Pipeline peripheral mechanical equipment identification and dangerous behavior detection method

Abstract

The application discloses a pipeline periphery mechanical equipment identification and dangerous behavior detection method, relates to the technical field of mechanical equipment identification, and aims to solve the problems that the existing methods are trained and tested based on public data sets, are not practically applied to engineering mechanical detection, are not fully functional, and cannot be completely applicable to large-scale installed engineering sites, and the technical scheme is as follows: S1, model establishment, an NNIE development package is preinstalled in Hi3559A, and model development and training are performed on a PC end before system running; S2, model transplantation, the trained model is transplanted to the Hi3559A, development is performed on the Hi3559A, and the support of the PC end is no longer needed; S3, system running, the Hi3559A is run according to the following steps to detect and identify the whole, and dangerous postures and behaviors are immediately alarmed. The effect of risk prompt and large-scale installation convenience is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical equipment identification, in particular to a pipeline periphery mechanical equipment identification and dangerous behavior detection method. BACKGROUND

[0002] The West-East Gas Pipeline Project traverses nine provinces and cities from east to west in China, passes through several major water systems, deserts, the Loess Plateau, the Taihang Mountains and the Jiangnan Water Network, and crosses above-grade highways, railways and large and medium-sized rivers 704 times, with a wide engineering front, long construction period, strong technicality and complex process.

[0003] Engineering machinery is an indispensable part of the West-East Gas Pipeline Project, and the most commonly used engineering machinery in the West-East Gas Pipeline Project includes excavating machinery, shovel loading machinery, compaction machinery, rock drilling machinery and pile driving machinery.

[0004] The prior art in the above has the following defects: these methods are based on public data sets for training and testing, and have not been actually applied to engineering machinery detection, and these methods cannot determine whether the region where the detected target object is located has a dangerous posture and behavior, and the overall functionality is not comprehensive, and cannot be completely applied to large-scale installation in the engineering site. SUMMARY

[0005] The purpose of the present application is to provide a pipeline periphery mechanical equipment identification and dangerous behavior detection method that does not require PC support during system operation and can be installed on a large scale in the engineering site.

[0006] To achieve the above purpose, the present application provides the following technical scheme:

[0007] A pipeline periphery mechanical equipment identification and dangerous behavior detection method, the steps of which are as follows:

[0008] S1: model establishment, the NNIE development package is pre-installed in Hi3559A, and model development and training work is performed on the PC side before system operation;

[0009] A1: the original engineering machinery pictures obtained on the construction site are cleaned and classified;

[0010] A2: the pictures in the data set are uniformly scaled to 800*800, and the scaled pictures are labeled;

[0011] A3: Obtain background pictures without construction machinery on the Internet, label and add to the dataset;

[0012] A4: Use multiple internal pictures in the dataset to piece together a single picture containing construction machinery, label and add to the dataset;

[0013] A5: Divide the dataset into training set, test set and validation set, the division ratio is 9:1:1, and data augmentation is performed on the data in the training set;

[0014] A6: Build a backbone network for target detection;

[0015] A7: Build a neural network model based on the SSD algorithm;

[0016] A8: Train the neural network, and perform validation during training. When the training loss and validation loss are lowest, save the optimal model of network training;

[0017] A9: Test the test set using the trained model to obtain the detection result of the construction machinery picture;

[0018] S2: Model transplantation: transplant the trained model to Hi3559A, develop on Hi3559A, and no longer need PC support;

[0019] S3: Run the system, and run Hi3559A to detect and identify the whole according to the following steps:

[0020] B1: Convert the detection model trained on the PC (.caffemodel format) to.wk format;

[0021] B2: Extract video frames and decode from the video stream file obtained from the camera by the VDEC module;

[0022] B3: The network file (i.e..wk file) provided by Sample of Hi provides only supports bgr format pictures. The.prototxt file corresponding to the wk file sets the input picture color space as BGR format. Each time a video frame or single picture is put in, it needs to be converted to bgr format;

[0023] B4: Pass the bgr format video frame through the VPSS module to the NNIE target recognition part for detection (.wk format VGG16 network model), and output the position and category of the detected target object;

[0024] B5: Calculate the dense optical flow of each pixel in the current frame combined with the next adjacent frame, and represent the motion direction and motion speed of the pixel with color and brightness respectively to obtain the optical flow image;

[0025] B6: convert the obtained optical flow image from HSV color space to RGB space;

[0026] B7: filter noise and remove small regions within the target area;

[0027] B8: count the range of dangerous postures and behaviors corresponding to the identified object categories within the target area (ignore other areas to reduce noise pollution);

[0028] B9: immediately alarm when dangerous postures and behaviors are found.

[0029] Further, the pictures of the original engineering machinery in A1 are divided into five categories: excavating machinery, shovel machinery, rock drilling machinery, compaction machinery and pile driving machinery.

[0030] Further, the background pictures without engineering machinery obtained on the Internet in A3 are used to generate corresponding.xml files by a python program and add them to the data set.

[0031] Further, the data enhancement method in A5 includes image mirroring, Gaussian blur, random rotation and random removal of some pixel points.

[0032] Further, the backbone of the SSD in A7 is MobileNet.

[0033] In summary, the beneficial technical effects of the present application are:

[0034] 1. The pipeline peripheral mechanical equipment recognition and dangerous behavior detection method, the system runs on Hi3559A, first detects the position and category of mechanical equipment in the video frame by SSD, then detects whether there is dangerous posture and behavior in the target object area by optical flow method, and generates a risk prompt effect.

[0035] 2. The pipeline peripheral mechanical equipment recognition and dangerous behavior detection method develops a set of independent engineering machinery vehicle detection and dangerous behavior recognition system, the system does not need PC support when running, can be installed in engineering site on a large scale, and has the effect of convenient large-scale installation. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The figure is a work flow diagram of the present application. DETAILED DESCRIPTION

[0037] The method of the present application will be further described in detail below with reference to the accompanying drawings.

[0038] Referring to Figure 1 A pipeline peripheral mechanical equipment recognition and dangerous behavior detection method, the steps are as follows:

[0039] The NNIE development package is preinstalled in Hi3559A. NNIE is the abbreviation of Neural Network Inference Engine, which is a hardware unit in the HiMedia SoC specially designed for neural network, especially deep learning convolutional neural network, and supports most of the existing public networks, such as Alexnet, VGG16, Googlenet, Resnet18, Resnet50, etc. Classification network, Faster R-CNN, YOLO, SSD, RFCN, etc. Detection network, and SegNet, FCN, etc. Scene segmentation network, the application develops a set of systems that can independently work on the detection pipeline of mechanical equipment and can alarm the dangerous behavior generated by the mechanical equipment;

[0040] Before developing on Hi3559A, the model development and training on the PC end are carried out, and the specific work is as follows:

[0041] S1: model establishment, the NNIE development package is preinstalled in Hi3559A, and the model development and training on the PC end are carried out before the system runs;

[0042] A1: data preparation and processing, the original engineering machinery picture obtained on the construction site is cleaned, and is classified according to the corresponding mechanical category, a total of five categories: excavating machinery, shovel loader, rock drilling machinery, compaction machinery, pile driving machinery, etc.;

[0043] A2: the pictures in the data set are uniformly scaled to 800*800, and the scaled pictures are labeled to obtain the labeled.xml annotation file;

[0044] A3: obtain the background picture without engineering machinery on the network, and generate the corresponding.xml file by using the python program, and add the data set;

[0045] A4: using a python program to generate 400 single pictures containing engineering machinery by splicing multiple pictures in the data set, and generating the corresponding.xml file to add to the data set;

[0046] A5: divide the data set into training set, test set and validation set, the division ratio is 9:1:1, and the data in the training set is subjected to data enhancement, the data enhancement methods are: image mirroring, Gaussian blur, random rotation and random removal of some pixels;

[0047] A6: build the backbone network of target detection, the backbone network of SSD used in the application is MobileNet;

[0048] A7: Build a neural network model based on the SSD algorithm. Use MobileNet as the backbone network of SSD to make the original network more lightweight and improve the detection speed of the network;

[0049] A8: Train the neural network using the back propagation algorithm. Verify the network during training, and save the optimal model of the network training when the training loss and validation loss are lowest. The network training is performed using Nvidia GPU;

[0050] A9: Test the test set using the saved optimal model to obtain the detection result of the engineering machinery picture;

[0051] The above network structure is built using the Caffe deep learning library. The programming language used is Python. In this embodiment, the convolutional layer, depth separable convolutional layer, fully connected layer, Layer and other convolutional neural network structures and internal operations

[0052] S2: Model transplantation: transplant the trained model to Hi3559A, and develop on Hi3559A, without the need for PC support;

[0053] S3: Run the system, and run the Hi3559A to detect and identify the whole according to the following steps:

[0054] B1: Convert the PC-side trained detection model (.caffemodel format) to.wk format. Since the NNIE supporting software and tool chain only supports the Caffe framework, network models using other frameworks need to be converted to models under the Caffe framework. RuyiStudio provided in the NNIE SVP component package converts the caffe model (.caffemodel format) to.wk format;

[0055] B2: The VDEC module obtains a video stream file from the camera and extracts video frames for decoding;

[0056] B3: The Sample network file (i.e.,.wk file) provided by HiSilicon supports only bgr format pictures. The.prototxt file corresponding to the wk file sets the input picture color space to BGR format. Each time a video frame or a single picture is put in, it needs to be converted to bgr format;

[0057] B4: The bgr format video frame is transmitted to the NNIE target recognition part through the VPSS module for detection (.wk format VGG16 network model), and the position and category of the detected target object are output;

[0058] B5: Calculate the dense optical flow for each pixel of the current frame combined with the next adjacent frame, and get the optical flow image, which represents the motion direction and speed of each pixel by color and brightness respectively. The input of the optical flow algorithm: two consecutive RGB images of size h (height) * w (width) * 3, or one gray image of size h * w. The output of the optical flow algorithm: one optical flow field of size h * w * 2, where each pixel value represents the displacement of the pixel in the x and y directions of the input frame. In order to represent the optical flow field more intuitively, we can convert the optical flow field of size h * w * 2 into an HSV image of size h * w * 3, where the H channel (color) represents the direction of movement of the pixel, and the S or V channel (brightness) represents the speed of movement of the pixel.

[0059] B6: Convert the optical flow image obtained in B5 from the HSV color space to the RGB space.

[0060] B7: Filter out small regions within the target area, and remove regions with less than 300 pixels.

[0061] B8: Count the range of dangerous postures and behaviors corresponding to the identified object categories within the target area (ignore other areas to reduce noise pollution), such as the upward and downward movement of excavators, and the left and right movement of compactors.

[0062] B9: Immediately alert when a dangerous posture or behavior is detected.

[0063] The embodiments of the present specific implementation are the preferred embodiments of the present application, but do not limit the protection scope of the present application, therefore: any equivalent changes made in accordance with the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. A pipeline perimeter mechanical equipment identification and dangerous behavior detection method, characterized in that, The steps are as follows: S1: model building, the NNIE development package is pre-installed in Hi3559A, and the model development and training work is carried out on the PC side before the system runs; A1: the original engineering machinery pictures obtained on the construction site are cleaned and classified; A2: the pictures in the data set are uniformly scaled to 800X800, and the scaled pictures are labeled; A3: obtain background pictures without engineering machinery on the network, label and add to the data set; A4: use multiple data set internal pictures to splice into a single picture containing engineering machinery, label and add to the data set; A5: divide the data set into training set, test set and validation set, the division ratio is 9:1:1, and data enhancement is carried out on the data in the training set; A6: build the backbone network of target detection; A7: build a neural network model based on SSD algorithm; A8: train the neural network, and save the optimal model of network training when the training loss and validation loss are lowest during training; A9: test the test set using the trained model, and the detection result of the engineering machinery picture can be obtained; S2: model transplantation: transplant the trained model to Hi3559A, develop on Hi3559A, and no longer need PC support; S3: run the system, and run the whole detection and recognition on Hi3559A according to the following steps: B1: convert the detection model trained on the PC into.wk format; B2: the VDEC module obtains video stream files from the camera and extracts video frames for decoding; B3: the network file provided by Sample of Hi provides only bgr format pictures, the.prototxt file corresponding to the wk file sets the color space of the input picture as BGR format, and the bgr format is converted every time the video frame or single picture is put in; B4: the bgr format video frame is transmitted to the NNIE target recognition part through the VPSS module for detection, and the position and category of the detected target object are output; B5: calculate the dense optical flow of each pixel in the current frame combined with the next adjacent frame, and represent the motion direction and motion speed of the pixel by color and brightness respectively to obtain the optical flow image; B6: convert the obtained optical flow image from HSV color space to RGB space; B7: filter noise and remove small areas in the target area; B8: count the range of dangerous postures and behaviors corresponding to the recognized object categories in the target area; B9: alarm immediately when dangerous postures and behaviors are found.

2. The method of claim 1, wherein: The original engineering machinery pictures in A1 are divided into five categories: excavating machinery, shovel loading machinery, rock drilling machinery, compaction machinery and pile driving machinery.

3. The method of claim 1, wherein: In A3, the background pictures without engineering machinery obtained on the network are used to generate corresponding.xml files by python program and added to the data set.

4. The method of claim 1, wherein: In A5, the data enhancement methods include image mirroring, Gaussian blur, random rotation and random removal of some pixels.

5. The method of claim 1, wherein: In A7, the backbone of SSD is MobileNet.

Images