A method for monitoring abnormal operation of goods in a dry coal shed

Abstract

The application discloses a kind of dry coal shed in cargo abnormal operation monitoring method, the method gathers the image information of measured object, constructs operation vehicle dataset, adds attention mechanism module on the basis of depth neural network model, improves the global modeling ability of model, improves the accuracy of real-time detection;The edge recognition and contour extraction of coal pile in image are carried out using image processing method, and then different coal piles are divided and numbered.According to the release instruction of customs and the movement of coal conveying belt, the change of operation vehicle corresponding coal pile is judged, and then it is judged whether there is irregular operation behavior, and an alarm is given for abnormal behavior, to achieve the purpose of real-time monitoring of dry coal shed interior operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of dry coal shed internal cargo supervision, in particular to using an image recognition method based on artificial intelligence to monitor abnormal cargo operations. BACKGROUND

[0002] According to the requirements of the General Administration of Customs, enterprises importing goods are not allowed to use and flow into the market before obtaining the customs release instruction in the customs supervision place. At present, after the enterprise imports coal into the supervision operation place, there is a lack of effective supervision means for un-released goods.

[0003] On the one hand, for traditional dry coal sheds, there is a lack of real-time monitoring method, and the customs cannot master the current state of the stockyard goods in real time, which leads to the inability to identify whether the enterprise has violated the rules and caused the un-released goods to flow into the market;

[0004] On the other hand, at the present stage, many dry coal sheds are increasing the number of cameras for real-time cargo state monitoring, but the customs supervision personnel need to check in real time, which consumes a lot of manpower. Sometimes, even if abnormal operations occur, these situations cannot be identified in a timely manner.

[0005] Therefore, considering the current development status of dry coal sheds, using video images for monitoring and combining artificial intelligence for automatic identification will help improve the accuracy and timeliness of customs supervision. SUMMARY

[0006] The present application uses image edge detection contour extraction technology and target detection technology to supervise the internal goods of dry coal sheds in view of the problems in the background art. The method has the following steps:

[0007] 1) Collect coal pile and operation vehicle image information through the camera fixed in the dry coal shed, and divide it into training set, validation set and data set according to the ratio of 8:1:1.

[0008] 2) Based on the coal pile image, the edge detection algorithm is used to obtain the points with strong pixel changes, and the contour information of the coal pile image is extracted through connectivity analysis.

[0009] 3) Analyze the coal pile contour, find the peak point through the sliding window method, and finally determine the position and number of the coal pile.

[0010] 4) Use the backbone network structure of the Yolov8 model, fuse the attention mechanism network based on the convolution module, add the CA attention mechanism module in each feature extraction block, and use the dynamic sample allocation strategy to select the optimal k positive samples according to the prediction class score and intersection over union weighted score t:

[0011] t=s α ×u β

[0012] Wherein s is the category score of each pixel point prediction, u is the regression score of the pixel point prediction frame and all target frames, and alpha and beta are weight hyperparameters, and the weighted score t is obtained by multiplying them, and the category classification loss and the frame regression loss are back propagated, and the coal mining operation vehicle network model is obtained through training.

[0013] 5) The network model is used for detecting the real-time collected dry coal shed internal image, and the coal mining operation vehicle is monitored in real time.

[0014] 6) After obtaining the position information of the coal mining operation vehicle, the coal pile position information is combined to determine the coal pile number corresponding to the operation vehicle. The frame difference method is used to determine whether the coal conveying belt is in a working state, whether the coal mining operation is being carried out, and if the threshold E th , it is determined that the coal mining operation is being carried out, otherwise step 5) is turned to.

[0015] 7) The current coal pile state corresponding to the operation vehicle is determined in combination with the release instruction issued by the customs, if the current coal pile state can be released, step 5) is turned to, otherwise an alarm is given and step 5) is turned to.

[0016] Beneficial effects:

[0017] In view of the problem that the current customs lacks effective real-time monitoring means for coal mining operation in the dry coal shed, the application provides a monitoring method for abnormal operation of goods in the dry coal shed, which adopts edge contour detection and target detection technology, can effectively identify the coal mining operation vehicle, can accurately alarm for the coal pile change in the abnormal operation period, and is convenient for the customs to timely control the coal mining operation in the dry coal shed, so as to effectively regulate the coal mining operation. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 It is a flow chart of the monitoring method for abnormal operation of goods in the dry coal shed of the application;

[0019] Figure 2 It is a coal pile image collected in the dry coal shed in the embodiment;

[0020] Figure 3 It is coal pile contour information after edge detection processing of the coal pile image in the embodiment;

[0021] Figure 4 It is a schematic diagram of the CA attention mechanism architecture in the model backbone network in the application;

[0022] Figure 5 It is a frame difference method schematic diagram for the coal conveying belt in the application. DETAILED DESCRIPTION

[0023] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The specific operational procedures illustrate the effectiveness of this method in monitoring abnormal operations within a dry coal shed. This embodiment is implemented based on the technical solution of the present invention, but the scope of protection of the present invention is not limited to the following embodiments.

[0024] This embodiment uses a specific enterprise as an example to illustrate the effectiveness and implementation process of the proposed method. Training, validation, and test sets are constructed using image data of the dry coal shed from March to May 2023. The effectiveness of the proposed method in detecting abnormal coal extraction operations is analyzed. The overall flowchart of the system is as follows: Figure 1 As shown.

[0025] The specific process is as follows:

[0026] A total of 25,000 historical images of the coal shed were acquired to form a coal extraction vehicle dataset. The dataset was divided into a training set of 20,000 images, a validation set of 2,500 images, and a test set of 2,500 images in an 8:1:1 ratio using the hold-out method.

[0027] To obtain image information of coal piles inside a dry coal shed, this example uses the Canny edge detection algorithm to identify points with strong pixel changes in the image, find the edge information of the coal pile, and extract the image contour of the coal pile through connectivity analysis.

[0028] Using the sliding window method, the extreme points of the profile curvature are obtained, thereby determining the peak points, and different coal piles are divided based on these peak points. Figure 2 For example, the camera captures coal piles in the left and right front areas of the dry coal shed. Using the Canny image processing algorithm, five coal piles are obtained, such as... Figure 3 As shown, and numbered accordingly.

[0029] After obtaining the image dataset of the work vehicle, we used Mosaic data augmentation, which involves randomly using four images, scaling them randomly, and then randomly distributing them for stitching. This enriched the work vehicle dataset, added many small objectives, and improved the robustness of the network.

[0030] Based on the Yolov8 model, a CA attention mechanism module is integrated into the backbone network, and multi-scale fusion training of the deep neural network is performed to obtain the optimized deep neural network model of the work vehicle.

[0031] In the Yolov8 model, the backbone network extracts the image features of the work car through convolution and pooling operations, and is divided into 5 layers P1-P5 in total, among which P3, P4 and P5 are the main three scales with sizes of 80*80, 40*40 and 20*20 respectively, which are used to focus on work car targets of different sizes in the image. At the same time, smaller scales are connected and fused with larger scales through upsampling and other processes to increase the number of feature channels and improve the recognition accuracy of work cars.

[0032] In view of the problems of weak light in the coal shed and wide movement range of the coal work car, the algorithm is improved in this case, and the global modeling capability is improved by fusing the attention mechanism module. The design idea of the attention mechanism module CA is as shown in Figure 4 After the residual network, the width and height of the input feature map are processed respectively, and the operation is performed from two dimensions instead of the previous single dimension, which better fuses the position information into the work car network.

[0033] In this embodiment, given an input feature map with a size of C*H*W=(256*80*80), the first step is to perform average pooling operation on the feature map in X and Y directions to obtain two new feature maps C*H*1=(256*80*1) and C*1*W=(256*1*80). The feature maps are connected and then input into a convolution block with a shared convolution kernel of 1*1 for Sigmoid operation. The feature vector D is obtained. 256 / 32×1×(80+80) Two convolutions are used to expand the feature map channels in X and Y directions to the same number as the input channel number, and the attention weights in the width and height directions are obtained through the Sigmoid function. After reassigning the weights of the input information, the final feature map is output as C*H*W=256*80*80.

[0034] After the three-scale output prediction result of the deep neural network, the class prediction result and the regression prediction result of 8400 anchors are obtained. According to the class prediction score and the regression prediction score of each anchor, a more fine selection strategy is performed based on the anchor inside the target box. Task-Aligned Assigner dynamic positive and negative sample distribution strategy is adopted, and the optimal k positive samples are selected according to the prediction class score and the intersection over union weighted score t:

[0035] t=s α ×u β

[0036] Where s is the class score predicted by each pixel point, u is the regression score of the pixel point prediction box and all target boxes, and a and β are weight hyperparameters. The weighted score t is obtained by multiplying them.

[0037] According to the category prediction score of each Anchor, and the ratio of the intersection and union between the predicted frame and the real target frame, the optimal k positive samples are obtained, k is 10 in this embodiment, to perform subsequent error loss calculation. The dynamic sample allocation method can make the model focus on the target faster, and finally obtain an optimized coal mining vehicle network model.

[0038] The real-time image is input into the optimized network model, the working vehicle can be recognized and positioned, and the corresponding coal pile number of the working vehicle can be obtained by combining the coal pile information. The frame difference method is used to judge the movement of the coal conveying belt, and the flowchart of the frame difference method is as shown in Figure 5 If the frame difference exceeds the threshold value E th = 10%, it indicates that the coal mining operation is being performed.

[0039] According to the instructions issued by the customs, it is determined which coal piles are in the operable state. If the current coal pile corresponding to the working vehicle is in the non-operable state, an abnormal operation alarm is performed. If the current coal pile corresponds to the number of coal piles that can be operated according to the instructions issued by the customs, it is determined that the operation is a normal operation, and the system does not perform an alarm. In order to verify the reliability of the method, the signal of the operable coal pile is manually changed, for example, the coal pile number is changed from No. 3 to No. 2, then the system determines that the current coal pile is not operable and alarms.

[0040] As can be seen from the above analysis, the monitoring method of the abnormal operation of the goods in the dry coal shed can effectively identify the coal mining vehicle, accurately alarm the change of the coal pile during the abnormal operation period, and facilitate the customs to timely control the coal mining operation in the dry coal shed, so as to effectively regulate the coal mining operation.

Claims

1. A method for monitoring abnormal operations of goods in a dry coal shed, characterized in that... Using contour extraction and target detection technology, irregular operations are identified through the following steps: 1) Collect images of coal piles and work vehicles using cameras fixed inside the dry coal shed, and construct a work vehicle image dataset; 2) Based on the coal pile image, the coal pile outline is extracted using an edge detection method; 3) Identify and number each coal pile based on the turning points of its outline; 4) An attention mechanism is integrated into the backbone network of the YOLO model, a dynamic sample allocation strategy is adopted, the category and regression losses are calculated, and the network parameters are trained to obtain the coal mining vehicle model. 5) Input the real-time images collected inside the dry coal shed into the network model to obtain the real-time location of the operating vehicles; 6) After determining the coal pile corresponding to the coal taking vehicle, use the frame difference method to determine whether the conveyor belt is in the coal transport state, and know the operating status of the corresponding coal pile. 7) If the corresponding coal pile is in operation but has not received a release order from customs, the system will alarm; otherwise, return to step 6.

2. The monitoring method for abnormal operations of goods in a dry coal shed according to claim 1, characterized in that... The image dataset of the work vehicle described in step 1) adopts a leave-out ratio of 8:1:

1.

3. The method for monitoring abnormal operations of goods in a dry coal shed according to claim 1, characterized in that... Step 2) describes the edge detection method as the Canny algorithm. After obtaining the edge information, connectivity analysis is performed to obtain the contour edge information of the coal pile.

4. The method for monitoring abnormal operations of goods in a dry coal shed according to claim 1, characterized in that... Step 3) Using the sliding window method, the extreme points of the edge contour curvature of the coal pile are obtained, and then the peak points are determined. Different coal piles are divided according to the peak points.

5. The method for monitoring abnormal operations of goods in a dry coal shed according to claim 1, characterized in that... Step 4) The deep learning model used is the Yolov8 model for object detection.

6. A method for monitoring abnormal operations of goods in a dry coal shed according to claim 5, characterized in that... Step 4) Based on the Yolov8 model, the CA attention mechanism is integrated, and the dynamic positive and negative sample allocation combines the class score and the crossover ratio score to select the optimal k positive samples.

7. A method for monitoring abnormal operations of goods in a dry coal shed according to claim 6, characterized in that... The CA attention mechanism is as follows: Given an input feature map of size C*H*W, where C represents the number of channels in the feature map, H represents the height of the feature map, and W represents the width of the feature map; the first step is to perform average pooling on the feature map in the X and Y directions to obtain two new feature maps C*H*1 and C*1*W. These feature maps are then concatenated and subsequently fed into a 1*1 convolutional block sharing a convolutional kernel for a sigmoid operation to obtain the feature vector D∈R. 256 / 32×1×(H+W) The feature map channels in the X and Y directions are expanded to the same number as the input channels by two convolutions. The attention weights in the width and height directions are obtained by the Sigmoid function. After redistributing the weights with the input information, the final output feature map is C*H*W=256*H*W. The three-scale output prediction results of the deep neural network yield the category prediction results and regression prediction results of the anchors. Based on the category prediction score and regression prediction score of each anchor, a selection strategy is applied to the anchors within the target bounding box. A Task-Aligned Assigner dynamic positive and negative sample allocation strategy is used to select the k optimal positive samples based on the predicted category score and the cross-union ratio (CUI) weighted score t. t=s α ×u β Where s is the predicted class score for each pixel, u is the regression score of the pixel's predicted bounding box with all target boxes, and α and β are weight hyperparameters, which are multiplied together to obtain the weighted score t. The optimal k positive samples are selected based on the category prediction score of each anchor and the ratio of the intersection and union between the predicted bounding box and the ground truth target bounding box. These samples are then used for subsequent error loss calculation. The dynamic allocation of samples allows the model to focus on the target more quickly, ultimately resulting in an optimized coal mining vehicle network model.

8. A method for monitoring abnormal operations of goods in a dry coal shed according to claim 6 or 7, characterized in that... k＝10。 9. A method for monitoring abnormal operations of goods in a dry coal shed according to claim 1, characterized in that... Step 6) Based on the changes in the coal conveyor belt before and after the video frames, if the frame value difference is greater than the frame value difference threshold E in step 6) th If so, it means that the coal pile is in operation.

10. A method for monitoring abnormal operations of goods in a dry coal shed according to claim 9, characterized in that... Step frame value difference threshold E th Take 5-10%.

Images