A slow device running state detection method, an electronic device, and a storage medium

Abstract

A slow device running state detection method, an electronic device and a storage medium, a slow device running state detection model is constructed, and real-time pictures monitored and photographed are read into the model for detection; wherein, the model collects monitoring videos, obtains slow device pictures in the videos as training sample data, replaces the C3 module of the backbone part in yolov5s with a CBAM module customized by the applicant for training, obtains a detection model; two pictures separated by a certain time are transmitted into the model for detection, the center point coordinates of the detection frame of the slow device in the two pictures are obtained, and the relative movement distance of the slow device in the time period is calculated through the center point coordinates, by comparing the distance value with the threshold value, it is judged whether the device is in normal operation or abnormal stop. The present application does not need to observe and judge the running state of the slow device in the monitoring picture artificially, thereby avoiding the safety hidden danger caused by misjudgment, increasing safety while reducing the cost of manual work.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to slow device motion state detection, in particular to a slow device running state detection method, an electronic device and a storage medium. BACKGROUND

[0002] Compared with general running devices, slow devices run very slowly, for example, a trolley in a sewage treatment plant, which is mainly responsible for discharging sludge outside the pool. The running state of the trolley directly affects the effect of the sewage treatment process and the safety of the treatment plant. Under normal circumstances, the moving distance per minute does not exceed 2 meters. Even in the normal running state, it is very time-consuming for a person to judge whether the device is in a normal running state or an abnormal stop state through monitoring video. In addition, due to the visual persistence of the human eye, it is easy to make a mistake. Because the running environment of the slow device is generally complex, some algorithms that perform well on the coco data set, such as the DETR algorithm based on the Transformer, the unmodified yolo series algorithm, and the anchor-free single-stage target detection algorithm FCOS, do not have good detection effects on slow devices in a complex background. Some existing methods, such as tracking algorithms, are also difficult to determine the running state of an object with very small motion amplitude and capture subtle motion changes, resulting in inaccurate data and thus the failure of the device to be discovered. Therefore, it is necessary to develop a slow device running state detection method to accurately detect the running state of the slow device. SUMMARY

[0003] The problem to be solved by the application is that the existing methods cannot accurately detect the running state of the device due to the slow running speed and small motion amplitude of the slow device.

[0004] The technical scheme of the application is a slow device running state detection method, which constructs a detection model to detect the running state of slow devices in monitoring pictures taken by a monitoring camera at a set time interval. The detection method comprises the following steps:

[0005] Step 1: obtaining slow device images collected by a monitoring camera in the past;

[0006] Step 2: training a slow device running state detection model using a data set obtained by labeling the pictures collected in Step 1, taking yolov5s as a basic detection network, and integrating a CBAM attention mechanism with a sinusoidal position encoding and a deformable convolution module to form a slow device detection model;

[0007] The slow device state detection model specifically comprises:

[0008] Step2.1: Construct a channel attention mechanism module, perform global average pooling and global maximum pooling on the input feature map to obtain two description vectors representing the average value and maximum value, respectively. After nonlinear transformation through a 1*1 convolution layer and a ReLU activation function, restore the vectors to the original channel dimension through a 1*1 convolution layer. Add the two nonlinearly transformed vectors, then generate channel attention weights through a Sigmoid function. Multiply the channel attention weights with the original feature map element by element to obtain the channel attention weighted feature map F CA ;

[0009] Step2.2: Perform a sinusoidal position encoding PE(x,y) operation on the feature map F CA obtained in Step2.1 to introduce position information for each spatial position (x,y). Add F CA and the sinusoidal position encoding result element by element to obtain the position encoding enhanced feature map F CA_PE (x,y);

[0010] Step2.3: Construct a deformable convolution module, which includes a 3*3 convolution layer to generate offsets, with an output channel number of 18, representing the horizontal and vertical offsets at each convolution kernel position, a 3*3 convolution layer to generate a weight mask at each position, and a Sigmoid activation function to ensure the weight is between 0 and 1, and an actual 3*3 convolution layer to adjust the position of the sampling point according to the generated offset and adjust the weight at each position according to the generated mask for convolution operation.

[0011] Step2.4: Construct a spatial attention mechanism module, perform average pooling and maximum pooling on the position encoding enhanced feature map F CA_PE in the channel dimension to obtain two spatial description feature maps. Concatenate the two spatial description feature maps in the channel dimension to form a new feature map. Apply the spatial attention weight generated by the deformable convolution module constructed in Step2.3 through the Sigmoid activation function to the position encoding enhanced feature map F CA_PE to obtain the final CBAM weighted feature map.

[0012] Step2.5: Change the network structure, replace the C3 module in the backbone of yolov5s with the custom CBAM module obtained in Step2.1-Step2.4 to obtain a slow device state detection model.

[0013] Step3: using the trained slow device state detection model to detect the slow device target in the two pictures taken by the monitoring camera at a set time interval;

[0014] Step4: obtaining the detection result, getting the center point coordinates of the slow device detection frame, and calculating the distance between the center point coordinates of the slow device in the front and rear two pictures to get the moving distance;

[0015] Step5: comparing the moving distance with the set threshold to get the running state of the slow device.

[0016] The application also provides an electronic device, which comprises a processor and a memory, and the memory stores at least one instruction or at least one program, which is loaded and executed by the processor to realize the slow device running state detection method.

[0017] The application also provides a computer readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or program is executed to realize the slow device running state detection method.

[0018] The slow device has the characteristics of slow running speed, small and subtle movement amplitude, for example, the trolley in a sewage treatment plant, which runs less than 2 meters per minute. If the slow device video collected by monitoring is checked manually, it is not easy to find the subtle position change of the slow device, and it is difficult to distinguish whether the slow device is in normal operation or abnormal stop in a short time, and it is easy to cause visual fatigue state due to long-time observation of the slow device on the screen. Therefore, the application aims at the above problems, first identifies the slow device according to the pictures taken by the monitoring camera, and then calculates the relative moving distance by the center point coordinates of the same slow device detection frame in the front and rear two pictures, and judges the running state of the slow device by comparing the distance and the threshold value. This can solve the problem that manual observation is not easy to find the subtle position change, and in the experiment, only the pictures taken at an interval of 5 seconds are needed to accurately judge the running state of the trolley, which not only improves the detection accuracy compared with manual observation, but also improves the detection efficiency and reduces the manual burden.

[0019] Compared with the prior art, the application has the following advantages:

[0020] (1) The existing yolov5s detection model combined with the CBAM attention mechanism is limited to a fixed convolution kernel shape, and the processing ability of the complex background is weak, and the target detection in the complex scene is general, and the C3 module of the backbone part in the yolov5s is replaced with a self-defined CBAM module introducing sinusoidal position encoding and deformable convolution. The sinusoidal position encoding injects explicit spatial position information into each feature point, helping the model better understand the global and local relationships in the image, improving the feature positioning ability of the model in complex scenes, and improving the accuracy of target positioning and recognition in complex backgrounds. In the spatial attention mechanism, deformable convolution is introduced to dynamically adjust the sampling position and weight of the convolution kernel, pay more attention to the target area, ignore background noise, improve the accuracy and stability of detection, and reduce the false detection and missed detection. The present application has stronger flexibility and robustness in processing complex background target detection tasks, overcomes the problem of poor detection effect caused by slow equipment running in complex background, and improves the detection accuracy.

[0021] (2) The present application overcomes the problem that the tracking algorithm is difficult to determine the running state of the object with small motion amplitude, and can accurately determine the running state of the slow equipment, whether it is in normal operation or abnormal stop.

[0022] (3) The problem of low efficiency and low accuracy of artificial observation of slow equipment running state in monitoring video is solved, and the problem of low sensitivity of artificial perception of the subtle position change of slow equipment is overcome, the determination accuracy is improved, the labor is saved, and the safety is increased. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The flow chart of the slow equipment running state detection method of the present application.

[0024] Figure 2 The CBAM module network structure diagram of the present application adding deformable convolution.

[0025] Figure 3 The detection schematic diagram of the present application. DETAILED DESCRIPTION

[0026] The present application provides a slow equipment running state detection method. The present application solves the problem of low efficiency, high cost and high misjudgment rate of artificial observation of slow equipment running state, and has strong practicality, saves labor, improves determination accuracy and equipment safety.

[0027] The application obtains a monitoring picture taken by a monitoring camera, and uses a yolov5s network with a CBAM attention mechanism integrated with a deformable convolution module to detect a slow device target in the picture, to obtain a slow device target detection result and a center point coordinate of a detection frame, calculate a relative movement distance from the center point coordinates of the same slow device detection frame in the two pictures, and determine the running state of the slow device by comparing the relative movement distance with a threshold value.

[0028] As shown in Figure 1 , the specific implementation process of the application is as follows.

[0029] Step1: Obtain the image of the slow device collected by the monitoring camera.

[0030] Step2: Improve the yolov5s network model, and obtain a slow device target detection model through training. Use the data set obtained by labeling the pictures collected in Step1 to train the slow device running state detection model, use yolov5s as the basic detection network, integrate the CBAM attention mechanism with sinusoidal position encoding and a deformable convolution module to form the slow device detection model. Since the running environment of the slow device is generally complex, the existing CBAM module spatial attention mechanism weights the spatial information of the feature map through fixed position convolution operation, and has weak processing ability for complex background targets, therefore, sinusoidal position encoding is added to inject explicit spatial position information to each feature point, to improve the feature positioning ability of the model in a complex scene, and improve the positioning and detection accuracy of the target in a complex background. A deformable convolution is added, which can more flexibly focus on the target area and ignore background noise, to improve the detection precision and stability. The specific implementation is as follows.

[0031] Step2.1: Define a channel attention mechanism module, perform global average pooling and global maximum pooling on the input feature map to obtain two description vectors representing the average value and the maximum value, respectively, perform nonlinear transformation on the two vectors through a 1*1 convolution layer and a ReLU activation function, and then restore the vectors to the original channel dimension through a 1*1 convolution layer, add the two nonlinearly transformed vectors, and then generate a channel attention weight through a Sigmoid function, multiply the channel attention weight with the original feature map element by element to obtain a weighted feature map F CA .

[0032] Step2.2: Perform sinusoidal position encoding on the feature map obtained in Step2.1 to introduce position information to each spatial position (x, y), and the sinusoidal position encoding formula is as follows:

[0033] When i is even:

[0034]

[0035] When i is odd:

[0036]

[0037] where i is the index of the channel, and C is the total number of channels of the feature map. The feature map obtained in Step 2.1 is element-wise added with the sinusoidal position encoding to obtain a feature map enhanced by position encoding;

[0038] F CA_PE (x,y)=F CA (x,y)+PE(x,y).

[0039] Step 2.3: Define a deformable convolution module. Since the present application adopts a 3*3 deformable convolution layer, there are two offset directions (x, y) at each position. Therefore, the module uses a 3*3 convolution layer to generate the offset, with an output channel number of 18. A 3*3 convolution layer generates the weight mask of each position, and the mask is activated by a Sigmoid function to ensure that the weight is between 0 and 1. An actual 3*3 convolution layer adjusts the position of the sampling point according to the generated offset, and adjusts the weight of each position according to the generated mask, and performs convolution operation.

[0040] Step 2.4: Define a spatial attention mechanism module. The feature map F CA_PE is enhanced by position encoding. Average pooling and maximum pooling are performed in the channel dimension to obtain two spatial description feature maps. The two feature maps are spliced in the channel dimension to form a new feature map. The new feature map is input into the 3*3 deformable convolution module to obtain the offset offset and the mask mask through the offset generation convolution layer and the mask generation convolution layer, respectively. Then, the input feature map, offset and mask are input into the actual convolution layer to perform convolution operation. The result is activated by a Sigmoid function to generate a spatial attention weight. Finally, the spatial attention weight is applied to the feature map F CA_PE enhanced by position encoding to obtain the final CBAM weighted feature map.

[0041] Step 2.5: Combine the channel attention mechanism module, the sinusoidal position encoder, and the spatial attention mechanism module in Step 2.1 to Step 2.4 to obtain a custom CBAM module. Replace the C3 module of the backbone part in yolov5s to obtain a slow device state detection model.

[0042] The image collected in Step 1 is labeled with a rectangular frame to form a data set, the learning rate and the number of iterations are set, and the data set is used to train the yolov5s network with a fusion self-defined CBAM module to obtain a trained slow device running state detection model.

[0043] Step3: Using the trained slow device state detection model, input two pictures separated by a certain time into the model, the certain time is determined by the specific device running speed, generally 5s, and obtain the center point coordinates of the slow device detection frame in the two pictures.

[0044] Step4: After obtaining the center point coordinates of the slow device detection frame, calculate the distance between the center point coordinates of the slow device in the front and back two pictures to obtain the moving distance. The present application can further be used in the scene where multiple devices exist in a single picture, that is, the target detection of Step3 obtains m detection results, and the relative moving distance of the slow device is obtained from the center point coordinates of the detection frame of each detection target. The specific implementation is as follows.

[0045] The center point coordinates of device 1 in the first picture are (x1, y1), and the center point coordinates of device 1 in the second picture are (X1, Y1); the center point coordinates of device 2 in the first picture are (x2, y2), and the center point coordinates of device 2 in the second picture are (X2, Y2); similarly, the center point coordinates of device m in the first picture are (xm, ym), and the center point coordinates of device m in the second picture are (Xm, Ym); and the center point coordinates of the same device in the front and back two pictures are calculated to obtain the relative moving distance of the ith device i .

[0046]

[0047] Step5: Compare the relative moving distance Δdi with the threshold value T, if Δdi ≥ T, then determine that device i is in normal running state, otherwise determine that it is abnormal stop. The setting method of threshold value T is: one-time send the collected n pictures of slow devices in normal running state under a certain time interval in a continuous time into the model for detection, obtain the relative moving distance of the slow device in each two pictures i , n is generally 100, and the average value and the standard deviation σ of the relative moving distance are calculated respectively:

[0048]

[0049] The threshold value T is calculated from the average value and the standard deviation σ:

[0050]

[0051] Because the slow device runs slowly, the motion amplitude is small, if the artificial observation monitoring video is used to judge the running state of the slow device, it is difficult to make accurate judgment on whether the slow device produces displacement in a short time, misjudgment is easy to appear, the high error rate brings safety problems and increases the burden of artificial, therefore the technology of deep learning is used to judge. The general tracking algorithm not only requires high computing power, but also is difficult to determine the running state of the object with very small motion amplitude and capture subtle motion changes, resulting in inaccurate data, so that the equipment failure cannot be found. The method based on target detection is used in the present application. Generally, the running environment of the slow device is relatively complex, which brings certain difficulties to the target detection of the slow device. The existing target detection algorithm is limited by the fixed convolution kernel shape, and the processing ability for complex background is weak, and the performance of target detection in complex scenes is general. Therefore, the yoloV5s network is improved in the present application, the sinusoidal position coding is introduced into the CBAM module to inject clear spatial position information for each feature point, the accuracy of positioning and recognition of the target in the complex background is improved, the deformable convolution is introduced, the spatial attention mechanism pays more attention to the target area flexibly, ignores the background interference, and the improved yoloV5s network can focus on the target area more accurately, so that the effect of target detection is improved. On the basis of accurately obtaining the target detection result, the relative moving distance of the slow device is calculated through the center point coordinates of the slow device detection frame in two pictures separated by a period of time, the size of the distance and the set threshold value is compared, and the running state of the slow device is calculated.

[0052] Figure 2 The network structure diagram of the CBAM module with added sinusoidal position coding and deformable convolution.

[0053] Figure 3 The detection process of an embodiment of the present application is shown, the slow device in the figure is a travelling crane of a sewage treatment plant, which shows the process of detecting the slow device using the yoloV5s fused with the custom CBAM module, calculating the relative moving distance of the slow device, judging, and obtaining the running state of the slow device, and the detection result is that the device is normally running.

[0054] The embodiment of the present application also provides an electronic device, including a processor and a memory, at least one instruction or at least one program is stored in the memory, the at least one instruction or the at least one program is loaded and executed by the processor, the slow device running state detection model described in the above embodiment is realized, and the slow device image photographed by the monitoring camera is received for running state detection.

[0055] The embodiment of the present application also provides a computer readable storage medium storing a computer program, wherein the computer program realizes the slow device running state detection model when executed by a processor of a computer. The program code for implementing the method of the present application can be written in any combination of one or more programming languages. The program code can be executed completely on a machine, partially on a machine, partially on a machine as a separate software package and partially on a remote machine, or completely on a remote machine or server.

Claims

1. A method of detecting a slow device operating state, characterized by The detection model is constructed to detect the running state of the slow equipment in the monitoring pictures taken by the monitoring camera at a set time interval. The detection method comprises the following steps: Step 1: obtaining the slow equipment images collected by the monitoring camera in history; Step 2: training the slow equipment running state detection model using the dataset obtained by labeling the images collected in Step 1, taking yolov5s as the basic detection network, and integrating the CBAM attention mechanism with added sinusoidal position encoding and deformable convolution module to form the slow equipment detection model; The slow equipment state detection model is specifically: Step2.1: Constructing the channel attention mechanism module, performing global average pooling and global maximum pooling on the input feature map to obtain two description vectors representing the average value and maximum value, respectively. After nonlinear transformation through a 1*1 convolution layer and a ReLU activation function, the vectors are restored to the original channel dimension through a 1*1 convolution layer. The two nonlinearly transformed vectors are added together, and then the channel attention weight is generated through a Sigmoid function. The channel attention weight is multiplied element by element with the original feature map to obtain the channel attention weighted feature map F CA ; Step2.2: Position encoding PE(x,y) operation is performed on the feature map F CA obtained in Step 2.1 to introduce position information for each spatial position (x,y) to F CA (x,y) by element-wise addition, to obtain the position-encoding-enhanced feature map F CA_PE (x,y). Step 2.3: constructing a deformable convolution module, which includes a 3*3 convolution layer for generating offset, with an output channel number of 18, representing the horizontal and vertical offset at each convolution kernel position, a 3*3 convolution layer for generating a weight mask at each position, the mask being activated by a Sigmoid function to ensure that the weight is between 0 and 1, and an actual 3*3 convolution layer for adjusting the position of the sampling point according to the generated offset and adjusting the weight at each position according to the generated mask for convolution operation; Step 2.4: constructing a spatial attention mechanism module, the feature map F enhanced by position encoding CA_PE The average pooling and maximum pooling are performed in the channel dimension to obtain two spatial description feature maps. The two spatial description feature maps are spliced in the channel dimension to form a new feature map. The new feature map is input into the deformable convolution module constructed in Step 2.3, and then is input into a Sigmoid activation function to generate a spatial attention weight. Finally, the spatial attention weight is applied to the feature map F enhanced by position encoding CA_PE The final CBAM weighted feature map is obtained. Step 2.5: changing the network structure, replacing the C3 module in the backbone of yolov5s with the custom CBAM module obtained in Step 2.1-Step 2.4 to obtain the slow equipment state detection model; Step 3: using the trained slow equipment state detection model to detect the slow equipment target in two pictures taken by the monitoring camera at a set time interval; Step 4: obtaining the detection result to obtain the center point coordinates of the slow equipment detection box, and calculating the distance between the center point coordinates of the slow equipment in the front and rear two pictures to obtain the moving distance; Step 5: comparing the moving distance with the set threshold to obtain the running state of the slow equipment.

2. The method of claim 1, wherein the method further comprises In Step 2, the images collected in Step 1 are labeled with a rectangular frame to mark the slow equipment in the image, which is put into the dataset as a label, and the learning rate and iteration number parameters are set. The improved yolov5s network is trained with the dataset to obtain the slow equipment target detection model for detecting the slow equipment in the pictures input by the monitoring camera.

3. The slow equipment running state detection method according to claim 1, wherein the specific implementation of Step 4 is: The slow device target detection obtains m detection results, obtains the center point coordinates of the target detection frame of the detection result, the coordinates of device 1 in the first picture are (x1, y1), and the coordinates in the second picture are (X1, Y1); the coordinates of device 2 in the first picture are (x2, y2), and the coordinates in the second picture are (X2, Y2); in this way, the coordinates of device m in the first picture are (xm, ym), and the coordinates in the second picture are (Xm, Ym); and the distance between the center point coordinates of the same device in the front and back two pictures is calculated to obtain the relative moving distance Δd of the i-th device i , 4. The method of claim 3, wherein the step of determining the operating state of the slow device is characterized by The specific implementation of Step 5 is: setting a threshold T in advance, and comparing the moving distance Δd obtained in Step 4 with T i If Δd i ≥ T, it is determined that the equipment i is in normal operation state, otherwise it is determined to be abnormal stop; wherein the setting method of the threshold T is: according to the time interval of Step 3, the n pictures collected in a continuous period of time under the normal operation state of the slow equipment are sent into the model for detection at one time, and the relative moving distance Δd i of the slow equipment in each two pictures is obtained, n is 100, and the average value and the standard deviation σ of the relative moving distance are calculated respectively: The threshold T is calculated from the average value and the standard deviation σ.

5. The method of claim 1, wherein the method further comprises In Step 3, the slow equipment target is detected in two pictures with a time interval of 5s.

6. An electronic device, characterized by The electronic device comprises a processor and a memory, and the memory stores at least one instruction or at least one program, which is loaded and executed by the processor to realize the slow equipment running state detection model of any one of claims 1-5 for receiving the slow equipment images taken by the monitoring camera for running state detection.

7. A computer readable storage medium characterized by The computer readable storage medium stores at least one instruction or at least one program, which is executed to realize the running state detection model in the slow equipment running state detection method according to any one of claims 1-5.

Images