Low-speed equipment running state detection method, electronic equipment and storage medium

Abstract

The invention discloses a low-speed equipment operation state detection method, electronic equipment and a storage medium, and the method comprises the steps: constructing a low-speed equipment operation state detection model, and reading a real-time picture obtained through monitoring and shooting into the model for detection; the method comprises the following steps: acquiring a monitoring video by a model, acquiring a low-speed equipment picture in the video as training sample data, and replacing a C3 module of a backbone part in yolov5s with a self-defined CBAM module for training to obtain a detection model; the method comprises the following steps: transmitting two pictures which are separated for a certain period of time into a model for detection, obtaining a center point coordinate of a detection frame of the low-speed equipment in the two pictures, calculating a relative movement distance of the low-speed equipment in the period of time through the center point coordinate, and judging whether the equipment runs normally or stops abnormally by comparing a distance value with a threshold value. According to the method, the operation state of the low-speed equipment in the monitoring picture does not need to be observed and judged manually, so that potential safety hazards caused by misjudgment are avoided, and the labor cost is reduced while the safety is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to the motion state detection of a slow device, and specifically to a method for detecting the running state of a slow device, an electronic device and a storage medium. Background Art

[0002] Compared with general running equipment, slow equipment runs very slowly. For example, the driving of a sewage treatment plant is mainly responsible for discharging sludge out of the pool. Its running state directly affects the effect of the sewage treatment process and the safety of the treatment plant. Under normal circumstances, the moving distance per minute does not exceed 2 meters. Even in normal operation, it is very time-consuming to manually judge whether the equipment is in normal operation or abnormal stop through monitoring video, and due to the visual persistence phenomenon of the human eye, it is easy to make misjudgments. Because the operating environment of slow equipment is generally complex, some algorithms that perform well on the coco dataset, such as the Transformer-based DETR algorithm, the unmodified yolo series algorithm, and the single-stage target detection algorithm FCOS without anchor boxes, have poor detection effects on slow equipment in complex backgrounds. Some current methods, such as tracking algorithms, are difficult to determine the running state of objects with very small motion amplitudes and capture subtle motion changes, resulting in inaccurate data, which leads to equipment failures that cannot be discovered. Therefore, it is necessary to develop a method for detecting the running state of slow equipment to accurately detect the running state of slow equipment. Summary of the invention

[0003] The problem to be solved by the present invention is that due to the slow running speed and small movement amplitude of the slow equipment, the existing method cannot accurately detect the running status of the equipment.

[0004] The technical solution of the present invention is: a method for detecting the running status of a slow device, constructing a detection model to detect the running status of the slow device on monitoring pictures taken by a monitoring camera at a set time interval, and the detection method comprises the following steps:

[0005] Step 1: Get the slow device images collected historically by the surveillance camera;

[0006] Step 2: Use the data set obtained by annotating the pictures collected in Step 1 to train the slow device running status detection model. The yolov5s-based detection network is integrated with the CBAM attention mechanism with added sinusoidal position encoding and deformable convolution modules to form a slow device detection model.

[0007] The slow device status detection model is specifically:

[0008] Step 2.1: Construct a channel attention mechanism module, perform global average pooling and global maximum pooling on the input feature map, and obtain two description vectors, representing the average value and the maximum value respectively. After nonlinear transformation of these two vectors through a 1*1 convolution layer and a ReLU activation function, the vectors are restored to the original channel dimension through a 1*1 convolution layer. The two nonlinearly transformed vectors are added together, and then the channel attention weight is generated through the Sigmoid function. The channel attention weight is multiplied element by element with the original feature map to obtain the feature map F after channel attention weighting. CA ;

[0009] Step 2.2: The feature map F obtained in Step 2.1 CA Perform sinusoidal position encoding PE(x,y) operation to introduce position information for each spatial position (x,y) and convert F CA Add the result of sinusoidal position coding element by element to obtain the feature map F enhanced by position coding. CA_PE (x,y);

[0010] Step 2.3: Construct a deformable convolution module. The deformable convolution module includes a 3*3 convolution layer for generating offsets. The number of output channels is 18, which represents the horizontal and vertical offsets of each convolution kernel position. A 3*3 convolution layer generates a weight mask for each position. The mask uses a Sigmoid activation function to ensure that the weight is between 0 and 1. An actual 3*3 convolution layer adjusts the position of the sampling point according to the generated offset, and adjusts the weight of each position according to the generated mask to perform a convolution operation.

[0011] Step 2.4: Construct a spatial attention mechanism module to enhance the feature map F after position encoding. CA_PE Perform average pooling and maximum pooling in the channel dimension to obtain two spatial description feature maps, which are then concatenated in the channel dimension to form a new feature map. The new feature map is passed through the deformable convolution module constructed in Step 2.3 and then through the Sigmoid activation function to generate spatial attention weights. Finally, the spatial attention weights are applied to the feature map F after position encoding enhancement. CA_PE The final CBAM weighted feature map is obtained;

[0012] Step 2.5: Change the network structure, replace the C3 module in the backbone of yolov5s with the custom CBAM module obtained in Step 2.1 to Step 2.4, and obtain the slow device status detection model;

[0013] Step 3: Use the trained slow device state detection model to perform slow device target detection on two pictures taken by the surveillance camera at a set time interval;

[0014] Step 4: Get the detection results, get the center coordinates of the slow device detection frame, and calculate the distance between the center coordinates of the slow device in the previous and next two pictures to get the moving distance;

[0015] Step 5: Compare the moving distance with the set threshold to obtain the operating status of the slow device.

[0016] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the above-mentioned slow device operation status detection method.

[0017] The present invention also provides a computer-readable storage medium, wherein at least one instruction or at least one program is stored in the computer-readable storage medium, and when the at least one instruction or at least one program is executed, a method for detecting the operating status of a slow device is implemented.

[0018] Slow equipment has the characteristics of slow running speed, small and subtle movement amplitude, such as the driving of sewage treatment plants, the running distance per minute does not exceed 2 meters. If the slow equipment video collected by monitoring is manually checked, it is not easy to find the subtle position change of the slow equipment, and it is impossible to distinguish whether the slow equipment is in normal operation or abnormal stop in a short time, and it is easy to produce visual fatigue due to staring at the screen for a long time. Therefore, the present invention is aimed at the above problems. First, according to the pictures taken by the monitoring camera, the slow equipment is identified by using the improved yolov5s network model, and then the relative moving distance is calculated by the coordinates of the center point of the same slow equipment detection frame in the front and back two pictures. By comparing the distance and the size of the threshold, the running state of the slow equipment is judged, which can solve the problem that it is difficult to find the subtle position change manually. In the experiment, for the running scene of the driving, only the pictures separated by 5 seconds before and after need to be input to accurately judge the running state of the driving. Compared with manual, not only the accuracy of detection is improved, but also the detection efficiency is improved, and the manual burden is reduced.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] (1) The existing yolov5s detection model combined with the CBAM attention mechanism is limited by the fixed convolution kernel shape, has weak processing capabilities for complex backgrounds, and performs generally well in target detection in complex scenes. The present invention replaces the C3 module of the backbone part of yolov5s with a custom CBAM module that introduces sinusoidal position coding and deformable convolution. Sinusoidal position coding injects clear spatial position information into each feature point, helping the model to better understand the global and local relationships in the image, improve the model's feature positioning capability in complex scenes, and improve the accuracy of positioning and identifying targets in complex backgrounds. Introducing deformable convolution in the spatial attention mechanism can dynamically adjust the sampling position and weight of the convolution kernel, more flexibly focus on the target area, ignore background noise, improve the accuracy and stability of detection, and reduce false detection and missed detection. The present invention has stronger flexibility and robustness when processing target detection tasks with complex backgrounds, overcomes the problem of poor detection effect caused by the complex background of slow equipment operation, and improves the accuracy of detection.

[0021] (2) The present invention overcomes the problem that it is difficult for the tracking algorithm to determine the operating status of an object with a small motion amplitude, and can accurately determine the operating status of a slow device, whether it is in normal operation or abnormal stop.

[0022] (3) It solves the problem of low efficiency and accuracy in judging the operating status of slow equipment in manual observation of monitoring videos, overcomes the problem of human insensitivity to subtle position changes of slow equipment, improves judgment accuracy, saves manpower and increases safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of the method for detecting the operating status of a slow device according to the present invention.

[0024] Figure 2 The network structure diagram of the CBAM module with deformable convolution added to the present invention.

[0025] Figure 3 It is a detection schematic diagram of the present invention. DETAILED DESCRIPTION

[0026] The present invention provides a method for detecting the operating status of a slow device. The present invention solves the problem of low efficiency, high cost and high misjudgment rate in manually watching monitoring videos to determine the operating status of a slow device. The present invention is highly practical and improves the accuracy of determination and the safety of the device while saving manpower.

[0027] The present invention obtains surveillance pictures taken by a surveillance camera, and uses a yolov5s network integrated with a deformable convolution module CBAM attention mechanism to perform slow device target detection on two pictures taken a certain time apart, so as to obtain the target detection result of the slow device and the center point coordinates of the detection frame, and calculates the relative movement distance from the center point coordinates of the same slow device detection frame in the two pictures, and judges the operating status of the slow device by comparing the relative movement distance and the threshold.

[0028] like Figure 1 As shown, the specific implementation process of the present invention is as follows.

[0029] Step 1: Get the image of the slow device captured by the surveillance camera.

[0030] Step 2: Improve the yolov5s network model and obtain the slow device target detection model through training. Use the data set obtained by annotating the pictures collected in Step 1 to train the slow device operation status detection model. Take yolov5s as the basic detection network, and integrate the CBAM attention mechanism with added sinusoidal position encoding and deformable convolution modules to form a slow device detection model. Since the operating environment of slow devices is generally complex, the existing CBAM module spatial attention mechanism weights the spatial information of the feature map through the convolution operation of fixed positions, and has weak processing capabilities for complex background targets. Therefore, consider adding sinusoidal position encoding to inject clear spatial position information into each feature point, improve the model's feature positioning capability in complex scenes, and improve the accuracy of positioning and detection of targets in complex backgrounds; add deformable convolution and use its dynamic sampling capability to more flexibly focus on the target area, ignore background noise, and improve the accuracy and stability of detection. The specific implementation is as follows.

[0031] Step 2.1: Define the channel attention mechanism module, perform global average pooling and global maximum pooling on the input feature map, and obtain two description vectors, representing the average value and the maximum value respectively. After the two vectors are nonlinearly transformed through a 1*1 convolution layer and a ReLU activation function, they are restored to the original channel dimension through a 1*1 convolution layer. The two nonlinearly transformed vectors are added together, and then the channel attention weight is generated through the Sigmoid function. The channel attention weight is multiplied element by element with the original feature map to obtain the weighted feature map F. CA .

[0032] Step 2.2: Perform sinusoidal position encoding on the feature map obtained in Step 2.1, and introduce position information for each spatial position (x, y). The sinusoidal position encoding formula is as follows:

[0033] When i is an even number:

[0034]

[0035] When i is an odd number:

[0036]

[0037] Where i is the index of the channel, and C is the total number of channels in the feature map. The feature map obtained in Step 2.1 is added element by element to the sinusoidal position code to obtain the feature map enhanced by the position code.

[0038] F CA_PE (x,y)=F CA (x,y)+PE(x,y).

[0039] Step 2.3: Define a deformable convolution module. Since the present invention adopts a 3*3 deformable convolution layer, there are two offset directions (x, y) at each position. Therefore, this module uses a 3*3 convolution layer to generate an offset, and the number of output channels is 18. A 3*3 convolution layer generates a weight mask for each position. The mask uses a Sigmoid activation function to ensure that the weight is between 0 and 1. An actual 3*3 convolution layer adjusts the position of the sampling point according to the generated offset, and adjusts the weight of each position according to the generated mask to perform a convolution operation.

[0040] Step 2.4: Define the spatial attention mechanism module to enhance the feature map F after position encoding. CA_PE Average pooling and maximum pooling are performed on the channel dimension to obtain two spatial description feature maps. The two feature maps are spliced ​​on the channel dimension to form a new feature map. The new feature map is input into the 3*3 deformable convolution module, and the offset offset and mask mask are obtained through the offset generation convolution layer and the mask generation convolution layer respectively. Then the input feature map, offset and mask are input into the actual convolution layer for convolution operation, and the result is generated by the Sigmoid activation function. The spatial attention weight is finally applied to the feature map F after position encoding enhancement. CA_PE The final CBAM weighted feature map is obtained.

[0041] Step 2.5: Combine the channel attention mechanism module, sinusoidal position encoder, and spatial attention mechanism module in Step 2.1 to Step 2.4 to obtain a custom CBAM module to replace the C3 module in the backbone part of yolov5s to obtain a slow device status detection model.

[0042] The images collected in Step 1 are marked with rectangular frames to form a data set. The learning rate and number of iterations are set. The data set is used to train the yolov5s network integrated with the custom CBAM module to obtain a trained slow equipment operation status detection model.

[0043] Step 3: Use the trained slow device status detection model to input two pictures separated by a period of time into the model. The period of time is determined by the specific device running speed, generally 5 seconds, and obtain the center point coordinates of the slow device detection box in the two pictures.

[0044] Step 4: After obtaining the center point coordinates of the slow device detection frame, calculate the distance between the center point coordinates of the slow device in the two pictures before and after to obtain the moving distance. The present invention can further be used in a scenario where multiple devices exist in a single picture, that is, the target detection in Step 3 obtains m detection results, and the relative moving distance of the slow device is obtained from the center point coordinates of the detection frame of each detected target. The specific implementation is as follows.

[0045] The center point coordinates of device 1 in the first picture are (x1, y1), and the center point coordinates in the second picture are (X1, Y1); the center point coordinates of device 2 in the first picture are (x2, y2), and the center point coordinates in the second picture are (X2, Y2); and so on, the center point coordinates of device m in the first picture are (xm, ym), and the center point coordinates in the second picture are (Xm, Ym); and the distance between the center point coordinates of the same device in the previous and next pictures is calculated to obtain the relative movement distance Δd of the i-th device i .

[0046]

[0047] Step 5: Compare the relative moving distance Δdi with the threshold T. If Δdi ≥ T, the device i is judged to be in normal operation, otherwise it is judged to be abnormally stopped. The setting method of threshold T is: send n pictures of the slow device in normal operation at a certain time interval in a continuous period of time to the model for detection at one time, and obtain the relative moving distance Δd of the slow device in every two pictures. i , n is generally taken as 100, and the average value of the relative moving distance is calculated respectively And standard deviation σ:

[0048]

[0049] By average And the standard deviation σ calculates the threshold T:

[0050]

[0051] Since slow devices run slowly and have small movements, it is difficult to make an accurate judgment on whether the slow devices have moved in a short time if the slow devices are manually observed and monitored by video. This can easily lead to misjudgment. The high error rate not only brings safety issues but also increases the burden on humans. Therefore, deep learning technology is used for judgment. General tracking algorithms not only have high computing power requirements, but also have difficulty in judging the running status of objects with very small movements and capturing subtle changes in movement, resulting in inaccurate data and thus equipment failures that cannot be discovered. The present invention adopts a method based on target detection. Under normal circumstances, the operating environment of slow devices is relatively complex, which brings certain difficulties to the target detection of slow devices. The existing target detection algorithm is limited by the fixed convolution kernel shape, and the processing ability of complex backgrounds is relatively weak. The target detection of complex scenes is generally performed. Therefore, the present invention improves the yolov5s network, introduces sinusoidal position coding in the CBAM module to inject clear spatial position information into each feature point, improves the accuracy of positioning and identifying targets in complex backgrounds, introduces deformable convolution, makes the spatial attention mechanism more flexible to focus on the target area, ignores background interference, and makes the improved yolov5s network focus on the target area more accurately, thereby improving the effect of target detection. On the basis of accurately obtaining the target detection result, the relative movement distance of the slow device is calculated by the center point coordinates of the slow device detection frame in two pictures within a period of time, and the distance is compared with the size of the set threshold to calculate the operating state of the slow device.

[0052] Figure 2 The network structure diagram of the CBAM module with added sinusoidal position encoding and deformable convolution.

[0053] Figure 3 This is a demonstration of the detection process of an embodiment of the present invention. The slow equipment in the figure is a vehicle in a sewage treatment plant. It shows the process of using yolov5s integrated with a custom CBAM module to detect the slow equipment, calculate the relative moving distance of the slow equipment, make a judgment, and obtain the operating status of the slow equipment. The detection result shows that the equipment is operating normally.

[0054] An embodiment of the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the slow device operation status detection model described in the above embodiment, and is used to receive slow device images taken by a surveillance camera for operation status detection.

[0055] The embodiment of the present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program implements the slow device operation status detection model when executed by a processor of a computer. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or entirely on a remote machine or server.

Claims

1. A method for detecting the running status of a slow device, characterized in that A detection model is constructed to detect the operating status of slow devices on monitoring images taken by monitoring cameras at set intervals. The detection method includes the following steps: Step 1: Get the slow device images collected historically by the surveillance camera; Step 2: Use the data set obtained by annotating the pictures collected in Step 1 to train the slow device running status detection model. The yolov5s-based detection network is integrated with the CBAM attention mechanism with added sinusoidal position encoding and deformable convolution modules to form a slow device detection model. The slow device status detection model is specifically: Step 2.1: Construct a channel attention mechanism module, perform global average pooling and global maximum pooling on the input feature map, and obtain two description vectors, representing the average value and the maximum value respectively. After nonlinear transformation of these two vectors through a 1*1 convolution layer and a ReLU activation function, the vectors are restored to the original channel dimension through a 1*1 convolution layer. The two nonlinearly transformed vectors are added together, and then the channel attention weight is generated through the Sigmoid function. The channel attention weight is multiplied element by element with the original feature map to obtain the feature map F after channel attention weighting. CA ; Step 2.2: The feature map F obtained in Step 2.1 CA Perform sinusoidal position encoding PE(x,y) operation to introduce position information for each spatial position (x,y) and convert F CA Add the result of sinusoidal position coding element by element to obtain the feature map F enhanced by position coding. CA_PE (x,y); Step 2.3: Construct a deformable convolution module. The deformable convolution module includes a 3*3 convolution layer for generating offsets. The number of output channels is 18, which represents the horizontal and vertical offsets of each convolution kernel position. A 3*3 convolution layer generates a weight mask for each position. The mask uses a Sigmoid activation function to ensure that the weight is between 0 and 1. An actual 3*3 convolution layer adjusts the position of the sampling point according to the generated offset, and adjusts the weight of each position according to the generated mask to perform a convolution operation. Step 2.4: Construct a spatial attention mechanism module to enhance the feature map F after position encoding. CA_PE Perform average pooling and maximum pooling in the channel dimension to obtain two spatial description feature maps, which are then concatenated in the channel dimension to form a new feature map. The new feature map is passed through the deformable convolution module constructed in Step 2.3 and then through the Sigmoid activation function to generate spatial attention weights. Finally, the spatial attention weights are applied to the feature map F after position encoding enhancement. CA_PE The final CBAM weighted feature map is obtained; Step 2.5: Change the network structure, replace the C3 module in the backbone of yolov5s with the custom CBAM module obtained in Step 2.1 to Step 2.4, and obtain the slow device status detection model; Step 3: Use the trained slow device state detection model to perform slow device target detection on two pictures taken by the surveillance camera at a set time interval; Step 4: Get the detection results, get the center coordinates of the slow device detection frame, and calculate the distance between the center coordinates of the slow device in the previous and next two pictures to get the moving distance; Step 5: Compare the moving distance with the set threshold to obtain the operating status of the slow device.

2. A method for detecting the running status of a slow device according to claim 1, characterized in that In Step 2, the slow device part of the image collected in Step 1 is marked with a rectangular frame and put into the dataset as a label. The learning rate and number of iterations are set, and the improved yolov5s network constructed with the dataset is trained to obtain a slow device target detection model, which is used to detect slow devices in the images transmitted by the surveillance camera.

3. According to the method for detecting the running status of a slow device according to claim 1, it is characterized in that the specific implementation of Step 4 is: The slow device target detection obtains m detection results, and obtains the center point coordinates of the target detection frame of the detection results. The coordinates of device 1 in the first picture are (x1, y1), and the coordinates in the second picture are (X1, Y1); the coordinates of device 2 in the first picture are (x2, y2), and the coordinates in the second picture are (X2, Y2); and so on, the coordinates of device m in the first picture are (xm, ym), and the coordinates in the second picture are (Xm, Ym); and the distance between the center point coordinates of the same device in the previous and next pictures is calculated to obtain the relative movement distance Δd of the i-th device. i , 4. A method for detecting the running status of a slow device according to claim 3, characterized in that The specific implementation of Step 5 is: pre-set the threshold T, and calculate the moving distance Δd obtained in Step 4 i Compared with T, if Δd i ≥T, the device i is judged to be in normal operation, otherwise it is judged to be abnormally stopped; the setting method of the threshold T is as follows: according to the time interval of Step 3, the n pictures collected in the normal operation state of the slow device within a continuous period of time are sent to the model for detection at one time, and the relative movement distance Δd of the slow device in every two pictures is obtained. i , n is 100, and the average value of the relative moving distance is calculated respectively And standard deviation σ: By average And the standard deviation σ calculates the threshold T:

5. A method for detecting the running status of a slow device according to claim 1, characterized in that In Step 3, slow device object detection is performed on two pictures taken 5 seconds apart.

6. An electronic device, characterized in that The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement a slow device operation status detection model as described in any one of claims 1 to 5, which is used to receive slow device images taken by a surveillance camera for operation status detection.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program. When the at least one instruction or the program is executed, the operation status detection model in the method for detecting the operation status of a slow device as described in any one of claims 1 to 5 is implemented.

Images