Excavator behavior identification method based on improved SlowFast

Abstract

The invention discloses an excavator behavior recognition method based on improved SlowFast, and the method comprises the following steps: (1) collecting an excavator behavior video of a water conservancy construction site in real time, carrying out the preprocessing, and dividing a training set and a verification set; (2) sending the training set into a target detector, namely a YOLOv5 model, and generating a plurality of regions of interest (RoI), thereby completing a target positioning task; (3) after target positioning is obtained, Deepsort serves as a target tracker, target coordinates are continuously marked, and the motion trail of an object is detected; (4) cutting a corresponding region in the feature map extracted by using an improved SlowFast model, and performing region-of-interest alignment processing on the region-of-interest alignment processing to unify the size of the region; (5) carrying out average pooling on the obtained feature vectors of the region of interest in a time dimension, and then connecting a full connection layer to complete a multi-label behavior classification task; the method is suitable for different construction scenes and has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior recognition, and in particular to an excavator behavior recognition method based on improved SlowFast. Background Art

[0002] In modern water conservancy project construction, excavators, as important construction equipment, undertake a large number of earth excavation, backfilling and site leveling tasks. Due to the complex and changeable construction environment and diverse operating behaviors, accurate identification and analysis of excavator behavior is of great significance to improving construction efficiency and safety. Traditional excavator behavior recognition methods mostly rely on manual observation and simple video analysis technology. These methods have problems such as low recognition accuracy, poor real-time performance and high labor costs, and are difficult to meet actual engineering needs.

[0003] In recent years, with the development of deep learning technology, behavior recognition technology based on computer vision has been widely used. Especially in the field of video analysis, deep learning models such as convolutional neural networks (CNN) and long short-term memory networks (LSTM) have performed well. However, when processing complex video data, these models still face challenges such as difficulty in extracting spatiotemporal features, large amount of calculation, and poor real-time performance. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide an excavator behavior recognition method based on improved SlowFast to solve the problems of difficulty in extracting spatiotemporal features, large amount of calculation and poor real-time performance.

[0005] Technical solution: The excavator behavior recognition method based on improved SlowFast described in the present invention comprises the following steps:

[0006] (1) Real-time collection of excavator behavior videos at water conservancy construction sites, preprocessing, and dividing them into training sets and validation sets;

[0007] (2) The training set is fed into the target detector, namely the YOLOv5 model, to generate multiple regions of interest (RoIs), thereby completing the target localization task;

[0008] (3) After obtaining the target location, Deepsort acts as a target tracker, continuously marking the target coordinates and detecting the object's motion trajectory;

[0009] (4) Crop the corresponding area from the feature map extracted by the improved SlowFast model and align it with the region of interest to unify the size of the area;

[0010] (5) The feature vector of the obtained region of interest is average pooled in the time dimension and then connected to a fully connected layer to complete the multi-label behavior classification task.

[0011] Furthermore, in step (1), the preprocessing is as follows: first, each segment of the collected video data is trimmed to a length of 3 seconds; then, at a frequency of 30 frames per second, each 3-second video is divided into 90 frames; then, each frame of data is annotated in detail using the web version of the via tool; finally, the annotated data is divided into a training set and a validation set in a ratio of 4:1.

[0012] Further, annotations include digging, rotating and dumping.

[0013] Furthermore, in step (3), Deepsort uses the Hungarian algorithm to associate the target prediction position of the Kalman filter, that is, the predicted position of the target in the next frame, with the actual target frame by frame, and uses the correlation degree as an indicator to determine the overlap of the target frame; if it is the same object, an ID is assigned.

[0014] Furthermore, in step (4), fast and slow dual channels are used to extract feature information; fusion is performed through lateral connection, and a cross-attention mechanism is introduced in the lateral connection process to capture the long-term behavior pattern and short-term action details of the excavator; wherein, the slow channel works at a lower frame rate to capture the static features of the RGB image; the fast channel processes the video frames at a higher frame rate to capture the fast motion details and short-term behavior characteristics of the excavator, thereby being able to capture the subtle action changes of the excavator in a short time; the loss function of the model uses a focal loss function and performs smoothing on the label.

[0015] Furthermore, in step (4), the fast channel feature f fast Fusion to the slow channel feature f slow In the example, 3D convolution is used, that is, the convolution kernel size is 5x1x1, the output channel is 2βC (the channel dimension is expanded by 2 times), the step size is α (the time dimension is compressed by α times), and the output dimension is {T,S 2 ,2βC}; bilinear interpolation is used to adjust the spatial dimension, and the dimensions of the fast and slow channel features are {T,S2,C}; the specific process is as follows:

[0016]

[0017] Slow channel feature f slow Fusion into fast channel feature f fast In the example, bilinear interpolation is used to increase the time dimension {αT,S 2 ,C}; 3D convolution is used, that is, the convolution kernel size is 5x1x1, the output channel is βC (the channel dimension is compressed β times), the step size is 1, and the output dimension is {αT,S 2 ,βC}; bilinear interpolation is used to adjust the spatial dimension, and the dimensions of the fast and slow channel features are {αT,S 2,βC}; the specific process is as follows:

[0018]

[0019] Furthermore, in step (4), the specific formula of the loss function is as follows:

[0020]

[0021] Where ∈ is the smoothing factor and C is the total number of categories.

[0022] The invention discloses an excavator behavior recognition system based on improved SlowFast, comprising:

[0023] Acquisition module: used to collect real-time videos of excavator behaviors at water conservancy construction sites, perform preprocessing, and divide them into training sets and validation sets;

[0024] YOLOv5 module: used to feed the training set into the target detector, namely the YOLOv5 model, to generate multiple regions of interest (RoIs) to complete the task of target positioning;

[0025] Detection module: After obtaining the target location, Deepsort acts as a target tracker to continuously mark the target coordinates and detect the object's motion trajectory;

[0026] SlowFast module: used to crop the corresponding area in the feature map extracted by the improved SlowFast model and align the area of ​​interest to unify the size of the area;

[0027] Classification module: It is used to average pool the feature vectors of the obtained region of interest in the time dimension, and then connect a fully connected layer to complete the multi-label behavior classification task.

[0028] An electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, any one of the excavator behavior recognition methods based on improved SlowFast is implemented.

[0029] A storage medium described in the present invention stores a computer program, and when the computer program is executed by a processor, it implements any one of the excavator behavior recognition methods based on improved SlowFast.

[0030] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: through the collaborative work of slow channels and fast channels, spatiotemporal features can be efficiently extracted to improve recognition accuracy; the cross-attention mechanism can more effectively combine feature information from fast channels and slow channels. By focusing on the correlation between different paths, the model can better capture spatiotemporal information, thereby improving the quality of overall feature representation; the YOLOv5 target detector and Deepsort target tracker introduced in the present invention further optimize target positioning and detection of motion trajectories; the system supports a variety of high-resolution camera equipment, adapts to different construction scenarios, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0032] Figure 2 It is the improved SlowFast basic structure of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0034] like Figure 1 As shown, an excavator behavior recognition method based on improved SlowFast disclosed in an embodiment of the present invention first collects various excavator behavior videos, divides the input video into continuous frames, uses the web version of the via tool to annotate each frame of data, and divides the training set and the verification set into a ratio of 4:1; enters the target detector, that is, the YOLOv5 model, and generates multiple regions of interest (RoI), thereby completing the task of target positioning. After obtaining the target positioning, Deepsort is used as a target tracker to continuously annotate the target coordinates and detect the motion trajectory of the object. Then, the improved SlowFast model is used to crop the corresponding area from the extracted feature map, and the region of interest is aligned to unify the size of the region. Finally, the feature vector of the obtained region of interest is average pooled in the time dimension, and then connected to a fully connected layer to complete the multi-label behavior classification task.

[0035] The following takes the behavior of an excavator at a water conservancy construction site as an example to describe the specific steps of the embodiment of the present invention in detail. It should be noted that the method of the present invention is not only applicable to water conservancy construction sites, but also to any scenario. The embodiment of the present invention discloses an excavator behavior recognition system based on improved SlowFast, comprising the following steps:

[0036] S1, collect excavator behavior videos through on-site shooting and online search. On-site shooting uses high-resolution camera equipment to shoot real-time excavator operations at water conservancy construction sites. These cameras include fixed cameras, drone cameras, and handheld devices, which can capture the details of excavator movements from different angles and heights. Collect videos related to excavator behavior through online search to ensure the authenticity and reliability of the video source and quality. Perform preliminary screening and evaluation of the searched videos to exclude low-quality and irrelevant videos.

[0037] S2, preprocess and label the collected data. First, trim each piece of collected video data to 3 seconds in length to ensure data consistency and ease of processing. Then, at a frequency of 30 frames per second, divide each 3-second video into 90 frames to prepare for subsequent labeling and processing. Then use the web version of the via tool to label each frame of data in detail to ensure accurate identification of each action. The labeling includes specific actions such as digging, rotating, and dumping, providing label data for subsequent classification. Finally, divide the labeled data into training set and validation set in a ratio of 4:1 to ensure the effectiveness of model training and validation.

[0038] S3, the training set is sent to the target detector, that is, the YOLOv5 model, to generate multiple regions of interest (RoI), thereby completing the task of target positioning. After obtaining the target location, Deepsort is used as a target tracker to continuously mark the target coordinates and detect the motion trajectory of the object. Then, the corresponding area is cropped from the feature map extracted by the improved SlowFast model, and the region of interest is aligned to unify the size of the area. Finally, the feature vector of the obtained region of interest is averaged pooled in the time dimension, and then connected to a fully connected layer to complete the multi-label behavior classification task; the specific process includes:

[0039] S3-1, send the training set to the target detector yolov5. This model has the advantages of small size, low computing resource consumption, and fast detection speed. It meets the needs of real-time applications and realizes the task of target positioning. Then Deepsort is used as a target tracker to continuously mark the target coordinates. Deepsort uses the Hungarian algorithm to associate the target prediction position of the Kalman filter (predicting the position of the target in the next frame) with the actual target frame by frame, and proposes to use the correlation degree as an indicator to determine the overlap of the target frame. If it is the same object, an ID is assigned. Deepsort adds the use of motion features and appearance features for data association, which improves the problem of failure when re-tracking after the target is occluded.

[0040] S3-2, in the improved SlowFast model, a cross-attention mechanism is added to the lateral connection process to better integrate the features of the fast channel and the slow channel, thereby achieving a multi-level and multi-scale comprehensive analysis of video information. Through this fusion mechanism, the model can simultaneously capture the long-term behavior patterns and short-term action details of the excavator.

[0041] First, the fast channel feature f fast Fusion to the slow channel feature f slow In the example, 3D convolution is used, that is, the convolution kernel size is 5x1x1, the output channel is 2βC (the channel dimension is expanded by 2 times), the step size is α (the time dimension is compressed by α times), and the output dimension is {T,S 2 ,2βC}. Then bilinear interpolation is used to adjust the spatial dimension, and the feature dimensions of the fast and slow channels are {T,S 2 ,C}. The specific process is as follows:

[0042]

[0043] Then the slow channel feature f slow Fusion into fast channel feature f fast In the example, bilinear interpolation is used to increase the time dimension {αT,S 2 ,C}. Then a 3D convolution is used, that is, the convolution kernel size is 5×1×1, the output channel is βC (the channel dimension is compressed β times), the step size is 1, and the output dimension is {αT,S 2 ,βC}. Then bilinear interpolation is used to adjust the spatial dimension, and the feature dimensions of the fast channel and the slow channel are {αT,S 2 ,βC}. The specific process is as follows:

[0044]

[0045] S3-3, in the improved SlowFast model, the focal loss function is used, which reduces the weight of easy-to-classify samples and increases the weight of difficult-to-classify samples to improve the model's learning effect on difficult samples. And based on this, further improvements are made: smoothing is performed on the labels to reduce the model's sensitivity to noise and improve generalization ability. The specific formula of the loss function is as follows:

[0046]

[0047] To verify the contribution of this method to model performance, we conducted an ablation experiment on a self-made dataset. The results are shown in Table 1. After introducing the smooth focus loss function, the mAP of the original SlowFast model increased by 1.4%. This result shows that the smooth focus loss function has a significant effect in improving model accuracy and robustness, thereby enhancing the overall model performance.

[0048] Table 1 Ablation experiment results

[0049] Deepsort Cross-Attention Smooth focal loss function mAP(%) × × × 78.7 √ × × 79.8 × × √ 80.1 × √ × 80.6

[0050] The embodiment of the present invention further provides an excavator behavior recognition system based on improved SlowFast, comprising:

[0051] Acquisition module: used to collect real-time videos of excavator behaviors at water conservancy construction sites, perform preprocessing, and divide them into training sets and validation sets;

[0052] YOLOv5 module: used to feed the training set into the target detector, namely the YOLOv5 model, to generate multiple regions of interest (RoIs) to complete the task of target positioning;

[0053] Detection module: After obtaining the target location, Deepsort acts as a target tracker to continuously mark the target coordinates and detect the object's motion trajectory;

[0054] SlowFast module: used to crop the corresponding area in the feature map extracted by the improved SlowFast model and align the area of ​​interest to unify the size of the area;

[0055] Classification module: It is used to average pool the feature vectors of the obtained region of interest in the time dimension, and then connect a fully connected layer to complete the multi-label behavior classification task.

[0056] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, any one of the excavator behavior recognition methods based on improved SlowFast is implemented.

[0057] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any one of the excavator behavior identification methods based on improved SlowFast.

Claims

1. An excavator behavior recognition method based on improved SlowFast, characterized in that: The following steps are involved: (1) Real-time collection of excavator behavior videos at water conservancy construction sites, preprocessing, and dividing them into training sets and validation sets; (2) The training set is fed into the target detector, namely the YOLOv5 model, to generate multiple regions of interest (RoIs), thereby completing the target localization task; (3) After obtaining the target location, Deepsort acts as a target tracker, continuously marking the target coordinates and detecting the object's motion trajectory; (4) Crop the corresponding area from the feature map extracted by the improved SlowFast model and align it with the region of interest to unify the size of the area; (5) The feature vector of the obtained region of interest is average pooled in the time dimension and then connected to a fully connected layer to complete the multi-label behavior classification task.

2. The excavator behavior recognition method based on improved SlowFast according to claim 1, characterized in that: In step (1), the preprocessing is as follows: first, each segment of the collected video data is trimmed to a length of 3 seconds; then, at a frequency of 30 frames per second, each 3-second video is divided into 90 frames; then, each frame of data is annotated in detail using the web version of the via tool; finally, the annotated data is divided into a training set and a validation set in a ratio of 4:

1.

3. The excavator behavior recognition method based on improved SlowFast according to claim 2 is characterized in that: Callouts include digging, rotating, and dumping.

4. The excavator behavior recognition method based on improved SlowFast according to claim 1, characterized in that: In step (3), Deepsort uses the Hungarian algorithm to associate the target prediction position of the Kalman filter, that is, the predicted position of the target in the next frame, with the actual target frame by frame, and uses the correlation degree as an indicator to determine the overlap of the target frame; if it is the same object, an ID is assigned.

5. The excavator behavior recognition method based on improved SlowFast according to claim 1, characterized in that: In step (4), fast and slow dual channels are used to extract feature information; fusion is performed through lateral connection, and a cross-attention mechanism is introduced in the lateral connection process to capture the long-term behavior pattern and short-term action details of the excavator; wherein, the slow channel works at a lower frame rate to capture the static features of the RGB image; the fast channel processes the video frames at a higher frame rate to capture the fast motion details and short-term behavior characteristics of the excavator, thereby being able to capture the subtle action changes of the excavator in a short time; the loss function of the model uses a focal loss function and performs smoothing on the label.

6. The excavator behavior recognition method based on improved SlowFast according to claim 5, characterized in that: In step (4), the fast channel feature f fast Fusion to the slow channel feature f slow In the example, 3D convolution is used, that is, the convolution kernel size is 5x1x1, the output channel is 2βC (the channel dimension is expanded by 2 times), the step size is α (the time dimension is compressed by α times), and the output dimension is {T,S 2 ,2βC}; bilinear interpolation is used to adjust the spatial dimension, and the dimensions of the fast and slow channel features are {T,S 2 ,C}; The specific process is as follows: Slow channel feature f slow Fusion into fast channel feature f fast In the example, bilinear interpolation is used to increase the time dimension {αT,S 2 ,C}; 3D convolution is used, that is, the convolution kernel size is 5x1x1, the output channel is βC (the channel dimension is compressed β times), the step size is 1, and the output dimension is {αT,S 2 ,βC}; bilinear interpolation is used to adjust the spatial dimension, and the dimensions of the fast and slow channel features are {αT,S 2 ,βC}; the specific process is as follows:

7. The excavator behavior recognition method based on improved SlowFast according to claim 5, characterized in that: In step (4), the specific formula of the loss function is as follows: Where ∈ is the smoothing factor and C is the total number of categories.

8. An excavator behavior recognition system based on improved SlowFast, characterized in that: include: Acquisition module: used to collect real-time videos of excavator behaviors at water conservancy construction sites, perform preprocessing, and divide them into training sets and validation sets; YOLOv5 module: used to feed the training set into the target detector, namely the YOLOv5 model, to generate multiple regions of interest (RoIs) to complete the task of target positioning; Detection module: After obtaining the target location, Deepsort acts as a target tracker to continuously mark the target coordinates and detect the object's motion trajectory; SlowFast module: used to crop the corresponding area in the feature map extracted by the improved SlowFast model and align the area of ​​interest to unify the size of the area; Classification module: It is used to average pool the feature vectors of the obtained region of interest in the time dimension, and then connect a fully connected layer to complete the multi-label behavior classification task.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into the processor, the method for identifying excavator behavior based on improved SlowFast according to any one of claims 1 to 7 is implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, an excavator behavior recognition method based on improved SlowFast is implemented according to any one of claims 1 to 7.

Images