A method for identifying work violation by combining multi-spatial scale features

Abstract

The present invention discloses a method for identifying work violations that combines multi-spatial scale features. The present invention first collects videos of multiple work scenes, and after pre-processing, divides the video data into a training set and a validation set; then constructs a multi-spatial scale action recognition neural network model, including a dense-residual backbone network and a multi-spatial scale module and a classification head; performs training based on the constructed network model; then collects work site monitoring videos, selects segments to be identified, obtains key frames, performs character detection on the key frames, and obtains the bounding boxes of the characters in the video screen; finally, inputs the obtained video segments, key frames, and bounding boxes into the multi-spatial scale action recognition neural network model to perform behavior or action recognition, identify character behavior, and determine whether it is a violation. The present invention can improve the accuracy of action recognition of work characters under long viewing angles, overexposure, and underexposure, thereby improving the accuracy and robustness of overall action recognition.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning computer vision, and specifically relates to a method for identifying work violations by combining multi-spatial scale features. Background Art

[0002] The various safety risk factors involved in power infrastructure construction are complex and diverse, encompassing 22 categories, including grid accidents, electric shock, and fire. Power project construction safety is directly related to the life safety of construction workers and the property safety of power companies. Identifying violations is a crucial component of safety oversight for power infrastructure construction projects. In existing power construction safety monitoring systems, violation identification is primarily accomplished through manual monitoring. Safety supervision center staff monitor on-site operations by spot-checking surveillance videos transmitted from construction sites, identifying violations, and taking appropriate measures based on their severity. However, this violation identification method suffers from high labor costs, a high risk of missed detections, low efficiency, and subjective misjudgments.

[0003] Deep learning-based video action recognition technology utilizes deep neural networks to analyze and learn from video sequences, enabling automatic recognition and classification of human movements in videos. By inputting video frame sequences into a deep learning model, the system learns and extracts the characteristic representations of the movements, enabling accurate action classification and recognition. Currently, few methods for identifying violations in power infrastructure projects employ neural network models. These methods also fail to specifically account for interfering factors such as viewing angle and lighting in construction site surveillance videos, resulting in low accuracy in recognizing subtle movements at long distances. Summary of the Invention

[0004] In view of this, the present invention provides a method for identifying work violations by combining multi-spatial scale features to solve or alleviate the technical problems existing in the prior art.

[0005] In a first aspect, the present invention provides a method for identifying work violations by combining multi-spatial scale features, the method comprising:

[0006] S1. Collect videos of multiple work scenes, crop the videos and adjust the resolution and frame rate, extract all image frames, and annotate key frames for actions;

[0007] S2, divide the video data into training set and validation set;

[0008] S3. Build a multi-spatial-scale action recognition neural network model, including a dense-residual backbone network, a multi-spatial-scale module, and a classification head.

[0009] The output of the dense-residual backbone network is divided into three branches. One branch serves as the input of the first stage of the multi-spatial scale module. Another branch and the output of the multi-spatial scale module are fed into the ROI Align layer for bounding box alignment to obtain four sets of outputs. The third branch is concatenated with the output of the ROI Align layer.

[0010] S4. Input the training samples into the multi-spatial-scale action recognition neural network model for supervised training, use the cross-entropy loss function, and solve the weights and bias parameters of the network model through iterative optimization; and use the samples in the validation set to verify the neural network model;

[0011] S5. Collect the surveillance video of the operation site, select the segment to be identified, obtain the key frame, perform person detection on the key frame, and obtain the bounding box of the person in the video image;

[0012] S6. Input the video clips, key frames, and bounding boxes obtained in S5 into a multi-spatial-scale action recognition neural network model to perform behavior or action recognition, identify the person's behavior, and determine whether it is a violation.

[0013] In a second aspect of the present invention, a computer device is provided, comprising a memory, a processor, and computer executable instructions stored in the memory and executable on the processor. When the processor executes the instructions, the method for identifying work violations combining multi-spatial scale features is implemented.

[0014] In a third aspect, the present invention provides a computer storage medium storing computer executable instructions, which, when executed, implement the method for identifying work violations combining multi-spatial scale features.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] This paper proposes a multi-scale action recognition neural network model. It designs a multi-scale module to extract multi-scale features of human actions in video streams. This model improves the accuracy of human action recognition under long-viewing conditions, thereby enhancing the overall accuracy and robustness of action recognition. The paper uses a purely convolutional neural network structure, reducing the number of training samples required for model training and making it easier to deploy. It also improves the accuracy and robustness of identifying illegal actions at work sites. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of a method for identifying violations in power engineering operations that combines multi-spatial scale features according to an embodiment of the present application.

[0018] Figure 2This is a framework diagram of a multi-spatial-scale action recognition neural network according to an embodiment of the present application.

[0019] Figure 3 This is a diagram of the multi-spatial-scale action recognition neural network structure of an embodiment of the present application.

[0020] Figure 4a This is a structural diagram of the input processing module of an embodiment of the present application.

[0021] Figure 4b Schematic diagram of the structure of four convolution submodules in an embodiment of the present application.

[0022] Figure 4c Schematic diagram of the dense-residual backbone network structure of an embodiment of the present application.

[0023] Figure 5 This is a specific structural diagram of the multi-spatial-scale module according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0025] The present application provides a method for identifying violations in power engineering operations by combining multi-spatial scale features, including the following steps:

[0026] S1. Video data of multiple operation scenes is collected through the surveillance camera at the power engineering operation site. The video data is downloaded to the safety supervision center, the video is cropped, and the video resolution is uniformly adjusted to 604×340 and the frame rate is uniformly adjusted to 30 frames per second. All image frames are extracted, and the key frames of the processed video are further extracted. Action annotation is performed based on this.

[0027] S2: Divide the video data in S1 into a training set and a validation set according to a total duration ratio of 6:4. At the same time, it is necessary to ensure that the cropped videos of the same time, place, and operation scene are not assigned to the training set and validation set separately.

[0028] S3. Constructing the backbone network of the multi-scale action recognition neural network model: We improve on the residual network by introducing dense-residual connections and adopting an optimized convolutional submodule structure. The backbone network consists of an input processing module, four convolutional modules composed of four convolutional submodules, and a downsampling module.

[0029] S4. Construct a multi-spatial-scale module for the multi-spatial-scale action recognition neural network model, and use the outputs i1, i2, i3, and i4 from the four convolutional stages of the backbone network for upsampling and fusion to obtain the multi-spatial-scale features o1, o2, and o3.

[0030] S5. The output of the backbone network is divided into three branches. One branch is used as the input of the first stage of the multi-spatial scale module. One branch is sent to the ROI Align layer together with the output of the multi-spatial scale module for bounding box alignment to obtain four sets of outputs. One branch is spliced ​​with the output of ROIAlign. The splicing layer is connected to the classification head. The classification head is composed of a kernel of 3×1 2 3D convolutional layer with stride=2,1,1 and kernel size 1×1 2 3D convolutional layer with stride=1,1,1 and kernel size 1×3 2 It consists of a three-dimensional convolutional layer with stride=1,1,1, an MLP linear layer, and a SoftMax classifier, where the first two convolutional layers use the ReLU activation function.

[0031] S6. Input the training samples in S2 into the multi-spatial-scale action recognition neural network model for supervised training, use the cross-entropy loss function, and solve the weights and bias parameters of the network model through iterative optimization; and use the samples in the validation set to verify the modified neural network model.

[0032] S7. Collect monitoring videos of power engineering operations, select segments to be identified, obtain key frames at a sampling rate of 1 frame per second, use the Faster R-CNN network to detect people in the key frames, and obtain the bounding boxes of N people in the video.

[0033] S8, the video clips, key frames, and bounding boxes obtained in S7 Input into the multi-spatial scale action recognition neural network model to perform behavior or action recognition, identify the person's behavior, and determine whether it is a violation.

[0034] In one embodiment, the detailed steps of S1 are:

[0035] S11. Download the operation site monitoring video;

[0036] S12, cropping the video footage that does not contain construction operations;

[0037] S13. Use the FFmpeg tool to extract frame images using the command ffmpeg -ss 1 -i video-path -qscale:v 1 -vf "scale = 604:340, fps = 30", where "-ss 1" specifies to start processing from the first second in the input file, "-i video-path" specifies the video path, "-qscale:v 1" sets the video quality to a constant rate factor (CRF) of 1, and -vf "scale = 604:340, fps = 30" sets the video filter, which first uses the scale filter to scale the video to a resolution of 604x340 pixels and then uses the fps filter to set the frame rate to 30 frames per second.

[0038] S14, select key frames: use a python script to extract the key frames of each video file at a sampling rate of 1 frame per second;

[0039] S15. Use the yolov5 model to perform target detection on the key frames to obtain a target detection file in json format;

[0040] S16, deleting the non-human bounding boxes in the target detection file to obtain a human target detection file;

[0041] S17. Import the person target detection file and key frames into the via annotation tool, and perform behavior annotation on each person boundary box one by one by checking the check box to obtain a person behavior annotation file.

[0042] In one embodiment, if Figure 2 and Figure 3 As shown, the multi-spatial-scale action recognition neural network model constructed in this application consists of three parts: a dense-residual backbone network, a multi-spatial-scale module, and a classification head.

[0043] The dense residual backbone network extracts spatiotemporal contextual features of video clips. The dense-residual backbone network includes an input processing module, four convolution modules consisting of four convolution submodules, and a downsampling module. The multi-spatial scale feature module upsamples and fuses the outputs of each convolution module of the backbone network in three stages to obtain multi-spatial scale features. The contextual features and multi-spatial scale features are each combined with the character bounding box to perform ROI Align to obtain their corresponding character action features, and these features are spliced ​​to obtain the final character action features. The classification head convolves and projects the character action features, and uses SoftMax for prediction and classification.

[0044] The multi-spatial-scale action recognition neural network model specifically includes:

[0045] (1) Dense-Residual Backbone Network:

[0046] Input processing module:

[0047] Convolutional layer 1: The convolution kernel size is 1×4×4, the number of convolution kernels is 96, stride=1, 4, 4, padding=0, and Layer Norm regularization is used;

[0048] Convolutional module 1:

[0049] Convolution Subblock 1-1 (Conv Subblock 1): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 96 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 96 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0050] Convolution Subblock 1-2 (Conv Subblock 1): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 96 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 96 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0051] Convolution Subblock 1-3 (Conv Subblock 1): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 96 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 96 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0052] Jump 1-1: the input of convolution submodule 1-1 and the output of convolution submodule 1-1 are cascaded;

[0053] Jump 1-2: the input of convolution submodule 1-2 and the output of convolution submodule 1-2 are cascaded;

[0054] Jump 1-3: the input of convolution submodule 1-3 and the output of convolution submodule 1-3 are cascaded;

[0055] Jump 1-4: The input of convolution submodule 1-1 and the output of convolution submodule 1-4 are cascaded.

[0056] Downsampling module 1:

[0057] This module contains Layer Norm regularization and three-dimensional convolution with a convolution kernel size of 1×2×2, 192 convolution kernels, stride=1, 2, 2, and padding=0.

[0058] Convolutional module 2:

[0059] Convolution Subblock 2-1 (Conv Subblock 2): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 192 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 3×1×1, 768 kernels, stride of 2,1,1, and padding of 1,0,0; and one 3D convolution with a kernel size of 1×1×1, 192 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0060] Convolution Subblock 2-2 (Conv Subblock 2): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 192 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 3×1×1, 768 kernels, stride of 2,1,1, and padding of 1,0,0; and one 3D convolution with a kernel size of 1×1×1, 192 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0061] Convolution Subblock 2-3 (Conv Subblock 2): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 192 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 3×1×1, 768 kernels, stride of 2,1,1, and padding of 1,0,0; and one 3D convolution with a kernel size of 1×1×1, 192 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0062] Jump 2-1: the input of convolution submodule 2-1 and the output of convolution submodule 2-1 are cascaded;

[0063] Jump 2-2: the input of convolution submodule 2-2 and the output of convolution submodule 2-2 are cascaded;

[0064] Jump 2-3: the input of convolution submodule 2-3 and the output of convolution submodule 2-3 are cascaded;

[0065] Jump 2-4: The input of convolution submodule 2-1 and the output of convolution submodule 2-4 are cascaded.

[0066] Downsampling module 2:

[0067] This module contains Layer Norm regularization and three-dimensional convolution with a convolution kernel size of 1×2×2, 384 convolution kernels, stride=1, 2, 2, and padding=0.

[0068] Convolutional module 3:

[0069] Convolution Subblock 3-1 (Conv Subblock 3): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 384 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 1536 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0070] Convolution Subblock 3-2 (Conv Subblock 3): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 384 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 1536 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0071] Convolution Subblock 3-3 (Conv Subblock 3): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 384 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 1536 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0072] Convolution Subblock 3-4 (Conv Subblock 3): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 384 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 1536 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0073] Convolution Subblock 3-5 (Conv Subblock 3): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 384 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 1536 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0074] Convolution Subblock 3-6 (Conv Subblock 3): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 384 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 1536 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0075] Convolution Subblock 3-7 (Conv Subblock 3): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 384 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 1536 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0076] Convolution Subblock 3-8 (Conv Subblock 3): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 384 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 1536 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0077] Convolution Subblock 3-9 (Conv Subblock 3): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 384 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 1×1×1, 1536 kernels, stride of 1,1,1, and padding of 0; and one 3D convolution with a kernel size of 1×1×1, 384 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0078] Jump 3-1: the input of convolution submodule 3-1 and the output of convolution submodule 3-1 are cascaded;

[0079] Jump 3-2: the input of convolution submodule 3-2 and the output of convolution submodule 3-2 are cascaded;

[0080] Jump 3-3: the input of convolution submodule 3-3 and the output of convolution submodule 3-3 are cascaded;

[0081] Jump 3-4: the input of convolution submodule 3-4 and the output of convolution submodule 3-4 are cascaded;

[0082] Jump 3-5: the input of convolution submodule 3-5 and the output of convolution submodule 3-5 are cascaded;

[0083] Jump 3-6: the input of convolution submodule 3-6 and the output of convolution submodule 3-6 are cascaded;

[0084] Jump 3-7: the input of convolution submodule 3-7 and the output of convolution submodule 3-7 are cascaded;

[0085] Jump 3-8: the input of convolution submodule 3-8 and the output of convolution submodule 3-8 are cascaded;

[0086] Jump 3-9: concatenate the input of convolution submodule 3-9 and the output of convolution submodule 3-9;

[0087] Jump 3-10: cascade the input of convolution submodule 3-1 and the output of convolution submodule 3-3;

[0088] Jump 3-11: cascade the input of convolution submodule 3-4 and the output of convolution submodule 3-6;

[0089] Jump 3-12: cascade the input of convolution submodule 3-7 and the output of convolution submodule 3-9;

[0090] Jump 3-13: cascade the input of convolution submodule 3-1 and the output of convolution submodule 3-9;

[0091] Downsampling module 3:

[0092] This module contains Layer Norm regularization and three-dimensional convolution with a convolution kernel size of 1×2×2, 768 convolution kernels, stride=1, 2, 2, and padding=0.

[0093] Convolutional module 4:

[0094] Convolution Subblock 4-1 (Conv Subblock 4): This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 768 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 3×1×1, 3072 kernels, stride of 2,1,1, and padding of 1,0,0; and one 3D convolution with a kernel size of 1×1×1, 768 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0095] Convolution Subblock 4-2: This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 768 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 3×1×1, 3072 kernels, stride of 2,1,1, and padding of 1,0,0; and one 3D convolution with a kernel size of 1×1×1, 768 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is used after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0096] Convolution Subblock 4-3: This subblock includes one depthwise separable 3D convolution with a kernel size of 1×7×7, 768 kernels, stride of 1,1,1, and padding of 0,3,3; one 3D convolution with a kernel size of 3×1×1, 3072 kernels, stride of 2,1,1, and padding of 1,0,0; and one 3D convolution with a kernel size of 1×1×1, 768 kernels, stride of 1,1,1, and padding of 0. Layer Norm regularization is applied after the depthwise separable convolution. The first 3D convolution uses the ReLU activation function, and the second 3D convolution uses Layer Scale regularization.

[0097] Jump 4-1: the input of the convolution submodule 4-1 and the output of the convolution submodule 4-1 are cascaded;

[0098] Jump 4-2: the input of the convolution submodule 4-2 and the output of the convolution submodule 4-2 are cascaded;

[0099] Jump 4-3: the input of the convolution submodule 4-3 and the output of the convolution submodule 4-3 are cascaded;

[0100] Jump 4-4: The input of convolution submodule 4-1 and the output of convolution submodule 4-3 are cascaded.

[0101] (2) Multi-spatial scale module:

[0102] Phase 1:

[0103] Convolutional layer 1-1: kernel size is 1×1×1, number of kernels is 256, stride=1,1,1, padding=0;

[0104] Upsampling layer 1-1: trilinear interpolation upsampling, sampling magnification 1, 2, 2;

[0105] Convolutional layer 1-2: convolution kernel size is 2×1×1, the number of convolution kernels is 256, stride=2, 1, 1, padding=0;

[0106] Convolutional layers 1-3: kernel size is 1×3×3, number of kernels is 256, stride=1,1,1, padding=0,1,1;

[0107] The organization of stage 1 is as follows: the output of the backbone network convolution module 4 passes through the convolution layer 1-1 and the upsampling layer 1-1 to obtain t1, the output of the backbone network convolution module 3 passes through the convolution layer 1-2 and is added to t1 to obtain m1, and m1 passes through the convolution layer 1-3 to obtain the output o1 of stage 1.

[0108] Phase 2:

[0109] Upsampling layer 2-1: trilinear interpolation upsampling, sampling magnification 1, 2, 2;

[0110] Convolutional layer 2-1: The convolution kernel size is 1×1×1, the number of convolution kernels is 256, stride=1,1,1, padding=0;

[0111] Convolutional layer 2-2: The convolution kernel size is 1×3×3, the number of convolution kernels is 256, stride=1,1,1, padding=0,1,1;

[0112] The organization of stage 2 is as follows: m1 passes through the upsampling layer 2-1 to obtain t2, the output of the backbone network convolution module 2 passes through the convolution layer 2-1 and is added to t2 to obtain m2, and m2 passes through the convolution layer 2-2 to obtain the output o2 of stage 2.

[0113] Phase 3:

[0114] Upsampling layer 3-1: trilinear interpolation upsampling, sampling magnification 1, 2, 2;

[0115] Convolutional layer 3-1: The convolution kernel size is 4×1×1, the number of convolution kernels is 256, stride=4, 1, 1, padding=0;

[0116] Convolutional layer 3-2: convolution kernel size is 1×3×3, the number of convolution kernels is 256, stride=1,1,1, padding=0,1,1;

[0117] The organization of stage 2 is as follows: m2 passes through the upsampling layer 3-1 to obtain t3, the output of the backbone network convolution module 1 passes through the convolution layer 3-1 and is added to t3 to obtain m3, and m3 passes through the convolution layer 3-2 to obtain the output o3 of stage 3.

[0118] (3) Classification header:

[0119] Convolutional layer 1: The convolution kernel size is 3×1×1, the number of convolution kernels is 4096, stride=2,1,1, padding=1,0,0, and ReLu activation function is used;

[0120] Convolutional layer 2: The convolution kernel size is 1×1×1, the number of convolution kernels is 2048, stride=1,1,1, padding=0, and ReLu activation function is used;

[0121] Convolutional layer 3: The convolution kernel size is 1×3×3, the number of convolution kernels is 2048, stride=1,1,1, padding=0,1,1;

[0122] MLP linear layer: including a hidden layer of 2048 neurons, a hidden layer of 1024 neurons, a hidden layer of 512 neurons, and an output layer of 30 neurons;

[0123] SoftMax layer.

[0124] The above is the composition of the multi-spatial-scale action recognition neural network of this application. The following further describes its internal topological relationship in more detail based on the direction of data flow.

[0125] S31, the input processing module input is a 32×224×224 3-channel RGB image sequence, where "32" refers to 32 frame images in time sequence and "224×224" refers to the spatial size of the frame image. The output of this layer is a 96-channel feature volume of 32×56×56;

[0126] The 96-channel feature volume of S32 and 32×56×56 is input into the convolution module 1 composed of three Conv Subblock1s, and the 96-channel feature volume of 32×56×56 is output, which is convenient for subsequent reference and is recorded as i4;

[0127] S33 and i4 input the downsampling module to obtain a 192-channel feature volume of 32×28×28;

[0128] S34, input the feature volume of the previous step into the convolution module 2 composed of three Conv Subblock2, and output a 16×28×28 192-channel feature volume, denoted as i3;

[0129] S35 and i3 input the downsampling module to obtain a 384-channel feature volume of 16×14×14;

[0130] S36, input the feature volume of the previous step into the convolution module 3 composed of nine Conv Subblock3, and output a 384-channel feature volume of 16×14×14, recorded as i2;

[0131] S37 and i2 input the downsampling module to obtain a 768-channel feature volume of 16×7×7;

[0132] S38. Input the feature body of the previous step into the convolution module 4 composed of three Conv Subblock4, and output an 8×7×7 768-channel feature body, denoted as i1. This is the output of the backbone network, which contains the dynamic context feature Context Feature in the image sequence.

[0133] For S3, in more detail,

[0134] In this embodiment, the input processing module of S3 is formed by a three-dimensional convolution connected to Layer Norm with a convolution kernel of 1×4×4 and stride=1, 4, and 4. The input is a 32×224×224 3-channel RGB image sequence, and the output is a 32×56×56 96-channel feature volume. Figure 4a As shown in Figure 1, the input processing module uses a large kernel 3D convolution with a size of 1×4×4 and stride=1, 4, 4, which can better capture the local features in the input feature map and retain the details of the action scene from a long distance perspective.

[0135] In this embodiment, the downsampling layer of S3 is connected to Layer Norm and the kernel size is 1×2 2 For a three-dimensional convolution with stride=1, 2, 2, the number of output channels is twice that of the input, the output spatial size is 1 / 2 of the input, and the scale in the time dimension remains unchanged. Figure 4aAs shown, the downsampling module adopts 2×2 convolution with stride=2, so that the downsampling module and other layers maintain basically the same calculation strategy. Compared with the residual network, the downsampling is integrated into the residual convolution module. The embodiment of the present application separates the downsampling from the four convolution modules and performs it separately, which can reduce the computational complexity of the model, and adds Layer Norm after downsampling, which helps to maintain the stability of the model.

[0136] In this embodiment, the input of the convolution submodule Conv Subblock1 of S3 is a 96-channel feature volume of 32×56×56, which is converted into a 1×7 feature volume after the kernel size is connected. 2 After the depth-separable 3D convolution and Layer Norm, a 96-channel feature volume of 32×56×56 is obtained, and then the input kernel size is 1×1 2 The three-dimensional convolution uses the ReLU activation function to output a 384-channel feature volume of 32×56×56, and then the input kernel size is 1×1 2 The three-dimensional convolution is performed, and after layer scaling, the output is a 96-channel feature volume of 32×56×56. The three Conv Subblock1s are connected front and back, and at the level of a single convolution submodule, the input of the Conv block is added to its output through a direct edge. Jump connections are also made at intervals of three convolution submodules to form a dense residual connection, forming Convolution Module 1. The connection method is expressed by the following formula:

[0137] x1=H1(x0)+x0, x2=H1(x1)+x1, x3=H1(x2)+x2+x0

[0138] Where x0 is the input of the first Conv Subblock1, x l is the output of the lth Conv Subblock1, and H1(·) is the nonlinear transformation function represented by Conv Subblock1.

[0139] In this embodiment, the input of the convolution submodule Conv Subblock2 of S3 is a 192-channel feature volume of 32×28×28, which is converted into a 1×7 feature volume after the kernel size is connected. 2 After the depth-separable 3D convolution and Layer Norm, a 192-channel feature volume of 32×28×28 is obtained, and then the input kernel size is 3×1 2 The step size stride=2,1,1 uses the three-dimensional convolution of the ReLU activation function, outputting a 768-channel feature volume of 16×28×28, followed by an input kernel size of 1×1 2The three-dimensional convolution is performed, and after LayerScale, it outputs a 192-channel feature volume of 16×28×28. The three Conv Subblock2s are connected front and back, and at the level of a single convolution submodule, the input of the Conv block is added to its output through a direct edge. Jump connections are also made at intervals of three convolution submodules to form a dense residual connection, forming Convolution Module 2. The connection method is expressed by the following formula:

[0140] x1=H2(x0))+x0, x2=H2(x1)+x1, x3=H2(x2)+x2+x0

[0141] Where x0 is the input of the first Conv Subblock2, x l is the output of the lth Conv Subblock2, and H2(·) is the nonlinear transformation function represented by Conv Subblock2.

[0142] In this embodiment, the input of the convolution subblock Conv Subblock3 of S3 is a 384-channel feature volume of 16×14×14, which is converted into a 1×7 kernel. 2 After the depth-separable 3D convolution and Layer Norm, a 384-channel feature volume of 16×14×14 is obtained, and then the input kernel size is 1×1 2 The three-dimensional convolution uses the ReLU activation function to output a 16×14×14 1536-channel feature volume, and then the input kernel size is 1×1 2 The three-dimensional convolution is layered and scaled to output a 16×14×14 384-channel feature volume. Nine Conv Subblocks3 are connected front and back, and at the level of a single convolution subblock, the input of the Convblock is added to its output via a direct edge. Jump connections are also made every three convolution subblocks, and every nine convolution subblocks. This forms a dense residual connection, forming the convolution module 3. The connection method is expressed by the following formula:

[0143] x k =H3(x k-1 )+x k-1

[0144] x j =H3(x j-1 )+x j-1 +x j-3

[0145] X9=H3(x8)+x8+x6+x0

[0146] Where k = 1, 2, 4, 5, 7, 8; j = 3, 6. x0 is the input of the first Conv Subblock3, xl is the output of the lth ConvSubblock3, and H3(·) is the nonlinear transformation function represented by Conv Subblock3.

[0147] In this embodiment, the input of the convolution submodule Conv Subblock4 of S3 is a 768-channel feature volume of 16×7×7, which is converted into a 1×7 feature volume after the kernel size is connected. 2 After the depth-separable 3D convolution and Layer Norm, a 768-channel feature volume of 16×7×7 is obtained, and then the input kernel size is 3×1 2 The step size stride=2,1,1 uses the three-dimensional convolution of the ReLU activation function, outputting a 3072-channel feature volume of 8×7×7, followed by an input kernel size of 1×1 2 The three-dimensional convolution is performed, and after layer scaling, it outputs a 768-channel feature volume of 8×7×7. The three Conv Subblocks 4 are connected front and back, and at the level of a single convolution submodule, the input of the Conv block is added to its output through a direct edge. Jump connections are also made at intervals of three convolution submodules to form a dense residual connection, forming a convolution module 4. The connection method is expressed by the following formula:

[0148] x1=H4(x0)+x0, x2=H4(x1))+x1, x3=H2(x2))+x2+x0

[0149] Where x0 is the input of the first Conv Subblock4, x l is the output of the lth Conv Subblock4, and H4() is the nonlinear transformation function represented by Conv Subblock4.

[0150] like Figure 4bAs shown in the figure, the convolution sub-module Conv Subblock1 first uses a large kernel 1×7×7 three-dimensional depth-separable convolution, then uses a kernel size of 1×1×1 three-dimensional convolution to increase the dimension, and finally uses a kernel size of 1×1×1 three-dimensional convolution to reduce the dimension to the original dimension. The depth-separable convolution focuses on the feature extraction between channels and the integration of spatial information, which helps to extract the features of small-amplitude motion changes in the time dimension. The convolution kernel of unit size is used to increase the dimension and then reduce the dimension. On the one hand, the small convolution kernel can improve the computational efficiency of the model. On the other hand, increasing the dimension first and then reducing the dimension can promote the model to make more subtle distinctions between features of different brightness levels. Subblock2 first uses a three-dimensional depth-wise separable convolution with a large kernel of 1×7×7, then uses a three-dimensional convolution with a kernel size of 3×1×1 and a stride of 2, 1, 1 to increase the dimension, and finally uses a three-dimensional convolution with a kernel size of 1×1×1 to reduce the dimension to the original dimension. Similarly, the large-kernel depth-wise separable convolution helps to extract the features of small-amplitude motion changes in the time dimension. Considering that the temporal resolution required to retain large-amplitude motion features is reduced after the model extracts small-amplitude motion features, a convolution with a kernel of 3×1×1 is used to increase the dimension while reducing the resolution of the time dimension; the structure of the convolution submodule Conv Subblock3 is the same as that of the convolution submodule Conv Subblock1, the difference is that the number of input and output channels increases. By gradually increasing the dimension, the network can gradually learn more complex and abstract feature representations. The structure of the convolution submodule Conv Subblock4 is the same as that of the convolution submodule Conv Subblock2. Similarly, the large-kernel depthwise separable convolution helps extract the features of small-scale motion changes in the temporal dimension. Here, the network is already very deep and focuses on large-scale, large-area motions. Therefore, a convolution with a kernel of 3×1×1 is used again to increase the dimension while reducing the resolution of the temporal dimension. The four convolution submodules can gradually learn the features from subtle to large-scale motions, helping the model improve the overall motion recognition accuracy.

[0151] like Figure 4c As shown, the four convolutional modules have a ratio of 3, 3, 9, and 3 convolutional submodules. Dense residual connections are used within the four convolutional modules. The ratio of 3, 3, 9, and 3 was determined experimentally and is an appropriate ratio for this embodiment, considering multiple factors such as model size, computational complexity, and behavior recognition accuracy.

[0152] Each convolution module uses dense-residual connections. Compared with simply using residual connections, every three Conv Subblocks in the convolution module receive one more input from the third Conv Subblock from the top, further reducing information loss. This is especially important for retaining information about fine-grained actions in different time and space. Large-scale actions can be composed of many small-scale and subtle actions. Retaining more information from previous layers helps improve model recognition performance. Compared with dense connections, using dense-residual connections significantly reduces memory usage and computational costs, thereby improving the deployability of the model.

[0153] In this embodiment, the multi-spatial scale module receives four inputs. The input i1 comes from the convolution module 4, which is a 768-channel feature volume of 8×7×7 and is processed by a kernel of 1×1 2 , the three-dimensional convolution layer with stride=1,1,1 outputs a 256-channel feature volume of 8×7×7, which is then upsampled by trilinear interpolation with sampling rates of 1×, 2×, and 2× to obtain a 256-channel feature volume t1 of 8×14×14; the input i2 comes from the convolution module 3, which is a 384-channel feature volume of 16×14×14, and is processed by a kernel of 3×1 2 , the three-dimensional convolution layer with stride=2,1,1 outputs a 256-channel feature volume of 8×14×14, which is added to t1. The result of the addition is divided into two branches, one of which is 1×3 after the kernel. 2 , the three-dimensional convolution layer with stride=1,1,1 outputs a 256-channel feature volume o1 of 8×14×14, and the other is upsampled by trilinear interpolation with sampling rates of 1×, 2×, 2× to obtain a 256-channel feature volume t2 of 8×28×28; the input i3 comes from the convolution module 2, which is a 192-channel feature volume of 16×28×28, and is processed by a kernel of 3×1 2 , the three-dimensional convolution layer with stride=2,1,1 outputs a 256-channel feature volume of 8×28×28, which is added to t2. The result of the addition is divided into two branches, one of which is 1×3 after the kernel. 2 , the three-dimensional convolution layer with stride=1,1,1 obtains the 256-channel feature volume o2 of 8×28×28, and the other one is up-sampled by trilinear interpolation with sampling rates of 1×, 2×, 2× to obtain the 256-channel feature volume t3 of 8×56×56; the input i4 comes from the convolution module 1, which is a 96-channel feature volume of 32×56×56 after the kernel is 4×1 2 , the three-dimensional convolutional layer with stride=4,1,1 outputs a 256-channel feature volume of 8×56×56, which is added to t3. The result of the addition is a 256-channel feature volume of 8×56×56. The calculation formula is:

[0154]

[0155] Where TU(·) is trilinear interpolation upsampling, d(·) is a 3D convolution with 256 output channels, and S(·) is a 3D convolution for smoothing.

[0156] like Figure 5 As shown, the multi-spatial scale module in the embodiment of the present application obtains feature maps of different scales with higher output resolution than the backbone network and with high-level semantics in the time dimension through a top-down feature fusion method. Based on this series of feature maps, spatial multi-scale motion features, especially the features of small parts and small movements, can be better extracted. Similarly, for overexposed or underexposed images, smaller-scale (larger resolution) feature maps can retain more detailed textures. Therefore, the introduction of this multi-spatial scale module can improve the model's ability to recognize subtle movements and enhance the model's ability to resist interference such as overexposure and underexposure introduced during the video acquisition process. All features from the backbone network will first be reduced to 256 channels through three-dimensional convolution of the unit convolution kernel, and the temporal resolution will also be adjusted to 8 through different convolution kernels and step size settings. The same dimension and the same temporal resolution are used to ensure that it can be numerically added with the upsampled upper-layer features (such as i2 has the same dimension as t2 after dimensionality reduction).

[0157] In the embodiment of the present application, the specific steps performed by the ROI Align layer of S5 are:

[0158] S51, obtain the bounding box position and size of N characters in the video image from the annotation file of S1

[0159] S52. Scale the bounding box to match the size of Context Feature, o1, o2, and o3. Extract features based on the positions of the four corner points of the bounding box and reshape the extracted feature space dimensions to 7×7 to obtain character features at multiple scales.

[0160] S53, concatenate the Context Feature with all the character features A to obtain the context-single-person association feature

[0161] S54. The context-single-person association features are input into the classification head. After further extraction of high-level semantic features through convolution, they are input into the linear layer, and then the behavior prediction is performed by the SoftMax layer.

[0162] The loss function formula of S6 is: Where K is the number of action label categories in the annotation, is the probability of the i-th category predicted by the model, y i is the actual label of the i-th category.

[0163] In this embodiment, after steps S1-S6, a trained multi-spatial scale action recognition neural network model can be obtained, which can be used to identify illegal behaviors at the work site. The specific process is as follows: Figure 1 As shown in the figure: the construction site operation monitoring screen is collected and imported into the backbone network to obtain contextual spatiotemporal feature extraction, which is then imported into the Faster R-CNN tool for person detection to obtain bounding boxes. The output of each stage of the backbone network enters the multi-spatial scale module for multi-spatial scale feature extraction. The bounding box is projected and cut (ROIAlign) on the contextual spatiotemporal feature map and the multi-spatial scale feature map according to the screen ratio to obtain a multi-scale feature map of a single working person. The multi-scale feature map of a single working person and the contextual spatiotemporal features are further spliced ​​to obtain a context-single-person feature map. The context-single-person feature map enters the 3D convolutional layer of the classification head, which fuses the spatiotemporal information contained in the contextual spatiotemporal features and the single-person feature map to abstract behavioral features containing high-level semantics. After the feature is dimensionalized by the fully connected layer, SoftMax is used for classification. Based on the classification result, it is determined whether the working person has violated the rules. If so, an alarm is issued. Otherwise, the next video is collected and judged.

[0164] An embodiment of the present application also provides a computer device, including a memory, a processor, and computer-executable instructions stored in the memory and executable on the processor, wherein the processor executes the method for identifying work violations that combines multi-spatial scale features.

[0165] An embodiment of the present application further provides a computer storage medium storing computer-executable instructions. When the computer-executable instructions are executed, the above-mentioned method for identifying work violations combining multi-spatial scale features is implemented.

[0166] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for identifying work violations by combining multi-spatial scale features, characterized by The following steps are involved: S1. Collect videos of multiple work scenes, crop the videos and adjust the resolution and frame rate, extract all image frames, and annotate key frames for actions; S2, divide the video data into training set and validation set; S3. Build a multi-spatial-scale action recognition neural network model, including a dense-residual backbone network, a multi-spatial-scale module, and a classification head. The output of the dense-residual backbone network is divided into three branches. One branch serves as the input of the first stage of the multi-spatial scale module. Another branch and the output of the multi-spatial scale module are fed into the ROI Align layer for bounding box alignment to obtain four sets of outputs. The third branch is concatenated with the output of the ROI Align layer. S4. Input the training samples into the multi-spatial-scale action recognition neural network model for supervised training, use the cross-entropy loss function, and solve the weights and bias parameters of the network model through iterative optimization; and use the samples in the validation set to verify the neural network model; S5. Collect the surveillance video of the operation site, select the segment to be identified, obtain the key frame, perform person detection on the key frame, and obtain the bounding box of the person in the video image; S6, inputting the video clips, key frames, and bounding boxes obtained in S5 into a multi-spatial-scale action recognition neural network model to perform behavior or action recognition, identify the person's behavior, and determine whether it is a violation; The specific steps of labeling key frames in S1 are: Use the yolov5 model to perform target detection on the key frames to obtain the target detection file in json format; Delete the non-human bounding boxes in the target detection file to obtain the human target detection file; Import the person target detection file and keyframes into the via annotation tool. By checking the checkbox, perform behavior annotation on each person bounding box one by one to obtain the person behavior annotation file. The dense-residual backbone network described in S3 consists of an input processing module, four convolutional submodules, and a downsampling module connected by dense-residual connections, and is used to extract spatiotemporal context features of video clips; The multi-spatial-scale feature module upsamples and fuses the outputs of each convolution module of the dense-residual backbone network in three stages to obtain multi-spatial-scale features. The spatiotemporal context features and multi-spatial scale features are each combined with the character bounding box to enter the ROI Align layer to obtain their respective corresponding character action features, and these features are spliced ​​to obtain the final character action features; The ROI Align layer performs the following operations: Get the bounding box position and size of the person in the video from the annotation file; The bounding box is scaled to match the spatiotemporal context features and multi-scale spatial features. Features are extracted based on the positions of the four corner points of the bounding box to obtain multi-scale character features. The spatiotemporal context features are concatenated with all the person features to obtain the context-single-person correlation features. The context-single-person association features are input into the classification head, and high-level semantic features are further extracted through convolution.

2. The method for identifying work violations by combining multi-spatial scale features according to claim 1, characterized in that: The classification head consists of three 3D convolutional layers, an MLP linear layer, and a SoftMax classifier, where the first two 3D convolutional layers use the ReLU activation function.

3. A computer device comprising a memory, a processor, and computer executable instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the instruction, the method for identifying work violations combining multi-spatial scale features as described in claim 1 or 2 is implemented.

4. A computer storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed, the method for identifying work violations combining multi-spatial scale features as described in claim 1 or 2 is implemented.

Images