Monitoring video action recognition method based on improved PoseC3D algorithm

Abstract

The invention discloses a monitoring video action recognition method based on an improved PoseC3D algorithm. The method mainly comprises the following steps: extracting a 2D skeleton through a target detection algorithm and a human body posture estimation algorithm according to a pre-segmented monitoring video frame sequence; and for the obtained 2D skeleton, performing a series of pipe line data preprocessing to obtain a 3D heat map body, and then inputting the 3D heat map body into a three-dimensional convolution model added with a mixed attention mechanism to obtain the type of action recognition. According to the method, improvement is carried out from the quality angle of obtaining the 2D skeleton, a target detection algorithm and a human body posture estimation algorithm are replaced by an end-to-end single-stage algorithm, the 2D skeleton is obtained by utilizing a YOLOX-Pose algorithm, and the accuracy of action recognition is further improved by fusing PoseC3D into a mixed attention mechanism.

Description

Technical Field

[0001] The invention belongs to the field of video monitoring, and in particular to a monitoring video action recognition method based on an improved PoseC3D algorithm. Background Art

[0002] With the continuous improvement of people's safety awareness and the widespread application of monitoring equipment, it is common to observe the unsafe production behaviors of people in the workshop through video monitoring. However, the fatigue caused by long-term observation of monitoring videos by video monitoring personnel will reduce their work efficiency, especially now that security monitoring is very developed. If an elderly person falls or a group fights under the camera, these things can be automatically detected using video action recognition algorithms. The basis of video action recognition is human posture estimation. First, the human skeleton joint features are constructed based on key points, and then his movements are judged based on the human skeleton joint features. This is very important. The application scenarios of human posture estimation are very wide, including action detection, sports guidance, acupoint positioning, body language understanding, gait analysis, motion capture, human-computer interaction, and now the very popular metaverse, digital human, VR, AR, and virtual anchors all use human posture estimation tasks. Therefore, the study of human movements in videos has great academic value and application value.

[0003] As the main component of people in smart workshops, workers have important responsibilities such as operating equipment, transporting materials, and optimizing operations. In the smart manufacturing process where the human-machine environment frequently interacts, safe and reliable operation and interaction is an important problem. According to statistics, 95% of industrial production safety accidents in the country are caused by unsafe operation processes or behaviors of workers. The traditional video surveillance unsafe production behavior recognition method mainly relies on manual feature extraction. Not only is the process very cumbersome, but the manually set feature extractor has poor robustness and is not necessarily scientific. The development of deep learning technology provides a direction for the recognition of unsafe production behaviors in intelligent workshops, especially as people are a key link in the five elements of workshop management (people, machines, materials, methods, and environment). How to quickly and accurately identify unsafe production behaviors in video surveillance in real time and issue corresponding warnings, and timely prevent safety accidents in advance, is of great research significance. Summary of the invention

[0004] In order to better analyze the human body movements in surveillance videos and issue timely warnings when certain unsafe movements occur, the present invention provides a surveillance video movement recognition method based on an improved PoseC3D algorithm, which can quickly identify the human body movements in the video, thereby providing support for the safety of personnel in the surveillance video.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0006] A surveillance video action recognition method based on an improved PoseC3D algorithm comprises the following steps:

[0007] Step 1: Obtain surveillance video and extract it into RGB frame sequences in sequence;

[0008] Step 2: Based on the end-to-end YOLOX-Pose algorithm, obtain 2D skeleton images and information from the RGB frame sequence to obtain a sequence data set of 17 key points of the human body {2D coordinates, confidence scores, action labels};

[0009] Step 3: Process the 2D skeleton image through pipeline data to obtain a 3D thermal image.

[0010] Step 4: Improve the PoseC3D algorithm: integrate the hybrid attention mechanism into the 3D convolutional model; use the 3D heat map to iteratively train the improved 3D convolutional model, so that it can fit the learning ability to distinguish the corresponding action type according to the pseudo heat map, and obtain the ideal 3D convolutional model when the fitting error converges;

[0011] Step 5: Collect surveillance video or RGB frame sequence in real time, convert it into a 3D thermal image and input it into an ideal 3D convolution model to automatically identify the output action type.

[0012] The 2D skeleton information includes: video identifier frame_dir, number of video frames total_frames, video frame shape img_shape, original video frame shape original_shape, action label label; key point keypoint, key point confidence score keypoint_score.

[0013] The pipeline data processing of the 2D skeleton image includes:

[0014] Uniformly sample the 2D skeleton image frame sequence and extract the sampling frame;

[0015] Load the sample frame and decode it. According to the index value of the current sample frame in the original frame sequence, map the corresponding pose as the action label of the current sample frame.

[0016] The region of interest is gradually narrowed down according to the coordinates of the key points to obtain a tight bounding box around all the key points of the human body in each frame;

[0017] Adjust image size and crop images;

[0018] Perform flip operation on the image to expand the data and obtain an enhanced image set;

[0019] According to the coordinates of the key points of the human body (x k ,y k ) and the confidence score c k The enhanced image set is calculated to generate pseudo heat maps, including two forms: key point heat map and limb heat map;

[0020] Convert the pseudo heat map resolution dimension to the set value;

[0021] Convert the pseudo heat map and the corresponding action type label to the Tensor data type to obtain a 3D heat map body.

[0022] The key point heat map and limb heat map are based on (x k ,y k ) as the center, c k is the maximum value, calculated separately;

[0023] a. Generate a keypoint heatmap using Gaussian distribution, defined by the following formula:

[0024]

[0025] Where σ represents the standard deviation of the Gaussian distribution;

[0026] b. Generate a limb heatmap between two key points using a Gaussian distribution, defined by the following formula:

[0027]

[0028] Where D(·) is the distance calculation function, which is used to calculate the distance between any point (x, y) on the skeleton and the skeleton line segment s[a k ,b k ]; a k ,b k are the key points at both ends of the bone; c ak ,c bk for a k ,b k Confidence of two points.

[0029] The improved three-dimensional convolution model includes: a convolution layer, several stacked ResNet layers, a global average pooling layer GAP, and a fully connected layer FC;

[0030] The convolutional layer is used for preliminary feature extraction and data processing;

[0031] The stacked ResNet layers are used to capture more complex features and improve the model's expressiveness;

[0032] The global average pooling layer GAP is used to calculate the average value of each feature map and reduce the dimension of the feature map;

[0033] The fully connected layer FC is used to realize complex nonlinear mapping by combining different features, and finally perform action recognition.

[0034] A hybrid attention mechanism CBAM module is added to the backbone network in the ResNet layer to fuse the temporal channel and the spatial channel, thereby reconstructing the features and outputting them.

[0035] The present invention has the following beneficial effects and advantages:

[0036] 1. Visualization: When reasoning about actions, the visualization function can identify specific actions and output action labels.

[0037] 2. Simplicity: Acquiring 2D skeletons in an end-to-end manner further simplifies the process of acquiring skeleton modal data.

[0038] 3. Accuracy: The recognition accuracy is further improved through the improved PoseC3D algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 To improve the overall framework diagram of PoseC3D;

[0040] Figure 2 This is a schematic diagram of the improved RestNetLayer network structure;

[0041] Figure 3 Schematic diagram of the attention mechanism CBAM module;

[0042] Figure 4 This is a schematic diagram of recognition accuracy data. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0045] The invention belongs to the field of video monitoring and can be used to identify and detect industrial workshops or other public places.

[0046] Figure 1 The flowchart is a method for surveillance video action recognition based on the improved PoseC3D algorithm, which includes the following steps:

[0047] Step 1: Data collection on unsafe production behaviors of workshop workers. The unsafe production behaviors of workers are monitored by the surveillance video equipment deployed in the workshop, and the data is obtained by summarizing the monitoring history information related to the unsafe production behavior information.

[0048] Step 2: RGB video data preprocessing. Set the frame rate to 30 frames / s and the resolution to 1920*1080.

[0049] Step 3: Divide the training set and test set by person ID, with 75% as the training set and 25% as the test set.

[0050] Step 4: Extract RGB frame sequence from the acquired RGB video.

[0051] Step 5: For the extracted RGB frame sequence, the 2D skeleton is obtained through the end-to-end YOLOX-Pose algorithm, and finally relevant information such as key points, confidence scores, action type labels, etc. can be obtained.

[0052] The present invention improves the original end-to-end YOLOX-Pose algorithm by modifying the original serial two-stage target detection and key point detection to simultaneously perform target detection and key point detection in one stage, obtaining 2D skeleton images and 2D skeleton information, and outputting key point position coordinates and key point confidence scores.

[0053] The 2D skeleton information includes 17 key points related to the human body: video identifier frame_dir, total_frames, shape of the video frame img_shape, shape of the original video frame original_shape, action label label; key point keypoint, key point confidence score keypoint_score. The shape of the video frame is the video resolution size. {Key point, confidence score, action type label} is used as the standard data set, {key point, confidence score} is used as input, and {action type label} is used as the predicted output. The model is used for fitting learning so that the model can predict unknown actions.

[0054] The action labels use 60 action labels in the NTU RGB+D public dataset of the ROSE Laboratory of Nanyang Technological University as a benchmark, including kicking things, playing with mobile phones, sneezing, falling, headaches, etc.

[0055] Step 6: Use the 2D skeleton to further generate a 3D heat map body through Gaussian distribution, and finally obtain a 3D heat map body data format in which pseudo heat maps are stacked frame by frame.

[0056] The improved PoseC3D algorithm comprises the following steps:

[0057] a) After a series of pipeline data preprocessing, a 3D thermal map is obtained.

[0058] 1) Uniformly sample the frame sequence in the video, divide the entire video duration into clip_len equal time periods, and randomly sample a frame in each time period;

[0059] 2) Load the sample frame and decode it. According to the index value of the current sample frame in the original frame sequence, map the corresponding pose as the action label of the sample frame.

[0060] 3) Further narrow down the region of interest based on the coordinates of the key points, find a tight bounding box around all human key points in each frame, and then expand the tight box by a given filling ratio.

[0061] 4) Resize the image resolution dimension to a specific size.

[0062] 5) Crop the image with random area ratio and aspect ratio.

[0063] 6) Resize the image to a specific size.

[0064] 7) Flip the input image with a certain probability (such as 80%), reverse the order of rows and columns of elements in a given frame image imgs in a specific direction, and save the flipped image imgs' to obtain a set of image frames after data enhancement.

[0065] 8) Calculate the fused image set according to the coordinates of the key points of the human body and the confidence scores to generate a pseudo heat map of the fused image, including the following two forms. They are in the form of (x k ,y k , c k ) format, where (x k ,y k ) is the predicted key point coordinate, c k is the confidence of the prediction of this key point.

[0066] With (x k ,y k ) as the center, with c k is the maximum value, and the corresponding key point heat map is generated using a Gaussian distribution, defined by the following formula:

[0067]

[0068] Here, σ represents the standard deviation of the Gaussian distribution.

[0069] Similarly, the limb heatmap between two keypoints can also be generated using a Gaussian distribution, defined by the following formula:

[0070]

[0071] Where D(·) is the distance calculation function, which is used to calculate the distance between the point (x, y) and the skeleton line segment s[a k ,b k ]; a k ,b k are the key points at both ends of the bone; c ak ,c bk for a k ,b k Confidence of two points.

[0072] 9) Format the pseudo heat map shape into the given required input format input_format;

[0073] 10) Collect data imgs and metadata img_metas related to the specific action recognition task from the loader.

[0074] 11) Convert pseudo heat map data and corresponding action type labels into Tensor data types. The standard data set of the 3D heat map volume includes {key points, confidence scores, action type labels}.

[0075] b) Input the acquired 3D heat map volume into the model of the improved PoseC3D algorithm.

[0076] Step 7: The 3D heat map obtained in step 6 can be further input into the 3D convolutional model fused with the attention mechanism for training to obtain the ideal model weights for pre-training.

[0077] The standard data set for 3D heatmaps includes {key points, confidence scores, action type labels}. The 3D convolutional model with attention mechanism takes {key points, confidence scores} as input and {action type labels} as prediction output for fitting learning. When the prediction accuracy meets the threshold requirement, the ideal model is obtained. The ideal model is used to predict unknown actions.

[0078] The improved 3D convolution model includes: convolution layer, several stacked ResNet layers, global average pooling layer GAP, and fully connected layer FC;

[0079] The convolution layer is responsible for preliminary feature extraction and data processing, laying the foundation for subsequent learning of complex features.

[0080] Stacked ResNet layers are used to capture more complex features and patterns, thereby improving the expressiveness of the model.

[0081] The global average pooling layer GAP is used to calculate the average value for each feature map and compress the spatial dimensions (width and height) of each feature map into a single value, thereby significantly reducing the dimension of the data.

[0082] The fully connected layer FC is used to achieve complex nonlinear mapping by combining different features, ultimately helping the model to perform action recognition.

[0083] in, Figure 2 The figure is a schematic diagram of the improved RestNetLayer network structure of the present invention. In this method, a hybrid attention mechanism CBAM module is added to the backbone network of the stacked ResNet layer to fully extract features and achieve improvement. Specifically, the CBAM module is inserted into each bottleneck block of the stacked ResNet layer. The bottleneck structure after embedding CBAM is as follows: Figure 2 As shown in the figure. The 3×1×1 convolution layer is used to capture local features in the time dimension and can effectively extract dynamic information from time series data. The combination of the BN layer (Batch Normalization layer) and the ReLU (Rectified Linear Unit) activation function enables the model to learn complex features and reduce the risk of overfitting by introducing nonlinearity. Then the 1×3×3 convolution layer uses 3×3 convolutions in the height and width directions to effectively extract local spatial features. The 1×1×1 convolution layer is used to reduce the number of parameters and reduce the dimension of the data. The BN layer can reduce the risk of overfitting to a certain extent. Adding CBAM enhances the model's attention to important features and suppresses unimportant features through channel and spatial attention mechanisms. Combining the original input features with the features processed by CBAM through short connections helps the gradient flow more smoothly during back propagation and alleviates the gradient vanishing problem. The ReLU activation function enables the model to learn complex functional relationships by introducing nonlinear transformations, thereby enhancing the model's expressive power.

[0084] Figure 3 Schematic diagram of the attention mechanism CBAM module; the hybrid attention mechanism is divided into two sub-modules:

[0085] a) Channel Attention Module (CAM)

[0086] The channel attention module compresses the feature map and performs a series of operations to learn the relationship between channels in the feature map. In the process of compressing the feature map, the feature map is aggregated by maximum pooling and average pooling. The pooled feature map is input into the Multilayer Perceptron (MLP). MLP is a neural network structure composed of multiple fully connected layers that can learn complex relationships between features. In the channel attention module, MLP further processes the feature map and learns the dependencies between channels. By adding the output of MLP element by element, the feature information of multiple fully connected layers is comprehensively considered. The interaction between channels can be accurately captured to obtain more representative channel attention features. Finally, the feature information is activated by applying the sigmoid(x) function and mapped to the range of 0 to 1. This step obtains the channel attention feature. Channels with larger values ​​represent higher importance in the feature map, while channels with smaller values ​​represent lower importance in the feature map. The formula is as follows:

[0087]

[0088] Where:

[0089] W a , W b ——Two weight dimensions of MLP;

[0090] D——Pooling features;

[0091] b) Spatial Attention Module (SAM)

[0092] The spatial attention mechanism module compresses in the channel dimension to reduce redundant information and storage costs. In this process, maximum pooling and average pooling are used to aggregate channel information and compress channel feature information. This extracts significant features and reduces computational costs. The spatial attention module integrates feature information by connecting the pooled features and generates compressed channel features. This process can capture richer target information and improve the model's recognition accuracy of the target. The sigmoid(x) function is applied to the pooled features to generate spatial attention features. The spatial attention module improves the accuracy of the model in target detection tasks and extracts more significant target features by compressing channel information and applying maximum pooling, average pooling, connection operations, and the sigmoid(x) function. The formula is as follows:

[0093]

[0094] Where:

[0095] D——Pooling features.

[0096] Step 8: Use the pre-trained ideal model weights obtained in step 7 to test on the test set to obtain the recognition accuracy.

[0097] Step 9: Collect surveillance video or RGB frame sequence in real time, convert it into a 3D heat map and input it into an ideal 3D convolution model to automatically identify the output action type.

[0098] Figure 4 This is a diagram of the recognition accuracy data. During the training process, the loss continued to decrease, and the recognition accuracy on the validation set gradually stabilized. Finally, the recognition accuracy on the test set reached 93.1%. Compared with the case without the CBAM module, the recognition accuracy increased by 1.9%.

[0099] In addition, it can be found from Table 1 that after adding the CBAM module, the recognition accuracy on the test set can be improved accordingly.

[0100] Table 1

[0101] Recognition model Recognition accuracy / % ST-GCN 81.5 2S-AGCN 88.5 PoseC3D 91.2 CBAM-PoseC3D 93.1

[0102] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A surveillance video action recognition method based on an improved PoseC3D algorithm, characterized in that: The following steps are involved: Step 1: Obtain surveillance video and extract it into RGB frame sequences in sequence; Step 2: Based on the end-to-end YOLOX-Pose algorithm, obtain 2D skeleton images and information from the RGB frame sequence to obtain a sequence data set of 17 key points of the human body {2D coordinates, confidence scores, action labels}; Step 3: Process the 2D skeleton image through pipeline data to obtain a 3D thermal image. Step 4: Improve the PoseC3D algorithm: integrate the hybrid attention mechanism into the 3D convolutional model; use the 3D heat map to iteratively train the improved 3D convolutional model, so that it can fit the learning ability to distinguish the corresponding action type according to the pseudo heat map, and obtain the ideal 3D convolutional model when the fitting error converges; Step 5: Collect surveillance video or RGB frame sequence in real time, convert it into a 3D thermal image and input it into an ideal 3D convolution model to automatically identify the output action type.

2. According to claim 1, a surveillance video action recognition method based on an improved PoseC3D algorithm is characterized in that: The 2D skeleton information includes: video identifier frame_dir, number of video frames total_frames, video frame shape img_shape, original video frame shape original_shape, action label label; key point keypoint, key point confidence score keypoint_score.

3. The surveillance video action recognition method based on the improved PoseC3D algorithm according to claim 1 is characterized in that: The pipeline data processing of the 2D skeleton image includes: Uniformly sample the 2D skeleton image frame sequence and extract the sampling frame; Load the sample frame and decode it. According to the index value of the current sample frame in the original frame sequence, map the corresponding pose as the action label of the current sample frame. The region of interest is gradually narrowed down according to the coordinates of the key points to obtain a tight bounding box around all the key points of the human body in each frame; Adjust image size and crop images; Perform flip operation on the image to expand the data and obtain an enhanced image set; According to the coordinates of the key points of the human body (x k ,y k ) and the confidence score c k The enhanced image set is calculated to generate pseudo heat maps, including two forms: key point heat map and limb heat map; Convert the pseudo heat map resolution dimension to the set value; Convert the pseudo heat map and the corresponding action type label to the Tensor data type to obtain a 3D heat map body.

4. The method for surveillance video action recognition based on the improved PoseC3D algorithm according to claim 3 is characterized in that: The key point heat map and limb heat map are based on (x k ,y k ) as the center, c k is the maximum value, calculated separately; a. Generate a keypoint heatmap using Gaussian distribution, defined by the following formula: Where σ represents the standard deviation of the Gaussian distribution; b. Generate a limb heatmap between two key points using a Gaussian distribution, defined by the following formula: Where D(·) is the distance calculation function, which is used to calculate the distance between any point (x, y) on the skeleton and the skeleton line segment s[a k ,b k ]; a k ,b k are the key points at both ends of the bone; c ak ,c bk for a k ,b k Confidence of two points.

5. The method for surveillance video action recognition based on the improved PoseC3D algorithm according to claim 1, characterized in that: The improved three-dimensional convolution model includes: a convolution layer, several stacked ResNet layers, a global average pooling layer GAP, and a fully connected layer FC; The convolutional layer is used for preliminary feature extraction and data processing; The stacked ResNet layers are used to capture more complex features and improve the model's expressiveness; The global average pooling layer GAP is used to calculate the average value of each feature map and reduce the dimension of the feature map; The fully connected layer FC is used to realize complex nonlinear mapping by combining different features, and finally perform action recognition.

6. The method for surveillance video action recognition based on the improved PoseC3D algorithm according to claim 5, characterized in that: A hybrid attention mechanism CBAM module is added to the backbone network in the ResNet layer to fuse the temporal channel and the spatial channel, thereby reconstructing the features and outputting them.

Images