Action recognition method based on skeleton key points and convolutional neural network

Abstract

The application discloses a motion recognition method based on skeleton key points and a convolutional neural network, which comprises the following steps: collecting a video; extracting two-dimensional posture key point coordinates of each person in each frame of image, performing heat map stacking, and obtaining a three-dimensional heat map; and finally, processing the three-dimensional heat map by using an improved slowonly network. The improved slowonly network is to remove the early downsampling operation in the original slowonly network, improve the time convolution module in the original slowonly network, and improve the channel attention mechanism in the original slowonly network. The output result obtained by processing by the improved slowonly network is the recognition result. The method can reduce the influence of the occlusion problem on the recognition accuracy without introducing additional parameters, and has certain application value in complex monitoring scenes.

Description

Technical Field

[0001] This invention relates to the field of human behavior recognition technology, specifically to an action recognition method based on skeletal key points and convolutional neural networks suitable for complex monitoring scenarios. Background Technology

[0002] Human motion recognition has long been a research hotspot in computer vision, with broad application prospects in fields such as intelligent surveillance, motion analysis, and human-computer interaction. Its aim is to extract motion information from videos and identify the main types of actions appearing in the video. Existing recognition methods employ different forms of feature representation, such as RGB frames, optical flow, and skeletal keypoints. Compared to other data types, skeletal data can more intuitively reflect the motion characteristics of the human body and is less affected by factors such as lighting changes, background complexity, and differences in human appearance, effectively avoiding noise interference. Therefore, motion recognition methods based on skeletal keypoints have received increasing attention in recent years.

[0003] Existing motion recognition methods based on skeletal data mainly employ the following networks: (1) Recurrent Neural Network (RNN); (2) Graph Convolutional Network (GCN); (3) Convolutional Neural Network (CNN). Among them:

[0004] Recurrent neural networks (RNNs) perform well in modeling time series, but they ignore the spatial structural information between key points in the skeleton, resulting in a weak ability to extract features in the spatial domain.

[0005] Graph convolutional networks represent skeletal keypoint data as a topological graph and perform graph convolution operations based on the constructed topological graph. Graph structures are richer in semantic information and are an effective representation of skeletal data. However, graph convolutional networks are more complex and their complexity is linearly related to the number of people being recognized. In addition, their recognition ability is greatly affected by the coordinate extraction effect and they are weak in modeling action information with a long time sequence.

[0006] Convolutional neural networks, due to their translation invariance, are able to effectively extract local features between skeletal key points when processing skeletal key point information, and have a strong ability to extract features in space. However, how to better extract features in the time dimension is a problem that needs to be considered. Summary of the Invention

[0007] Purpose of the invention: To provide a human motion recognition method suitable for complex scenarios, which can identify human motion in original videos and improve the recognition accuracy to a certain extent.

[0008] Technical solution: The action recognition method based on skeletal key points and convolutional neural networks adopts a slowonly network structure that removes the early downsampling operation, improves the temporal convolution module, and improves the channel attention mechanism to process the 3D heatmap that has been estimated and preprocessed.

[0009] An action recognition method based on skeletal keypoints and convolutional neural networks is characterized by first acquiring video; then extracting the coordinates of two-dimensional pose keypoints of each person in each frame of the image, stacking the heatmaps to obtain a three-dimensional heatmap; and finally using an improved slowonly network to process the three-dimensional heatmap.

[0010] The improved slowonly network removes the early downsampling operation from the original slowonly network, improves the temporal convolution module in the original slowonly network, and improves the channel attention mechanism in the original slowonly network.

[0011] The processing procedure for 3D heatmaps in the improved slowonly network is as follows:

[0012] 1) The 3D heatmap is constructed using convolutional layers of an improved slowonly network and the first to third ResNet layers;

[0013] 2) After dividing the input features into four equal parts along the channel dimension, they are then processed using an improved temporal convolution module. The steps are as follows:

[0014] 2.1) Convolve the first two parts directly using a 3×1×1 time convolution kernel and then output the result;

[0015] 2.2) The inputs of the last two convolutional parts respectively contain the output features of the first part and the input features of this part, that is, the third part contains the output features of the second part, and the fourth part contains the output features of the third part;

[0016] 2.3) After merging these four outputs, pass them sequentially through a 1×1×3 time convolutional kernel network and a 1×1×1 time convolutional kernel network before outputting the result.

[0017] 3) The output of step 2) is passed through the improved attention mechanism module, the steps are as follows;

[0018] 3.1) Soft pooling is used for feature compression, which transforms the two-dimensional feature channel into a real number with a global receptive field;

[0019] 3.2) Perform activation operations to generate weights for each feature channel;

[0020] 3.3) Multiply the normalized weights and the original input feature map channel by channel to generate a weighted feature map;

[0021] 4) Pass the output of step 3) through an average pooling layer to obtain the recognition result.

[0022] Among convolutional neural network-based methods, 3D-CNN uses three-dimensional convolution to model spatiotemporal features, and is a natural extension of 2D-CNN. Compared with 2D-CNN, it can capture temporal context information more effectively. The action recognition method proposed in this invention, without introducing additional parameters, is based on an improved 3D-CNN network and comprehensively utilizes image processing techniques such as pose estimation and 3D heatmap processing to reduce the impact of occlusion on recognition accuracy, and has certain application value in complex monitoring scenarios. Attached Figure Description

[0023] Figure 1 The network structure diagram for the improved slowonly algorithm;

[0024] Figure 2 A diagram of the improved temporal convolution module;

[0025] Figure 3 This is a diagram of the S-SEnet network structure. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0027] In this example, the action recognition method based on skeletal keypoints and convolutional neural networks consists of the following steps:

[0028] Step 1: Use a top-down pose estimator to extract the 2D pose of each person in each frame. Each frame has K keypoints, each with a position (xk, yk) and a confidence level of ck. Save the coordinates as (xk, yk, ck) and generate a joint pseudo-heatmap using the following formula.

[0029] The frame's height and width are H and W, respectively. By overlaying the time dimension, a three-dimensional heatmap J of K×T×H×W is obtained. kij As input to the network.

[0030]

[0031] In the formula, (i, j) represents the coordinates of a certain position in the image, and σ 2 It is adaptive variance.

[0032] Step 2: Sample the video using a sparse temporal sampling strategy. Divide the entire video into T equal segments, and then randomly select one frame from each segment to form T frames as input.

[0033] Step 3: Crop the 3D heatmap to reduce spatial redundancy.

[0034] Find the smallest bounding box that can cover all people in the video frame, crop the heatmap of each frame based on the found bounding box, and rescale it to the target size.

[0035] Step 4: As Figure 1 The 3D heatmap processed in steps 1 to 3 is then passed through the convolutional layers, resNet layer 1, resNet layer 2, and resNet layer 3 of the improved slowonly network.

[0036] Step 5: After dividing the input features into four equal parts along the channel dimension, the improved temporal convolution module is then applied, such as... Figure 2 .

[0037] Step 5.1) Convolve the first two parts directly using a 3×1×1 time convolution kernel and then output the result;

[0038] Step 5.2) The input of the last two parts of the convolution includes the output features of the previous part and the input features of this part;

[0039] Step 5.3) After merging these four parts, they are sequentially passed through a 1×1×3 temporal convolutional kernel network ( Figure 1 Improved res4) and 1×1×1 temporal convolutional kernel network ( Figure 1 The improved res5) output.

[0040] Step 6: Pass the output of Step 5 through the improved attention mechanism module ( Figure 1 (S-SEnet network in the middle), such as Figure 3 .

[0041] Step 6.1) Use soft pooling to compress features, turning the two-dimensional feature channel into a real number with a global receptive field;

[0042] Step 6.2) Perform the activation operation to generate weights for each feature channel;

[0043] Step 6.3) Multiply the normalized weights and the original input feature map channel by channel to generate the weighted feature map.

[0044] Step 7: Pass the output of Step 6 through an average pooling layer to obtain the recognition result.

[0045] Appendix Explanation

[0046] Table 1 compares the detection results of the improved slowonly network with the current mainstream skeleton-based action recognition network.

[0047] Table 1

[0048] method CS (Cross Subject) / % CV (Cross View) / % ST-GCN 81.5 88.3 AGCN 88.5 95.1 Shift-GCN 90.7 96.5 MS-G3D 91.5 96.2 This article (Improved slowonly network) 94.3 97.4 。

Claims

1. An action recognition method based on skeletal keypoints and convolutional neural networks, characterized by: First, video is captured; then, the coordinates of the two-dimensional pose key points of each person in each frame are extracted, and heatmaps are stacked to obtain a three-dimensional heatmap; finally, an improved slowonly network is used to process the three-dimensional heatmap. The improved slowonly network consists of a pooling layer and a immediately following resNet layer after removing the first convolutional layer from the original slowonly network; an improved temporal convolutional module from the original slowonly network; and an improved channel attention mechanism from the original slowonly network. The processing procedure for 3D heatmaps in the improved slowonly network is as follows: 1) The 3D heatmap is constructed using convolutional layers of an improved slowonly network and the first to third ResNet layers; 2) After dividing the input features into four equal parts along the channel dimension, they are then processed using an improved temporal convolution module. The steps are as follows: 2.1) Convolve the first two parts directly using a 3×1×1 time convolution kernel and then output the result; 2.2) The inputs of the last two convolutional parts respectively contain the output features of the first part and the input features of this part, that is, the third part contains the output features of the second part, and the fourth part contains the output features of the third part; 2.3) After merging these four outputs, pass them sequentially through a 1×1×3 time convolutional kernel network and a 1×1×1 time convolutional kernel network before outputting the result. 3) The output of step 2) is passed through the improved attention mechanism module, the steps are as follows; 3.1) Soft pooling is used for feature compression, which transforms the two-dimensional feature channel into a real number with a global receptive field; 3.2) Perform activation operations to generate weights for each feature channel; 3.3) Multiply the normalized weights and the original input feature map channel by channel to generate a weighted feature map; 4) Pass the output of step 3) through an average pooling layer to obtain the recognition result.

2. The action recognition method based on skeletal key points and convolutional neural networks according to claim 1, characterized in that: The three-dimensional heatmap is a three-dimensional heatmap that has undergone attitude estimation and preprocessing.

3. The action recognition method based on skeletal key points and convolutional neural networks according to claim 1, characterized in that: The method for obtaining a 3D heatmap is as follows: A top-down pose estimator is used to extract the 2D pose of each person in each frame; Each frame contains K key points for human pose, with each key point located at (xk, yk) and having a confidence level of ck. These key points are saved as coordinates (xk, yk, ck) and then used to generate joint pseudo-heatmaps, i.e., 3D heatmaps, as input to the improved slowonly network.

4. The action recognition method based on skeletal key points and convolutional neural networks according to claim 3, characterized in that: The method for generating joint pseudo-heatmaps is as follows: Let the height and width of the frame be H and W respectively, and overlay the time dimension T to obtain a three-dimensional heatmap J of size K×T×H×W. kij As input to the improved slowonly network, (i, j) represents the coordinates of a location in the image. It is adaptive variance: 。 5. The action recognition method based on skeletal key points and convolutional neural networks according to claim 1, characterized in that: Cropping the 3D heatmap reduces spatial redundancy.

6. The action recognition method based on skeletal key points and convolutional neural networks according to claim 1, characterized in that: The video is sampled using a sparse temporal sampling strategy: the entire video is divided into T segments on average, and then one frame is randomly selected from each segment to form T frames as input.

7. The action recognition method based on skeletal key points and convolutional neural networks according to claim 5, characterized in that: The method for cropping 3D heatmaps is as follows: find a bounding box with the smallest possible range that can cover all people in the video frame, crop each frame of the heatmap based on the found bounding box, and then rescale it to the target size.

Images