A lightweight behavior recognition method based on skeleton data

Abstract

The application discloses a light behavior recognition method based on skeleton data, comprising the following steps: 1) preprocessing the obtained behavior data set; 2) adopting a double-branch structure to fuse spatial and time sequence information, and through the idea of dimension conversion, respectively placing time sequence T and joint node N in respective channel dimensions, and extracting behavior features through an established efficient space-time graph convolution model. For the efficient space-time graph convolution model, 1*1 convolution and depth separable convolution are introduced to reduce the calculation cost of parameter adjustment and model inference, and residual links are introduced to reduce the difficulty of model training. After setting model parameters, a loss function, an optimization function and the like, model weight initialization is performed, data is loaded, a model optimizer is used, and the network is trained with a training set. 3) The trained skeleton behavior recognition model is used to predict the data set, and corresponding action categories are given. The method can effectively reduce the model complexity on the basis of ensuring a high recognition accuracy, and has a wide application prospect.

Description

Technical Field

[0001] This invention relates to the field of behavior recognition technology, and in particular to a lightweight behavior recognition method based on skeletal data. Background Technology

[0002] Currently, human motion recognition technology has wide applications in fields such as intelligent surveillance, human-computer interaction, and video retrieval. Skeleton data can be viewed as a topological graph, typically obtained from video sequences by human pose estimation algorithms. Existing research indicates that simply representing skeleton data as vector sequences processed by RNNs or 2D / 3D data processed by CNNs cannot fully simulate the complex spatiotemporal correlations of body joints. Inspired by the booming development of graph-based methods, many human motion recognition methods based on GNNs and GCNs have been proposed. Significant progress has been made in treating skeleton data as a graph structure of edges and nodes, and modeling the human skeleton as a spatiotemporal graph using convolutional networks.

[0003] However, graph convolution-based methods generally suffer from two problems. First, the receptive fields of spatial and temporal graphs are not flexible enough, limiting their feature representation capabilities. Second, over-parameterization of the model leads to high computational costs, which affects its practical application in real-world scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a lightweight behavior recognition method based on skeletal data, which classifies and recognizes behaviors through data acquisition, preprocessing, and a lightweight deep learning model for practical application.

[0005] The technical solution adopted by this invention to achieve the above objectives is: a lightweight behavior recognition method based on skeletal data, used to identify the behavior of a subject to be detected in a scene with unknown behavior, the method comprising the following steps:

[0006] Step S1: Obtain action recognition scene data, preprocess and label it, establish a {action label-behavior feature image pair} dataset, and divide it into training set and validation set;

[0007] Step S2: Establish an efficient spatiotemporal graph convolutional network based on skeletal data, set the network and training parameters, use behavioral features in the training set as input and action labels as output to iteratively fit and train the network model, adjust the network parameters through backpropagation, and then use validation set data to verify the model accuracy to obtain an ideal model for recognizing and predicting unknown behaviors.

[0008] Step S3: Collect scene data containing unknown behaviors on site, preprocess the data to obtain joint joint position features, input them into the trained skeletal behavior recognition network to predict the behavior, and output the action category of the subject to be detected in the scene.

[0009] The scene data is either video or a single-frame image.

[0010] The behavior recognition image data is either cross-view and cross-subject images of the scene to be detected captured by a camera, or an open-source behavior recognition dataset with subject action labels of a known scene.

[0011] The preprocessing and labeling includes processing the cross-viewpoint and cross-subject images of the scene to be detected captured by the camera as follows:

[0012] S1.1 Extract the absolute position data of the joints of the human skeleton using skeletal detection methods;

[0013] S1.2 Calculate the relative position data between the joints of the subject to be detected; label the actions.

[0014] S1.3 Calculate the joint joint position features: the concatenated vector of the absolute and relative positions of each joint;

[0015] S1.4. Establish a dataset of {action label-behavioral feature pairs} as input for the next step of the network model.

[0016] The efficient spatiotemporal graph convolutional network based on skeletal data includes:

[0017] a) BatchNorm layer, used to accelerate network convergence;

[0018] b) Initialize the spatiotemporal graph convolutional module InitialGCN Block, which is used to transform the joint key point location features;

[0019] c) Spatiotemporal extraction module, which is used to place the temporal information of bones and the joint features in the input features into the temporal and spatial channel dimensions of the network, respectively, to obtain features in different dimensions: temporal information Transpose(T,C,N) and joint features Transpose(N,T,C);

[0020] d) A dual-branch high-efficiency convolution module is used to further extract spatiotemporal features from features of different dimensions to obtain spatiotemporal feature maps with enhanced depths.

[0021] e) Fusion operation, used to concatenate and fuse the bi-branch spatiotemporal feature maps after feature extraction;

[0022] f) The mainstream branch of efficient convolutional networks, used to further extract features from the fused feature map;

[0023] g) Global average pooling layer, used to operate on the output feature map to obtain feature vectors representing the behavior;

[0024] g) Fully connected layer, used to predict and identify the action category of the subject to be detected in unknown scene data based on feature vectors.

[0025] The dual-branch high-efficiency convolutional module includes a temporal channel and a spatial channel; the spatial channel includes multiple stacked depthwise separable graph convolutional layer modules; the temporal channel includes multiple stacked depthwise separable convolutional layer modules.

[0026] The mainstream branch of the efficient convolutional network includes multiple stacked depthwise separable graph convolutional layer modules.

[0027] The network and training parameters settings include:

[0028] Set model parameters: maximum training epochs, initial learning rate, Swish activation function, cross-entropy loss function, momentum random descent as the optimization function, and top-1 and top-5 metrics for accuracy.

[0029] Weight initialization, loading of data, model, and optimizer, and end-to-end training.

[0030] The beneficial effects and advantages of this invention are as follows:

[0031] 1. This invention provides a lightweight behavior recognition method based on skeletal data, which achieves top-1 performance of 84.6% and 92.5% in behavior recognition accuracy on the NTU 60 large dataset;

[0032] 2. The method of the present invention significantly reduces the number of model parameters and the algorithm complexity, which is beneficial to the practical application of skeleton-based action recognition. Attached Figure Description

[0033] Figure 1 This is a flowchart of the behavior recognition method based on skeletal data of the present invention;

[0034] Figure 2 It is a module detail diagram of the efficient spatiotemporal graph convolution;

[0035] Figure 3 It is an infographic of 25 skeletal joints. Detailed Implementation

[0036] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0038] Please see Figure 1 The embodiments of this invention include:

[0039] A lightweight behavior recognition method based on skeletal data includes the following steps:

[0040] S1: Preprocess the acquired behavior recognition dataset. A large NTU-RGB-D dataset is selected as the behavior recognition dataset. This dataset provides 3D skeletal coordinates, including two references: cross-viewpoints and cross-subjects. Cross-viewpoints are scene images containing multiple subjects captured by cameras from different perspectives. Cross-subjects are scene images containing multiple subjects, where the actions of multiple subjects are independent or there is interaction between them. Specific steps include:

[0041] S1.1: The 25 skeletal joints in the acquired behavior recognition dataset are labeled with the numbers 0-24;

[0042] S1.2: Based on the human body structure, determine the connections between the key bones and calculate the relative position coordinates between the joints; connect the absolute and relative position coordinates of each joint to obtain the joint position features of a single body;

[0043] S2: Input the joint joint coordinates into the established efficient spatiotemporal graph convolutional network, optimize the network parameters, and obtain an efficient spatiotemporal convolutional network model based on skeletal data; specific steps include:

[0044] S2.1: The BatchNorm layer is used to batch normalize the coordinates of the joint joint points, thereby accelerating network convergence;

[0045] S2.2: Input the normalized joint keypoint coordinate data into the initialization spatiotemporal graph convolution module proposed in the existing ST-GCN algorithm. Please refer to [link to relevant documentation]. Figure 2 (a) Convolution is performed using an adjacency matrix established based on the human body's topology to perform multi-layer spatiotemporal graph convolution operations on the input data, extracting the main action features. The initialization spatiotemporal graph convolution module includes a BatchNorm layer, an SGC module, and a TC module.

[0046] S2.3: Next, the input main action features are transformed in dimension. The temporal information Transpose(T,C,N) and the keypoint features Transpose(N,T,C) are placed in their respective channel dimensions of the network to obtain features of different dimensions. Where T is the number of frames, C is the number of channels, and N is the number of keypoints;

[0047] S2.4: Temporal Branch of Efficient Convolutional Networks: Temporal information Transpose(T,C,N) is placed in the feature map along the temporal channel dimension as input; firstly, a 1*1 convolutional (1*1 Conv) layer is used to reduce the temporal dimension. Secondly, two advanced depthwise separable graph convolutional layers are employed, such as... Figure 2 As shown in (b), the model is composed of ordered stacked depth map convolutions (Depth_SimpGCN) and pointwise convolutions (Point_Conv), and residual connections (adding residual branches to the final output of 1*1 convolutions) are introduced to reduce the difficulty of model training. In the figure, the two numbers in each module represent the number of input Cin and output Cout channels, respectively, where Crd = Cin / 2, and / 2 indicates a stride of 2.

[0048] S2.5: Spatial Branch of Efficient Convolutional Networks: The keypoint information Transpose(N,T,C) is placed in the feature map along the spatial channel dimension and used as input to another branch. It also employs two advanced depthwise separable convolutional layers and introduces residual links at the beginning and end. (See [link to previous section]). Figure 2 (c) This model is composed of stacked standard depthwise convolutions (Depth_Conv) and pointwise convolutions (Point_Conv);

[0049] S2.6: The extracted bi-branch feature maps are fused through a concatenation operation;

[0050] S2.5: Input the dual-branch joint feature map into two modules composed of advanced depthwise separable map convolutional layers to extract temporal features again;

[0051] S2.6: Perform a global averaging (GAP) operation on the output feature map to obtain the feature vector;

[0052] S2.7: After passing through the final fully connected layer (FC), the predicted action category and the main action regression box are obtained.

[0053] S3: After setting the model parameters, loss function, optimization function, etc., the training set data is input into the network model in S2. After initializing the model weights, loading the data and model optimizer, end-to-end training is performed to obtain an ideal model for skeletal behavior recognition.

[0054] S3.1: Set model parameters. This includes setting the maximum training epochs to 70, the initial learning rate to 0.1, selecting the Swish function as the activation function, selecting cross-entropy as the loss function, setting momentum random descent as the optimization function, and using top-1 and top-5 metrics to measure accuracy.

[0055] S3.2: Weight initialization, loading data, model, and optimizer, and performing end-to-end training.

[0056] S4: Use the trained skeletal behavior recognition model to predict the dataset and give the corresponding action category.

[0057] The NTU RGB+D 60 dataset was released by Nanyang Technological University in Singapore in 2016. This dataset consists of 60 behaviors performed by 40 participants aged 10 to 35 years. The specific behavior labels are shown in Table 1 below. The behaviors in this dataset are divided into three categories: 1) 40 daily behaviors; 2) 9 health-related behaviors; and 3) 11 interactive behaviors. This dataset utilizes a Kinect v2 camera to obtain information on 25 skeletal joints, the distribution of which is illustrated in the diagram below. Figure 3 As shown.

[0058] Table 1

[0059]

[0060] Our method is compared with current classic and state-of-the-art algorithms on the X-sub and X-view benchmarks of the NTU 60 dataset to achieve the same recognition accuracy. The results are shown in Table 2. It can be seen that our model achieves the best performance of 84.6% and 92.5% on the X-Sub and X-View benchmarks, respectively, outperforming other algorithms. Two typical methods are worth noting. The first is the ST-GCN algorithm, currently the most popular skeleton-based action recognition backbone model. Compared to ST-GCN, our method leads by approximately 3% and 4% on the X-sub and X-view benchmark datasets. The second is NewPartition-STGCN, a recent improvement based on ST-GCN. Our method achieves approximately 1% higher accuracy than NewPartition-STGCN on the NTU 60 benchmark. These results demonstrate that our proposed method is a competitively performing algorithm compared to the comparison methods.

[0061] Table 2

[0062] HBRNN

[16] 59.1 64.0 ST-LSTM

[17] 69.2 77.7 STA-LSTM

[18] 73.4 81.2 HCN

[19] 86.5 91.1 ST-GCN[2] 81.5 88.3 SR-TSL

[20] 84.8 92.4 Motif-STGCN

[21] 84.2 90.2 New Partition-STGCN[8] 83.7 91.6 Our method 84.6 92.5

[0063] To verify the efficiency of the model, our method was compared with other methods on the NTU 60 dataset in terms of accuracy and model complexity (FLOPs and number of parameters). The experimental results are shown in Table 3. The ratios following FLOPs and number of parameters represent the ratios between other models and our proposed model.

[0064] It can be seen that there is a significant gap in efficiency between the previous model and the model in this paper. Compared with the first GCN baseline for skeleton-based action recognition (i.e., ST-GCN

[11] ), our accuracy is improved by 3-4%, the number of FLOPs is reduced by 17.18 times, and the number of parameters is reduced by 13.48 times. SR-TSL achieves the same accuracy as the model in this paper, but its number of trainable parameters is much larger, about 82.6 times that of the model in this paper. The results clearly show that the proposed method brings significant improvements in both model accuracy and complexity, which will be beneficial to the practical application of skeleton-based action recognition.

[0065] Table 3

[0066]

[0067] This invention is in Figure 2 (b) The network module features a newly proposed structure: Considering the balance between the excellent learning ability of spatial graph convolution on skeleton data and computational cost, this invention introduces spatial graph convolution (GCN) into the deep convolutional layer, simplifies the model parameters, reduces the parameters that adjust the correlation between key points in the graph convolution operation, and adds residual branches. This enables the localization of unknown behaviors and action recognition.

[0068] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A lightweight behavior recognition method based on skeletal data, characterized in that, Includes the following steps: Step S1: Obtain action recognition scene data, preprocess and label it, establish a {action label-behavior feature image pair} dataset, and divide it into training set and validation set; Step S2: Establish a spatiotemporal graph convolutional network based on skeletal data, set the network and training parameters, use behavioral features in the training set as input and action labels as output to iteratively fit and train the network model, adjust the network parameters through backpropagation, and then use validation set data to verify the model accuracy to obtain an ideal model for recognizing and predicting unknown behaviors. The spatiotemporal graph convolutional network based on skeletal data includes: a) BatchNorm layer, used to accelerate network convergence; b) Initialize the spatiotemporal graph convolutional module InitialGCN Block to perform feature transformation on the joint key point location features; c) Spatiotemporal extraction module, which is used to place the temporal information of bones and the joint features in the input features into the temporal and spatial channel dimensions of the network, respectively, to obtain features in different dimensions: temporal information Transpose(T,C,N) and joint features Transpose(N,T,C). d) A dual-branch convolution module is used to further extract spatiotemporal features from features of different dimensions to obtain spatiotemporal feature maps with enhanced depths. e) Fusion operation, used to concatenate and fuse the bi-branch spatiotemporal feature maps after feature extraction; f) The mainstream branch of convolutional networks, used to further extract features from the fused feature map; g) Global average pooling layer, used to operate on the output feature map to obtain feature vectors representing the behavior; g) Fully connected layer, used to predict and identify the action category of the subject to be detected in unknown scene data based on feature vectors; Step S3: Collect scene data containing unknown behaviors on site, preprocess the data to obtain joint joint position features, input them into the trained skeletal behavior recognition network to predict the behavior, and output the action category of the subject to be detected in the scene.

2. The lightweight behavior recognition method based on skeletal data according to claim 1, characterized in that, The scene data is either video or a single-frame image.

3. The lightweight behavior recognition method based on skeletal data according to claim 1, characterized in that, The behavior recognition image data is either cross-view and cross-subject images of the scene to be detected captured by a camera, or an open-source behavior recognition dataset with subject action labels of a known scene.

4. The lightweight behavior recognition method based on skeletal data according to claim 1, characterized in that, The preprocessing and labeling includes processing the cross-viewpoint and cross-subject images of the scene to be detected captured by the camera as follows: S1.1 Extract the absolute position data of the joints of the human skeleton using skeletal detection methods; S1.2 Calculate the relative position data between the joints of the subject to be detected; label the actions. S1.3 Calculate the joint joint position characteristics: The concatenated vector of the absolute and relative positions of each joint; S1.

4. Establish a dataset of {action label-behavioral feature pairs} as input for the next step of the network model.

5. The lightweight behavior recognition method based on skeletal data according to claim 1, characterized in that, The dual-branch convolutional module includes a temporal channel and a spatial channel; the spatial channel includes multiple stacked depthwise separable graph convolutional layer modules; the temporal channel includes multiple stacked depthwise separable convolutional layer modules.

6. The lightweight behavior recognition method based on skeletal data according to claim 1, characterized in that, The main branch of the convolutional network includes multiple stacked depthwise separable graph convolutional layer modules.

7. The lightweight behavior recognition method based on skeletal data according to claim 1, characterized in that, The network and training parameters settings include: Set model parameters: maximum training epochs, initial learning rate, Swish activation function, cross-entropy loss function, momentum random descent as the optimization function, and top-1 and top-5 metrics for accuracy. Weight initialization, loading of data, model, and optimizer, and end-to-end training.

Images