Lightweight behavior recognition method based on skeleton data

Abstract

The invention discloses a lightweight behavior recognition method based on skeleton data. The lightweight behavior recognition method comprises the following steps: 1) preprocessing an obtained behavior data set; and 2) fusing space and time sequence information by adopting a double-branch structure, respectively putting a time sequence T and a joint point N in respective channel dimensions through a dimension conversion thought, and extracting behavior characteristics through an established efficient space-time diagram convolution model. Wherein for the efficient space-time diagram convolution model, 1 * 1 convolution and depth separable convolution are introduced, the calculation cost of parameter adjustment and model reasoning is reduced, and the difficulty of model training is reduced by introducing a residual link. After model parameters, a loss function, an optimization function and the like are set, the network is trained by a training set through model weight initialization, data loading and a model optimizer. And 3) predicting the data set by using the trained skeleton behavior recognition model, and giving a corresponding action category. According to the method, the model complexity can be effectively reduced on the basis of ensuring relatively high recognition accuracy, and the method has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior recognition, and in particular to a lightweight behavior recognition method based on skeleton data. Background Art

[0002] At present, human action recognition technology has a wide range of application values ​​in the fields of intelligent monitoring, human-computer interaction, video retrieval, etc. Skeleton data can be regarded as a topological graph, which is usually obtained from video sequences by human posture estimation algorithms. Existing studies have shown that simply representing skeleton data as a vector sequence processed by RNN or 2D / 3D data processed by CNN cannot fully simulate the complex spatiotemporal correlation of body joints. Inspired by the booming graph-based methods, many human action recognition methods based on GNN and GCN have been proposed, which regard skeleton data as a graph structure of edges and nodes, and make significant progress in convolutional networks that model the human skeleton as a spatiotemporal graph.

[0003] However, there are two common problems with graph convolution-based methods. On the one hand, the receptive fields of spatial and temporal graphs are not flexible enough, which limits their feature expression capabilities. On the other hand, the model is over-parameterized, resulting in high computational costs, which affects its practical application in real-world scenarios. Summary of the invention

[0004] The purpose of the present invention is to provide a lightweight behavior recognition method based on skeleton data, which classifies and recognizes behaviors through data acquisition, preprocessing and lightweight deep learning models to facilitate practical applications.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a lightweight behavior recognition method based on skeleton data, which is used to recognize the behavior of a subject to be detected in a scene of unknown behavior, and the method comprises the following steps:

[0006] Step S1, obtain behavior recognition scene data, preprocess and label it, establish an {action label-behavior feature image pair} data set, and divide it into a training set and a validation set;

[0007] Step S2, establish an efficient spatiotemporal graph convolutional network based on skeletal data, set network and training parameters, iteratively fit and train the network model with behavioral features in the training set as input and action labels as output, and adjust the network parameters by back propagation, and then use the validation set data to verify the model accuracy to obtain an ideal model for identifying and predicting unknown behaviors;

[0008] Step S3, collecting scene data containing unknown behaviors on site, obtaining joint joint position features after preprocessing, inputting them into the trained skeletal behavior recognition network to predict behaviors, and outputting the action category of the subject to be detected in the scene.

[0009] The scene data is a video or a single-frame image.

[0010] The behavior recognition image data is an open source behavior recognition dataset of cross-viewing angles and cross-subject images of a scene to be detected taken by a camera, or subject action labels of a known scene.

[0011] The preprocessing and labeling includes performing the following processing on the cross-viewing angles and cross-subject images of the scene to be detected captured by the camera:

[0012] S1.1, using the bone detection method to extract the absolute position data of the joint points of the human skeleton;

[0013] S1.2, calculate the relative position data between the joints of the subject to be detected; mark the action label;

[0014] S1.3, calculate the joint joint point position features: the splicing vector of the absolute position and relative position of each joint point;

[0015] S1.4. Establish a dataset of {action label-behavior feature pairs} as the input of the next network model.

[0016] The efficient spatiotemporal graph convolutional network based on skeleton data includes:

[0017] a) Batch normalization BatchNorm layer, used to accelerate network convergence;

[0018] b) Initialize the spatiotemporal graph convolution module InitialGCN Block, which is used to perform feature conversion on the joint joint point position features;

[0019] c) The spatiotemporal extraction module is used to place the temporal information of the skeleton and the joint features in the input features in the time and space channel dimensions of the network respectively, and obtain features of different dimensions: temporal information Transpose (T, C, N) and joint feature Transpose (N, T, C);

[0020] d) A dual-branch efficient convolution module is used to further extract spatiotemporal features from features of different dimensions and obtain spatiotemporal feature maps after their respective enhanced depths;

[0021] e) Fusion operation, used to perform serial operation and fusion of the dual-branch temporal and spatial feature maps after feature extraction;

[0022] f) The mainstream branch of the efficient convolutional network is used to further extract features from the fused feature map;

[0023] g) Global average pooling layer, used to operate on the output feature map to obtain a feature vector representing the behavior;

[0024] g) Fully connected layer, used to predict and identify the action category of the subject to be detected in unknown scene data based on the feature vector.

[0025] The dual-branch efficient convolution module includes a branch time channel and a branch space channel; the space channel includes a plurality of stacked depth-separable graph convolution layer modules; the time channel includes a plurality of stacked depth-separable convolution layer modules.

[0026] The mainstream branch of the efficient convolutional network includes multiple stacked depth-wise separable graph convolutional layer modules.

[0027] The setting of network and training parameters includes:

[0028] Set model parameters: maximum training cycle, initial learning rate, Swish activation function, cross entropy loss function, momentum random descent as optimization function, top1 and top5 as accuracy measurement;

[0029] Initialize weights, load data, models, and optimizers, and perform end-to-end training.

[0030] Beneficial effects and advantages of the present invention:

[0031] 1. The present invention provides a lightweight behavior recognition method based on skeleton data, and its behavior recognition accuracy achieves the best top1 performance of 84.6% and 92.5% on the NTU 60 large dataset;

[0032] 2. The method of the present invention significantly reduces the amount of model parameters and the complexity of the algorithm, which is beneficial to the practical application of skeleton-based action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the behavior recognition method based on skeleton data of the present invention;

[0034] Figure 2 This is a detailed diagram of the module for establishing efficient spatiotemporal graph convolution;

[0035] Figure 3 It is an infographic of 25 skeleton joints. DETAILED DESCRIPTION

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0038] See also Figure 1 , this embodiment includes:

[0039] A lightweight behavior recognition method based on skeleton data comprises the following steps:

[0040] S1: Preprocess the acquired behavior recognition dataset and select the large NTU-RGB-D dataset as the behavior recognition dataset. The dataset provides 3D skeleton coordinates, including two benchmarks: cross-view and cross-subject. The cross-view refers to a scene image containing multiple subjects taken by cameras with different viewpoints. The cross-subject refers to a scene image containing multiple subjects, and the actions of multiple subjects in the image are independent or there are action interactions. The specific steps include:

[0041] S1.1: Use numbers 0-24 to label the 25 skeletal joints in the obtained behavior recognition dataset;

[0042] S1.2: According to the human body structure, determine the connection between the key bones and calculate the relative position coordinates between the joints; connect the absolute position and relative position coordinates of each joint point in series to obtain the joint joint position characteristics of a single subject;

[0043] S2: Input the joint joint position coordinates into the established efficient spatiotemporal graph convolutional network, optimize the network parameters, and obtain an efficient spatiotemporal convolutional network model based on skeleton data; the specific steps include:

[0044] S2.1: Through the BatchNorm layer, the joint point position coordinates are batch normalized to accelerate network convergence;

[0045] S2.2: Input the normalized joint joint point position coordinate data into the initialized spatiotemporal graph convolution module proposed in the existing ST-GCN algorithm, see Figure 2 (a) Convolution is performed using the adjacency matrix established by the human body topology structure, and multi-layer spatiotemporal graph convolution operations are performed on the input data to extract the subject's action features. The spatiotemporal graph convolution module is initialized, including the BatchNorm layer, SGC module, and TC module.

[0046] S2.3: Then transform the input subject action features, and put the feature temporal information Transpose (T, C, N) and joint point feature Transpose (N, T, C) in the respective channel dimensions of the network to obtain features of different dimensions. T is the number of frames, C is the number of channels, and N is the number of joint points;

[0047] S2.4: Time branch of efficient convolutional network: Transpose the time information (T, C, N) to the feature map of the time channel dimension as input; first, use 1*1 convolution (1*1Conv) layer to reduce the time dimension. Secondly, use two advanced depth-separable graph convolution layers, such as Figure 2 (b) is shown. It is composed of ordered stacking of depth graph convolution (Depth_SimpGCN) and point-by-point convolution (Point_Conv), and the introduction of residual links (1*1 convolution adds residual branches to the final output) reduces the difficulty of model training. The two numbers in each module in the figure represent the number of input Cin and output Cout channels, where Crd = Cin / 2, / 2 means the step size is 2;

[0048] S2.5: Spatial branch of efficient convolutional network: Transpose (N, T, C) the joint point information in the feature map of the spatial channel dimension as the input of another branch. It also uses two advanced depth-separable convolutional layers and introduces residual links at the beginning and end, see Figure 2 (c). The model is composed of a stack of standard depth convolution (Depth_Conv) and point-wise convolution (Point_Conv);

[0049] S2.6: The dual-branch feature maps after feature extraction are fused through a series operation;

[0050] S2.5: The dual-branch joint feature map is input into two modules consisting of two stacked advanced depthwise separable graph convolutional layers to extract temporal features again;

[0051] S2.6: Perform a global average GAP operation on the output feature map to obtain a feature vector;

[0052] S2.7: After the final fully connected layer FC, the predicted action category and the main action regression box are obtained.

[0053] S3: After setting the model parameters, loss function, optimization function, etc., the training set data is input into the network model of S2, the model weight is initialized, the data and model optimizer are loaded, and end-to-end training is performed to obtain an ideal model for skeletal behavior recognition.

[0054] S3.1: Set model parameters, including the maximum training period of 70, the initial learning rate of 0.1, the activation function of Swish function, the loss function of cross entropy, the momentum random descent as the optimization function, and the top1 and top5 as the accuracy.

[0055] S3.2: Initialize weights, load data, models, optimizers, and perform end-to-end training.

[0056] S4: Use the trained skeletal behavior recognition model to predict the data set and give the corresponding action category.

[0057] The NTU RGB+D 60 dataset was released by Nanyang Technological University, Singapore in 2016. The dataset consists of 40 subjects aged between 10 and 35 performing 60 behaviors. The specific behavior labels are shown in Table 1. The behavior categories of this dataset are divided into three categories: 1) 40 daily behaviors; 2) 9 health-related behaviors; and 3) 11 interactive behaviors. This dataset uses the Kinect v2 camera to obtain information with 25 skeleton joints. The distribution diagram is shown in the figure below. Figure 3 shown.

[0058] Table 1

[0059]

[0060] Our method is compared with the current classic algorithms and advanced algorithms on the X-sub and X-view benchmarks of the NTU 60 dataset in terms of recognition accuracy. The results are shown in Table 2. It can be seen that the best performance of our model is 84.6% and 92.5% in the X-Sub and X-View benchmarks, respectively, which is better than other algorithms. Two typical methods are worth noting here. The first is the ST-GCN algorithm, which is currently the most popular backbone model for skeleton-based action recognition. Compared with the ST-GCN algorithm, our method leads by about 3% and 4% on the X-sub and X-view benchmark datasets. The second is NewPartition-STGCN, which is a recent improvement based on ST-GCN. The accuracy of this paper on the NTU 60 benchmark is about 1% higher than that of the New Partition-STGCN algorithm. These results show that the method proposed in this paper is an algorithm with competitive performance compared with the comparison methods.

[0061] Table 2

[0062] method X-Sub X-View HBRNN

[16] 59.1 64.0 ST-LSTM

[17] 69.2 77.7 STA-LSTM

[18] 73.4 81.2 HCN

[19] 86.5 91.1 ST-GCN[2] 81.5 88.3 SR-TSL

[20] 84.8 92.4 Motif-STGCN

[21] 84.2 90.2 New Partition-STGCN[8] 83.7 91.6 Our method 84.6 92.5

[0063] In order to verify the efficiency of the model, this paper is compared with other methods in terms of accuracy and model complexity (FLOPs and number of parameters) on the NTU 60 dataset. The experimental results are shown in Table 3. The ratios after FLOPs and the number of parameters represent the ratios between other models and the model in this paper.

[0064] It can be seen that there is a large gap in efficiency between the previous models and our model. Compared with the first GCN baseline for skeleton-based action recognition (i.e., ST-GCN

[11] ), we achieve a 3-4% improvement in accuracy, 17.18 times fewer FLOPs, and 13.48 times fewer parameters. SR-TSL achieves the same accuracy as our model, but its number of trainable parameters is much higher, about 82.6 times that of our model. The results clearly show that the proposed method brings significant improvements in both model accuracy and complexity, which will benefit the practical applications of skeleton-based action recognition.

[0065] Table 3

[0066]

[0067] The present invention Figure 2 The network module of (b) is a newly proposed structure: considering the balance between the good learning ability of spatial graph convolution for skeleton data and the computational cost, the present invention introduces the spatial graph convolution operation GCN into the deep convolution layer, simplifies the model parameters, deletes the parameters that adjust the correlation between joint points in the graph convolution operation, and adds residual branches. The positioning and action recognition of unknown behaviors are realized.

[0068] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structures or equivalent process changes made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A lightweight behavior recognition method based on skeleton data, characterized in that: The following steps are involved: Step S1, obtain behavior recognition scene data, preprocess and label it, establish an {action label-behavior feature image pair} data set, and divide it into a training set and a validation set; Step S2, establish an efficient spatiotemporal graph convolutional network based on skeletal data, set network and training parameters, iteratively fit and train the network model with behavioral features in the training set as input and action labels as output, and adjust the network parameters by back propagation, and then use the validation set data to verify the model accuracy to obtain an ideal model for identifying and predicting unknown behaviors; Step S3, collecting scene data containing unknown behaviors on site, obtaining joint joint position features after preprocessing, inputting them into the trained skeletal behavior recognition network to predict behaviors, and outputting the action category of the subject to be detected in the scene.

2. A lightweight behavior recognition method based on skeleton data according to claim 1, characterized in that: The scene data is a video or a single-frame image.

3. The lightweight behavior recognition method based on skeleton data according to claim 1 is characterized in that: The behavior recognition image data is an open source behavior recognition dataset of cross-viewing angles and cross-subject images of a scene to be detected taken by a camera, or subject action labels of a known scene.

4. The lightweight behavior recognition method based on skeleton data according to claim 1 is characterized in that: The preprocessing and labeling includes performing the following processing on the cross-viewing angles and cross-subject images of the scene to be detected captured by the camera: S1.1, using the bone detection method to extract the absolute position data of the joint points of the human skeleton; S1.2, calculate the relative position data between the joints of the subject to be detected; mark the action label; S1.

3. Calculate the joint joint position features: The splicing vector of the absolute and relative positions of each joint point; S1.

4. Establish a dataset of {action label-behavior feature pairs} as the input of the next network model.

5. The lightweight behavior recognition method based on skeleton data according to claim 1 is characterized in that: The efficient spatiotemporal graph convolutional network based on skeleton data includes: a) Batch normalization BatchNorm layer, used to accelerate network convergence; b) Initialize the spatiotemporal graph convolution module InitialGCN Block, which is used to perform feature conversion on the joint joint point position features; c) The spatiotemporal extraction module is used to place the temporal information of the skeleton and the joint features in the input features in the time and space channel dimensions of the network respectively, and obtain features of different dimensions: temporal information Transpose (T, C, N) and joint feature Transpose (N, T, C); d) A dual-branch efficient convolution module is used to further extract spatiotemporal features from features of different dimensions and obtain spatiotemporal feature maps after their respective enhanced depths; e) Fusion operation, used to perform serial operation and fusion of the dual-branch temporal and spatial feature maps after feature extraction; f) The mainstream branch of the efficient convolutional network is used to further extract features from the fused feature map; g) Global average pooling layer, used to operate on the output feature map to obtain a feature vector representing the behavior; g) Fully connected layer, used to predict and identify the action category of the subject to be detected in unknown scene data based on the feature vector.

6. The lightweight behavior recognition method based on skeleton data according to claim 5 is characterized in that: The dual-branch efficient convolution module includes a branch time channel and a branch space channel; the space channel includes a plurality of stacked depth-separable graph convolution layer modules; the time channel includes a plurality of stacked depth-separable convolution layer modules.

7. The lightweight behavior recognition method based on skeleton data according to claim 5 is characterized in that: The mainstream branch of the efficient convolutional network includes multiple stacked depth-wise separable graph convolutional layer modules.

8. The lightweight behavior recognition method based on skeleton data according to claim 1, characterized in that: The setting of network and training parameters includes: Set model parameters: maximum training cycle, initial learning rate, Swish activation function, cross entropy loss function, momentum random descent as optimization function, top1 and top5 as accuracy measurement; Initialize weights, load data, models, and optimizers, and perform end-to-end training.

Images