A skeleton motion recognition model construction method and a skeleton motion recognition method based on a multi-head MLP architecture

Abstract

The application discloses a skeleton motion recognition model construction method and a skeleton motion recognition method based on a multi-head MLP architecture. A multi-head space MLP module is constructed to model the spatial information of a skeleton sequence, so that the model has less prior knowledge and no longer depends on an explicit and complex graph topology structure, and meanwhile, the spatial information of any skeleton joint is interacted in a global modeling mode. A random batch mask strategy is constructed to randomly mask the skeleton sequence in each training batch, so as to simulate the collection of three-dimensional skeleton data in an actual scene, improve the recognition ability of the model, and strengthen the robustness of the model to incomplete skeleton sequences. The application significantly improves the skeleton motion recognition accuracy, effectively reduces the dependence of the model on prior knowledge, provides a brand-new benchmark model for the field of skeleton motion recognition, and has a wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of skeleton action recognition, and in particular to a skeleton action recognition model construction method based on a multi-head MLP architecture and a skeleton action recognition method based on the skeleton action recognition model. BACKGROUND

[0002] Skeleton-based action recognition is an important task in the field of computer vision, which has attracted more and more researchers in recent years. This topic has extremely wide applications in video monitoring, human-computer interaction, autonomous driving cars, etc. In particular, skeleton data can be easily obtained from depth cameras (such as Microsoft Kinect V2) and current mature 2D or 3D pose estimation algorithms, and compared with RGB video data, skeleton data has more rich semantics, and better robustness and flexibility to dynamic environment and complex background.

[0003] With the continuous development of deep learning, skeleton-based action recognition mainly includes three types of mainstream methods, namely RNN-based, CNN-based and GCN-based. The RNN-based method can model the time dependence of the skeleton sequence by capturing the context information, and these methods usually input the human skeleton information as a vector sequence into the model. The CNN-based method generally converts the skeleton data into a pseudo-image according to the hand-designed conversion rule, that is, the width of the image represents the skeleton joint and the length represents the frame number. However, the RNN and CNN-based methods both ignore the physical structure of the human body in the skeleton data, so they cannot effectively model irregular data. Therefore, the GCN-based method emerges as the times require. The graph convolution network (GCN) can be regarded as an extension of the CNN on non-Euclidean space data, which takes the human skeleton joints as points and the physical connection of the human body as edges to design a graph topology with prior knowledge, so as to better integrate the skeleton data information and improve the performance of skeleton action recognition. Yan et al. first introduced the graph convolution network to model the skeleton data and proposed the spatio-temporal graph convolution network (ST-GCN). ST-GCN includes a spatial graph convolution and a temporal convolution module, wherein the spatial graph convolution is designed according to the physical connection of the human body to capture the spatial information of the skeleton data, and the temporal convolution module is used to model the time dependence of the skeleton data, so as to improve the understanding of the skeleton data and achieve better recognition performance. Subsequently, the adaptive graph convolution method is further proposed in 2s-AGCN, which uses a self-attention mechanism to learn an attention graph topology to more flexibly model the relationship between skeleton joints. Peng et al. use a neural architecture search graph convolution network to search for different topologies for each graph convolution layer.

[0004] However, the GCN-based method largely relies on prior knowledge to design a complex graph topology structure, or adopts a deep network to learn the optimal adjacency matrix of the skeleton in the graph convolution layer, and these methods have failed to approach a unified architecture. Therefore, how to construct a more simple and efficient skeleton action recognition model with less prior knowledge and without relying on an explicit and complex graph topology structure, to flexibly model the relationship between the skeleton data and improve the recognition performance, still needs further exploration. SUMMARY

[0005] In view of the deficiencies and shortcomings of the existing skeleton action recognition method, the application provides a skeleton action recognition model construction method based on a multi-head MLP architecture and a skeleton action recognition method.

[0006] To achieve the above purpose, the technical scheme adopted by the application is as follows:

[0007] A skeleton action recognition model construction method based on a multi-head MLP architecture, comprising the following steps:

[0008] Construct a skeleton action recognition model based on a multi-head MLP architecture; the skeleton action recognition model comprises a batch normalization layer, an MLP skeleton layer, a multi-head MLP skeleton layer, a global average pooling layer and a linear classifier connected in sequence, wherein each MLP skeleton layer is composed of a spatial MLP module and a multi-scale time convolution module, and each multi-head MLP skeleton layer is composed of a multi-head spatial MLP module and a multi-scale time convolution module; wherein the spatial MLP module is composed of an MLP module and a linear mapping layer, and the MLP module is composed of two linear mapping layers and a nonlinear activation function; the multi-head spatial MLP module is composed of a 1x1 convolution layer, a multi-head MLP module and a linear mapping layer; the multi-scale time convolution module is composed of multiple branches, and each branch contains two convolution layers, wherein the convolution kernel size of the first convolution layer of each branch is 1x1, and the second convolution layer of each branch adopts different sizes of convolution kernel and convolution dilation factor, wherein the output results of the multiple branches are spliced into fusion and residual operation with the input; the linear classifier is used to output the classification result of the skeleton action recognition;

[0009] Train the skeleton action recognition model based on the training data set to obtain a trained skeleton action recognition model based on the multi-head MLP architecture.

[0010] Further, the training data set is obtained by collecting skeleton motion data from a skeleton motion recognition data set; the skeleton motion recognition data set is NTU-RGB+D 60, NTU-RGB+D 120, and Northwestern-UCLA.

[0011] Further, the method further comprises randomly masking the skeleton sequences in the skeleton motion recognition data set, and the random masking method comprises: randomly sampling the skeleton sequences of each training batch according to a uniform distribution and performing random spatial masking and / or temporal masking.

[0012] Further, the multi-scale temporal convolution module is composed of four branches, wherein the first two branches include two time convolution layers with a convolution kernel size of 1*3, and the convolution dilation factors are 1 and 2, respectively; the last two branches include two time convolution layers with a convolution kernel size of 1*5, and the convolution dilation factors are 1 and 2, respectively.

[0013] Further, the number of layers of the MLP skeleton layer of the skeleton motion recognition model is 4, and the number of layers of the multi-head MLP skeleton layer is 6; the output channel dimensions of each layer are 64, 64, 64, 64, 128, 128, 128, 256, 256, and 256, respectively.

[0014] Further, the linear classifier mainly consists of a linear mapping layer and a Softmax function.

[0015] Further, the loss function used for training the skeleton motion recognition model based on the training data set is cross-entropy loss.

[0016] A skeleton motion recognition method based on a multi-head MLP architecture, specifically comprising:

[0017] Extracting the skeleton sequence from the video and inputting it into the trained skeleton motion recognition model based on the multi-head MLP architecture obtained by the construction method to obtain a skeleton motion recognition result.

[0018] The beneficial effects of this invention are: 1) This invention models the spatial information of skeleton sequences by constructing a multi-head spatial MLP module. Compared with existing technologies, it has less prior knowledge, eliminates the model's dependence on complex graph topology, and realizes the interaction of global spatial information between skeleton joints, enabling more flexible and efficient modeling of skeleton spatial information; 2) This invention constructs a random batch masking strategy to randomly mask skeleton sequences in each training batch to simulate the acquisition of 3D skeleton data in real-world scenarios, improving the model's recognition ability and strengthening its robustness to incomplete skeleton sequences; 3) This invention provides a novel multi-head MLP skeleton network. Compared with most previous GCN-based methods, it has more powerful modeling capabilities, effectively reducing the model's dependence on prior knowledge while ensuring good performance in skeleton action recognition. It provides a new benchmark model for the field of skeleton action recognition and can be widely applied in this technical field. Attached Figure Description

[0019] Figure 1 This is a flowchart of the skeleton motion model construction method based on a multi-head MLP architecture in an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of the skeleton action recognition model structure based on a multi-head MLP architecture in an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the random batch masking strategy in an embodiment of the present invention;

[0022] Figure 4 These are schematic diagrams of a space module based on MLP in an embodiment of the present invention; wherein, (a) is a schematic diagram of a space MLP module; (b) is a schematic diagram of an MLP module; and (c) is a schematic diagram of a multi-head space MLP module;

[0023] Figure 5 This is a schematic diagram of the multi-scale temporal convolution module in an embodiment of the present invention; Detailed Implementation

[0024] To make the objectives, technical methods, creative features, and advantages of this invention clearer, the skeleton action recognition method based on a multi-head MLP architecture of this invention will be further described in detail below with reference to embodiments and accompanying drawings. Obviously, the specific embodiments described herein are only for explaining this invention and are not intended to limit this invention. Furthermore, the embodiments described below are only some embodiments of this invention, not all embodiments, wherein the technical features involved in the various embodiments can be combined with each other as long as there is no conflict between them.

[0025] Example

[0026] The application discloses a skeleton action recognition model construction method based on a multi-head MLP architecture.

[0027] Figure 1 is a flow chart of the skeleton action recognition model construction method based on the multi-head MLP architecture in the embodiment of the application. Figure 2 is a structural schematic diagram of the skeleton action recognition model based on the multi-head MLP architecture in the embodiment of the application.

[0028] As shown in Figure 1 and Figure 2 , the skeleton action recognition model construction method based on the multi-head MLP architecture comprises the following steps:

[0029] Step 1, collect skeleton action data and construct a training data set.

[0030] In the embodiment, mainstream skeleton action recognition data sets NTU-RGB+D 60, NTU-RGB+D 120 and Northwestern-UCLA are selected to construct the training set, and part of the test set is divided to be used for subsequent training and verification of the model. Specifically, skeleton action data is obtained by extracting skeleton data and corresponding action labels of each frame of image from the skeleton action recognition data set. wherein i, j and k are respectively coordinate values of corresponding human skeleton joints in three-dimensional coordinates, x is a joint vector, that is, the corresponding triple (i, j, k), C is a coordinate dimension, T is a time length, and N is a node number.

[0031] Referring to Figure 3 , the embodiment further comprises random masking of the skeleton sequence in the skeleton action data. The random batch mask is executed before the skeleton sequence is input into the model to simulate the collection of three-dimensional skeleton data in an actual scene, thereby improving the recognition ability of the model and strengthening the robustness of the model to incomplete skeleton sequences. Specifically, the following sub-steps are included:

[0032] (1) Randomly sample the skeleton sequence of each training batch according to uniform distribution to obtain a skeleton sequence;

[0033] (2) Randomly mask the skeleton sequence obtained by random sampling according to a given mask strategy to obtain incomplete skeleton data;

[0034] Figure 3 is a schematic diagram of a random batch mask strategy in the embodiment of the application, here only a mask strategy with a mask rate of 50% is given, which can be determined according to specific circumstances in practice. Figure 3 The black solid square in the middle represents unmasked, and the other squares represent masked. As shown in Figure 3 , the mask strategy in the embodiment specifically includes:

[0035] Masking strategy one: spatial masking, see Figure 3 (a), a one-dimensional mask graph is randomly sampled in the spatial dimension, and then the mask graph is repeated in the time dimension to expand the one-dimensional mask graph, wherein the spatial mask region is the same for each frame in the time dimension.

[0036] Masking strategy two: time masking, see Figure 3 (b), a one-dimensional mask graph is randomly sampled in the time dimension, and then the mask graph is repeated in the spatial dimension to expand the one-dimensional mask graph, wherein the time mask region is the same for each skeleton joint in the spatial dimension.

[0037] Masking strategy three: agnostic masking, see Figure 3 (c), a two-dimensional mask graph is randomly sampled in the time and space dimensions to randomly mask the skeleton joints, wherein the spatial and time mask regions are random.

[0038] For the above three masking strategies, the three-dimensional coordinate values of each masked skeleton joint are set to (0, 0, 0), i.e., the triple (i, j, k) = (0, 0, 0).

[0039] Step two, construct a skeleton action recognition model based on a multi-head MLP architecture.

[0040] Referring to Figure 2 , the skeleton action recognition model based on the multi-head MLP architecture is constructed in the embodiment, specifically including the following sub-steps:

[0041] (2.1) Construct a spatial MLP module for modeling the spatial information between any skeleton joints, extracting the global dependency relationship of the skeleton sequence, compared to the spatial graph convolution operation, this module has less prior knowledge, no longer depends on explicit and complex graph topology structure, and can model the global spatial information of the skeleton sequence in a simpler and more efficient way.

[0042] Figure 4 is a schematic diagram of the MLP-based spatial module in the embodiments of the present application, wherein, Figure 4 (a) is a schematic diagram of a spatial MLP module, Figure 4 (b) is a schematic diagram of an MLP module. Specifically, as shown in (a) and (b), the spatial MLP module is composed of an MLP module and a linear mapping layer, wherein the MLP module is composed of two linear mapping layers and a nonlinear activation function, and the process is described as follows: Figure 4

[0043] Y”=MLP Es (X”)W3

[0044] MLP Es (X”)=X+W2δ(W1BN(X”))

[0045] In the formula, X” represents the input of the MLP module, MLP Es (·) represents the MLP module, Es represents the expansion factor of the MLP module, W3 is the linear mapping layer in the spatial MLP module, i.e., the learnable linear parameter weight matrix, Y” represents the output skeleton sequence of the spatial MLP module, W1 and W2 are respectively the two linear mapping layers of the MLP module, i.e., two learnable linear parameter weight matrices, which can be regarded as a filtering operation with a global receptive field, BN represents batch normalization, and δ represents the GELU nonlinear activation function.

[0046] In particular, when the expansion factor Es in the MLP module is equal to 1, the sizes of W1 and W2 are N×N, and at this time the spatial MLP module is similar to the adaptive graph convolution operation, and the matrix is learned adaptively in the training process, so it can be simply explained as twice sampling of the spatial information of the input skeleton sequence X, i.e., using the adaptive matrix to perform twice matrix multiplication operation in the spatial dimension of the skeleton data. In addition, the spatial MLP module is shared with the same MLP module in the channel dimension and the time dimension.

[0047] (2.2) The spatial MLP module can realize the spatial information interaction between any skeleton joints, but one skeleton joint may have potential relevance with different joints at different levels. Therefore, a multi-head spatial MLP module is further constructed on the basis of the spatial MLP module to capture the spatial correlation between skeleton joints in different representation subspaces at different positions and enhance the feature expression ability of the model.

[0048] Figure 4 (c) is a schematic diagram of a multi-head spatial MLP module in the embodiments of the present application, wherein the MLP module head n ​MLP module of the nth head. Specifically, as shown in Figure 4 (c) The multi-head spatial MLP module is composed of a 1x1 convolution layer, multiple parallel MLP modules and a linear mapping layer, wherein the input skeleton sequence will pass through the 1x1 convolution layer to reduce the channel dimension, then the reduced skeleton sequence will be input into the multi-head MLP module respectively, for each MLP module, it is independent and parallel calculation, then the output of the multi-head MLP module is spliced in the channel dimension, thereby enriching the expression ability of the feature, finally, a linear mapping layer is used to obtain the output skeleton sequence of the multi-head spatial MLP module, and the process is described as follows:

[0049] X'' = Conv1x1(X')

[0050] Y' = Concat(MLP 1 Es (X”),…, MLP n Es (X”))W4

[0051] In the formula, X' represents the input of the multi-head spatial MLP module, Conv1x1 represents the 1x1 convolution layer, X'' represents the reduced skeleton sequence, which corresponds to the input of the MLP module, n represents the number of MLP heads, Concat(·) represents the splicing operation, MLP n Es MLP module of the nth head, W4 represents the linear mapping layer, that is, the learnable linear parameter weight matrix, which can be in the form of 1x1 convolution, Y' represents the output of the multi-head spatial MLP module. Unlike the multi-subset operation in the previous graph convolution module, the multi-head MLP module shares a 1x1 convolution kernel after spatial modeling, which does not introduce a large number of parameters.

[0052] (2.3) A multi-scale time convolution module is constructed to model the time information of the skeleton sequence, a multi-branch convolution structure is designed by using different size convolution kernels to capture the time dependence relationship from different scales, improve the modeling ability in the time dimension, and reduce the calculation amount and parameter amount of the model.

[0053] Figure 5 is a schematic diagram of the multi-scale time convolution module in the embodiment of the present application. As shown in Figure 5As shown, the multi-scale time convolution module is composed of multiple branches, and each branch includes two convolution layers, wherein the convolution kernel size of the first convolution layer of each branch is 1x1 to reduce the channel dimension, and the second convolution layer of each branch uses different size of convolution kernel and convolution dilation factor to capture time dependence from different scales. The number of branches shown in the figure is 4, preferably, the first two branches include two time convolution layers with a convolution kernel size of 1x3, and the convolution dilation factors are 1 and 2 respectively, and the last two branches include two time convolution layers with a convolution kernel size of 1x5, and the convolution dilation factors are 1 and 2 respectively. The input skeleton sequence is input into the four branches in parallel for modeling in the time dimension, and the outputs of the four branches are fused using a concatenation operation, and then the output features after concatenation are subjected to residual operation with the input skeleton sequence of the multi-scale time convolution module, while using ReLU activation function for nonlinear activation, thereby obtaining the output skeleton sequence of the multi-scale time convolution module.

[0054] (2.4) The skeleton action recognition model based on the multi-head MLP architecture has a whole architecture as shown in Figure 2 The whole architecture is composed of batch normalization layer, MLP skeleton layer, multi-head MLP skeleton layer, global average pooling layer and linear classifier, and one MLP skeleton layer and two multi-head MLP skeleton layers are shown in the figure, and four MLP skeleton layers and six multi-head MLP skeleton layers are preferred in the embodiment, and the channel dimension of each layer of skeleton sequence is 64, 64, 64, 64, 128, 128, 128, 256, 256, 256 in turn. In particular, each MLP skeleton layer is composed of spatial MLP module and multi-scale time convolution module, and each multi-head MLP skeleton layer is composed of multi-head spatial MLP module and multi-scale time convolution module. The deep features extracted at the end of the model are globally averaged and input into the linear classifier, thereby obtaining the final classification result of skeleton action recognition. In addition, the linear classifier is mainly composed of linear mapping layer and Softmax function, wherein the Softmax function is used to convert the scores of each part of the skeleton action into probability, and the skeleton action with the maximum probability is the final recognition result.

[0055] Step three, training the skeleton action recognition model based on the training data set to obtain the trained skeleton action recognition model based on the multi-head MLP architecture.

[0056] In this embodiment, the training set of the data set mentioned in step one is used for model training, and the training model is performed on an RTX 2080Ti GPU, using a stochastic gradient descent (SGD) optimizer to learn, with a momentum of 0.9 and a weight decay factor of 0.0001. The model is trained for a total of 65 batches, and the first 5 batches use the Warm up strategy. The initial learning rate is set to 0.1, and is decayed by a factor of 0.1 at the 35th and 55th batches. Label smoothing with a smoothing factor of 0.1 is used in training. In particular, for the NTU RGB+D 60 and NTU RGB+D 120 data sets, the batch size is set to 64, and the number of frames for training and testing is set to 48. For the Northwestern-UCLA data set, the batch size is set to 16, and a total of 80 batches are trained.

[0057] When training the model, the trained skeleton data is input to the skeleton action recognition model based on the multi-head MLP architecture for forward propagation, to obtain the classification probability of each skeleton data for each action category, and finally the cross-entropy loss is used for back propagation to adjust the network parameters.

[0058] The trained skeleton action recognition model based on the multi-head MLP architecture can be used for skeleton action recognition; specifically, the skeleton action recognition method based on the multi-head MLP architecture provided by the present application can extract skeleton sequences from videos and input them into the trained skeleton action recognition model based on the multi-head MLP architecture, to obtain the skeleton action recognition result.

[0059] In this embodiment, the skeleton action recognition model based on the multi-head MLP architecture is verified on the test set of the data set mentioned in step one. During the verification process, all experiments are evaluated based on recognition accuracy. Among them, for the NTU RGB+D 60 data set, the method of the present application follows two evaluation indicators, cross-view (X-View) and cross-subject (X-Sub), for evaluation, and for the NTU RGB+D 120 data set, follows two evaluation indicators, cross-setting (X-Set) and cross-subject (X-Sub), for evaluation.

[0060] The skeleton action recognition model based on the multi-head MLP architecture is input with samples in the test set, and the classification probability of each skeleton data for each action category is obtained through forward propagation. Finally, the category with the maximum classification probability is selected as the recognition category of the skeleton data. For each skeleton data, if the recognized category is consistent with the true label, the method correctly recognizes the skeleton action; otherwise, the method incorrectly recognizes the skeleton action. Experimental results show that the skeleton action recognition model based on the multi-head MLP architecture disclosed in the present application achieves an identification accuracy of 91.6% (X-Sub) and 96.3% (X-View) on the NTU RGB+D 60 dataset, an identification accuracy of 87.8% (X-Sub) and 89.7% (X-Set) on the NTU RGB+D 120 dataset, and a Top-1 identification accuracy of 95.7% on the Northwestern-UCLA dataset. These results show that the method of the present application can achieve advanced recognition performance, and compared with the previous GCN-based method, it has less prior knowledge and eliminates the dependence of the model on complex graph topological structures, and is a simple and efficient method for the skeleton action recognition task.

[0061] In the above embodiments, it can be shown that the construction method of the present application can effectively reduce the dependence of the model on prior knowledge, realize the interaction of the global spatial information between the skeleton joints, and more flexibly and efficiently model the spatial information of the skeleton. Therefore, the trained skeleton action recognition model based on the multi-head MLP architecture can be used to recognize skeleton actions, and as a new benchmark model in the field of skeleton action recognition, it can be widely used in this technical field.

[0062] The above embodiments are used to explain and illustrate the present application, but not to limit the present application. Any modifications and changes made to the present application within the spirit and protection scope of the claims fall within the protection scope of the present application.

Claims

1. A method for constructing a skeleton action recognition model based on a multi-head MLP architecture, characterized in that, The method comprises the following steps: constructing a skeleton action recognition model based on a multi-head MLP architecture; the skeleton action recognition model comprises a batch normalization layer, an MLP skeleton layer, a multi-head MLP skeleton layer, a global average pooling layer and a linear classifier connected in sequence, wherein each MLP skeleton layer is composed of a spatial MLP module and a multi-scale temporal convolution module, and each multi-head MLP skeleton layer is composed of a multi-head spatial MLP module and a multi-scale temporal convolution module; wherein the spatial MLP module is composed of an MLP module and a linear mapping layer, and the MLP module is composed of two linear mapping layers and a nonlinear activation function; the multi-head spatial MLP module is composed of a 1×1 convolution layer, a multi-head MLP module and a linear mapping layer; the multi-scale temporal convolution module is composed of multiple branches, and each branch comprises two convolution layers, wherein the convolution kernel size of the first convolution layer of each branch is 1×1, and the second convolution layer of each branch uses different sizes of convolution kernels and convolution dilation factors, wherein the output results of the multiple branches are spliced into fusion and simultaneously perform residual operation with the input; the linear classifier is used to output the classification results of the skeleton action recognition; training the skeleton action recognition model based on a training data set to obtain a trained skeleton action recognition model based on a multi-head MLP architecture; The training data set is obtained by collecting skeleton action data from a skeleton action recognition data set; wherein the skeleton sequences in the skeleton action recognition data set are randomly masked, and the method of random masking is specifically as follows: randomly sampling the skeleton sequences of each training batch according to a uniform distribution and performing random spatial masking and / or temporal masking.

2. The method of claim 1, wherein, The skeleton action recognition data set is NTU-RGB+D60, NTU-RGB+D120 and Northwestern-UCLA.

3. The method of claim 1, wherein, The multi-scale temporal convolution module is composed of four branches, wherein the first two branches comprise two time convolution layers with a convolution kernel size of 1×3 and a convolution dilation factor of 1 and 2, respectively, and the last two branches comprise two time convolution layers with a convolution kernel size of 1×5 and a convolution dilation factor of 1 and 2, respectively.

4. The method of claim 1, wherein, The number of layers of the MLP skeleton layer of the skeleton action recognition model is 4, and the number of layers of the multi-head MLP skeleton layer is 6, and the output channel dimensions of each layer are 64, 64, 64, 64, 128, 128, 128, 256, 256, 256, respectively.

5. The method of claim 1, wherein, The linear classifier is mainly composed of a linear mapping layer and a Softmax function.

6. The method of claim 1, wherein, The loss function used for training the skeleton action recognition model based on the training data set is cross-entropy loss.

7. A skeleton motion recognition method based on a multi-headed MLP architecture, characterized by, Specifically, the skeleton action recognition result is obtained by extracting the skeleton sequence from the video and inputting it into the trained skeleton action recognition model based on the multi-head MLP architecture obtained by the construction method of any one of claims 1-6.

Images