A method for human skeleton action recognition based on parallel convolutional neural network

Abstract

The present invention relates to a human skeleton action recognition method based on a parallel convolutional neural network, comprising the following steps: 1) obtaining skeleton data and preprocessing, removing irrelevant skeleton data and incompletely repaired data and performing normalization processing; 2) designing a basic module structure of a parallel convolutional neural network; 3) setting training parameters and performing error back propagation; 4) inserting the basic module of the parallel convolutional neural network into a semantic guided neural network SGN network, constructing a human skeleton action recognition network model, and performing cross-view and cross-object training and testing on a skeleton data set; 5) using the processed skeleton data as input, performing human skeleton action recognition according to the trained human skeleton action recognition network model. Compared with the prior art, the present invention has the advantages of greatly reducing the amount of data while improving recognition accuracy, wide application range, and plug-and-play.

Description

Technical Field

[0001] The present invention relates to the field of robot learning and computer vision, and in particular to a method for recognizing human skeleton movements based on a parallel convolutional neural network. Background Art

[0002] Action recognition is widely used in video surveillance, human-computer interaction, intelligent driving and other fields. Video-based action recognition can be divided into RGB pixel-based action recognition and skeleton-based action recognition. Skeleton-based action recognition is widely studied because of its low computational complexity and ability to retain rich action behavior information.

[0003] Skeleton-based algorithms are currently divided into traditional feature extraction methods, such as the spatiotemporal interest point method, the dense trajectory method, and the energy histogram method, as well as deep learning algorithms, such as the convolutional neural network algorithm and the graph convolutional neural network. The convolutional neural network requires that the input data is regular and has Euclidean distance, while the human skeleton topology is irregular and the distance between bone points is non-Euclidean distance. The graph convolutional network solves this problem through the adjacency matrix. However, the existing graph convolutional network often requires many layers of stacking to achieve a relatively good recognition accuracy. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a human skeleton motion recognition method based on parallel convolutional neural network.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A human skeleton action recognition method based on parallel convolutional neural network includes the following steps:

[0007] 1) Obtain bone data and perform preprocessing, remove irrelevant bone data and incompletely repaired data, and then perform normalization processing;

[0008] 2) Design the basic module structure of parallel convolutional neural network;

[0009] 3) Set training parameters and perform error back propagation;

[0010] 4) Insert the basic module of parallel convolutional neural network into the semantic guided neural network (SGN) network to build a human skeleton action recognition network model, and perform cross-view and cross-object training and testing on the skeleton dataset;

[0011] 5) Using the processed skeleton data as input, human skeleton action recognition is performed based on the trained human skeleton action recognition network model.

[0012] In the step 1), the bone data that does not meet the specifications or the data scale is not within the set range is eliminated, and the interpolation method is used to generate new bone data, which is normalized by the BatchNorm2d method in the deep learning framework Pytorch.

[0013] In the step 2), the basic module is composed of a graph convolutional network (GCN) and a two-dimensional convolutional network (CNN) in parallel. The input bone data is independently convolved and then nonlinearly processed through an activation function (ReLU) to increase the information extraction capability of the network. The graph convolutional network (GCN) is used to extract spatial information features, and the action connection between the bone points of the human body is extracted through a learnable adjacency matrix. The two-dimensional convolutional network (CNN) is used to extract optical flow information between different frames, that is, regular time frame information.

[0014] The parallel convolutional neural network is composed of one or more basic modules through stacking to achieve plug-and-play feature extraction.

[0015] In the step 3), the training parameters include learning rate (lr), loss function (loss), batch size (batch_size), number of iterations (epoch) and loss function.

[0016] In the step 4), inserting the basic module of the parallel convolutional neural network into the semantic guided neural network SGN is specifically as follows:

[0017] The plug-and-play parallel convolutional neural network replaces the GCN module in the original SGN network to form a human skeleton action recognition network model.

[0018] In the step 4), cross-view and cross-object training and testing are performed on the skeleton dataset NTU-RGB+D 60.

[0019] In step 4), the cross-view and cross-object training and testing specifically include:

[0020] Adjust the convolution kernel size of the two-dimensional convolutional network (CNN) and repeat step 4) until the convolution kernel size corresponding to the highest cross-view and cross-object accuracy is obtained, and the corresponding size parameter is used as the model parameter of the basic module;

[0021] Fix the size parameters of the basic module, adjust the number of stack layers and the channel dimension of the middle layer of the parallel convolutional network, and repeat step 4) until the number of stack layers and the channel dimension of the middle layer corresponding to the highest cross-view and cross-object accuracy are obtained, and the model parameter design of the parallel convolutional network is completed.

[0022] In the step 5), the adjustment of the convolution kernel size of the two-dimensional convolutional network (CNN) includes 1×1 size, 1×3 size and 3×3 size.

[0023] In the step 6), the adjustment of the number of stack layers of the parallel convolutional network includes 1, 2, 3, 4 and 5, and the adjustment of the channel dimension of the middle layer includes 64, 64*2 and 64*4.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] 1. The parallel convolutional network of the present invention can be plug-and-play in any feature extraction and recognition network structure. The basic module is realized by parallel connection of graph convolution (GCN) and two-dimensional convolution (CNN). The input bone point data are independently convolved. GCN extracts the action connection between each bone point of the human body through a learnable adjacency matrix, and CNN is used to extract regular time frame information. The two directly process the input data, reducing information loss, so as to extract rich spatiotemporal information to the greatest extent and increase recognition accuracy. The present invention implants the parallel convolutional network into the latest semantic guided neural network (SGN) network. The CV and CS test accuracies on the NTU RGB+D 60 data set are greatly improved, with the accuracy increased by 0.5%, while the parameter amount of the entire network is only 1.69M, which greatly reduces the amount of calculation.

[0026] 2. The model parameter optimization design of the present invention is novel. First, the parameters of the basic module are designed to find the optimal convolution kernel size. Then, the optimal basic module parameters are fixed. Through multiple iterative training, the optimal number of basic module stack layers and the channel dimension of the intermediate layer are found. By changing the number of channel layers of different layers, multi-scale channel information is extracted to enhance the information extraction capability of the network. Through training at different levels, the optimal network parameter model is obtained in two times.

[0027] 3. Plug and play, can replace the graph convolution (GCN) module of any network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a basic module diagram of the method of the present invention.

[0029] Figure 2 This is a basic module stack diagram of the method of the present invention.

[0030] Figure 3 This is a structural diagram of the human skeleton action recognition network model. DETAILED DESCRIPTION

[0031] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0032] In order to give full play to the learning ability of graph convolutional networks and two-dimensional convolutional neural networks and complete high-precision action recognition with the least stack, the present invention provides a human skeleton action recognition method based on parallel convolutional neural networks. The basic module structure of the parallel convolutional neural network is as follows: Figure 1 As shown in the figure, each basic module is composed of a graph convolutional network (GCN) and a two-dimensional convolutional network (CNN) in parallel. The input skeleton data is processed by GCN and CNN independently, and then added and calculated. The ReLU function is then used for nonlinear activation to improve the learning ability of the module. Figure 2 As shown in the figure, the parallel convolutional neural network is formed by multiple basic module stacks, which are composed of three basic modules connected in series. Experiments show that the stack of three modules can fully extract the spatiotemporal information of the input data and reduce overfitting. Figure 3 Figure 1 is a schematic diagram of the human skeleton action recognition network model structure. The model framework comes from the semantic guided neural network (SGN). In the figure, θ(·) and φ(·) are implantation functions. The multiplication of the two is used to generate the adjacency matrix of the graph convolutional network (GCN). STM is a spatiotemporal aggregation module used to aggregate spatiotemporal information. FC is a fully connected layer used to complete action classification. The dashed box is the parallel convolutional network module designed by the present invention, which replaces the original GCN module.

[0033] The specific steps include:

[0034] S1. Preprocess the skeleton data according to the existing method. The skeleton data used in the present invention is the NTU RGB 60 skeleton data set, which includes 60 types of daily action skeleton data. The data set of each type of action mainly records the number of video frames that complete the action, the position coordinates of the human skeleton points in each frame, and other information. The preprocessing mainly includes operations such as eliminating irrelevant skeleton data and repairing incomplete data, and normalizing the data of each dimension in the skeleton data to the [0,1] interval;

[0035] S2. Design the basic module structure and model parameters of the parallel convolutional neural network, connect the graph convolutional network (GCN) and the two-dimensional convolutional network (CNN) in parallel, so that the two modules can directly perform convolution learning on the input data, reduce information loss, and give full play to the powerful spatial information extraction ability of GCN and the optical flow information extraction ability of CNN between different frames;

[0036] S3. Set training parameters, such as learning rate (lr), loss function (loss), batch size (batch_size), number of iterations (epoch), loss function, etc., and perform error back propagation;

[0037] S4, insert the parallel convolution basic module into the semantic guided neural network (SGN) to form a human skeleton action recognition network model (such as Figure 3 As shown), and cross-view (CV) and cross-object (CS) training and testing are performed on the skeleton dataset NTU-RGB+D 60;

[0038] S5, adjust the size of the two-dimensional convolution kernel, repeat step S4, until the size of the two-dimensional convolution kernel with the highest CV and CS accuracy is found, and the corresponding parameters are used as model parameters of the parallel convolution basic module;

[0039] S6, change the number of stack layers and the channel dimension of the middle layer of the parallel convolutional neural network, repeat step S4 until the CV and CS accuracy rates are maximized, record the corresponding number of stack layers and the channel dimension of the middle layer, and complete the model parameter design of the parallel convolutional network;

[0040] S7. Using the processed skeleton data as input, human skeleton action recognition is performed according to the trained human skeleton action recognition network model.

[0041] The specific process of identification is as follows Figure 3 As shown in the figure, the skeleton data is processed by a three-layer parallel convolution module to obtain spatiotemporal data with certain semantic information, and then the spatiotemporal information is aggregated by the spatiotemporal information aggregation module STM, and finally the classification of 60 types of daily actions is completed through the fully connected layer FC.

[0042] During the implementation of step S1, when the bone data does not meet the specifications, the data scale is not within the set range, etc., the data that does not meet the requirements needs to be eliminated, and new bone data needs to be generated using interpolation, and normalized using the BatchNorm2d() method in the deep learning framework Pytorch.

[0043] During the implementation of step S2, the size of the two-dimensional convolution kernel is initially designed to be 1×1, and the outputs of the two convolutions are added together and then nonlinearly processed using the activation function ReLU to further increase the network's information extraction capability and complete the network architecture design.

[0044] In the implementation process of step S3, the learning rate (lr) is set to 0.001, the cross entropy function is used as the loss function (loss), the batch_size is set to 64, the epoch is set to 140, and every 580 batches are used as an epoch.

[0045] During the implementation of step S4, the parallel convolutional neural network of the present invention as a plug-and-play module replaces the original SGN network GCN module to form a human skeletal action recognition network model. The parallel convolutional neural network module designed by the present invention can also technically replace the graph convolution (GCN) modules of other network models, such as the graph convolution (GCN) modules of the dual-stream adaptive graph convolution (2s-AGCN) and action structure graph convolution (AS-GCN) networks.

[0046] During the implementation of step S5, the convolution kernel size of CNN is changed, and the CV and CS accuracy are tested at 1×1, 1×3 and 3×3 sizes respectively. The size parameters of the convolution kernel under the highest accuracy are used as the model parameters of the parallel convolutional network basic module to complete the parameter design of the parallel convolutional neural network basic module.

[0047] During the implementation of step S6, the model parameters of the basic module are fixed, and the CV and CS accuracies are tested when the number of stack layers is 1, 2, 3, 4, and 5. The test results are shown in Table 1. The CS and CV accuracies corresponding to different channel dimensions of the intermediate layer are tested. The channel dimension can be set to 64, 64*2, and 64*4. The number of stack layers and channel dimension at the highest accuracy are used as model parameters of the parallel convolutional neural network, and finally the network structure and parameter design are completed.

[0048] Table 1 Action recognition accuracy table under different stack layers

[0049]

[0050]

[0051] Compared with the existing graph convolutional action recognition method, the human skeleton action recognition method based on parallel convolutional neural network provided by the present invention has three major innovations:

[0052] 1. High accuracy. The graph convolutional network (GCN) is connected in parallel with the two-dimensional convolutional network (CNN), which reduces the information loss of the input data after passing through different convolution modules, thereby giving full play to the ability of GCN to learn spatial information and CNN to learn temporal information;

[0053] 2. The optimal design of network model parameters is novel. First, the optimal basic module parameters, i.e., the convolution kernel size of CNN, are found. Then, the basic module parameters are fixed, basic modules with different numbers of layers are stacked, and different channel dimensions of the middle layer are designed. Training is repeated until the cross-domain and cross-view recognition accuracy on the NTU-RGB+D 60 dataset is the highest. The corresponding number of layers and channel dimensions are recorded to complete the optimal network parameter design.

[0054] 3. Plug and play, can replace the GCN module of any network, and has wide applicability.

[0055] In order to extract richer skeleton information, the present invention makes full use of the strong spatial information extraction ability of graph convolution and the strong temporal frame information extraction ability of two-dimensional convolutional network, designs a plug-and-play parallel convolution module, and implants the module into the latest network module semantic guided neural network (SGN). Cross-view (CV) and cross-subject (CS) tests are performed on NTU RGB+D 60, which improves the accuracy by 0.5% compared with the network before implantation, as shown in Table 2.

[0056] Table 2 Comparison of action recognition accuracy of different recognition networks

[0057] Method Year CS CV VA-LSTM 2017 79.4 87.6 ST-GCN 2018 81.5 88.3 HCN 2018 86.5 91.1 AS-GCN 2019 86.8 94.2 GR-GCN 2019 87.5 94.3 2s-AGCN 2019 88.5 95.1 SGN 2020 89.0 94.5 Full HSQ 2021 87.8 93.7 OURS - 89.5 95.0

[0058] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A human skeleton action recognition method based on parallel convolutional neural network, characterized in that: The following steps are involved: 1) Obtain bone data and perform preprocessing, remove irrelevant bone data and incompletely repaired data, and then perform normalization processing; 2) Design the basic module structure of parallel convolutional neural network; 3) Set training parameters and perform error back propagation; 4) Insert the basic module of the parallel convolutional neural network into the semantic guided neural network (SGN) network to build a human skeleton action recognition network model, and perform cross-view and cross-object training and testing on the skeleton dataset; 5) Taking the processed skeleton data as input, human skeleton action recognition is performed according to the trained human skeleton action recognition network model; In the step 2), the basic module is composed of a graph convolutional network GCN and a two-dimensional convolutional network CNN in parallel, and the input bone data is independently convolved and then nonlinearly processed by the activation function ReLU to increase the information extraction ability of the network. The graph convolutional network GCN is used to extract spatial information features, and the action connection between the bone points of the human body is extracted through a learnable adjacency matrix. The two-dimensional convolutional network CNN is used to extract optical flow information between different frames, that is, regular time frame information; In the step 4), cross-view and cross-object training and testing are performed on the skeleton dataset NTU-RGB+D 60; In step 4), the cross-view and cross-object training and testing specifically include: Adjust the convolution kernel size of the two-dimensional convolutional network CNN and repeat step 4) until the convolution kernel size corresponding to the highest cross-view and cross-object accuracy is obtained, and the corresponding size parameter is used as the model parameter of the basic module; Fix the size parameters of the basic module, adjust the number of stack layers and the channel dimension of the middle layer of the parallel convolutional network, and repeat step 4) until the number of stack layers and the channel dimension of the middle layer corresponding to the highest cross-view and cross-object accuracy are obtained, and the model parameter design of the parallel convolutional network is completed.

2. A human skeleton action recognition method based on parallel convolutional neural network according to claim 1, characterized in that: In the step 1), the bone data that does not meet the specifications or the data scale is not within the set range is eliminated, and the interpolation method is used to generate new bone data, which is normalized by the BatchNorm2d method in the deep learning framework Pytorch.

3. The method for human skeleton motion recognition based on parallel convolutional neural network according to claim 1, characterized in that: The parallel convolutional neural network is composed of one or more basic modules through stacking to achieve plug-and-play feature extraction.

4. The method for human skeleton motion recognition based on parallel convolutional neural network according to claim 1, characterized in that: In the step 3), the training parameters include learning rate, loss function, batch size, number of iterations and loss function.

5. The method for human skeleton motion recognition based on parallel convolutional neural network according to claim 3, characterized in that: In the step 4), inserting the basic module of the parallel convolutional neural network into the semantic guided neural network SGN is specifically as follows: The plug-and-play parallel convolutional neural network replaces the GCN module in the original SGN network to form a human skeleton action recognition network model.

6. The method for human skeleton motion recognition based on parallel convolutional neural network according to claim 1, characterized in that: In the step 5), the adjustment of the convolution kernel size of the two-dimensional convolutional network CNN includes 1×1 size, 1×3 size and 3×3 size.

7. The method for human skeleton motion recognition based on parallel convolutional neural network according to claim 1, characterized in that: In the step 6), the adjustment of the number of stack layers of the parallel convolutional network includes 1, 2, 3, 4 and 5, and the adjustment of the channel dimension of the middle layer includes 64, 64*2 and 64*4.

Images