Dynamic hand gesture recognition method based on multi-view 3D skeleton information fusion

Abstract

The present invention proposes a dynamic gesture recognition method based on multi-view three-dimensional skeleton information fusion. It includes: using a sliding window-based detection method to segment the original dynamic multi-gesture sequence to obtain multiple single gesture sequences; performing a three-dimensional space coordinate transformation on each single gesture sequence to obtain the corresponding multi-view three-dimensional skeleton information; encoding each multi-view three-dimensional skeleton information to obtain a single-view total skeleton map corresponding to multiple perspectives; inputting the single-view total skeleton map of each perspective into the corresponding branch convolutional neural network for feature extraction, and then inputting it into the aggregation network based on the perspective attention mechanism, and then inputting it into the flattening layer and the fully connected layer in turn, and the fully connected layer outputs the single gesture classification result. The present invention can solve the problems of insufficient utilization of spatial information, difficult recognition of complex gestures, poor robustness, etc. in the traditional gesture recognition method using a single perspective, and greatly improves the recognition accuracy.

Description

Technical Field

[0001] The invention belongs to the field of human-computer interaction and relates to a dynamic gesture recognition method, in particular to a dynamic gesture recognition method based on multi-view three-dimensional skeleton information fusion. Background Art

[0002] In the field of human-computer interaction, dynamic gestures, as a natural and efficient communication medium, have been applied to robot remote control, virtual assembly, sign language recognition and many other aspects. As the most important link in intelligent gesture interaction, dynamic gesture recognition has received more and more attention from researchers. Existing dynamic recognition methods can be divided into image-based recognition methods and hand skeleton data-based recognition methods according to different input data. Among them, the image-based method is easily affected by changes in illumination, making it difficult to extract hand features. The method based on three-dimensional skeleton data is very robust to changes in illumination and background and has a small amount of computation, so it has become a mainstream research method.

[0003] Scholars at home and abroad have conducted a series of studies on gesture recognition based on 3D skeleton data. Some researchers have classified gestures by artificially designing features. However, the features designed in this way have poor distinguishing ability, resulting in a low gesture recognition rate. With the rapid development of deep neural networks, some scholars have used recurrent neural networks (RNN) and long short-term memory units (LSTM) to extract features from 3D skeleton data, achieving good classification results. However, when such networks encounter longer sequences, they will have a gradient vanishing problem, resulting in the inability to further optimize the network. With the introduction of convolutional neural networks (CNN) into the field of image classification, many scholars have used CNN to extract deep features from 3D skeleton data, which are further used for gesture classification, and have made some progress.

[0004] However, existing gesture recognition methods based on CNN three-dimensional skeleton data all use gesture sequences observed from a single perspective as input. This method ignores the impact of the important factor of "perspective" on the accuracy of gesture recognition. The impact of "perspective" on gesture recognition rate is mainly concentrated in two aspects. On the one hand, observing gesture sequences from different perspectives can better capture the spatial structure information of gesture sequences, and this spatial information helps to more comprehensively describe the characteristics of gesture sequences; on the other hand, some gestures are difficult to recognize from a single perspective, but may be easier to recognize from another perspective. Summary of the invention

[0005] In order to solve the problems existing in the above-mentioned background technology, the present invention proposes a dynamic gesture image recognition method based on multi-view three-dimensional skeleton information fusion. By making full use of multi-view skeleton information, the present invention can solve the problems of insufficient utilization of spatial information, difficulty in recognizing complex gestures, poor robustness, etc. in the traditional gesture recognition method using a single view. The method proposed in the present invention is of great significance for improving the accuracy of dynamic gesture recognition, can be simply and effectively deployed in other classification tasks, and has a wide range of application value.

[0006] The technical solution of the present invention is as follows, and mainly comprises the following steps:

[0007] S1: After segmenting and processing the original dynamic multi-gesture sequence using a sliding window-based detection method, multiple single-gesture sequences of equal length are obtained;

[0008] S2: Transform the three-dimensional space coordinates of each single gesture sequence to obtain the corresponding multi-view three-dimensional skeleton information;

[0009] S3: Encode the current multi-view 3D skeleton information to obtain a single-view total skeleton map corresponding to multiple views, where the single-view total skeleton map of each view consists of an X-coordinate, a Y-coordinate, and a Z-coordinate skeleton map;

[0010] S4: Inputting the single-view total skeleton graph of each view into the corresponding branch convolutional neural network for feature extraction, respectively, to obtain the corresponding single-view depth features, and forming the multi-view depth features of the current single gesture sequence from multiple single-view depth features;

[0011] S5: Input the multi-view deep features of the current single gesture sequence into the aggregation network based on the view attention mechanism for feature aggregation to generate the global features of the current single gesture sequence;

[0012] S6: inputting the global features of the current single gesture sequence into the flattening layer and the fully connected layer in turn for gesture classification, and the fully connected layer outputs the classification result of the current single gesture sequence;

[0013] S7: Repeat S2-S6 to classify the remaining single gesture sequences to obtain corresponding gesture classification results.

[0014] The S1 is specifically:

[0015] S11: performing sliding detection on the original dynamic multi-gesture sequence using a sliding detection window of a fixed length to determine the start position and the end position of each gesture, locating the corresponding gesture according to the start position and the end position of each gesture, and then dividing the original dynamic multi-gesture sequence into multiple gesture sequences with single gestures as a single gesture sequence;

[0016] S12: using a two-dimensional linear interpolation method to adjust the length of each single gesture sequence so that the lengths of each single gesture sequence are equal.

[0017] In S2, each single gesture sequence uses the following formula to perform three-dimensional space coordinate transformation, and the specific formula is as follows:

[0018]

[0019]

[0020]

[0021] Among them, s i t,j Represents the set of skeleton point coordinates of the i-th perspective of the t-th frame in the current single gesture sequence, They represent the X, Y, and Z coordinates of the jth bone point in the ith view of the tth frame in the current single gesture sequence, i∈[1,N], N represents the total number of viewpoints; t∈[1,T], T represents the total length of the current single gesture sequence; j∈[1,J], J represents the total number of bone points, T represents the transposition operation, and x t,j ,y t,j ,z t,j represents the X-coordinate, Y-coordinate, and Z-coordinate of the j-th bone point in the t-th frame of the current single gesture sequence, and represents the Z-axis rotation matrix. Indicates the rotation angle around the Z axis of the current single gesture sequence at the i-th viewing angle.

[0022] In S3, the current multi-perspective three-dimensional skeleton information is split in the perspective dimension to obtain three-dimensional skeleton information of multiple single perspectives. Based on the three-dimensional skeleton information of each single perspective, the spatial order of each skeleton point is re-encoded according to the link relationship between the skeleton points to obtain a single-perspective total skeleton map of each perspective.

[0023] The structures of the branch convolutional neural networks in S4 are the same, including six convolutional layers, a dimension transformation layer and four pooling layers.

[0024] The input of the branched convolutional neural network is input into the first convolutional layer, the first convolutional layer is connected to the second convolutional layer, the second convolutional layer is connected to the dimension transformation layer, the dimension transformation layer is connected to the fourth pooling layer after passing through the third convolutional layer, the first pooling layer, the fourth convolutional layer, the second pooling layer, the fifth convolutional layer, the third pooling layer, and the sixth convolutional layer in sequence. The output of the fourth pooling layer is used as the output of the branched convolutional neural network, and the branched convolutional neural network outputs the single-view depth features of the current single gesture sequence.

[0025] The multi-branch convolutional neural networks are trained in a parameter-sharing manner.

[0026] The aggregation network based on the perspective attention mechanism in S5 includes multiple convolutional layers, average pooling layers, maximum pooling layers and activation layers. Multiple single-perspective depth features in the multi-perspective depth features of the current single gesture sequence are respectively input into the corresponding convolutional layers for feature dimension compression. The dimension compression features output by each convolutional layer are spliced ​​to obtain a mixed feature. The mixed feature is respectively input into the average pooling layer and the maximum pooling layer for attention weight calculation to obtain the average perspective attention weight and the maximum perspective attention weight, respectively. The average perspective attention weight and the maximum perspective attention weight are element-wise summed and input into the activation layer. The activation layer outputs the perspective attention weight. The perspective attention weight and the multi-perspective depth feature of the current single gesture sequence are fused into a global feature by vector dot multiplication and output.

[0027] In each of the multi-branch convolutional neural networks, a first dropout layer is further arranged between the third convolutional layer and the first pooling layer, and a second dropout layer is further arranged between the fourth convolutional layer and the second pooling layer.

[0028] In each of the multi-branch convolutional neural networks, a first LeakyRelu activation function layer is further arranged between the fifth convolutional layer and the third pooling layer, a second LeakyRelu activation function layer is further arranged between the sixth convolutional layer and the fourth pooling layer, and the output of the fourth pooling layer is used as the output of the multi-branch convolutional neural network.

[0029] The beneficial effects of the present invention are:

[0030] 1. High classification accuracy of the model: Taking the three-dimensional skeleton information of multiple perspectives as input, rich spatial structure information of dynamic gestures can be obtained. Compared with a single perspective, the available information increases, which is equivalent to a way of information augmentation, which helps to classify gestures more accurately. By designing an aggregation network based on the perspective attention mechanism, the multi-perspective deep features can be organically integrated to generate more distinguishable global information. Through testing on public datasets, this method has a higher accuracy than existing methods.

[0031] 2. Strong model adaptability: By effectively utilizing spatial information from various perspectives, both rough and fine gestures commonly used in human-computer interaction can be accurately classified. The parameter sharing mechanism used in the multi-branch convolutional neural network can effectively reduce the number of parameters, allowing the model to be deployed on other classification tasks with strong adaptability.

[0032] In summary, the method proposed in the present invention can solve the problems of insufficient utilization of spatial information, difficulty in recognizing complex gestures, and poor robustness in traditional gesture recognition methods using a single perspective by making full use of multi-view skeleton information. The method proposed in the present invention can be simply and effectively deployed in other classification tasks and has a wide range of application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the process of the present invention;

[0034] Figure 2 It is the overall framework diagram of the method of the present invention;

[0035] Figure 3 This is a structural diagram of a branched convolutional neural network used to extract single-view features in the present invention;

[0036] Figure 4 This is a diagram of the aggregation network structure based on the perspective attention mechanism in the present invention;

[0037] Figure 5 It is a convergence analysis diagram of the network during the training process of the present invention;

[0038] Figure 6 A schematic diagram of the confusion matrix of the gesture classification result of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below with reference to the accompanying drawings and specific experiments conducted using the public data set SHREC'17 Track Dataset.

[0040] The embodiment of the present invention uses the public data set SHREC'17 Track Dataset for model training and testing. The data set uses the Intel Real Sense depth camera for data collection and contains 2800 gesture sequences from 28 participants. The data set contains a total of 28 gesture categories, which are divided into two major categories: single-finger gestures and full-hand gestures. Each major category contains 14 commonly used gestures. In this embodiment, the training set contains 1960 gesture sequences and the test set contains 840 gesture sequences.

[0041] like Figure 1 and Figure 2 As shown, the present invention comprises the following steps:

[0042] S1: The original dynamic multi-gesture sequence is segmented using a sliding window-based detection method to obtain multiple single gesture sequences of equal length;

[0043] S1 is specifically:

[0044] S11: performing sliding detection on the original dynamic multi-gesture sequence using a sliding detection window of a fixed length to determine the start position and the end position of each gesture, locating the corresponding gesture according to the start position and the end position of each gesture, and then dividing the original dynamic multi-gesture sequence into multiple gesture sequences with single gestures as a single gesture sequence, so as to facilitate subsequent gesture recognition of the single gesture;

[0045] S12: In order to solve the problem of different frame numbers of gesture sequences in the original data set, a two-dimensional linear interpolation method is used to adjust the length of each single gesture sequence so that the lengths of each single gesture sequence are equal, which is convenient for subsequent feature extraction by convolutional neural network. In this embodiment, the length of each single gesture sequence is 32.

[0046] S2: In order to avoid the influence of the initial coordinate position on gesture recognition, the coordinate data is transformed into a coordinate system. The origin of each sequence is set at the wrist joint point of the first frame.

[0047] Then, a circular uniform arrangement method is adopted to arrange N=8 virtual cameras around the gesture sequence, with an interval angle of 45° between each camera, so as to observe the spatial structure of the gesture sequence from multiple angles.

[0048] Since observing the gesture sequence from multiple angles is equivalent to directly rotating the gesture sequence around the coordinate axis, each single gesture sequence is transformed into a three-dimensional space coordinate to obtain the corresponding multi-view three-dimensional skeleton information;

[0049] In S2, each single gesture sequence uses the following formula to transform the three-dimensional space coordinates. The specific formula is as follows:

[0050]

[0051]

[0052]

[0053] Among them, s i t,j Represents the set of skeleton point coordinates of the i-th perspective of the t-th frame in the current single gesture sequence, They represent the X, Y, and Z coordinates of the jth bone point in the ith view of the tth frame in the current single gesture sequence, i∈[1,N], N represents the total number of viewpoints; t∈[1,T], T represents the total length of the current single gesture sequence; j∈[1,J], J represents the total number of bone points, T represents the transposition operation, and x t,j ,y t,j ,z t,j represents the X-coordinate, Y-coordinate, and Z-coordinate of the j-th bone point in the t-th frame of the current single gesture sequence, and represents the Z-axis rotation matrix. Indicates the rotation angle of the current single gesture sequence around the Z axis at the i-th viewing angle. This method is equivalent to arranging multiple virtual cameras around the gesture sequence, achieving the purpose of observing the spatial structure of the gesture sequence from multiple viewing angles. The X-axis indicates the direction of movement of the gesture, the Y-axis indicates the viewing direction of the camera, and the Z-axis indicates the vertical upward direction perpendicular to the XY plane.

[0054] S3: Encode the current multi-view 3D skeleton information to obtain a single-view total skeleton map corresponding to multiple views. The single-view total skeleton map of each view is composed of X-coordinate, Y-coordinate and Z-coordinate skeleton maps; the dimension size of the single-view total skeleton map of each view is T×J×3, where T and J are the total length of the sequence and the total number of skeleton points, respectively.

[0055] In S3, the current multi-perspective three-dimensional skeleton information is split in the perspective dimension to obtain multiple single-perspective three-dimensional skeleton information. Based on the three-dimensional skeleton information of each single perspective, the spatial order of each skeleton point is re-encoded according to the link relationship between the skeleton points to obtain the single-perspective total skeleton map of each perspective.

[0056] S4: Inputting the single-view total skeleton graph of each view into the corresponding branch convolutional neural network for feature extraction, respectively, to obtain the corresponding single-view depth features, and forming the multi-view depth features of the current single gesture sequence from multiple single-view depth features;

[0057] like Figure 3 As shown in Figure 1, the structures of the convolutional neural networks of each branch in S4 are the same, including six convolutional layers (Conv), a dimension transformation layer and four pooling layers (Pooling).

[0058] The input of the branched convolutional neural network is input into the first convolutional layer, which is connected to the second convolutional layer. The convolution kernel sizes of the first and second convolutional layers are set to 1×1 and 3×1 respectively, where "1" indicates that the dimensions of the two layers in the bone joints are 1. The second convolutional layer is connected to the dimension transformation layer, which is connected to the fourth pooling layer after passing through the third convolutional layer, the first pooling layer, the fourth convolutional layer, the second pooling layer, the fifth convolutional layer, the third pooling layer, and the sixth convolutional layer in sequence. The output of the fourth pooling layer is used as the output of the branched convolutional neural network, which outputs the single-view depth features of the current single gesture sequence. The first and second convolutional layers are used to extract shallow features; the dimension transformation layer swaps the channel dimension and the bone joint dimension through matrix transposition transformation; the third, fourth, fifth, and sixth convolutional layers are used to extract depth information, and the four pooling layers are used to ensure feature invariance.

[0059] In each multi-branch convolutional neural network, a first dropout layer is provided between the third convolutional layer and the first pooling layer, and a second dropout layer is provided between the fourth convolutional layer and the second pooling layer.

[0060] In each multi-branch convolutional neural network, a first LeakyRelu activation function layer is provided between the fifth convolutional layer and the third pooling layer, a second LeakyRelu activation function layer is provided between the sixth convolutional layer and the fourth pooling layer, and the output of the fourth pooling layer is used as the output of the multi-branch convolutional neural network.

[0061] The multi-branch convolutional neural networks are trained by sharing parameters, which helps reduce the number of parameters and improves the training efficiency of the neural network.

[0062] S5: Input the multi-view deep features of the current single gesture sequence into the aggregation network based on the view attention mechanism for feature aggregation to generate the global features of the current single gesture sequence;

[0063] like Figure 4 As shown in FIG. 1 , the aggregation network based on the perspective attention mechanism in S5 includes multiple convolutional layers, average pooling layers, maximum pooling layers and activation layers. The convolution kernel of the convolutional layer is 1*1. Multiple single-perspective depth features in the multi-perspective depth features of the current single gesture sequence are respectively input into the corresponding convolutional layers for feature dimension compression. The dimension compression features output by each convolutional layer are concatenated to obtain mixed features. The mixed features are respectively input into the average pooling layer and the maximum pooling layer for attention weight calculation to obtain the average perspective attention weight and the maximum perspective attention weight, respectively. The average perspective attention weight and the maximum perspective attention weight are element-wise summed and then input into the activation layer. The activation layer outputs the perspective attention weight. The perspective attention weight and the multi-perspective depth features of the current single gesture sequence are fused into a global feature by vector dot multiplication and output.

[0064] S6: inputting the global features of the current single gesture sequence into the flattening layer and the fully connected layer in turn for gesture classification, and the fully connected layer outputs the classification result of the current single gesture sequence;

[0065] S7: Repeat S2-S6 to classify the remaining single gesture sequences to obtain corresponding gesture classification results.

[0066] In the model training phase, this method uses Pytorch as the development platform and uses NVIDIA 1060TI GPU for parallel computing to speed up the model training. The training set is divided into different batches for training, and the data size of each batch is set to 64. During the training process, the cross entropy function is used to calculate the loss value of the model. The Adam algorithm is used to optimize the model parameters, the learning rate is set to 0.001, and a total of 50 cycles are trained.

[0067] During the testing phase, 840 gesture sequences in the test set were input into the trained model for testing to obtain the gesture classification results and recognition accuracy.

[0068] In order to verify the effectiveness of this method, three different multi-view feature fusion methods were compared. Method 1 is the element-wise addition method. This method adds the feature values ​​of each view feature, which has the advantage of ensuring that the spatial position and feature dimension of the feature remain unchanged. Method 2 is the feature concatenation method. This method concatenates multi-view features in the channel dimension, which can increase feature information while ensuring that the features of the spatial dimension remain unchanged. Method 3 is the maximum pooling method (max_pooling). This method takes the maximum feature value of the corresponding position of each view feature as the final feature value.

[0069] like Figure 5 As shown in Figure 2, this method is compared with the above three methods to obtain the convergence analysis diagram of the network during the model training process. Figure 5 It can be seen that method 2 has improved convergence speed and recognition accuracy compared with method 1. The reason is that feature splicing can retain the original information of each perspective. Method 3 has further improved compared with the first two methods, proving that the use of maximum pooling can better highlight the information of each position in the feature. Compared with the above three methods, this method has higher accuracy, faster convergence speed and smaller data jitter. It proves that this method can more accurately extract useful information from multi-view features and has strong robustness.

[0070] In order to further verify the superiority of this method over the traditional single-view method, several methods that use traditional features to describe gestures and use recurrent neural networks (RNN) and convolutional neural networks (CNN) to extract gesture information are compared. These methods include HON4D, HOG2, SoCJ, RNN and CNN. As can be seen from Table 1, the method proposed in this invention is significantly better than the traditional method in gesture recognition effect because it makes full use of the skeleton information of multiple views, which further verifies the effectiveness of this method.

[0071] Table 1 compares the gesture recognition accuracy of different methods

[0072]

[0073]

[0074] like Figure 6 As shown in the figure, the recognition result of each gesture is displayed through the confusion matrix. The gestures used by this method to recognize include 14 common gestures: 1. Grab (G); 2. Tap (T); 3. Expand (E); 4. Pinch (P); 5. Rotate clockwise (R-CW); 6. Rotate counterclockwise (R-CCW); 7. Slide right (SR); 8. Slide left (SL); 9. Slide up (SU); 10. Slide down (SD); 11. X-shaped slide (SX); 12. V-shaped slide (SV); 13. +-shaped slide (S-+); 14. Waving (Sh). For 11 of the gestures, the recognition effect of this method exceeds 90%. For all gestures, the average recognition rate of this method reaches 93.62%.

[0075] The method proposed in the present invention can accurately classify the SHREC'17Track Dataset. At the same time, the dynamic gesture recognition method based on multi-view 3D skeleton information fusion proposed in the present invention can solve the problems of insufficient spatial information utilization, difficult recognition of complex gestures, poor robustness, etc. in the traditional gesture recognition method using a single view by making full use of multi-view skeleton information. The method proposed in the present invention can be simply and effectively deployed in other classification tasks and has a wide range of application value.

[0076] The above-mentioned example is only the result of the present invention on this example, but the specific implementation of the present invention is not limited to this example. All alternative solutions with similar effects proposed according to the principles and ideas of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A dynamic gesture recognition method based on multi-view three-dimensional skeleton information fusion, characterized in that: The following steps are involved: S1: After segmenting and processing the original dynamic multi-gesture sequence using a sliding window-based detection method, multiple single-gesture sequences of equal length are obtained; S2: Transform the three-dimensional space coordinates of each single gesture sequence to obtain the corresponding multi-view three-dimensional skeleton information; S3: Encode the current multi-view 3D skeleton information to obtain a single-view total skeleton map corresponding to multiple views, where the single-view total skeleton map of each view consists of an X-coordinate, a Y-coordinate, and a Z-coordinate skeleton map; S4: Inputting the single-view total skeleton graph of each view into the corresponding branch convolutional neural network for feature extraction, respectively, to obtain the corresponding single-view depth features, and forming the multi-view depth features of the current single gesture sequence from multiple single-view depth features; S5: Input the multi-view deep features of the current single gesture sequence into the aggregation network based on the view attention mechanism for feature aggregation to generate the global features of the current single gesture sequence; S6: inputting the global features of the current single gesture sequence into the flattening layer and the fully connected layer in turn for gesture classification, and the fully connected layer outputs the classification result of the current single gesture sequence; S7: repeat S2-S6 to classify the remaining single gesture sequences to obtain corresponding gesture classification results; The aggregation network based on the perspective attention mechanism in S5 includes multiple convolutional layers, average pooling layers, maximum pooling layers and activation layers. Multiple single-perspective depth features in the multi-perspective depth features of the current single gesture sequence are respectively input into the corresponding convolutional layers for feature dimension compression. The dimension compression features output by each convolutional layer are spliced ​​to obtain a mixed feature. The mixed feature is respectively input into the average pooling layer and the maximum pooling layer for attention weight calculation to obtain the average perspective attention weight and the maximum perspective attention weight, respectively. The average perspective attention weight and the maximum perspective attention weight are element-wise summed and input into the activation layer. The activation layer outputs the perspective attention weight. The perspective attention weight and the multi-perspective depth feature of the current single gesture sequence are fused into a global feature by vector dot multiplication and output.

2. The method for dynamic gesture recognition based on multi-view three-dimensional skeleton information fusion according to claim 1, characterized in that: The S1 is specifically: S11: performing sliding detection on the original dynamic multi-gesture sequence using a sliding detection window of a fixed length to determine the start position and the end position of each gesture, locating the corresponding gesture according to the start position and the end position of each gesture, and then dividing the original dynamic multi-gesture sequence into multiple gesture sequences with single gestures as a single gesture sequence; S12: using a two-dimensional linear interpolation method to adjust the length of each single gesture sequence so that the lengths of each single gesture sequence are equal.

3. The method for dynamic gesture recognition based on multi-view three-dimensional skeleton information fusion according to claim 1, characterized in that: In S2, each single gesture sequence uses the following formula to perform three-dimensional space coordinate transformation, and the specific formula is as follows: Among them, s i t,j Represents the set of skeleton point coordinates of the i-th perspective of the t-th frame in the current single gesture sequence, They represent the X, Y, and Z coordinates of the jth bone point in the ith view of the tth frame in the current single gesture sequence, i∈[1,N], N represents the total number of viewpoints; t∈[1,CH], CH represents the total length of the current single gesture sequence; j∈[1,J], J represents the total number of bone points, T represents the transposition operation, x t,j ,y t,j ,z t,j represents the X, Y, and Z coordinates of the jth bone point in the tth frame of the current single gesture sequence. represents the Z-axis rotation matrix, Indicates the rotation angle around the Z axis of the current single gesture sequence at the i-th viewing angle.

4. The method for dynamic gesture recognition based on multi-view three-dimensional skeleton information fusion according to claim 1, characterized in that: In S3, the current multi-perspective three-dimensional skeleton information is split in the perspective dimension to obtain three-dimensional skeleton information of multiple single perspectives. Based on the three-dimensional skeleton information of each single perspective, the spatial order of each skeleton point is re-encoded according to the link relationship between the skeleton points to obtain a single-perspective total skeleton map of each perspective.

5. The method for dynamic gesture recognition based on multi-view three-dimensional skeleton information fusion according to claim 1, characterized in that: The structures of the convolutional neural networks of each branch in S4 are the same, including six convolutional layers, a dimension transformation layer and four pooling layers. The input of the branched convolutional neural network is input into the first convolutional layer, the first convolutional layer is connected to the second convolutional layer, the second convolutional layer is connected to the dimension transformation layer, the dimension transformation layer is connected to the fourth pooling layer after passing through the third convolutional layer, the first pooling layer, the fourth convolutional layer, the second pooling layer, the fifth convolutional layer, the third pooling layer, and the sixth convolutional layer in sequence. The output of the fourth pooling layer is used as the output of the branched convolutional neural network, and the branched convolutional neural network outputs the single-view depth features of the current single gesture sequence.

6. The method for dynamic gesture recognition based on multi-view three-dimensional skeleton information fusion according to claim 1, characterized in that: Each multi-branch convolutional neural network is trained by sharing parameters.

7. The method for dynamic gesture recognition based on multi-view three-dimensional skeleton information fusion according to claim 5, characterized in that: In each multi-branch convolutional neural network, a first dropout layer is provided between the third convolutional layer and the first pooling layer, and a second dropout layer is provided between the fourth convolutional layer and the second pooling layer.

8. The method for dynamic gesture recognition based on multi-view three-dimensional skeleton information fusion according to claim 5, characterized in that: In each multi-branch convolutional neural network, a first LeakyRelu activation function layer is provided between the fifth convolutional layer and the third pooling layer, a second LeakyRelu activation function layer is provided between the sixth convolutional layer and the fourth pooling layer, and the output of the fourth pooling layer is used as the output of the multi-branch convolutional neural network.

Images