Gesture recognition method and device based on dynamic hypergraph convolution

Abstract

The invention discloses a gesture recognition method and device based on dynamic hypergraph convolution, and the method comprises a network structure with a dynamic hypergraph convolution block and a full connection layer, the network structure executes the method, and the method comprises the following steps: obtaining a to-be-recognized gesture image; extracting a high-order relation feature between hand nodes in the gesture image to be recognized through the dynamic hypergraph convolution block to obtain a first hypergraph incidence matrix; randomly generating a second hypergraph incidence matrix, wherein element values in the second hypergraph incidence matrix are generated through network training; calculating the linear sum of the first hypergraph incidence matrix and the second hypergraph incidence matrix to obtain a third hypergraph incidence matrix; and outputting a gesture prediction result through the full connection layer according to the third hypergraph incidence matrix. And elements of the third hypergraph incidence matrix can calculate different element values according to different input samples, so that dynamic hypergraph convolution is realized, and the gesture recognition effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human motion recognition, and in particular to a gesture recognition method and device based on dynamic hypergraph convolution. Background Art

[0002] With the advancement of artificial intelligence technology, the use of deep learning models for gesture recognition has become a trend. For example, gesture recognition is currently performed through a graph convolutional network (GCN). GCN is mainly divided into spectral methods and spatial methods. The spectral method uses Fourier transform based on the Laplace matrix to perform graph convolution operations in the spectral domain. The spatial method directly aggregates information from a node and its neighboring nodes, and extracts and normalizes features through pre-designed rules. Since the spatial method is simpler and more efficient, it has become the mainstream method.

[0003] In spatial methods, adjacency matrices are often used to represent the relationships between graph nodes. For example, ST-GCN (Spatiotemporal Graph Convolutional Network) proposed in prior art 1 (YanS, Xiong Y, Lin D. "Spatial temporal graph convolutional networks for skeleton-based action recognition") is a typical spatial method that applies the adjacency matrix of the skeleton graph to GCN. Prior art 2 (K. He, X. Zhang, S. Ren, J. Sun, "Deepestual learning for image recognition") proposed a network structure consisting of several stacked spatial blocks and temporal blocks, and used residual connections to prevent gradient disappearance. The spatial block uses GCN to aggregate spatial information from a single frame, while the temporal block uses one-dimensional convolution to connect information between frames along the temporal dimension. ST-GCN outperforms the CNN (Convolutional Neural Network / RNN (Recurrent Neural Network) method, but the adjacency matrix of its graph remains unchanged during training. This makes it difficult to integrate node information with a long jump distance, resulting in the inability to solve the long-distance dependency problem.

[0004] To solve the above problems, subsequent studies introduced learnable adjacency matrices that continuously adjust the relationship between nodes during training. For example, prior art 3 (Shi L, Zhang Y, Cheng J, et al. "Two-stream adaptive graph convolutional networks for skeleton-based action recognition") first proposed to discover the relationship between nodes through network training. Prior art 4 (Chen Y, Zhang Z, Yuan C, et al. "Channel-wise topology refinement graph convolution for skeleton-based action recognition") proposed CTR-GCN (Channel Topology Graph Convolutional Neural Network) to find a specific adjacency matrix for each channel. Prior art 5 (J. Liu, X. Wang, C. Wang, Y. Gao, M. Liu, "Temporal decoupling graph convolutional network for skeleton-based gesture recognition") considers that the dependency between joints changes over time, and proposes TD-GCN (Temporal Decoupling Graph Convolutional Network), which adopts the strategy of calculating the adjacency matrix frame by frame.

[0005] Although the above research works have alleviated the long-distance dependency problem to a certain extent, they all initialize the adjacency matrix based on the physiological bone structure. This results in the adjacency matrix finally trained to be more biased towards the relationship between nodes in the physiological skeleton. Since the key nodes in the input data are not real joints, this initialization method introduces some bias.

[0006] In order to further solve the long-distance dependency problem and eliminate such bias, the existing technology 6 (Lee J, Lee M, Lee D, et al. "Hierarchically decomposed graph convolutional networks for skeleton-based action recognition") defines multiple derived skeleton graph structures that directly connect nodes within a certain area. The existing technology 7 (H. Duan, J. Wang, K. Chen, D. Lin, "Dg-stgcn: Dynamic spatial-temporal modeling for skeleton-based action recognition") randomly initializes K adjacency matrices and performs training to avoid the tedious process of initializing the adjacency matrix. However, in practical applications, the correlation between multiple joint points is very important, but due to the property that an edge in the graph can only connect two points, the graph cannot express the correlation between multiple points at the same time. Therefore, the graph convolutional neural network cannot extract high-order correlation features between multiple points. Summary of the invention The technical problem to be solved by the present invention is to provide a gesture recognition method and device based on dynamic hypergraph convolution, which can extract high-order correlation features to improve the gesture recognition effect.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: A gesture recognition method based on dynamic hypergraph convolution, having a network structure of a dynamic hypergraph convolution block and a fully connected layer, wherein the network structure executes the method, comprising the following steps: Obtaining a gesture image to be recognized; Extracting high-order relationship features between hand nodes in the gesture image to be recognized through the dynamic hypergraph convolution block to obtain a first hypergraph association matrix; Randomly generating a second hypergraph association matrix, wherein element values ​​in the second hypergraph association matrix are generated through network training; Calculate the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix; The gesture prediction result is outputted through the fully connected layer according to the third hypergraph association matrix.

[0008] In order to solve the above technical problems, another technical solution adopted by the present invention is: A gesture recognition device based on dynamic hypergraph convolution includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Obtaining a gesture image to be recognized; Extracting high-order relationship features between hand nodes in the gesture image to be recognized through the dynamic hypergraph convolution block to obtain a first hypergraph association matrix; Randomly generating a second hypergraph association matrix, wherein element values ​​in the second hypergraph association matrix are generated through network training; Calculate the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix; The gesture prediction result is outputted through the fully connected layer according to the third hypergraph association matrix.

[0009] The beneficial effects of the present invention are as follows: a first hypergraph association matrix is ​​obtained by using a dynamic hypergraph convolution block to express an input gesture image to be recognized, which can naturally express a non-Euclidean structure, and a hyperedge in the dynamic hypergraph convolution block can connect multiple points, so compared with ordinary graph convolutional neural networks, high-order association relationships between multiple points can be extracted; at the same time, the element values ​​in the second hypergraph association matrix are obtained randomly, and the relationship between each point is completely obtained by training, and is not affected by the hand skeleton graph, so the association information between farther points on the graph can be associated to solve the long-distance dependency problem; thus, after the first hypergraph association matrix and the second hypergraph association matrix are linearly and calculated to obtain a third hypergraph association matrix, the elements of the third hypergraph association matrix can calculate different element values ​​according to different input samples, thereby realizing dynamic hypergraph convolution and improving gesture recognition effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flowchart of a method for gesture recognition based on dynamic hypergraph convolution in an embodiment of the present invention; Figure 2 A network structure diagram of a gesture recognition method based on dynamic hypergraph convolution in an embodiment of the present invention; Figure 3 Detailed structural diagram of a dynamic hypergraph convolution block in a network structure according to an embodiment of the present invention; Figure 4 This is a structural diagram of a multi-stream fusion strategy of a gesture recognition method based on dynamic hypergraph convolution in an embodiment of the present invention.

[0011] Figure 5 4 is a structural diagram of a gesture recognition device based on dynamic hypergraph convolution in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0013] A gesture recognition method based on dynamic hypergraph convolution, having a network structure of a dynamic hypergraph convolution block and a fully connected layer, wherein the network structure executes the method, comprising the following steps: Obtaining a gesture image to be recognized; Extracting high-order relationship features between hand nodes in the gesture image to be recognized through the dynamic hypergraph convolution block to obtain a first hypergraph association matrix; Randomly generating a second hypergraph association matrix, wherein element values ​​in the second hypergraph association matrix are generated through network training; Calculate the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix; The gesture prediction result is outputted through the fully connected layer according to the third hypergraph association matrix.

[0014] From the above description, it can be seen that the beneficial effects of the present invention are: the first hypergraph association matrix is ​​obtained by using a dynamic hypergraph convolution block to express the input gesture image to be recognized, which can naturally express the non-Euclidean structure, and a hyperedge in the dynamic hypergraph convolution block can connect multiple points. Therefore, compared with the ordinary graph convolutional neural network, the high-order association relationship between multiple points can be extracted; at the same time, the element values ​​in the second hypergraph association matrix are obtained randomly, and the relationship between each point is completely obtained by training, and is not affected by the hand skeleton graph. Therefore, the association information between farther points on the graph can be associated to solve the long-distance dependency problem; thereby, the first hypergraph association matrix and the second hypergraph association matrix are linearly and calculated to obtain the third hypergraph association matrix, so that the elements of the third hypergraph association matrix can calculate different element values ​​according to different input samples, thereby realizing dynamic hypergraph convolution and improving the gesture recognition effect.

[0015] Furthermore, extracting high-order relationship features between hand nodes in the gesture image to be identified by hypergraph convolution to obtain a first hypergraph association matrix includes: ; in, represents the first hypergraph incidence matrix; Represents a 1x1 convolution kernel; represents average pooling along the time dimension; Represents the gesture image to be recognized; The weight matrix representing the first hypergraph incidence matrix.

[0016] From the above description, it can be seen that by calculating the above formula for each sample, the element value in the first hypergraph association matrix can be accurately obtained.

[0017] Further, the calculating the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix includes: ; Outputting the gesture prediction result according to the third hypergraph association matrix through the fully connected layer includes: ; in, represents the third hypergraph incidence matrix; represents the second hypergraph incidence matrix; Indicates H a With H b When linear summation H b The coefficient of represents the first hypergraph incidence matrix; Represents the processing result of the dynamic hypergraph convolution block; represents the vertex degree matrix; A weight matrix representing the third hypergraph incidence matrix; represents the hyperedge degree matrix; Represents the activation function ReLU; Represents the gesture image to be recognized; Represents the weight matrix used to change the number of input data channels.

[0018] From the above description, it can be seen that after obtaining the third hypergraph association matrix through the linear sum of the first hypergraph association matrix and the second hypergraph association matrix, the third hypergraph association matrix is ​​processed through the vertex degree matrix, the hyperedge degree matrix and the weight matrix to output accurate gesture prediction results.

[0019] Further, outputting the gesture prediction result according to the third hypergraph association matrix includes: Creating the network structure with different numbers of hyperedges; Inputting the gesture image to be recognized into different network structures to obtain different third hypergraph association matrices and prediction results corresponding to each third hypergraph association matrix; All the prediction results are combined to obtain a gesture prediction result.

[0020] From the above description, it can be seen that by changing the number of hyperedges to create different structures for the network structure model, and after predicting the gesture image to be recognized based on the different structures, the prediction results are combined to obtain the gesture prediction results, thereby providing more diverse and complementary information during fusion, and improving the accuracy and robustness of the overall gesture recognition.

[0021] Further, comprising at least four of said network structures; Inputting the gesture image to be recognized into different network structures to obtain different prediction results includes: Processing the gesture image to be recognized to obtain joint data, bone data and motion data; Inputting the joint data, the bone data and the motion data into three different network structures respectively, and inputting the joint data or the motion data into another different network structure, to obtain four sets of data prediction results; The gesture prediction result obtained by combining all the prediction results includes: The four groups of data prediction results are multiplied by preset weights respectively and then added together to obtain the gesture prediction result.

[0022] From the above description, it can be seen that by processing the gesture image to be recognized to obtain joint data, bone data and motion data, and inputting the corresponding data into different network structures for prediction, it is possible to make predictions for different types of data separately, and finally multiply the different data prediction results by preset weights to obtain more accurate gesture prediction results.

[0023] Another embodiment of the present invention provides a gesture recognition device based on dynamic hypergraph convolution, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Obtaining a gesture image to be recognized; Extracting high-order relationship features between hand nodes in the gesture image to be recognized through the dynamic hypergraph convolution block to obtain a first hypergraph association matrix; Randomly generating a second hypergraph association matrix, wherein element values ​​in the second hypergraph association matrix are generated through network training; Calculate the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix; The gesture prediction result is outputted through the fully connected layer according to the third hypergraph association matrix.

[0024] From the above description, it can be seen that the use of dynamic hypergraph convolution blocks to express the input gesture image to be recognized to obtain the first hypergraph association matrix can naturally express non-Euclidean structures, and a hyperedge in the dynamic hypergraph convolution block can connect multiple points. Therefore, compared with ordinary graph convolutional neural networks, high-order association relationships between multiple points can be extracted; at the same time, the element values ​​in the second hypergraph association matrix are obtained randomly, and the relationship between each point is completely obtained by training, and is not affected by the hand skeleton graph. Therefore, the association information between farther points on the graph can be associated to solve the long-distance dependency problem; thus, the first hypergraph association matrix and the second hypergraph association matrix are linearly and calculated to obtain the third hypergraph association matrix, so that the elements of the third hypergraph association matrix can calculate different element values ​​according to different input samples, thereby realizing dynamic hypergraph convolution and improving gesture recognition effects.

[0025] Furthermore, extracting high-order relationship features between hand nodes in the gesture image to be identified by hypergraph convolution to obtain a first hypergraph association matrix includes: ; in, represents the first hypergraph incidence matrix; Represents a 1x1 convolution kernel; represents average pooling along the time dimension; Represents the processing result of the dynamic hypergraph convolution block; The weight matrix representing the first hypergraph incidence matrix.

[0026] From the above description, it can be seen that by calculating the above formula for each sample, the element value in the first hypergraph association matrix can be accurately obtained.

[0027] Further, the calculating the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix includes: ; Outputting the gesture prediction result according to the third hypergraph association matrix through the fully connected layer includes: ; in, represents the third hypergraph incidence matrix; represents the second hypergraph incidence matrix; Indicates H a With H b When linear summation H b The coefficient of represents the first hypergraph incidence matrix; Indicates the gesture prediction result; represents the vertex degree matrix; A weight matrix representing the third hypergraph incidence matrix; represents the hyperedge degree matrix; Represents the activation function ReLU; Represents the gesture image to be recognized; Represents the weight matrix used to change the number of input data channels.

[0028] From the above description, it can be seen that after obtaining the third hypergraph association matrix through the linear sum of the first hypergraph association matrix and the second hypergraph association matrix, the third hypergraph association matrix is ​​processed through the vertex degree matrix, the hyperedge degree matrix and the weight matrix to output accurate gesture prediction results.

[0029] Further, outputting the gesture prediction result according to the third hypergraph association matrix includes: Creating the network structure with different numbers of hyperedges; Inputting the gesture image to be recognized into different network structures to obtain different third hypergraph association matrices and prediction results corresponding to each third hypergraph association matrix; All the prediction results are combined to obtain a gesture prediction result.

[0030] From the above description, it can be seen that by changing the number of hyperedges to create different structures for the network structure model, and after predicting the gesture image to be recognized based on the different structures, the prediction results are combined to obtain the gesture prediction results, thereby providing more diverse and complementary information during fusion, and improving the accuracy and robustness of the overall gesture recognition.

[0031] Further, comprising at least four of said network structures; Inputting the gesture image to be recognized into different network structures to obtain different prediction results includes: Processing the gesture image to be recognized to obtain joint data, bone data and motion data; Inputting the joint data, the bone data and the motion data into three different network structures respectively, and inputting the joint data or the motion data into another different network structure, to obtain four sets of data prediction results; The gesture prediction result obtained by combining all the prediction results includes: The four groups of data prediction results are multiplied by preset weights respectively and then added together to obtain the gesture prediction result.

[0032] From the above description, it can be seen that by processing the gesture image to be recognized to obtain joint data, bone data and motion data, and inputting the corresponding data into different network structures for prediction, it is possible to make predictions for different types of data separately, and finally multiply the different data prediction results by preset weights to obtain more accurate gesture prediction results.

[0033] The gesture recognition method and device based on dynamic hypergraph convolution provided by the present invention can be applied to the recognition scenario of human posture, especially the recognition of gestures, which is described below through specific implementation methods: Embodiment 1 Please refer to Figure 1 , a gesture recognition method based on dynamic hypergraph convolution, with a network structure of dynamic hypergraph convolution blocks and fully connected layers; please refer to Figure 2The network structure is a dynamic hyper graph convolutional neural network (Dynamic Hyper Graph Convolutional Network, DHGCN) which specifically includes a dynamic hyper graph convolution block (DHGC Block), BN (BatchNormolization), T module and ReLU activation function stacked S times, for example, set to 10 times, and a subsequent T-pooling layer, N-pooling layer and a fully connected (FC) layer; the network structure execution method includes the following steps: S1. Acquire a gesture image to be recognized. The gesture image to be recognized may be an image set formed by multiple images, such as continuous image frames.

[0034] S2. Extract the high-order relationship features between the hand nodes in the gesture image to be identified through the dynamic hypergraph convolution block to obtain the first hypergraph association matrix; let the hypergraph ,in V represents a set of vertices, E Represents a set of edges, and the dynamic hypergraph convolution is defined as follows: ; in, represents the first hypergraph incidence matrix; Represents a 1x1 convolution kernel, whose input channel is , the output channel is ; represents average pooling along the time dimension; Represents the gesture image to be recognized; A weight matrix representing the first hypergraph incidence matrix; Please refer to Figure 3 , AvgPool in the figure corresponds to the average pooling operation , Conv_h corresponds to the convolution operation , Corresponding to the weight matrix operation, softmax is a normalization operation; that is, after the image input, it undergoes average pooling, convolution, weight matrix and normalization processing in sequence, and finally outputs the first hypergraph association matrix .

[0035] S3, randomly generate a second hypergraph association matrix, and the element values ​​in the second hypergraph association matrix are generated through network training; Figure 3 As shown, the second hypergraph association matrix Ha is randomly generated; the initial value of the second hypergraph association matrix Ha is obtained randomly, and the values ​​of each element are changed through network training, and each element is learnable; at the same time, the second hypergraph association matrix Ha is shared by all inputs. The value of Ha initialized at the beginning of training is random, and it is fixed after the training of the training set is completed.

[0036] S4. Calculate the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix. Specifically: ; The gesture prediction results outputted by the fully connected layer according to the third hypergraph association matrix include: ; in, represents the third hypergraph incidence matrix; represents the second hypergraph incidence matrix, is the hypergraph incidence matrix shared by all inputs, is the number of vertices, is the number of hyperedges; Indicates H a With H b When linear summation H b The coefficient can be automatically learned and adjusted through network training; Represents the output result of the dynamic hypergraph convolution block; represents the vertex degree matrix; Represents the weight matrix of the third hypergraph association matrix, which is used to convert D -1 H performs linear mapping; represents the hyperedge degree matrix; Represents the gesture image to be recognized; The weight matrix used to change the number of input data channels; Represents the activation function ReLU; Figure 3 As shown, Norm_row represents row normalization, Norm_col represents column normalization, W represents weight matrix operation, Conv_p represents the operation of the weight matrix that changes the number of input data channels, and the final output is .

[0037] S5: Output the gesture prediction result according to the third hypergraph association matrix through the fully connected layer; that is, the output result of the dynamic hypergraph convolution block , and then pass through the T-pooling layer, N-pooling layer and fully connected layer to get the gesture prediction result. That is, the dynamic hypergraph convolution block output Then enter BN->TM->ReLU and then enter the dynamic hypergraph convolution block, and so on, repeated ten times.

[0038] Embodiment 2 The difference between this embodiment and the first embodiment is that, on the basis of the first embodiment, a different model structure is used to obtain significantly different prediction vectors to solve the homogeneity problem. Specifically: 1. Create a network structure with different numbers of hyperedges; that is, based on the dynamic hypergraph convolutional neural network model (DHGCN) in Example 1, create different structures for the DHGCN model by changing the number of hyperedges; Figure 4 As shown, in an optional embodiment, four network structures with different numbers of hyperedges are created respectively; wherein different network structures can be trained by different training sets, for example, different network structures are trained respectively by joint data, bone data and motion data.

[0039] 2. Input the gesture image to be recognized into different network structures to obtain different third hypergraph association matrices and the prediction results corresponding to each third hypergraph association matrix. Specifically: 21. Input joint data, bone data and motion data into three different network structures respectively, and input joint data or motion data into another different network structure to obtain four sets of data prediction results. That is, before inputting the gesture images to be recognized into different network structures, the gesture images to be recognized need to be processed into different types; joint data is the original data without any processing; bone data is obtained by subtracting the nodes separated in the original data, such as the hand skeleton; motion data is obtained by subtracting the same nodes in consecutive frames.

[0040] 22. Input joint data, bone data and motion data into three different network structures respectively, and input joint data or motion data into another different network structure to obtain four sets of data prediction results. Each type of data is input into its corresponding model for training; Figure 4 As shown, j stream, jm stream, b stream and o stream are obtained respectively; wherein, o stream indicates that its input data type can be joint data or motion data, and the number of hyperedges is different from other streams; thus, j stream prediction results, jm stream prediction results, b stream prediction results and o stream prediction results are obtained.

[0041] 3. Combine all prediction results to get the gesture prediction result. That is, multiply the j stream prediction result, jm stream prediction result, b stream prediction result and o stream prediction result by the preset weights respectively, and add them together to get the gesture prediction result. The final prediction vector The calculation formula is as follows: ; in, , , The prediction vectors representing the predictions made by different data streams; Represents the weight values ​​corresponding to different data streams. By using different model structures to obtain significantly different prediction vectors, it can be ensured that each model structure learns different feature representations, thereby providing more diverse and complementary information when fused, and improving the accuracy and robustness of overall gesture recognition.

[0042] In this embodiment, the concept of hypergraph is used to represent the relationship between multiple points, so that the relationship between multiple points can be expressed more naturally; and the high-order correlation features between multiple points on the hand are extracted through the designed dynamic hypergraph convolution, making the model's ability to extract features better than other methods; and DHGCN with different numbers of hyperedges is used through different data stream training, so that the final prediction vectors of each stream have significant differences, thereby improving the robustness and accuracy of the final network.

[0043] The specific experimental results are shown on the DHG-14 / 28 and SHREC17 datasets: Table 1. Prediction performance on DHG14 / 28 compared with other models

[0044] Table 2 Prediction performance on SHREC17 compared with other models

[0045] It can be seen from Table 1 and Table 2 that the DHGCN model in this embodiment has better performance than other existing models.

[0046] Embodiment 3 Please refer to Figure 5 A gesture recognition device based on dynamic hypergraph convolution includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: S1, obtaining a gesture image to be recognized; S2. Extract high-order relationship features between hand nodes in the gesture image to be recognized through a dynamic hypergraph convolution block to obtain a first hypergraph association matrix. Specifically: ; in, represents the first hypergraph incidence matrix; Represents a 1x1 convolution kernel; represents average pooling along the time dimension; Represents the gesture image to be recognized; The weight matrix representing the first hypergraph incidence matrix.

[0047] S3, randomly generating a second hypergraph association matrix, wherein the element values ​​of each in the second hypergraph association matrix are generated through network training; S4. Calculate the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix. Specifically: ; The gesture prediction results outputted by the fully connected layer according to the third hypergraph association matrix include: ; in, represents the third hypergraph incidence matrix; represents the second hypergraph incidence matrix; Indicates H a With H b When linear summation H b The coefficient of represents the second hypergraph incidence matrix; Indicates the gesture prediction result; represents the vertex degree matrix; A weight matrix representing the third hypergraph incidence matrix; represents the hyperedge degree matrix; Represents the activation function ReLU; Represents the gesture image to be recognized; Represents the weight matrix used to change the number of input data channels.

[0048] S5. Output the gesture prediction result according to the third hypergraph association matrix through the fully connected layer.

[0049] In another optional embodiment, the method further comprises the following steps: A1. Create network structures with different numbers of hyperedges; for example, create at least four network structures; A2. Input the gesture image to be recognized into different network structures to obtain different third hypergraph association matrices and the prediction results corresponding to each third hypergraph association matrix. Specifically: process the gesture image to be recognized to obtain joint data, bone data and motion data; input the joint data, bone data and motion data into three different network structures respectively, and input the joint data or motion data into another different network structure to obtain four sets of data prediction results.

[0050] A3. All prediction results are combined to obtain a gesture prediction result. That is, the four sets of data prediction results are multiplied by preset weights respectively, and then added to obtain the gesture prediction result.

[0051] In summary, the present invention discloses a gesture recognition method and device based on dynamic hypergraph convolution, which uses a dynamic hypergraph convolution block to express the input gesture image to be recognized to obtain a first hypergraph association matrix, which can naturally express a non-Euclidean structure, and a hyperedge in the dynamic hypergraph convolution block can connect multiple points, so compared with the ordinary graph convolution neural network, it can extract high-order association relationships between multiple points; at the same time, the element values ​​in the second hypergraph association matrix are obtained randomly, and the relationship between each point is completely obtained by training, which is not affected by the hand skeleton map, so the association information between the farther points on the association map can be associated to solve the long-distance dependency problem; thereby, the first hypergraph association matrix and the second hypergraph association matrix are linearly and calculated to obtain a third hypergraph association matrix, so that the elements of the third hypergraph association matrix can calculate different element values ​​according to different input samples, realize dynamic hypergraph convolution, and improve the gesture recognition effect. At the same time, by training hypergraph convolution networks with different numbers of hyperedges for fusion prediction, the gesture recognition effect is further improved.

[0052] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A gesture recognition method based on dynamic hypergraph convolution, characterized in that: A network structure having a dynamic hypergraph convolution block and a fully connected layer, wherein the network structure performs the method, comprising the following steps: Obtaining a gesture image to be recognized; Extracting high-order relationship features between hand nodes in the gesture image to be recognized through the dynamic hypergraph convolution block to obtain a first hypergraph association matrix; Randomly generating a second hypergraph association matrix, wherein element values ​​in the second hypergraph association matrix are generated through network training; Calculate the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix; The gesture prediction result is outputted through the fully connected layer according to the third hypergraph association matrix.

2. The gesture recognition method based on dynamic hypergraph convolution according to claim 1, characterized in that: The step of extracting high-order relationship features between hand nodes in the gesture image to be identified by hypergraph convolution to obtain a first hypergraph association matrix includes: ； in, represents the first hypergraph incidence matrix; Represents a 1x1 convolution kernel; represents average pooling along the time dimension; Represents the gesture image to be recognized; The weight matrix representing the first hypergraph incidence matrix.

3. The gesture recognition method based on dynamic hypergraph convolution according to claim 1, characterized in that: The calculating the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix comprises: ； Outputting the gesture prediction result according to the third hypergraph association matrix through the fully connected layer includes: ； in, represents the third hypergraph incidence matrix; represents the second hypergraph incidence matrix; Indicates H a With H b When linear summation H b The coefficient of represents the first hypergraph incidence matrix; Represents the processing result of the dynamic hypergraph convolution block; represents the vertex degree matrix; A weight matrix representing the third hypergraph incidence matrix; represents the hyperedge degree matrix; Represents the activation function ReLU; Represents the gesture image to be recognized; Represents the weight matrix used to change the number of input data channels.

4. The gesture recognition method based on dynamic hypergraph convolution according to claim 1, characterized in that: Outputting the gesture prediction result according to the third hypergraph association matrix includes: Creating the network structure with different numbers of hyperedges; Inputting the gesture image to be recognized into different network structures to obtain different third hypergraph association matrices and prediction results corresponding to each third hypergraph association matrix; All the prediction results are combined to obtain a gesture prediction result.

5. The gesture recognition method based on dynamic hypergraph convolution according to claim 4, characterized in that: comprising at least four of the network structures; Inputting the gesture image to be recognized into different network structures to obtain different prediction results includes: Processing the gesture image to be recognized to obtain joint data, bone data and motion data; Inputting the joint data, the bone data and the motion data into three different network structures respectively, and inputting the joint data or the motion data into another different network structure, to obtain four sets of data prediction results; The gesture prediction result obtained by combining all the prediction results includes: The four groups of data prediction results are multiplied by preset weights respectively and then added together to obtain the gesture prediction result.

6. A gesture recognition device based on dynamic hypergraph convolution, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: Obtaining a gesture image to be recognized; Extracting high-order relationship features between hand nodes in the gesture image to be recognized through the dynamic hypergraph convolution block to obtain a first hypergraph association matrix; Randomly generate a second hypergraph association matrix, wherein the element value of each in the second hypergraph association matrix is ​​generated through network training; Calculate the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix; The gesture prediction result is outputted through the fully connected layer according to the third hypergraph association matrix.

7. The gesture recognition device based on dynamic hypergraph convolution according to claim 6, characterized in that: The step of extracting high-order relationship features between hand nodes in the gesture image to be identified by hypergraph convolution to obtain a first hypergraph association matrix includes: ； in, represents the first hypergraph incidence matrix; Represents a 1x1 convolution kernel; represents average pooling along the time dimension; Represents the processing result of the dynamic hypergraph convolution block; The weight matrix representing the first hypergraph incidence matrix.

8. The gesture recognition device based on dynamic hypergraph convolution according to claim 6, characterized in that: The calculating the linear sum of the first hypergraph association matrix and the second hypergraph association matrix to obtain a third hypergraph association matrix comprises: ； Outputting the gesture prediction result according to the third hypergraph association matrix through the fully connected layer includes: ； in, represents the third hypergraph incidence matrix; represents the second hypergraph incidence matrix; Indicates H a With H b When linear summation H b The coefficient of represents the first hypergraph incidence matrix; Represents the processing result of the dynamic hypergraph convolution block; represents the vertex degree matrix; A weight matrix representing the third hypergraph incidence matrix; represents the hyperedge degree matrix; Represents the activation function ReLU; Represents the gesture image to be recognized; Represents the weight matrix used to change the number of input data channels.

9. The gesture recognition device based on dynamic hypergraph convolution according to claim 6, characterized in that: Outputting the gesture prediction result according to the third hypergraph association matrix includes: Creating the network structure with different numbers of hyperedges; Inputting the gesture image to be recognized into different network structures to obtain different third hypergraph association matrices and prediction results corresponding to each third hypergraph association matrix; All the prediction results are combined to obtain a gesture prediction result.

10. The gesture recognition device based on dynamic hypergraph convolution according to claim 9, characterized in that: comprising at least four of the network structures; Inputting the gesture image to be recognized into different network structures to obtain different prediction results includes: Processing the gesture image to be recognized to obtain joint data, bone data and motion data; Inputting the joint data, the bone data and the motion data into three different network structures respectively, and inputting the joint data or the motion data into another different network structure, to obtain four sets of data prediction results; The gesture prediction result obtained by combining all the prediction results includes: The four groups of data prediction results are multiplied by preset weights respectively and then added together to obtain the gesture prediction result.

Images