A gesture recognition method and device based on dynamic hypergraph convolution

Abstract

The application discloses a gesture recognition method and device based on dynamic hypergraph convolution, and has a network structure with a dynamic hypergraph convolution block and a full connection layer, wherein the network structure performs the method and comprises the following steps: acquiring a gesture image to be recognized; extracting high-order relationship features between hand nodes in the gesture image to be recognized through the dynamic hypergraph convolution block to obtain a first hypergraph correlation matrix; randomly generating a second hypergraph correlation matrix, wherein element values in the second hypergraph correlation matrix are generated through network training; calculating a linear sum of the first hypergraph correlation matrix and the second hypergraph correlation matrix to obtain a third hypergraph correlation matrix; and outputting a gesture prediction result according to the third hypergraph correlation matrix through the full connection layer. The element values of the third hypergraph correlation matrix can be calculated according to different input samples, dynamic hypergraph convolution is realized, and the gesture recognition effect is improved.

Description

Technical Field

[0001] This invention relates to the field of human motion recognition technology, and in particular to a gesture recognition method and apparatus based on dynamic hypergraph convolution. Background Technology

[0002] With the advancement of artificial intelligence technology, using deep learning models for gesture recognition has become a trend. For example, gesture recognition is currently performed using Graph Convolutional Networks (GCNs). GCNs are mainly divided into spectral methods and spatial methods. Spectral methods utilize Fourier transforms based on the Laplacian matrix to perform graph convolution operations in the spectral domain. Spatial methods directly aggregate information from nodes and their neighboring nodes, extracting and normalizing features through pre-designed rules. Due to their simplicity and efficiency, spatial methods have become the mainstream approach.

[0003] In spatial methods, adjacency matrices are typically used to represent the relationships between graph nodes. For example, the ST-GCN (Spatiotemporal Graph Convolutional Network) proposed in Existing Technique 1 (Yan S, Xiong Y, Lin D. "Spatial temporal graph convolutional networks for skeleton-based action recognition") is a typical spatial method that applies the adjacency matrix of the skeleton graph to a GCN. Existing Technique 2 (K. He, X. Zhang, S. Ren, J. Sun. "Deepresidual learning for image recognition") proposes a network structure consisting of several stacked spatial and temporal blocks, using residual connections to prevent gradient vanishing. Spatial blocks use a GCN to aggregate spatial information from a single frame, while temporal blocks utilize one-dimensional convolutions to connect information between frames along the temporal dimension. ST-GCN outperforms CNN (Convolutional Neural Network) methods, but its graph adjacency matrix remains unchanged during training. This makes it difficult to integrate information from nodes with large jump distances, resulting in an inability to solve long-range dependency problems.

[0004] To address the aforementioned issues, subsequent research introduced learnable adjacency matrices, which continuously adjust the relationships between nodes during training. For example, Existing Technique 3 (Shi L, Zhang Y, Cheng J, et al., "Two-stream adaptive graph convolutional networks for skeleton-based action recognition") first proposed discovering relationships between nodes through network training. Existing Technique 4 (Chen Y, Zhang Z, Yuan C, et al., "Channel-wise topology refinement graph convolution for skeleton-based action recognition") proposed CTR-GCN (Channel Topology Graph Convolutional Network) to find a specific adjacency matrix for each channel. Existing Technique 5 (J. Liu, X. Wang, C. Wang, Y. Gao, M. Liu, "Temporal decoupling graph convolutional network for skeleton-based gesture recognition") considered the temporal changes in dependencies between joints and proposed TD-GCN (Temporal Decoupling Graph Convolutional Network), which employs a strategy of calculating the adjacency matrix frame-by-frame.

[0005] While the aforementioned studies have alleviated the long-distance dependency problem to some extent, they all initialize the adjacency matrix based on the physiological skeletal structure. This results in the final trained adjacency matrix being more biased towards the relationships between nodes in the physiological skeleton. Since the key nodes in the input data are not true joints, this initialization method introduces some biases.

[0006] To further address the long-distance dependency problem and eliminate such biases, existing technique 6 (Lee J, Lee M, Lee D, et al., "Hierarchically decomposed graph convolutional networks for skeleton-based action recognition") defines multiple derived skeleton graph structures that directly connect nodes within a certain region. Existing technique 7 (H. Duan, J. Wang, K. Chen, D. Lin, "Dg-stgcn: Dynamic spatial-temporal modeling for skeleton-based action recognition") avoids the tedious process of initializing adjacency matrices by randomly initializing K adjacency matrices during training. However, in practical applications, the correlation between multiple key points is crucial, but due to the property that an edge in a graph can only connect two points, a graph cannot simultaneously represent the correlation between multiple points. Therefore, graph convolutional neural networks cannot extract high-order correlation features between multiple points. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a gesture recognition method and apparatus based on dynamic hypergraph convolution, which can extract high-order correlation features to improve the gesture recognition effect.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A gesture recognition method based on dynamic hypergraph convolution, comprising a network structure with dynamic hypergraph convolutional blocks and fully connected layers, wherein the network structure executes the method, including the following steps:

[0010] Acquire the image of the gesture to be recognized;

[0011] The high-order relationship features between hand nodes in the gesture image to be recognized are extracted by the dynamic hypergraph convolutional block to obtain the first hypergraph association matrix;

[0012] A second hypergraph association matrix is ​​randomly generated, and the element values ​​in the second hypergraph association matrix are generated through network training.

[0013] Calculate the linear sum of the first hypergraph incidence matrix and the second hypergraph incidence matrix to obtain the third hypergraph incidence matrix;

[0014] The gesture prediction result is output through the fully connected layer based on the third hypergraph association matrix.

[0015] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:

[0016] A gesture recognition device based on dynamic hypergraph convolution includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0017] Acquire the image of the gesture to be recognized;

[0018] The high-order relationship features between hand nodes in the gesture image to be recognized are extracted by the dynamic hypergraph convolutional block to obtain the first hypergraph association matrix;

[0019] A second hypergraph association matrix is ​​randomly generated, and the element values ​​in the second hypergraph association matrix are generated through network training.

[0020] Calculate the linear sum of the first hypergraph incidence matrix and the second hypergraph incidence matrix to obtain the third hypergraph incidence matrix;

[0021] The gesture prediction result is output through the fully connected layer based on the third hypergraph association matrix.

[0022] The beneficial effects of this invention are as follows: A first hypergraph association matrix is ​​obtained by using dynamic hypergraph convolutional blocks to represent the input gesture image to be recognized. This matrix can naturally represent non-Euclidean structures, and a single hyperedge in the dynamic hypergraph convolutional block can connect multiple points. Therefore, compared to ordinary graph convolutional neural networks, it can extract higher-order relationships between multiple points. Simultaneously, the element values ​​in the second hypergraph association matrix are randomly obtained, entirely based on the relationships between points acquired during training, and are unaffected by the hand skeleton map. Therefore, it can associate the association information between distant points on the graph, solving the long-distance dependency problem. Furthermore, after linearly summing the first and second hypergraph association matrices to obtain the third hypergraph association matrix, the elements of the third hypergraph association matrix can calculate different element values ​​based on different input samples, realizing dynamic hypergraph convolution and improving gesture recognition performance. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the steps of a gesture recognition method based on dynamic hypergraph convolution in an embodiment of the present invention.

[0024] Figure 2 This is a network structure diagram of a gesture recognition method based on dynamic hypergraph convolution in an embodiment of the present invention;

[0025] Figure 3 This is a detailed structural diagram of the dynamic hypergraph convolutional block within the network structure in an embodiment of the present invention;

[0026] Figure 4 This is a structural diagram of a multi-stream fusion strategy for a gesture recognition method based on dynamic hypergraph convolution, as described in an embodiment of the present invention.

[0027] Figure 5 This is a structural diagram of a gesture recognition device based on dynamic hypergraph convolution in an embodiment of the present invention. Detailed Implementation

[0028] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0029] A gesture recognition method based on dynamic hypergraph convolution, comprising a network structure with dynamic hypergraph convolutional blocks and fully connected layers, wherein the network structure executes the method, including the following steps:

[0030] Acquire the image of the gesture to be recognized;

[0031] The high-order relationship features between hand nodes in the gesture image to be recognized are extracted by the dynamic hypergraph convolutional block to obtain the first hypergraph association matrix;

[0032] A second hypergraph association matrix is ​​randomly generated, and the element values ​​in the second hypergraph association matrix are generated through network training.

[0033] Calculate the linear sum of the first hypergraph incidence matrix and the second hypergraph incidence matrix to obtain the third hypergraph incidence matrix;

[0034] The gesture prediction result is output through the fully connected layer based on the third hypergraph association matrix.

[0035] As described above, the beneficial effects of this invention are as follows: A first hypergraph association matrix is ​​obtained by using dynamic hypergraph convolutional blocks to represent the input gesture image to be recognized. This matrix can naturally represent non-Euclidean structures, and a single hyperedge in the dynamic hypergraph convolutional block can connect multiple points. Therefore, compared to ordinary graph convolutional neural networks, it can extract higher-order relationships between multiple points. Simultaneously, the element values ​​in the second hypergraph association matrix are obtained randomly, entirely based on the relationships between points obtained through training, and are unaffected by the hand skeleton map. Therefore, it can associate the association information between distant points on the graph, solving the long-distance dependency problem. Finally, a third hypergraph association matrix is ​​obtained by linearly summing the first and second hypergraph association matrices. This allows the elements of the third hypergraph association matrix to calculate different element values ​​based on different input samples, realizing dynamic hypergraph convolution and improving gesture recognition performance.

[0036] Furthermore, the step of extracting high-order relationship features between hand nodes in the hand gesture image to be recognized through hypergraph convolution to obtain the first hypergraph association matrix includes:

[0037] ;

[0038] in, Represents the first hypergraph incidence matrix; This represents a 1x1 convolution kernel; This indicates average pooling along the time dimension; This represents the image of the gesture to be recognized. This represents the weight matrix of the first hypergraph correlation matrix.

[0039] As can be seen from the above description, by calculating for each sample using the above formula, the element values ​​in the first hypergraph correlation matrix can be obtained accurately.

[0040] Further, the step of calculating the linear sum of the first hypergraph incidence matrix and the second hypergraph incidence matrix to obtain the third hypergraph incidence matrix includes:

[0041] ;

[0042] The step of outputting the gesture prediction result through the fully connected layer based on the third hypergraph correlation matrix includes:

[0043] ;

[0044] in, Represents the third hypergraph incidence matrix; Represents the second hypergraph incidence matrix; H represents a With H b H when linearly summing b The coefficient; Represents the first hypergraph incidence matrix; This represents the processing result of the dynamic hypergraph convolutional block; Represents the vertex degree matrix; The weight matrix represents the correlation matrix of the third hypergraph; Represents the hypermarginality matrix; Represents the activation function ReLU; This represents the image of the gesture to be recognized. This represents the weight matrix used to change the number of input data channels.

[0045] As described above, after obtaining the third hypergraph correlation matrix by linearly summing the first and second hypergraph correlation matrices, the third hypergraph correlation matrix is ​​then processed by the vertex degree matrix, hyperedge degree matrix, and weight matrix to output accurate gesture prediction results.

[0046] Furthermore, the step of outputting the gesture prediction result based on the third hypergraph association matrix includes:

[0047] Create the network structures with different numbers of hyperedges;

[0048] By inputting the gesture image to be recognized into different network structures, different third hypergraph correlation matrices are obtained, as well as the prediction results corresponding to each third hypergraph correlation matrix;

[0049] All the prediction results are combined to obtain the gesture prediction result.

[0050] As described above, by changing the number of hyperedges to create different structures for the network structure model, and by predicting the gesture image to be recognized based on different structures, the prediction results are combined to obtain the gesture prediction result. This provides more diverse and complementary information during fusion, thereby improving the accuracy and robustness of the overall gesture recognition.

[0051] Furthermore, it includes at least four of the aforementioned network structures;

[0052] The step of inputting the gesture image to be recognized into different network structures to obtain different prediction results includes:

[0053] The image of the gesture to be recognized is processed to obtain joint data, bone data, and motion data;

[0054] The joint data, bone data, and motion data are respectively input into three different network structures, and the joint data or motion data is input into another different network structure to obtain four sets of data prediction results;

[0055] The step of combining all the prediction results to obtain the gesture prediction result includes:

[0056] The gesture prediction result is obtained by multiplying the four sets of data prediction results by preset weights and then summing them.

[0057] As described above, by processing the image of the gesture to be recognized to obtain joint data, skeletal data, and motion data, and inputting the corresponding data into different network structures for prediction, predictions can be made for different types of data. Finally, the prediction results of different data are multiplied by preset weights to obtain more accurate gesture prediction results.

[0058] Another embodiment of the present invention provides a gesture recognition device based on dynamic hypergraph convolution, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0059] Acquire the image of the gesture to be recognized;

[0060] The high-order relationship features between hand nodes in the gesture image to be recognized are extracted by the dynamic hypergraph convolutional block to obtain the first hypergraph association matrix;

[0061] A second hypergraph association matrix is ​​randomly generated, and the element values ​​in the second hypergraph association matrix are generated through network training.

[0062] Calculate the linear sum of the first hypergraph incidence matrix and the second hypergraph incidence matrix to obtain the third hypergraph incidence matrix;

[0063] The gesture prediction result is output through the fully connected layer based on the third hypergraph association matrix.

[0064] As described above, using dynamic hypergraph convolutional blocks to represent the input gesture image to be recognized to obtain the first hypergraph association matrix can naturally represent non-Euclidean structures. Furthermore, a single hyperedge in the dynamic hypergraph convolutional block can connect multiple points, thus enabling the extraction of higher-order relationships between multiple points compared to ordinary graph convolutional neural networks. Simultaneously, the element values ​​in the second hypergraph association matrix are randomly obtained, entirely based on the relationships between points acquired during training, and are unaffected by the hand skeleton map. Therefore, it can associate the association information between distant points on the graph, solving the long-distance dependency problem. Finally, the first and second hypergraph association matrices are linearly summed to obtain the third hypergraph association matrix. This allows the elements of the third hypergraph association matrix to calculate different element values ​​based on different input samples, achieving dynamic hypergraph convolution and improving gesture recognition performance.

[0065] Furthermore, the step of extracting high-order relationship features between hand nodes in the hand gesture image to be recognized through hypergraph convolution to obtain the first hypergraph association matrix includes:

[0066] ;

[0067] in, Represents the first hypergraph incidence matrix; This represents a 1x1 convolution kernel; This indicates average pooling along the time dimension; This represents the processing result of the dynamic hypergraph convolutional block; This represents the weight matrix of the first hypergraph correlation matrix.

[0068] As can be seen from the above description, by calculating for each sample using the above formula, the element values ​​in the first hypergraph correlation matrix can be obtained accurately.

[0069] Further, the step of calculating the linear sum of the first hypergraph incidence matrix and the second hypergraph incidence matrix to obtain the third hypergraph incidence matrix includes:

[0070] ;

[0071] The step of outputting the gesture prediction result through the fully connected layer based on the third hypergraph correlation matrix includes:

[0072] ;

[0073] in, Represents the third hypergraph incidence matrix; Represents the second hypergraph incidence matrix; H represents a With H b H when linearly summing b The coefficient; Represents the first hypergraph incidence matrix; Indicates the gesture prediction result; Represents the vertex degree matrix; The weight matrix represents the correlation matrix of the third hypergraph; Represents the hypermarginality matrix; Represents the activation function ReLU; This represents the image of the gesture to be recognized. This represents the weight matrix used to change the number of input data channels.

[0074] As described above, after obtaining the third hypergraph correlation matrix by linearly summing the first and second hypergraph correlation matrices, the third hypergraph correlation matrix is ​​then processed by the vertex degree matrix, hyperedge degree matrix, and weight matrix to output accurate gesture prediction results.

[0075] Furthermore, the step of outputting the gesture prediction result based on the third hypergraph association matrix includes:

[0076] Create the network structures with different numbers of hyperedges;

[0077] By inputting the gesture image to be recognized into different network structures, different third hypergraph correlation matrices are obtained, as well as the prediction results corresponding to each third hypergraph correlation matrix;

[0078] All the prediction results are combined to obtain the gesture prediction result.

[0079] As described above, by changing the number of hyperedges to create different structures for the network structure model, and by predicting the gesture image to be recognized based on different structures, the prediction results are combined to obtain the gesture prediction result. This provides more diverse and complementary information during fusion, thereby improving the accuracy and robustness of the overall gesture recognition.

[0080] Furthermore, it includes at least four of the aforementioned network structures;

[0081] The step of inputting the gesture image to be recognized into different network structures to obtain different prediction results includes:

[0082] The image of the gesture to be recognized is processed to obtain joint data, bone data, and motion data;

[0083] The joint data, bone data, and motion data are respectively input into three different network structures, and the joint data or motion data is input into another different network structure to obtain four sets of data prediction results;

[0084] The step of combining all the prediction results to obtain the gesture prediction result includes:

[0085] The gesture prediction result is obtained by multiplying the four sets of data prediction results by preset weights and then summing them.

[0086] As described above, by processing the image of the gesture to be recognized to obtain joint data, skeletal data, and motion data, and inputting the corresponding data into different network structures for prediction, predictions can be made for different types of data. Finally, the prediction results of different data are multiplied by preset weights to obtain more accurate gesture prediction results.

[0087] The gesture recognition method and apparatus based on dynamic hypergraph convolution provided by this invention can be applied to human posture recognition scenarios, especially for gesture recognition. The following is a description of specific implementation methods:

[0088] Example 1

[0089] Please refer to Figure 1 A gesture recognition method based on dynamic hypergraph convolution, featuring a network structure with dynamic hypergraph convolutional blocks and fully connected layers; please refer to... Figure 2 The network structure is a Dynamic Hyper Graph Convolutional Network (DHGCN), specifically including a Dynamic Hyper Graph Convolutional Block (DHGC Block), Batch Normalization (BN), a T module, and ReLU activation functions stacked S times (e.g., 10 times), followed by a T-pooling layer, an N-pooling layer, and a fully connected (FC) layer. The network structure execution method includes the following steps:

[0090] S1. Obtain the gesture image to be recognized; wherein, the gesture image to be recognized can be an image set formed by multiple images, such as consecutive image frames.

[0091] S2. Extract high-order relationship features between hand nodes in the hand gesture image to be recognized through dynamic hypergraph convolutional blocks to obtain the first hypergraph correlation matrix; let the hypergraph... ,in V Represents the set of vertices. ERepresenting a set of edges, dynamic hypergraph convolution is defined as follows:

[0092] ;

[0093] in, Represents the first hypergraph incidence matrix; This represents a 1x1 convolution kernel with 12 input channels. The output channel is ; This indicates average pooling along the time dimension; This represents the image of the gesture to be recognized. The weight matrix represents the correlation matrix of the first hypergraph;

[0094] Please refer to Figure 3 In the diagram, AvgPool corresponds to the average pooling operation. Conv_h corresponds to the convolution operation. , For the corresponding weight matrix operations, softmax is the normalization operation; that is, after the image input, it goes through average pooling, convolution, weight matrix, and normalization processing in sequence, and finally outputs the first hypergraph correlation matrix. .

[0095] S3. Randomly generate the second hypergraph incidence matrix. The element values ​​in the second hypergraph incidence matrix are generated through network training; for example... Figure 3 As shown, a second hypergraph association matrix Ha is randomly generated. The initial value of Ha is randomly obtained, and the values ​​of each element are subsequently changed during network training. Each element is learnable. Furthermore, the second hypergraph association matrix Ha is shared by all inputs. Initially, Ha is randomly initialized during training, but becomes a fixed value after training on the training set.

[0096] S4. Calculate the linear sum of the first hypergraph incidence matrix and the second hypergraph incidence matrix to obtain the third hypergraph incidence matrix. Specifically:

[0097] ;

[0098] The gesture prediction results output by the fully connected layer based on the third hypergraph correlation matrix include:

[0099] ;

[0100] in, Represents the third hypergraph incidence matrix; Represents the second hypergraph incidence matrix. It is a hypergraph correlation matrix shared by all inputs. It is the number of vertices. It is the number of superedges; H represents a With H b H when linearly summing b The coefficient can be automatically learned and adjusted through network training; This represents the output of a dynamic hypergraph convolution block. Represents the vertex degree matrix; The weight matrix represents the third hypergraph's correlation matrix, and its function is to weight D. -1 H is linearly mapped; Represents the hypermarginality matrix; This represents the image of the gesture to be recognized. A weight matrix used to change the number of input data channels; Represents the activation function ReLU; for example Figure 3 As shown, Norm_row represents row normalization, Norm_col represents column normalization, W represents weight matrix operation, Conv_p represents the operation of changing the number of input data channels in the weight matrix, and the final output... .

[0101] S5. Output the gesture prediction result through the fully connected layer based on the third hypergraph correlation matrix; that is, the output result of the dynamic hypergraph convolutional block. The gesture prediction result is then obtained after passing through a T-pooling layer, an N-pooling layer, and a fully connected layer. This is the output of the dynamic hypergraph convolutional block. Then it enters BN->TM->ReLU and then enters the dynamic hypergraph convolution block, and so on, repeating ten times.

[0102] Example 2

[0103] The difference between this embodiment and Embodiment 1 is that, based on Embodiment 1, a different model structure is used to obtain significantly different prediction vectors in order to solve the homogeneity problem. Specifically:

[0104] 1. Create network structures with different numbers of hyperedges; that is, based on the Dynamic Hypergraph Convolutional Neural Network (DHGCN) model in Example 1, create different structures for the DHGCN model by changing the number of hyperedges; such as... Figure 4 As shown, in an optional implementation, four network structures with different numbers of hyperedges are created; wherein, different network structures can be trained using different training sets, such as joint data, bone data and motion data.

[0105] 2. Input the gesture image to be recognized into different network structures to obtain different third hypergraph correlation matrices, and the prediction results corresponding to each third hypergraph correlation matrix. Specifically:

[0106] 21. Input joint data, skeletal data, and motion data into three different network structures respectively, and input either joint data or motion data into another different network structure to obtain four sets of data prediction results. That is, before inputting the gesture image to be recognized into different network structures, the gesture image to be recognized needs to be processed into different types; joint data is the raw data without any processing; skeletal data is obtained by subtracting nodes that are separated in the raw data, such as the hand skeleton; motion data is obtained by subtracting the same nodes in consecutive frames.

[0107] 22. Joint data, bone data, and motion data are input into three different network structures respectively, and either joint data or motion data is input into another different network structure, resulting in four sets of data prediction results. Each type of data is then input into its corresponding model for training; for example... Figure 4 As shown, the j-flow, jm-flow, b-flow, and o-flow are obtained respectively; where o-flow indicates that its input data type can be joint data or motion data, and the number of hyperedges is different from other flows; thus, the prediction results of j-flow, jm-flow, b-flow, and o-flow are obtained.

[0108] 3. Combine all prediction results to obtain the gesture prediction result. That is, multiply the j-flow prediction result, jm-flow prediction result, b-flow prediction result, and o-flow prediction result by preset weights, and then add them together to obtain the gesture prediction result, which is the final prediction vector. The calculation formula is as follows:

[0109] ;

[0110] in, , , The prediction vectors represent the predictions made by different data streams; This represents the weight values ​​corresponding to different data streams. By using different model structures to obtain significantly different prediction vectors, it is ensured that each model structure learns different feature representations, thereby providing more diverse and complementary information during fusion and improving the overall accuracy and robustness of gesture recognition.

[0111] In this embodiment, the concept of a hypergraph is used to represent the relationship between multiple points, so that the relationship between multiple points can be expressed more naturally. Furthermore, a well-designed dynamic hypergraph convolution is used to extract high-order correlation features between multiple points in the hand, making the model's feature extraction ability superior to other methods. In addition, DHGCN with different numbers of hyperedges is trained through different data streams, so that the prediction vectors of each stream have significant differences, improving the robustness and accuracy of the final network.

[0112] The specific experimental results are shown on the DHG-14 / 28 and SHREC17 datasets:

[0113] Table 1. Prediction performance on DHG14 / 28 compared with other models

[0114]

[0115] Table 2 Comparison of prediction performance on SHREC17 with other models

[0116]

[0117] As shown in Tables 1 and 2, the DHGCN model in this embodiment has better performance compared with other existing models.

[0118] Example 3

[0119] Please refer to Figure 5 A gesture recognition device based on dynamic hypergraph convolution includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0120] S1. Obtain the image of the gesture to be recognized;

[0121] S2. Extract high-order relationship features between hand nodes in the hand gesture image to be recognized through dynamic hypergraph convolutional blocks to obtain the first hypergraph association matrix. Specifically:

[0122] ;

[0123] in, Represents the first hypergraph incidence matrix; This represents a 1x1 convolution kernel; This indicates average pooling along the time dimension; This represents the image of the gesture to be recognized. This represents the weight matrix of the first hypergraph correlation matrix.

[0124] S3. Randomly generate the second hypergraph correlation matrix. The element values ​​in the second hypergraph correlation matrix are generated through network training.

[0125] S4. Calculate the linear sum of the first hypergraph incidence matrix and the second hypergraph incidence matrix to obtain the third hypergraph incidence matrix. Specifically:

[0126] ;

[0127] The gesture prediction results output by the fully connected layer based on the third hypergraph correlation matrix include:

[0128] ;

[0129] in, Represents the third hypergraph incidence matrix; Represents the second hypergraph incidence matrix; H represents a With H b H when linearly summing b The coefficient; Represents the second hypergraph incidence matrix; Indicates the gesture prediction result; Represents the vertex degree matrix; The weight matrix represents the correlation matrix of the third hypergraph; Represents the hypermarginality matrix; Represents the activation function ReLU; This represents the image of the gesture to be recognized. This represents the weight matrix used to change the number of input data channels.

[0130] S5. Output gesture prediction results based on the association matrix of the third hypergraph through the fully connected layer.

[0131] In another alternative implementation, the following steps are also included:

[0132] A1. Create network structures with different numbers of hyperedges; for example, create at least four network structures.

[0133] A2. Input the gesture image to be recognized into different network structures to obtain different third hypergraph correlation matrices and the prediction results corresponding to each third hypergraph correlation matrix. Specifically, process the gesture image to be recognized to obtain joint data, bone data and motion data; input the joint data, bone data and motion data into three different network structures respectively, and input the joint data or motion data into another different network structure to obtain four sets of data prediction results.

[0134] A3. Combine all prediction results to obtain the gesture prediction result. This involves multiplying each of the four sets of data prediction results by a preset weight and then summing them together to obtain the gesture prediction result.

[0135] In summary, this invention discloses a gesture recognition method and apparatus based on dynamic hypergraph convolution. It uses dynamic hypergraph convolution blocks to represent the input gesture image to be recognized, obtaining a first hypergraph association matrix. This matrix naturally represents non-Euclidean structures, and a single hyperedge in the dynamic hypergraph convolution block can connect multiple points. Therefore, compared to ordinary graph convolutional neural networks, it can extract higher-order relationships between multiple points. Simultaneously, the element values ​​in the second hypergraph association matrix are randomly obtained, entirely based on the relationships between points acquired during training, and are unaffected by the hand skeleton image. Therefore, it can associate association information between distant points in the graph, solving the long-distance dependency problem. A third hypergraph association matrix is ​​obtained by linearly summing the first and second hypergraph association matrices. The elements of the third hypergraph association matrix can calculate different values ​​based on different input samples, realizing dynamic hypergraph convolution and improving gesture recognition performance. Furthermore, by training hypergraph convolutional networks with different numbers of hyperedges for fusion prediction, the gesture recognition performance is further improved.

[0136] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A gesture recognition method based on dynamic hypergraph convolution, characterized in that, A network structure with a dynamic hypergraph convolution block and a full connection layer, which performs the method, comprising the following steps: Obtaining a gesture image to be recognized; Extracting high-order relationship features between hand nodes in the gesture image to be recognized through the dynamic hypergraph convolution block to obtain a first hypergraph correlation matrix: ； wherein, represents a first hypergraph incidence matrix; represents a 1x1 convolution kernel; represents an average pooling along the time dimension; represents a gesture image to be recognized; represents a weight matrix of the first hypergraph incidence matrix; Randomly generating a second hypergraph correlation matrix, the element values in the second hypergraph correlation matrix being generated through network training; Calculating the linear sum of the first hypergraph correlation matrix and the second hypergraph correlation matrix to obtain a third hypergraph correlation matrix; Outputting a gesture prediction result according to the third hypergraph correlation matrix through the full connection layer, comprising: Creating the network structure with different numbers of hyperedges; Inputting the gesture image to be recognized into different network structures to obtain different third hypergraph correlation matrices and prediction results corresponding to each third hypergraph correlation matrix; Combining all the prediction results to obtain a gesture prediction result. 2.The gesture recognition method based on dynamic hypergraph convolution of claim 1, wherein, The calculation of the linear sum of the first hypergraph correlation matrix and the second hypergraph correlation matrix to obtain a third hypergraph correlation matrix comprises: ； The output of a gesture prediction result according to the third hypergraph correlation matrix through the full connection layer comprises: ； wherein, denotes a third hypergraph incidence matrix; denotes a second hypergraph incidence matrix; denotes H a and H b coefficients of H b in linear summation; denotes a first hypergraph incidence matrix; denotes a processing result of a dynamic hypergraph convolution block; denotes a vertex degree matrix; denotes a weight matrix of the third hypergraph incidence matrix; denotes a hyperedge degree matrix; denotes an activation function ReLU; denotes a hand gesture image to be recognized; denotes a weight matrix for changing the number of input data channels. 3.The gesture recognition method based on dynamic hypergraph convolution of claim 1, wherein, Including at least four network structures; The input of the gesture image to be recognized into different network structures to obtain different prediction results comprises: Processing the gesture image to be recognized to obtain joint data, skeletal data and motion data; Inputting the joint data, skeletal data and motion data into three different network structures respectively, and inputting the joint data or motion data into another different network structure to obtain four groups of data prediction results; The combination of all the prediction results to obtain a gesture prediction result comprises: Multiplying four groups of data prediction results by preset weights and then adding them to obtain the gesture prediction result.

4. A gesture recognition apparatus based on dynamic hypergraph convolution, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize a gesture recognition method based on dynamic hypergraph convolution according to any one of claims 1-3.

Images