A method for intelligent dynamic gesture recognition

Abstract

This invention discloses an intelligent dynamic gesture recognition method, comprising: constructing initial node features of a dynamic gesture sequence obtained based on Leap Motion into a tensor form as subsequent input; designing a spatial feature extraction module based on a hypergraph neural network to learn the spatial features of the dynamic gesture sequence; designing a temporal feature extraction module based on a hypergraph neural network to learn the temporal features of the dynamic gesture sequence; constructing a spatiotemporal feature fusion classification module based on node spatial and temporal features to effectively fuse the spatiotemporal features of nodes and obtain the final gesture recognition result; and using a cross-entropy loss function to constrain network training based on the spatial feature extraction module, temporal feature extraction module, and spatiotemporal feature fusion classification module of the hypergraph neural network to obtain a dynamic gesture recognition model based on the hypergraph neural network. This invention utilizes the hypergraph neural network to effectively mine the potential higher-order correlations of gesture data, achieving more accurate dynamic gesture recognition.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and human-computer interaction, and in particular to an intelligent dynamic gesture recognition method. Background Technology

[0002] Gesture recognition, as a natural and intuitive non-verbal interaction technology, has been widely applied in various fields such as virtual games, smart homes, and autonomous driving. However, in practical applications, gesture recognition still faces many challenges, such as changes in lighting, background interference, and the difficulty in customizing and expanding gesture types. Therefore, improving the accuracy and generalization ability of gesture recognition models is of significant research importance and practical application value.

[0003] Existing gesture recognition methods are mainly divided into wearable device-based methods and computer vision algorithm-based methods. Wearable device-based gesture recognition methods typically require users to wear bulky gloves and use mechanical or electronic sensors attached to the hand to achieve gesture recognition. These methods are generally poorly adaptable to changes in ambient light and background, and have low ease of use and flexibility. Unlike wearable device-based methods, computer vision algorithm-based gesture recognition methods can achieve gesture recognition without external sensors, greatly improving the ease of use and flexibility of gesture recognition, and are gradually becoming a current research hotspot.

[0004] In practical applications, gesture recognition methods based on computer vision algorithms typically utilize Leap Motion controllers to acquire 3D pose information of the hand skeleton for effective gesture recognition. Early Leap Motion-based gesture recognition methods often employed traditional machine learning algorithms. In recent years, with the rise of deep learning, methods that construct 3D hand pose information as a graph structure and process it using Graph Neural Networks (GNNs) have gradually demonstrated superior performance in gesture recognition. For example, Hou et al. constructed a residual graph convolutional neural network based on the 3D pose information of the hand skeleton for dynamic gesture recognition. Chen et al. proposed a spatiotemporally correlated hand skeleton graph structure, learning node and edge features through a self-attention mechanism for gesture recognition. However, GNNs primarily rely on the propagation and aggregation of information from local neighbor nodes, which has limitations in extracting the complex intra-frame and inter-frame high-order correlations of 3D pose information in dynamic gesture sequences. In contrast, applying Hypergraph Neural Networks (HGNNs) to gesture recognition methods can simultaneously process the relationships between multiple nodes in the hand, effectively uncovering potential higher-order associations in dynamic gesture data and improving the accuracy and robustness of dynamic gesture recognition models. In conclusion, exploring effective gesture recognition methods based on Hypergraph Neural Networks is feasible. Summary of the Invention

[0005] Common gesture recognition methods based on ordinary graph neural networks struggle to effectively capture the complex intra- and inter-frame node correlations in dynamic gesture sequences, given the 3D hand pose information provided by Leap Motion. This invention proposes an intelligent dynamic gesture recognition method that leverages hypergraph neural networks to effectively mine the latent higher-order correlations in gesture data, achieving more accurate dynamic gesture recognition. Details are described below:

[0006] A method for intelligent dynamic gesture recognition, the method comprising:

[0007] The dynamic gesture sequences acquired based on Leap Motion are sampled at equal intervals to extract t frames of data. The initial node features of each dynamic gesture sequence are constructed into a tensor form as subsequent input.

[0008] Design a spatial feature extraction module based on a hypergraph neural network to learn the spatial features of dynamic gesture sequences;

[0009] Design a temporal feature extraction module based on a hypergraph neural network to learn the temporal features of dynamic gesture sequences;

[0010] Based on the spatial and temporal features of nodes, a spatiotemporal feature fusion classification module is constructed to effectively fuse the spatiotemporal features of nodes and obtain the final gesture recognition result.

[0011] Based on the spatial feature extraction module, temporal feature extraction module, and spatiotemporal feature fusion classification module of the hypergraph neural network, and using the cross-entropy loss function to constrain network training, a dynamic gesture recognition model based on the hypergraph neural network can be obtained.

[0012] By loading the best model obtained from training and taking the three-dimensional pose information of the dynamic gesture sequence to be tested as input, the gesture classification result can be obtained.

[0013] The spatial feature extraction module is as follows:

[0014] The initial node feature X is added to the corresponding spatial location encoding vector Ps to obtain the spatially encoded node feature.

[0015] Using spatial attention Att S Learning the spatial features of nodes, and applying spatial attention Att S The spatial attention weights corresponding to the degree of association between nodes at different time steps are set to 0, and spatial attention Att is used to achieve this. S Preliminary acquisition of node spatial features X As ;

[0016] Each finger's joint points are combined into a hyperedge, and the hyperedges in each frame of the sampled dynamic gesture sequence are concatenated to form a hypergraph correlation matrix H, in order to model the higher-order correlation of the finger's spatial structure.

[0017] The spatial features of the nodes extracted by spatial attention X As The hypergraph association matrix H is used as the input to the hypergraph convolutional layer, and the hypergraph convolution is used to further extract the spatial features of the gesture sequence. The hypergraph convolutional layer can be expressed by the following formula:

[0018]

[0019]

[0020] Among them, D e and D v Let represent the degree matrices of the edges and vertices, respectively; W is the weight matrix. θ represents the node spatial features of the l-th layer input; θ represents the parameters to be learned during training; σ represents the nonlinear activation function.

[0021] The temporal feature extraction module of the hypergraph neural network is as follows:

[0022] Using the initial node feature X as input, a temporal position encoding operation is performed on the initial node feature X. The initial node feature X is added to the corresponding temporal position embedding vector Pt to obtain the temporally position encoded node feature.

[0023] Using time attention Att T Learning the temporal characteristics of nodes, and applying temporal attention Att T In this context, the temporal attention weights corresponding to the correlation between nodes with the same time step are set to 0 to block the transmission of spatial domain information. This is achieved through temporal attention (Att). T Preliminary acquisition of node temporal features X At ;

[0024] Time attention Att T Extracted node temporal features X At The hypergraph association matrix H is used as the input to the hypergraph convolutional layer, and the hypergraph convolution is used to further extract the temporal features of the dynamic gesture sequence. The formula is expressed as follows:

[0025]

[0026]

[0027] in, The node temporal features are input to the l-th layer.

[0028] The beneficial effects of the technical solution provided by this invention are:

[0029] 1. This invention proposes an intelligent dynamic gesture recognition method by utilizing the ability of hypergraph neural networks to model higher-order correlations between multiple nodes. This method fully explores the higher-order correlations of hand joints in the temporal and spatial domains, thereby achieving more effective dynamic gesture recognition.

[0030] 2. This invention designs a spatial feature extraction module and a temporal feature extraction module based on a hypergraph neural network. The hypergraph neural network is used to better capture the complex dependencies and interactions between three-dimensional hand joints in dynamic gesture sequences. This module effectively models the spatiotemporal correlation of joints and has better expressive and generalization capabilities. Attached Figure Description

[0031] Figure 1 This is a flowchart of an intelligent dynamic gesture recognition method. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0033] I. Processing dynamic gesture data based on Leap Motion

[0034] Based on Leap Motion, dynamic gesture sequences are obtained. Each frame of the sequence contains three-dimensional pose information of 21 joints of the hand, including the x, y, and z coordinates of the finger joints in space. In this embodiment of the invention, each dynamic gesture sequence is sampled at equal intervals to extract t frames of data. The initial node feature X of each dynamic gesture sequence is constructed into a tensor form of t×21×3 as subsequent input.

[0035] For example, in this embodiment of the invention, each dynamic gesture sequence is sampled at equal intervals to extract 8 frames of data. The initial node features X of each dynamic gesture sequence are constructed as an 8×21×3 tensor and used as the input of the subsequent hypergraph neural network.

[0036] II. Design of a spatial feature extraction module based on a hypergraph neural network

[0037] For the initial node features X input, this embodiment of the invention designs a spatial feature extraction module based on a hypergraph neural network to learn the spatial features of dynamic gesture sequences.

[0038] Since the initial node features X do not contain spatial location information of the hand joints, spatial location encoding is performed on the initial node features X to introduce the spatial location information of the joints, in order to facilitate the model's understanding and processing of the features and relationships of different joints. Specifically, the initial node features X are added to the corresponding spatial location encoding vector Ps to obtain the spatially encoded node features.

[0039] Then, using spatial attention Att S Learn the spatial features of nodes. To achieve spatial attention Att... S Focusing on the spatial domain, att spatial attention S The spatial attention weights corresponding to the correlation between nodes at different time steps are set to 0 to block the transmission of temporal information. This is achieved through spatial attention Att. S Preliminary acquisition of node spatial features X As .

[0040] Simultaneously, a hypergraph correlation matrix is ​​constructed based on the anatomical structure of the hand skeleton, namely the skeletal structure of the palm and fingers. Specifically, the joints of each finger are grouped into a hyperedge, and the hyperedges in each frame of the sampled dynamic gesture sequence are concatenated to form a hypergraph correlation matrix H, in order to model the higher-order correlations of the finger spatial structure.

[0041] The spatial features of the nodes extracted by spatial attention X AsThe hypergraph association matrix H is used as the input to the hypergraph convolutional layer, and the hypergraph convolution is used to further extract the spatial features of the gesture sequence. The hypergraph convolutional layer can be expressed by the following formula:

[0042]

[0043]

[0044] Among them, D e and D v Let represent the degree matrices of the edges and vertices, respectively; W is the weight matrix. θ represents the node spatial features of the l-th layer input; θ represents the parameters to be learned during training; σ represents the nonlinear activation function.

[0045] III. Design of a Temporal Feature Extraction Module Based on Hypergraph Neural Network

[0046] Similar to the spatial feature extraction module, the temporal feature extraction module based on the hypergraph neural network takes the initial node features X as input and performs temporal position encoding on the initial node features X. Specifically, the initial node features X are added to the corresponding temporal position embedding vector Pt to obtain the temporally position encoded node features.

[0047] Then use time attention Att T Learn the temporal characteristics of nodes. To achieve temporal attention Att... T Focusing on the time domain, att of time attention T The temporal attention weights corresponding to the correlation between nodes with the same time step are set to 0 to block the transmission of spatial domain information. This is achieved through temporal attention (Att). T Preliminary acquisition of node temporal features X At .

[0048] Time attention Att T Extracted node temporal features X At The hypergraph association matrix H is used as the input to the hypergraph convolutional layer, and the hypergraph convolution is used to further extract the temporal features of the dynamic gesture sequence. The formula is expressed as follows:

[0049]

[0050]

[0051] in, The node temporal features are input to the l-th layer.

[0052] IV. Constructing a Spatiotemporal Feature Fusion Classification Module

[0053] By utilizing the proposed spatial and temporal feature extraction modules based on hypergraph neural networks, temporal and spatial node features of dynamic gesture sequences are effectively extracted. Based on node spatial features... and node time series characteristics This invention constructs a spatiotemporal feature fusion classification module to effectively fuse the spatiotemporal features of nodes and obtain the final gesture recognition result. Specifically, it first fuses spatial features... and time series characteristics The network is cascaded, and then average pooling is used to obtain distributed feature maps. Finally, a fully connected layer is used to map the distributed features learned by the network into the sample label space to obtain the dynamic gesture recognition classification result. The formula is expressed as follows:

[0054]

[0055] Here, cat(·) is the feature concatenation operation, and ap(·) is the average pooling operation. It is a fully connected layer, where pred is the probability distribution of the input sample mapped to different categories of dynamic gestures, that is, the result of the distributed features learned by the network being mapped to the sample label space.

[0056] V. Training the Dynamic Gesture Recognition Network

[0057] The dynamic gesture recognition network proposed in this embodiment of the invention includes: a spatial feature extraction module, a temporal feature extraction module, and a spatiotemporal feature fusion classification module based on a hypergraph neural network. A cross-entropy loss function is used to constrain network training, resulting in a dynamic gesture recognition model based on a hypergraph neural network. The formula for the cross-entropy loss function is as follows:

[0058]

[0059]

[0060] Where M is the number of dynamic gesture categories; N is the number of samples; y nm The sign function is set to 1 if sample n belongs to dynamic gesture category m, and 0 otherwise; p nm This is the probability predicted by the model that sample n belongs to class m.

[0061] During testing, the best model obtained from training is loaded, and the three-dimensional pose information of the dynamic gesture sequence to be tested is used as input to obtain the gesture classification result.

[0062] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0063] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A smart dynamic gesture recognition method, characterized in that, The method comprises: The dynamic gesture sequence based on the Leap Motion is taken for equal interval sampling, and the frame data is extracted The initial node features of each dynamic gesture sequence are constructed in a tensor form as subsequent input. a spatial feature extraction module based on a hypergraph neural network is designed to learn spatial features of a dynamic gesture sequence; a temporal feature extraction module based on a hypergraph neural network is designed to learn temporal features of the dynamic gesture sequence; based on the node spatial features and the node temporal features, a spatio-temporal feature fusion classification module is constructed to effectively fuse the spatio-temporal features of the nodes and obtain a final gesture recognition result; based on the spatial feature extraction module, the temporal feature extraction module and the spatio-temporal feature fusion classification module based on the hypergraph neural network, a cross-entropy loss function is used to constrain network training, and a dynamic gesture recognition model based on the hypergraph neural network can be obtained; the best model obtained by training is loaded, and three-dimensional posture information of a dynamic gesture sequence to be tested is taken as input, so that a gesture classification result can be obtained; wherein the spatial feature extraction module is: obtaining initial node features adding the corresponding spatial position encoding vector obtaining the node features after spatial position encoding ; Utilizing spatial attention learning the spatial features of the nodes, setting the spatial attention weights corresponding to the degrees of association of the nodes at different time steps in the middle to 0, and utilizing spatial attention learning the spatial features of the nodes, setting the spatial attention weights corresponding to the degrees of association of the nodes at different time steps in the middle to 0, and utilizing spatial attention preliminarily acquiring the spatial features of the nodes ; The joint of each finger is composed of a hyperedge, and the hyperedges in each frame of the sampled dynamic gesture sequence are cascaded to form a hypergraph association matrix to model high-order correlations of finger spatial structure; The node spatial features extracted by the spatial attention and the hypergraph adjacency matrix As the input of the hypergraph convolution layer, the gesture sequence spatial features are further extracted by hypergraph convolution The hypergraph convolution layer is expressed by the following formula: ； ； wherein, and denote the edge and vertex degree matrices, respectively; is the weight matrix; is the first layer input node space feature; is the parameter to be learned during the training process; denotes a nonlinear activation function; wherein the temporal feature extraction module of the hypergraph neural network is: The initial node features are input The initial node features are input The initial node features are input The initial node features are input The initial node features are input ; Utilizing time attention Learning the temporal characteristics of nodes and applying temporal attention In this context, the temporal attention weights corresponding to the correlation between nodes with the same time step are set to 0 to block the transmission of spatial domain information. Preliminary acquisition of node temporal features ; Attending to time extracted node temporal features and hypergraph adjacency matrix As the input of the hypergraph convolution layer, the hypergraph convolution is further used to extract the dynamic gesture sequence temporal features , which can be expressed as follows: ； ； wherein, is the first node timing feature input to the layer.

Images