Gesture recognition method based on attention-guided spatial graph convolution simple recurrent unit

Abstract

The present invention discloses a gesture recognition method based on a simple recurrent unit with attention-guided spatial graph convolution. First, the present invention embeds the spatial graph convolution into the gate structure of the simple recurrent unit (SRU), so that the SRU with high-speed parallel computing capability can model the time domain and spatial domain information of complex gestures. Secondly, a finger joint attention guidance module is introduced to give more important finger joints a higher degree of attention. Then, a regularization method of attention-enhanced graph discarding is introduced to alleviate the disadvantage of overfitting of gesture data. The present invention has a high recognition accuracy for fine dynamic gestures while maintaining excellent recognition calculation efficiency, and can have broad application prospects in the fields of human-computer interaction, virtual reality, sign language translation, etc.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and pattern recognition. It uses an attention-guided spatial graph convolution simple recurrent unit to perform spatiotemporal modeling of dynamic gestures represented by joint points, and combines a regularization method of attention-enhanced spatial graph discarding to alleviate the overfitting problem and realize dynamic gesture classification. Background Art

[0002] As an intuitive and effective interactive tool, gesture recognition plays an important role in many interactive experiences and is a mainstream interactive method. Its application areas include sign language translation, virtual reality, assisted living, etc. Gestures can be divided into static gestures and dynamic gestures. Static gesture recognition is to recognize gestures from a single image, so it runs faster, but its application scenarios have great limitations. Dynamic gesture recognition recognizes gestures from a series of sequential images, so it has higher practicality. With the rapid development of hardware, dynamic recognition methods are gaining more and more attention.

[0003] Existing dynamic gesture recognition methods are mainly divided into two categories according to the type of input: image-based methods and hand skeleton-based methods. Among them, image-based methods take RGB or RGB-D image sequences as input, but these sequences are easily affected by factors such as lighting, background, and occlusion. Hand skeleton-based methods alleviate the difficulties caused by background clutter and have lower computational costs, so they can better realize real-time gesture interaction on mobile devices. Nowadays, with the rapid development of cost-effective depth sensors such as Microsoft Kinect and Intel RealSense, and the emergence of high-precision posture estimation algorithms, high-precision skeleton data can be effectively obtained, so skeleton-based dynamic gesture recognition has become an active research field.

[0004] With the rapid development of hardware devices, deep neural network learning has attracted attention from various research fields. Similarly, in the field of gesture recognition, replacing traditional artificial feature extraction algorithms with end-to-end deep neural networks has become the current mainstream development trend.

[0005] Nunez et al. proposed a method that uses CNN to extract features for each frame and uses LSTM to aggregate the output of CNN, but it is difficult to parse the structural information of the features and the network inference speed is slow. Hou et al. designed an end-to-end spatiotemporal attention residual temporal convolutional network (STA-Res-TCN), which modified the temporal convolutional network for recognizing dynamic gestures based on skeletons. STA-Res-TCN models the connectivity between joints by multiplying the output of the additional branch with the original branch to form a soft attention mechanism. However, this method piles up the three-dimensional coordinates of the X, Y, and Z axes in the same channel, limiting its performance. Nguyen et al. proposed a neural network based on symmetric positive definite (SPD) manifold learning for skeleton-based gesture recognition and aggregated the features in the spatial and temporal domains. It is divided into three stages: convolutional layer to improve the recognition ability of learned features; spatial and temporal Gaussian aggregation of joint features; learning the final SPD matrix from skeleton data. Li et al. proposed a gesture graph convolutional network, adding three types of edges to the graph to finely describe the connection between joints and expanding the dimension of joint coordinates, but lacking the importance of different joints. Miao et al. divided the movement of dynamic gestures into two parts: the global movement of the hand in space and the local movement of the fingers in the hand, and used keyframe extraction to solve the problem of time domain information processing. Although it has high accuracy, it is difficult to avoid the cumbersome cost of manual feature extraction and keyframe extraction, and it is difficult to achieve real-time gesture recognition.

[0006] With the widespread application of graph convolutional networks in human motion recognition, it can be seen that graph convolution has a strong fitting ability for graph structure data such as human skeleton. In many fine dynamic gestures, spatiotemporal graph convolution can still show the spatial and temporal characteristics of gestures, but it inevitably brings too high recognition calculation costs, thus limiting the application of gesture recognition in real-time human-computer interaction scenarios. The present invention combines spatial graph convolution with SRU, and proposes a dynamic gesture recognition method of attention-guided spatial graph convolution SRU, while ensuring the accuracy and speed of gesture recognition, so that the two are better balanced. Summary of the invention

[0007] To solve the above problems, the present invention proposes a method based on attention-guided spatial graph convolutional simple recurrent unit (ASGC-SRU) and applies it to dynamic gesture recognition.

[0008] Firstly, the present invention embeds the spatial graph convolution into the gate structure of the simple recurrent unit (SRU), so that the SRU with high-speed parallel computing capability can model the temporal and spatial information of complex gestures. Secondly, a knuckle attention guidance module is introduced to make more important knuckles have higher attention. Then, a regularization method of attention-enhanced graph discarding (ASD) is introduced to alleviate the drawback of overfitting of gesture data. Finally, the proposed method is used for dynamic gesture recognition.

[0009] In order to achieve the above object, the method of the present invention mainly comprises the following steps:

[0010] Step (1) inputs the input 3D gesture skeleton data and inter-frame motion features into the model after fusion enhancement. The basic unit of the model is the SGC-SRU layer formed by combining the gesture skeleton spatial graph convolution operator with SRU, which better captures the single-frame spatial static features and multi-frame temporal dynamic features of the hand skeleton data. Average pooling is introduced before each layer to speed up the fitting speed.

[0011] Step (2) introduces the knuckle attention guidance mechanism in SGC-SRU to guide attention to more important nodes and improve recognition accuracy.

[0012] Step (3) designs an attention-enhanced spatial graph discarding (ASD) regularization method in the spatial graph convolution operator, which alleviates the disadvantage of overfitting of gesture data and further improves the recognition accuracy of complex gestures.

[0013] Step (4) The hidden layer features of the last layer ASGC-SRU+ASD output and the attention hidden layer features are fused after passing through the fully connected layer, and the softmax function is used for classification.

[0014] Beneficial effects of the present invention:

[0015] 1. The hand skeleton spatial domain graph convolution is combined with a simple recurrent unit, and a knuckle attention guidance network is introduced.

[0016] Some methods model the gesture skeleton data as a series of vectors, which are then processed by convolutional neural networks or recurrent neural networks to directly learn the spatiotemporal features of the hand, but ignore the spatial dependencies between the hand joints. The present invention selects spatial graph convolution and SRU to combine them, which not only makes good use of the spatial features of complex gestures, but also uses SRU with powerful parallel computing capabilities to extract temporal features, thereby efficiently spatiotemporally modeling the gesture skeleton graph structure sequence. The introduction of the finger joint attention guidance module gives more attention to important joints, further improving the accuracy of the model.

[0017] 2. Regularization method of spatial domain graph discarding using attention enhancement

[0018] For the hand skeleton graph convolution, the information of some nodes is relatively redundant. In the fitting process of the multi-layer network, redundant nodes will aggravate the network overfitting to a certain extent. Most methods usually use dropout to randomly discard nodes to alleviate overfitting. Due to the strong linkage of joints in gesture movements, even if a node is discarded, information about the node can still be obtained from its neighboring nodes, which cannot alleviate overfitting well. Therefore, in order to alleviate the overfitting phenomenon while reducing the impact of the randomness of node discarding, the traditional dropout regularization method is replaced by the spatial domain dropout with attention enhancement. For fine gesture data with limited data volume, the overfitting of the network is alleviated to a certain extent, and the network calculation time is hardly increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The figure is a flow chart of an embodiment of the present invention.

[0020] Figure 2 Schematic diagram of the node subset partitioning for graph convolution.

[0021] Figure 3 This is the ASGC-SRU structure diagram.

[0022] Figure 4 It is the knuckle attention guidance module.

[0023] Figure 5 Discard schematic diagram for airspace diagram.

[0024] Figure 6 It is a comparison chart under different node sampling probabilities μ.

[0025] Figure 7 This is the confusion matrix diagram of 14 categories for ASGC-SRU+ASD in SHREC'17.

[0026] Figure 8 28-class confusion matrix diagram for ASGC-SRU+ASD in SHREC'17.

[0027] Fig. 9 This is the 14-category confusion matrix diagram for ASGC-SRU+ASD in DHG14 / 28.

[0028] Fig.10 This is the 28-category confusion matrix diagram for ASGC-SRU+ASD in DHG14 / 28.

[0029] Fig.11 This is a comparison chart of gesture recognition calculation rates. DETAILED DESCRIPTION

[0030] The following describes in detail the dynamic gesture recognition method based on the attention-guided spatial graph convolution simple recurrent unit of the present invention with reference to the accompanying drawings. Figure 1 Implementation flow chart.

[0031] like Figure 1 The implementation of the method of the present invention mainly includes four steps: (1) inputting the gesture data after feature fusion enhancement into the spatial graph convolution SRU network (SGC-SRU); (2) introducing the finger joint attention guidance module to obtain the attention guidance spatial graph convolution SRU network (ASGC-SRU); (3) introducing the attention enhancement spatial graph discarding mechanism (ASD); (4) finally aggregating the features and identifying the dynamic gesture category.

[0032] The following is a detailed description of each step.

[0033] Step 1: Embed the gesture spatial domain graph convolution operator into the simple recurrent unit, construct the spatial domain graph convolution simple recurrent unit, and perform spatiotemporal modeling of the dynamic gesture after motion feature fusion and enhancement.

[0034] Dynamic gesture skeleton sequences are usually represented by the 2D or 3D coordinates of the hand joints in each frame. A complete gesture contains multiple frames of different lengths with different samples. In the spatial domain, for a hand skeleton sequence with N joints and t frames, an undirected spatial graph G is constructed. t =(V t ,E t ), represents the skeleton graph of the hand at the tth frame. Among them, the node set V t = {v t1 ,v t2 ,v t3 …,v tN} includes N joints in the entire hand skeleton, node set E t ={(v ti ,v tj )|v ti ,v tj ∈V t} represents the undirected connection between the i-th and j-th joint points in the hand skeleton, that is, the set of bone edges. ti The adjacent node set of is defined as α(v ti )={v tj |d(v ti ,v tj )≤1},d(v ti ,v tj ) represents the slave node v ti to v tj The minimum path, that is, the sampling distance, is less than or equal to 1 by default in the present invention.

[0035] like Figure 2As shown in the figure, according to the experience of human skeleton graph, the adjacent node set α(v ti ) is divided into three subsets, namely: (1) Vertex v ti itself; (2) centripetal collection, closer to the adjacent knuckles of the palm; (3) eccentric collection, away from the adjacent knuckles of the palm. Figure 2 In the above example, graph convolution operation is used to extract high-level features of gesture graph in the air domain. For the vertex v at time t, ti The graph convolution operation formula on is as follows:

[0036]

[0037] Among them, v tj represents the vertices of the input hand graph, X in , X out Represents adjacent vertices v tj 、v ti The input and output features of w are similar to the weight function in the original convolution operation, which provides a weight vector through the given input. represents a graph labeling function that labels the node v tj The adjacent node set of is divided into Q subsets, so that each subset has a corresponding weight vector, and the number of subsets Q is equal to 3. Finally, the feature parameter is divided by the number of subsets to balance the influence of each subset.

[0038] In formula (1), by introducing the adjacency matrix and transforming it, we can get the following graph convolution operation formula:

[0039]

[0040] Here, Q follows the above partitioning strategy and is set to 3, i.e., three different subsets. The graph labeling function of formula (1) Get the label index q∈{1,2,3} of the subset. q Represents the adjacency matrix corresponding to the subsets of different label indices. When the node v in the corresponding qth subset ti With v tj When there is an undirected connection, A i,j =1; otherwise A i,j = 0, and there is a node self-connection, that is, when i = j, A i,j =1. D q Represents the normalized diagonal matrix for different subsets of label indices.

[0041] Figure 3The ASGC-SRU structure diagram is given. The combination of graph convolution and SRU has spatiotemporal characteristics, and can perform spatiotemporal joint modeling of dynamic gestures based on bones. The network is input in the form of a hand skeleton graph, and the hidden layer state and cell state are continuously updated through the forget gate and reset gate. The expression formula of the ASGC-SRU model is as follows:

[0042]

[0043] f t =sigmoid(w f (γ*)X t +b f ) (4)

[0044] r t =sigmoid(w r (γ*)X t +b r ) (5)

[0045]

[0046]

[0047] Among them, (γ*) represents the spatial graph convolution operator, ⊙ represents the Hadamard product, X t represents the input features at time t, w, w f 、w r 、b f 、b r are the weight parameters and biases that can be learned during training. t 、r t represents the forget gate and reset gate, c t , Represents the cell state and hidden layer state.

[0048] Step 2: Introduce the knuckle attention guidance module into the spatial graph convolution simple recurrent unit.

[0049] The knuckle attention guidance module is as follows Figure 4 As shown in the figure, the hidden state of ASGC-SRU contains rich spatial structural information and temporal dynamic information. Input the attention guidance module, the feature information of each joint in the hidden layer passes through the fully connected layer and the tanh activation function to form the joint aggregation feature, and then passes through a fully connected layer, and the nonlinear activation function sigmoid is used to calculate the attention weight score between 0 and 1. The calculation formula is as follows:

[0050]

[0051] Among them, at Represents the attention weight score of each node, w1 and w2 are the trainable weight parameters in the fully connected layer, and b1 and b2 are the biases.

[0052] The attention guidance module is represented as f a , and introduce the spatial domain graph convolution simple cycle unit, the hidden layer output h of the unit can be obtained t :

[0053]

[0054] Step 3: Design a regularization method for attention-enhanced spatial graph discarding in the spatial graph convolution operator.

[0055] For traditional Dropout, the nodes to be discarded are directly sampled with a certain drop probability μ, but for spatial domain graph dropout, such as Figure 5 As shown in the figure, when a node is discarded, its neighbor node set is also discarded. The discarding of nodes depends on the parameters p and K, where p controls the sampling probability and K controls the size of the neighbor set to be discarded. Given a hand skeleton graph with n joints and v edges, estimate the expected number of nodes in the k-order neighborhood of its randomly sampled node:

[0056]

[0057] in, Indicates the average degree of the node, k∈{1,2,…,K}. In the hand skeleton graph, the average expansion multiple of the sampled nodes that need to be discarded after expanding to the k-order neighborhood:

[0058]

[0059] Then the sampling probability of the root node discarded in the hand skeleton airspace graph is:

[0060]

[0061] Next, the present invention introduces an attention enhancement mechanism to give a higher sampling probability to redundant root nodes, and designs an attention-enhanced spatial graph discarding (ASD). The present invention inputs the gesture features X∈R for spatial graph convolution in each gate structure of ASGC-SRU n×T×C , calculate the average of its absolute value in the channel dimension and time dimension to obtain the attention enhancement feature map β∈R n×T×C . Normalize the attention-enhanced feature map as:

[0062]

[0063] where num β represents the number of elements in β.

[0064] The normalized attention enhancement map is sampled with probability p. In the hand skeleton map with n nodes, the probability p of sampling node i as the root node of the discarded area of ​​ASD is i for:

[0065]

[0066] The ASD steps are as follows:

[0067] 1. Input gesture features X∈R in spatial graph convolution n×T×C , p, K, adjacency matrix A. Calculate the average of the absolute values ​​of the input gesture feature X in the time dimension T and the channel dimension C to obtain the attention enhanced feature map β, and normalize it.

[0068] 2. Multiply the normalized feature map by p to obtain the sampling probability p of the i-th root node in ASD i . Randomly sample the root nodes of n samples, and each sampled root node obeys the probability p i Bernoulli distribution.

[0069] 3. Combined with the adjacency matrix A, calculate the mask matrix M of the sampled root node and the node with the maximum step length K from it. The values ​​of all sampled nodes in the matrix are 0.

[0070] 4. Multiply the gesture feature X by the matrix M, perform mask calculation, discard the sampling nodes, and output the normalized feature X.

[0071] Step 4: The hidden features of the last layer output and the attention hidden features are fused after passing through the fully connected layer, and the softmax function is used to realize gesture classification.

[0072] At the end of the third ASGC-SRU layer, the hidden joint feature information is aggregated with the hidden joint feature information weighted by the attention weight score to obtain the feature F t .

[0073]

[0074] where a ti is the attention weight score of the i-th joint at time t, h ti is the output of the hidden layer of ASGC-SRU.

[0075] The aggregate feature F at time t t Convert to the scores s of J categories t ∈{s t1 ,s t2 ,…,s tJ}, then the expression of the predicted probability function of the i-th category is:

[0076]

[0077] Therefore, the loss function obtained by the model of the present invention is as follows:

[0078]

[0079] Among them, T m represents the time step of the mth ASGC-SRU network layer, Y i represents the true label of the i-th category, Represents the predicted label of the i-th category; λ1 and λ2 are weight attenuation coefficients, which are used to alleviate the gradient vanishing problem in the gradient back propagation process; α tnm Represents the attention weight score of the nth joint at time t in the mth layer of the ASGC-SRU network.

[0080] The present invention conducts ablation experiments on the SHREC'17 dataset to verify the rapidity of SRU and the effectiveness of the spatial domain graph convolution and attention guidance modules. Figure 1 The gesture feature fusion enhancement after the data input remains unchanged, and the three ASGC-SRU layers are replaced by LSTM, SGC-LSTM, SRU, and SGC-SRU respectively, with the same number of layers and parameters. The recognition accuracy of each method for 14 and 28 categories of gestures in the SHREC'17 dataset is evaluated, and the average recognition time of 10 tests is calculated for the 840 dynamic skeleton gesture data in the test set, as shown in Table 1.

[0081] Table 1 Effectiveness of attention-guided spatial graph convolution SRU

[0082]

[0083] As can be seen from Table 1, compared with LSTM, SRU has similar recognition accuracy in dynamic gesture recognition based on hand bones. After introducing spatial graph convolution in 28 types of fine gestures, the accuracy has been greatly improved by 16.67% and 17.66%. Although the performance of SGC-SRU is slightly lower than that of SGC-LSTM by 0.72% in fine gestures, in terms of test time, although gesture feature fusion enhancement and spatial graph convolution will significantly increase the test time, the test efficiency of SGC-SRU is improved to 1.83 times that of SGC-LSTM. Therefore, in dynamic gesture recognition based on bones, SRU has greatly improved the computational efficiency compared with LSTM without losing accuracy. On this basis, the finger joint attention guidance module is further introduced to make important finger joints have higher attention. The accuracy of ASGC-SRU in 14 and 28 types of gestures is improved to 95.48% and 90.12%.

[0084] In order to verify the effectiveness of attention-enhanced spatial domain graph dropout, Dropout and ASD are introduced to ASGC-SRU respectively. Since the node sampling probability μ before node expansion is a common parameter of the two, 14 categories and 28 categories are experimentally compared under different μ values ​​to select the μ value with the best effect. For different μ values, the present invention conducts three experiments on the two methods respectively and selects the best result. Figure 6 The comparison between Dropout and ASD under different sampling probabilities in 28 types of gestures is given. When μ is between 0.05 and 0.2, it has an effective regularization effect, and when it is 0.15, it has the best effect. Table 2 shows the results of 14 and 28 categories, and it can be seen that ASD has a more significant effect on the more complex 28 types of gestures.

[0085] Table 2 Effectiveness of attention-enhanced spatial map discarding mechanism %

[0086]

[0087] Table 3 shows the comparison results of the proposed ASGC-SRU+ASD with the mainstream methods on the dynamic gesture dataset SHREC'17. The recognition accuracy of 14 types of gestures is 95.8%, and the recognition accuracy of 28 more complex gestures is 91.1%.

[0088] Table 3 Comparison of methods on the SHREC'17 dataset%

[0089]

[0090]

[0091] Figure 7 and Figure 8 The confusion matrix diagrams of the prediction results of 14 types of gestures and 28 types of gestures are shown. It can be observed that the present invention achieves a recognition accuracy rate higher than 90% in 13 gestures out of the 14 types of gestures, and a recognition accuracy rate higher than 85% in 20 gestures out of the 28 types of gestures.

[0092] In order to further verify the effectiveness of the proposed method, the present invention also compares the results with the mainstream method on the DHG14 / 28 dataset. The recognition accuracy of the present invention for 14 types of gestures is 90.1%, and the recognition accuracy for 28 more complex types of gestures is 86.9%, which shows that the proposed method has good generalization ability.

[0093] Table 4 Comparison of methods on DHG14 / 28 dataset%

[0094]

[0095] Fig. 9 and Fig.10 The confusion matrix diagrams of the prediction results of 14 types of gestures and 28 types of gestures in DHG14 / 28 are shown. On this dataset, the proposed method achieves an accuracy rate higher than 90.0% in 10 of the 14 types of gestures and an accuracy rate higher than 85% in 21 of the 28 types of gestures.

[0096] The recognition calculation rate of the model built by the present invention and the mainstream method in dynamic gesture recognition, in order to ensure the uniformity of the computing hardware equipment, the GPU model used for recognition calculation in the comparison method is the same as the NVIDIA GeForce GTX 1080 of the present invention. Since there are certain differences in the number of sampling frames of gesture data to be predicted input between the methods, the number of hand bones recognized and calculated per second is used as the rate evaluation index, that is, the number of hand bones recognized and calculated per second (skeleton / s) is equal to the number of gestures recognized per second (gesture / s) multiplied by the number of sampling frames (T).

[0097] Fig.11 The calculation rate comparison results of five gesture recognition methods are given. It can be seen that the recognition calculation rate of the method of the present invention far exceeds that of CNN-LSTM and STA-Res-TCN, reaching 54857 skeleton / s, which is 2.55 times and 1.73 times of the recognition calculation rate of AGC-LSTM and HG-GCN respectively, and far exceeds the standard of real-time video analysis (30 hand skeletons per frame). While ensuring the advantage of recognition accuracy, the method of the present invention can significantly reduce the recognition calculation time and has good practicality.

Claims

1. A gesture recognition method based on attention-guided spatial graph convolution simple recurrent unit, characterized by The method includes the following main steps: Step (1) embeds the gesture spatial domain graph convolution operator into a simple recurrent unit, constructs a spatial domain graph convolution simple recurrent unit, and performs spatiotemporal modeling of the dynamic gesture after motion feature fusion enhancement: In the spatial domain, for a hand skeleton sequence with N joints and t frames, an undirected spatial graph G is constructed t =(V t ,E t ), represents the skeleton graph of the hand at the tth frame; where the node set V t = {v t1 ,v t2 ,v t3 …,v tN } includes N joints in the entire hand skeleton, node set E t ={(v ti ,v tj )|v ti ,v tj ∈V t } represents the undirected connection between the i-th and j-th joint points in the hand skeleton, that is, the set of bone edges; At the same time, the node v ti The adjacent node set of is defined as α(v ti )={v tj |d(v ti ,v tj )≤1},d(v ti ,v tj ) represents the slave node v ti to v tj The minimum path of According to the human skeleton graph, the adjacent node set α(v ti ) is divided into three subsets, namely: vertex v ti itself, centripetal set and centrifugal set; The graph convolution operation is used to extract the high-level features of the gesture graph in the air domain. For the vertex v at time t ti The graph convolution operation formula on is as follows: Among them, v tj represents the vertices of the input hand graph, X in , X out Represents adjacent vertices v tj 、v ti The input and output features of ; w is similar to the weight function in the original convolution operation, providing a weight vector through the given input; represents a graph labeling function that labels the node v tj The adjacent node set of is divided into Q subsets, so that each subset has a corresponding weight vector, and the number of subsets Q is equal to 3; In formula (1), by introducing the adjacency matrix and transforming it, we can get the following graph convolution operation formula: The graph labeling function of formula (1) Get the label index q∈{1,2,3} of the subset; A q Represents the adjacency matrix corresponding to the subsets of different label indices, D q Represents the normalized diagonal matrix of different label index subsets; The multi-layer spatial graph convolution simple recurrent unit layer is input in the form of a hand skeleton graph, and the expression formula is as follows: f t =sigmoid(w f (γ*)X t +b f ) (4) r t =sigmoid(w r (γ*)X t +b r ) (5) Among them, (γ*) represents the spatial graph convolution, ⊙ represents the Hadamard product, X t represents the input features at time t, w, w f 、w r 、b f 、b r are the weight parameters and biases that can be learned during training; f t 、r t represents the forget gate and reset gate, c t , Represents the cell state and hidden layer state; Step (2), introduce the knuckle attention guidance module into the spatial domain graph convolution simple cycle unit: Attention guides the spatial domain graph convolution simple recurrent network to transform the hidden layer state Input the attention guidance module, the feature information of each joint in the hidden layer passes through the fully connected layer and the tanh activation function to form the joint aggregation feature, and then passes through a fully connected layer, and the nonlinear activation function sigmoid is used to calculate the attention weight score between 0 and 1. The calculation formula is as follows: Among them, a t Represents the attention weight score of each node, w1 and w2 are the trainable weight parameters in the fully connected layer, and b1 and b2 are the bias values; The attention-guided network module is represented as f a , and introduce the spatial domain graph convolution simple cycle unit to obtain the hidden layer output h of the unit t : Step (3), design an attention-enhanced regularization method for spatial graph discarding in the spatial graph convolution operator: Given a hand skeleton graph with n joints and v edges, estimate the expected number of nodes in the k-order neighborhood of a randomly sampled node: in, represents the average degree of the node, k∈{1,2,…,K}, K represents the size of the neighbor set to be discarded; in the hand skeleton graph, the average expansion multiple of the sampled nodes to be discarded after expanding to the k-order neighborhood: Then the sampling probability of the root node discarded in the hand skeleton airspace graph is: The attention enhancement mechanism is introduced to give a higher sampling probability to redundant root nodes, and an attention-enhanced spatial graph discarding is designed; the gesture features X∈R used for spatial graph convolution in each gate structure of the input simple recurrent unit are n×T×C , calculate the average of its absolute value in the channel dimension and time dimension to obtain the attention enhancement feature map β∈R n×T×C ; Normalize the attention-enhanced feature map to: where num β represents the number of elements in β; The normalized attention enhancement map is sampled with probability p. In the hand skeleton map with n nodes, the probability p of sampling node i as the root node of the discarded area of ​​ASD is i for: Step (4), the hidden layer features of the last layer output and the attention hidden layer features are fused after passing through the fully connected layer, and the softmax function is used to implement gesture classification: At the end of the third network unit layer, the hidden joint feature information is aggregated with the hidden joint feature information weighted by the attention weight score to obtain the feature F t ; where a ti is the attention weight score of the i-th joint at time t, h ti is the output of the hidden layer of ASGC-SRU; The aggregate feature F at time t t Convert to the scores s of J categories t ∈{s t1 ,s t2 ,…,s tJ }, then the expression of the predicted probability function of the i-th category is: The loss function of the simple recurrent network of attention-guided spatial graph convolution is as follows: Among them, T m represents the time step of the mth network unit layer, Y i represents the true label of the i-th category, represents the predicted label of the i-th category; λ1 and λ2 are weight attenuation coefficients, which are used to alleviate the gradient vanishing problem in the gradient back propagation process; α tnm Represents the attention weight score of the nth joint at time t in the mth layer of the ASGC-SRU network.

2. The gesture recognition method based on attention-guided spatial graph convolution simple recurrent unit according to claim 1 is characterized in that: The attention-enhanced airspace map discarding process is as follows: 3-1 Input spatial graph convolution gesture feature X∈R n×T×C , p, K, adjacency matrix A; calculate the average of the absolute values ​​of the input gesture feature X in the time dimension T and the channel dimension C to obtain the attention enhancement feature map β, and normalize it; 3-2 Multiply the normalized feature map by p to obtain the sampling probability p of the i-th root node in ASD i ; Randomly sample the root nodes of n samples, and each sampled root node obeys the probability p i Bernoulli distribution of ; 3-3 Combined with the adjacency matrix A, calculate the mask matrix M of the sampled root node and the node with the maximum step length K from it, and all the sampled node values ​​in the matrix are 0; 3-4 Multiply the gesture feature X by the matrix M, perform mask calculation, and discard the sampling nodes. Output after standardizing feature X.

Images