Dual-stream Cyclic Attention Dynamic Gesture Recognition Method Based on Motion Guidance

Abstract

The present invention proposes a two-stream cyclic attention dynamic gesture recognition method based on motion guidance. The method includes: using a video codec tool to extract motion vectors, and combining a prediction frame residual analysis mechanism to quantify the dynamic intensity of the hand and generate a motion mask; using a gated network and a graph neural network for the RGB features and the hand motion contour mask to perform gated fusion and graph convolution calculations in sequence; using a dynamic spatio-temporal decay module and a multi-scale wavelet spectrum-spatial block to perform dilated convolution processing, as well as multi-scale frequency domain reconstruction and channel interaction respectively to obtain a gesture classification result. The present invention proposes a cyclic attention mechanism based on a fully convolutional architecture, combined with a learnable time decay strategy, to achieve efficient and low-complexity temporal modeling. At the same time, a wavelet decomposition and reconstruction and multi-scale pooling method are adopted to capture fine-grained hand motion changes, enabling the network to effectively model short-term temporal dependence features and further improving the recognition accuracy of dynamic gestures.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and human-computer interaction, and particularly to a dual-stream recurrent attention dynamic gesture recognition method based on motion guidance. Background Art

[0002] Dynamic Hand Gesture Recognition (DHGR), as an important research direction in the fields of computer vision and human-computer interaction, has received extensive attention in recent years. The core goal of DHGR is to accurately identify different types of gesture actions by analyzing the hand dynamic trajectories in video data. This technology has great potential in many practical application scenarios such as smart home control, augmented reality (AR), virtual reality (VR) interaction, and medical rehabilitation. For example, in a human-computer interaction system, the interaction method based on dynamic hand gesture recognition can greatly improve the user experience, enabling users to control devices through natural hand movements without relying on physical contact. However, limited by the locality of gestures in the video, complex background interference, and the computational cost of temporal modeling, existing methods still face many challenges.

[0003] On the one hand, since the gesture area accounts for a small proportion in the video, existing CNN-based feature extraction methods often extract a large amount of irrelevant background information, resulting in redundant feature representations and affecting the robustness of the model. In addition, the rapid changes of dynamic gestures are prone to cause motion blur, and a single RGB image only contains color and texture information, lacking the ability to directly model the gesture motion pattern and difficult to capture the complete gesture motion trajectory, thus reducing the recognition accuracy. On the other hand, existing methods mainly rely on Transformer for long-term temporal modeling, but the global self-attention mechanism in Transformer brings a problem of high computational complexity and is difficult to meet the requirements of real-time applications in long-term temporal video tasks. In addition, multi-modal fusion methods (such as RGB + optical flow) can enhance the temporal modeling ability to a certain extent, but the optical flow calculation cost is high and sensitive to light changes, restricting its popularization in practical applications.

[0004] For most of the current mainstream dynamic gesture recognition methods, CNN is mostly used as the backbone network for feature extraction, and combined with RNN, LSTM or Transformer for temporal modeling. The features extracted by the CNN backbone network contain a lot of background noise, resulting in the gesture information being easily submerged. Recurrent neural networks such as RNN / LSTM transmit time information through hidden states and lack the ability to dynamically fuse spatial features. In dynamic gesture recognition, the spatial changes of hand key points (such as joint positions) need to be modeled synchronously with the temporal evolution, and it is difficult for the hidden states of recurrent neural networks to encode spatio-temporal features simultaneously. The computational cost of the traditional Transformer structure is relatively high, which affects the deployment ability of the model on embedded devices. In addition, the existing methods lack effective global modeling means, resulting in difficulty in maintaining the spatio-temporal consistency of gestures during cross-frame modeling, thus affecting the recognition performance. Summary of the Invention

[0005] In view of the above situation, the main object of the present invention is to propose a dual-stream recurrent attention dynamic gesture recognition method based on motion guidance to solve the above technical problems.

[0006] The present invention proposes a dual-stream recurrent attention dynamic gesture recognition method based on motion guidance, and the method includes the following steps:

[0007] Step 1: Construct a motion perception data augmentation module based on the predicted frame residual analysis mechanism, construct a motion-guided gated graph neural network fusion module based on the gated network and the graph neural network, and construct an efficient multi-scale recurrent attention network based on the recurrent attention mechanism. The motion perception data augmentation module, the motion-guided gated graph neural network fusion module, and the efficient multi-scale recurrent attention network constitute a classification model; wherein, the motion perception data augmentation module includes a bilateral filter, the motion-guided gated graph neural network fusion module includes a multi-layer perceptron, and the efficient multi-scale recurrent attention network includes a dynamic spatio-temporal attenuation module and a multi-scale wavelet spectrum-spatial block;

[0008] Step 2: Input the video frame sequence into the motion perception data augmentation module, use the video encoding and decoding tool to extract the motion vector, and combine the predicted frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask to obtain the hand motion contour mask and the RGB image;

[0009] Step 3: Extract the RGB features based on the RGB image, input the hand motion contour mask and the RGB features into the motion-guided gated graph neural network fusion module, and use the gated network and the graph neural network for the RGB features and the hand motion contour mask to perform gated fusion and graph convolution calculations in sequence to obtain the features output by the graph neural network;

[0010] Step 4: Input the features output by the graph neural network into the efficient multi-scale recurrent attention network, perform dilated convolution processing using the dynamic spatio-temporal decay module to obtain the features output by the dynamic spatio-temporal decay module, and input them into the multi-scale wavelet spectrum-spatial block for multi-scale frequency domain reconstruction and channel interaction to obtain the gesture classification result;

[0011] Construct a classification loss based on the gesture classification result, use the classification loss to optimize the classification model to obtain the optimized classification model, and input the video frame sequence into the optimized classification model to obtain the final gesture classification result.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0013] 1. The present invention uses the motion vector (MV) in the video compression domain to construct a prediction frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask. The hand contour is reconstructed through dynamic threshold filtering and morphological closing operation, enabling the model to more accurately focus on the hand region while effectively suppressing background noise, thereby providing accurate spatio-temporal information for the subsequent two-stream network and improving the robustness of gesture recognition;

[0014] 2. The present invention designs a two-stream fusion framework combining gated MLP and graph neural network (GCN), dynamically adjusts the contribution degrees of the RGB and motion mask (Mask) modalities through spatio-temporal perception weights, and models the cross-frame dependence relationship through fully connected temporal graph. This method can effectively enhance the adaptability of the gesture recognition system to light changes, background interference, and complex hand deformations, and improve the temporal modeling ability across time steps;

[0015] 3. The present invention proposes a recurrent attention mechanism based on a fully convolutional architecture, combined with a learnable time decay strategy, to achieve efficient and low-complexity temporal modeling. At the same time, wavelet decomposition reconstruction and multi-scale pooling methods are used to capture fine-grained hand motion changes, enabling the network to effectively model short-term temporal dependence features and further improving the recognition accuracy of dynamic gestures.

[0016] The additional aspects and advantages of the present invention will be partially given in the following description, partially become apparent from the following description, or be understood through the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of the steps of the dynamic gesture recognition method based on motion-guided two-stream recurrent attention proposed by the present invention.

[0018] Figure 2 It is an overall architecture diagram of the dynamic gesture recognition method based on motion-guided two-stream recurrent attention proposed by the present invention.

[0019] Figure 3Flowchart of the motion perception data enhancement strategy for the dual-stream recurrent attention dynamic gesture recognition method based on motion guidance proposed by the present invention.

[0020] Figure 4 Framework diagram of the motion-guided gated neural network fusion module for the dual-stream recurrent attention dynamic gesture recognition method based on motion guidance proposed by the present invention.

[0021] Figure 5 Framework diagram of the efficient multi-scale recurrent attention network for the dual-stream recurrent attention dynamic gesture recognition method based on motion guidance proposed by the present invention.

[0022] Figure 6 Framework diagram of the dynamic spatio-temporal decay module for the dual-stream recurrent attention dynamic gesture recognition method based on motion guidance proposed by the present invention.

[0023] Figure 7 Framework diagram of the multi-scale wavelet spectrum-spatial block for the dual-stream recurrent attention dynamic gesture recognition method based on motion guidance proposed by the present invention.

[0024] Figure 8 Framework diagram of the wavelet feature deconstruction for the dual-stream recurrent attention dynamic gesture recognition method based on motion guidance proposed by the present invention. Detailed implementation manners

[0025] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0026] These and other aspects of the embodiments of the present invention will be clear with reference to the following description and drawings. In these descriptions and drawings, some specific embodiments of the embodiments of the present invention are specifically disclosed as some ways to implement the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0027] Please refer to Figure 1 , the embodiments of the present invention propose a dual-stream recurrent attention dynamic gesture recognition method based on motion guidance, and the method includes the following steps:

[0028] Step 1: Construct a motion perception data enhancement module based on the predicted frame residual analysis mechanism, construct a motion-guided gated graph neural network fusion module based on the gated network and the graph neural network, and construct an efficient multi-scale recurrent attention network based on the recurrent attention mechanism. The motion perception data enhancement module, the motion-guided gated graph neural network fusion module, and the efficient multi-scale recurrent attention network constitute a classification model. Among them, the motion perception data enhancement module includes a bilateral filter, the motion-guided gated graph neural network fusion module includes a multi-layer perceptron, and the efficient multi-scale recurrent attention network includes a dynamic spatio-temporal attenuation module and a multi-scale wavelet spectrum-spatial block.

[0029] Step 2: Input the video frame sequence into the motion perception data enhancement module, use the video codec tool to extract the motion vectors, and combine the predicted frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask to obtain the hand motion contour mask and the RGB image.

[0030] Please refer to Figure 2 and Figure 3 In Step 2, input the video frame sequence into the motion perception data enhancement module, use the video codec tool to extract the motion vectors, and combine the predicted frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask, which specifically includes the following steps:

[0031] Input the video frame sequence into the motion perception data enhancement module, use the FFmpeg video codec tool to extract the vectors to obtain the extracted motion vectors;

[0032] Perform a dynamic vector filtering operation on the extracted motion vectors to obtain the filtered motion vectors;

[0033] Perform motion compensation based on the filtered motion vectors and the video frame sequence to construct a predicted frame to obtain the constructed predicted frame;

[0034] Calculate the residual of the constructed predicted frame to obtain the residual image;

[0035] Perform grayscale operation, bilateral filtering operation of the bilateral filter, and threshold binary operation on the residual image in sequence to obtain the hand motion contour mask.

[0036] It should be noted that the image encoder and the mask encoder in the appendix Figure 2 are two backbone networks (FastNet and MobileNetV3 respectively, where the parameters of the image encoder are learnable (participate in gradient update), and the parameters of the mask encoder are frozen (do not participate in gradient update)).

[0037] Perform a dynamic vector filtering operation on the extracted motion vectors to obtain the filtered motion vectors. The corresponding relationship in the process is as follows:

[0038] ;

[0039] wherein, represents the Euclidean amplitude, represents the motion vector, represents the filtered motion vector, represents the motion amplitude threshold, represents the macroblock index of the video frame;

[0040] It should be noted that controls the filtering strength.

[0041] In the step of performing motion compensation based on the filtered motion vector and the video frame sequence to construct a predicted frame and obtaining the constructed predicted frame, the relational expressions in the corresponding process are as follows:

[0042] ;

[0043] wherein, represents the predicted frame, represents the th frame image, represents the displacement compensation value of the pixel by the filtered motion vector for the pixel, represents the pixel index;

[0044] In the step of calculating the residual of the constructed predicted frame to obtain the residual image, the relational expressions in the corresponding process are as follows:

[0045] ;

[0046] where, represents the residual image, represents the th frame RGB image, represents the dimension of the image matrix of, represents the height of the image, represents the width of the image, represents the number of channels of the image;

[0047] It should be noted that is to highlight the pixel difference in the motion area.

[0048] In the step of sequentially performing grayscale operation, bilateral filtering operation of the bilateral filter, and threshold binary operation on the residual image to obtain the hand motion contour mask, the relational expressions in the corresponding process are as follows:

[0049] ;

[0050] Among them, represents the grayscale image, represents the standard RGB to grayscale mapping, represents the image matrix The dimension of , represents the bilaterally filtered image, represents the bilateral wave operation, represents the standard deviation in the spatial domain, represents the standard deviation in the intensity domain, represents the hand motion contour mask, represents the binarization threshold.

[0051] Furthermore, in order to improve the robustness of dynamic gesture recognition, reduce the computational complexity, and precisely focus on the hand motion area, the present invention uses the motion vectors in the video compression domain to extract hand motion features, and constructs a prediction frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask. At the same time, dynamic threshold filtering and morphological closing operations are introduced to remove background noise and ensure that the model focuses on the effective features of the hand.

[0052] Step 3: Extract RGB features from the RGB image, input the hand motion contour mask and RGB features into the motion-guided gated graph neural network fusion module, and use the gated network and graph neural network to perform gated fusion and graph convolution calculations on the RGB features and the hand motion contour mask in sequence to obtain the features output by the graph neural network.

[0053] Please refer to Figure 4 , in Step 3, input the hand motion contour mask and RGB features into the motion-guided gated graph neural network fusion module, and use the gated network and graph neural network to perform gated fusion and graph convolution calculations on the RGB features and the hand motion contour mask in sequence to obtain the features output by the graph neural network, which specifically includes the following steps:

[0054] Perform splicing operations and feature fusion processing on the RGB features and the hand motion contour mask using the gated network in sequence to obtain the fused features;

[0055] Construct a temporal graph based on the graph neural network, and perform graph convolution calculations on the fused features using the graph convolution network in combination with the temporal graph to obtain the features output by the graph neural network.

[0056] Perform splicing operations and feature fusion processing on the RGB features and the hand motion contour mask using the gated network in sequence to obtain the fused features, and the relational expressions existing in the corresponding process are as follows:

[0057] ;

[0058] Among them, represents the first gating weight, represents function, represents the weight of the first linear layer, represents the weight of the second linear layer, represents the RGB feature, represents the bias term of the first linear layer, represents the bias term of the second linear layer, represents the feature after fusion, represents matrix multiplication;

[0059] It should be noted that in the appendix Figure 4 in The function is used to obtain the gating coefficient by normalizing the output of the multi-layer perceptron.

[0060] In the step of constructing a temporal graph based on a graph neural network and performing graph convolution calculation on the fused feature by combining the graph convolution network with the temporal graph to obtain the feature output by the graph neural network, the following relational expressions exist in the corresponding process:

[0061] ;

[0062] Among them, represents the constructed temporal graph, represents the node of the fused feature corresponding to each time step, represents the edge in the graph neural network, represents the graph convolution calculation of the layer linear layer, represents the node of the fused feature corresponding to each time step neighbor set of represents the graph convolution calculation of the layer linear layer, represents the mean normalization function, represents the second graph convolution calculation, represents the first graph convolution calculation, represents the index of the linear layer.

[0063] Step 4: Input the feature output by the graph neural network into the efficient multi-scale recurrent attention network, perform dilated convolution processing using the dynamic spatio-temporal decay module to obtain the feature output by the dynamic spatio-temporal decay module, and input it into the multi-scale wavelet spectrum-spatial block for multi-scale frequency domain reconstruction and channel interaction to obtain the gesture classification result;

[0064] Construct a classification loss based on the gesture classification results, optimize the classification model using the classification loss to obtain an optimized classification model, and input the video frame sequence into the optimized classification model to obtain the final gesture classification result.

[0065] Please refer to Figures 5 to 8 , in step 4, input the features output by the graph neural network into the efficient multi-scale recurrent attention network, perform dilated convolution processing using the dynamic spatio-temporal decay module to obtain the features output by the dynamic spatio-temporal decay module, and input them into the multi-scale wavelet spectrum-spatial block for multi-scale frequency domain reconstruction and channel interaction to obtain the gesture classification result, which specifically includes the following steps:

[0066] Input the features output by the graph neural network into the efficient multi-scale recurrent attention network, and perform a time shift operation using the dynamic spatio-temporal decay module to obtain the features after time shift;

[0067] Perform dilated convolution processing on the features after time shift along the time dimension to obtain the features after dilated convolution;

[0068] For the features after dilated convolution, combine the learnable time decay mechanism and the past information reception gate, and perform linear attention mechanism processing and gated output projection processing in sequence to obtain the output features of the dynamic spatio-temporal decay module;

[0069] Perform a time shift operation and a wavelet transform operation on the output features of the dynamic spatio-temporal decay module in sequence to obtain the low-frequency information and high-frequency information of the features;

[0070] Input the low-frequency information and high-frequency information of the features into the dilated convolution group respectively for feature enhancement to obtain the enhanced low-frequency component and enhanced high-frequency component;

[0071] Perform inverse wavelet transform operation and multi-scale pooling operation on the enhanced low-frequency component and enhanced high-frequency component in sequence to obtain the tensor after multi-scale pooling;

[0072] Perform element-wise multiplication calculation, weight convolution mapping operation and non-linear activation processing on the tensor after multi-scale pooling in sequence to obtain the intermediate tensor;

[0073] Obtain the dynamic gating coefficient, perform element-wise multiplication calculation on the intermediate tensor and the dynamic gating coefficient to obtain the output weighted features;

[0074] Obtain the gesture classification result by passing the output weighted features through a fully connected layer.

[0075] Input the features output by the graph neural network into the efficient multi-scale recurrent attention network, and perform a time shift operation using the dynamic spatio-temporal decay module to obtain the features after time shift. The relational expressions existing in the corresponding process are as follows:

[0076] ;

[0077] Among them, represents the first feature after time offset, represents the time offset operation, represents the feature output from the th layer of the graph neural network as the input ;

[0078] In the step of performing dilated convolution processing on the feature after time offset along the time dimension to obtain the feature after dilated convolution, the relationship in the corresponding process is as follows:

[0079] ;

[0080] Among them, represents the feature after multiple dilated convolutions, represents the dilated convolution operation, represents different dilation factors, represents the dilation factor, represents the th convolution operation with different dilation factors;

[0081] In the step of performing linear attention mechanism processing and gated output projection processing on the feature after dilated convolution by combining the learnable time decay mechanism and the past information reception gate to obtain the output feature of the dynamic spatio-temporal decay module, the relationship in the corresponding process is as follows:

[0082] ;

[0083] Among them, represents the learnable time weight of each attention head, represents the normalization function, represents the learnable parameter, represents the reception gate, represents the one-dimensional convolution operation, represents that the dimension of the feature is , represents the dimension of each batch, represents the number of video frames in the input, represents the key vector, represents the value vector, represents that the dimension of the feature is , represents the dimension of each head, represents the intermediate tensor, represents the current output decay factor, represents the cumulative-product function, represents the learnable event decay parameter, represents the feature dimension, indicates that the dimension of the feature is , represents the tensor after time decay, represents the cumulative-sum function, represents the output feature of the dynamic spatio-temporal decay module, represents a one-dimensional convolution operation.

[0084] It should be noted that to reduce the computational complexity, the traditional dot-product attention is replaced by a vectorized cumulative operation. This mechanism first performs an element-wise multiplication between the key and value matrices to generate an intermediate tensor with fused features. Then, a cumulative-product function is used along the time dimension to generate a sequence of decay factors, simulating the gradual decay of historical information over time; finally, a cumulative sum is used to linearly accumulate the weighted features, effectively capturing the long-range dependencies in the gesture sequence while reducing the complexity in the traditional attention mechanism from to .

[0085] The output features of the dynamic spatio-temporal decay module are sequentially subjected to a time shift operation and a wavelet transform operation to obtain the low-frequency information and high-frequency information of the features. The corresponding relationships in the process are as follows:

[0086] ;

[0087] where, represents the second time shift operation, represents the low-frequency information of the feature, represents the high-frequency information of the feature, represents the wavelet transform operation;

[0088] It should be noted that this process is the process of wavelet feature deconstruction.

[0089] In the step of separately inputting the low-frequency information and high-frequency information of the feature into the dilated convolution group for feature enhancement to obtain the enhanced low-frequency component and enhanced high-frequency component, the corresponding relationships in the process are as follows:

[0090] ;

[0091] where, represents the enhanced low-frequency component, represents the enhanced high-frequency component, represents a two-dimensional convolution operation, represents the size of the convolution kernel, represents the dilation rate of the convolution kernel, Indicates the high-frequency component after the th convolution operation;

[0092] In the step of performing inverse wavelet transform operation and multi-scale pooling operation on the enhanced low-frequency component and the enhanced high-frequency component in sequence to obtain the tensor after multi-scale pooling, the relational expressions existing in the corresponding process are as follows:

[0093] ;

[0094] Among them, Indicates the output tensor of the inverse wavelet transform, Indicates the inverse wavelet transform operation, Indicates the 1st high-frequency component, Indicates the 2nd high-frequency component, Indicates the 3rd high-frequency component, Indicates the tensor after multi-scale pooling, Indicates the multi-scale pooling operation, Indicates the set of pooling sizes, Indicates the pooling kernel sizes of different scales;

[0095] In the step of performing element-wise multiplication calculation, weight convolution mapping operation and non-linear activation processing on the tensor after multi-scale pooling in sequence to obtain the intermediate tensor, the relational expressions existing in the corresponding process are as follows:

[0096] ;

[0097] Among them, Indicates the key tensor in the process of inter-channel information interaction, Indicates that the dimension of the feature is , Indicates the number of hidden layers, Indicates the value tensor in the process of inter-channel information interaction, Indicates the receiving gate in the process of inter-channel information interaction, Indicates that the dimension of the feature is , Indicates the number of features, Indicates the intermediate tensor generated in the process of inter-channel information interaction, Indicates Activation function;

[0098] In the step of obtaining the dynamic gating coefficient and performing element-wise multiplication calculation on the intermediate tensor and the dynamic gating coefficient to obtain the output weighted feature, the relational expressions existing in the corresponding process are as follows:

[0099] ;

[0100] Among them, Represents the second dynamic gating coefficient, represents the acceptance gate, represents the output weighted feature;

[0101] In the step of obtaining the gesture classification result by passing the output weighted feature through the fully connected layer, the relational expressions existing in the corresponding process are as follows:

[0102] ;

[0103] Among them, represents the gesture classification result, represents the fully connected layer.

[0104] Based on the gesture classification result, a classification loss is constructed, and the relational expressions existing in the corresponding process are as follows:

[0105] ;

[0106] Among them, represents the classification loss, represents the number of batches of training samples, represents the th sample belonging to the true value of the th class, represents the th sample belonging to the predicted probability of the th class, represents the logarithmic function.

[0107] Furthermore, the efficient multi-scale recurrent attention network, as a temporal modeling network, receives the feature input after the fusion of the backbone network and the motion-guided gating graph neural network fusion module, and adopts a recurrent structure similar to RNN but based on full convolution calculation. The efficient multi-scale recurrent attention network is composed of alternating stacks of a dynamic spatio-temporal decay module (DSTD) and a multi-scale wavelet spectrum-spatial block (MSWSS). The dynamic spatio-temporal decay module efficiently models global and local temporal dependencies through multi-scale dilated convolution and convolutional attention; the multi-scale wavelet spectrum-spatial block combines wavelet transform and multi-scale pooling to enhance channel interaction and capture fine-grained gesture features. The overall architecture combines residual connections and normalization to achieve efficient parallel training and improve the dynamic robustness of the model.

[0108] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0109] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0110] The above-described embodiments merely represent several implementation manners of the present invention, and the description thereof is relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be subject to the appended claims.

Claims

1. A dual-stream cyclic attention dynamic gesture recognition method based on motion guidance, characterized in that The method includes the following steps: Step 1: Construct a motion perception data enhancement module based on a prediction frame residual analysis mechanism, construct a motion-guided gated graph neural network fusion module based on a gated network and a graph neural network, and construct an efficient multi-scale recurrent attention network based on a recurrent attention mechanism. The motion perception data enhancement module, the motion-guided gated graph neural network fusion module, and the efficient multi-scale recurrent attention network constitute a classification model. Among them, the motion perception data enhancement module includes a bilateral filter, the motion-guided gated graph neural network fusion module includes a multi-layer perceptron, and the efficient multi-scale recurrent attention network includes a dynamic spatio-temporal attenuation module and a multi-scale wavelet spectrum-spatial block; Step 2: Input the video frame sequence into the motion perception data enhancement module, use a video codec tool to extract motion vectors, and combine the prediction frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask to obtain a hand motion contour mask and an RGB image; Step 3: Extract RGB features based on the RGB image, input the hand motion contour mask and the RGB features into the motion-guided gated graph neural network fusion module, and use the gated network and the graph neural network for the RGB features and the hand motion contour mask to perform gated fusion and graph convolution calculations in sequence to obtain the features output by the graph neural network; Step 4: Input the features output by the graph neural network into the efficient multi-scale recurrent attention network, perform dilated convolution processing using the dynamic spatio-temporal attenuation module to obtain the features output by the dynamic spatio-temporal attenuation module, and input them into the multi-scale wavelet spectrum-spatial block for multi-scale frequency domain reconstruction and channel interaction to obtain the gesture classification result; Construct a classification loss based on the gesture classification result, use the classification loss to optimize the classification model to obtain an optimized classification model, and input the video frame sequence into the optimized classification model to obtain the final gesture classification result.

2. The method for dynamic gesture recognition based on motion-guided dual-stream cyclic attention according to claim 1, wherein In the said Step 2, inputting the video frame sequence into the motion perception data enhancement module, using a video codec tool to extract motion vectors, and combining the prediction frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask specifically includes the following steps: Input the video frame sequence into the motion perception data enhancement module, use the FFmpeg video codec tool to extract vectors to obtain the extracted motion vectors; Perform a dynamic vector filtering operation on the extracted motion vectors to obtain the filtered motion vectors; Perform motion compensation based on the filtered motion vectors and the video frame sequence to construct a prediction frame to obtain the constructed prediction frame; Perform residual calculation on the constructed prediction frame to obtain a residual image; Perform a grayscale operation, a bilateral filtering operation of the bilateral filter, and a threshold binary operation on the residual image in sequence to obtain a hand motion contour mask.

3. The method for dynamically recognizing gesture based on motion-guided dual-stream cyclic attention according to claim 2, wherein Perform a dynamic vector filtering operation on the extracted motion vectors to obtain the filtered motion vectors. The corresponding relationship existing in the process is as follows: ； Among them, represents the Euclidean amplitude, represents the motion vector, represents the filtered motion vector, represents the motion amplitude threshold, represents the macroblock index of the video frame; In the step of performing motion compensation based on the filtered motion vectors and the video frame sequence to construct a prediction frame to obtain the constructed prediction frame, the corresponding relationship existing in the process is as follows: ； Among them, represents a predicted frame, represents the th frame image, represents the displacement compensation value of the pixel by the filtered motion vector , represents the pixel index; In the step of calculating the residual for the predicted frame of the structure to obtain the residual image, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the residual image, represents the th frame of RGB image, represents the dimension of the image matrix and represents the height of the image, and represents the width of the image; In the step of successively performing grayscale operation, bilateral filtering operation of the bilateral filter, and threshold binaryzation operation on the residual image to obtain the hand movement contour mask, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the grayscale image, represents the standard RGB to grayscale mapping, represents the image matrix whose dimension is , represents the bilaterally filtered image, represents the bilateral filtering operation, represents the standard deviation in the spatial domain, represents the standard deviation in the intensity domain, represents the hand motion contour mask, represents the binarization threshold.

4. The method for dynamic gesture recognition based on motion-guided dual-stream cyclic attention according to claim 3, wherein In step 3, input the hand movement contour mask and RGB features into the motion-guided gated graph neural network fusion module, and use the gated network and graph neural network to perform gated fusion and graph convolution calculation on the RGB features and the hand movement contour mask in sequence to obtain the features output by the graph neural network. The specific steps are as follows: Use the gated network to perform concatenation operation and feature fusion processing on the RGB features and the hand movement contour mask in sequence to obtain the fused features; Based on the graph neural network, construct a temporal graph, and use the graph convolution network to perform graph convolution calculation on the fused features in combination with the temporal graph to obtain the features output by the graph neural network.

5. The method for dynamic gesture recognition based on motion-guided two-stream cyclic attention according to claim 4, wherein Use the gated network to perform concatenation operation and feature fusion processing on the RGB features and the hand movement contour mask in sequence to obtain the fused features. The relational expressions existing in the corresponding process are as follows: ； Among them, represents the first gating weight, represents a function, represents the weight of the first linear layer, represents the weight of the second linear layer, represents the RGB feature, represents the bias term of the first linear layer, represents the bias term of the second linear layer, represents the feature after fusion, represents matrix multiplication; In the step of constructing a temporal graph based on the graph neural network and using the graph convolution network to perform graph convolution calculation on the fused features in combination with the temporal graph to obtain the features output by the graph neural network, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the constructed timing diagram, represents the node of the fusion feature corresponding to each time step, represents the edge in the graph neural network, represents the graph convolution calculation of the layer linear layer, represents the learnable parameter, represents the node of the fusion feature corresponding to each time step of the neighbor set, represents the graph convolution calculation of the layer linear layer, represents the mean normalization function, represents the second graph convolution calculation, represents the first graph convolution calculation, represents the feature output by the graph neural network.

6. The method for dynamic gesture recognition based on motion-guided dual-stream cyclic attention according to claim 5, wherein In step 4, input the features output by the graph neural network into the efficient multi-scale recurrent attention network, use the dynamic spatio-temporal decay module to perform dilated convolution processing to obtain the features output by the dynamic spatio-temporal decay module, and input them into the multi-scale wavelet spectrum-spatial block for multi-scale frequency domain reconstruction and channel interaction to obtain the gesture classification result. The specific steps are as follows: Input the features output by the graph neural network into the efficient multi-scale recurrent attention network, and use the dynamic spatio-temporal decay module to perform time shift operation to obtain the features after time shift; Perform dilated convolution processing on the features after time shift along the time dimension to obtain the features after dilated convolution; For the features after dilated convolution, combine the learnable time decay mechanism and the past information receiving gate, and perform linear attention mechanism processing and gated output projection processing in sequence to obtain the output features of the dynamic spatio-temporal decay module; Perform time shift operation and wavelet transform operation on the output features of the dynamic spatio-temporal decay module in sequence to obtain the low-frequency information and high-frequency information of the features; Input the low-frequency information and high-frequency information of the features into the dilated convolution group respectively for feature enhancement to obtain the enhanced low-frequency component and enhanced high-frequency component; Perform inverse wavelet transform operation and multi-scale pooling operation on the enhanced low-frequency component and enhanced high-frequency component in sequence to obtain the tensor after multi-scale pooling; Perform element-wise multiplication calculation, weight convolution mapping operation, and non-linear activation processing on the tensor after multi-scale pooling in sequence to obtain the intermediate tensor; Obtain the dynamic gating coefficient, and perform element-wise multiplication calculation on the intermediate tensor and the dynamic gating coefficient to obtain the output weighted features; The gesture classification result is obtained by passing the output weighted features through a fully connected layer.

7. The method for dynamic gesture recognition based on motion-guided dual-stream cyclic attention according to claim 6, wherein The features output by the graph neural network are input into an efficient multi-scale recurrent attention network, and a time shift operation is performed using a dynamic spatio-temporal decay module to obtain the features after time shift. The relational expressions for the corresponding process are as follows: ； Among them, represents the first feature after time offset, represents the time offset operation, represents the feature output by the graph neural network as input ; In the step of performing dilated convolution processing on the features after time shift along the time dimension to obtain the features after dilated convolution, the relational expressions for the corresponding process are as follows: ； Among them, represents the feature after multiple dilated convolutions, represents the dilated convolution operation, represents different dilation coefficients, represents the dilation coefficient, represents the convolution operation with the In the step of performing linear attention mechanism processing and gated output projection processing on the features after dilated convolution in combination with a learnable time decay mechanism and a past information reception gate to obtain the output features of the dynamic spatio-temporal decay module, the relational expressions for the corresponding process are as follows: ； Among them, represents the learnable temporal weight of each attention head, represents the normalization function, represents the learnable parameter, represents the acceptance gate, represents the one-dimensional convolution operation, represents that the dimension of the feature is , represents the dimension of each batch, represents the number of input video frames, represents the key vector, represents the value vector, represents that the dimension of the feature is , represents the dimension of each head, represents the intermediate tensor, represents the decay factor of the current output, represents the cumulative-product function, represents the learnable event decay parameter, represents the feature dimension, represents that the dimension of the feature is , represents the tensor after temporal decay, represents the cumulative-sum function, represents the output feature of the dynamic spatio-temporal decay module, represents the one-dimensional convolution operation.

8. The method for dynamically recognizing hand gestures based on motion-guided dual-stream cyclic attention according to claim 7, characterized in that Time shift operation and wavelet transform operation are sequentially performed on the output features of the dynamic spatio-temporal decay module to obtain the low-frequency information of the features and the high-frequency information of the features. The relational expressions for the corresponding process are as follows: ； Among them, represents the second time offset operation, represents the low-frequency information of the feature, represents the high-frequency information of the feature, represents the wavelet transform operation; In the step of respectively inputting the low-frequency information of the features and the high-frequency information of the features into a dilated convolution group for feature enhancement to obtain the enhanced low-frequency component and the enhanced high-frequency component, the relational expressions for the corresponding process are as follows: ； Among them, represents the enhanced low-frequency component, represents the enhanced high-frequency component, represents a two-dimensional convolution operation, represents the size of the convolution kernel, represents the dilation rate of the convolution kernel, represents the high-frequency component after the th convolution operation; In the step of performing inverse wavelet transform operation and multi-scale pooling operation on the enhanced low-frequency component and the enhanced high-frequency component in sequence to obtain the tensor after multi-scale pooling, the relational expressions for the corresponding process are as follows: ； Among them, represents the output tensor of the inverse wavelet transform, represents the inverse wavelet transform operation, represents the first high-frequency component, represents the second high-frequency component, represents the third high-frequency component, represents the tensor after multi-scale pooling, represents the multi-scale pooling operation, represents the set of pooling sizes, represents the pooling kernel sizes of different scales; In the step of performing element-wise multiplication calculation, weight convolution mapping operation and non-linear activation processing on the tensor after multi-scale pooling in sequence to obtain the intermediate tensor, the relational expressions for the corresponding process are as follows: ； Among them, represents the key tensor in the process of information interaction between channels, represents that the dimension of the feature is , represents the number of hidden layers, represents the value tensor in the process of information interaction between channels, represents the receiving gate in the process of information interaction between channels, represents that the dimension of the feature is , represents the number of features, represents the intermediate tensor generated in the process of information interaction between channels, represents the activation function; In the step of obtaining the dynamic gating coefficient and performing element-wise multiplication calculation on the intermediate tensor and the dynamic gating coefficient to obtain the output weighted features, the relational expressions for the corresponding process are as follows: ； Among them, represents the second dynamic gating coefficient, represents the acceptance gate, represents the output weighted feature; In the step of obtaining the gesture classification result by passing the output weighted features through a fully connected layer, the relational expressions for the corresponding process are as follows: ； Among them, represents the gesture classification result, represents the fully connected layer.

9. The method for dynamic gesture recognition based on motion-guided dual-stream cyclic attention according to claim 8, wherein A classification loss is constructed based on the gesture classification result. The relational expressions for the corresponding process are as follows: ； Among them, represents the classification loss, represents the number of batches of training samples, represents the -th sample belonging to the true value of the -th class, represents the -th sample belonging to the -th class's predicted probability, represents the logarithmic function.

Images