Double-flow circulation attention dynamic gesture recognition method based on motion guidance

Abstract

The invention provides a double-flow circulation attention dynamic gesture recognition method based on motion guidance, and the method comprises the steps: carrying out the motion vector extraction through employing a video coding and decoding tool, quantifying the hand dynamic intensity through combining with a prediction frame residual analysis mechanism, and generating a motion mask; sequentially performing gating fusion and graph convolution calculation on the RGB features and the hand motion contour mask by using a gating network and a graph neural network; and utilizing a dynamic space-time attenuation module and a multi-scale wavelet spectrum-space block to respectively carry out expansion convolution processing and multi-scale frequency domain reconstruction and channel interaction to obtain a gesture classification result. The invention provides a circulation attention mechanism based on a full convolution architecture, and efficient and low-complexity time sequence modeling is realized in combination with a learnable time decay strategy. Meanwhile, a wavelet decomposition reconstruction and multi-scale pooling method is adopted to capture fine-grained hand motion changes, so that the network can effectively model short time sequence dependence features, and the recognition precision of dynamic gestures is further improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and human-computer interaction, and particularly to a dual-stream recurrent attention dynamic gesture recognition method based on motion guidance. Background Art

[0002] Dynamic Hand Gesture Recognition (DHGR), as an important research direction in the fields of computer vision and human-computer interaction, has received extensive attention in recent years. The core goal of DHGR is to accurately recognize different categories of gesture actions by analyzing the dynamic trajectories of hands in video data. This technology has great potential in many practical application scenarios such as smart home control, augmented reality (AR), virtual reality (VR) interaction, and medical rehabilitation. For example, in a human-computer interaction system, the interaction method based on dynamic hand gesture recognition can greatly improve the user experience, enabling users to control devices through natural hand movements without relying on physical contact. However, limited by the locality of gestures in the video, complex background interference, and the computational cost of temporal modeling, existing methods still face many challenges.

[0003] On the one hand, since the gesture area accounts for a small proportion in the video, existing CNN-based feature extraction methods often extract a large amount of irrelevant background information, resulting in redundant feature representations and affecting the robustness of the model. In addition, the rapid changes in dynamic gestures are prone to cause motion blur, and a single RGB image only contains color and texture information, lacking the direct modeling ability of gesture motion patterns and being difficult to capture the complete gesture motion trajectory, thus reducing the recognition accuracy. On the other hand, existing methods mainly rely on Transformer for long-term temporal modeling, but the global self-attention mechanism in Transformer brings a problem of high computational complexity and is difficult to meet the requirements of real-time applications in long-term temporal video tasks. In addition, multi-modal fusion methods (such as RGB + optical flow) can enhance the temporal modeling ability to a certain extent, but the optical flow calculation cost is high and it is sensitive to light changes, restricting its popularization in practical applications.

[0004] For current mainstream dynamic gesture recognition methods, most of them adopt CNN as the backbone network for feature extraction and combine RNN, LSTM or Transformer for temporal modeling. The features extracted by the CNN backbone network contain a lot of background noise, resulting in the gesture information being easily submerged. Recurrent neural networks such as RNN / LSTM transmit time information through hidden states and lack the ability to dynamically fuse spatial features. In dynamic gesture recognition, the spatial changes of hand key points (such as joint positions) need to be modeled synchronously with the temporal evolution, and it is difficult for the hidden states of recurrent neural networks to encode spatio-temporal features simultaneously. The computational cost of traditional Transformer structures is relatively high, affecting the deployment ability of the model on embedded devices. In addition, existing methods lack effective global modeling means, resulting in difficulty in maintaining spatio-temporal consistency of gestures during cross-frame modeling, thus affecting the recognition performance. Summary of the Invention

[0005] In view of the above situation, the main purpose of the present invention is to propose a two-stream recurrent attention dynamic gesture recognition method based on motion guidance to solve the above technical problems.

[0006] The present invention proposes a two-stream recurrent attention dynamic gesture recognition method based on motion guidance, and the method includes the following steps: Step 1: Construct a motion perception data augmentation module based on the predicted frame residual analysis mechanism, construct a motion-guided gated graph neural network fusion module based on the gated network and the graph neural network, and construct an efficient multi-scale recurrent attention network based on the recurrent attention mechanism. The motion perception data augmentation module, the motion-guided gated graph neural network fusion module, and the efficient multi-scale recurrent attention network constitute a classification model; wherein, the motion perception data augmentation module includes a bilateral filter, the motion-guided gated graph neural network fusion module includes a multi-layer perceptron, and the efficient multi-scale recurrent attention network includes a dynamic spatio-temporal attenuation module and a multi-scale wavelet spectrum-spatial block; Step 2: Input the video frame sequence into the motion perception data augmentation module, extract motion vectors using video coding and decoding tools, and combine the predicted frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask to obtain a hand motion contour mask and an RGB image; Step 3: Extract RGB features based on the RGB image, input the hand motion contour mask and the RGB features into the motion-guided gated graph neural network fusion module, and use the gated network and the graph neural network for the RGB features and the hand motion contour mask to perform gated fusion and graph convolution calculations in sequence to obtain the features output by the graph neural network; Step 4: Input the features output by the graph neural network into the efficient multi-scale recurrent attention network, perform dilated convolution processing using the dynamic spatio-temporal decay module to obtain the features output by the dynamic spatio-temporal decay module, and input them into the multi-scale wavelet spectrum-spatial block for multi-scale frequency-domain reconstruction and channel interaction to obtain the gesture classification result; Construct a classification loss based on the gesture classification result, optimize the classification model using the classification loss to obtain an optimized classification model, and input the video frame sequence into the optimized classification model to obtain the final gesture classification result.

[0007] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The present invention uses the motion vector (MV) in the video compression domain to construct a prediction frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask. By reconstructing the hand contour through dynamic threshold filtering and morphological closing operation, the model can more accurately focus on the hand region while effectively suppressing background noise, thereby providing accurate spatio-temporal information for the subsequent two-stream network and improving the robustness of gesture recognition; 2. The present invention designs a two-stream fusion framework combining gated MLP and graph neural network (GCN), dynamically adjusts the contribution degrees of the RGB and motion mask (Mask) modalities through spatio-temporal perception weights, and models the cross-frame dependency relationship through fully connected temporal graph. This method can effectively enhance the adaptability of the gesture recognition system to illumination changes, background interference, and complex hand deformations, and improve the temporal modeling ability across time steps; 3. The present invention proposes a recurrent attention mechanism based on a fully convolutional architecture, combined with a learnable time decay strategy, to achieve efficient and low-complexity temporal modeling. At the same time, wavelet decomposition reconstruction and multi-scale pooling methods are used to capture fine-grained hand motion changes, enabling the network to effectively model short-term temporal dependency features and further improving the recognition accuracy of dynamic gestures.

[0008] The additional aspects and advantages of the present invention will be partially given in the following description, partially become apparent from the following description, or be understood through the embodiments of the present invention. Description of the Drawings

[0009] Figure 1 It is a flowchart of the steps of the dynamic gesture recognition method based on motion-guided two-stream recurrent attention proposed by the present invention.

[0010] Figure 2 It is an overall architecture diagram of the dynamic gesture recognition method based on motion-guided two-stream recurrent attention proposed by the present invention.

[0011] Figure 3 It is a flowchart of the motion perception data augmentation strategy of the dynamic gesture recognition method based on motion-guided two-stream recurrent attention proposed by the present invention.

[0012] Figure 4 It is a framework diagram of the motion-guided gated neural network fusion module for the dual-stream recurrent attention dynamic gesture recognition method proposed by the present invention.

[0013] Figure 5 It is a framework diagram of the efficient multi-scale recurrent attention network for the dual-stream recurrent attention dynamic gesture recognition method proposed by the present invention.

[0014] Figure 6 It is a framework diagram of the dynamic spatio-temporal attenuation module for the dual-stream recurrent attention dynamic gesture recognition method proposed by the present invention.

[0015] Figure 7 It is a framework diagram of the multi-scale wavelet spectrum-spatial block for the dual-stream recurrent attention dynamic gesture recognition method proposed by the present invention.

[0016] Figure 8 It is a framework diagram of the wavelet feature deconstruction for the dual-stream recurrent attention dynamic gesture recognition method proposed by the present invention. Detailed implementation manners

[0017] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals are the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0018] Referring to the following description and drawings, these and other aspects of the embodiments of the present invention will be clear. In these descriptions and drawings, some specific implementation manners in the embodiments of the present invention are specifically disclosed as some ways to implement the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0019] Please refer to Figure 1 , an embodiment of the present invention proposes a dual-stream recurrent attention dynamic gesture recognition method based on motion guidance, and the method includes the following steps: Step 1, construct a motion perception data enhancement module based on the prediction frame residual analysis mechanism, construct a motion-guided gated graph neural network fusion module based on the gated network and the graph neural network, and construct an efficient multi-scale recurrent attention network based on the recurrent attention mechanism. The motion perception data enhancement module, the motion-guided gated graph neural network fusion module, and the efficient multi-scale recurrent attention network constitute a classification model; wherein, the motion perception data enhancement module includes a bilateral filter, the motion-guided gated graph neural network fusion module includes a multi-layer perceptron, and the efficient multi-scale recurrent attention network includes a dynamic spatio-temporal attenuation module and a multi-scale wavelet spectrum-spatial block.

[0020] Step 2: Input the video frame sequence into the motion perception data enhancement module, extract motion vectors using video coding and decoding tools, and combine the predicted frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask, so as to obtain the hand motion contour mask and RGB image.

[0021] Please refer to Figure 2 and Figure 3 , in Step 2, input the video frame sequence into the motion perception data enhancement module, extract motion vectors using video coding and decoding tools, and combine the predicted frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask, which specifically includes the following steps: Input the video frame sequence into the motion perception data enhancement module, and use the FFmpeg video coding and decoding tool to extract vectors to obtain the extracted motion vectors; Perform a dynamic vector filtering operation on the extracted motion vectors to obtain the filtered motion vectors; Perform motion compensation based on the filtered motion vectors and the video frame sequence to construct a predicted frame, and obtain the constructed predicted frame; Calculate the residual of the constructed predicted frame to obtain the residual image; Perform a grayscale operation, a bilateral filtering operation of a bilateral filter, and a threshold binary operation on the residual image in sequence to obtain the hand motion contour mask.

[0022] It should be noted that the Figure 2 image encoder and mask encoder in the attachment are two backbone networks (FastNet and MobileNetV3 respectively, where the parameters of the image encoder are learnable (participate in gradient update), and the parameters of the mask encoder are frozen (do not participate in gradient update)).

[0023] Perform a dynamic vector filtering operation on the extracted motion vectors to obtain the filtered motion vectors. The corresponding relationship in the process is as follows: ; Among them, represents the Euclidean amplitude, represents the motion vector, represents the filtered motion vector, represents the motion amplitude threshold, represents the macroblock index of the video frame; It should be noted that controls the filtering intensity.

[0024] In the step of performing motion compensation based on the filtered motion vectors and the video frame sequence to construct a predicted frame and obtaining the constructed predicted frame, the corresponding relationship in the process is as follows: ; Among them, represents a predicted frame, represents the th frame image, represents the displacement compensation value of the pixel by the filtered motion vector , represents the pixel index; In the step of calculating the residual of the constructed predicted frame to obtain the residual image, the relational expressions existing in the corresponding process are as follows: ; Among them,[[]] represents the residual image, represents the th frame RGB image, represents the dimension of the image matrix , represents the height of the image, represents the width of the image, represents the number of channels of the image; It should be noted that is to highlight the pixel difference in the motion area.

[0025] In the step of sequentially performing grayscale operation, bilateral filtering operation of the bilateral filter, and threshold binary operation on the residual image to obtain the hand motion contour mask, the relational expressions existing in the corresponding process are as follows: ; Among them, represents the grayscale image, represents the standard RGB-to-grayscale mapping, represents the dimension of the image matrix is , represents the bilaterally filtered image, represents the bilateral filtering operation, represents the standard deviation in the spatial domain, represents the standard deviation in the intensity domain, represents the hand motion contour mask, represents the binary threshold.

[0026] Furthermore, in order to improve the robustness of dynamic gesture recognition, reduce the computational complexity, and accurately focus on the hand motion area, the present invention uses the motion vectors in the video compression domain to extract hand motion features, constructs a predicted frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask, and at the same time introduces dynamic threshold filtering and morphological closing operation to remove background noise and ensure that the model focuses on the effective features of the hand.

[0027] Step 3: Extract RGB features from the RGB image, input the hand motion contour mask and the RGB features into the motion-guided gated graph neural network fusion module, and use the gated network and the graph neural network to perform gated fusion and graph convolution calculations on the RGB features and the hand motion contour mask in sequence to obtain the features output by the graph neural network.

[0028] Please refer to Figure 4 , in Step 3, input the hand motion contour mask and the RGB features into the motion-guided gated graph neural network fusion module, and use the gated network and the graph neural network to perform gated fusion and graph convolution calculations on the RGB features and the hand motion contour mask in sequence to obtain the features output by the graph neural network, which specifically includes the following steps: Perform a concatenation operation and feature fusion process on the RGB features and the hand motion contour mask using the gated network in sequence to obtain the fused features; Construct a temporal graph based on the graph neural network, and perform graph convolution calculations on the fused features using the graph convolution network in combination with the temporal graph to obtain the features output by the graph neural network.

[0029] Perform a concatenation operation and feature fusion process on the RGB features and the hand motion contour mask using the gated network in sequence to obtain the fused features. The relational expressions for the corresponding process are as follows: ; Among them, represents the first gated weight, represents function, represents the weight of the first linear layer, represents the weight of the second linear layer, represents the RGB features, represents the bias term of the first linear layer, represents the bias term of the second linear layer, represents the fused features, represents matrix multiplication; It should be noted that in Appendix Figure 4 the function is used to obtain the gated coefficient by normalizing the output of the multi-layer perceptron.

[0030] In the step of constructing a temporal graph based on the graph neural network and performing graph convolution calculations on the fused features using the graph convolution network in combination with the temporal graph to obtain the features output by the graph neural network, the relational expressions for the corresponding process are as follows: ; Among them, represents the constructed temporal graph, Nodes representing the fused features corresponding to each time step. Edges in the graph neural network. Represents the Graph convolution calculation of the Learnable parameters. Nodes representing the fused features corresponding to each time step Neighbor set of Represents the Graph convolution calculation of the Represents the mean normalization function. Represents the second graph convolution calculation. Represents the first graph convolution calculation. Features output by the graph neural network. Represents the index of the linear layer.

[0031] Step 4: Input the features output by the graph neural network into the efficient multi-scale recurrent attention network, perform dilated convolution processing using the dynamic spatio-temporal decay module to obtain the features output by the dynamic spatio-temporal decay module, and input them into the multi-scale wavelet spectrum-spatial block for multi-scale frequency domain reconstruction and channel interaction to obtain the gesture classification result; Construct a classification loss based on the gesture classification result, optimize the classification model using the classification loss to obtain the optimized classification model, and input the video frame sequence into the optimized classification model to obtain the final gesture classification result.

[0032] Please refer to Figures 5 to 8 , in Step 4, input the features output by the graph neural network into the efficient multi-scale recurrent attention network, perform dilated convolution processing using the dynamic spatio-temporal decay module to obtain the features output by the dynamic spatio-temporal decay module, and input them into the multi-scale wavelet spectrum-spatial block for multi-scale frequency domain reconstruction and channel interaction to obtain the gesture classification result, which specifically includes the following steps: Input the features output by the graph neural network into the efficient multi-scale recurrent attention network and perform a time shift operation using the dynamic spatio-temporal decay module to obtain the features after time shift; Perform dilated convolution processing on the features after time shift along the time dimension to obtain the features after dilated convolution; For the features after dilated convolution, combine the learnable time decay mechanism and the past information reception gate, and perform linear attention mechanism processing and gated output projection processing in sequence to obtain the output features of the dynamic spatio-temporal decay module; Perform time shift operation and wavelet transform operation on the output features of the dynamic spatio-temporal decay module in sequence to obtain the low-frequency information and high-frequency information of the features; The low-frequency information of the feature and the high-frequency information of the feature are respectively input into the dilated convolution group for feature enhancement to obtain the enhanced low-frequency component and the enhanced high-frequency component; Inverse wavelet transform operations and multi-scale pooling operations are sequentially performed on the enhanced low-frequency component and the enhanced high-frequency component to obtain the tensor after multi-scale pooling; Element-wise multiplication calculations, weighted convolution mapping operations, and non-linear activation processing are sequentially performed on the tensor after multi-scale pooling to obtain the intermediate tensor; A dynamic gating coefficient is obtained, and element-wise multiplication calculation is performed on the intermediate tensor and the dynamic gating coefficient to obtain the output weighted feature; The gesture classification result is obtained through a fully connected layer for the output weighted feature.

[0033] The features output by the graph neural network are input into the efficient multi-scale recurrent attention network, and a time shift operation is performed using the dynamic spatio-temporal decay module to obtain the features after time shift. The relational expressions existing in the corresponding process are as follows: ; Among them, represents the first feature after time shift, represents the time shift operation, represents the feature output by the th layer of the graph neural network as the input ; In the step of performing dilated convolution processing on the features after time shift along the time dimension to obtain the features after dilated convolution, the relational expressions existing in the corresponding process are as follows: ; Among them, represents the features after multiple dilated convolutions, represents the dilated convolution operation, represents different dilation coefficients, represents the dilation coefficient, represents the th convolution operation with different dilation coefficients; In the step of sequentially performing linear attention mechanism processing and gated output projection processing on the features after dilated convolution by combining the learnable time decay mechanism and the past information reception gate to obtain the output features of the dynamic spatio-temporal decay module, the relational expressions existing in the corresponding process are as follows: ; Among them, represents the learnable time weight of each attention head, represents the normalization function, represents the learnable parameter, Indicates an acceptance gate, Indicates a one-dimensional convolution operation, Indicates that the dimension of the feature is , Indicates the dimension of each batch, Indicates the number of input video frames, Indicates the key vector, Indicates the value vector, Indicates that the dimension of the feature is , Indicates the dimension of each head, Indicates the intermediate tensor, Indicates the decay factor of the current output, Indicates the cumulative-product function, Indicates the learnable event decay parameter, Indicates the feature dimension, Indicates that the dimension of the feature is , Indicates the tensor after time decay, Indicates the cumulative sum function, Indicates the output feature of the dynamic spatio-temporal decay module, Indicates a one-dimensional convolution operation.

[0034] It should be noted that to reduce the computational complexity, the traditional dot product attention is replaced by a vectorized cumulative operation. This mechanism first performs an element-wise multiplication between the key and value matrices to generate an intermediate tensor with fused features. Then, a cumulative-product function is used along the time dimension to generate a sequence of decay factors, simulating the gradual decay of historical information over time; finally, a cumulative sum is used to linearly accumulate the weighted features, effectively capturing the long-range dependencies in the gesture sequence while reducing the complexity in the traditional attention mechanism from to .

[0035] The output features of the dynamic spatio-temporal decay module are sequentially subjected to a time shift operation and a wavelet transform operation to obtain the low-frequency information and high-frequency information of the features. The corresponding relationships in the process are as follows: ; Among them, Indicates the second time shift operation, Indicates the low-frequency information of the feature, Indicates the high-frequency information of the feature, Indicates the wavelet transform operation; It should be noted that this process is the process of wavelet feature deconstruction.

[0036] In the step of separately inputting the low-frequency information of the feature and the high-frequency information of the feature into the dilated convolution group for feature enhancement to obtain the enhanced low-frequency component and the enhanced high-frequency component, the relational expressions existing in the corresponding process are as follows: ; Among them, represents the enhanced low-frequency component, represents the enhanced high-frequency component, represents the two-dimensional convolution operation, represents the size of the convolution kernel, represents the dilation rate of the convolution kernel, represents the th high-frequency component processed by the convolution operation; In the step of successively performing the inverse wavelet transform operation and the multi-scale pooling operation on the enhanced low-frequency component and the enhanced high-frequency component to obtain the tensor after multi-scale pooling, the relational expressions existing in the corresponding process are as follows: ; Among them, represents the output tensor of the inverse wavelet transform, represents the inverse wavelet transform operation, represents the first high-frequency component, represents the second high-frequency component, represents the third high-frequency component, represents the tensor after multi-scale pooling, represents the multi-scale pooling operation, represents the set of pooling sizes, represents the pooling kernel sizes of different scales; In the step of successively performing element-wise multiplication calculation, weight convolution mapping operation and non-linear activation processing on the tensor after multi-scale pooling to obtain the intermediate tensor, the relational expressions existing in the corresponding process are as follows: ; Among them, represents the key tensor in the inter-channel information interaction process, represents that the dimension of the feature is , represents the number of hidden layers, represents the value tensor in the inter-channel information interaction process, represents the receiving gate in the inter-channel information interaction process, represents that the dimension of the feature is , represents the number of features, represents the intermediate tensor generated in the inter-channel information interaction process, represents the activation function; In the step of obtaining the dynamic gating coefficient and performing element-wise multiplication on the intermediate tensor and the dynamic gating coefficient to obtain the output weighted feature, the relational expressions in the corresponding process are as follows: ; Among them, represents the second dynamic gating coefficient, represents the acceptance gate, represents the output weighted feature; In the step of obtaining the gesture classification result by passing the output weighted feature through the fully connected layer, the relational expressions in the corresponding process are as follows: ; Among them, represents the gesture classification result, represents the fully connected layer.

[0037] Based on the gesture classification result, a classification loss is constructed, and the relational expressions in the corresponding process are as follows: ; Among them, represents the classification loss, represents the number of batches of training samples, represents the th sample belonging to the true value of the th class, represents the th sample belonging to the predicted probability of the th class, represents the logarithmic function.

[0038] Furthermore, the efficient multi-scale recurrent attention network, as a temporal modeling network, receives the feature input after the fusion of the backbone network and the motion-guided gating graph neural network fusion module, and adopts a recurrent structure similar to RNN but based on fully convolutional computation. The efficient multi-scale recurrent attention network is composed of alternating stacks of a dynamic spatio-temporal decay module (DSTD) and a multi-scale wavelet spectrum-spatial block (MSWSS). The dynamic spatio-temporal decay module efficiently models global and local temporal dependencies through multi-scale dilated convolution and convolutional attention; the multi-scale wavelet spectrum-spatial block combines wavelet transform and multi-scale pooling to enhance channel interaction and capture fine-grained gesture features. The overall architecture combines residual connections and normalization to achieve efficient parallel training and improve the dynamic robustness of the model.

[0039] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one of the following techniques known in the art or a combination thereof can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0040] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0041] The above-described embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be subject to the appended claims.

Claims

1. A two-stream cyclic attention dynamic gesture recognition method based on motion guidance, characterized in that The method includes the following steps: Step 1: Construct a motion perception data enhancement module based on a prediction frame residual analysis mechanism, construct a motion-guided gated graph neural network fusion module based on a gated network and a graph neural network, and construct an efficient multi-scale recurrent attention network based on a recurrent attention mechanism. The motion perception data enhancement module, the motion-guided gated graph neural network fusion module, and the efficient multi-scale recurrent attention network constitute a classification model. Among them, the motion perception data enhancement module includes a bilateral filter, the motion-guided gated graph neural network fusion module includes a multi-layer perceptron, and the efficient multi-scale recurrent attention network includes a dynamic spatio-temporal attenuation module and a multi-scale wavelet spectrum-spatial block; Step 2: Input the video frame sequence into the motion perception data enhancement module, use video codec tools to extract motion vectors, and combine the prediction frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask to obtain a hand motion contour mask and an RGB image; Step 3: Extract RGB features based on the RGB image, input the hand motion contour mask and the RGB features into the motion-guided gated graph neural network fusion module, and use the gated network and the graph neural network for the RGB features and the hand motion contour mask to perform gated fusion and graph convolution calculations in sequence to obtain the features output by the graph neural network; Step 4: Input the features output by the graph neural network into the efficient multi-scale recurrent attention network, perform dilated convolution processing using the dynamic spatio-temporal attenuation module to obtain the features output by the dynamic spatio-temporal attenuation module, and input them into the multi-scale wavelet spectrum-spatial block for multi-scale frequency domain reconstruction and channel interaction to obtain a gesture classification result; Construct a classification loss based on the gesture classification result, use the classification loss to optimize the classification model to obtain an optimized classification model, and input the video frame sequence into the optimized classification model to obtain the final gesture classification result.

2. The method for dynamic gesture recognition based on motion-guided dual-stream cyclic attention according to claim 1, wherein In the said Step 2, when inputting the video frame sequence into the motion perception data enhancement module, using video codec tools to extract motion vectors, and combining the prediction frame residual analysis mechanism to quantify the hand dynamic intensity and generate a motion mask, it specifically includes the following steps: Input the video frame sequence into the motion perception data enhancement module, use the FFmpeg video codec tool to extract vectors to obtain the extracted motion vectors; Perform a dynamic vector filtering operation on the extracted motion vectors to obtain filtered motion vectors; Perform motion compensation based on the filtered motion vectors and the video frame sequence to construct a prediction frame to obtain the constructed prediction frame; Perform residual calculation on the constructed prediction frame to obtain a residual image; Perform grayscale operation, bilateral filtering operation of the bilateral filter, and threshold binary operation on the residual image in sequence to obtain a hand motion contour mask.

3. The method for dynamic gesture recognition based on motion-guided dual-stream cyclic attention according to claim 2, wherein Perform a dynamic vector filtering operation on the extracted motion vectors to obtain filtered motion vectors. The corresponding relationship existing in the process is as follows: ； Among them, represents the Euclidean amplitude, represents the motion vector, represents the filtered motion vector, represents the motion amplitude threshold, represents the macroblock index of the video frame; In the step of performing motion compensation based on the filtered motion vectors and the video frame sequence to construct a prediction frame to obtain the constructed prediction frame, the corresponding relationship existing in the process is as follows: ； Among them, represents a predicted frame, represents the th frame image, represents the displacement compensation value of the pixel by the filtered motion vector, and represents the pixel index;​ In the step of calculating the residual for the predicted frame of the structure to obtain the residual image, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the residual image, represents the th frame of RGB image, represents the dimension of the image matrix and represents the height of the image, represents the width of the image; In the step of successively performing grayscale operation, bilateral filtering operation of the bilateral filter, and threshold binary operation on the residual image to obtain the hand motion contour mask, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the grayscale image, represents the standard RGB to grayscale mapping, represents the image matrix whose dimension is , represents the bilaterally filtered image, represents the bilateral filtering operation, represents the standard deviation in the spatial domain, represents the standard deviation in the intensity domain, represents the hand motion contour mask, represents the binarization threshold.

4. The method for dynamic gesture recognition based on motion-guided dual-stream cyclic attention according to claim 3, wherein In step 3, the hand motion contour mask and the RGB feature are input into the motion-guided gated graph neural network fusion module. The RGB feature and the hand motion contour mask are successively subjected to gated fusion and graph convolution calculation using the gated network and the graph neural network to obtain the feature output by the graph neural network. The specific steps are as follows: The RGB feature and the hand motion contour mask are successively subjected to a concatenation operation and a feature fusion process using the gated network to obtain the fused feature; Based on the graph neural network, a temporal graph is constructed, and the fused feature is subjected to graph convolution calculation using the graph convolution network in combination with the temporal graph to obtain the feature output by the graph neural network.

5. The method for dynamically recognizing gesture based on motion-guided dual-stream cyclic attention according to claim 4, wherein The RGB feature and the hand motion contour mask are successively subjected to a concatenation operation and a feature fusion process using the gated network to obtain the fused feature. The relational expressions existing in the corresponding process are as follows: ； Among them, represents the first gating weight, represents function, represents the weight of the first linear layer, represents the weight of the second linear layer, represents the RGB feature, represents the bias term of the first linear layer, represents the bias term of the second linear layer, represents the feature after fusion, represents matrix multiplication; In the step of constructing a temporal graph based on the graph neural network and performing graph convolution calculation on the fused feature using the graph convolution network in combination with the temporal graph to obtain the feature output by the graph neural network, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the constructed timing diagram, represents the node of the fusion feature corresponding to each time step, represents the edge in the graph neural network, represents the graph convolution calculation of the layer linear layer, represents the learnable parameter, represents the neighbor set of the node of the fusion feature corresponding to each time step, represents the mean normalization function, represents the second graph convolution calculation, represents the first graph convolution calculation, represents the feature output by the graph neural network.

6. The method for dynamic gesture recognition based on motion-guided dual-stream cyclic attention according to claim 5, wherein In step 4, the feature output by the graph neural network is input into the efficient multi-scale recurrent attention network, and dilated convolution processing is performed using the dynamic spatio-temporal decay module to obtain the feature output by the dynamic spatio-temporal decay module, and then input into the multi-scale wavelet spectrum-spatial block for multi-scale frequency domain reconstruction and channel interaction to obtain the gesture classification result. The specific steps are as follows: The feature output by the graph neural network is input into the efficient multi-scale recurrent attention network, and a time shift operation is performed using the dynamic spatio-temporal decay module to obtain the feature after time shift; The feature after time shift is subjected to dilated convolution processing along the time dimension to obtain the feature after dilated convolution; The feature after dilated convolution is successively subjected to linear attention mechanism processing and gated output projection processing in combination with the learnable time decay mechanism and the past information receiving gate to obtain the output feature of the dynamic spatio-temporal decay module; The output feature of the dynamic spatio-temporal decay module is successively subjected to a time shift operation and a wavelet transform operation to obtain the low-frequency information of the feature and the high-frequency information of the feature; The low-frequency information of the feature and the high-frequency information of the feature are respectively input into the dilated convolution group for feature enhancement to obtain the enhanced low-frequency component and the enhanced high-frequency component; The enhanced low-frequency component and the enhanced high-frequency component are successively subjected to an inverse wavelet transform operation and a multi-scale pooling operation to obtain the tensor after multi-scale pooling; The tensor after multi-scale pooling is successively subjected to element-wise multiplication calculation, weight convolution mapping operation, and non-linear activation processing to obtain the intermediate tensor; The dynamic gating coefficient is obtained, and the intermediate tensor and the dynamic gating coefficient are subjected to element-wise multiplication calculation to obtain the output weighted feature; The gesture classification result is obtained by passing the output weighted features through a fully connected layer.

7. The method for dynamic gesture recognition based on motion-guided dual-stream cyclic attention according to claim 6, wherein The features output by the graph neural network are input into an efficient multi-scale recurrent attention network, and a time shift operation is performed using a dynamic spatio-temporal decay module to obtain the features after time shift. The relational expressions for the corresponding process are as follows: ； Among them, represents the first feature after time offset, represents the time offset operation, represents the feature output by the input graph neural network ; In the step of performing dilated convolution processing on the features after time shift along the time dimension to obtain the features after dilated convolution, the relational expressions for the corresponding process are as follows: ； Among them, represents the feature after multiple dilated convolutions, represents the dilated convolution operation, represents different dilation coefficients, represents the dilation coefficient, represents the convolution operation with the th different dilation coefficient; In the step of performing linear attention mechanism processing and gated output projection processing on the features after dilated convolution in combination with a learnable time decay mechanism and a past information reception gate to obtain the output features of the dynamic spatio-temporal decay module, the relational expressions for the corresponding process are as follows: ； Among them, represents the learnable temporal weight of each attention head, represents the normalization function, represents the learnable parameter, represents the acceptance gate, represents a one-dimensional convolution operation, represents that the dimension of the feature is , represents the dimension of each batch, represents the number of input video frames, represents the key vector, represents the value vector, represents that the dimension of the feature is , represents the dimension of each head, represents the intermediate tensor, represents the decay factor of the current output, represents the cumulative-product function, represents the learnable event decay parameter, represents the feature dimension, represents that the dimension of the feature is , represents the tensor after temporal decay, represents the cumulative-sum function, represents the output feature of the dynamic spatio-temporal decay module, represents a one-dimensional convolution operation.

8. The method for dynamic gesture recognition based on motion-guided two-stream cyclic attention according to claim 7, wherein Time shift operations and wavelet transform operations are sequentially performed on the output features of the dynamic spatio-temporal decay module to obtain the low-frequency information and high-frequency information of the features. The relational expressions for the corresponding process are as follows: ； Among them, represents the second time offset operation, represents the low-frequency information of the feature, represents the high-frequency information of the feature, represents the wavelet transform operation; In the step of inputting the low-frequency information and high-frequency information of the features into a dilated convolution group respectively for feature enhancement to obtain the enhanced low-frequency component and enhanced high-frequency component, the relational expressions for the corresponding process are as follows: ； Among them, represents the enhanced low-frequency component, represents the enhanced high-frequency component, represents a two-dimensional convolution operation, represents the size of the convolution kernel, represents the dilation rate of the convolution kernel, represents the high-frequency component after the th convolution operation; In the step of performing inverse wavelet transform operations and multi-scale pooling operations on the enhanced low-frequency component and enhanced high-frequency component sequentially to obtain the tensor after multi-scale pooling, the relational expressions for the corresponding process are as follows: ； Among them, represents the output tensor of the inverse wavelet transform, represents the inverse wavelet transform operation, represents the first high-frequency component, represents the second high-frequency component, represents the third high-frequency component, represents the tensor after multi-scale pooling, represents the multi-scale pooling operation, represents the set of pooling sizes, represents the pooling kernel sizes of different scales; In the step of performing element-wise multiplication calculation, weight convolution mapping operation, and non-linear activation processing on the tensor after multi-scale pooling sequentially to obtain an intermediate tensor, the relational expressions for the corresponding process are as follows: ； Among them, represents the key tensor in the process of information interaction between channels, represents that the dimension of the feature is , represents the number of hidden layers, represents the value tensor in the process of information interaction between channels, represents the receiving gate in the process of information interaction between channels, represents that the dimension of the feature is , represents the number of features, represents the intermediate tensor generated in the process of information interaction between channels, represents the activation function; In the step of obtaining a dynamic gating coefficient and performing element-wise multiplication calculation on the intermediate tensor and the dynamic gating coefficient to obtain the output weighted features, the relational expressions for the corresponding process are as follows: ； Among them, represents the second dynamic gating coefficient, represents the acceptance gate, represents the output weighted feature; In the step of obtaining the gesture classification result by passing the output weighted features through a fully connected layer, the relational expressions for the corresponding process are as follows: ； Among them, represents the gesture classification result, represents the fully connected layer.

9. The method for dynamic gesture recognition based on motion-guided dual-stream cyclic attention according to claim 8, wherein, A classification loss is constructed based on the gesture classification result. The relational expressions for the corresponding process are as follows: ； in, represents the classification loss, represents the batch number of training samples, Indicates The samples belong to The true value of the class, Indicates The samples belong to The predicted probability of the class, Represents a logarithmic function.

Images