A dynamic gesture recognition method based on spatiotemporal interaction and rate awareness
By combining 3D convolution and Transformer models and introducing visual rhythm modeling and feature interaction modules, the problem of insufficient spatiotemporal modeling in dynamic gesture recognition is solved, thereby improving recognition accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SICHUAN UNIV
- Filing Date
- 2024-11-20
- Publication Date
- 2026-05-26
AI Technical Summary
Existing dynamic gesture recognition methods fail to effectively consider visual rhythm features during spatiotemporal modeling, and the Transformer model is insufficient in capturing local features, resulting in insufficient recognition accuracy.
A 3D convolutional neural network is used to extract spatial features, combined with Transformer to process temporal relationships, and gesture visual rhythm modeling is introduced. At the same time, a spatial feature injection module and a temporal feature interaction module are embedded to enhance the capture of local texture details and temporal information between adjacent frames.
It improves the classification accuracy of dynamic gesture recognition, especially with competitive performance on the Jester and DHG28 datasets.
Smart Images
Figure CN119600684B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of human-computer interaction, dynamic gesture recognition, deep learning, and computer vision; and particularly to image processing technologies such as convolutional neural networks, Transformer, and attention mechanisms. Background Technology
[0002] Gestures are intentional actions and symbols that involve upper body movements, particularly the arms, hands, and fingers. Video-based classification is a crucial component of gesture recognition. It has been applied to many human-centered tasks, such as epigenetic analysis, sign language recognition, and human-computer interaction. Compared to static gestures, dynamic gestures can convey more information. Dynamic gesture recognition employs various methods, including those based on wearable sensing devices, hand skeletal keypoint extraction, and vision-based approaches. Considering the cost of algorithm implementation and the need for portable interaction, vision-based dynamic gesture recognition presents a challenging yet worthy research topic.
[0003] Vision-based gesture recognition can be broadly categorized into three methods: First, methods based on convolutional neural networks (CNNs). Extensive research in computer vision in recent years has demonstrated that convolution is the most suitable tool for extracting image features; therefore, CNNs have wide applications in dynamic gesture recognition. Furthermore, different gestures in dynamic gestures may exhibit the same image-level features. In this case, temporal features become the effective basis for distinguishing them, so 3D convolution methods are generally superior to 2D convolution. Second, methods based on LSTM. Since dynamic gesture recognition is highly sensitive to temporal features, another method has emerged: first, image-level features are extracted from the video using convolution, and then the feature map is input into an LSTM network to obtain relevant temporal information. This approach is logically feasible, and the number of model parameters is much smaller than the first method. However, in actual experiments, this method's performance is suboptimal. Third, methods based on transformers. The Transformer architecture was proposed for natural language processing tasks and has become a de facto standard in the field. Recent research shows that Transformer methods also have high research value in the field of vision, and in the field of dynamic gesture recognition, Transformer-based methods have been a research hotspot in recent years. Despite the emergence of numerous excellent research results in the field, these results also have their limitations. Existing research focuses on spatiotemporal modeling. Although spatiotemporal modeling is a key factor in gesture recognition performance, dynamic gestures not only have spatiotemporal dimensions but also gesture visual tempo. Gesture visual tempo characterizes the dynamics and time scale of the action, which helps to distinguish gestures that are highly similar in visual dynamics and appearance. However, current research has not taken into account the visual tempo aspect.
[0004] Therefore, it is necessary to design a gesture recognition method that incorporates visual rhythm modeling on the basis of spatiotemporal modeling to fill the research gap in this field. Secondly, with technological advancements, the application of Transformers in computer vision is becoming increasingly widespread. Indeed, when processing image tasks, Transformer models tend to capture global features, while their ability to capture local features is relatively weak. This is mainly due to the design characteristics of their self-attention mechanism.
[0005] The Transformer's self-attention mechanism directly establishes global element relationships by calculating the attention weights between each element in the input sequence and other elements. This makes the Transformer particularly adept at capturing long-distance dependencies and global contextual information. However, this global operation may be insufficient when capturing detailed or local features of an image. In image processing tasks, Convolutional Neural Networks (CNNs) effectively capture local features and spatial information by sliding convolutional kernels across local regions. The standard Transformer lacks this local awareness mechanism. In the field of dynamic gesture recognition, capturing fine-grained changes in hand pose is a reliable guarantee of recognition accuracy.
[0006] Therefore, it is necessary to design a spatiotemporal feature injection interaction mechanism to enhance the transformer's attention to local texture detail features. Summary of the Invention
[0007] To address the shortcomings of existing technologies, the present invention aims to achieve the above objectives. The technical solution adopted by the present invention is as follows: Figure 1 As shown, a 3D convolutional neural network is used to extract features in the spatial dimension, and a transformer is used to process these features to obtain the temporal relationships. At the same time, gesture visual rhythm modeling is introduced to obtain the rate features of dynamic gestures.
[0008] Based on the above technical solutions, this method embeds a spatial feature injection module (SFIM) and a temporal feature interaction module (TFIM) into the Transformer to refine small-scale local features, enhance temporal information between adjacent frames, and capture the continuity of video in the temporal dimension.
[0009] Compared with existing technologies, the advantages of this invention are: a complete feature extraction mechanism that considers feature extraction from multiple dimensions and scales; and a more complete feature enhancement mechanism compared with other methods. Experiments have shown that this method achieves competitive classification accuracy on the Jester dataset (RGB modality) and DHG28 (deep modality).
[0010] Furthermore, the technical solution of the present invention is as follows:
[0011] Step 1: Obtain the dynamic gesture dataset and perform data preprocessing;
[0012] Step 2: Initially extract texture features through a series of 3D convolution operations to obtain a low-level semantic feature map;
[0013] Step 3: Extract video rate features through the rate feature injection module, and overlay the rate features with the low-level semantic feature map;
[0014] Step 4: Further extract multi-scale spatial features through a series of extended convolution operations;
[0015] Step 5: After feature sampling of the feature map obtained in Step 4, feature extraction is performed through a Transformer network that integrates the STFIM module.
[0016] Step 6: Input the feature map obtained in Step 5 into a multilayer perceptron to obtain the classification result.
[0017] Furthermore, the dynamic gesture dataset in step 1 is in the form of gesture videos, which can be composed of RGB or depth modalities. Data preprocessing methods include cropping, rotation, and shuffle-mixing. These operations can enrich the data and fully utilize the model's expressive power, especially on smaller datasets, effectively mitigating overfitting issues.
[0018] Furthermore, in step 2, the texture feature extraction uses the first five layers of the I3D network to output the feature map dimension. .
[0019] The rate feature extraction in step 3 includes the following steps:
[0020] Step 3.1: Using the texture feature map obtained in Step 2 As input, image sampling and similarity calculation are performed. This involves selecting a region at the same location in each frame of the image for similarity calculation. Higher inter-frame similarity indicates a lower visual rate. The computational complexity and sampling radius are also considered. Related, in the input feature map dimension of In this case, the dimensions of the similarity calculation results are: For each frame, the closest and farthest frames are selected for similarity calculation. The frame sampling method used in this method avoids the excessive computation caused by pyramid sampling and the high data dimensionality that leads to feature maps.
[0021] Step 3.2: Using the similarity feature map obtained in step 3.1 as input, perform optical flow estimation on the correlation tensor between frame pairs, and then use the calculated optical flow and the corresponding confidence map to extract motion information;
[0022] Step 3.3: Concatenate the dual-channel optical flow with the single-channel confidence map to form a pixel-level displacement map to further depict motion features. Estimate the longest and shortest scale motions and merge them to adapt to slow and fast downstream feature transformations, thus better understanding gesture movements. The final displacement map has the following dimensions: ;
[0023] Step 3.4: Using the displacement map obtained in Step 3.3 as input, first transform the displacement map using depthwise separable convolution to align it with the original feature map. Then, use global average pooling to capture aggregated features, given a time period for the aggregated features. Where T represents the feature time dimension, rate attention can be learned without dimensionality reduction using the following formula:
[0024]
[0025] Where W is a general parameter matrix with T × T elements;
[0026] Step 3.5: Superimpose the original shallow feature map with the weights learned through the rate attention mechanism to obtain enhanced features containing rate information, and output the feature map. .
[0027] Furthermore, the multi-scale spatial feature extraction in step 4 is characterized by the feature map... A series of spatially centered 3D Inception modules, with convolutional kernels of size 1 in the time dimension, will extract multi-scale features at the k-th layer in the following manner:
[0028]
[0029] in This represents the feature map of the nth layer. This represents an Inception module with a learnable parameter matrix W.
[0030] Furthermore, the spatiotemporal feature injection interaction module in step 5 consists of multiple spatial feature injection modules (SFIM) and temporal feature interaction modules (TFIM). The spatial feature injection module uses local detail features. (No. Each SFIM module serves as the key and value, while global features... As a query, CrossAttention is applied to obtain spatial features that enhance local detail awareness. :
[0031]
[0032] Where LN(·) represents layer normalization. CrossAttention represents a cross-attention layer. This design aims to enhance the transformer structure's perception of important local features. In dynamic gesture recognition tasks, the importance of each local detail in the entire image to the classification task varies, while the transformer has a weak perception of local features. In the field of dynamic gesture recognition, capturing fine-grained changes in hand posture is a reliable guarantee of recognition accuracy. Therefore, the spatial feature injection module uses cross-attention to integrate local features into global features.
[0033] Furthermore, the time feature interaction module starts from frame t and frame (t... 1) The shadow features extracted from the frame are and , and The bidirectional interaction is used for feature integration. The network adopts a twin structure in the time dimension, with feature maps from two adjacent frames being input simultaneously, and the feature maps from the t-th frame being integrated. In each TFIM, As a query, and As keys and values; in the In frame feature extraction, their roles are reversed. This mechanism is designed to focus on feature interactions between adjacent frames, because in the field of dynamic gesture recognition, the temporal relationship of actions between frames is an important basis for classification tasks. Therefore, interacting between adjacent frames through a cross-attention mechanism helps the model capture complete spatiotemporal information and make the most accurate classification.
[0034] Furthermore, the multilayer perceptron in step six consists of a series of fully connected layers. The multilayer perceptron is a commonly used classification method in classification tasks in the field of deep learning, which classifies the features obtained that contain multi-dimensional and multi-scale semantic information. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings described below are merely an abstract process structure of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 This is a schematic diagram of the backbone network structure;
[0037] Figure 2 for Figure 1 A schematic diagram of the RFIM rate feature injection module structure;
[0038] Figure 3 for Figure 1 A schematic diagram of the Transformer-STFIM structure in the image;
[0039] Figure 4 for Figure 3 A schematic diagram of the Spatial Feature Injection Module (SFIM) structure;
[0040] Figure 5 for Figure 3 A schematic diagram of the Time Feature Interaction Module (TFIM) structure in the system. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0042] For step 1, the video frames are cropped to 224*224 pixels using the Jester public dataset, and the ShuffleMix image enhancement operation is performed. Specifically, the ShuffleMix value is set to 0.2 and the Mixup value is set to 0.8.
[0043] For step 2, the first five layers of the I3D network are used to extract shallow texture features from the model.
[0044] For step 3, follow these steps:
[0045] Step 3.1: Using the texture feature map obtained in Step 2 As input, image sampling and similarity calculation involve selecting a region at the same location in each frame of the image for similarity calculation. The computational complexity and sampling radius are also relevant. Related, in the input feature map dimension of In this case, the dimensions of the similarity calculation results are: For each frame, the nearest and farthest frames are selected for similarity calculation. Similarity calculation can be performed using techniques such as dot product, SSIM, or PSNR. This method uses SSIM similarity for rate representation. in, , ;
[0046] Step 3.2: Using the similarity feature map obtained in step 3.1 as input, a lightweight MotionSqueeze method can be used to estimate the optical flow of the correlation tensor between frame pairs. Then, the calculated optical flow and the corresponding confidence map are used to extract motion information.
[0047] Step 3.3: Concatenate the dual-channel optical flow with the single-channel confidence map to form a pixel-level displacement map to further depict motion features. Estimate the longest and shortest scale motions and merge them to adapt to slow and fast downstream feature transformations, thus better understanding gesture movements. The final displacement map has the following dimensions: ;
[0048] Step 3.4: Using the displacement map obtained in Step 3.3 as input, first transform the displacement map using depthwise separable convolution to align it with the original feature map. Then, use global average pooling to capture aggregated features, given a time period for the aggregated features. Where T represents the feature time dimension, rate attention can be learned without dimensionality reduction using the following formula: , where W is a general parameter matrix with T × T elements.
[0049] For step 4, feature map It will extract multi-scale features at the k-th layer by using a series of spatially centered 3D Inception modules with convolution kernels of size 1 in the time dimension: in This represents the feature map of the nth layer. This represents an Inception module with a learnable parameter matrix W.
[0050] For step 5, multi-scale feature sampling is first performed in the following manner: ,in This indicates taking a random integer between m and n. Obtain the downsampled feature map. We then pass it through a 3D Inception module with a convolutional kernel of size 1 × 1 in spatial dimension and a max pooling layer to capture local fine-grained features. Within the Transformer module, four Spatial Feature Injection (SFIM) modules and a Temporal Feature Interaction (TFIM) module are used, stacked in a linear series. The Spatial Feature Injection module uses local detail features. (No. Each SFIM module serves as the key and value, while global features... As a query, CrossAttention is applied to obtain spatial features that enhance local detail awareness. , where N(·) represents layer normalization. CrossAttention represents the cross-attention layer.
[0051] For step six, add three fully connected layers, and set the output dimension of the last layer to the number of gesture categories. The number of categories in the Jester dataset is 27.
[0052] The formulas for steps five and six are expressed as follows: ,in, This represents a multilayer perceptron used for feature mapping. Indicates the normalization layer. This represents a k-NN multi-head self-attention layer. Furthermore, to avoid overfitting of the transformer branch at a specific time and resolution, hyperparameters are used. We control the weights of the output distribution of each sub-branch and apply a constrained loss to it. The output feature will be obtained by superimposing the individual branches with varying scaling factors. .in, This indicates the number of child branches of the transformer.
[0053] During model training, random parameters were initialized, the cross-entropy loss function was used, and the SGD optimizer was employed. The initial learning rate was 0.001, and the learning rate was set to cosine decay with a minimum learning rate of 1e-6. After training on the training set for 50 epochs, the learning effect of the model was verified using the test set.
[0054] The above description is merely one example of a specific implementation of this method, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A dynamic gesture recognition method based on spatiotemporal interaction and rate perception, characterized in that, This method performs spatiotemporal decoupling feature extraction, visual rhythm feature extraction, spatiotemporal feature enhancement, and interaction enhancement on dynamic gesture videos, and classifies them using multi-scale features. The method includes the following steps: Step 1: Obtain the dynamic gesture dataset and perform data preprocessing; Step 2: Initially extract texture features through a series of 3D convolution operations to obtain a low-level semantic feature map; Step 3: Extract video rate features through the rate feature injection module, and overlay the rate features with the low-level semantic feature map; Step 4: Further extract multi-scale spatial features through a series of extended convolution operations; Step 5: After feature sampling of the feature map obtained in Step 4, feature extraction is performed using an improved Transformer network that performs interactive modeling of features in the temporal and spatial dimensions. The specific interactive modeling is implemented by the spatiotemporal feature injection interaction module, which consists of several spatial feature injection modules (SFIM) and temporal feature interaction modules (TFIM). The spatial feature injection module uses local detail features... As Key and Value, and global features As a query, CrossAttention is applied to obtain spatial features that enhance local detail awareness. : Where LN(·) represents the normalization layer, and CrossAttention represents the cross-attention layer. This represents the local detail features in the x-th SIFM module; The time feature interaction module starts from frame t and frame (t) 1) The shadow features extracted from the frame are and , and The bidirectional interaction is used for feature integration. The network adopts a twin structure in the time dimension, where feature maps from two adjacent frames are input simultaneously, and at frame t... In each TFIM, As a query, and As keys and values; in the In the feature extraction of frames, their roles are reversed; Step 6: Input the feature map obtained in Step 5 into a multilayer perceptron to obtain the classification result.
2. The dynamic gesture recognition method based on spatiotemporal interaction and rate perception as described in claim 1, characterized in that, The dynamic gesture dataset in step 1 is in the form of gesture videos, which can be composed of RGB or depth modalities. Data preprocessing methods include cropping, rotation, mixup, and shufflemix.
3. The dynamic gesture recognition method based on spatiotemporal interaction and rate perception as described in claim 1, characterized in that, The texture feature extraction in step 2 uses the first five layers of the I3D network to output the feature map dimension. .
4. The dynamic gesture recognition method based on spatiotemporal interaction and rate perception as described in claim 1, characterized in that, The rate feature extraction in step 3 includes the following steps: Step 3.1: Using the texture feature map obtained in Step 2 As input, image sampling and similarity calculation involve selecting a region at the same location in each frame of the image for similarity calculation. The computational complexity and sampling radius are also relevant. Related, in the input feature map dimension of In this case, the dimensions of the similarity calculation results are: For each frame, the closest and farthest frames are selected for similarity calculation; Step 3.2: Using the similarity feature map obtained in Step 3.1 as input, the lightweight MotionSqueeze method is used to estimate the optical flow of the correlation tensor between frame pairs. Then, the calculated optical flow and the corresponding confidence map are used to extract motion information. Step 3.3: Concatenate the dual-channel optical flow with the single-channel confidence map to form a pixel-level displacement map to further depict motion features. Estimate the longest and shortest scale motions and merge them to adapt to slow and fast downstream feature transformations, thus better understanding gesture movements. The final displacement map has the following dimensions: ; Step 3.4: Using the displacement map obtained in Step 3.3 as input, first transform the displacement map using depthwise separable convolution to align it with the original feature map. Then, use global average pooling to capture aggregated features, given a time period for the aggregated features. Where T represents the feature time dimension, rate attention can be learned without dimensionality reduction using the following formula: Where W is a general parameter matrix with T × T elements; Step 3.5: Superimpose the original shallow feature map with the weights learned through the rate attention mechanism to obtain enhanced features containing rate information, and output the feature map. .
5. The dynamic gesture recognition method based on spatiotemporal interaction and rate perception as described in claim 1, characterized in that, The multi-scale spatial feature extraction in step 4 is characterized by feature maps. It will extract multi-scale features at the k-th layer by using a series of spatially centered 3D Inception modules with convolution kernels of size 1 in the time dimension: in This represents the feature map of the nth layer. This represents an Inception module with a learnable parameter matrix W.