Dynamic gesture recognition method based on space-time interaction and rate perception

Abstract

The invention discloses a dynamic gesture recognition method based on space-time interaction and rate perception. The method comprises a space-time decoupling feature extraction module, a rate feature injection module and a space-time feature injection interaction module. The space-time decoupling feature extraction is formed by three-dimensional convolution, an Inception module and a multi-scale Transform, and the rate feature injection module realizes visual rhythm feature extraction based on similarity calculation and an optical flow estimation algorithm, performs feature enhancement on an initial feature by using the visual rhythm feature, and performs feature extraction on the initial feature. The spatial-temporal feature injection interaction module is composed of a spatial feature injection module and a time feature interaction module, and the spatial feature injection module and the time feature interaction module are realized based on an attention mechanism. The space-time decoupling feature extraction is a backbone network, and other modules are embedded in the backbone network and play roles of feature injection, feature enhancement and the like. The method relates to the fields of man-machine interaction, dynamic gesture recognition algorithms, deep learning, computer vision and the like, and has the advantages of comprehensive features and accurate recognition.

Description

Technical Field

[0001] The present invention relates to the fields of human-computer interaction, dynamic gesture recognition, deep learning, computer vision, etc.; in particular, it relates to image processing technologies such as convolutional neural networks, Transformer, and attention mechanisms. Background Art

[0002] Gestures are produced as part of intentional movements and signs, involving the movement of the upper body, especially the arms, hands and fingers. Video-based classification is an important part of gesture recognition. It has been applied to many human-centric tasks, such as apparent personality analysis, sign language recognition and human-computer interaction. Compared with static gestures, dynamic gestures can express more information. There are different methods for dynamic gesture recognition, such as wearable sensor-based, hand skeleton key point extraction-based and vision-based. Considering the implementation cost of the algorithm and the portability of the interaction, vision-based dynamic gesture recognition is a very challenging and worthwhile research topic.

[0003] Visual gesture recognition can be roughly divided into three categories: First, methods based on convolutional neural networks. A large number of studies in the field of computer vision in recent years have shown that convolution is the most suitable tool for extracting image features. Therefore, convolutional neural networks are also widely used in the field of dynamic gesture recognition. At the same time, different gestures in dynamic gestures may reflect the same image-level features. At this time, the features of the time dimension will be an effective basis for distinguishing them, so the 3D convolution method is usually better than the 2D convolution method. Second, methods based on LSTM. Since dynamic gesture recognition is very sensitive to the features of the time dimension, another method has been born. First, the image-level features of the video are extracted by convolution, and then the feature map is input into the LSTM network to obtain relevant time series information. This approach is logically very feasible, and the model parameters are much smaller than the first method. However, in actual experiments, the performance of this method is suboptimal. Third, methods based on transformers. The transformer architecture was proposed for natural language processing tasks and has become a de facto standard in the field. Recent studies have shown that the application of transformer methods in the field of vision also has high research value. In the field of dynamic gesture recognition, transformer-based methods are a hot topic in recent years. Although a large number of excellent research results have emerged in the field, these results also have their limitations. The existing results focus on spatiotemporal modeling. Although spatiotemporal modeling is a key factor in gesture recognition performance, dynamic gestures not only have the characteristics of spatiotemporal dimensions, but also have gesture visual tempo. Gesture visual tempo characterizes the dynamics and time scale of the action, which helps to distinguish gestures with high similarity in visual dynamics and appearance, but current research does not take into account the visual rhythm level.

[0004] Therefore, it is necessary to design a gesture recognition method that introduces visual rhythm modeling on the basis of spatiotemporal modeling to fill the research gap in the field. Secondly, with the development of technology, the application of transformers in the field of computer vision is becoming more and more extensive. Yes, when processing image tasks, the Transformer model does tend to capture global features, while its ability to capture local features is relatively weak. This is mainly due to the design characteristics of its self-attention mechanism:

[0005] The self-attention mechanism of Transformer can directly establish global element associations by calculating the attention weights between each element in the input sequence and other elements. This makes Transformer particularly good at capturing long-distance dependencies and global contextual information. However, this global operation may be insufficient in capturing the details or local features of an image. In image processing tasks, convolutional neural networks (CNNs) can capture local features and spatial information well by sliding convolution kernels in local areas. The standard Transformer does not have this local perception mechanism. In the field of dynamic gesture recognition, capturing fine-grained changes in hand posture is a reliable guarantee for recognition accuracy.

[0006] Therefore, it is necessary to design a spatiotemporal feature injection interaction mechanism to enhance the transformer's attention to local texture detail features. Summary of the invention

[0007] In view of the defects existing in the prior art, the purpose of the present invention is to achieve the above purpose, and the technical solution adopted by the present invention is: Figure 1 As shown in the figure, a 3D convolutional neural network is used to extract features in the spatial dimension, and transformer is used to process these features to obtain the temporal relationship therein. At the same time, gesture visual rhythm modeling is introduced to obtain the rate characteristics of dynamic gestures.

[0008] On the basis of the above technical scheme, this method embeds the spatial feature injection module (SFIM) and the temporal feature interaction module (TFIM) in the Transformer to achieve the purpose of refining the spatial small-scale local features, enhancing the timing information between adjacent frames, and capturing the continuity of the video time dimension. Compared with the prior art, the advantages of the present invention are: the feature extraction mechanism is complete, multi-dimensional and multi-scale feature extraction is considered, and the feature enhancement mechanism is more perfect than other methods. Experiments have proved that this method has achieved competitive classification accuracy on the jester dataset (RGB mode) and DHG28 (depth mode).

[0009] Furthermore, the technical solution of the present invention is as follows:

[0010] Step 1: Obtain dynamic gesture dataset and perform data preprocessing;

[0011] Step 2: Preliminary extraction of texture features through a series of 3D convolution operations to obtain a low-level semantic feature map;

[0012] Step 3: extracting the video rate feature through the rate feature injection module, and superimposing the rate feature with the low-level semantic feature map;

[0013] Step 4: Further extract multi-scale spatial features through a series of dilated convolution operations;

[0014] Step 5: After feature sampling, the feature map obtained in step 4 is passed through the Transformer network integrated with the STFIM module for feature extraction.

[0015] Step 6: Input the feature map obtained in step 5 into a multi-layer perceptron to obtain the classification result.

[0016] Furthermore, the dynamic gesture dataset in step 1 is in the form of gesture video, which can be composed of RGB or depth modality, and the data preprocessing means include cropping, rotation, ShuffleMix and other methods. The relevant operations can enrich the data and give full play to the expression ability of the model, especially on smaller datasets, which can fully reduce the overfitting problem.

[0017] Furthermore, the texture feature extraction in step 2 uses the first five layers of the I3D network to output the feature map dimension X1∈R T×C×H×W .

[0018] The rate feature extraction in step 3 includes the following steps:

[0019] Step 3.1: Take the texture feature map X1∈R obtained in step 2 T×C×H×W As input, image sampling and similarity calculation, that is, taking an area at the same position in each frame of the image for similarity calculation. The greater the similarity between frames, the lower the visual rate. The amount of calculation is related to the sampling radius R. When the input feature map dimension is C×T×H×W, the result dimension of the similarity calculation is H×W×R 2 , for each frame, the nearest and farthest frames are selected to calculate the similarity. The frame sampling method used in this method avoids the excessive amount of calculation caused by pyramid sampling and the high data dimension leading to feature maps;

[0020] Step 3.2: Using the similarity feature map obtained in step 3.1 as input, perform optical flow estimation on the correlation tensor between the frame pairs, and then use the calculated optical flow and the corresponding confidence map to extract motion information;

[0021] Step 3.3: Connect the dual-channel optical flow with the single-channel confidence map to form a pixel-level displacement map to further describe the motion features, estimate the longest and shortest scale motions, and merge them to adapt to slow and fast-paced downstream feature conversions for better understanding of gestures. The final displacement map has a dimension of 2×T×H×W×3;

[0022] Step 3.4: Taking the displacement map obtained in step 3.3 as input, first transform the displacement map through depthwise separable convolution to align it with the original feature map, and then use global average pooling to capture the aggregated features. Given the time aggregated feature F∈R T , where T represents the feature time dimension, the rate attention can be learned without dimensionality reduction according to the following formula:

[0023] ω=σ(WF0

[0024] where W is a general parameter matrix with T×T elements;

[0025] Step 3.5: Superimpose the original input shallow feature map with the weights learned through the rate attention mechanism to obtain enhanced features containing rate information, and output the feature map X2∈R T×C×H×W .

[0026] Furthermore, the multi-scale spatial feature extraction in step 4 is characterized in that the feature map X2 = {I1, I2…I t} will pass through a series of space-centric 3D Inception modules whose convolution kernel size in the time dimension is 1, which extracts multi-scale features at layer k in the following way:

[0027] X k =maxpool(Inception(X k-1 ,W))

[0028] Where X n represents the feature map of the nth layer, and Inception(·,W) represents the Inception module with a learnable parameter matrix W.

[0029] Furthermore, the spatiotemporal feature injection interaction module in step 5 is composed of a plurality of spatial feature injection modules (SFIM) and a temporal feature interaction module (TFIM), wherein the spatial feature injection module is based on local detail features. (xth SFIM module) as Key and Value, and the global feature As a query, perform CrossAttention to obtain spatial features that enhance local detail attention

[0030]

[0031] Where LN(·) represents layer normalization. CrossAttention represents the cross attention layer. This design aims to enhance the transformer structure's perception of important local features. This is because in the dynamic gesture recognition task, the importance of each local detail of the entire image to the classification task is different, and the transformer has a weak perception of local features. In the field of dynamic gesture recognition, capturing the fine-grained changes in hand posture is a reliable guarantee for recognition accuracy. Therefore, the spatial feature injection module uses the form of cross attention to integrate local features into global features.

[0032] Furthermore, the shadow features extracted by the time feature interaction module from the tth frame and the (t-1)th frame are: and and The bidirectional interaction is used for feature integration. The network adopts a twin structure in the time dimension. The feature maps of two adjacent frames are input simultaneously. In the xth TFIM of the tth frame, As a query, As the key and value; in the feature extraction of the t-1th frame, their roles are reversed. The design of this mechanism focuses on the feature interaction between adjacent frames. Because in the field of dynamic gesture recognition, the temporal relationship between frame actions is an important basis for classification tasks, so the interaction between adjacent frames in the form of a cross-attention mechanism helps the model capture complete spatiotemporal information and make the most accurate classification.

[0033] Furthermore, the multilayer perceptron in step six is ​​composed of a series of fully connected layers. The multilayer perceptron is a commonly used classification method in classification tasks in the field of deep learning, and classifies the obtained features containing multi-dimensional and multi-scale semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. The drawings described below are merely abstract process structures of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 It is a schematic diagram of the backbone network structure;

[0036] Figure 2 for Figure 1 Schematic diagram of the RFIM rate feature injection module structure;

[0037] Figure 3 for Figure 1 Schematic diagram of the Transformer-STFIM structure in;

[0038] Figure 4 for Figure 3 Schematic diagram of the spatial feature injection module (SFIM) structure in;

[0039] Figure 5 for Figure 3 Schematic diagram of the temporal feature interaction module (TFIM) structure in . DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] For step 1, the Jester public dataset is used to crop the video frame to a size of 224*224 pixels, and the ShuffleMix image enhancement operation is performed. The specific parameters are set to ShuffleMix value of 0.2 and Mixup value of 0.8.

[0042] For step 2, the first five layers of the I3D network are used to extract shallow texture features of the model.

[0043] For step 3, follow these steps:

[0044] Step 3.1: Take the texture feature map X1∈R obtained in step 2 T×C×H×W As input, image sampling and similarity calculation, that is, taking an area at the same position in each frame image for similarity calculation, the amount of calculation is related to the sampling radius R. When the input feature map dimension is C×T×H×W, the result dimension of the similarity calculation is H×W×R 2 , for each frame, select the nearest and farthest frames to calculate the similarity. The similarity calculation can use dot product or SSIM, PSNR and other technologies. This method uses SSIM similarity to characterize the rate: S = SSIM(f1,f2) = [l(f1,f2)] · [c(f1,f2)] · [s(f1,f2)] where,

[0045] Step 3.2: Using the similarity feature map obtained in step 3.1 as input, a lightweight MotionSqueeze method can be used to estimate the optical flow of the correlation tensor between the frame pairs, and then the motion information is extracted using the calculated optical flow and the corresponding confidence map;

[0046] Step 3.3: Connect the dual-channel optical flow with the single-channel confidence map to form a pixel-level displacement map to further describe the motion features, estimate the longest and shortest scale motions, and merge them to adapt to slow and fast-paced downstream feature conversions for better understanding of gestures. The final displacement map has a dimension of 2×T×H×W×3;

[0047] Step 3.4: Taking the displacement map obtained in step 3.3 as input, first transform the displacement map through depthwise separable convolution to align it with the original feature map, and then use global average pooling to capture the aggregated features. Given the time aggregated feature F∈R T , where T represents the feature time dimension, the rate attention can be learned without dimensionality reduction according to the following formula: ω = σ(WF), where W is a general parameter matrix with T×T elements.

[0048] For step 4, the feature map X2 = {I1, I2…I t} will pass through a series of space-centric 3D Inception modules, whose convolution kernel size in the time dimension is 1, which extracts multi-scale features at the kth layer in the following way: X k =maxpool(Inception(X k-1 ,W)) where X n represents the feature map of the nth layer, and Inception(·,W) represents the Inception module with a learnable parameter matrix W.

[0049] For step 5, multi-scale feature sampling is first performed in the following way: Rand(m,n) represents a random integer between m and n. After obtaining the downsampled feature map Y1, we pass it through a 3D Inception module with a convolution kernel of size 1×1 in the spatial dimension and a maximum pooling layer to capture local fine-grained features, Y2 = maxpool(Inception(Y1,W)). Inside the Transformer module, four spatial feature injection modules (SFIM) and temporal feature interaction modules (TFIM) are used. They are stacked in a linear series manner. The spatial feature injection module is based on local detail features. (xth SFIM module) as Key and Value, and the global feature As a query, perform CrossAttention to obtain spatial features that enhance local detail attention Where N(·) represents layer normalization. CrossAttention represents the cross attention layer.

[0050] For step 6, add three fully connected layers and set the output dimension of the last layer to the number of gesture categories. The number of categories in the Jester dataset is 27.

[0051] The formulas for step five and step six are expressed as follows: n =MLP(LN(MSA knn (Y n-1 )))+Y n-1 , where MLP represents a multi-layer perceptron for feature mapping, LN represents a normalization layer, and MSA knn Represents a k-NN multi-head self-attention layer. At the same time, in order to avoid overfitting of a transformer branch at a certain time and resolution, a hyperparameter ξ is used to control the weight of the output distribution of each sub-branch and impose a constraint loss on it. The output feature will be the result of superimposing each branch with a varying scale factor. Among them, K represents the number of transformer sub-branches.

[0052] During the model training process, random parameter initialization was selected, the cross entropy loss function was adopted, and the SGD optimizer was used. The initial learning rate was 0.001, and the learning rate cosine decay was set. The minimum learning rate was 1e-6. After 50 cycles of training on the training set, the test set was used to verify the learning effect of the model.

[0053] The above is only an example of the specific implementation of the method, so that those skilled in the art can understand or implement the application. It will be apparent to those skilled in the art that various modifications to these embodiments are possible, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

Claims

1. A dynamic gesture recognition method based on spatiotemporal interaction and rate perception, characterized in that: The method extracts spatiotemporal decoupling features, extracts visual rhythm features, and enhances and interacts spatiotemporal features for dynamic gesture videos, and classifies them using multi-scale features. The method includes the following steps: Step 1: Obtain dynamic gesture dataset and perform data preprocessing; Step 2: Preliminary extraction of texture features through a series of 3D convolution operations to obtain a low-level semantic feature map; Step 3: extracting the video rate feature through the rate feature injection module, and superimposing the rate feature with the low-level semantic feature map; Step 4: Further extract multi-scale spatial features through a series of dilated convolution operations; Step 5: After feature sampling, the feature map obtained in step 4 is passed through the Transformer network integrated with the STFIM module for feature extraction; Step 6: Input the feature map obtained in step 5 into a multi-layer perceptron to obtain the classification result.

2. A dynamic gesture recognition method based on spatiotemporal interaction and rate perception as claimed in claim 1, characterized in that: The dynamic gesture dataset in step 1 is in the form of gesture video, which can be composed of RGB or depth modality. The data preprocessing means include cropping, rotation, ShuffleMix and other methods.

3. A dynamic gesture recognition method based on spatiotemporal interaction and rate perception as claimed in claim 1, characterized in that: The texture feature extraction in step 2 uses the first five layers of the I3D network to output the feature map dimension X1∈R T × C × H × W .

4. A dynamic gesture recognition method based on spatiotemporal interaction and rate perception as claimed in claim 1, characterized in that: The rate feature extraction in step 3 includes the following steps: Step 3.1: Take the texture feature map X1∈R obtained in step 2 T × C × H × W As input, image sampling and similarity calculation, that is, taking an area at the same position in each frame image for similarity calculation, the amount of calculation is related to the sampling radius R. When the input feature map dimension is C×T×H×W, the result dimension of the similarity calculation is H×W×R2. For each frame, the nearest and farthest frames are selected for similarity calculation; Step 3.2: Using the similarity feature map obtained in step 3.1 as input, a lightweight MotionSqueeze method is used to perform optical flow estimation on the correlation tensor between the frame pairs, and then the motion information is extracted using the calculated optical flow and the corresponding confidence map; Step 3.3: Connect the dual-channel optical flow with the single-channel confidence map to form a pixel-level displacement map to further describe the motion features, estimate the longest and shortest scale motions, and merge them to adapt to slow and fast-paced downstream feature conversions for better understanding of gestures. The final displacement map has a dimension of 2×T×H×W×3; Step 3.4: Taking the displacement map obtained in step 3.3 as input, first transform the displacement map through depthwise separable convolution to align it with the original feature map, and then use global average pooling to capture the aggregated features. Given the temporal aggregated feature F∈R T , where T represents the feature time dimension, the rate attention can be learned without dimensionality reduction according to the following formula: ω=σ(WF) where W is a general parameter matrix with T×T elements; Step 3.5: Superimpose the original input shallow feature map with the weights learned through the rate attention mechanism to obtain enhanced features containing rate information, and output the feature map X2∈R T × C × H × W .

5. The method for dynamic gesture recognition based on spatiotemporal interaction and rate perception as claimed in claim 1, characterized in that: The multi-scale spatial feature extraction in step 4 is characterized in that the feature map X2 = {I1, I2...It} will pass through a series of space-centric 3D Inception modules, whose convolution kernel size in the time dimension is 1, which extracts multi-scale features at the kth layer in the following way: Xk=maxpool(Inception(Xk-1,W)) Where Xn represents the feature map of the nth layer, and Inception(·,W) represents the Inception module with a learnable parameter matrix W.

6. A dynamic gesture recognition method based on spatiotemporal interaction and rate perception as claimed in claim 1, characterized in that: The spatiotemporal feature injection interaction module in step 5 is composed of several spatial feature injection modules (SFIM) and temporal feature interaction modules (TFIM). (xth SFIM module) as Key and Value, and the global feature As a query, perform CrossAttention to obtain spatial features that enhance local detail attention Where LN(·) represents the normalization layer, and CrossAttention represents the cross attention layer; The shadow features extracted by the temporal feature interaction module from the tth frame and the (t-1th frame are: and and The bidirectional interaction is used for feature integration. The network adopts a twin structure in the time dimension. The feature maps of two adjacent frames are input simultaneously. In the xth TFIM of the tth frame, As a query, as key and value; in the feature extraction of the t-1th frame, their roles are reversed.

Images