Multi-modal dynamic gesture recognition method based on deep learning

Abstract

The invention discloses a multi-modal dynamic gesture recognition method based on deep learning. RGB image and hand key point features are respectively extracted through a double-branch network: an RGB branch adopts ShuffleNetV2 to extract spatial features, and a time sequence feature is captured in combination with GRU; the key point branch is based on improved ST-GCN modeling space-time correlation. The two branch features are input into a classification network after channel / space fusion, a multi-scale feature fusion module is introduced to enhance the recognition precision, and shallow feature extraction is optimized to improve the small target detection effect. According to six dynamic gestures of left sliding, right sliding, up sliding, down sliding, grabbing and fist clenching, model parameters are compressed to be below 8MB on the premise of keeping precision through technologies of decomposition convolution, frame compression, channel rearrangement and combination and the like, the reasoning speed is increased, and the method is suitable for various embedded devices such as AR glasses and smart home or scenes with low computing power.

Description

Technical Field

[0001] The present invention relates to the field of gesture recognition, and in particular to a multimodal dynamic gesture recognition method based on deep learning. Background Art

[0002] Since 2014, dynamic gesture recognition has become a key research area in both academia and industry, with a variety of emerging technologies emerging. Early research focused on recognition methods based on static RGB images, but these methods were limited by their limited adaptability to directional changes, low recognition rates in partially occluded scenes, and difficulty in acquiring gesture depth and spatial dimensions, resulting in significant bottlenecks in recognition performance.

[0003] With technological advancements, dynamic gesture recognition methods based on RGB video sequences have gradually emerged. Building on this foundation, researchers have successively proposed recognition techniques based on different information carriers, such as skeleton modalities, depth data modalities, electromyography modalities, electroencephalography signal modalities, and audio signal modalities. However, single-modal data, due to the limited dimensions of gesture features contained, cannot meet the requirements for high-precision recognition in complex scenarios. To this end, multimodal fusion technology has emerged. By integrating multi-source heterogeneous data to achieve complementary and enhanced gesture features, it has become a mainstream research direction in the field of dynamic gesture recognition and has shown broad application prospects in various human-computer interaction scenarios, such as virtual reality interaction, sign language translation, smart healthcare, and somatosensory gaming. Compared with single-modal methods, multimodal fusion technology can significantly improve the recognition accuracy and generalization performance of the model due to its ability to comprehensively process multi-dimensional information.

[0004] However, some modalities (such as EEG and EMG) are limited by high signal acquisition complexity, strong sensitivity to environmental noise, and low device universality, making it difficult to meet actual deployment requirements. In view of this, the present invention selects RGB video frame images and hand key points, two widely available modalities with high acquisition stability and strong compatibility, to construct a dynamic gesture detection system. This system specifically addresses the key technical challenges of current multimodal dynamic gesture detection, namely, how to achieve efficient feature extraction and deep fusion.

[0005] In order to improve the feature extraction capability of the multimodal dynamic gesture network, increase the inference speed and reduce the number of parameters while ensuring detection accuracy, so that the model can be deployed on lightweight devices, the present invention improves the ST-GCN, ShuffleNetV2 and GRU networks. By introducing the AsFFV5 module and depthwise separable convolution and point convolution, the number of network layers and network structure are modified, and a multimodal dynamic gesture detection network based on deep learning is designed. This effectively improves the feature extraction capability of the model, greatly reduces the number of parameters, improves the detection accuracy and inference speed of the model, and greatly facilitates the deployment of the model on embedded devices. Summary of the Invention

[0006] The present invention proposes a multimodal dynamic gesture recognition method based on deep learning to solve the problems of low multimodal dynamic gesture detection accuracy, large number of model parameters, slow inference speed, and difficulty in deployment on embedded platforms.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A multimodal dynamic gesture recognition method based on deep learning, comprising the following steps:

[0009] Step 1: Construct a dataset of six dynamic gestures and expand and preprocess the dataset;

[0010] Step 2: Build a bimodal feature extraction network;

[0011] Wherein, the step 2 specifically includes the following steps:

[0012] Step 2.1: Build a dynamic gesture 2D hand key point data extraction network based on the improved ST-GCN network;

[0013] Step 2.2: Build a dynamic gesture RGB image data extraction network based on the improved ShuffleNetV2 and GRU network;

[0014] Step 3: Fuse the features extracted by the two feature extraction networks spatially and temporally respectively;

[0015] Step 4: training the multimodal dynamic gesture recognition model;

[0016] Step 5: Use the trained model to evaluate the performance of the dynamic gesture test set.

[0017] Preferably, step 1 specifically includes the following steps:

[0018] Step 1.1: Use public dynamic gesture datasets, including Chalearn IsoGD, HaGRID, and MSRAHand, as well as three methods: online search and self-filming to obtain video clips of six dynamic gestures;

[0019] Step 1.2: To enrich the original data, perform data enhancement on the original dataset, mainly including adding Gaussian noise, zooming in and out, rotating at random angles, and symmetric transformation.

[0020] Step 1.3: The enriched data is further preprocessed and labeled, including normalizing coordinates, building an adjacency matrix, and extracting 21 key points of the hand. For the 2D hand key point modality of dynamic gestures, the original video clips are first unified into 1.5-second videos. Then, openpose is used to extract the 21 key points of the hand from the video frame by frame. The dynamic gesture category is then labeled for a complete dynamic gesture video. The 21 key points of the hand include fingertips, middle joints, proximal joints, and wrists, where 0 is the wrist, and then each finger is numbered from the base to the fingertip. Key point 1 is the base of the thumb, 2 is the proximal joint of the thumb, 3 is the distal joint, 4 is the tip of the thumb, followed by 5 to 8 of the index finger, 9 to 12 of the middle finger, 13 to 16 of the ring finger, and 17 to 20 of the little finger, for a total of 21 points. There are six categories of dynamic gestures: up, down, left, right, grab, and fist.

[0021] Step 1.4: For the RGB video frame image data, unify the original video clips into 1.5-second videos, split each video into 16 frames, and normalize the pixel values. Then use LabelMe to label each frame with the dynamic gesture category.

[0022] Preferably, step 2.1 specifically includes the following steps:

[0023] Step 2.1.1: Decompose the standard convolution in the time dimension of ST-GCN into two operations: depthwise convolution and 1×1 point convolution. The depthwise convolution performs temporal convolution on each input channel independently to generate a feature map with the same number of input channels. The 1×1 point convolution fuses features across channels through 1×1 convolution and outputs the target number of channels.

[0024] Step 2.1.2: To further reduce memory usage and improve inference speed, use bilinear interpolation to compress the original frame rate from 64 to 16.

[0025] Preferably, step 2.2 specifically includes the following steps:

[0026] Step 2.2.1: To reduce the number of parameters and lightweight feature extraction model, adjust the number of layers of ShuffleNetV2. Specifically, reduce the number of repetitions of stage2, stage3, and stage4 of ShuffleNetV2 from 3x, 7x, and 3x to 1x, 5x, and 1x respectively;

[0027] Step 2.2.2: To reduce the resolution while retaining more details, the first max pooling in shufflenetv2 is changed to a 3×3 DW convolution plus a 1×1 point convolution;

[0028] Step 2.2.3: To reduce the amount of computation, at the expense of a small degree of accuracy, the channels of adjacent blocks are rearranged and merged into one.

[0029] Step 2.2.4: To enhance feature expression, add the AsFFv5 module after stage2, stage3, and stage4 of shufflenetv2 to perform multi-scale feature fusion. The fused features are then fed into CBR. The feature maps output by stage2, 3, and 4 are X2, X3, and X4, and the resolutions are aligned after upsampling:

[0030] α=σ(Wα*[X2,X3,X4])

[0031] β=σ(Wβ*[X2,X3,X4])

[0032] γ=σ(Wγ*[X2,X3,X4])

[0033] Where σ is the activation function, Wα, Wβ, and Wγ are 1×1 convolution kernels, and the formula for AsFFv5 weighted fusion is as follows:

[0034] Y=α⊙X2+β⊙X3+γ⊙X4

[0035] Preferably, step 3 specifically includes the following steps:

[0036] Step 3.1: Feature format alignment and dimensionality reduction. After the ST-GCN branch output features and the shufflenetv2 plus GRU branch output formats are unified into two dimensions using Reshape, the two branch channels are aligned using 1D convolution:

[0037] C min =min(C1,C2)

[0038] Step 3.2: Concat the two branch features along the channel dimension and use 1×1 convolution to fuse information across channels to avoid causing large kernel convolution parameters:

[0039] F 合并特征 =Concat(F 分支1 ,F 分支2 )

[0040] Step 3.3: In order to preserve the original features, add learnable weights to fuse different features:

[0041]

[0042] where μ, τ is the learnable weight.

[0043] Preferably, step 4 specifically includes the following steps:

[0044] Step 4.1: Split the pre-processed dynamic gesture dataset into two branches, and set the training set and test set of each branch in a 7:3 ratio.

[0045] Step 4.2: Initially, the learning rate of the branch network is set to 0.002, and the fusion layer is set to 0.01. The threshold of the GRU and fusion layer gradients is set to 0.5 to prevent the difference in gradient magnitude between branches. The two feature extraction branches use weighted cross entropy loss with an initial weight of 0.5. They are then learned independently. The cross entropy loss is also used for the feature fusion part.

[0046] Step 4.3: In order to constrain the distribution consistency of the two-branch features before fusion and improve the fusion effect, the cosine similarity loss is used to directly align the feature directions. In order to avoid excessive constraints that destroy branch independence, the weight of this loss function is fixed to 0.1;

[0047] Step 4.4: The total loss function is the linear sum of the above loss functions, and the training model is subsequently set according to the above parameters. 7. In the above step 5, the following steps are specifically included:

[0048] Step 5.1: Evaluate the final trained model using the test set and some unused data. The performance evaluation of the present invention includes three key indicators: precision (p), recall (R), and mean average precision (mAP). The specific formula is as follows:

[0049]

[0050] Among them, TP (True Positives) is the number of correctly predicted positive samples, FN (False Negatives) is the number of incorrectly predicted negative samples, and FP (False Positives) is the number of incorrectly predicted positive samples.

[0051] The beneficial effects of the present invention are as follows:

[0052] (1) The present invention simplifies the ST-GCN network, reduces the number of network layers, and thus reduces computational complexity. It uses a depthwise separable convolution module to replace traditional convolution, reducing computational redundancy. It also uses frame compression technology (64 frames → 16 frames) to reduce memory usage, thereby reducing the amount of computation and parameters while improving speed.

[0053] (2) The present invention reduces the amount of computation by reducing the number of layers and rearranging channels in the ShuffleNetV2+GRU branch, improves the inference speed, and adapts to the deployment requirements of mobile terminals. In addition, the AsFFv5 module is added in the middle of the network to perform multi-scale fusion, enhance feature expression, and use depthwise separable convolution and point convolution to replace the maximum pooling operation in the original ShuffleNetV2 network, thereby reducing the resolution while retaining more details.

[0054] (3) By combining channel alignment, splicing fusion and learnable weight residual enhancement with minimal parameter introduction, cross-modal feature complementarity is achieved, thereby maximizing the model detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The following will provide detailed description through specific implementation methods and drawings.

[0056] Figure 1 This is a schematic diagram of the overall structure of a multimodal dynamic gesture recognition system based on deep learning;

[0057] Figure 2 It is an improved shufflenetV2 network structure;

[0058] Figure 3 Mark the 21 key points of the hand with serial numbers;

[0059] Figure 4 The convolution operation process of the improved ST-GCN network in the time dimension;

[0060] Figure 5 This is a flow chart of the multimodal dynamic gesture recognition method based on deep learning proposed in this invention. DETAILED DESCRIPTION

[0061] In order to make the embodiments of the present invention and the prior art solutions clearer and easier to understand, the present invention is described in detail below with reference to the accompanying drawings.

[0062] Reference Figure 5 The multimodal dynamic gesture recognition method based on deep learning provided by the present invention comprises the following steps:

[0063] Step 1: Obtain datasets of 2D hand key points and RGB video frame images of six dynamic gestures and preprocess the initial data;

[0064] Step 1.1: Use public dynamic gesture datasets, including Chalearn IsoGD, HaGRID, and MSRAHand, as well as online search and self-filmed data to obtain video clips of six dynamic gestures. The public dynamic gesture datasets and online search data can be organized into a test set, and the self-filmed data can be annotated and organized into a training set.

[0065] Step 1.2: To enrich the original data, perform data enhancement on the original dataset, mainly including adding Gaussian noise, zooming in and out, rotating at random angles, and symmetric transformation.

[0066] Step 1.3: Further preprocess and annotate the enriched data. For the dynamic gesture 2D hand key point modality, first unify the original video clips into 1.5 seconds of video, and then use openpose to extract 21 key points of the hand from the video frame by frame. Figure 3 , including fingertips, middle joints, proximal joints and wrists, where 0 is the wrist, and then each finger is numbered from the base to the fingertip. Key point 1 is the base of the thumb, 2 is the proximal joint of the thumb, 3 is the distal joint, 4 is the tip of the thumb, followed by 5 to 8 of the index finger, 9 to 12 of the middle finger, 13 to 16 of the ring finger, and 17 to 20 of the little finger, for a total of 21 points. There are six categories of dynamic gestures, namely, swipe up, swipe down, swipe left, swipe right, grab, and clench a fist. Dynamic gestures are uniformly performed by the right hand;

[0067] Step 1.4: Normalize the coordinates of the processed data. First, calculate the coordinate system parameters of the video frame with an original resolution of W×H. Then, linearly map the coordinates to the interval [-1, 1]. The calculation formula is as follows:

[0068]

[0069] After normalizing the coordinates, a data file in the format of (N×C×T×V×M) is generated, where N is the number of videos, C is the joint feature (x, y, acc), x, y are the position coordinates of the node joint, acc is the confidence, T is the number of video segments, V is the number of joints, that is, 21, and M is the number of people in a frame, which is 1;

[0070] Step 1.5: For the RGB video frame image data, the original video clips are first unified into 1.5-second videos. 16 key frames in each video are extracted using OpenCV. The resolution of all images is adjusted to 224×224. The final data is saved as a (N×T×C×H×W) data file, where N is the batch size, T is the number of video frames, C is the number of input channels, and (H, W) is the image resolution.

[0071] Step 1.6: Organize the two modal datasets, hand keypoints and RGB video frame images, and store them separately. The hand keypoint data file is stored in the h5 format of N×16×3×21×1. Each sample is the time series data of the keypoints of a video clip. The label file contains six types of dynamic gesture labels. In addition, the adjacency matrix between the hand keypoints should be defined and saved as a .npy file in the dataset.

[0072] Step 1.7: The RGB video frame image data file is stored as a single video segment data in .npy format, with shape = (N × 16 × 3 × 224 × 224). The label file contains six types of dynamic gesture labels.

[0073] Step 1.8: For joint training of the two modal datasets, the two datasets share the same time step T = 16 and labels. During single batch training, the two modal data are loaded synchronously by indexing and input into the two-branch network.

[0074] Step 2: Construct a 2D hand skeleton key point feature extraction network;

[0075] Step 2.1: The original temporal convolution layer of ST-GCN is a standard 3D convolution, which processes both time and space dimensions. However, the number of parameters of 3D convolution is large, which is not conducive to deployment on embedded platforms. Therefore, the standard convolution in the time dimension of ST-GCN is decomposed into two steps: depthwise convolution and 1×1 point convolution. The convolution kernel of the depthwise convolution is 3×3 in size, and with the ReLU activation function, each input channel is independently convolved in time to generate features of shape (N, C_in, T, V, M). The 1×1 point convolution fuses features across channels through 1×1 convolution, and the output shape is (N, C_out, T, V, M).

[0076] Step 2.2: Assuming that the number of parameters of the original temporal convolution is C_in × C_out × convolution kernel size, then the number of parameters of the decomposed module is C_in × convolution kernel size + C_out × C_in, which will be greatly reduced;

[0077] Step 2.3: Adjust the learning rate of the improved ST-GCN branch network to 0.003;

[0078] Step 2.4: To further reduce memory usage and improve inference speed, use bilinear interpolation to compress the original frame rate from 64 to 16.

[0079] Step 2.4 includes the following steps:

[0080] Step 2.4.1: Map the original 64-frame time axis to the [0,1] interval. The time position of the t-th frame is:

[0081]

[0082] The time positions of the target 16 frames are evenly distributed in [0,1], and the time position of the t'th frame is:

[0083]

[0084] Perform interpolation calculation for each target time point t';

[0085] Step 2.4.2: Determine the adjacent original frames, find the two closest frame indices t_1 and t_2 (t_1≤t_2) on the original time axis, and calculate the weight w to represent the proximity between the target time point and t_1:

[0086]

[0087] Step 2.4.3: Interpolate the coordinates and confidence of each keypoint V:

[0088] X(t′,V)=(1-w)X(t1,V)+wX(t2,V)

[0089] Where X(t,V) represents the data (x, y, confidence) of the original t-th frame and the V-th key point;

[0090] Step 2.4.4: Interpolate all target time points t' and key points V to generate 16 compressed frames of data. The output format remains (N, 16, 3, 21, 1). After interpolation, the amount of data is greatly reduced, and the inference speed is improved.

[0091] Step 3: Build a dynamic gesture RGB video frame image feature extraction network;

[0092] Step 3.1: To reduce the number of parameters and lightweight feature extraction model, adjust the number of layers of ShuffleNetV2. Specifically, reduce the number of repetitions of stage2, stage3, and stage4 of ShuffleNetV2 from 3x, 7x, and 3x to 1x, 5x, and 1x respectively.

[0093] Step 3.2: To reduce the resolution while retaining more details, the first maximum pooling in ShuffleNetV2 is changed to a 3×3 DW convolution plus a 1×1 point convolution;

[0094] Step 3.3: To reduce the amount of computation, at the expense of a small degree of accuracy, the channels of adjacent blocks are rearranged and merged into one;

[0095] Step 3.4: To enhance feature expression, add the AsFFv5 module after stage2, stage3, and stage4 of shufflenetv2 to perform multi-scale feature fusion. The fused features are then fed into CBR. The feature maps output by stage2, 3, and 4 are X2, X3, and X4, and the resolutions are aligned after upsampling:

[0096] α=σ(Wα*[X2,X3,X4])

[0097] β=σ(Wβ*[X2,X3,X4])

[0098] γ=σ(Wγ*[X2,X3,X4])

[0099] Where σ is the activation function, Wα, Wβ, and Wγ are 1×1 convolution kernels, and the formula for AsFFv5 weighted fusion is as follows:

[0100] Y=α⊙X2+β⊙X3+γ⊙X4

[0101] Step 4: Construct feature fusion network;

[0102] Step 4.1: Feature format alignment and dimensionality reduction. After the ST-GCN branch output features and the shufflenetv2 plus GRU branch output formats are unified into two dimensions using Reshape, the two branch channels are aligned using 1D convolution:

[0103] C min =min(C1,C2)

[0104] Step 4.2: Concat the two branch features along the channel dimension and use 1×1 convolution to fuse information across channels to avoid causing large kernel convolution parameters:

[0105] F 合并特征 =Concat(F 分支1 ,F 分支2 )

[0106] Step 4.3: In order to preserve the original features, add learnable weights to fuse different features:

[0107]

[0108] where μ, τ is the learnable weight.

[0109] Step 5: Set initial parameters and loss function;

[0110] Step 5.1: Initially, the learning rate of the branch network is set to 0.002, and the fusion layer is set to 0.01. The threshold of the GRU and fusion layer gradients is set to 0.5 to prevent the difference in gradient magnitude between branches. The two feature extraction branches use weighted cross entropy loss with an initial weight of 0.5. They are then learned independently. The cross entropy loss is also used for the feature fusion part.

[0111] Step 5.2: In order to constrain the distribution consistency of the two-branch features before fusion and improve the fusion effect, the cosine similarity loss is used to directly align the feature directions. In order to avoid excessive constraints that destroy branch independence, the weight of this loss function is fixed to 0.1;

[0112] Step 5.3: Perform training according to the set parameters and calculate the total loss function to optimize the model parameters to obtain the trained multimodal dynamic gesture recognition model;

[0113] The total loss function consists of three parts: the weighted cross entropy loss of the two branches, the cross entropy loss of the feature fusion part, and the cosine similarity loss that constrains feature consistency.

[0114] In order to supervise the output of the two branches of ST-GCN and shufflenetv2+GRU, a weighted cross entropy loss is set. The loss formula is as follows:

[0115]

[0116] where y c is the one-hot encoding of the true label, p c is the predicted probability of the branch output after softmax, w i is the branch loss weight, with the initial value w1=w2=0.5, which is automatically updated by the optimizer;

[0117] The cross entropy loss of the feature fusion part supervises the final output after fusion, and its fusion loss function formula is as follows:

[0118]

[0119] where p 融合，c The predicted probability of the fused output after softmax;

[0120] In order to force the features of the two branches to be aligned in spatial distribution before feature fusion and improve the fusion effect, the cosine similarity loss is set to perform the feature map of the two branches (F 分支1 、F 分支2 ) Calculate the cosine similarity, the formula is as follows:

[0121]

[0122] Where · represents the vector inner product, ||·|| is the L2 norm, and the weight is fixed at 0.1 and does not participate in learning;

[0123] The total loss function is:

[0124] L 总损失 =w1L 分支1 +w2L 分支2 +L 融合 +0.1L 余弦相似度

[0125] Step 6: Evaluate the final trained model using the test set and some unused data. The performance evaluation of the present invention includes three key indicators: precision (p), recall (R), and mean average precision (mAP). The specific formula is as follows:

[0126]

[0127] Among them, TP (True Positives) is the number of correctly predicted positive samples, FN (False Negatives) is the number of incorrectly predicted negative samples, and FP (False Positives) is the number of incorrectly predicted positive samples.

[0128] The deep learning-based dynamic gesture detection method provided by the present invention optimizes the ST-GCN and ShuffleNetV2+GRU networks, which not only enhances the feature extraction capability of the model, but also greatly reduces the number of parameters while ensuring detection accuracy, improves the inference speed, and improves the stability of the model in dynamic gesture recognition. This allows the model to be deployed on lightweight devices, expands the scope of use of dynamic gestures, and is particularly suitable for recognizing some fast gestures.

[0129] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, can make equivalent replacements or changes based on the technical solution and inventive concept of the present invention, which should be within the scope of protection of the present invention.

Claims

1. A multimodal dynamic gesture recognition method based on deep learning, characterized in that: The method comprises: Step 1: Construct a dataset of six dynamic gestures and expand and preprocess the dataset; Step 2: Build a bimodal feature extraction network; Wherein, the step 2 specifically includes the following steps: Step 2.1: Build a dynamic gesture 2D hand key point data extraction network based on the improved ST-GCN network; Step 2.2: Build a dynamic gesture RGB image data extraction network based on the improved ShuffleNetV2 and GRU network; Step 3: Fuse the features extracted by the two feature extraction networks spatially and temporally respectively; Step 4: training the multimodal dynamic gesture recognition model; Step 5: Use the trained model to evaluate the performance of the dynamic gesture test set.

2. The method according to claim 1, wherein: Step 1 specifically includes the following steps: Step 1.1: Use public dynamic gesture datasets, including Chalearn IsoGD, HaGRID, and MSRA Hand, as well as three methods: online search and self-filming to obtain video clips of six dynamic gestures; Step 1.2: To enrich the original data, perform data augmentation on the original dataset, including adding Gaussian noise, zooming in and out, rotating at random angles, and performing symmetric transformation operations; Step 1.3: Further preprocess and label the enriched data, including normalizing coordinates, building an adjacency matrix, and extracting 21 key points of the hand. For the 2D hand key point modality of dynamic gestures, first unify the original video clips into 1.5-second videos, then use openpose to extract the 21 key points of the hand from the video frame by frame, and then label the dynamic gesture category for a complete dynamic gesture video. The 21 key points of the hand include fingertips, middle joints, proximal joints and wrists, where 0 is the wrist, and then each finger is numbered from the base to the fingertip. Key point 1 is the base of the thumb, 2 is the proximal joint of the thumb, 3 is the distal joint, 4 is the tip of the thumb, followed by 5 to 8 of the index finger, 9 to 12 of the middle finger, 13 to 16 of the ring finger, and 17 to 20 of the little finger, for a total of 21 points. There are six types of dynamic gestures: swipe up, swipe down, swipe left, swipe right, grab, and clench a fist. Step 1.4: For RGB video frame image data, first unify the original video clips into 1.5-second videos, cut each video into 16 frames, and normalize the pixel values. Then use labelme to annotate each frame with the dynamic gesture category.

3. The method according to claim 1, wherein: Step 2.1 specifically includes the following steps: Step 2.1.1: Decompose the standard convolution in the time dimension of ST-GCN into two operations: depthwise convolution and 1×1 point convolution. The depthwise convolution performs temporal convolution on each input channel independently to generate a feature map with the same number of input channels. The 1×1 point convolution fuses features across channels through 1×1 convolution and outputs the target number of channels. Step 2.1.2: To further reduce memory usage and improve inference speed, use bilinear interpolation to compress the original frame rate from 64 to 16.

4. The method according to claim 1, wherein: Step 2.2 specifically includes the following steps: Step 2.2.1: To reduce the number of parameters and lightweight feature extraction model, adjust the number of layers of ShuffleNetV2. Specifically, reduce the number of repetitions of stage2, stage3, and stage4 of ShuffleNetV2 from 3x, 7x, and 3x to 1x, 5x, and 1x respectively; Step 2.2.2: To reduce the resolution while retaining more details, the first max pooling in shufflenetv2 is changed to a 3×3 DW convolution plus a 1×1 point convolution; Step 2.2.3: To reduce the amount of computation, at the expense of a small degree of accuracy, the channels of adjacent blocks are rearranged and merged into one. Step 2.2.4: To enhance feature expression, add the AsFFv5 module after stage2, stage3, and stage4 of shufflenetv2 to perform multi-scale feature fusion. The fused features are then fed into CBR. The feature maps output by stage2, 3, and 4 are X2, X3, and X4, and the resolutions are aligned after upsampling: α=σ(Wα*[X2,X3,X4]) β=σ(Wβ*[X2,X3,X4]) γ=σ(Wγ*[X2,X3,X4]) Where σ is the activation function, Wα, Wβ, and Wγ are 1×1 convolution kernels, and the formula for AsFFv5 weighted fusion is as follows: Y=α⊙X2+β⊙X3+γ⊙X4.

5. The method according to claim 1, wherein: Step 3 specifically includes the following steps: Step 3.1: Feature format alignment and dimensionality reduction. After the ST-GCN branch output features and the shufflenetv2 plus GRU branch output formats are unified into two dimensions using Reshape, the two branch channels are aligned using 1D convolution: C min =min(C1,C2) Step 3.2: Concat the two branch features along the channel dimension and use 1×1 convolution to fuse information across channels to avoid causing large kernel convolution parameters: F 合并特征 =Concat(F 分支1 ,F 分支2 ) Step 3.3: In order to preserve the original features, add learnable weights to fuse different features: Among them, μ, τ is the learnable weight.

6. The method according to claim 1, wherein: Step 4 specifically includes the following steps: Step 4.1: Split the pre-processed dynamic gesture dataset into two branches, and set the training set and test set of each branch according to a 7:3 ratio; Step 4.2: Initially, the learning rate of the branch network is set to 0.002, and the fusion layer is set to 0.

01. The threshold of the GRU and fusion layer gradients is set to 0.5 to prevent the difference in gradient magnitude between branches. The two feature extraction branches use weighted cross entropy loss with an initial weight of 0.

5. They are then learned independently. The cross entropy loss is also used for the feature fusion part. Step 4.3: In order to constrain the distribution consistency of the two-branch features before fusion and improve the fusion effect, the cosine similarity loss is used to directly align the feature directions. In order to avoid excessive constraints that destroy branch independence, the weight of this loss function is fixed to 0.1; Step 4.4: The total loss function is the linear sum of the above loss functions. The model is then trained according to the initial settings of the above parameters.

Images