A method for constructing a lightweight dynamic gesture recognition neural network model

Abstract

The application relates to the fields of artificial intelligence research and gesture recognition, and relates to a dynamic gesture recognition neural network model with light weight design. The most important feature components are an attention mechanism module and a light weight feature extraction LW-ConvNeXt module. The attention mechanism module mainly comprises a motion attention mechanism Diff module and a channel attention mechanism ECA module; the light weight feature extraction LW-ConvNeXt module constitutes a basic module of the neural network and is a new module obtained by light weight design on a feature extraction ConvNeXt module.

Description

TECHNICAL FIELD

[0001] The application relates to the field of artificial intelligence and gesture recognition, and particularly relates to a construction method of a light-weight dynamic gesture recognition neural network model. BACKGROUND

[0002] With the development of artificial intelligence and deep learning, and the rise of digital twins in recent years, gesture recognition and human-computer interaction have become a research hotspot. At present, most digital twin systems use intelligent glasses, handles and other interactive devices for interaction. Vision-based gesture interaction can free users from the shackles of input devices, and has the advantages of low attachment, weak invasiveness and more natural interaction experience. Gesture recognition can be divided into static gesture recognition and dynamic gesture recognition. The field of static gesture recognition is relatively mature, while dynamic gesture recognition has great development space due to its larger parameter quantity, more complex structure and higher dimension. SUMMARY

[0003] In order to solve the deficiencies of the existing model, a construction method of a dynamic gesture recognition model with high precision, light weight and real-time requirement is provided.

[0004] In order to achieve the above purpose, the application provides a construction method of a gesture recognition neural network model based on an improved attention mechanism and a ConvNeXt module, and the specific design scheme is as follows:

[0005] The specific structure of the model is as follows: the input is a video to be recognized, after preprocessing, the input is input into a down-sampling Stem module, then the whole structure is divided into four stages, each stage has a down-sampling module and a plurality of light-weight feature extraction modules LW-ConvNeXt modules; the attention mechanism module is interlaced between each stage; finally, gesture recognition is completed through global average pooling and a fully connected layer.

[0006] The preprocessing process of the input data is as follows: due to the high similarity between adjacent frames, there are a large number of redundant features, so three-frame difference method is used to process the video, and frames higher than the similarity threshold are deleted, so that the dynamic features between frames are more obvious and the image is clearer; the processed continuous frames are divided into n segments, m frames are taken from each segment, and 2 frames are added at the beginning and the end to form a group of input data, so that a group of data is (n*m+2) frames. Each frame of image is subjected to pixel normalization.

[0007] In two-dimensional attention mechanism, generally spatial-channel attention mechanism, on the one hand, highlight the features in the image space, on the other hand, highlight the features in different channels, while in three-dimensional convolution, simple spatial-channel attention mechanism cannot well highlight the features in time sequence, and the motion features in time sequence are often the most important in video; therefore, a time-space hybrid attention mechanism module suitable for three-dimensional video input is designed.

[0008] The attention mechanism module is specifically designed as follows: divided into a motion attention mechanism Diff module and a channel attention mechanism ECA module; firstly, the motion attention mechanism Diff module; the motion features between adjacent frames are obtained through inter-frame difference, and the motion features are extracted through convolution on the re-spliced data after difference, to highlight the motion feature expression in the data; then the dimension is adjusted through maximum pooling, and finally the channel attention mechanism ECA module is used for channel attention feature extraction, which together constitutes the attention mechanism module; the specific structure is as shown in Figure 2 .

[0009] The convolutional neural network model ConvNeXt is a pure convolutional neural network model that approaches the transformer neural network model, and provides a large number of training techniques to improve the accuracy of the model; the lightweight neural network model Shufflev2 is a typical lightweight convolutional neural network model. Combining the advantages of ConvNeXt and Shuffle v2 two neural network models, the ConvNeXt neural network is lightened, and a light-weight ConvNeXt module is designed as the basic module of the network; named as LW-ConvNeXt module.

[0010] The specific design of the light-weight feature extraction LW-ConvNeXt module is as follows: the specific structure is as shown in Figure 3 , on the basis of the feature extraction ConvNeXt module, in order to reduce the model parameter amount, the convolution kernel size is changed from 7 to 3; the depth separable convolution is used instead of the general three-dimensional convolution; the inverse bottleneck layer is removed; the residual structure is used, and the features after channel-by-channel convolution and point-by-point convolution are added to the input; then the data is normalized, and activated by the activation function, and finally the channel is mixed, and the feature data is output; the module is the basic module of the LW-ConvNeXt neural network; the channel number and the feature map size of the output and the input do not change.

[0011] Compared with the prior art, the advantages of the present application are:

[0012] This invention addresses the weakness in extracting temporal motion features by designing a motion attention mechanism module. This module can better extract motion features, focusing on the motion features of the video while suppressing unnecessary background features. It effectively improves the accuracy of neural network models and is applicable to various 3D convolutional neural network models. It is also plug-and-play and does not affect the overall structure of the model. It has a small number of parameters and does not introduce redundancy into the model.

[0013] This invention combines the lightweight neural network shuffle v2 and the convolutional neural network ConNeXt to design a lightweight feature extraction module, LW-ConvNeXt. Compared with the original model, the new network model based on this module is only 6.76M in size, and the number of parameters is reduced by 94.7%. Experimental results on the dynamic gesture dataset 20-bn-jester show an accuracy of 94.6% and a frame rate of 174 frames per second, meeting the requirements of real-time performance, accuracy, and lightweight design.

[0014] This invention uses convolutional kernels of convolutional neural networks to replace general pooling layers, so the parameters of pooling layers can also be changed through training; by setting different convolutional kernel sizes and different strides, different data dimensions can be set after sampling. Attached Figure Description

[0015] Figure 1 This is a diagram of the overall network structure of the present invention;

[0016] Figure 2 for Figure 1 Schematic diagram of the motor attention module;

[0017] Figure 3 for Figure 1 Schematic diagram of the stem module structure;

[0018] Figure 4 for Figure 1 Schematic diagram of the downsampling module;

[0019] Figure 5 for Figure 1 Schematic diagram of the lightweight feature extraction module LW-ConvNeXt;

[0020] Figure 6 Input examples for the experiment;

[0021] Figure 7 For training curves;

[0022] Figure 8 The resulting confusion matrix. Detailed Implementation Plan

[0023] To make the technical solution and structural features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples. The described examples are one example of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0024] Figure 1 The diagram shows the overall network structure of this invention. The input is a dynamic gesture video. This example uses the 20-bn-jester open-source dynamic gesture dataset, which includes approximately 140,000 data points and 27 gestures, categorized as shown in Table 1. The dataset is divided into training, validation, and test sets according to a certain ratio. The detailed model training process is as follows:

[0025] Table 1: 27 categories of gestures

[0026] 1 Swipe left 2 Swipe right 3 Swipe down 4 Swipe up 5 Push hands away 6 Pull hands in 7 Two-finger swipe left 8 Two-finger swipe right 9 Two-finger swipe down 10 Two-finger swipe up 11 Two-finger push away 12 Two-finger pull in 13 Scroll hand forward 14 Scroll hand backward 15 Turn hand clockwise 16 Turn hand counterclockwise 17 Zoom in with full hand 18 Zoom out with full hand 19 Zoom in with two fingers 20 Zoom out with two fingers 21 Thumb up 22 Thumb down 23 Wave hand repeatedly 24 Stop gesture 25 Clap hands gesture 26 No hands 27 Other gestures

[0027] The input data is a video clip. First, it enters the data preprocessing module, where the input video undergoes three-frame differential processing to remove frames with excessive repetition. The processed video is then divided into n segments based on time, with each segment randomly selecting m consecutive frames, plus the first and last two frames. In this example, n=4 and m=4, forming a group of 18 input data. In this experiment, the height and width of each frame are both set to 112. After normalization, the data is input into the downsampling stem module.

[0028] This module takes N×C×T×H×W dimension feature data as input, which is very large for a direct input image model. Therefore, the size needs to be adjusted first. The main function of this module is to adjust the input feature size, increase the number of channels, decrease the channel size, and extract features simultaneously. Its structure is as follows: Figure 3 As shown, the convolution operation adjusts the dimensionality and normalizes the result.

[0029] After passing through the downsampling stem module, the input is fed into the four stages of the backbone network. Each stage mainly consists of a downsampling module and several lightweight feature extraction LW-ConvNeXt modules, which are the basic modules of this neural network. In this experiment, the ratio of the number of lightweight feature extraction LW-ConvNeXt modules in each stage is 3:3:9:3, which is used to repeatedly extract features. Motion attention modules are inserted between each stage to effectively improve the accuracy.

[0030] The main function of the downsampling module is to replace the pooling layer. The downsampling module primarily achieves the purpose of pooling through convolution operations. In this example, a 3D grouped convolution with a kernel size of 2 and a stride of 2 is used, with the number of groups matching the number of input channels. While achieving the same downsampling effect as the pooling layer, pointwise convolution and channel shuffling are used to exchange data information between different channels. Finally, data normalization is performed. The number of output channels is twice the number of input channels, and the size of the output feature map can be adjusted according to the stride. In this example, the number of channels from input to the end of feature extraction is [3, 64, 128, 256, 512]. Except for the first transformation, which is completed by the stem module, all other dimensionality transformations are performed by the downsampling module.

[0031] After downsampling, the system proceeds to the lightweight feature extraction module LW-ConvNeXt, which is the foundational module of this neural network; its specific structure is as follows: Figure 5 As shown, this module does not change the number of channels or the size of the feature map, and adopts a residual structure, using channel-wise convolution and pointwise convolution to replace general 3D convolution. Since the dimensionality increase operation of the inverse bottleneck layer in the ConvNeXt module will lead to too many parameters, increasing the model complexity while having limited improvement in accuracy, the inverse bottleneck layer structure in the feature extraction ConvNeXt module is abandoned. Finally, the data is normalized, activated using an activation function, and the extracted features are output.

[0032] The attention mechanism module inserted between each adjacent stage can be specifically divided into two parts: the motion attention mechanism Diff module and the channel attention mechanism ECA module, which are connected in series. First, the motion attention mechanism Diff module performs difference on the features of two adjacent frames. The feature map obtained after difference is concatenated with the first feature map in the temporal dimension. After concatenation, the same dimension as the input feature is obtained. The difference features are then input into a 1×3×3 three-dimensional group convolution module for convolution, with the number of groups being the number of input feature channels. After feature extraction, global max pooling is performed in the H×W dimensions. Finally, inter-frame convolution is performed through a 3×1×1 three-dimensional convolution module to facilitate data exchange between channels.

[0033] The output data of the motion attention mechanism Diff module is multiplied by its input data, and the result is input into the channel attention mechanism ECA module. This module is designed to implement the attention mechanism at the channel level. Different channels often represent different features, and these features are often very abstract. Individuals cannot distinguish which channel's features are more important based on experience alone, so a channel attention mechanism is needed. First, global average pooling is performed in the T×H×W dimension to obtain N×C×1×1×1 feature data. The feature dimensions are then reshaped to N×1×C dimensions and input into a one-dimensional convolution module for convolution to obtain attention features. After multiplying with the input features, the output is completed, thus completing the attention mechanism.

[0034] After completing the four stages, the obtained feature data is averaged and pooled across the three dimensions of T×H×W to obtain two-dimensional data. The output is then classified through a fully connected layer to complete the gesture recognition process.

[0035] The AdamW optimizer was used in this experiment during training. The initial learning rate was set to 0.01, and the learning rate decay factor was 0.1. 200 training rounds were performed, and the best result from each experiment was taken. The highest accuracy achieved on the 20-bn-jester dynamic gesture recognition dataset was 94.6%. The training process was recorded as follows: Figure 7 As shown; the detailed confusion matrix is ​​as follows. Figure 8 As shown in the figure; the results are compared with those of other commonly used algorithms in Table 2.

[0036] With an NVIDIA 3060 graphics card, taking 16 pixels at a time from a 3×18×112×112 pixel array, the floating-point computation time is only 0.6105 GFLOPs, the frame rate is 4.89ms, and the average frame rate is 174 frames per second. The current network using 18 frames of input takes 81.5ms. This fully meets the requirements for real-time performance, accuracy, and lightweight design.

[0037] Table 2 Comparison of Common Algorithms

[0038]

Claims

1. A method for constructing a lightweight dynamic gesture recognition neural network model, characterized in that, Its overall structure and steps are as follows: This dynamic gesture recognition neural network model is a lightweight convolutional neural network model improved on the ConvNeXt convolutional neural network model, named LW-ConvNeXt neural network. The size of the new network model is only 6.76M. The input is a gesture video, and the input data is five-dimensional data N×C×T×H×W, representing the batch size, channel size, depth (frames), height, and width, respectively. First, data preprocessing is performed using a three-frame difference method. The processed consecutive frames are divided into n segments, each segment taking m frames, plus the first and last two frames, forming a set of input data. Therefore, a set of data consists of (n*m+2) frames, and pixel normalization is performed on each frame. After processing, the data is input into the LW-ConvNeXt neural network, whose structure is as follows: First, it enters the downsampling stem module to adjust the input feature size. The process involves adjusting the number of channels, increasing the channel size, and extracting features simultaneously. Then, it proceeds in a series of four stages, each consisting of a downsampling module and several lightweight feature extraction LW-ConvNeXt modules. The ratio of the number of lightweight feature extraction LW-ConvNeXt modules in the four stages is 3:3:9:

3. An attention mechanism module is interspersed between the four stages. After these four stages, the obtained features are averaged across the T×H×W dimensions to obtain data of dimension N×C. Finally, the data is input into a fully connected layer for classification, yielding the classification result and completing the gesture recognition process. The lightweight feature extraction LW-ConvNeXt module, based on the feature extraction ConvNeXt module, reduces the number of model parameters by changing the convolution kernel size from 7 to 3; using depthwise separable convolution instead of general 3D convolution; removing the inverse bottleneck layer; using a residual structure overall, performing addition operations between the features and input after channel-wise convolution and pointwise convolution; then performing data normalization and activation by an activation function; finally, channel shuffling to output feature data. This module is the foundational module of the LW-ConvNeXt neural network; the number of output and input channels and the size of the feature map remain unchanged, and it is used for feature extraction.

2. The method for constructing a lightweight dynamic gesture recognition neural network model according to claim 1, characterized in that: The attention mechanism module is divided into two parts: a motion attention mechanism (Diff) module and a channel attention mechanism (ECA) module, which are connected in series. Feature data first enters the motion attention mechanism (Diff) module, where features from two adjacent frames are differencing. The resulting feature map is then concatenated with the first feature map in the temporal dimension, resulting in a concatenation with the same dimension as the input features. The differencing features are then input into a 1×3×3 three-dimensional group convolution module for convolution, with the number of groups equal to the number of input feature channels. After extraction, global max pooling is performed in the H×W dimensions, and finally, inter-frame convolution is performed through a 3×1×1 three-dimensional convolution module to facilitate data exchange between channels. The output data of the motion attention mechanism Diff module is multiplied by its input data, and the result is input into the channel attention mechanism ECA module. Then, global average pooling is performed in the T×H×W dimensions to obtain N×C×1×1×1 feature data. The feature dimensions are reshaped into N×1×C dimensions and then input into a one-dimensional convolution module for convolution to obtain attention features. After multiplying with the input features, the output is completed, thus completing the attention mechanism.

3. The method for constructing a lightweight dynamic gesture recognition neural network model according to claim 1, characterized in that: Convolutional kernels from convolutional neural networks are used instead of regular pooling layers, so the parameters of pooling layers can also be changed through training; different data dimensions can be set after sampling by setting different kernel sizes and different strides.

Images