Lightweight gesture recognition method and system, electronic equipment and storage medium

Abstract

The invention discloses a lightweight gesture recognition method and system, electronic equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring to-be-detected data; the to-be-detected data comprises picture data and video data; inputting the to-be-detected gesture into a trained gesture recognition model for prediction, and outputting a gesture position and a gesture category of a current frame; the gesture types comprise a one gesture, a five gesture, a first gesture, an ok gesture and a yeah gesture; the gesture recognition model is constructed on the basis of a YOLOv8n basic network, a GB-C2f target detection module and an F-FPN fusion type feature pyramid network. According to the invention, the reasoning speed and detection precision of gesture recognition can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a lightweight gesture recognition method, system, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence and the increasing complexity of human-computer interaction application scenarios, traditional human-computer interaction methods have gradually exposed their limitations. In order to meet these challenges, new interaction methods continue to emerge. Among them, gesture recognition has received widespread attention and in-depth research in many fields due to its flexibility, speed and efficiency. The visual technology of gesture recognition provides users with a natural and intuitive way of interaction, which can be used in application scenarios such as interactive entertainment, smart home, sign language recognition, and smart cars, improving the user experience.

[0003] The application of gesture recognition technology presents diverse advantages and potentials. In order to improve the accuracy and efficiency of gesture recognition, deep learning technology has become a powerful solution. Commonly used deep learning technologies are mainly divided into single-stage and two-stage target detection algorithms. Among them, single-stage algorithms, such as SSD, YOLO, etc., have faster calculation speeds but slightly lose accuracy; two-stage target detection algorithms, such as Fast R-CNN, CornerNet, etc., have more complete feature extraction, but are relatively slow. Summary of the invention

[0004] The purpose of the present invention is to provide a lightweight gesture recognition method, system, electronic device and storage medium, which can improve the reasoning speed and detection accuracy of gesture recognition.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A lightweight gesture recognition method, comprising:

[0007] Acquire data to be detected; the data to be detected includes image data and video data;

[0008] The gesture to be detected is input into the trained gesture recognition model for prediction, and the gesture position and gesture category of the current frame are output; the gesture categories include one gesture, five gesture, fist gesture, ok gesture and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n basic network, GB-C2f target detection module and F-FPN fusion feature pyramid network.

[0009] Optionally, the process of constructing the gesture recognition model includes:

[0010] First, build the GB-C2f target detection module:

[0011] The original Bottelneck structure in the GB-C2f target detection module is replaced with the Ghost Bottelneck structure, wherein the Ghost Bottelneck structure includes two stacked Ghost Conv modules, and the two Ghost Conv modules are connected using a shortcut, and the Ghost Conv module includes a 1×1 convolution kernel and a 5×5 convolution kernel connected in sequence, and the output result of the 5×5 convolution kernel is channel-joined with the output result of the 1×1 convolution kernel in the previous step;

[0012] Then, construct the F-FPN fusion feature pyramid network:

[0013] The F-FPN fusion feature pyramid network is proposed based on the ASFF algorithm and the AFPN algorithm, and the F-FPN fusion feature pyramid network replaces the FPN layer in the YOLOv8n basic network.

[0014] Optionally, the specific construction process of the GB-C2f target detection module is:

[0015] First, use a 1×1 convolution kernel to perform a standard convolution operation with a step size of 1 on the input data to halve the number of channels of the input data. Then use a 5×5 convolution kernel to perform a standard convolution with a step size of 1 on the output data generated in the previous step, and concatenate the convolution result with the output data obtained in the previous step to restore the original number of feature channels to obtain the GhostConv module.

[0016] The Conv module in the original Bottleneck structure is replaced with the Ghost Conv module to form a new Ghost Bottleneck structure; the Ghost Bottleneck structure includes two stacked Ghost Conv modules, wherein the first Ghost Conv module is used as an extension layer, and the second Ghost Conv module is used to reduce the number of channels of output data so that the number of output channels matches the number of input channels; a Shortcut connection is used between the two Ghost Conv modules, and after the second Ghost Conv module processes the feature information, a ReLU activation operation is not performed;

[0017] The original Bottelneck structure is replaced with the Ghost Bottelneck structure to obtain a lightweight GB-C2f target detection module.

[0018] Optionally, the operation process of the F-FPN fusion feature pyramid network is:

[0019] Extract the features of the 4th, 6th and 9th layers from the backbone network to obtain a set of features at different scales; before feature fusion, in order to align the feature dimensions and prepare for feature fusion, use 1×1 convolution and bilinear interpolation method to upsample the low-resolution features, and use 2×2 convolution with a stride of 2 to downsample the high-resolution feature maps twice and 4×4 convolution with a stride of 4 to downsample them four times;

[0020] The ASFF algorithm is used to perform consistency processing on the sampling data for feature fusion at different spatial positions, and then feature fusion is performed. The low-level features of the 4th layer and the middle-level features of the 6th layer are first input into the feature pyramid network for preliminary feature fusion, and then the high-level features of the 9th layer are added to the preliminary fusion results for feature fusion again; after each feature fusion is completed, the GB-C2f target detection module is used to learn the features to generate a set of multi-scale features, and the recognition task is completed based on the features.

[0021] Optionally, the training process of the gesture recognition model is:

[0022] The static hand gesture handpose_x_gesture_v1 data set is screened and sorted, and the data after the initial screening is manually annotated, and the annotated gesture data is rotated, denoised and blurred to obtain training data; the training data includes gesture images and corresponding recognition results;

[0023] Build a pre-trained model;

[0024] The training data is input into the pre-training model for parameter optimization, and a model that meets the set number of training rounds and accuracy is determined as a trained gesture recognition model.

[0025] The present invention also provides a lightweight gesture recognition system, which is applied to the above method, comprising:

[0026] A data acquisition unit, used to acquire data to be detected; the data to be detected includes image data and video data;

[0027] The gesture recognition unit is used to input the gesture to be detected into a trained gesture recognition model for prediction, and output the gesture position and gesture category of the current frame; the gesture categories include one gesture, five gesture, fist gesture, ok gesture and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n basic network, GB-C2f target detection module and F-FPN fusion feature pyramid network.

[0028] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the lightweight gesture recognition method described above.

[0029] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program implements the lightweight gesture recognition method as described above when executed by a processor.

[0030] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0031] The present invention discloses a lightweight gesture recognition method, system, electronic device and storage medium, the method comprises obtaining data to be detected; the data to be detected comprises picture data and video data; the gesture to be detected is input into a trained gesture recognition model for prediction, and the gesture position and gesture category of the current frame are output; the gesture category comprises one gesture, five gesture, fist gesture, ok gesture and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n basic network, GB-C2f target detection module and F-FPN fusion feature pyramid network. The present invention can improve the reasoning speed and detection accuracy of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 This is a structural diagram of the gesture recognition model GA-YOLO in this embodiment;

[0034] Figure 2 This is a structural diagram of the lightweight target detection module GB-C2f in this embodiment;

[0035] Figure 3 : is a diagram of the Ghost Conv module in this embodiment;

[0036] Figure 4 It is the structure diagram of Ghost Bottelneck in this embodiment;

[0037] Figure 5 This is a structural diagram of the fused feature pyramid network F-FPN in this embodiment. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The purpose of the present invention is to provide a lightweight gesture recognition method, system, electronic device and storage medium, which can improve the reasoning speed and detection accuracy of gesture recognition.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] like Figure 1-Figure 5 As shown, the present invention provides a lightweight gesture recognition method, comprising:

[0042] Step 100: Acquire data to be detected; the data to be detected includes image data and video data.

[0043] Step 200: Input the gesture to be detected into a trained gesture recognition model for prediction, and output the gesture position and gesture category of the current frame; the gesture categories include one gesture, five gesture, fist gesture, ok gesture and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n basic network, GB-C2f target detection module and F-FPN fusion feature pyramid network.

[0044] Based on the above technical solution, the following embodiments are provided.

[0045] Taking the model construction and training process as an example, the steps of lightweight gesture recognition are as follows:

[0046] Step 1: Dataset collection and processing

[0047] By screening and sorting the static hand gesture handpose_x_gesture_v1 dataset, the LabelImg annotation tool is used to manually annotate the gesture images, and the image data is converted into the YOLO dataset format containing the target category and target position coordinates. The images are expanded to 5,500 using data enhancement methods such as rotation, noise addition, and blurring as a dataset for neural network training. The gesture dataset is divided into training set, validation set, and test set in a ratio of 7:2:1.

[0048] Step 2: Build a lightweight target detection module GB-C2f

[0049] Figure 2 The structure diagram of the lightweight target detection module GB-C2f is shown. Figure 3 This is the Ghost Conv module diagram. Figure 4 The structure diagram of GhostBottelneck is as follows:

[0050] (1) First, a Ghost Conv module is designed. This module first uses a 1×1 convolution kernel to perform a standard convolution operation with a stride of 1 on the input feature map, effectively halving the number of channels of the feature map, thereby reducing the amount of computation while retaining key information. Then, a 5×5 convolution kernel is used to perform a standard convolution with a stride of 1 on the previously generated feature map, and the result is concatenated with the feature map obtained in the previous step to restore the original number of feature channels.

[0051] (2) Replace Conv in the Bottleneck structure with Ghost Conv to form a new Ghost Bottleneck module. The Ghost Bottleneck module combines the advantages of multiple convolutional layers and shortcut connections to improve the efficiency and performance of the network. This module is mainly composed of two stacked Ghost Conv modules. The first Ghost Conv module serves as an extension layer. This extension layer not only increases the number of channels, but also controls the amount of calculation and parameters through the design of Ghost Conv, avoiding excessive calculation burden. The second Ghost module reduces the number of channels of the output feature map to match the number of input channels, ensuring that the input and output of the module can be connected smoothly and maintaining the continuity and consistency of the network structure. A shortcut connection is used between the two Ghost Conv modules, so that the input feature map can be directly added to the output feature map, thereby retaining more original information and accelerating the convergence of the network. In addition, after the second Ghost Conv module processes the feature information, no ReLU activation operation is performed. Batch normalization and ReLU nonlinear activation are used after other layers process the feature information.

[0052] (3) The Bottelneck structure in the C2f module is replaced with the Ghost Bottelneck structure to fully utilize the advantages of GhostNet in reducing computational complexity and parameter count. The GB-C2f module significantly reduces the use of 3×3 ordinary convolutions, thereby significantly compressing the model size. It not only reduces the number of parameters and computation, but also enables the model to run more efficiently in resource-constrained environments. The GB-C2f module reduces the model's floating-point operations by 27.2% and the number of parameters by 30.2%.

[0053] Among them, the experimental results comparison of YOLOv8n adding GB-C2f module is shown in Table 1.

[0054] Table 1 Comparison of experimental results

[0055] Model mAP GFLOPs Parameters YOLOv8 96.8 8.1 3,006,623 YOLOv8+GB-C2f 96.5 5.9 2,099,611

[0056] Step 3: Construct a fused feature pyramid network F-FPN

[0057] Figure 5 Shown is the structure diagram of the fusion feature pyramid network F-FPN:

[0058] (1) First, the features of the 4th, 6th and 9th layers are extracted from the backbone network to obtain a set of features at different scales {C3, C4, C5}. Then, feature fusion is performed. The low-level feature C3 and the middle-level feature C4 are first input into the feature pyramid network for preliminary fusion, and then the high-level feature C5 is added. After the feature fusion is completed, the GB-C2f unit is used to learn the features to generate a set of multi-scale features {P3, P4, P5}.

[0059] (2) In order to align the dimensions of the feature maps and prepare for feature fusion, the low-resolution feature maps are upsampled using 1×1 convolution and bilinear interpolation, and the high-resolution feature maps are downsampled twice using 2×2 convolution with a stride of 2 and 4×4 convolution with a stride of 4.

[0060] (3) Since feature fusion at each spatial position may cause conflicts in multi-target information, the ASFF operation is introduced to alleviate these inconsistencies. In addition, F-FPN optimizes the model by reducing the network width to one-fourth of the original, reducing the number of model parameters while ensuring accuracy. F-FPN reduces the model's floating-point operations by 24.7%, the number of parameters by 25.5%, and increases mAP by 0.6%.

[0061] Among them, the experimental results of YOLOv8n adding F-FPN module are shown in Table 2.

[0062] Table 2 Comparison of experimental results

[0063] Model mAP GFLOPs Parameters YOLOv8 96.8 8.1 3,006,623 YOLOv8+F-FPN 97.4 6.1 2,240,580

[0064] Step 4: Build a lightweight, high-precision target detection network GA-YOLO

[0065] First, two Ghost Conv modules are stacked to form a GhostBottleneck module. By replacing the Bottleneck module in the original C2f with the GhostBottleneck module, the lightweight GB-C2f module is redesigned to compress the model and reduce the amount of calculation; secondly, in order to compensate for the reduced detection accuracy due to lightweight, the F-FPN network is proposed based on the ASFF and AFPN algorithms, and it is used to replace the FPN in the original YOLOv8. The feature fusion is performed in an adaptive and progressive manner to enable better interaction between non-adjacent layers to extract more useful information, improve the model's perception of gesture features, and reduce the complexity of the model. The floating-point operations of the GA-YOLO network are reduced by 46.9%, and the number of parameters is reduced by 44.9%.

[0066] Among them, the comparison of GA-YOLO network training experimental results is shown in Table 3.

[0067] Table 3 Comparison of GA-YOLO network training experimental results

[0068]

[0069]

[0070] Step 5: Model training

[0071] Use the lightweight improved GA-YOLO network for model training, set the number of training rounds to 400, the input image size to 640×640, and save the best model in the training process. After the model training is completed, save the model. Among them, after 400 rounds of training, you will get the experimental results and two weight files, which are the best training weight file and the weight file of the last training. Import the obtained weight files into the system to detect the picture or video, mark the gesture position and gesture category, and finally display the detection results on the display window. The display window will recognize the gesture and frame it with a rectangular frame. The upper left corner of the frame displays the gesture category and recognition confidence.

[0072] Therefore, it can be seen that the technical solution in this embodiment solves the problems of complex background and limited hardware computing power based on visual recognition algorithms, solves the problem that wearable device recognition methods cannot be separated from the support of external devices, and can achieve efficient and accurate human-computer interaction. A static gesture dataset was selected and expanded, and the mainstream target detection algorithms were evaluated and compared. Finally, YOLOv8n was selected as the basic network and improved for training.

[0073] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0074] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A lightweight gesture recognition method, characterized in that: include: Acquire data to be detected; the data to be detected includes image data and video data; The gesture to be detected is input into the trained gesture recognition model for prediction, and the gesture position and gesture category of the current frame are output; the gesture categories include one gesture, five gesture, fist gesture, ok gesture and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n basic network, GB-C2f target detection module and F-FPN fusion feature pyramid network.

2. The lightweight gesture recognition method according to claim 1, characterized in that: The construction process of the gesture recognition model includes: First, build the GB-C2f target detection module: The original Bottelneck structure in the GB-C2f target detection module is replaced with the Ghost Bottelneck structure, wherein the Ghost Bottelneck structure includes two stacked Ghost Conv modules, and the two Ghost Conv modules are connected using a shortcut, and the Ghost Conv module includes a 1×1 convolution kernel and a 5×5 convolution kernel connected in sequence, and the output result of the 5×5 convolution kernel is channel-joined with the output result of the 1×1 convolution kernel in the previous step; Then, construct the F-FPN fusion feature pyramid network: The F-FPN fusion feature pyramid network is proposed based on the ASFF algorithm and the AFPN algorithm, and the F-FPN fusion feature pyramid network replaces the FPN layer in the YOLOv8n basic network.

3. The lightweight gesture recognition method according to claim 2, characterized in that: The specific construction process of the GB-C2f target detection module is as follows: First, use a 1×1 convolution kernel to perform a standard convolution operation with a step size of 1 on the input data to halve the number of channels of the input data. Then use a 5×5 convolution kernel to perform a standard convolution with a step size of 1 on the output data generated in the previous step, and concatenate the convolution result with the output data obtained in the previous step to restore the original number of feature channels to obtain the GhostConv module. The Conv module in the original Bottleneck structure is replaced with the Ghost Conv module to form a new Ghost Bottleneck structure; the Ghost Bottleneck structure includes two stacked Ghost Conv modules, wherein the first Ghost Conv module serves as an extension layer, and the second Ghost Conv module is used to reduce the number of channels of output data so that the number of output channels matches the number of input channels; a Shortcut connection is used between the two Ghost Conv modules, and after the second Ghost Conv module processes the feature information, a ReLU activation operation is not performed; The original Bottelneck structure is replaced with the Ghost Bottelneck structure to obtain a lightweight GB-C2f target detection module.

4. The lightweight gesture recognition method according to claim 2, characterized in that: The operation process of the F-FPN fusion feature pyramid network is as follows: Extract the features of the 4th, 6th and 9th layers from the backbone network to obtain a set of features at different scales; before feature fusion, in order to align the feature dimensions and prepare for feature fusion, use 1×1 convolution and bilinear interpolation method to upsample the low-resolution features, and use 2×2 convolution with a stride of 2 to downsample the high-resolution feature maps twice and 4×4 convolution with a stride of 4 to downsample them four times; The ASFF algorithm is used to process the consistency of the sampled data for feature fusion at different spatial positions, and then feature fusion is performed. The low-level features of the 4th layer and the middle-level features of the 6th layer are first input into the feature pyramid network for preliminary feature fusion, and then the high-level features of the 9th layer are added to the preliminary fusion results for feature fusion again. After each feature fusion is completed, the GB-C2f target detection module is used to learn the features, generate a set of multi-scale features, and complete the recognition task based on the features.

5. The lightweight gesture recognition method according to claim 1, characterized in that: The training process of the gesture recognition model is as follows: The static hand gesture handpose_x_gesture_v1 data set is screened and sorted, and the data after the initial screening is manually annotated, and the annotated gesture data is rotated, denoised and blurred to obtain training data; the training data includes gesture images and corresponding recognition results; Build a pre-trained model; The training data is input into the pre-training model for parameter optimization, and a model that meets the set number of training rounds and accuracy is determined as a trained gesture recognition model.

6. A lightweight gesture recognition system, applied to the method according to any one of claims 1 to 5, characterized in that: include: A data acquisition unit, used for acquiring data to be detected; The data to be detected includes picture data and video data; The gesture recognition unit is used to input the gesture to be detected into a trained gesture recognition model for prediction, and output the gesture position and gesture category of the current frame; the gesture categories include one gesture, five gesture, fist gesture, ok gesture and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n basic network, GB-C2f target detection module and F-FPN fusion feature pyramid network.

7. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the lightweight gesture recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the lightweight gesture recognition method as described in any one of claims 1 to 5.

Images