A lightweight gesture recognition method, system, electronic device and storage medium

Abstract

The present invention discloses a lightweight gesture recognition method, system, electronic device and storage medium, which relates to the field of computer vision technology. The method includes: obtaining data to be detected; the data to be detected includes picture data and video data; inputting the gesture to be detected into a trained gesture recognition model for prediction, and outputting the gesture position and gesture category of the current frame; the gesture categories include one gesture, five gesture, fist gesture, ok gesture and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n basic network, the GB-C2f object detection module and the F-FPN fusion feature pyramid network. The present invention can improve the inference speed and detection accuracy of gesture recognition.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly to a lightweight gesture recognition method, system, electronic device, and storage medium. Background Art

[0002] With the rapid development of artificial intelligence and the increasing complexity of human-computer interaction application scenarios, the limitations of traditional human-computer interaction methods have gradually emerged. To address these challenges, new interaction methods have emerged continuously. Among them, gesture recognition has received extensive attention and in-depth research in multiple fields due to its flexible, fast, and efficient characteristics. The visual technology of gesture recognition provides users with a natural and intuitive interaction method and can be used in application scenarios such as interactive entertainment, smart home, sign language recognition, and intelligent vehicle, improving the user experience.

[0003] The application of gesture recognition technology presents diverse advantages and potential. To improve the accuracy and efficiency of gesture recognition, deep learning technology has become a powerful solution. The commonly used deep learning technologies are mainly divided into single-stage and two-stage object detection algorithms. Among them, single-stage algorithms, such as SSD, YOLO, etc., although the calculation speed is faster, there is a slight loss in accuracy; two-stage object detection algorithms, such as Fast R-CNN, CornerNet, etc., extract features more fully, but the speed is relatively slow. Summary of the Invention

[0004] The purpose of the present invention is to provide a lightweight gesture recognition method, system, electronic device, and storage medium, which can improve the inference speed and detection accuracy of gesture recognition.

[0005] To achieve the above purpose, the present invention provides the following solutions:

[0006] A lightweight gesture recognition method includes:

[0007] Obtain data to be detected; the data to be detected includes picture data and video data;

[0008] Input the gesture to be detected into a trained gesture recognition model for prediction, and output the gesture position and gesture category of the current frame; the gesture category includes one gesture, five gesture, fist gesture, ok gesture, and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n basic network, GB-C2f object detection module, and F-FPN fusion feature pyramid network.

[0009] Optionally, the construction process of the gesture recognition model includes:

[0010] First, construct a GB-C2f object detection module:

[0011] Replace the original Bottleneck structure in the GB-C2f object detection module with the Ghost Bottelneck structure. The Ghost Bottelneck structure includes two stacked Ghost Conv modules, and a Shortcut connection is used between the two Ghost Conv modules. The Ghost Conv module includes a 1×1 convolutional kernel and a 5×5 convolutional kernel connected in sequence, and the output result of the 5×5 convolutional kernel is concatenated with the output result of the previous 1×1 convolutional kernel in the channel dimension.

[0012] Then, construct an F-FPN fusion feature pyramid network:

[0013] Propose the F-FPN fusion feature pyramid network based on the ASFF algorithm and the AFPN algorithm, and replace the FPN layer in the YOLOv8n base network with the F-FPN fusion feature pyramid network.

[0014] Optionally, the specific construction process of the GB-C2f object detection module is as follows:

[0015] First, perform a standard convolution operation with a stride of 1 on the input data using a 1×1 convolutional kernel to halve the number of channels of the input data. Then, perform a standard convolution with a stride of 1 on the output data generated in the previous step using a 5×5 convolutional kernel, and concatenate the convolution result with the output data obtained in the previous step in the channel dimension to restore the original number of feature channels, obtaining a Ghost Conv module.

[0016] Replace the Conv module in the original Bottleneck structure with the Ghost Conv module to form a new Ghost Bottleneck structure. The Ghost Bottleneck structure includes two stacked Ghost Conv modules. Among them, the first Ghost Conv module serves as an expansion layer, and the second Ghost Conv module is used to reduce the number of channels of the output data to match the number of input channels. A Shortcut connection is used between the two Ghost Conv modules, and after the second Ghost Conv module processes the feature information, no ReLU activation operation is performed.

[0017] Replace the original Bottelneck structure with the Ghost Bottelneck structure to obtain a lightweight GB-C2f object detection module.

[0018] Optionally, the operation process of the F-FPN fusion feature pyramid network is as follows:

[0019] Extract the features of the 4th, 6th, and 9th layers from the backbone network to obtain a set of features at different scales; before feature fusion, in order to align the feature dimensions and prepare for feature fusion, use 1×1 convolution and bilinear interpolation methods to upsample the low-resolution features, and use 2×2 convolution with a stride of 2 for 2 downsamplings and 4×4 convolution with a stride of 4 for 4 downsamplings on the high-resolution feature map;

[0020] Use the ASFF algorithm to perform consistency processing on the sampling data for feature fusion at different spatial positions, and then perform feature fusion. First, input the low-level features of the 4th layer and the middle-level features of the 6th layer into the feature pyramid network for preliminary feature fusion, and then add the high-level features of the 9th layer to the preliminary fusion result for further feature fusion; after each feature fusion is completed, use the GB-C2f object detection module to learn the features, generate a set of multi-scale features, and complete the recognition task according to the features.

[0021] Optionally, the training process of the gesture recognition model is as follows:

[0022] Screen and organize the static gesture handpose_x_gesture_v1 dataset, manually annotate the data after preliminary screening, and perform rotation, noise addition, and blurring processing on the annotated gesture data to obtain training data; the training data includes gesture images and corresponding recognition results;

[0023] Construct a pre-trained model;

[0024] Input the training data into the pre-trained model for parameter optimization, and determine the model that meets the set number of training epochs and accuracy as the trained gesture recognition model.

[0025] The present invention also provides a lightweight gesture recognition system, which is applied to the method described above and includes:

[0026] A data acquisition unit for acquiring data to be detected; the data to be detected includes picture data and video data;

[0027] A gesture recognition unit for inputting the gesture to be detected into the trained gesture recognition model for prediction, and outputting the gesture position and gesture category of the current frame; the gesture categories include one gesture, five gesture, fist gesture, ok gesture, and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n backbone network, the GB-C2f object detection module, and the F-FPN fusion type feature pyramid network.

[0028] The present invention also provides an electronic device, including a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the lightweight gesture recognition method according to the above.

[0029] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the lightweight gesture recognition method as described above is implemented.

[0030] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:

[0031] The present invention discloses a lightweight gesture recognition method, system, electronic device and storage medium. The method includes obtaining data to be detected; the data to be detected includes picture data and video data; inputting the gesture to be detected into a trained gesture recognition model for prediction, and outputting the gesture position and gesture category of the current frame; the gesture category includes one gesture, five gesture, fist gesture, ok gesture and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n basic network, GB-C2f object detection module and F-FPN fusion feature pyramid network. The present invention can improve the inference speed and detection accuracy of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0033] Figure 1 It is the structure diagram of the gesture recognition model GA-YOLO in this embodiment;

[0034] Figure 2 It is the structure diagram of the lightweight object detection module GB-C2f in this embodiment;

[0035] Figure 3 It is the Ghost Conv module diagram in this embodiment;

[0036] Figure 4 It is the structure diagram of the Ghost Bottelneck in this embodiment;

[0037] Figure 5 It is the structure diagram of the fusion feature pyramid network F-FPN in this embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0039] The object of the present invention is to provide a lightweight gesture recognition method, system, electronic device and storage medium, which can improve the inference speed and detection accuracy of gesture recognition.

[0040] In order to make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] As Figures 1 - 5 shown, the present invention provides a lightweight gesture recognition method, including:

[0042] Step 100: Obtain the data to be detected; the data to be detected includes picture data and video data.

[0043] Step 200: Input the gesture to be detected into the trained gesture recognition model for prediction, and output the gesture position and gesture category of the current frame; the gesture categories include one gesture, five gesture, fist gesture, ok gesture and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n basic network, GB-C2f object detection module and F-FPN fusion feature pyramid network.

[0044] Based on the above technical solutions, the following embodiments are provided.

[0045] Taking the model construction and training process as an example, the method steps of lightweight gesture recognition are as follows:

[0046] Step 1: Dataset collection and processing

[0047] By screening and sorting from the static gesture handpose_x_gesture_v1 dataset, and then using the LabelImg annotation tool to manually annotate the gesture images, converting the image data into the YOLO dataset format containing the target category and target position coordinates, and using data augmentation methods such as rotation, adding noise and blurring to expand the images to 5500, as the dataset available for neural network training. The gesture dataset is divided into a training set, a validation set and a test set according to the ratio of 7:2:1.

[0048] Step 2: Construct the lightweight object detection module GB-C2f

[0049] Figure 2 The figure shows the structural diagram of the lightweight object detection module GB-C2f. Figure 3 It is the diagram of the Ghost Conv module. Figure 4 It is the structural diagram of Ghost Bottleneck:

[0050] (1) First, the Ghost Conv module is designed. This module first performs a standard convolution operation with a stride of 1 on the input feature map using a 1×1 convolutional kernel, effectively halving the number of channels of the feature map, thus maintaining key information while reducing the computational load. Then, a 5×5 convolutional kernel is used to perform a standard convolution with a stride of 1 on the previously generated feature map, and the result is concatenated with the feature map obtained in the previous step in the channel dimension to restore the original number of feature channels.

[0051] (2) Replace the Conv in the Bottleneck structure with Ghost Conv to form a new Ghost Bottleneck module. The Ghost Bottleneck module combines the advantages of multiple convolutional layers and shortcut connections, aiming to improve the efficiency and performance of the network. This module is mainly composed of two stacked Ghost Conv modules. The first Ghost Conv module serves as an expansion layer, which not only increases the number of channels but also controls the computational load and the number of parameters through the design of Ghost Conv, avoiding excessive computational burden. The second Ghost module reduces the number of channels of the output feature map to match the number of input channels, ensuring the smooth connection between the input and output of the module and maintaining the continuity and consistency of the network structure. Between these two Ghost Conv modules, a Shortcut connection is used, and the input feature map can be directly added to the output feature map, thus retaining more original information and accelerating the convergence speed of the network. In addition, after the second Ghost Conv module processes the feature information, no ReLU activation operation is performed. While batch normalization and ReLU non-linear activation are used after other layers process the feature information.

[0052] (3) Replace the Bottleneck structure in the C2f module with the Ghost Bottleneck structure to make full use of the advantages of GhostNet in reducing computational complexity and the number of parameters. The GB-C2f module significantly reduces the use of 3×3 ordinary convolutions, thus significantly compressing the model size. It not only reduces the number of parameters and the computational load but also enables the model to run more efficiently in resource-constrained environments. The GB-C2f module reduces the floating-point operation amount of the model by 27.2% and the number of parameters by 30.2%.

[0053] Among them, the experimental results of adding the GB-C2f module to YOLOv8n are compared as shown in Table 1.

[0054] Table 1 Comparison of Experimental Results

[0055] Model mAP GFLOPs Parameters YOLOv8 96.8 8.1 3,006,623 YOLOv8 + GB - C2f 96.5 5.9 2,099,611

[0056] Step 3: Construct the fusion feature pyramid network F-FPN

[0057] Figure 5 The structure diagram of the fusion feature pyramid network F-FPN is shown as follows:

[0058] (1) First, extract the features of the 4th, 6th, and 9th layers from the backbone network to obtain a set of features {C3, C4, C5} at different scales; then perform feature fusion. First, input the low-level feature C3 and the middle-level feature C4 into the feature pyramid network for preliminary fusion, and then add the high-level feature C5 to it; after the feature fusion is completed, use the GB-C2f unit to learn the features and generate a set of multi-scale features {P3, P4, P5}.

[0059] (2) To align the dimensions of the feature maps and prepare for feature fusion, use 1×1 convolution and bilinear interpolation methods to upsample the low-resolution feature maps, and use 2×2 convolution with a stride of 2 for 2 downsamplings and 4×4 convolution with a stride of 4 for 4 downsamplings on the high-resolution feature maps.

[0060] (3) Since feature fusion at each spatial position may cause conflicts in multi-object information, introduce the ASFF operation to alleviate these inconsistencies. Moreover, F-FPN optimizes the model by reducing the network width to one-fourth of the original, reducing the model parameter quantity while ensuring the accuracy. F-FPN reduces the model's floating-point operation amount by 24.7%, the parameter quantity by 25.5%, and increases the mAP by 0.6%.

[0061] Among them, the experimental results of adding the F-FPN module to YOLOv8n are compared as shown in Table 2.

[0062] Table 2 Comparison of Experimental Results

[0063] Model mAP GFLOPs Parameters YOLOv8 96.8 8.1 3,006,623 YOLOv8 + F - FPN 97.4 6.1 2,240,580

[0064] Step 4: Construct the lightweight and high-precision object detection network GA-YOLO

[0065] First, two Ghost Conv modules are stacked to form a GhostBottleneck module. By replacing the Bottleneck module in the original C2f with the GhostBottleneck module, a lightweight GB-C2f module is redesigned to compress the model and reduce the computational load. Second, to compensate for the reduction in detection accuracy due to lightweighting, an F-FPN network is proposed based on the ASFF and AFPN algorithms and used to replace the FPN in the original YOLOv8. Adaptive and progressive feature fusion is employed to enable better interaction between non-adjacent levels, extract more useful information, improve the model's perception ability of gesture features, and reduce the model complexity. The floating-point operation volume of the GA-YOLO network is reduced by 46.9%, and the number of parameters is reduced by 44.9%.

[0066] Among them, the comparison of the training experiment results of the GA-YOLO network is shown in Table 3.

[0067] Table 3 Comparison of the training experiment results of the GA-YOLO network

[0068]

[0069]

[0070] Step 5: Model training

[0071] The lightweight improved GA-YOLO network is used for model training. Set the number of training rounds to 400, the input image size to 640×640, and save the best model during the training process. After the model training is completed, save the model. Among them, after 400 rounds of training, experimental results and two weight files will be obtained, namely the weight file with the best training and the weight file of the last training. Import the obtained weight files into the system to detect pictures or videos, mark the gesture positions and gesture categories, and finally display the detection results on the display window. The display window will recognize the made gestures and frame them with a rectangular box. The gesture category and recognition confidence are displayed in the upper left corner of the box.

[0072] Therefore, it can be seen that the technical solution in this embodiment solves problems such as complex backgrounds and limited hardware computing power existing in visual recognition algorithms, solves the problem that the wearable device recognition method cannot be separated from the support of external devices, and can achieve efficient and accurate human-computer interaction. A static gesture dataset is selected and expanded, and mainstream object detection algorithms are evaluated and compared. Finally, YOLOv8n is selected as the basic network and improved for training.

[0073] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other.

[0074] Specific examples are used in this article to illustrate the principles and implementation manners of the present invention. The descriptions of the above embodiments are only used to help understand the core idea of the present invention. At the same time, for those of ordinary skill in the art, based on the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation on the present invention.

Claims

1. A lightweight gesture recognition method, characterized in that, Including: Obtain the data to be detected; the data to be detected includes image data and video data; Input the gesture to be detected into the trained gesture recognition model for prediction, and output the gesture position and gesture category of the current frame; the gesture categories include one gesture, five gesture, fist gesture, ok gesture, and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n basic network, the GB-C2f object detection module, and the F-FPN fusion feature pyramid network; The construction process of the gesture recognition model includes: First, construct the GB-C2f object detection module: Replace the original Bottelneck structure in the GB-C2f object detection module with the Ghost Bottelneck structure. The Ghost Bottelneck structure includes two stacked Ghost Conv modules, and a Shortcut connection is used between the two Ghost Conv modules. The Ghost Conv module includes a 1×1 convolutional kernel and a 5×5 convolutional kernel connected in sequence, and the output result of the 5×5 convolutional kernel is concatenated with the output result of the previous 1×1 convolutional kernel in channels; Then, construct the F-FPN fusion feature pyramid network: Propose the F-FPN fusion feature pyramid network based on the ASFF algorithm and the AFPN algorithm, and replace the FPN layer in the YOLOv8n basic network with the F-FPN fusion feature pyramid network; The specific construction process of the GB-C2f object detection module is: First, perform a standard convolution operation with a 1×1 convolutional kernel on the input data with a stride of 1 to halve the number of channels of the input data. Then, perform a standard convolution with a 5×5 convolutional kernel on the output data generated in the previous step with a stride of 1, and concatenate the convolution result with the output data obtained in the previous step in channels to restore the original number of feature channels, obtaining the GhostConv module; Replace the Conv module in the original Bottleneck structure with the Ghost Conv module to form a new GhostBottleneck structure; the Ghost Bottleneck structure includes two stacked Ghost Conv modules. Among them, the first Ghost Conv module serves as an expansion layer, and the second Ghost Conv module is used to reduce the number of channels of the output data to match the number of input channels; between the two Ghost Conv modules, a Shortcut connection is used, and after the second Ghost Conv module processes the feature information, no ReLU activation operation is performed; Replace the original Bottelneck structure with the Ghost Bottelneck structure to obtain the lightweight GB-C2f object detection module; The operation process of the F-FPN fusion feature pyramid network is: Extract the features of the 4th, 6th, and 9th layers from the backbone network to obtain a set of features at different scales; before feature fusion, in order to align the feature dimensions and prepare for feature fusion, use 1×1 convolution and bilinear interpolation methods to upsample the low-resolution features, and use 2×2 convolution with a stride of 2 for 2 downsamplings and 4×4 convolution with a stride of 4 for 4 downsamplings on the high-resolution feature map. Use the ASFF algorithm to perform consistency processing on the sampling data for feature fusion at different spatial positions, and then perform feature fusion. First, input the low-level features of the 4th layer and the middle-level features of the 6th layer into the feature pyramid network for preliminary feature fusion, and then add the high-level features of the 9th layer to the preliminary fusion result for further feature fusion; after each feature fusion is completed, use the GB-C2f object detection module to learn the features, generate a set of multi-scale features, and complete the recognition task according to the features.

2. The lightweight gesture recognition method according to claim 1, wherein The training process of the gesture recognition model is as follows: Screen and organize the static gesture handpose_x_gesture_v1 dataset, manually annotate the data after preliminary screening, and perform rotation, noise addition, and blurring processing on the annotated gesture data to obtain training data; the training data includes gesture images and corresponding recognition results. Construct a pre-trained model. Input the training data into the pre-trained model for parameter optimization, and determine the model that meets the set number of training epochs and accuracy as the trained gesture recognition model.

3. A lightweight gesture recognition system, applied to the method described in any one of claims 1-2, characterized in that, It includes: A data acquisition unit for obtaining data to be detected. The data to be detected includes picture data and video data. A gesture recognition unit for inputting the gesture to be detected into the trained gesture recognition model for prediction, and outputting the gesture position and gesture category of the current frame; the gesture categories include one gesture, five gesture, fist gesture, ok gesture, and yeah gesture; the gesture recognition model is constructed based on the YOLOv8n backbone network, GB-C2f object detection module, and F-FPN fusion type feature pyramid network.

4. An electronic device, characterized in that, It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the lightweight gesture recognition method according to any one of claims 1-2.

5. A computer-readable storage medium, characterized in that, It stores a computer program, and when the computer program is executed by the processor, it implements the lightweight gesture recognition method according to any one of claims 1-2.

Images