Gesture recognition method, system and device and storage medium

Abstract

The invention provides a gesture recognition method, system and device and a storage medium, and belongs to the technical field of deep learning, and the method comprises the steps: constructing a gesture recognition model which comprises a feature extraction module, a parallel attention module, a multi-scale feature aggregation module and a detection head which are connected in sequence; and obtaining a to-be-recognized gesture image, inputting the to-be-recognized image into the trained gesture recognition model, and recognizing and positioning gesture targets of different scales. In complex scenes of multi-gesture coexistence and the like, the precision and robustness of gesture recognition and the distinguishing capability of fine actions are remarkably improved. The problem of insufficient feature robustness and discrimination caused by complex backgrounds, illumination changes and inter-class similarity is solved.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to a gesture recognition method, system, device, and storage medium. Background Technology

[0002] Against the backdrop of a fundamental era of change in human-computer interaction, gesture recognition, as an intuitive and efficient method of human-computer interaction, has developed out of the need for the evolution of human-computer interaction paradigms from traditional command lines and graphical interfaces to more natural haptic interactions. It is a crucial touchstone for advancing artificial intelligence's perception capabilities to human levels, and a powerful means to achieve technological inclusion and promote social inclusion. Therefore, gesture recognition research has extremely important strategic value and practical significance. However, relying on expensive peripheral devices such as data gloves to directly acquire the angle and position information of hand joints through built-in sensors, while highly accurate, is costly and restrictive to users, hindering widespread adoption.

[0003] In the field of gesture recognition research, existing multi-view skeleton feature fusion methods improve the model's discrimination ability by integrating hand information from different sources or perspectives, or by using convolutional neural networks to extract spatial features of the hand skeleton from different perspectives. These methods primarily focus on the local movements and spatial relationships of hand joints, lacking the integration and processing of semantic information about gesture actions. This results in the model struggling to maintain detection accuracy for targets of different scales in complex scenes with varying backgrounds and lighting conditions, leading to insufficient recognition capabilities. Summary of the Invention

[0004] To address the shortcomings of existing deep learning methods for gesture recognition, this invention provides a gesture recognition method, system, device, and storage medium.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A gesture recognition method includes the following steps: Acquire the image of the gesture to be recognized; The image to be recognized is input into the trained gesture recognition model to identify and locate gesture targets at different scales; the gesture recognition model includes a feature extraction module, a parallel attention module, a multi-scale feature aggregation module, and a detection head connected in sequence; The feature extraction module processes the image to be identified, extracting features step by step to obtain multiple feature maps at different scales. The feature maps at different scales are then input into a parallel attention module for feature enhancement. The enhanced multi-scale features are then input into a multi-scale feature aggregation module to perform channel splicing and adaptive weight learning on features at different levels, resulting in multi-scale fused features. The multi-scale features are then input into the detection head to obtain the gesture target and position at different scales.

[0006] Preferably, the feature extraction module uses ResNet18 as the backbone feature extraction network to process the input image, extracts features step by step using convolutional layers, and generates multiple feature maps of different scales.

[0007] Preferably, the parallel attention module includes a spatial attention submodule and a channel attention submodule; the spatial attention submodule generates a spatial attention weight map by calculating the similarity matrix between any two positions in the feature map; the channel attention submodule generates a channel attention weight vector by calculating the correlation between channels, enhances features at different scales, and then fuses the enhanced features of spatial attention and channel attention, connects them to the residual inverted MLP, and obtains the enhanced multi-scale features.

[0008] Preferably, the multi-scale feature aggregation module sequentially employs global average pooling (GAP), convolutional layers, and activation functions to perform channel concatenation and adaptive weight learning on features at different levels.

[0009] Preferably, the method further includes introducing the WIoU v3 dynamic bounding box loss function during the training of the gesture recognition model, specifically: ; in, As a dynamic adjustment factor, For Wise-IoU loss, To focus parameters, For intersection, union, and comparison.

[0010] Preferably, the original gesture images are obtained based on the gesture dataset HaGRID, the original gesture images are downsampled, and then labeled using LabelImg to label the category label and bounding box information of each gesture instance to obtain a sequence dataset; the gesture recognition model is trained using the data in the training dataset to obtain the trained gesture recognition model.

[0011] The present invention also provides a gesture recognition system, specifically comprising: The data module is used to acquire images of the gestures to be recognized.

[0012] The gesture recognition module is used to input the image to be recognized into the trained gesture recognition model to recognize and locate gesture targets at different scales; the gesture recognition model includes a feature extraction module, a parallel attention module, a multi-scale feature aggregation module, and a detection head connected in sequence. The feature extraction module processes the image to be identified, extracting features step by step to obtain multiple feature maps at different scales. The feature maps at different scales are then input into a parallel attention module for feature enhancement. The enhanced multi-scale features are then input into a multi-scale feature aggregation module to perform channel splicing and adaptive weight learning on features at different levels, resulting in multi-scale fused features. The multi-scale features are then input into the detection head to obtain the gesture target and position at different scales.

[0013] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps described in the gesture recognition method.

[0014] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute the steps described in the gesture recognition method.

[0015] The gesture recognition method provided by this invention has the following beneficial effects: This invention utilizes ResNet18 as the backbone feature extraction network to obtain basic feature maps containing spatial details and semantic information at different levels. A parallel attention module captures cross-regional positional dependencies, achieving global correlation of hand features, optimizing channel-dimensional feature responses, adaptively focusing on key hand regions, and suppressing background interference. A multi-scale feature aggregation module fully integrates high-level and low-level feature information, improving the ability of existing deep features to reduce information dilution during feature fusion and enhancing the model's sensitivity to subtle gesture differences and multi-scale adaptability. By outputting gesture target detection results at different scales through a detection head, the invention accurately identifies gesture targets of different sizes and distances in the same image, enhancing the model's ability to detect gestures of different scales in complex scenes. Attached Figure Description

[0016] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a network architecture diagram of the gesture recognition model in an embodiment of the present invention.

[0018] Figure 2 This is a structural diagram of the Parallel Attention Module (PAM) in an embodiment of the present invention.

[0019] Figure 3This is a structural diagram of the multi-scale feature aggregation module MSFA in an embodiment of the present invention.

[0020] Figure 4 This is a flowchart of a gesture recognition method according to an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0022] Example This invention provides a gesture recognition method, such as Figure 1 As shown, the specific steps include: S1. Obtain original gesture images based on the HaGRID gesture dataset. Each gesture image includes gesture recognition objects at different distance scales. Downsample the original gesture images to a resolution of 640×640. Use the LabelImg tool to annotate the images, and annotate the category label and bounding box information of each gesture instance in PASCAL VOC format.

[0023] 13,500 images were selected from the preprocessed images and divided into training, validation and test sets in an 8:1:1 ratio.

[0024] S2. Construct a gesture recognition model, which includes a feature extraction module, a parallel attention module, a multi-scale feature aggregation module, and a detection head connected in sequence. The feature extraction module uses ResNet18.

[0025] S3. Input the training set image data into the gesture recognition model for training.

[0026] Initial setup: Configure the CUDA 11.8 graphics driver, torch 2.0.1, and scientific computing and image processing libraries on the Windows system. Configure the network hyperparameters, setting the initial learning rate to 0.001, batch size to 16, training epochs to 100, and using a cosine annealing learning rate strategy.

[0027] The gesture recognition model processes image data in the following steps: (1) The image is scaled proportionally to the standard size of 640×640 through the input layer.

[0028] (2) ResNet18 was used as the backbone feature extraction network to process the input image. Features were extracted step by step using convolutional layers to generate four basic feature maps (C2, C3, C4, C5) at different scales, which contain spatial details and semantic information at different levels. Among them, shallow features (such as C2) retain rich hand edge and texture details, which is beneficial for accurate localization; deep features (such as C5) contain stronger semantic information, which is helpful for the discrimination of gesture categories.

[0029] (3) Input the basic feature maps of different scales into the parallel attention module (PAM), which includes a spatial attention submodule and a channel attention submodule. The spatial attention submodule generates a spatial attention weight map by calculating the similarity matrix between any two positions in the feature map, which is used to model long-range spatial dependencies and enhance the feature responses of key parts of the gesture. The channel attention submodule generates a channel attention weight vector by calculating the correlation between channels, which is used to adaptively recalibrate the importance of each feature channel. By enhancing features at different scales, the spatial attention submodule captures the spatial dependencies of key regions of the gesture, and the channel attention submodule adaptively calibrates the weight distribution of each channel, thus suppressing background interference while highlighting the feature responses most relevant to gesture discrimination. The structure of the parallel attention module PAM is as follows: Figure 3 As shown.

[0030] Spatial attention module, given local features The input convolutional layer produces two new feature maps. Then, through vectorization operations, Transform into ,in The similarity matrix is ​​obtained by multiplying the transposes of feature map B and feature map C to represent the number of pixels in the spatial range. The spatial attention matrix is ​​calculated after passing through the softmax layer. ,in Measurement of the first Position relative to the first The influence of each position is calculated using the following formula: ; Generate a spatial weight map using the sigmoid function. And reshape the weight graph to Multiply the features at all locations by the scale parameter. With original features Weighted summation yields the first... eigenvectors of columns , This is the scale parameter, initially set to 0. The calculation formula is as follows:

[0031] ; ; The channel attention module directly draws from the input feature map Remodeling ,Will and Multiplying the transposes of the matrices yields the channel similarity matrix. ,right Each row is processed by softmax to obtain the channel attention matrix. ,in, Measurement of the first Position relative to the first The influence of each position. The calculation formula is as follows:

[0032] ; The sigmoid function is used to normalize the weights to a range of 0 to 1, generating a spatial weighted graph. And reshape the weight graph to Multiply the features at all locations by With original features The multiplication operation yields the final output. The formula is as follows:

[0033] ; ; Finally, the spatial and channel attention features are added element-wise to obtain the fused features. The calculation formula is as follows:

[0034] ; To prevent gradient vanishing and network degradation to some extent, a residual inverted MLP is finally connected, effectively capturing global and local information at different levels. This avoids gradient vanishing and network degradation, ensuring that deep networks can stably learn high-quality features that fuse global and local data, further enhancing feature representation capabilities. First, a 1×1 convolution is used to expand the number of feature channels by four times, and a non-linear activation function GELU is introduced to improve the representation capacity of the feature space.

[0035] ; ; Then, 3×3 convolution and GELU were used for local space extraction.

[0036] ; Finally, a 1×1 convolution is used to compress the number of channels back to the original dimensions, and a residual connection is made with the module input to obtain the final output. .

[0037] .

[0038] (4) To address the issue of insufficient feature representation caused by the gradual dilution of information during feature fusion in deep features, a multi-scale feature aggregation module (MSFA) is adopted, with the structure as follows: Figure 1 As shown, MSFA enables the full fusion of high-level and low-level features, providing ample target location information for high-level features and rich semantic information for low-level features. This enhances the model's sensitivity to subtle gesture differences and its multi-scale adaptability.

[0039] The parallel attention module outputs four feature maps of size 1. The four feature channel maps are concatenated to obtain... .

[0040] ; Then The branch performs global average pooling to compress spatial information, and then generates an initial weight vector through 1×1 convolution and activation function.

[0041] ; ; To ensure that the weight coefficients fall within the [0,1] interval, they are typically normalized using the Sigmoid function to obtain adaptive weight coefficients for different levels. The calculation process is as follows: ; Then, F2 is multiplied by F1 to obtain feature maps with different weight coefficients. : .

[0042] (5) Based on the fused features, a multi-scale detection head structure is adopted as the classifier. The detection head adopts a downsampling structure and fuses multi-scale features through top-down paths and lateral connections to perform gesture category prediction and gesture position estimation, and detect small, medium and large-sized gesture targets. Target detection and classification are performed on three image sizes: 20×20, 40×40 and 80×80, to achieve accurate localization and recognition of gestures at different scales and ensure the robustness of the model in large-scale changing scenarios.

[0043] By introducing a dynamic non-monotonic focusing mechanism loss function The prediction results are optimized using a differentiated weighting method. This strategy effectively suppresses harmful gradients generated by low-quality samples while reducing the competitive advantage of high-quality anchor frames.

[0044] First, define the predicted bounding box. and the true bounding box ,in Indicates the coordinates of the bounding box center. and These represent the width and length, respectively.

[0045] ; ; Area of ​​the intersection region between the predicted bounding box and the ground truth bounding box The calculation formula is as follows.

[0046] ; Calculate the area of ​​the union region between the predicted bounding box and the ground truth bounding box. The calculation formula is as follows.

[0047] ; ; ; Intersection over Union (IoU) is used to measure the predicted bounding box. With real frame degree of overlap: ; Since low-quality samples inevitably exist in training data, traditional geometric penalty terms based on distance or aspect ratio tend to over-penalize these samples, leading to a decrease in the model's generalization ability. Therefore, a WIoU penalty term is introduced. This design aims to reduce excessive intervention in training, thereby improving the model's generalization performance.

[0048] ; in, The width and height of the minimum bounding rectangle. The coordinates of the center of the predicted bounding box. These are the coordinates of the center of the true bounding box.

[0049] Based on this, a Wise-IoU loss is constructed, where Significantly magnified ordinary quality anchor boxes .

[0050] ; To ensure non-monotonicity, a dynamic adjustment factor is introduced. ,in Current number of training iterations, Total number of iterations For smoothing parameters.

[0051] ; Finally, the dynamic non-monotonic focusing loss is defined as ,in This is a focusing parameter used to adjust the level of attention given to difficult samples.

[0052] ; An end-to-end optimization strategy is adopted during training, and gesture classification and bounding box regression are learned as a joint task, which enables the network to extract more discriminative multi-scale features. Ultimately, high-precision and robust gesture recognition is achieved in challenging scenarios such as complex backgrounds and coexistence of multiple gestures.

[0053] S4. The performance of the trained gesture recognition model is evaluated using the test set, and the recognition accuracy of the model is quantitatively verified using the precision, recall, and mAP metrics.

[0054] The present invention also provides a gesture recognition system, comprising: The data module is used to acquire images of the gestures to be recognized.

[0055] The gesture recognition module is used to input the image to be recognized into the trained gesture recognition model to identify and locate gesture targets at different scales. The gesture recognition model includes a feature extraction module, a parallel attention module, a multi-scale feature aggregation module, and a detection head connected in sequence. The feature extraction module processes the image to be recognized, extracting features step by step to obtain feature maps at multiple different scales. The feature maps at multiple different scales are then input into the parallel attention module for feature enhancement. The enhanced multi-scale features are then input into the multi-scale feature aggregation module to perform channel splicing and adaptive weight learning on features at different levels, resulting in multi-scale fused features. The multi-scale features are then input into the detection head to obtain the gesture target and position at different scales.

[0056] The various modules in the aforementioned gesture recognition system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0057] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps in an embodiment of a gesture recognition method. Specific implementation methods can be found in the method embodiments, and will not be repeated here.

[0058] Furthermore, the present invention also provides a non-transitory computer-readable storage medium containing instructions on which a computer program is stored. For example, a memory containing instructions that can be executed by a processor of a computer device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. When the computer program is executed by the processor, it can implement the steps in an embodiment of a gesture recognition method. Specific implementation methods can be found in the method embodiments, which will not be repeated here.

[0059] Those skilled in the art will understand that embodiments of the present invention can provide methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0063] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the present invention patent. No reference numerals in the claims should be construed as limiting the scope of the claims. Any simple variations or equivalent substitutions of technical solutions that can be readily obtained by those skilled in the art within the scope of the technology disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A gesture recognition method, characterized in that, Includes the following steps: Acquire the image of the gesture to be recognized; The image to be recognized is input into the trained gesture recognition model to identify and locate gesture targets at different scales; the gesture recognition model includes a feature extraction module, a parallel attention module, a multi-scale feature aggregation module, and a detection head connected in sequence; The feature extraction module processes the image to be identified, extracting features step by step to obtain multiple feature maps at different scales. The feature maps at different scales are then input into a parallel attention module for feature enhancement. The enhanced multi-scale features are then input into a multi-scale feature aggregation module to perform channel splicing and adaptive weight learning on features at different levels, resulting in multi-scale fused features. The multi-scale features are then input into the detection head to obtain the gesture target and position at different scales.

2. The gesture recognition method according to claim 1, characterized in that, The feature extraction module specifically uses ResNet18 as the backbone feature extraction network to process the input image, extracting features step by step using convolutional layers to generate multiple feature maps of different scales.

3. The gesture recognition method according to claim 1, characterized in that, The parallel attention module includes a spatial attention submodule and a channel attention submodule; The spatial attention submodule generates a spatial attention weight map by calculating the similarity matrix between any two locations in the feature map; The channel attention submodule generates channel attention weight vectors by calculating the correlation between channels, enhances features at different scales, and then fuses the enhanced features of spatial attention and channel attention, connects them to the residual inverted MLP, and obtains the enhanced multi-scale features.

4. The gesture recognition method according to claim 1, characterized in that, The multi-scale feature aggregation module sequentially employs global average pooling (GAP), convolutional layers, and activation functions to perform channel concatenation and adaptive weight learning on features at different levels.

5. The gesture recognition method according to claim 1, characterized in that, It also includes introducing the WIoU v3 dynamic bounding box loss function during the training of the gesture recognition model, specifically: ； in, As a dynamic adjustment factor, For Wise-IoU loss, To focus parameters, For intersection, union, and comparison.

6. The gesture recognition method according to claim 1, characterized in that, The original gesture images are obtained based on the gesture dataset HaGRID. The original gesture images are downsampled and labeled using LabelImg to label the category label and bounding box information of each gesture instance, thus obtaining a sequence dataset. The gesture recognition model is trained using the data in the training dataset to obtain the trained gesture recognition model.

7. A gesture recognition system, characterized in that, include: The data module is used to acquire the image of the gesture to be recognized; The gesture recognition module is used to input the image to be recognized into the trained gesture recognition model to recognize and locate gesture targets at different scales; the gesture recognition model includes a feature extraction module, a parallel attention module, a multi-scale feature aggregation module, and a detection head connected in sequence. The feature extraction module processes the image to be identified, extracting features step by step to obtain multiple feature maps at different scales. The feature maps at different scales are then input into a parallel attention module for feature enhancement. The enhanced multi-scale features are then input into a multi-scale feature aggregation module to perform channel splicing and adaptive weight learning on features at different levels, resulting in multi-scale fused features. The multi-scale features are then input into the detection head to obtain the gesture target and position at different scales.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is loaded by the processor, it is able to perform the steps of the method according to any one of claims 1 to 6.

Images