Gesture recognition method and system based on deep learning

Abstract

The present invention discloses a gesture recognition method and system based on deep learning, and the method includes gesture image acquisition, data preprocessing, establishing a gesture recognition model and gesture recognition. The present invention belongs to the field of gesture recognition, and specifically refers to a gesture recognition method and system based on deep learning. The scheme combines multi-scale convolution and angle adaptive convolution to dynamically adjust the feature integration at different scales to improve the recognition ability of complex gesture details; based on the introduction of spatial and channel attention mechanisms, the gesture area is automatically focused to suppress background interference, thereby improving the accuracy of gesture recognition; by integrating global features and context features, dynamic gesture changes are captured, the processing ability of dynamic gesture image sequences is improved, and the gesture recognition ability under complex backgrounds is improved; based on classification accuracy, reconstruction quality and detail enhancement design loss, the recognition ability of complex actions and detail features is improved, thereby improving the accuracy of final gesture recognition.

Description

Technical Field

[0001] The present invention relates to the field of gesture recognition technology, and specifically to a gesture recognition method and system based on deep learning. Background Art

[0002] Gesture recognition methods enable computers to recognize and understand gestures through various technical means, and are widely used in virtual reality, smart home, medical health, robot control and other fields. The key to gesture recognition lies in how to extract gesture features and how to accurately identify the target gesture in noise and background interference. However, general gesture recognition methods are unable to adapt to gesture images of different sizes and angles, resulting in poor accuracy of gesture recognition in complex environments; general gesture recognition methods have poor gesture recognition effects under changing backgrounds and low gesture recognition accuracy in scenes with continuous gesture movements. Summary of the invention

[0003] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides a gesture recognition method and system based on deep learning. In view of the fact that general gesture recognition methods cannot adapt to gesture images of different sizes and angles, resulting in poor accuracy of gesture recognition in complex environments, this solution combines multi-scale convolution and angle adaptation convolution to dynamically adjust feature integration at different scales to improve the recognition ability of complex gesture details; based on the introduction of spatial and channel attention mechanisms, the gesture area is automatically focused to suppress background interference, improve the recognition accuracy of the gesture area, and thus improve the accuracy of gesture recognition; in view of the fact that general gesture recognition methods have poor gesture recognition effect under changing backgrounds and low gesture recognition accuracy in scenes with continuous gesture actions, this solution integrates global features and context features, and combines 3D convolution with jump connections to capture dynamic gesture changes, improve the processing capability of dynamic gesture image sequences, and improve the gesture recognition capability under complex backgrounds. Based on classification accuracy, reconstruction quality and detail enhancement design loss, the recognition capability of complex actions and detail features is improved, thereby improving the accuracy of final gesture recognition.

[0004] The technical solution adopted by the present invention is as follows: The gesture recognition method based on deep learning provided by the present invention comprises the following steps:

[0005] Step S1: gesture image acquisition;

[0006] Step S2: gesture image optimization;

[0007] Step S3: Establishing a gesture recognition model;

[0008] Step S4: Gesture recognition.

[0009] Furthermore, in step S1, the gesture image acquisition is to acquire historical gesture recognition image data; the historical gesture recognition image data includes gesture image data and gesture type; and the gesture type is used as a label.

[0010] Furthermore, in step S2, the gesture image optimization specifically includes the following steps:

[0011] Step S21: construct a global feature integration subnetwork; receive gesture image data , use multi-scale convolution to extract features, adapt to gesture images of different sizes and action ranges, and add angle adaptation convolution to the multi-scale convolution to adapt to the gesture angle, expressed as: ; Where M is the feature map processed by angle-adaptive convolution; DAC(·) is angle-adaptive convolution, which is used to adapt the gesture angle; Conv(·) is the convolution operation; PReLU(·) is the PReLU activation function; The global features of the gesture are extracted through multi-scale receptive fields, and dynamic weights are introduced, which are expressed as: ;in, It is the dynamic weight calculated by Softmax; It is the output multi-scale angle adaptation feature map; , and They are to directly extract features from the image through convolution, extract features after 1 / 2 downsampling, and extract features after 1 / 4 downsampling; combined with spatial and channel attention, it is expressed as: ; ; Where SAM(·) and CAM(·) are spatial attention module and channel attention module respectively; Sigmoid is Sigmoid activation function; GAP(·) and GMP(·) are global average pooling and global maximum pooling respectively; Concat(·) is concatenation; the output feature DAU is expressed as: ; According to the saliency of the resolution gesture feature, the feature integration is dynamically adjusted, expressed as: ; ; Among them, SKFF is the integrated output of multi-scale dynamic features; V1, V2 and V3 are the significance scores of the original scale, downsampled 1 / 2 scale and downsampled 1 / 4 scale respectively; and is an upsampling operation; by enhancing attention layer by layer, the global features of the gesture area are optimized, expressed as: ; ;in, and It is the feature map of the previous layer; is the attention weight map after layer-by-layer fusion; CFFB is the output of the context feature integration submodule; the context feature integration module is introduced to generate a globally enhanced gesture image , expressed as: ; Where CFF(·) is a context feature integration module; FFM(·) is a multi-scale dynamic feature integration module;

[0012] Step S22: construct a local detail enhancement subnetwork; used to restore the detailed features of the fingertip contour and joint edge in the gesture image; receive the gesture image output after global enhancement, use the channel attention module to extract local features, highlight the details of the gesture area, and add an inter-layer fusion module in the local feature extraction process, which is expressed as: ; Where N is the initial local feature map of local feature extraction; ILF(·) is the inter-layer fusion module; Through the cascade of detail enhancement modules, the edge and local features of the gesture image are gradually refined, and a recursive multi-scale recovery mechanism is introduced, which is expressed as: ; Wherein, DRM(·) is the detail enhancement module; , is a multi-level channel attention module; RMSR(·) is a recursive multi-scale recovery mechanism; using multi-scale convolution kernels and pixel-level attention mechanism, the local details of the gesture are restored, which can be expressed as: ; Where N1 and N9 are the feature maps generated by N in the local detail enhancement subnetwork; LDRB(·) is the local detail enhancement block; The gesture image after detail enhancement is obtained , expressed as: ;

[0013] Step S23: loss function design; construct reconstruction loss , the difference between the enhanced result and the normal lighting gesture image is constrained by L1 loss, expressed as: ;in, is the reference gesture image; is the L1 norm; C(·) is the brightness contrast filter; α is the contrast perception weight; construct the perceptual loss , constraining the similarity between the enhanced image and the reference image in the feature space, expressed as: ; Where N2 is the number of feature layers; , and are the number of channels, height and width of the j-th layer features respectively; is the feature map; is the square of the L2 norm; K1 is the number of multi-scale feature layers; is the feature extraction function; β is the multi-scale perception weight; constructs the structural loss , the structural similarity of gesture images is improved by SSIM measurement, which is expressed as: ; Where T(·) is the texture extraction operation; γ is the texture consistency weight; SSIM(·) is the structural similarity index; For the global feature integration subnetwork, the loss function It is expressed as: ; For the local detail enhancement sub-network, the loss function It is expressed as: ;in, and is the loss weight.

[0014] Furthermore, in step S3, the establishment of the gesture recognition model specifically includes the following steps:

[0015] Step S31: Feature extraction; inputting optimized gesture image , including the results of global feature enhancement and local detail enhancement; multi-scale convolution is used to adapt to gestures and action ranges of different sizes, spatial features are extracted, and a dynamic context feature integration module DCF is introduced, which is expressed as: ; Introduce context enhancement features, expressed as: ;in, is the global feature map; is the context-enhanced feature output; DWConv is a deep separable convolution; a lightweight channel attention module is used to perform channel weighting on each convolution feature and perform global feature calibration; it is expressed as: ;in, is the global feature calibration output; is the Sigmoid activation function; It is a one-dimensional convolution operation; it fuses global space and channel information and is expressed as: ;in, It is the fused multimodal feature map;

[0016] Step S32: Dimensionality reduction of gesture features; introducing an attention-aware feature selection module to remove redundant information, expressed as: ; Use multi-scale deconvolution operation to restore the complete information of the feature, expressed as: ;in, is the output of the feature selection module; A-SelectNet is a convolutional network combined with ECA attention; M-RecNet is a multi-scale deconvolutional network; is the restored feature map;

[0017] Step S33: Sample classification; extracting high-dimensional spatial and temporal features of gesture images , expressed as: ;in, It is a 3D convolution; the gesture image sequence is regarded as a time series input; a skip connection is introduced after each convolution layer to retain multi-scale feature information, which is expressed as: ;in, is the skip connection output; the P-Net model is used to classify the gesture category, expressed as: ; Determine the gesture category by nearest neighbor calculation; where d(·) is the distance; c r is the rth category; x i is a sample of the skip connection output; is the embedded feature representation; is the Euclidean distance; is the balance coefficient; c j2 is the j2th category; R2 is the total category;

[0018] Step S34: Loss function construction; constructing classification loss function , expressed as: ; Where Y is the true category; M is the total number of samples; is the weight coefficient of contrast loss; l(·) is the contrast loss; is the j3rd sample; use the Adam optimizer to adjust the model parameters;

[0019] Step S35: model determination; the image data after gesture image optimization is randomly divided into a training set and a test set. When the gesture recognition model converges to the training set loss or reaches the maximum number of training times, the gesture recognition model training is completed. A classification threshold is set in advance. When the classification accuracy of the trained gesture recognition model for the test set is higher than the classification threshold, the gesture recognition model is established; otherwise, the data set is re-divided to adjust the initial gesture recognition model parameters and train again.

[0020] Furthermore, in step S4, the gesture recognition is based on the established gesture recognition model, and the gesture image data to be recognized is collected in real time, and is input into the gesture recognition model after image optimization, and the gesture type output by the model is used as the gesture recognition result.

[0021] The deep learning-based gesture recognition system provided by the present invention includes a gesture image acquisition module, a gesture image optimization module, a gesture recognition model establishment module and a gesture recognition module;

[0022] The gesture image acquisition module acquires historical gesture recognition image data and sends the data to the gesture image optimization module;

[0023] The gesture image optimization module enhances the global features and details of the gesture image through multi-scale convolution, angle-adaptive convolution, attention mechanism and context fusion technology; and sends the data to the gesture recognition model building module;

[0024] The gesture recognition model building module uses the extracted optimized gesture image features, combined with sample classification, spatiotemporal feature modeling and contrast learning, to build a gesture recognition model; and sends the data to the gesture recognition module;

[0025] The gesture recognition module recognizes gesture types of gesture images collected in real time based on the established gesture recognition model.

[0026] The beneficial effects achieved by the present invention using the above scheme are as follows:

[0027] (1) In view of the problem that general gesture recognition methods cannot adapt to gesture images of different sizes and angles, resulting in poor accuracy of gesture recognition in complex environments, this solution combines multi-scale convolution and angle-adaptive convolution to dynamically adjust the feature integration at different scales to improve the recognition ability of complex gesture details; based on the introduction of spatial and channel attention mechanisms, it automatically focuses on the gesture area, suppresses background interference, improves the recognition accuracy of the gesture area, and thus improves the accuracy of gesture recognition.

[0028] (2) In order to address the problems of poor gesture recognition effect in changing backgrounds and low gesture recognition accuracy in scenes with continuous gesture movements in general gesture recognition methods, this solution integrates global features and context features and combines 3D convolution with jump connections to capture dynamic gesture changes, improve the processing capabilities of dynamic gesture image sequences, and improve the gesture recognition capabilities in complex backgrounds. Based on the classification accuracy, reconstruction quality and detail enhancement design loss, it improves the recognition capabilities of complex movements and detail features, thereby improving the accuracy of final gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the flow of a gesture recognition method based on deep learning provided by the present invention;

[0030] Figure 2 A schematic diagram of a gesture recognition system based on deep learning provided by the present invention;

[0031] Figure 3 It is a schematic diagram of the process of step S3.

[0032] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0034] In the description of the present invention, it is necessary to understand that terms such as “upper”, “lower”, “front”, “back”, “left”, “right”, “top”, “bottom”, “inside” and “outside” indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred system or element must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0035] Example 1, see Figure 1 The present invention provides a gesture recognition method based on deep learning, which comprises the following steps:

[0036] Step S1: Gesture image acquisition: acquiring historical gesture recognition image data;

[0037] Step S2: gesture image optimization; enhance the global features and details of the gesture image through multi-scale convolution, angle-adaptive convolution, attention mechanism and context fusion technology;

[0038] Step S3: Establishing a gesture recognition model; using the extracted optimized gesture image features, combined with sample classification, spatiotemporal feature modeling and contrastive learning, to construct a gesture recognition model;

[0039] Step S4: gesture recognition: based on the established gesture recognition model, gesture type recognition is performed on the gesture images collected in real time.

[0040] Example 2, see Figure 1 This embodiment is based on the above embodiment. In step S1, the historical gesture recognition image data includes gesture images and gesture types; the gesture types are used as labels; the gesture types include basic gesture types, natural language gestures, body gestures and emotional gestures; the basic gesture types include clenching fists, opening palms and pointing; the natural language gestures include sign language gestures and digital gestures; the body gestures include arm waving and beckoning gestures; the emotional gestures include angry gestures, happy gestures and sad gestures.

[0041] Example 3, see Figure 1 This embodiment is based on the above embodiment. In step S2, gesture image optimization specifically includes the following steps:

[0042] Step S21: construct a global feature integration subnetwork; used to improve the overall brightness of the gesture image, enhance feature contrast, ensure the visibility of gesture contours and key joint parts; receive gesture image data , use multi-scale convolution to extract features, adapt to gesture images of different sizes and action ranges, and add angle adaptation convolution to the multi-scale convolution to adapt to the gesture angle, expressed as: ; Where M is the feature map processed by angle-adaptive convolution; DAC(·) is angle-adaptive convolution, which is used to adapt the gesture angle and improve the expressiveness in the feature direction; Conv(·) is the convolution operation; PReLU(·) is the PReLU activation function; The global features of the gesture are extracted through multi-scale receptive fields, and dynamic weights are introduced, which are expressed as: ;in, It is a dynamic weight calculated by Softmax, which ensures that small-scale features receive more attention in gesture recognition scenarios; It is the output multi-scale angle adaptation feature map; , and They are: directly extracting convolution features from the image, extracting features after 1 / 2 downsampling, and extracting features after 1 / 4 downsampling; combining spatial and channel attention, enhancing gesture area features and suppressing background interference, expressed as: ; ; Where SAM(·) and CAM(·) are spatial attention module and channel attention module respectively; Sigmoid is Sigmoid activation function; GAP(·) and GMP(·) are global average pooling and global maximum pooling respectively; Concat(·) is concatenation; the output feature DAU is expressed as: ; According to the saliency of the resolution gesture feature, the feature integration is dynamically adjusted, expressed as: ; ; Among them, SKFF is the integrated output of multi-scale dynamic features; V1, V2 and V3 are the significance scores of the original scale, downsampled 1 / 2 scale and downsampled 1 / 4 scale respectively; and is an upsampling operation; by enhancing the attention layer by layer, the global features of the gesture area are optimized, which can be expressed as: ; ;in, and It is the feature map of the previous layer; is the attention weight map after layer-by-layer fusion; CFFB is the output of the context feature integration submodule; the context feature integration module is introduced to generate a globally enhanced gesture image , expressed as: ; Where CFF(·) is a context feature integration module; FFM(·) is a multi-scale dynamic feature integration module;

[0043] Step S22: construct a local detail enhancement subnetwork; used to restore the detailed features of the fingertip contour and joint edge in the gesture image, and suppress noise to ensure high resolution of the gesture area; receive the gesture image output after global enhancement, use the channel attention module to extract local features, highlight the details of the gesture area, and add an inter-layer fusion module in the local feature extraction process, which is expressed as: ; Where N is the initial local feature map of local feature extraction; ILF(·) is the inter-layer fusion module; Through the cascade of detail enhancement modules, the edge and local features of the gesture image are gradually refined, and a recursive multi-scale recovery mechanism is introduced, which is expressed as: ; Wherein, DRM(·) is the detail enhancement module; , is a multi-level channel attention module; RMSR(·) is a recursive multi-scale recovery mechanism; using multi-scale convolution kernels and pixel-level attention mechanism, the local details of the gesture are restored, which can be expressed as: ; Where N1 and N9 are the feature maps generated by N in the local detail enhancement subnetwork; LDRB(·) is the local detail enhancement block; The gesture image after detail enhancement is obtained , expressed as: ;

[0044] Step S23: loss function design; construct reconstruction loss , the difference between the enhanced result and the normal lighting gesture image is constrained by L1 loss, expressed as: ;in, is the reference gesture image; is the L1 norm; C(·) is the brightness contrast filter; α is the contrast perception weight; construct the perceptual loss , constraining the similarity between the enhanced image and the reference image in the feature space, expressed as: ; Where N2 is the number of feature layers; , and are the number of channels, height and width of the j-th layer features respectively; is the feature map; is the square of the L2 norm; K1 is the number of multi-scale feature layers; is the feature extraction function; β is the multi-scale perception weight; constructs the structural loss , the structural similarity of gesture images is improved by SSIM measurement, which is expressed as: ; Where T(·) is the texture extraction operation; γ is the texture consistency weight; SSIM(·) is the structural similarity index; For the global feature integration subnetwork, the loss function It is expressed as: ; For the local detail enhancement sub-network, the loss function It is expressed as: ;in, and is the loss weight.

[0045] By performing the above operations, the general gesture recognition method cannot adapt to gesture images of different sizes and angles, resulting in poor accuracy of gesture recognition in complex environments. This solution combines multi-scale convolution and angle adaptive convolution to dynamically adjust the feature integration at different scales to improve the recognition ability of complex gesture details; based on the introduction of spatial and channel attention mechanisms, it automatically focuses on the gesture area, suppresses background interference, improves the recognition accuracy of the gesture area, and thus improves the accuracy of gesture recognition.

[0046] Example 4, see Figure 1 and Figure 3 This embodiment is based on the above embodiment. In step S3, establishing a gesture recognition model specifically includes the following steps:

[0047] Step S31: Feature extraction; inputting optimized gesture image , including the results of global feature enhancement and local detail enhancement; multi-scale convolution is used to adapt to gestures and action ranges of different sizes, spatial features are extracted, and a dynamic context feature integration module DCF is introduced, which is expressed as: ; Introduce context enhancement features, expressed as: ;in, is the global feature map; is the context-enhanced feature output; DWConv is a deep separable convolution; a lightweight channel attention module is used to perform channel weighting on each convolution feature and perform global feature calibration; it is expressed as: ;in, is the global feature calibration output; is the Sigmoid activation function; It is a one-dimensional convolution operation; it integrates global spatial and channel information to adapt to diverse gestures, expressed as: ;in, It is the fused multimodal feature map;

[0048] Step S32: Dimensionality reduction of gesture features; introducing an attention-aware feature selection module to remove redundant information, expressed as: ; Use multi-scale deconvolution operation to restore the complete information of the feature, expressed as: ;in, is the output of the feature selection module; A-SelectNet is a convolutional network combined with ECA attention; M-RecNet is a multi-scale deconvolutional network; is the restored feature map;

[0049] Step S33: Sample classification; extracting high-dimensional spatial and temporal features of gesture images , expressed as: ;in, It is a 3D convolution; the gesture image sequence is regarded as a time series input; a skip connection is introduced after each convolution layer to retain multi-scale feature information, which is expressed as: ;in, is the skip connection output; the P-Net model is used to classify the gesture category, expressed as: ; Determine the gesture category by nearest neighbor calculation; where d(·) is the distance; c r is the rth category; x i is a sample of the skip connection output; is the embedded feature representation; is the Euclidean distance; is the balance coefficient; c j2 is the j2th category; R2 is the total category;

[0050] Step S34: loss function construction; constructing classification loss function , expressed as: ; Where Y is the true category; M is the total number of samples; is the weight coefficient of contrast loss; l(·) is the contrast loss; is the j3th sample; use the Adam optimizer to adjust the model parameters;

[0051] Step S35: model determination; the image data after gesture image optimization is randomly divided into a training set and a test set. When the gesture recognition model converges to the training set loss or reaches the maximum number of training times, the gesture recognition model training is completed. A classification threshold is set in advance. When the classification accuracy of the trained gesture recognition model for the test set is higher than the classification threshold, the gesture recognition model is established; otherwise, the data set is re-divided to adjust the initial gesture recognition model parameters and train again.

[0052] By performing the above operations, in order to solve the problems of poor gesture recognition effect in changing backgrounds and low gesture recognition accuracy in scenes with continuous gesture actions in general gesture recognition methods, this solution integrates global features and context features and combines 3D convolution with jump connections to capture dynamic gesture changes, improve the processing capability of dynamic gesture image sequences, and improve the gesture recognition capability in complex backgrounds. Based on classification accuracy, reconstruction quality and detail enhancement design loss, this solution improves the recognition capability of complex actions and detail features, thereby improving the accuracy of final gesture recognition.

[0053] Example 5, see Figure 1 This embodiment is based on the above embodiment. In step S4, gesture recognition is based on the established gesture recognition model, and gesture image data to be recognized is collected in real time. After image optimization, it is input into the gesture recognition model, and the gesture type output by the model is used as the gesture recognition result.

[0054] Example 6, see Figure 2 , this embodiment is based on the above embodiment, and the gesture recognition system based on deep learning provided by the present invention includes a gesture image acquisition module, a gesture image optimization module, a gesture recognition model establishment module and a gesture recognition module;

[0055] The gesture image acquisition module acquires historical gesture recognition image data and sends the data to the gesture image optimization module;

[0056] The gesture image optimization module enhances the global features and details of the gesture image through multi-scale convolution, angle-adaptive convolution, attention mechanism and context fusion technology; and sends the data to the gesture recognition model building module;

[0057] The gesture recognition model building module uses the extracted optimized gesture image features, combined with sample classification, spatiotemporal feature modeling and contrast learning, to build a gesture recognition model; and sends the data to the gesture recognition module;

[0058] The gesture recognition module recognizes gesture types of gesture images collected in real time based on the established gesture recognition model.

[0059] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0060] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.

[0061] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A gesture recognition method based on deep learning, characterized in that: The method comprises the following steps: Step S1: Gesture image acquisition: acquiring historical gesture recognition image data; Step S2: gesture image optimization; Step S3: Establishing a gesture recognition model; Step S4: gesture recognition: based on the established gesture recognition model, gesture type recognition is performed on the gesture images collected in real time; Step S3 includes step S31: feature extraction; inputting the optimized gesture image , including the results of global feature enhancement and local detail enhancement; multi-scale convolution is used to adapt to gestures and action ranges of different sizes, spatial features are extracted, and a dynamic context feature integration module DCF is introduced, which is expressed as: ; Introduce context enhancement features, expressed as: ;in, is the global feature map; , and They are: directly extracting convolution features from the image, extracting features after 1 / 2 downsampling, and extracting features after 1 / 4 downsampling; is the context-enhanced feature output; DWConv is a depth-separable convolution; Conv(·) is a convolution operation; Concat(·) is concatenation; a lightweight channel attention module is used to perform channel weighting on each convolution feature and perform global feature calibration; it is expressed as: ;in, is the global feature calibration output; GAP(·) is the global average pooling; is the Sigmoid activation function; It is a one-dimensional convolution operation; it integrates global spatial and channel information to adapt to diverse gestures, expressed as: ;in, is the fused multimodal feature map; SAM(·) and CAM(·) are the spatial attention module and channel attention module respectively; In step S2, the gesture image optimization specifically includes the following steps: Step S21: construct a global feature integration subnetwork; receive gesture image data , use multi-scale convolution to extract features, adapt to gesture images of different sizes and action ranges, and add angle adaptation convolution to the multi-scale convolution to adapt to the gesture angle, expressed as: ; Where M is the feature map processed by angle-adaptive convolution; DAC(·) is angle-adaptive convolution, which is used to adapt the gesture angle; Conv(·) is the convolution operation; PReLU(·) is the PReLU activation function; The global features of the gesture are extracted through multi-scale receptive fields, and dynamic weights are introduced, which are expressed as: ;in, It is the dynamic weight calculated by Softmax; is the output multi-scale angle-adapted feature map; combined with spatial and channel attention, it is expressed as: ; ; Where SAM(·) and CAM(·) are spatial attention module and channel attention module respectively; Sigmoid is Sigmoid activation function; GAP(·) and GMP(·) are global average pooling and global maximum pooling respectively; Concat(·) is concatenation; the output feature DAU is expressed as: ; According to the saliency of the resolution gesture feature, the feature integration is dynamically adjusted, expressed as: ; ; Among them, SKFF is the integrated output of multi-scale dynamic features; V1, V2 and V3 are the significance scores of the original scale, downsampled 1 / 2 scale and downsampled 1 / 4 scale respectively; and is an upsampling operation; by enhancing attention layer by layer, the global features of the gesture area are optimized, expressed as: ; ;in, It is the feature map of the previous layer; is the attention weight map after layer-by-layer fusion; CFFB is the output of the context feature integration module; the context feature integration module is introduced to generate a globally enhanced gesture image , expressed as: ; Where CFF(·) is a context feature integration module; FFM(·) is a multi-scale dynamic feature integration module; Step S22: construct a local detail enhancement subnetwork; used to restore the detailed features of the fingertip contour and joint edge in the gesture image; receive the gesture image output after global enhancement, use the channel attention module to extract local features, highlight the details of the gesture area, and add an inter-layer fusion module in the local feature extraction process, which is expressed as: ; Where N is the initial local feature map of local feature extraction; ILF(·) is the inter-layer fusion module; Through the cascade of detail enhancement modules, the edge and local features of the gesture image are gradually refined, and a recursive multi-scale recovery mechanism is introduced, which is expressed as: ; Wherein, DRM(·) is the detail enhancement module; , is a multi-level channel attention module; RMSR(·) is a recursive multi-scale recovery mechanism; using multi-scale convolution kernels and pixel-level attention mechanism, the local details of the gesture are restored, which can be expressed as: ; Where N1 and N9 are the feature maps generated by N in the local detail enhancement subnetwork; LDRB(·) is the local detail enhancement block; The gesture image after detail enhancement is obtained , expressed as: ; Step S23: loss function design.

2. The gesture recognition method based on deep learning according to claim 1, characterized in that: In step S23, the loss function is designed to construct the reconstruction loss , the difference between the enhanced result and the normal lighting gesture image is constrained by L1 loss, expressed as: ;in, is the reference gesture image; is the L1 norm; C(·) is the brightness contrast filter; α is the contrast perception weight; construct the perceptual loss , constraining the similarity between the enhanced image and the reference image in the feature space, expressed as: ; Where N2 is the number of feature layers; , and are the number of channels, height and width of the j-th layer features respectively; is the feature map; is the square of the L2 norm; K1 is the number of multi-scale feature layers; is the feature extraction function; β is the multi-scale perception weight; constructs the structural loss , the structural similarity of gesture images is improved by SSIM measurement, which is expressed as: ; Where T(·) is the texture extraction operation; γ is the texture consistency weight; SSIM(·) is the structural similarity index; For the global feature integration subnetwork, the loss function It is expressed as: ; For the local detail enhancement sub-network, the loss function It is expressed as: ;in, and is the loss weight.

3. The gesture recognition method based on deep learning according to claim 2, characterized in that: In step S3, the step of establishing the gesture recognition model further includes the following steps: Step S32: Dimensionality reduction of gesture features; introducing an attention-aware feature selection module to remove redundant information, expressed as: ; Use multi-scale deconvolution operation to restore the complete information of the feature, expressed as: ;in, is the output of the feature selection module; A-SelectNet is a convolutional network combined with ECA attention; M-RecNet is a multi-scale deconvolutional network; is the restored feature map; Step S33: Sample classification; extracting high-dimensional spatial and temporal features of gesture images , expressed as: ;in, It is a 3D convolution; the gesture image sequence is regarded as a time series input; a skip connection is introduced after each convolution layer to retain multi-scale feature information, which is expressed as: ;in, is the skip connection output; the P-Net model is used to classify the gesture category, expressed as: ; Determine the gesture category by nearest neighbor calculation; where d(·) is the distance; c r is the rth category; x i is a sample of the skip connection output; is the embedded feature representation; is the balance coefficient; c j2 is the j2th category; R2 is the total category; Step S34: loss function construction; constructing classification loss function , expressed as: ; Where Y is the true category; M is the total number of samples; is the weight coefficient of contrast loss; l(·) is the contrast loss; is the j3th sample; use the Adam optimizer to adjust the model parameters; Step S35: model determination; the image data after gesture image optimization is randomly divided into a training set and a test set. When the gesture recognition model converges to the training set loss or reaches the maximum number of training times, the gesture recognition model training is completed. A classification threshold is set in advance. When the classification accuracy of the trained gesture recognition model for the test set is higher than the classification threshold, the gesture recognition model is established; otherwise, the data set is re-divided to adjust the initial gesture recognition model parameters and train again.

4. The gesture recognition method based on deep learning according to claim 1, characterized in that: In step S1, the historical gesture recognition image data includes gesture image data and gesture types; the gesture types are used as labels.

5. The gesture recognition method based on deep learning according to claim 1, characterized in that: In step S4, the gesture recognition is based on the established gesture recognition model, and the gesture image data to be recognized is collected in real time. After image optimization, it is input into the gesture recognition model, and the gesture type output by the model is used as the gesture recognition result.

6. A gesture recognition system based on deep learning, used to implement the gesture recognition method based on deep learning as described in any one of claims 1 to 5, characterized in that: It includes a gesture image acquisition module, a gesture image optimization module, a gesture recognition model building module and a gesture recognition module; The gesture image acquisition module acquires historical gesture recognition image data and sends the data to the gesture image optimization module; The gesture image optimization module enhances the global features and details of the gesture image through multi-scale convolution, angle-adaptive convolution, attention mechanism and context fusion technology; and sends the data to the gesture recognition model building module; The gesture recognition model building module uses the extracted optimized gesture image features, combined with sample classification, spatiotemporal feature modeling and contrast learning, to build a gesture recognition model; and sends the data to the gesture recognition module; The gesture recognition module recognizes gesture types of gesture images collected in real time based on the established gesture recognition model.

Images