Gesture recognition system and method based on LHGDet model

Abstract

The invention discloses a gesture recognition system and method based on an LHGDet model. The system comprises a data collection module, a model construction module, a model training module and a gesture recognition module. The data collection module is used for acquiring a gesture data image set and marking the gesture data image set to obtain training data; the model construction module is used for improving a NanoDet-Plus model to obtain an initial gesture recognition model; the model training module is used for training the initial gesture recognition model by using the training data to obtain an LHGDet gesture recognition model; and the gesture recognition module completes gesture recognition by using an LHGDet gesture recognition model. According to the method, the NanoDet-Plus model is improved by using a CA attention mechanism, an SE attention mechanism and an FF attention mechanism, so that the network can better capture a long-distance spatial dependency relationship in an image, and the detection capability on a small target is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and in particular relates to a gesture recognition system and method based on an LHGDet model. Background Art

[0002] In the field of human-computer interaction, gestures have many application scenarios, such as game control, smart home, real-time sign language translation, etc. Therefore, it is very important to detect and recognize gestures quickly and accurately. In recent years, gesture recognition (HGR) has attracted the research interest of more and more scholars. Since the end of the 20th century, research on gesture recognition interaction has begun, which is mainly divided into gesture recognition based on external devices and gesture recognition based on computer vision. Compared with gesture recognition based on external devices, gesture recognition based on computer vision is more convenient and flexible, and reduces equipment costs.

[0003] Traditional gesture recognition methods based on computer vision are divided into three stages: gesture detection and segmentation, gesture feature extraction, and gesture recognition. The skin color segmentation model is often used to segment the gesture area, and then the features are extracted based on the information obtained by the image processing algorithm, such as the gradient direction histogram algorithm, the scale-invariant feature transformation algorithm, the local binary pattern algorithm, etc., and then combined with machine learning classifiers, such as support vector machines and adaptive enhancement, for classification. However, this type of method has poor robustness and low recognition accuracy under complex backgrounds, and is only suitable for single-target recognition, while multi-target recognition is relatively difficult. In addition, the manual feature extraction process is cumbersome and complicated, relying on the experience of engineers themselves, and the application of the algorithm is relatively limited.

[0004] With the research and development of deep learning, deep learning-based methods have fast recognition speed and high accuracy, becoming the mainstream gesture recognition algorithm. Oyedotun et al. binarized the image, used threshold segmentation to obtain the gesture area, and used convolutional neural network and encoder for gesture recognition. Wang Long et al. used the skin color model for gesture detection, reconstructed the obtained gesture area into a grayscale image, and finally input the gesture area into CNN for feature extraction and recognition. Mohanty et al. did not perform gesture detection processing, and directly input the original image into a simple 2-layer CNN for feature extraction and recognition. In the above gesture recognition classification algorithm, when there is complex background interference such as human skin color in the image, the CNN recognition rate drops significantly.

[0005] With the development of target detection algorithms, many scholars have converted the classification problem of gesture recognition into a target detection problem, which is mainly divided into two types: a one-stage target detection method based on regression and a two-stage target detection method based on region recommendation. The main one-stage target detection algorithms include YOLO, RetinaNet, and CornerNet, and the main two-stage target detection algorithms include R-CNN, FastR-CNN, FasterR-CNN, and R-FCN. These algorithms have greatly improved the accuracy of gesture recognition, but due to limited memory and computing resources, it is difficult to deploy convolutional neural networks (CNNs) on embedded devices. Summary of the invention

[0006] The present invention aims to solve the deficiencies of the prior art and provides the following solutions:

[0007] A gesture recognition system based on the LHGDet model, comprising: a data collection module, a model building module, a model training module and a gesture recognition module;

[0008] The data collection module is used to obtain a gesture data image set and mark the gesture data image set to obtain training data;

[0009] The model building module improves the NanoDet-Plus model to obtain an initial gesture recognition model;

[0010] The model training module trains the initial gesture recognition model using the training data to obtain a LHGDet gesture recognition model;

[0011] The gesture recognition module uses the LHGDet gesture recognition model to complete gesture recognition.

[0012] Preferably, the data collection module includes: an image processing unit, a labeling unit and a data division unit;

[0013] The image processing unit is used to rotate and scale the gesture data image set to obtain a processed image set;

[0014] The labeling unit is used to label the processed image set according to different gesture types to obtain a labeled image set;

[0015] The data division unit is used to divide the annotated image set into a training set and a test set according to a preset ratio to obtain the training data.

[0016] Preferably, the initial gesture recognition model comprises: an improved backbone network, an improved neck network, an auxiliary training network and a detection network;

[0017] The method for obtaining the improved backbone network includes: adding SE attention mechanism and CA attention mechanism to the backbone network of the NanoDet-Plus model;

[0018] The method for obtaining the improved neck network includes: adding a FF attention mechanism to the neck network of the NanoDet-Plus model;

[0019] The auxiliary training network consists of 4 3×3 convolution kernels, using group normalization as the normalization layer;

[0020] The detection network is a detection head with shared weights.

[0021] Preferably, the improved backbone network includes: block1 and block2;

[0022] The block1 is composed of a first branch and a second branch, the first branch is composed of a 1×1 convolution kernel, a 3×3 convolution kernel, an SE attention mechanism, and a 1×1 convolution kernel connected in sequence, the second branch does not process the input data, and the outputs of the first branch and the second branch are connected through a splicing operation;

[0023] The block2 is composed of a third branch and a fourth branch. The third branch is composed of a 1×1 convolution kernel, a 3×3 convolution kernel and a 1×1 convolution kernel connected in sequence. The fourth branch is composed of a 3×3 convolution kernel, a CA attention mechanism and a 1×1 convolution kernel connected in sequence. The outputs of the third branch and the fourth branch are connected through a splicing operation.

[0024] Preferably, the improved neck network comprises: 2 upsampling layers, 2 FF attention mechanisms and 2 downsampling layers;

[0025] One upsampling, one FF attention mechanism and one splicing operation are sequentially connected to form one subunit, two subunits are connected end to end, and the output of the second subunit is spliced ​​with the output of the first subunit through one downsampling layer, and the result is spliced ​​with the network input through one downsampling layer.

[0026] The present invention also provides a gesture recognition method based on the LHGDet model, and the recognition method is applied to any of the above-mentioned application systems, comprising the following steps:

[0027] Acquire a gesture data image set, and mark the gesture data image set to obtain training data;

[0028] Improve the NanoDet-Plus model to obtain the initial gesture recognition model;

[0029] Using the training data to train the initial gesture recognition model to obtain a LHGDet gesture recognition model;

[0030] The LHGDet gesture recognition model is used to complete gesture recognition.

[0031] Preferably, the method for obtaining the training data includes:

[0032] Rotating and scaling the gesture data image set to obtain a processed image set;

[0033] Annotating the processed image set according to different gesture types to obtain an annotated image set;

[0034] The annotated image set is divided into a training set and a test set according to a preset ratio to obtain the training data.

[0035] Preferably, the initial gesture recognition model comprises: an improved backbone network, an improved neck network, an auxiliary training network and a detection network;

[0036] The method for obtaining the improved backbone network includes: adding SE attention mechanism and CA attention mechanism to the backbone network of the NanoDet-Plus model;

[0037] The method for obtaining the improved neck network includes: adding a FF attention mechanism to the neck network of the NanoDet-Plus model;

[0038] The auxiliary training network consists of 4 3×3 convolution kernels, using group normalization as the normalization layer;

[0039] The detection network is a detection head with shared weights.

[0040] Preferably, the improved backbone network includes: block1 and block2;

[0041] The block1 is composed of a first branch and a second branch, the first branch is composed of a 1×1 convolution kernel, a 3×3 convolution kernel, an SE attention mechanism, and a 1×1 convolution kernel connected in sequence, the second branch does not process the input data, and the outputs of the first branch and the second branch are connected through a splicing operation;

[0042] The block2 is composed of a third branch and a fourth branch. The third branch is composed of a 1×1 convolution kernel, a 3×3 convolution kernel and a 1×1 convolution kernel connected in sequence. The fourth branch is composed of a 3×3 convolution kernel, a CA attention mechanism and a 1×1 convolution kernel connected in sequence. The outputs of the third branch and the fourth branch are connected through a splicing operation.

[0043] Preferably, the improved neck network comprises: 2 upsampling layers, 2 FF attention mechanisms and 2 downsampling layers;

[0044] One upsampling, one FF attention mechanism and one splicing operation are sequentially connected to form one subunit, two subunits are connected end to end, and the output of the second subunit is spliced ​​with the output of the first subunit through one downsampling layer, and the result is spliced ​​with the network input through one downsampling layer.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention improves the NanoDet-Plus network for small targets such as gestures, and uses the CA attention mechanism and the SE attention mechanism to improve the backbone network, so that the network can better capture the long-distance spatial dependencies in the image and enhance the detection ability of small targets. The FF attention mechanism is embedded in the original PAN so that it can better integrate features of different scales, so that the present invention achieves good performance on multi-scale gestures with complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 A schematic diagram of the system structure of an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of the structure of an improved backbone network according to an embodiment of the present invention;

[0050] Figure 3 Schematic diagram of the structure of the SE attention mechanism of an embodiment of the present invention;

[0051] Figure 4 Schematic diagram of the structure of the CA attention mechanism of an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of the structure of the improved neck network according to an embodiment of the present invention

[0053] Figure 6 It is a structural diagram of the FF attention mechanism of an embodiment of the present invention;

[0054] Figure 7Schematic diagram of the structure of the LHGDet gesture recognition model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Embodiment 1

[0058] In this embodiment, if Figure 1 As shown, a gesture recognition system based on the LHGDet model includes: a data collection module, a model construction module, a model training module and a gesture recognition module.

[0059] The data collection module is used to obtain a gesture data image set and mark the gesture data image set to obtain training data.

[0060] The data collection module includes: an image processing unit, a labeling unit and a data division unit; the image processing unit is used to rotate and scale the gesture data image set to obtain a processed image set; the labeling unit is used to label the processed image set according to different gesture types to obtain a labeled image set; the data division unit is used to divide the labeled image set into a training set and a test set according to a preset ratio to obtain training data.

[0061] In this embodiment, the latest open-source ultra-large public gesture image dataset is used: the HaGRID dataset. This dataset consists of 500,000 RGB images of size 1920×1080, including 18 types of common gestures. This dataset is designed for human-computer interaction. The 18 types of gestures are all functional gestures and are widely used in daily life. At the same time, all images are different in background, lighting, scenes, and themes, and have good generalization. According to the actual application situation, this embodiment performs sample simplification and image resolution reduction on the original dataset to obtain the dataset HaGRID-tiny, which contains a total of 18,000 images, divided into 18 gestures such as call, peace, dislike, etc., and then divided into training set and test set in a ratio of 4:1 to obtain training data.

[0062] The model building module improves the NanoDet-Plus model to obtain the initial gesture recognition model.

[0063] The initial gesture recognition model includes: an improved backbone network, an improved neck network, an auxiliary training network and a detection network; the method for obtaining the improved backbone network includes: adding SE attention mechanism and CA attention mechanism to the backbone network of the NanoDet-Plus model; the method for obtaining the improved neck network includes: adding FF attention mechanism to the neck network of the NanoDet-Plus model; the auxiliary training network consists of 4 3×3 convolution kernels, using group normalization as the normalization layer; the detection network is a detection head with shared weights.

[0064] The improved backbone network includes: block1 and block2; block1 consists of the first branch and the second branch, the first branch consists of a 1×1 convolution kernel, a 3×3 convolution kernel, an SE attention mechanism and a 1×1 convolution kernel connected in sequence, the second branch does not process the input data, and the outputs of the first branch and the second branch are connected through a splicing operation; block2 consists of the third branch and the fourth branch, the third branch consists of a 1×1 convolution kernel, a 3×3 convolution kernel and a 1×1 convolution kernel connected in sequence, the fourth branch consists of a 3×3 convolution kernel, a CA attention mechanism and a 1×1 convolution kernel connected in sequence, and the outputs of the third branch and the fourth branch are connected through a splicing operation.

[0065] In this embodiment, the backbone network used by NanoDet-Plus is the lightweight network ShuffleNetV2. The internal shuffle structure of ShuffleNetV2 makes ShuffleNetV2 have a good effect on problems such as graphic classification. However, in the network structure of ShuffleNetV2, there is no effective attention mechanism to assist the model in learning features. Based on this, in order to better identify areas with important information in the input image, such as the edges and contours of gestures, and enhance the feature representation of these areas, so that the model can more effectively capture important information related to the task, this embodiment adds SE attention mechanism and CA attention mechanism to the two blocks of the backbone network respectively. The blocks after adding the attention mechanism are as follows: Figure 2 shown.

[0066] Among them, the SE attention mechanism can help obtain the information inside these channels and the relationship between each channel, and then add the normalized weights obtained before to the features of each channel to obtain the required features. Although the SE attention mechanism does not consider space, its structure is simpler. In block 1, the model accuracy can be improved while keeping the number of model parameters roughly the same. Its specific structure is as follows Figure 3As shown in the figure, specifically, the input feature map X becomes the output feature map U after passing through Ftr. Ftr can be regarded as a standard convolution operator, and the formula is defined as follows:

[0067]

[0068] Where X∈R H×W×C Represents the input feature map, U∈R H×W×C represents the output feature map, R represents the feature map, U C It is synonymous with U, X s represents the sth channel of X, H represents the height of the feature map, W represents the width of the feature map, C represents the number of channels of the feature map, V represents a set of learned filter kernels, V C represents the parameters of the Cth filter, Represents a 2D spatial kernel, and * represents a convolution operation. In order to solve the problem of utilizing channel dependency, the global spatial information is compressed into a channel descriptor, that is, the global average pooling of the channel is used to directly compress the W×H×C feature map containing global information into a 1×1×C feature vector Z. The channel features of C feature maps are compressed into a value, so that the generated channel-level feature vector Z contains context information, alleviating the problem of channel dependency. The definition of the feature vector Z is as follows:

[0069]

[0070] Among them, Z c represents the cth element of Z, F sq Represents global average pooling. In order to utilize the information gathered in the compression operation, the Excitation operation is then used to fully capture channel dependencies. Specifically, the SE attention mechanism uses a gate mechanism consisting of two fully connected layers. The first fully connected layer compresses C channels into C / r channels to reduce the amount of computation, and then passes through a RELU nonlinear activation layer. The second fully connected layer restores the number of channels back to C channels, and then uses Sigmoid activation to obtain the weight s. The dimension of the final s is 1×1×C, which is used to characterize the weights of the C feature maps in the feature map U, and r represents the compression ratio. The definition of weight s is as follows:

[0071] s=F ex (Z, W)=σ(g(Z, W))=σ(W2δ(W1Z))

[0072] Among them, F exIndicates the name of the function, W1 indicates the weight matrix of the fully connected layer used for dimensionality reduction, W2 indicates the weight matrix of another fully connected layer, which is used to map the channel description vector after dimensionality reduction back to the original number of channels C, σ indicates the output activation function, usually the Sigmoid function, which is used to limit the output to the range of (0,1), δ is the nonlinear activation function, usually ReLU, and g indicates the function of the activation stage (that is, the intermediate result of the channel weight calculation process). The last is the Scale operation, which weights the attention weights obtained above to the features of each channel to obtain the final output of the SE module.

[0073]

[0074] Among them, F scale Indicates the Scale operation, s C Represents the weight of the Cth channel.

[0075] The CA attention mechanism splits channel attention into two 1D feature encoding processes that aggregate features along different directions. Its advantage is that it can capture long-distance dependencies in one spatial direction, while retaining precise position information in another spatial direction. The generated feature maps are then encoded to form a pair of direction-aware and position-sensitive feature maps, which can be applied to complement the input feature map to enhance the target features of interest. The two different channel attention directions are coordinate information embedding and coordinate attention generation, and their specific structures are as follows: Figure 4 As shown. In this embodiment, the CA attention mechanism transforms global pooling into an encoding operation of two 1-dimensional vectors. For input x, the pooling kernels (1, W) and (1, W) are used to encode horizontal and vertical features, that is, the output of the c-th dimension feature is:

[0076]

[0077] in, Represents the value of the hth row after pooling in channel C, Represents the value of the wth column after pooling in channel C. In order to make better use of the above coordinate information, in this embodiment, CA proposes a matching coordinate attention generation operation, concatenates the output of the cth dimension feature, and uses 1×1 convolution, BN and nonlinear activation for feature transformation:

[0078] f=δ(F1(z h , z w ))

[0079] Where f∈R C / r×(H+W)represents the intermediate feature containing horizontal and vertical spatial information, r represents the compression ratio, and F1 represents the feature fusion function, which is used to transform the height direction feature z h and width direction feature z w Combined, z h Represents the features obtained by pooling in the height direction (row direction), z w represents the feature obtained by pooling in the width direction (column direction). Here, the features in the two directions are not fused intensively, and then f is divided into two independent features f h ∈R C / r×H and f w ∈R C / r×W , use another two 1×1 convolutions and the sigmoid function to transform the features so that their dimensions are consistent with the input x:

[0080] g h =σ(F h (f h ))

[0081] g w =σ(F w (f w ))

[0082] The output g h and g w Merge into a weight matrix for calculating the coordinate attention block output:

[0083]

[0084] Each weight of the CA attention mechanism contains inter-channel information, horizontal spatial information, and vertical spatial information, which can help the network locate target information more accurately and enhance recognition capabilities.

[0085] The improved neck network includes: 2 upsampling layers, 2 FF attention mechanisms and 2 downsampling layers; 1 upsampling, 1 FF attention mechanism and 1 splicing operation are sequentially connected to form a subunit, the 2 subunits are connected end to end, and the output of the second subunit is spliced ​​with the output of the first subunit through a downsampling layer, and the result is spliced ​​with the network input through a downsampling layer.

[0086] In this embodiment, NanoDet-Plus uses GhostModule for feature fusion to create Ghost-PAN, which enhances the performance of multi-scale target detection without increasing too many parameters and computational complexity. However, gestures are extremely diverse, and different individuals may show different hand postures and hand colors when performing the same gesture. This diversity makes it difficult to generalize the model, resulting in a decrease in recognition accuracy. In order to further improve the model's ability to detect gestures, we combine the FF (Filtration-Fusion) attention mechanism with Ghost-PAN to obtain GF-PAN. In GF-PAN, high-level features are used as weights to filter low-level feature information through the channel attention module. The filtered information is merged with the high-level features to enhance the model's feature expression capabilities. The specific structure of GF-PAN and FF (Filtration-Fusion) attention mechanism is shown below. Figure 5 , Figure 6 shown.

[0087] Specifically, the FF attention mechanism consists of a 3×3 transposed convolution kernel, an upsampling layer / downsampling layer, and a CA attention module. Given an input high-level feature f high ∈R C×H×W and an input low-scale feature f low ∈R C ×H1×W1 , the high-level features are first expanded using a transposed convolution (T-Conv) of size 2 and kernel size 3×3 to obtain the feature f′ high ∈R C×2H×2W Then, in order to unify the dimensions of high-level features and low-scale features, bilinear interpolation is used to up- or down-sample the high-level features to obtain the feature f att ∈R C×H1×W1 Next, the CA module is used to convert high-level features into corresponding attention weights to filter low-scale features, and after obtaining features with the same dimension. Finally, the filtered low-scale features are fused with high-level features to enhance the feature representation of the model and obtain f out ∈R C×H1×W1 , the fusion process of feature selection is as follows:

[0088] f att =BL(T-Conv(f high ))

[0089] f out =f low *CA(f att +f att )

[0090] Among them, BL stands for bilinear interpolation and T-Conv stands for transposed convolution.

[0091] In this embodiment, the auxiliary training network (AGM) consists of only four 3×3 convolutions, uses group normalization as the normalization layer, and shares parameters between feature maps of different scales. Due to the shared number and the fact that it is not a depthwise separable convolution (depthwise separable convolution is not GPU-friendly), the training resources consumed by AGM are very small, far less than a teacher model, and this module is only used during training and is discarded after training, which does not affect the inference speed at all.

[0092] In the detection network of this embodiment, NanoDet-Plus uses a detection head with shared weights, that is, the same set of convolutions are used to predict the detection frame for the multi-scale Feature Map from FPN, and each layer uses a learnable Scale value as a coefficient to scale the predicted frame.

[0093] The model training module uses the training data to train the initial gesture recognition model to obtain the LHGDet gesture recognition model, such as Figure 7 shown.

[0094] The gesture recognition module uses the LHGDet gesture recognition model to complete gesture recognition.

[0095] Embodiment 2

[0096] In this embodiment, in order to evaluate the constructed LHGDet gesture recognition model, experiments were conducted on two datasets, one is a self-made gesture dataset IHGD, and the other is a public dataset HaGRID.

[0097] The IHGD dataset collects 2,000 images of 9 different gestures, including 0, 1, 2, 3, 4, 5, 8, good, and no. In order to enhance the performance of the model, the images are collected under complex backgrounds and changing lighting conditions. Afterwards, the gesture part in the image is annotated using a bounding box using an open source data annotation tool (LabelImg). There are about 220 images for each gesture, which are divided into a training set and a test set in a ratio of 4:1. The gestures in this dataset are collected while wearing protective gloves. This particularity makes it a unique data support for gesture recognition problems in specific environments. This provides a valuable resource for studying and solving the challenges of gesture recognition under similar conditions. In order to solve the problems of data enhancement and limited dataset size, the original images of IHGD are also operated, so that the 2,000 images contained in the dataset are rotated, horizontally or vertically moved, horizontally cut, scaled, and flipped, and further converted into training data to enhance the number of training images. These operations will be performed before model training, and since only some geometric processing is performed on the images, the characteristics of the images will not be changed. The public data set HaGRID is the same as that in the first embodiment.

[0098] The experimental environment of this embodiment is based on the Windows 10 operating system. The hardware conditions of the experiment are: i9-11900kCPU, RTX4090GPU. This embodiment uses the PyTorch deep learning framework to complete the experiment. The PyTorch version is 1.10.0, and the CUDA version is 11.3. In order to comprehensively evaluate the accuracy of the model, in this embodiment, different types of MSCOCO style mAP, that is, the average of the average precision of all categories under a certain intersection-over-union (IoU) threshold is used as a metric, including AP@.50:.95 corresponding to the average mAP of IoU from 0.50 to 0.95, with a step size of 0.05, AP@.50 corresponding to the mAP of IoU=0.50, as well as the model parameter quantities params and FLOPs, and mA@.50.:95P is used as the primary evaluation indicator. The above-mentioned metrics are defined as:

[0099]

[0100]

[0101] Among them, TP represents the positive samples judged as positive samples by the model; FP represents the negative samples judged as positive samples; p(r) represents the precision rate, which is a function of the recall rate, n represents the number of a certain category; APi represents the detection accuracy of category i.

[0102] In order to prove the effectiveness of the proposed method, the invention and several other classic lightweight target detection models (NanoDet-Plus, YOLOX, YOLOv5n, Picodet-S) are used to perform detection on the IHGR and HaGRID-tiny datasets. The results are shown in Table 1:

[0103] Table 1

[0104]

[0105] As can be seen from the table, the improved model is equivalent to NanoDet-Plus with almost the same number of parameters and FLOPs, but mAP has increased by 5.5%. Compared with other lightweight models, LHGDet performs better in terms of accuracy. From the data in the table, it can be seen that LHGDet has achieved a good balance between accuracy and model size, and its performance is much better than NanoDet-Plus.

[0106] To further verify the detection performance of the model, we conducted comparative experiments on the HaGRID-tiny dataset. As shown in Table 2, our method achieved an mAP of 75.3, which is 1.1 higher than NanoDet-Plus and higher than other SOTA models.

[0107] Table 2

[0108]

[0109] To further explore the contribution of SE attention mechanism, CA attention mechanism and FF attention mechanism in the model, we conducted an ablation experiment on the IHGD dataset. The experimental results are shown in Table 3.

[0110] Table 3

[0111]

[0112] As shown in Table 3, when different modules are added separately, the improvement of model accuracy is not obvious, but if they are combined, the effect of the model will be greatly improved, and the number of parameters remains basically unchanged. When the number of parameters changes slightly, the model inference speed will not be affected, and the accuracy is greatly improved.

[0113] Embodiment 3

[0114] In this embodiment, a gesture recognition method based on the LHGDet model includes the following steps:

[0115] S1. Obtain a gesture data image set, and mark the gesture data image set to obtain training data.

[0116] The method for obtaining training data includes: rotating and scaling a gesture data image set to obtain a processed image set; annotating the processed image set according to different gesture types to obtain an annotated image set; dividing the annotated image set into a training set and a test set according to a preset ratio to obtain training data.

[0117] S2. Improve the NanoDet-Plus model to obtain the initial gesture recognition model.

[0118] The initial gesture recognition model includes: an improved backbone network, an improved neck network, an auxiliary training network and a detection network; the method for obtaining the improved backbone network includes: adding SE attention mechanism and CA attention mechanism to the backbone network of the NanoDet-Plus model; the method for obtaining the improved neck network includes: adding FF attention mechanism to the neck network of the NanoDet-Plus model; the auxiliary training network consists of 4 3×3 convolution kernels, using group normalization as the normalization layer; the detection network is a detection head with shared weights.

[0119] The improved backbone network includes: block1 and block2; block1 consists of the first branch and the second branch, the first branch consists of a 1×1 convolution kernel, a 3×3 convolution kernel, an SE attention mechanism and a 1×1 convolution kernel connected in sequence, the second branch does not process the input data, and the outputs of the first branch and the second branch are connected through a splicing operation; block2 consists of the third branch and the fourth branch, the third branch consists of a 1×1 convolution kernel, a 3×3 convolution kernel and a 1×1 convolution kernel connected in sequence, the fourth branch consists of a 3×3 convolution kernel, a CA attention mechanism and a 1×1 convolution kernel connected in sequence, and the outputs of the third branch and the fourth branch are connected through a splicing operation.

[0120] The improved neck network includes: 2 upsampling layers, 2 FF attention mechanisms and 2 downsampling layers; 1 upsampling, 1 FF attention mechanism and 1 splicing operation are sequentially connected to form a subunit, the 2 subunits are connected end to end, and the output of the second subunit is spliced ​​with the output of the first subunit through a downsampling layer, and the result is spliced ​​with the network input through a downsampling layer.

[0121] S3. Use the training data to train the initial gesture recognition model to obtain the LHGDet gesture recognition model.

[0122] S4. Use the LHGDet gesture recognition model to complete gesture recognition.

[0123] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A gesture recognition system based on the LHGDet model, characterized in that: include: Data collection module, model building module, model training module and gesture recognition module; The data collection module is used to obtain a gesture data image set and mark the gesture data image set to obtain training data; The model building module improves the NanoDet-Plus model to obtain an initial gesture recognition model; The model training module trains the initial gesture recognition model using the training data to obtain a LHGDet gesture recognition model; The gesture recognition module uses the LHGDet gesture recognition model to complete gesture recognition.

2. The gesture recognition system based on the LHGDet model according to claim 1, characterized in that: The data collection module includes: an image processing unit, a labeling unit and a data division unit; The image processing unit is used to rotate and scale the gesture data image set to obtain a processed image set; The labeling unit is used to label the processed image set according to different gesture types to obtain a labeled image set; The data division unit is used to divide the annotated image set into a training set and a test set according to a preset ratio to obtain the training data.

3. The gesture recognition system based on the LHGDet model according to claim 1, characterized in that: The initial gesture recognition model includes: an improved backbone network, an improved neck network, an auxiliary training network and a detection network; The method for obtaining the improved backbone network includes: adding SE attention mechanism and CA attention mechanism to the backbone network of the NanoDet-Plus model; The method for obtaining the improved neck network includes: adding a FF attention mechanism to the neck network of the NanoDet-Plus model; The auxiliary training network consists of 4 3×3 convolution kernels, using group normalization as the normalization layer; The detection network is a detection head with shared weights.

4. The gesture recognition system based on the LHGDet model according to claim 3, characterized in that: The improved backbone network includes: block1 and block2; The block1 is composed of a first branch and a second branch, the first branch is composed of a 1×1 convolution kernel, a 3×3 convolution kernel, an SE attention mechanism, and a 1×1 convolution kernel connected in sequence, the second branch does not process the input data, and the outputs of the first branch and the second branch are connected through a splicing operation; The block2 is composed of a third branch and a fourth branch. The third branch is composed of a 1×1 convolution kernel, a 3×3 convolution kernel and a 1×1 convolution kernel connected in sequence. The fourth branch is composed of a 3×3 convolution kernel, a CA attention mechanism and a 1×1 convolution kernel connected in sequence. The outputs of the third branch and the fourth branch are connected through a splicing operation.

5. The gesture recognition system based on the LHGDet model according to claim 3, characterized in that: The improved neck network includes: 2 upsampling layers, 2 FF attention mechanisms and 2 downsampling layers; One upsampling, one FF attention mechanism and one splicing operation are sequentially connected to form one subunit, two subunits are connected end to end, and the output of the second subunit is spliced ​​with the output of the first subunit through one downsampling layer, and the result is spliced ​​with the network input through one downsampling layer.

6. A gesture recognition method based on the LHGDet model, the recognition method being applied to the application system according to any one of claims 1 to 5, characterized in that: The following steps are involved: Acquire a gesture data image set, and mark the gesture data image set to obtain training data; Improve the NanoDet-Plus model to obtain the initial gesture recognition model; Using the training data to train the initial gesture recognition model to obtain a LHGDet gesture recognition model; The LHGDet gesture recognition model is used to complete gesture recognition.

7. The gesture recognition method based on the LHGDet model according to claim 6, characterized in that: The method for obtaining the training data includes: Rotating and scaling the gesture data image set to obtain a processed image set; Annotating the processed image set according to different gesture types to obtain an annotated image set; The annotated image set is divided into a training set and a test set according to a preset ratio to obtain the training data.

8. The gesture recognition method based on the LHGDet model according to claim 6, characterized in that: The initial gesture recognition model includes: an improved backbone network, an improved neck network, an auxiliary training network and a detection network; The method for obtaining the improved backbone network includes: adding SE attention mechanism and CA attention mechanism to the backbone network of the NanoDet-Plus model; The method for obtaining the improved neck network includes: adding a FF attention mechanism to the neck network of the NanoDet-Plus model; The auxiliary training network consists of 4 3×3 convolution kernels, using group normalization as the normalization layer; The detection network is a detection head with shared weights.

9. The gesture recognition method based on the LHGDet model according to claim 8, characterized in that: The improved backbone network includes: block1 and block2; The block1 is composed of a first branch and a second branch, the first branch is composed of a 1×1 convolution kernel, a 3×3 convolution kernel, an SE attention mechanism, and a 1×1 convolution kernel connected in sequence, the second branch does not process the input data, and the outputs of the first branch and the second branch are connected through a splicing operation; The block2 is composed of a third branch and a fourth branch. The third branch is composed of a 1×1 convolution kernel, a 3×3 convolution kernel and a 1×1 convolution kernel connected in sequence. The fourth branch is composed of a 3×3 convolution kernel, a CA attention mechanism and a 1×1 convolution kernel connected in sequence. The outputs of the third branch and the fourth branch are connected through a splicing operation.

10. The gesture recognition method based on the LHGDet model according to claim 8, characterized in that: The improved neck network includes: 2 upsampling layers, 2 FF attention mechanisms and 2 downsampling layers; One upsampling, one FF attention mechanism and one splicing operation are sequentially connected to form one subunit, two subunits are connected end to end, and the output of the second subunit is spliced ​​with the output of the first subunit through one downsampling layer, and the result is spliced ​​with the network input through one downsampling layer.

Images