Gesture recognition system and method based on LHGDet model

Abstract

The application discloses a gesture recognition system and method based on an LHGDet model, and the system comprises a data collection module, a model construction module, a model training module and a gesture recognition module; the data collection module is used for acquiring a gesture data image set and marking the gesture data image set to obtain training data; the model construction module improves a NanoDet-Plus model to obtain an initial gesture recognition model; the model training module trains the initial gesture recognition model by using the training data to obtain an LHGDet gesture recognition model; and the gesture recognition module completes gesture recognition by using the LHGDet gesture recognition model. The NanoDet-Plus model is improved by using CA attention mechanisms, SE attention mechanisms and FF attention mechanisms, so that the network can better capture the long-distance spatial dependence relationship in an image and enhance the detection capability for small targets.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of human-computer interaction, and particularly relates to a gesture recognition system and method based on an LHGDet model. BACKGROUND

[0002] In the field of human-computer interaction, gestures have many application scenarios, such as game control, smart home, sign language real-time translation, etc. Therefore, it is very important to quickly and accurately detect and recognize gestures. In recent years, gesture recognition (HGR) has attracted more and more research interest of scholars. Since the end of the 20th century, research on gesture recognition interaction has been carried out, which is mainly divided into gesture recognition based on external devices and gesture recognition based on computer vision. Compared with the gesture recognition based on external devices, the gesture recognition based on computer vision is more convenient and flexible, and reduces the cost of devices.

[0003] The traditional gesture recognition method based on computer vision is divided into three stages of gesture detection and segmentation, gesture feature extraction and gesture recognition. The skin color segmentation model is often used to segment the gesture region, and then the information obtained by the image processing algorithm is used to extract features, such as the gradient direction histogram algorithm, the scale invariant feature transform algorithm, the local binary pattern algorithm, etc., and then combined with the classifier of machine learning, such as support vector machine, adaptive boosting, etc. for classification. But this kind of method has poor robustness and low recognition accuracy in complex background, and is only suitable for single target recognition, and it is difficult to recognize multiple targets. Moreover, the manual feature extraction method is tedious and complex, depends on the experience of engineering personnel, and has large application limitations of the algorithm.

[0004] With the research and development of deep learning, the method based on deep learning has fast recognition speed and high accuracy, and has become the mainstream gesture recognition algorithm. Oyedotun et al. binarize the image, use threshold segmentation to obtain the gesture region, and use convolutional neural network and encoder for gesture recognition. Wang Long et al. use a skin color model to detect gestures, and reconstruct the obtained gesture region into a gray image, and finally input the gesture region into CNN for feature extraction and recognition. Mohanty et al. do not perform gesture detection processing, but directly input the original image into a simple 2-layer CNN for feature extraction and recognition. In the above gesture recognition classification algorithms, when there are complex background interferences such as human skin color in the image, the recognition rate of CNN decreases significantly.

[0005] With the development of target detection algorithm, many scholars convert the classification problem of gesture recognition into target detection problem, which mainly includes two kinds: 1-stage target detection method based on regression and 2-stage target detection method based on region recommendation. The 1-stage target detection algorithm mainly includes YOLO, RetinaNet and CornerNet, and the 2-stage target detection algorithm mainly includes R-CNN, FastR-CNN, FasterR-CNN and R-FCN. These algorithms greatly improve the accuracy of gesture recognition, but due to the limitation of memory and computing resources, it is difficult to deploy convolutional neural network (CNN) on embedded devices. SUMMARY

[0006] The present application aims to solve the problems of the prior art and provides the following solutions:

[0007] A gesture recognition system based on LHGDet model, comprising: a data collection module, a model construction module, a model training module and a gesture recognition module;

[0008] The data collection module is used to obtain a gesture data image set and mark the gesture data image set to obtain training data;

[0009] The model construction module improves the NanoDet-Plus model to obtain an initial gesture recognition model;

[0010] The model training module trains the initial gesture recognition model using the training data to obtain an LHGDet gesture recognition model;

[0011] The gesture recognition module completes gesture recognition using the LHGDet gesture recognition model.

[0012] Preferably, the data collection module comprises an image processing unit, a labeling unit and a data division unit;

[0013] The image processing unit is used to rotate and scale the gesture data image set to obtain a processed image set;

[0014] The labeling unit is used to label the processed image set according to different gesture types to obtain a labeled image set;

[0015] The data division unit is used to divide the labeled image set into a training set and a test set according to a preset proportion to obtain the training data.

[0016] Preferably, the initial gesture recognition model comprises an improved backbone network, an improved neck network, an auxiliary training network and a detection network.

[0017] The method for obtaining the improved backbone network comprises: adding SE attention mechanism and CA attention mechanism to the backbone network of the NanoDet-Plus model;

[0018] The method for obtaining the improved neck network comprises: adding FF attention mechanism to the neck network of the NanoDet-Plus model;

[0019] The auxiliary training network is composed of four 3*3 convolution kernels, and uses group normalization as a normalization layer.

[0020] The detection network is a detection head sharing weights.

[0021] Preferably, the improved backbone network comprises: block1 and block2.

[0022] The block1 is composed of a first branch and a second branch, the first branch is sequentially connected by one 1*1 convolution kernel, one 3*3 convolution kernel, SE attention mechanism and one 1*1 convolution kernel, the second branch does not process the input data, and the outputs of the first branch and the second branch are connected through a splicing operation.

[0023] The block2 is composed of a third branch and a fourth branch, the third branch is sequentially connected by one 1*1 convolution kernel, one 3*3 convolution kernel and one 1*1 convolution kernel, the fourth branch is sequentially connected by one 3*3 convolution kernel, CA attention mechanism and one 1*1 convolution kernel, and the outputs of the third branch and the fourth branch are connected through a splicing operation.

[0024] Preferably, the improved neck network comprises: two up-sampling layers, two FF attention mechanisms and two down-sampling layers.

[0025] One up-sampling, one FF attention mechanism and one splicing operation are sequentially connected to form one subunit, two subunits are connected in head-tail mode, the output of the second subunit is spliced with the output of the first subunit through one down-sampling layer, and the obtained result is spliced with the network input through one down-sampling layer.

[0026] The application further provides a gesture recognition method based on the LHGDet model, which is applied to the application system.

[0027] A set of gesture data images is acquired, and the set of gesture data images is labeled to obtain training data;

[0028] The NanoDet-Plus model is improved to obtain an initial gesture recognition model.

[0029] training the initial gesture recognition model by using the training data, to obtain an LHGDet gesture recognition model;

[0030] completing gesture recognition by using the LHGDet gesture recognition model.

[0031] Preferably, the method for obtaining the training data comprises:

[0032] rotating and scaling the gesture data image set to obtain a processed image set;

[0033] annotating the processed image set according to different gesture types to obtain an annotated image set;

[0034] dividing the annotated image set into a training set and a test set according to a preset proportion to obtain the training data.

[0035] Preferably, the initial gesture recognition model comprises an improved backbone network, an improved neck network, an auxiliary training network and a detection network.

[0036] The method for obtaining the improved backbone network comprises adding an SE attention mechanism and a CA attention mechanism to a backbone network of the NanoDet-Plus model.

[0037] The method for obtaining the improved neck network comprises adding an FF attention mechanism to a neck network of the NanoDet-Plus model.

[0038] The auxiliary training network is composed of four 3x3 convolution kernels and uses group normalization as a normalization layer.

[0039] The detection network is a detection head sharing weights.

[0040] Preferably, the improved backbone network comprises a block1 and a block2.

[0041] The block1 is composed of a first branch and a second branch, the first branch is composed of one 1x1 convolution kernel, one 3x3 convolution kernel, an SE attention mechanism and one 1x1 convolution kernel connected in sequence, the second branch does not process the input data, and the outputs of the first branch and the second branch are connected through a splicing operation.

[0042] The block2 is composed of a third branch and a fourth branch, the third branch is composed of one 1x1 convolution kernel, one 3x3 convolution kernel and one 1x1 convolution kernel connected in sequence, the fourth branch is composed of one 3x3 convolution kernel, a CA attention mechanism and one 1x1 convolution kernel connected in sequence, and the outputs of the third branch and the fourth branch are connected through a splicing operation.

[0043] Preferably, the improved neck network comprises: 2 up-sampling layers, 2 FF attention mechanisms and 2 down-sampling layers.

[0044] 1 up-sampling, 1 FF attention mechanism and 1 concatenation operation are sequentially connected to form a subunit, 2 subunits are connected head to tail, and the output of the second subunit is spliced with the output of the first subunit through 1 down-sampling layer, and the obtained result is spliced with the network input through 1 down-sampling layer.

[0045] Compared with the prior art, the present application has the following beneficial effects:

[0046] The present application improves the NanoDet-Plus network for gestures, a small target, uses CA attention mechanism and SE attention mechanism to improve the backbone network, so that the network can better capture the long-distance spatial dependence in the image, enhance the detection ability of small targets, and embed FF attention mechanism in the original PAN to better fuse different scale features, so that the present application has good performance on multi-scale gestures with complex background. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the present application, the following briefly introduces the drawings needed in the embodiments, and obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0048] Figure 1 The system structure diagram of the embodiment of the present application is shown in the figure;

[0049] Figure 2 The structure diagram of the improved backbone network of the embodiment of the present application is shown in the figure;

[0050] Figure 3 The structure diagram of the SE attention mechanism of the embodiment of the present application is shown in the figure;

[0051] Figure 4 The structure diagram of the CA attention mechanism of the embodiment of the present application is shown in the figure;

[0052] Figure 5 The structure diagram of the improved neck network of the embodiment of the present application is shown in the figure

[0053] Figure 6 The structure diagram of the FF attention mechanism of the embodiment of the present application is shown in the figure;

[0054] Figure 7A structural schematic diagram of an LHGDet gesture recognition model of an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0056] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0057] Embodiment one

[0058] In this embodiment, as shown in the figure, a gesture recognition system based on an LHGDet model includes a data collection module, a model construction module, a model training module and a gesture recognition module. Figure 1 The data collection module is configured to obtain a gesture data image set and mark the gesture data image set to obtain training data.

[0059] The data collection module includes an image processing unit, a marking unit and a data division unit. The image processing unit is configured to rotate and scale the gesture data image set to obtain a processed image set. The marking unit is configured to mark the processed image set according to different gesture types to obtain a marked image set. The data division unit is configured to divide the marked image set into a training set and a test set according to a preset ratio to obtain the training data.

[0060] In this embodiment, the latest open-source super-large gesture image public data set, HaGRID data set, is used. The data set is composed of 500,000 RGB images with a size of 1920x1080, containing 18 common gestures. This data set is designed for human-computer interaction. The 18 gestures are all functional gestures and are widely used in daily life. At the same time, all images are different in background, light, scene and theme, etc., and have good generalization. According to the actual application situation, the original data set is simplified and the picture resolution is reduced to obtain the data set HaGRID-tiny, which contains 18,000 pictures, including 18 gestures such as call, peace, dislike, etc. Then, the training set and the test set are divided according to the ratio of 4:1 to obtain the training data.

[0061]

[0062] ​The model construction module improves the NanoDet-Plus model to obtain an initial gesture recognition model.

[0063] The initial gesture recognition model comprises an improved backbone network, an improved neck network, an auxiliary training network and a detection network; the method for obtaining the improved backbone network comprises adding an SE attention mechanism and a CA attention mechanism to the backbone network of the NanoDet-Plus model; the method for obtaining the improved neck network comprises adding an FF attention mechanism to the neck network of the NanoDet-Plus model; the auxiliary training network is composed of four 3x3 convolution kernels and uses group normalization as a normalization layer; and the detection network is a detection head with shared weights.

[0064] The improved backbone network comprises a block1 and a block2; the block1 is composed of a first branch and a second branch, the first branch is composed of a 1x1 convolution kernel, a 3x3 convolution kernel, an SE attention mechanism and a 1x1 convolution kernel connected in sequence, the second branch does not process the input data, and the outputs of the first branch and the second branch are connected through a splicing operation; and the block2 is composed of a third branch and a fourth branch, the third branch is composed of a 1x1 convolution kernel, a 3x3 convolution kernel and a 1x1 convolution kernel connected in sequence, the fourth branch is composed of a 3x3 convolution kernel, a CA attention mechanism and a 1x1 convolution kernel connected in sequence, and the outputs of the third branch and the fourth branch are connected through a splicing operation.

[0065] In the embodiment, the backbone network used by the NanoDet-Plus is a lightweight network ShuffleNetV2, the shuffle structure inside the ShuffleNetV2 makes the ShuffleNetV2 have good effects on problems such as image classification, but in the network structure of the ShuffleNetV2, there is no effective attention mechanism to assist the model in learning features, based on this, in order to better identify the regions with important information in the input image, such as the edges and contours of gestures, and enhance the feature representation of these regions, so that the model can more effectively capture important information related to the task, the embodiment adds an SE attention mechanism and a CA attention mechanism to two blocks of the backbone network respectively, and the blocks after adding the attention mechanisms are as shown in Figure 2 .

[0066] The SE attention mechanism can help obtain the information inside these channels and the relationship between the channels, and then weight the normalized weights obtained before to the features of each channel to obtain the required features. The SE attention mechanism does not consider space, but its structure is simpler. In the block1, the model precision can be improved while the number of model parameters remains approximately the same. The specific structure is as shown in Figure 3As shown, specifically, the input feature map X is changed into the output feature map U after Ftr. Ftr can be regarded as a standard convolution operator, which is defined as follows:

[0067]

[0068] wherein X∈R H×W×C represents the input feature map, U∈R H×W×C represents the output feature map, R represents the feature map, U C represents the same as U, X s represents the s-th channel of X, H represents the height of the feature map, W represents the width of the feature map, C represents the number of channels of the feature map, V represents a set of learned filter kernels, V C represents the parameters of the C-th filter, represents a 2D spatial kernel, and * represents a convolution operation. In order to solve the problem of channel dependence, the global spatial information is compressed into a channel descriptor, i.e. using the global average pooling of the channel, the feature map of WxHxC containing global information is directly compressed into a 1x1xC feature vector Z, and the channel features of C feature maps are compressed into a numerical value, so that the generated channel-level feature vector Z contains context information, which alleviates the problem of channel dependence, and the definition of the feature vector Z is as follows:

[0069]

[0070] wherein Z c represents the c-th element of Z, F sq represents the global average pooling. In order to utilize the information converged in the compression operation, the channel dependence is comprehensively captured through the Excitation operation next, specifically, the SE attention mechanism adopts a gate mechanism composed of two fully connected layers, the first fully connected layer compresses C channels into C / r channels to reduce the calculation amount, and then passes through a RELU nonlinear activation layer, and the second fully connected layer restores the number of channels to C channels, and then obtains the weight s through the Sigmoid activation, and finally the dimension of this s is 1x1xC, which is used to depict the weight of C feature maps in the feature map U, and r represents the compression ratio. The definition of the weight s is as follows:

[0071] s=F ex (Z,W)=σ(g(Z,W))=σ(W2δ(W1Z))

[0072] wherein F exThe name of the function represents, W1 represents a full connection layer weight matrix for dimension reduction, W2 represents another full connection layer weight matrix, which is used to map the channel description vector after dimension reduction back to the original channel number C, σ represents an output activation function, which is usually a Sigmoid function, used to limit the output to the range of (0, 1), δ is a nonlinear activation function, usually ReLU is selected, g represents the function of the activation stage (that is, the intermediate result of the channel weight calculation process). Finally, the Scale operation weights the attention weight obtained in the foregoing to the feature of each channel to obtain the final output of the SE module

[0073]

[0074] wherein F scale represents the Scale operation, s C represents the weight of the Cth channel.

[0075] The CA attention mechanism splits the channel attention into two 1D feature encoding processes that aggregate features in different directions. The advantage is that long-distance dependencies can be captured in one spatial direction, while precise position information can be preserved in the other spatial direction, and then the generated feature maps are respectively encoded to form a pair of direction-aware and position-sensitive feature maps, which can be applied to the input feature map to enhance the target features of interest. Two different channel attention directions are coordinate information embedding and coordinate attention generation, and the specific structure is as shown in Figure 4 In this embodiment, the CA attention mechanism changes the global pooling into two 1D vector encoding operations. For the input x, use the pooling kernel (1, W) and (1, W) to encode the horizontal direction and vertical direction features, that is, the output of the cth dimension feature is:

[0076]

[0077] wherein, represents the value of the hth row after pooling in the Cth channel, represents the value of the wth column after pooling in the Cth channel; In order to better utilize the above coordinate information, the CA in this embodiment proposes a matching coordinate attention generation operation, which performs concatenate operation on the output of the cth dimension feature, uses 1x1 convolution, BN and nonlinear activation for feature transformation:

[0078] f = δ (F1(z h , z w ))

[0079] wherein f ∈ R C / r×(H+W)denotes the intermediate feature containing lateral and longitudinal spatial information, r denotes the compression ratio, F1 denotes the feature fusion function, which is used to fuse the height direction feature z h and the width direction feature z w . Here, the features of the two directions are not fused intensively, and then f is divided into two independent features f h denotes the feature obtained from height direction (row direction) pooling, and z w denotes the feature obtained from width direction (column direction) pooling. Here, the features of the two directions are not fused intensively, and then f is divided into two independent features f h ∈R C / r×H and f w ∈R C / r×W , and another two 1x1 convolution and sigmoid function are used for feature conversion, so that the dimension is consistent with the input x:

[0080] g h =σ(F h (f h ))

[0081] g w =σ(F w (f w ))

[0082] The outputs g h and g w are combined into a weight matrix, which is used to calculate the output of the coordinate attention block:

[0083]

[0084] Each weight of the CA attention mechanism contains inter-channel information, lateral spatial information and longitudinal spatial information, which can help the network to locate the target information more accurately and enhance the recognition ability.

[0085] The improved neck network includes: 2 up-sampling layers, 2 FF attention mechanisms and 2 down-sampling layers; 1 up-sampling, 1 FF attention mechanism and 1 concatenation operation are sequentially connected to form a sub-unit, 2 sub-units are connected at the head and tail, and the output of the second sub-unit is spliced with the output of the first sub-unit through a down-sampling layer, and the obtained result is spliced with the network input through a down-sampling layer.

[0086] In this embodiment, NanoDet-Plus uses GhostModule for feature fusion to create Ghost-PAN, which enhances the performance of multi-scale object detection without increasing the number of parameters and computational load. However, gestures have extremely high diversity; different individuals may exhibit different hand postures and hand colors when performing the same gesture. This diversity makes it difficult for the model to generalize, leading to a decrease in recognition accuracy. To further improve the model's ability to detect gestures, we combine the FF (Filtration-Fusion) attention mechanism with Ghost-PAN to obtain GF-PAN. In GF-PAN, high-level features are used as weights to filter low-level feature information through the channel attention module. The filtered information is then merged with the high-level features, thereby enhancing the model's feature representation ability. The specific structures of GF-PAN and the FF (Filtration-Fusion) attention mechanism are as follows: Figure 5 , Figure 6 As shown.

[0087] Specifically, the FF attention mechanism consists of a 3×3 transposed convolutional kernel, an upsampling / downsampling layer, and a CA attention module. Given an input high-level feature f... high ∈R C×H×W And an input low-scale feature f low ∈R C ×H1×W1 The high-level features are first expanded using a transposed convolution (T-Conv) with a size of 2 and a kernel size of 3×3 to obtain feature f′. high ∈R C×2H×2W Then, to unify the dimensionality of high-level and low-scale features, bilinear interpolation is used to upsample or downsample the high-level features, resulting in feature f. att ∈R C×H1×W1 Next, the CA module is used to convert high-level features into corresponding attention weights to filter low-scale features, resulting in features with the same dimension. Finally, the filtered low-scale features are fused with the high-level features to enhance the model's feature representation and obtain f. out ∈R C×H1×W1 The feature selection fusion process is as follows:

[0088] f att =BL(T-Conv(f) high ))

[0089] f out =f low *CA(f att +f att )

[0090] where BL denotes bilinear interpolation, and T-Conv denotes transpose convolution.

[0091] In this embodiment, the auxiliary training network (AGM) is only composed of 4 3x3 convolutions, uses group normalization as the normalization layer, and shares parameters between Feature Maps of different scales. Since the number is shared, and it is not a depth separable convolution (depth separable convolution is not friendly to GPU), the training resources consumed by AGM are very small, far less than a teacher model, and this module is only used during training, and is discarded after training, which does not affect the inference speed at all.

[0092] In the detection network of this embodiment, NanoDet-Plus uses a shared weight detection head, that is, the same set of convolutions is used to predict the detection frame of the multi-scale Feature Map output by FPN, and a learnable Scale value is used as a coefficient for each layer to scale the predicted frame.

[0093] The model training module trains the initial gesture recognition model using training data to obtain the LHGDet gesture recognition model, as shown in Figure 7 .

[0094] The gesture recognition module completes gesture recognition using the LHGDet gesture recognition model.

[0095] Embodiment Two

[0096] In this embodiment, in order to evaluate the LHGDet gesture recognition model constructed, experiments are performed on two data sets, one is a self-made gesture data set IHGD, and the other is a public data set HaGRID.

[0097] The dataset IHGD contains 2000 images of 9 different gestures, including 0, 1, 2, 3, 4, 5, 8, good, and no. To enhance the performance of the model, the images were captured in complex backgrounds and varying lighting conditions. Then, the open-source data labeling tool (LabelImg) was used to label the gesture part of the image using a bounding box. There are about 220 images for each gesture, and they are divided into training and test sets in a 4:1 ratio. The gestures in this dataset were captured while wearing protective gloves, which makes it unique in the context of gesture recognition under similar conditions. This provides valuable resources for researching and solving gesture recognition challenges under similar conditions. To address the problem of data augmentation and limited dataset size, the original images of IHGD were manipulated to include 2000 images in the dataset, which were rotated, horizontally or vertically moved, horizontally cut, scaled, and flipped, and further converted into training data to enhance the number of training images. These operations will be performed before model training, and since only geometric processing is performed on the images, the features of the images will not be changed. The public dataset HaGRID is the same as in Embodiment One.

[0098] The experimental environment of this embodiment is based on the Windows 10 operating system. The hardware conditions of the experiment are: i9-11900k CPU, RTX4090 GPU. This embodiment uses the PyTorch deep learning framework to complete the experiment. The PyTorch version is 1.10.0, and the CUDA version is 11.3. To comprehensively evaluate the accuracy of the model, in this embodiment, different types of MSCOCO style mAP are used, i.e. the average value of the average precision of all classes under a certain intersection over union (IoU) threshold is used as a measure, including AP@.50:.95 corresponding to the average mAP of IoU from 0.50 to 0.95 with a step of 0.05, AP@.50 corresponding to the mAP of IoU=0.50, and the model parameter quantity params and FLOPs, and mA@.50.:95P is used as the primary evaluation index. The above measurement index is defined as:

[0099]

[0100]

[0101] where TP represents the positive samples that the model judges as positive samples; FP represents the negative samples that are judged as positive samples; p(r) represents the precision, which is a function of the recall; n represents the number of a certain class; and APi represents the detection accuracy of class i.

[0102] To prove the effectiveness of the method, the present application and several other classic lightweight target detection models (NanoDet-Plus, YOLOX, YOLOv5n, Picodet-S) are used for detection on the IHGR, HaGRID-tiny datasets. The results are shown in Table 1:

[0103] Table 1

[0104]

[0105] As can be seen from the table, the improved model has almost the same parameter quantity and FLOPs as NanoDet-Plus, but the mAP has increased by 5.5%. Compared with other lightweight models, LHGDet performs better in terms of accuracy. As can be seen from the data in the table, LHGDet achieves a good balance between accuracy and model size, and has a higher performance improvement than NanoDet-Plus.

[0106] To further verify the detection performance of the model, we conducted comparative experiments of various models on the HaGRID-tiny dataset. As shown in Table 2, our method achieved an mAP of 75.3, an improvement of 1.1 over NanoDet-Plus, and higher than other SOTA models.

[0107] Table 2

[0108]

[0109] To further explore the contribution of SE attention mechanism, CA attention mechanism and FF attention mechanism in the model, we conducted an ablation experiment on the IHGD dataset. The experimental results are shown in Table 3.

[0110] Table 3

[0111]

[0112] As shown in Table 3, when adding different modules alone, the model accuracy does not improve significantly, but if they are combined together, the model effect will be greatly improved, and the parameter quantity remains basically unchanged. In the case of small changes in the number of parameters, the model inference speed will not be affected, and the accuracy is greatly improved.

[0113] Embodiment Three

[0114] In this embodiment, a gesture recognition method based on the LHGDet model includes the following steps:

[0115] S1. Obtain a set of gesture data images and label the set of gesture data images to obtain training data.

[0116] The method for obtaining the training data comprises: rotating and scaling the gesture data image set to obtain a processed image set; labeling the processed image set according to different gesture types to obtain a labeled image set; and dividing the labeled image set into a training set and a test set according to a preset ratio to obtain the training data.

[0117] S2. The initial gesture recognition model is obtained by improving the NanoDet-Plus model.

[0118] The initial gesture recognition model comprises an improved backbone network, an improved neck network, an auxiliary training network and a detection network; the method for obtaining the improved backbone network comprises adding an SE attention mechanism and a CA attention mechanism to the backbone network of the NanoDet-Plus model; the method for obtaining the improved neck network comprises adding an FF attention mechanism to the neck network of the NanoDet-Plus model; the auxiliary training network is composed of four 3x3 convolution kernels and uses group normalization as a normalization layer; and the detection network is a detection head with shared weights.

[0119] The improved backbone network comprises a block1 and a block2; the block1 is composed of a first branch and a second branch, the first branch is composed of a 1x1 convolution kernel, a 3x3 convolution kernel, an SE attention mechanism and a 1x1 convolution kernel connected in sequence, the second branch does not process the input data, and the outputs of the first branch and the second branch are connected through a splicing operation; the block2 is composed of a third branch and a fourth branch, the third branch is composed of a 1x1 convolution kernel, a 3x3 convolution kernel and a 1x1 convolution kernel connected in sequence, the fourth branch is composed of a 3x3 convolution kernel, a CA attention mechanism and a 1x1 convolution kernel connected in sequence, and the outputs of the third branch and the fourth branch are connected through a splicing operation.

[0120] The improved neck network comprises two up-sampling layers, two FF attention mechanisms and two down-sampling layers; one up-sampling layer, one FF attention mechanism and one splicing operation are sequentially connected to form one subunit, two subunits are connected in head-tail mode, the output of the second subunit is spliced with the output of the first subunit through one down-sampling layer, and the obtained result is spliced with the network input through one down-sampling layer.

[0121] S3. The initial gesture recognition model is trained using the training data to obtain an LHGDet gesture recognition model.

[0122] S4. The gesture recognition is completed using the LHGDet gesture recognition model.

[0123] The above described embodiments are only to illustrate the preferred modes of the present application, and are not intended to limit the scope of the present application. Any modification and improvement made by those skilled in the art to the technical solutions of the present application without departing from the design spirit of the present application shall fall within the protection scope of the present application as defined by the claims.

Claims

1. A gesture recognition system based on the LHGDet model, characterized in that, include: The module includes a data collection module, a model building module, a model training module, and a gesture recognition module. The data collection module is used to acquire a set of gesture data images and to label the set of gesture data images to obtain training data. The model building module improves the NanoDet-Plus model to obtain an initial gesture recognition model; The model training module uses the training data to train the initial gesture recognition model to obtain the LHGDet gesture recognition model; The gesture recognition module uses the LHGDet gesture recognition model to complete gesture recognition; The initial gesture recognition model includes: an improved backbone network, an improved posterior neck network, an auxiliary training network, and a detection network; The method for obtaining the improved backbone network includes: adding SE attention mechanism and CA attention mechanism to the backbone network of the NanoDet-Plus model; The method for obtaining the improved posterior neck network includes: adding an FF attention mechanism to the neck network of the NanoDet-Plus model; The auxiliary training network consists of four 3×3 convolutional kernels, and group normalization is used as the normalization layer; The detection network consists of a detection head with shared weights; The improved backbone network includes: block1 and block2; The block1 consists of a first branch and a second branch. The first branch consists of a 1×1 convolutional kernel, a 3×3 convolutional kernel, an SE attention mechanism, and a 1×1 convolutional kernel connected in sequence. The second branch does not process the input data. The outputs of the first branch and the second branch are connected by a concatenation operation. The block2 consists of a third branch and a fourth branch. The third branch is composed of a 1×1 convolutional kernel, a 3×3 convolutional kernel, and a 1×1 convolutional kernel connected in sequence. The fourth branch is composed of a 3×3 convolutional kernel, a CA attention mechanism, and a 1×1 convolutional kernel connected in sequence. The outputs of the third branch and the fourth branch are connected by a concatenation operation. The improved post-neck network includes: two upsampling layers, two FF attention mechanisms, and two downsampling layers; One upsampling layer, one FF attention mechanism, and one concatenation operation are sequentially connected to form one subunit. Two subunits are connected end to end, and the output of the second subunit is concatenated with the output of the first subunit through one downsampling layer. The result is then concatenated with the network input through one downsampling layer.

2. The gesture recognition system based on the LHGDet model according to claim 1, characterized in that, The data collection module includes: an image processing unit, an annotation unit, and a data partitioning unit; The image processing unit is used to rotate and scale the gesture data image set to obtain a processed image set. The annotation unit is used to annotate the processed image set according to different gesture types to obtain an annotated image set; The data partitioning unit is used to divide the labeled image set into a training set and a test set according to a preset ratio to obtain the training data.

3. A gesture recognition method based on the LHGDet model, wherein the recognition method is applied to the recognition system according to any one of claims 1-2, characterized in that, Includes the following steps: Acquire a set of gesture data images and label the set of gesture data images to obtain training data; The NanoDet-Plus model was improved to obtain the initial gesture recognition model; The initial gesture recognition model is trained using the training data to obtain the LHGDet gesture recognition model; Gesture recognition is performed using the LHGDet gesture recognition model.

4. The gesture recognition method based on the LHGDet model according to claim 3, characterized in that, The methods for obtaining the training data include: The gesture data image set is rotated and scaled to obtain a processed image set; The processed image set is labeled according to different gesture types to obtain a labeled image set; The labeled image set is divided into a training set and a test set according to a preset ratio to obtain the training data.

5. The gesture recognition method based on the LHGDet model according to claim 3, characterized in that, The initial gesture recognition model includes: an improved backbone network, an improved posterior neck network, an auxiliary training network, and a detection network; The method for obtaining the improved backbone network includes: adding SE attention mechanism and CA attention mechanism to the backbone network of the NanoDet-Plus model; The method for obtaining the improved posterior neck network includes: adding an FF attention mechanism to the neck network of the NanoDet-Plus model; The auxiliary training network consists of four 3×3 convolutional kernels, and group normalization is used as the normalization layer; The detection network consists of a detection head with shared weights.

6. The gesture recognition method based on the LHGDet model according to claim 5, characterized in that, The improved backbone network includes: block1 and block2; The block1 consists of a first branch and a second branch. The first branch consists of a 1×1 convolutional kernel, a 3×3 convolutional kernel, an SE attention mechanism, and a 1×1 convolutional kernel connected in sequence. The second branch does not process the input data. The outputs of the first branch and the second branch are connected by a concatenation operation. The block2 consists of a third branch and a fourth branch. The third branch is composed of a 1×1 convolutional kernel, a 3×3 convolutional kernel, and a 1×1 convolutional kernel connected in sequence. The fourth branch is composed of a 3×3 convolutional kernel, a CA attention mechanism, and a 1×1 convolutional kernel connected in sequence. The outputs of the third branch and the fourth branch are connected by a concatenation operation.

7. The gesture recognition method based on the LHGDet model according to claim 5, characterized in that, The improved post-neck network includes: two upsampling layers, two FF attention mechanisms, and two downsampling layers; One upsampling layer, one FF attention mechanism, and one concatenation operation are sequentially connected to form one subunit. Two subunits are connected end to end, and the output of the second subunit is concatenated with the output of the first subunit through one downsampling layer. The result is then concatenated with the network input through one downsampling layer.

Images