Gesture recognition method based on lightweight neural network
By using multi-level connected backbone network modules and lightweight detection head modules in gesture recognition networks, combined with deep separation convolution and expansion convolution, the large number of parameters and low accuracy in gesture recognition methods of ultra-lightweight neural networks is solved. Problem, efficient feature extraction and gesture recognition are achieved.
Patent Information
- Application Number
- CN202310536600.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-12
AI Technical Summary
The existing gesture recognition method based on ultra-lightweight neural networks still has a large number of parameters and low accuracy, making it difficult to achieve satisfactory results in deployment and efficiency.
Using multi-level connection backbone network module and lightweight detection head module, combining depth separation convolution and expansion convolution, setting smaller input resolution and grayscale graph input, improving feature extraction and detection capabilities, reducing calculation amount and Parameter quantity.
It realizes efficient feature extraction and gesture recognition, reduces the calculation amount and parameter amount of the gesture recognition network, and maintains a high accuracy and efficiency of gesture recognition.
Smart Images

Figure CN118942149A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to gesture recognition, and in particular to a gesture recognition method based on a lightweight neural network. Background Art
[0002] Gesture recognition is an important part of human-computer interaction, and the traditional neural network method is one of the most widely used methods in gesture recognition. However, there is a problem with using the neural network method for gesture recognition, that is, the number of parameters and the amount of calculation are large, and it is not easy to deploy. Neural network models usually contain multiple layers, and each layer is composed of multiple neurons. In order to train the model and improve the accuracy, it is necessary to optimize the weights between each neuron, which leads to a large number of parameters and calculations, which need to be stored and processed during deployment, and also have high requirements for the deployed hardware.
[0003] An ultra-lightweight neural network refers to a neural network model with very small parameters and computational complexity. This type of neural network model can not only significantly reduce model size, memory consumption, and speed, but also achieve high-precision gesture recognition in low-power environments such as mobile devices. Due to its small number of parameters and computational complexity, ultra-lightweight neural networks are easier to deploy on mobile or embedded devices, and can quickly respond to user operations and achieve real-time gesture recognition, which is very suitable for gesture control applications. In addition, since the amount of training data for ultra-lightweight neural networks is relatively small, it saves the cost of data collection and annotation, and also makes its training more efficient.
[0004] However, the existing gesture recognition methods based on ultra-lightweight neural networks still have the problems of a large number of parameters and low accuracy. Summary of the invention
[0005] The present invention is made to solve the above-mentioned problem, and its purpose is to provide a gesture recognition method based on a lightweight neural network.
[0006] The present invention provides a gesture recognition method based on a lightweight neural network, which has the following characteristics and comprises the following steps: step S1, constructing a neural network model, and using a plurality of existing gesture images as a training set to train the neural network model, so as to obtain the trained neural network model as a gesture recognition model; step S2, acquiring an original image containing a hand through an image sensor; step S3, preprocessing the original image to obtain a preprocessed image; step S4, inputting the preprocessed image into the gesture recognition model, so as to obtain a gesture recognition result of the original image, wherein the gesture recognition model comprises a hand detection network and a hand classification network, the hand detection network comprises a backbone network module and a lightweight detection head module connected in sequence, the backbone network module comprises a first two-dimensional convolutional layer, a first inverted residual linear bottleneck block, a second inverted residual linear bottleneck block, a third inverted residual linear bottleneck block, a fourth inverted residual linear bottleneck block, a fifth inverted residual linear bottleneck block, and a fifth inverted residual linear bottleneck block. The inverted residual linear bottleneck block and the sixth inverted residual linear bottleneck block, the lightweight detection head module includes a first point convolution layer, a first yolo output layer and a second yolo output layer, the output of the first point convolution layer is used as the input of the first yolo output layer and the second yolo output layer respectively, the first yolo output layer includes a first depth convolution layer, a seventh inverted residual linear bottleneck block, a second point convolution layer, a second depth convolution layer, a third depth convolution layer and a third point convolution layer connected in sequence, the second yolo output layer includes a fourth depth convolution layer, an eighth inverted residual linear bottleneck block, a fourth point convolution layer, a fifth depth convolution layer, a sixth depth convolution layer and a fifth point convolution layer connected in sequence, the hand classification network includes a second two-dimensional convolution layer, a third two-dimensional convolution layer, a first depth separable convolution layer, a second depth separable convolution layer, a third depth separable convolution layer, a fourth two-dimensional convolution layer, a fifth two-dimensional convolution layer and a global average pooling layer connected in sequence.
[0007] The gesture recognition method based on a lightweight neural network provided by the present invention may also have the following features: wherein, in step S1, the existing gesture images include gesture images obtained by a web crawler, gesture images obtained by cleaning open source data, and gesture images obtained by collecting real images of the hands.
[0008] The gesture recognition method based on a lightweight neural network provided by the present invention may also have the following features: wherein, in step S3, the specific process of preprocessing is: compressing and stretching the original image to the input resolution size of the hand detection network to obtain a preprocessed image.
[0009] In the gesture recognition method based on lightweight neural network provided by the present invention, it can also have the following characteristics: wherein, step S4 includes the following sub-steps: step S4-1, inputting the preprocessed image into the hand detection network to obtain the position information of the hand in the preprocessed image; step S4-2, cropping the preprocessed image according to the position information to obtain a cropped image containing the hand; step S4-3, compressing and stretching the cropped image to the input resolution size of the hand classification network, and then converting the cropped image into a grayscale image to obtain a grayscale image; step S4-4, inputting the grayscale image into the hand classification network, and converting the output of the hand classification network through a softmax function to obtain the category probabilities of n gestures; step S4-5, obtaining the gesture recognition result of the original image according to the category probabilities of the n gestures.
[0010] The gesture recognition method based on a lightweight neural network provided by the present invention may also have the following characteristics: wherein the gesture recognition results include an “up” gesture, a “left” gesture, a “right” gesture and an “OK” gesture.
[0011] In the gesture recognition method based on lightweight neural network provided by the present invention, it can also have the following characteristics: wherein, the first depth-wise separable convolution layer, the second depth-wise separable convolution layer and the third depth-wise separable convolution layer are all depth-wise separable convolutions including depth-wise convolution and point-wise convolution.
[0012] The gesture recognition method based on a lightweight neural network provided by the present invention may also have the following characteristics: F ×D F ×M feature maps are input to the depthwise separable convolution and output D F ×D F ×N feature map, the convolution kernel of the depthwise separable convolution consists of M convolution kernels with the same input feature channel number as 1 and the same size as the conventional convolution, and the convolution kernel of the point convolution of the depthwise separable convolution consists of N convolution kernels with the size of 1×1 and the same number of output feature channels as the conventional convolution. The computational complexity of the depthwise convolution is C. depth and parameter P depth The formula is as follows: depth =D K ×D K ×D F ×D F ×M,P depth =D K ×D K ×M, where D K ×D K is the size of the depth convolution, the computational cost of the point convolution C depth and parameter P depth The formula is as follows: point =DF ×D F ×M×N,P point =M×N.
[0013] Functions and Effects of the Invention
[0014] According to the gesture recognition method based on a lightweight neural network involved in the present invention, in the construction of the hand detection network, a smaller input resolution is set for the hand detection network, and a depth-separable convolution and an expanded convolution structure are used to achieve efficient feature extraction and reduce the amount of calculation, and the detection capability of small targets is improved by setting the first yolo output layer and the second yolo output layer, and the input is set as a grayscale image in the hand classification network to improve the network's feature learning of gesture shapes and edges, and further reduce the complexity and amount of calculation of the hand classification network. Therefore, the gesture recognition method based on a lightweight neural network of the present invention can reduce the amount of calculation and the amount of parameters of the gesture recognition network while maintaining a high gesture recognition accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 is a flow chart of a gesture recognition method based on a lightweight neural network in an embodiment of the present invention;
[0016] Figure 2 is a schematic diagram of the structure of block 1 in an embodiment of the present invention;
[0017] Figure 3 is a schematic diagram of the structure of block2 in an embodiment of the present invention;
[0018] Figure 4 is a schematic diagram of a hand frame in an embodiment of the present invention;
[0019] Figure 5 is a schematic diagram of a loss function curve of the detection model of the present invention during the training process in an embodiment of the present invention;
[0020] Figure 6 is a schematic diagram of hand detection results of the detection model of the present invention that has been trained in an embodiment of the present invention;
[0021] Figure 7 is a loss function curve of the recognition model of the present invention when a classification training set is used in an embodiment of the present invention;
[0022] Figure 8 is a loss function curve of the recognition model of the present invention when a classification validation set is used in an embodiment of the present invention;
[0023] Fig. 9 is the accuracy curve of the recognition model of the present invention when the classification training set is used in the embodiment of the present invention;
[0024] Fig.10 is the accuracy curve of the recognition model of the present invention when the classification verification set is used in the embodiment of the present invention;
[0025] Fig.11 Schematic diagram of the recognition result of gesture by the recognition model of the present invention in an embodiment of the present invention. DETAILED DESCRIPTION
[0026] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and accompanying drawings specifically illustrate the gesture recognition method based on a lightweight neural network of the present invention.
[0027] Figure 1 4 is a flow chart of a gesture recognition method based on a lightweight neural network in an embodiment of the present invention.
[0028] like Figure 1 As shown, the gesture recognition method based on a lightweight neural network of this embodiment includes the following steps:
[0029] Step S1, constructing a neural network model, and using a plurality of existing gesture images as a training set to train the neural network model, and obtaining the trained neural network model as a gesture recognition model.
[0030] Among them, the existing gesture images include gesture images obtained by web crawlers, gesture images obtained by cleaning open source data, and gesture images obtained by collecting real images of the hands.
[0031] Among them, the gesture recognition model includes a hand detection network and a hand classification network.
[0032] The hand detection network includes a backbone network module and a lightweight detection head module connected in sequence.
[0033] The backbone network module includes a first two-dimensional convolutional layer, a first inverted residual linear bottleneck block, a second inverted residual linear bottleneck block, a third inverted residual linear bottleneck block, a fourth inverted residual linear bottleneck block, a fifth inverted residual linear bottleneck block and a sixth inverted residual linear bottleneck block connected in sequence.
[0034] The lightweight detection head module includes a first point convolution layer, a first yolo output layer and a second yolo output layer. The output of the first point convolution layer is used as the input of the first yolo output layer and the second yolo output layer respectively.
[0035] The first yolo output layer includes a first depthwise convolutional layer, a seventh inverted residual linear bottleneck block, a second pointwise convolutional layer, a second depthwise convolutional layer, a third depthwise convolutional layer, and a third pointwise convolutional layer, which are connected sequentially.
[0036] The second yolo output layer includes the fourth depth convolution layer, the eighth inverse residual linear bottleneck block, the fourth point convolution layer, the fifth depth convolution layer, the sixth depth convolution layer and the fifth point convolution layer, which are connected sequentially.
[0037] The hand classification network includes a second two-dimensional convolution layer, a third two-dimensional convolution layer, a first depth-separable convolution layer, a second depth-separable convolution layer, a third depth-separable convolution layer, a fourth two-dimensional convolution layer, a fifth two-dimensional convolution layer and a global average pooling layer, which are sequentially connected.
[0038] Among them, the first depth-wise separable convolution layer, the second depth-wise separable convolution layer and the third depth-wise separable convolution layer are all depth-wise separable convolutions including depth-wise convolution and point-wise convolution.
[0039] Set the size to D F ×D F ×M feature maps are input to the depthwise separable convolution and output D F ×D F ×N feature map, the convolution kernel of the depthwise separable convolution consists of M convolution kernels with the same size as the conventional convolution and the number of input feature channels is 1, and the convolution kernel of the point convolution of the depthwise separable convolution consists of N convolution kernels with the size of 1×1 and the same number of output feature channels as the conventional convolution.
[0040] The computational cost of depthwise convolution C depth and parameter P depth The formula is as follows:
[0041] C depth =D K ×D K ×D F ×D F ×M,
[0042] P depth =D K ×D K ×M,
[0043] Where D K ×D K is the size of the depthwise convolution.
[0044] The computational cost of point convolution C depth and parameter P depth The formula is as follows:
[0045] C point =D F ×D F ×M×N,
[0046] P point =M×N.
[0047] Step S2: acquiring an original image including the hand through an image sensor.
[0048] Step S3, preprocessing the original image to obtain a preprocessed image.
[0049] Among them, in step S3, the specific process of preprocessing is: compressing and stretching the original image to the input resolution size of the hand detection network to obtain a preprocessed image.
[0050] In this embodiment, the input resolution of the hand detection network is set to 256*192 according to the structure of the model, the distribution of training data, the inference speed and the computing power deployment environment, thereby effectively improving the inference speed of the model.
[0051] Step S4, inputting the preprocessed image into a gesture recognition model to obtain a gesture recognition result of the original image.
[0052] Wherein, step S4 includes the following sub-steps:
[0053] Step S4-1, input the preprocessed image into the hand detection network to obtain the position information of the hand in the preprocessed image.
[0054] Table 1 Structure of the backbone network module of the embodiment of the present invention
[0055]
[0056]
[0057] As shown in Table 1, the first column is the name of each layer in the backbone network module, the second column is the input feature map size data of each layer, the third column is the convolution module contained in each layer, the fourth column is the number of output channels corresponding to each layer, and the fifth column is the final step size of each layer.
[0058] In this embodiment, conv2d 3*3 is a 3*3 two-dimensional convolution, block1 is an inverted residual structure with a stride of 2, and block2 is an inverted residual structure with a stride of 1.
[0059] Figure 2 It is a schematic diagram of the structure of block 1 in an embodiment of the present invention.
[0060] like Figure 2 As shown in the figure, block1 includes 1*1 convolution, 3*3 depth-separable convolution and 1*1 convolution connected in sequence. There is BN+Relu6 function between each layer of convolution. Only the Relu6 function after the 1*1 convolution of the last layer is replaced by the linear function Linear, that is, the linear bottleneck structure. By using the linear bottleneck structure to retain low-dimensional information, the feature extraction ability of the network is improved and the number of parameters and calculation of the network is reduced.
[0061] Figure 3 It is a schematic diagram of the structure of block2 in an embodiment of the present invention.
[0062] like Figure 3 As shown in the figure, block2 includes a 1*1 convolution, a 3*3 depth-separable convolution, a 1*1 convolution and a dropout layer connected in sequence, and the output of the dropout layer is concatenated with the input of block2 to obtain the output of block2.
[0063] Table 2 Structure of the lightweight detection head module of the embodiment of the present invention
[0064]
[0065]
[0066] As shown in Table 2, the first column is the name of each layer in the lightweight detection head module, the second column is the input feature map size data of each layer, the third column is the convolution module contained in each layer, the fourth column is the number of output channels corresponding to each layer, and the fifth column is the final step size of each layer.
[0067] In this embodiment, Point conv is point convolution, depthwise conv 3*3 is 3*3 depth-separable convolution, and depthwise conv 5*5 is 5*5 depth-separable convolution. By reusing 3*3 depth-separable convolution and 5*5 depth-separable convolution in the lightweight detection head module, a larger receptive field is obtained, thereby extracting higher-dimensional features.
[0068] In this embodiment, the first yolo output layer outputs a feature map of 8*6*18, and the second yolo output layer outputs a feature map of 16*12*18. The larger feature map is responsible for detecting small targets, and the smaller feature map is responsible for detecting large targets, thereby making up for the ability to detect small targets. Among the 18 output channels, 3*4=12 channels are responsible for predicting the position of the rectangular box, 3 channels are responsible for predicting the category, and 3 channels are responsible for predicting the confidence.
[0069] Step S4-2, cropping the preprocessed image according to the position information to obtain a cropped image containing the hand.
[0070] Step S4-3, compress and stretch the cropped image to the input resolution size of the hand classification network, and then convert the cropped image into a grayscale image to obtain a grayscale image. The grayscale image can enable the hand classification network to focus more on learning the shape and edge features of the gesture.
[0071] Step S4-4, input the grayscale image into the hand classification network, and transform the output of the hand classification network through the softmax function to obtain the category probabilities of n types of gestures.
[0072] Table 3 Structure of the hand classification network of the embodiment of the present invention
[0073]
[0074]
[0075] As shown in Table 3, the first column is the name of each layer in the hand classification network, the second column is the input feature map size data of each layer, the third column is the convolution module contained in each layer, the fourth column is the number of output channels corresponding to each layer, and the fifth column is the final step size of each layer.
[0076] In this embodiment, Global_average_pooling is a global average pooling layer, and the output of the hand classification network is a 1*1*5 feature map, which is then transformed by a softmax function to obtain the category probabilities of the four gestures.
[0077] Step S4-5, obtaining the gesture recognition result of the original image according to the category probabilities of the n gestures.
[0078] Among them, the gesture recognition results include "up" gesture, "left" gesture, "right" gesture and "OK" gesture.
[0079] In this embodiment, 6504 pictures containing human hands that are close to actual application scenarios are obtained through web crawlers, open source data set cleaning, real image collection, etc., and then the labelImg software is used to generate a label file corresponding to the picture for the hand frame in each picture. Then all pictures and the corresponding label files are used as hand detection data sets, and part of the data is randomly selected from the hand detection data set as detection training data, and the remaining part is used as detection test data.
[0080] Figure 4 Schematic diagram of a hand frame in an embodiment of the present invention.
[0081] like Figure 4 As shown, the white rectangular frame surrounding the hand in the picture in the labelImg software is the hand frame.
[0082] A hardware environment with a training graphics card model of NVDIA GeForce RTX 3080, a memory of 10GB, a processor model of Intel(R) Core(TM) i7-10700K CPU@3.80GHz, and a memory of 16GB is constructed. In the hardware environment, the darknet training framework is compiled under the windows operating system. After configuring the cfg file of the detection network, the hand detection model constructed by the hand detection network according to the present invention, i.e., the detection model of the present invention, the existing YOLOv3-tiny hand detection model, i.e., the YOLOv3-tiny model, and the existing YOLOv4-tiny hand detection model, i.e., the YOLOv4-tiny model, are trained according to the detection training data. The training settings include: single batch (batchsize) The initial learning rate (learning_rate) is 48, the initial learning rate (learning_rate) is 0.001, the number of iterations is set to 100000, the Adam optimizer is used, the weight decay coefficient is set to 0.0005, the data enhancement setting enhancement exposure coefficient and saturation coefficient are 1.5, the IoU loss uses CIOU, the nms algorithm uses greedynms, the nms threshold is set to 0.6, and the k-means clustering method is used to calculate the prior Anchor for the data set label before training, so as to obtain the trained detection model of the present invention, the YOLOv3-tiny model and the YOLOv4-tiny model.
[0083] Figure 5 It is a schematic diagram of the loss function curve of the detection model of the present invention during the training process in an embodiment of the present invention.
[0084] like Figure 5 As shown, the horizontal axis is the number of iterations, and the vertical axis is the value of the loss function. After 100,000 iterations during the training process, the value of the loss function has completely converged, and the detection model of the present invention has been trained.
[0085] Figure 6 It is a schematic diagram of the hand detection results of the detection model of the present invention that has been trained in an embodiment of the present invention.
[0086] like Figure 6 As shown, the black rectangular box is the detection result of the hand in the picture by the detection model of the present invention, and the "hand 1.00" above the black rectangular box is the estimated accuracy of the detection result by the detection model of the present invention, that is, the estimated accuracy of the detection model of the present invention in judging that the black rectangular box is a hand is 100%.
[0087] The detection test data is input into the three trained models to obtain the hand detection accuracy of each model. The model input resolution, floating point operations per second FLOPs (i.e., calculation amount), weight size, parameter amount, and detection accuracy of the three models are shown in the following table:
[0088]
[0089] In the above table, the first column is the model name, the second column is the resolution requirement of the image input corresponding to the model, the third column is the calculation amount of each trained model, the fourth column is the weight size of each trained model, the fifth column is the parameter amount of each trained model, and the sixth column is the hand detection accuracy of each model trained under the test data. For example, the cell in the fourth row and sixth column indicates that the hand detection accuracy of the detection model of the present invention for the test data is 77.40%. It can be seen from the above table that the calculation amount, weight size and parameter amount of the detection model of the present invention are much smaller than those of the YOLOv3-tiny model and the YOLOv4-tiny model, and the detection accuracy is better than the YOLOv3-tiny model and slightly lower than the YOLOv4-tiny model, so the hand detection network of the present invention can achieve hand detection accuracy similar to that of the existing model, and exponentially reduce the complexity and volume of the model construction, thereby obtaining an ultra-lightweight hand detection model.
[0090] In this embodiment, 8,000 palm-up pictures and 7,000 "OK" gestures are obtained by means of web crawlers, open source data set cleaning, real image collection, etc., and then the 8,000 palm-up pictures are flipped left and right to obtain three types of gestures: "up" gesture, "left" gesture and "right" gesture. The "up" gesture, "left" gesture, "right" gesture and "OK" gesture correspond to the "pause", "fast rewind", "fast forward" and "play" command operations of the music player respectively. Then, negative pictures, i.e., various gesture pictures other than the above four gestures, are added, and the pictures are augmented by adjusting exposure, contrast, translation, zooming, etc., and each type of picture is augmented to 15,000 pictures, so that 60,000 pictures are obtained as the classification data set, and the classification data set is divided into a classification training set, a classification verification set and a classification test set according to the ratios of 85%, 10% and 5%.
[0091] Intel-cpu (Intel (R) Core (TM) i7-10700K CPU @ 3.80GHz) single-threaded reasoning (windows10, x64), i.e., intel-cpu, and microcontroller i.MX RT1170 (ARM Cortex-M71GHz), i.e., rt1170, are selected as test platforms respectively, and tensorflow framework is selected as training framework, and a hand recognition model constructed according to the gesture recognition method based on a lightweight neural network of the present invention, i.e., the recognition model of the present invention, an existing YOLOv3-tiny hand recognition model, i.e., the YOLOv3-tiny model, and an existing YOLOv4-tiny hand recognition model, i.e., the YOLOv4-tiny model, are trained through classification training sets and classification verification sets. The training settings include: a single batch batchsize is set to 48, an iteration epoch is set to 300, and a softmax function is selected as a loss function, so as to obtain the trained recognition model of the present invention, the YOLOv3-tiny model, and the YOLOv4-tiny model.
[0092] Figure 7 It is the loss function curve of the recognition model of the present invention when the classification training set is used in the embodiment of the present invention.
[0093] Figure 8 It is the loss function curve of the recognition model of the present invention when the classification verification set is used in the embodiment of the present invention.
[0094] like Figure 7 , 8 As shown, Figure 7 , 8 The horizontal axis is the number of iterations, i.e. Epoch, and the vertical axis is the value of the loss function, i.e. Loss. After 300 iterations, the loss function values of the classification training set and the classification verification set have converged, and the recognition model training of the present invention is completed.
[0095] Fig. 9 It is the accuracy curve of the recognition model of the present invention when the classification training set is used in the embodiment of the present invention.
[0096] Fig.10 It is the accuracy curve of the recognition model of the present invention when the classification verification set is used in the embodiment of the present invention.
[0097] like Fig. 9 , 10 As shown, Fig. 9 , 10The horizontal axis is the number of iterations, and the vertical axis is the accuracy value. After 300 iterations, the accuracy of gesture recognition of the recognition model of the present invention on the training set is 99.8%, and the accuracy of gesture recognition on the verification set is 99.6%, indicating that the trained recognition model of the present invention has good gesture recognition accuracy.
[0098] Each model is tested on two test platforms using the classification test set, and the inference time of each model is shown in the following table:
[0099] Test platform YOLOv3-tiny model YOLOv4-tiny model Identification model of the present invention intel-cpu 88.28ms 90.09ms 7.52ms rt1170 4442ms 5531ms 161ms
[0100] The first column in the table is the name of the test platform, the second column is the average reasoning time of the YOLOv3-tiny model on the classification test set in each test platform, the third column is the average reasoning time of the YOLOv4-tiny model on the classification test set in each test platform, and the fourth column is the average reasoning time of the recognition model of the present invention on the classification test set in each test platform. For example, the cell in the third row and fourth column indicates that the average reasoning time of the recognition model of the present invention on the classification test set in the rt1170 test platform is 161ms. It can be seen from the above table that under the same test platform, the average reasoning time consumption of the classification model of the present invention is exponentially reduced compared with the YOLOv4-tiny model and the YOLOv3-tiny model, so the hand recognition model constructed by the gesture recognition method based on the lightweight neural network of the present invention has the ability of lightweight and fast reasoning.
[0101] Fig.11 Schematic diagram of the recognition result of gesture by the recognition model of the present invention in an embodiment of the present invention.
[0102] like Fig.11 As shown, the recognition model of the present invention detects the hand position in the picture according to the picture on the left side of (a), (b), (c) and (d) and marks it with a black box, and then performs gesture recognition on the hand in the black box to obtain the gesture recognition results on the right side of (a), (b), (c) and (d), that is, gesture recognition. The gesture recognition result of (a) is a "right" gesture, which corresponds to the "fast forward" instruction operation of the music player, the gesture recognition result of (b) is a "left" gesture, which corresponds to the "fast rewind" instruction operation of the music player, the gesture recognition result of (c) is an "OK" gesture, which corresponds to the "play" instruction operation of the music player, and the gesture recognition result of (d) is an "up" gesture, which corresponds to the "pause" instruction operation of the music player.
[0103] Functions and Effects of the Embodiments
[0104] According to the gesture recognition method based on a lightweight neural network involved in this embodiment, in the construction of the hand detection network, a smaller input resolution is set for the hand detection network, and a depth-separable convolution and an expanded convolution structure are used to achieve efficient feature extraction and reduce the amount of calculation, and the detection capability of small targets is improved by setting the first yolo output layer and the second yolo output layer, and the input is set as a grayscale image in the hand classification network to improve the network's feature learning of gesture shapes and edges, and further reduce the complexity and amount of calculation of the hand classification network, so the parameter amount of the hand detection network constructed in this embodiment is 0.17M, the amount of calculation is 0.094BFLOPs, the parameter amount of the hand classification network is 0.09M, and the amount of calculation is 0.032BFLOPs, thereby obtaining a gesture recognition model that exceeds the existing lightweight network. In short, this method can reduce the amount of calculation and parameter amount of the gesture recognition network while maintaining a high gesture recognition accuracy and efficiency.
[0105] The above-mentioned embodiments are preferred examples of the present invention and are not intended to limit the protection scope of the present invention.
Claims
1. A gesture recognition method based on a lightweight neural network, characterized in that: The following steps are involved: Step S1, constructing a neural network model, and using a plurality of existing gesture images as a training set to train the neural network model, to obtain the trained neural network model as a gesture recognition model; Step S2, acquiring an original image including the hand through an image sensor; Step S3, preprocessing the original image to obtain a preprocessed image; Step S4, inputting the pre-processed image into the gesture recognition model to obtain a gesture recognition result of the original image, The gesture recognition model includes a hand detection network and a hand classification network. The hand detection network includes a backbone network module and a lightweight detection head module connected in sequence. The backbone network module includes a first two-dimensional convolutional layer, a first inverted residual linear bottleneck block, a second inverted residual linear bottleneck block, a third inverted residual linear bottleneck block, a fourth inverted residual linear bottleneck block, a fifth inverted residual linear bottleneck block and a sixth inverted residual linear bottleneck block, which are connected in sequence. The lightweight detection head module includes a first point convolution layer, a first yolo output layer and a second yolo output layer, wherein the output of the first point convolution layer is used as the input of the first yolo output layer and the second yolo output layer respectively. The first yolo output layer includes a first depth convolution layer, a seventh inverse residual linear bottleneck block, a second point convolution layer, a second depth convolution layer, a third depth convolution layer and a third point convolution layer connected in sequence. The second YOLO output layer includes a fourth depth convolution layer, an eighth inverse residual linear bottleneck block, a fourth point convolution layer, a fifth depth convolution layer, a sixth depth convolution layer and a fifth point convolution layer connected in sequence. The hand classification network includes a second two-dimensional convolutional layer, a third two-dimensional convolutional layer, a first depth-separable convolutional layer, a second depth-separable convolutional layer, a third depth-separable convolutional layer, a fourth two-dimensional convolutional layer, a fifth two-dimensional convolutional layer and a global average pooling layer, which are connected in sequence.
2. The gesture recognition method based on a lightweight neural network according to claim 1, characterized in that: in, In the step S1, the existing gesture images include gesture images obtained by web crawlers, gesture images obtained by cleaning open source data, and gesture images obtained by collecting real images of the hands.
3. The gesture recognition method based on a lightweight neural network according to claim 1, characterized in that: in, In the step S3, the specific process of the preprocessing is: compressing and stretching the original image to the input resolution size of the hand detection network, thereby obtaining the preprocessed image.
4. The gesture recognition method based on a lightweight neural network according to claim 1, Features: Wherein, the step S4 includes the following sub-steps: Step S4-1, inputting the preprocessed image into the hand detection network to obtain the position information of the hand in the preprocessed image; Step S4-2, cropping the pre-processed image according to the position information to obtain a cropped image containing the hand; Step S4-3, compressing and stretching the cropped image to the input resolution size of the hand classification network, and then converting the cropped image into a grayscale image to obtain a grayscale image; Step S4-4, inputting the grayscale image into the hand classification network, and transforming the output of the hand classification network through a softmax function to obtain the category probabilities of n types of gestures; Step S4-5: obtaining the gesture recognition result of the original image according to the category probabilities of the n gestures.
5. The gesture recognition method based on a lightweight neural network according to claim 1 or 4, characterized in that: in, The gesture recognition results include an “up” gesture, a “left” gesture, a “right” gesture and an “OK” gesture.
6. The gesture recognition method based on a lightweight neural network according to claim 1, characterized in that: in, The first depth-wise separable convolution layer, the second depth-wise separable convolution layer, and the third depth-wise separable convolution layer are all depth-wise separable convolutions including depth-wise convolution and point-wise convolution.
7. The gesture recognition method based on a lightweight neural network according to claim 6, characterized in that: in, Set the size to D F ×D F The feature map of ×M is input into the depth-wise separable convolution and outputs D F ×D F ×N feature graph, the convolution kernel of the depthwise separable convolution consists of M convolution kernels whose input feature channels are 1 and whose size is the same as that of the conventional convolution, and the convolution kernel of the point convolution of the depthwise separable convolution consists of N convolution kernels whose size is 1×1 and whose output feature channels are the same as that of the conventional convolution. The computational cost of the depthwise convolution is C depth and parameter P depth The formula is as follows: C depth =D K ×D K ×D F ×D F ×M, P depth =D K ×D K ×M, Where D K ×D K is the size of the depthwise convolution, The computational cost of the point convolution is C depth and parameter P depth The formula is as follows: C point =D F ×D F ×M×N, P point =M×N。
