A gesture recognition method and system based on ShuffleNet-SSD

Abstract

The present invention discloses a gesture recognition method and system based on ShuffleNet-SSD. The gesture images are collected, converted into fixed sizes, labeled, and a data set is constructed with a certain ratio of training set to test set; a ShuffleNet-SSD network model is constructed; the ShuffleNet-SSD model is trained, and parameters are adjusted and optimized; the trained ShuffleNet-SSD model is used to realize gesture recognition. The present invention proposes a lightweight network framework to replace the VGG16 framework to reduce computing resource consumption. Feature extraction is performed through the ShuffleNet network to improve recognition accuracy and reduce resource consumption.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and mainly relates to a gesture recognition method and system based on ShuffleNet-SSD. Background Art

[0002] As the most human-friendly interaction method, gestures have performed well in the field of virtual simulation. With the increasing pursuit of interactive experience, higher requirements are also placed on gesture recognition technology used in gesture interaction.

[0003] At present, gesture recognition methods based on deep learning are mainly divided into two categories: two-step algorithm and single-step algorithm. The two-step algorithm divides the recognition process into two stages: first, generating candidate regions, and then classifying candidate regions. Typical representatives of this type of algorithm are R-CNN algorithms, such as R-CNN, SPPNet, Fast R-CNN, Faster R-CNN, FPN, R-FCN, etc. The single-step algorithm does not need to generate candidate regions. It directly generates the category probability and position coordinate value of the object. After a single detection, the final detection result can be directly obtained. Therefore, it has a faster detection speed. Typical algorithms include YOLO, SSD, and Retina-Net. Among them, SSD is a very excellent gesture recognition method. The SSD algorithm uses the VGG16 network for feature extraction. Although the model has good performance, it has a bloated structure and a large number of parameters, which reduces the speed of model detection. For human-computer interaction, due to space, cost and other limitations and the requirement for real-time performance, the SSD algorithm needs to be further optimized.

[0004] Non-maximum suppression (NMS) is an important component of gesture recognition. The NMS algorithm first sorts the detection frames from high to low according to the score, then the detection frame with the highest score is selected, and other frames that have obvious overlap with the selected frame are suppressed. This process is continuously recursively applied to the remaining detection frames. According to the design of the algorithm, if an object is within the preset overlap threshold, it may result in the failure to detect the object to be detected. That is, when two target frames are close, the frame with a lower score will be deleted because of the large overlap area with it. Summary of the invention

[0005] The purpose of the present invention is to provide a gesture recognition method based on ShuffleNet-SSD in view of the deficiencies in the prior art.

[0006] The present invention comprises the following steps:

[0007] Step (1), collect gesture images, convert them into a fixed size of 224*224, annotate them, and construct a dataset with a ratio of 9:1 between the training set and the test set;

[0008] Step (2), constructing the ShuffleNet-SSD network model;

[0009] The ShuffleNet-SSD network includes the feature extraction network ShuffleNet and the softened non-maximum suppression block Soft-NMS;

[0010] As a preferred method, the ShuffleNet network uses the idea of ​​grouped convolution and channel shuffling, and performs well in small networks. Grouped convolution greatly reduces the computational cost by ensuring that each convolution operation only acts on the corresponding input channel group. Channel shuffling allows grouped convolution to obtain input data from different groups, avoiding the situation where the output of a channel only comes from a small part of the input channel. The ShuffleNet network contains two units, namely the ShuffleNet unit with a step size of 1 and the ShuffleNet unit with a step size of 2.

[0011] The ShuffleNet unit with a stride of 1 uses a 1*1 grouped convolution, followed by a channel shuffle, followed by a 3*3 depth-wise convolution, and then a second grouped convolution to restore the channel dimension. At the same time, a BN layer is connected after each convolution layer, and a ReLU function is required after the first grouped convolution in addition to the BN layer.

[0012] The ShuffleNet unit with a step size of 2 adds a 3*3 average pooling layer on the bypass of the ShuffleNet unit with a step size of 1, and replaces the addition operation with a cascade operation to increase the dimension.

[0013] The feature extraction network ShuffleNet network structure includes a Conv1 layer, a MaxPool layer, a Stage2 layer, a Stage3 layer, and a Stage4 layer.

[0014] The Conv1 layer inputs a 224*224 image through a 3*3 convolution and outputs a 112*112 feature map to the MaxPool layer;

[0015] The MaxPool layer outputs the 112*112 feature map as a 56*56 feature map to the Stage2 layer through maximum pooling;

[0016] The Stage2 layer includes 1 ShuffleNet unit with a step size of 2 and 3 ShuffleNet units with a step size of 1, and outputs a 28*28 feature map to the Stage3 layer;

[0017] The Stage3 layer includes 1 ShuffleNet unit with a step size of 2 and 7 ShuffleNet units with a step size of 1, and outputs a 14*14 feature map to the Stage4 layer;

[0018] The Stage 4 layer includes 1 ShuffleNet unit with a step size of 2 and 3 ShuffleNet units with a step size of 1, and outputs a 7*7 feature map.

[0019] The softened non-maximum suppression block Soft-NMS is used to remove highly overlapping detection frames in the feature map output by the feature extraction network ShuffleNet;

[0020] As a preferred method, compared with the traditional NMS algorithm, the detection boxes are first sorted from high to low according to the scores, and then the detection box with the highest score is selected, and other boxes that have obvious overlap with the selected suggested box are suppressed. This process is continuously recursively applied to the remaining detection boxes. According to the design of the algorithm, if an object is within the preset overlap threshold, it may cause the object to be detected to not be detected. During the execution of the algorithm, Soft-NMS does not simply delete the detection boxes whose IoU is greater than the given threshold, but reduces the score. The core idea of ​​the Soft-NMS algorithm is

[0021] si←sif(iou(M,bi))(1)

[0022] Wherein, M is the set of detection frames in the feature map output by the feature extraction network ShuffleNet, bi is the other frames except the highest confidence frame, Si is the score, f(iou(M,bi)) is the weight function, which will attenuate the detection scores of adjacent detection frames that overlap with the detection frame M. The more highly overlapping the detection frames with M, the more serious their score attenuation. The result of iou and threshold is used as the basis for reducing the score Si. Finally, the threshold is set to remove the scores Si less than the threshold. The weight function selected by the present invention is the Gaussian function, and the formula is as follows:

[0023]

[0024] Among them, D is the detection box suppressed by Soft-NMS, σ is a manually specified parameter, and Si is the score of the current detection box.

[0025] Step (3), training the ShuffleNet-SSD model, and adjusting and optimizing its parameters;

[0026] Step (4): Use the trained ShuffleNet-SSD model to realize gesture recognition.

[0027] Another object of the present invention is to provide a gesture recognition system based on ShuffleNet-SSD, comprising:

[0028] Dataset creation module, creating training sets for training ShuffleNet-SSD;

[0029] A model training module, used for inputting the preprocessed gesture images in the training set into the ShuffleNet-SSD model for training;

[0030] The gesture recognition module is used to predict the gesture to be recognized through the trained ShuffleNet-SSD model and generate recognition results.

[0031] The beneficial effects of the present invention are:

[0032] 1) This paper proposes a lightweight network framework to replace the VGG16 framework to reduce computing resources. The ShuffleNet network is used to extract features, improve recognition accuracy, and reduce resource consumption.

[0033] 2) The Soft-NMS function is proposed to improve the recognition accuracy while only requiring simple changes to the traditional NMS algorithm without adding additional parameters. The Soft-NMS algorithm has the same algorithm complexity as the traditional NMS, but is more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Gesture recognition flow chart;

[0035] Figure 2 ShuffleNet-SSD network structure diagram;

[0036] Figure 3 Schematic diagram of ShuffleNet unit with a step size of 1;

[0037] Figure 4 Schematic diagram of a ShuffleNet unit with a stride of 2. DETAILED DESCRIPTION

[0038] The present invention will be further analyzed below in conjunction with the accompanying drawings.

[0039] A gesture recognition method based on ShuffleNet-SSD includes the following steps:

[0040] Step (1), collect gesture images, convert them into a fixed size of 224*224, annotate them, and construct a dataset with a ratio of 9:1 between the training set and the test set;

[0041] Step (2), constructing the ShuffleNet-SSD network model;

[0042] like Figure 2-4 The ShuffleNet-SSD network includes the ShuffleNet network and the softened non-maximum suppression block Soft-NMS. The ShuffleNet network uses the idea of ​​grouped convolution and channel shuffling, and has good performance in small networks. Grouped convolution greatly reduces the computational cost by ensuring that each convolution operation only acts on the corresponding input channel group. Channel shuffling allows grouped convolution to obtain input data from different groups, avoiding the situation where the output of a channel only comes from a small part of the input channel. The ShuffleNet network contains two units, namely Figure 3 The ShuffleNet unit with stride 1 is shown and Figure 4 The ShuffleNet unit shown has a step size of 2. The ShuffleNet network structure includes Conv1 layer, MaxPool layer, Stage2 layer, Stage3 layer, and Stage4 layer. The Conv1 layer inputs a 224*224 image through 3*3 convolution and outputs a 112*112 feature map to the MaxPool layer;

[0043] The MaxPool layer outputs the 112*112 feature map as a 56*56 feature map to the Stage2 layer through maximum pooling;

[0044] The Stage2 layer includes 1 ShuffleNet unit with a step size of 2 and 3 ShuffleNet units with a step size of 1, and outputs a 28*28 feature map to the Stage3 layer;

[0045] The Stage3 layer includes 1 ShuffleNet unit with a step size of 2 and 7 ShuffleNet units with a step size of 1, and outputs a 14*14 feature map to the Stage4 layer;

[0046] The Stage 4 layer includes 1 ShuffleNet unit with a step size of 2 and 3 ShuffleNet units with a step size of 1, and outputs a 7*7 feature map.

[0047] The output feature maps of the above layers are input into the SSD prediction module, and then the results are output through the softened non-maximum suppression block Soft-NMS.

[0048] Step (3), training the ShuffleNet-SSD model, and adjusting and optimizing its parameters;

[0049] Step (4): Use the trained ShuffleNet-SSD model to realize gesture recognition.

[0050] The above embodiments are not limitations of the present invention, and the present invention is not limited to the above embodiments. As long as the requirements of the present invention are met, they belong to the protection scope of the present invention.

Claims

1. A gesture recognition method based on ShuffleNet-SSD, characterized in that The following steps are involved: Step (1), collect gesture images, convert them into a fixed size, annotate them with labels, and construct a dataset with a certain ratio of training set to test set; Step (2), constructing the ShuffleNet-SSD network model; The ShuffleNet-SSD network includes the feature extraction network ShuffleNet and the softened non-maximum suppression block Soft-NMS; The feature extraction network ShuffleNet network structure includes Conv1 layer, MaxPool layer, Stage2 layer, Stage3 layer, and Stage4 layer; The Conv1 layer inputs a 224*224 image through a 3*3 convolution and outputs a 112*112 feature map to the MaxPool layer; The MaxPool layer outputs the 112*112 feature map as a 56*56 feature map to the Stage2 layer through maximum pooling; The Stage2 layer includes 1 ShuffleNet unit with a step size of 2 and 3 ShuffleNet units with a step size of 1, and outputs a 28*28 feature map to the Stage3 layer; The Stage3 layer includes 1 ShuffleNet unit with a step size of 2 and 7 ShuffleNet units with a step size of 1, and outputs a 14*14 feature map to the Stage4 layer; The Stage 4 layer includes 1 ShuffleNet unit with a step size of 2 and 3 ShuffleNet units with a step size of 1, outputting a 7*7 feature map; The softened non-maximum suppression block Soft-NMS is specifically as follows: s i ←s i f(iou(M,b i )) (1) Where M is the set of detection boxes in the feature map output by the feature extraction network ShuffleNet, bi is the other boxes except the highest confidence box, Si is the score of the current detection box, and iou represents the confidence; f(iou(M,bi)) is the weight function, specifically: Among them, D is the detection box suppressed by Soft-NMS, and σ is a manually specified parameter; Step (3), training the ShuffleNet-SSD model, and adjusting and optimizing its parameters; Step (4): Use the trained ShuffleNet-SSD model to realize gesture recognition.

2. The method according to claim 1, characterized in that The softened non-maximum suppression block Soft-NMS is used to remove highly overlapping detection boxes in the feature map output by the feature extraction network ShuffleNet.

3. A gesture recognition system based on ShuffleNet-SSD that implements the gesture recognition method according to claim 1, characterized in that include: Dataset creation module, creating training sets for training ShuffleNet-SSD; A model training module, used for inputting the preprocessed gesture images in the training set into the ShuffleNet-SSD model for training; The gesture recognition module is used to predict the gesture to be recognized through the trained ShuffleNet-SSD model and generate recognition results.

Images