A static gesture recognition method and system based on deep learning

Abstract

The present invention relates to a static gesture recognition method and system based on deep learning, the recognition method comprising the following steps: inputting an image to be recognized with a static gesture into the backbone network for feature extraction to obtain a feature map of the image to be recognized; inputting the feature map into the detection network and the segmentation network respectively, using the detection network to perform target detection on the feature map to obtain a prediction frame, using the segmentation network to perform foreground segmentation on the feature map to obtain a foreground heat map, and using the prediction frame to crop the foreground heat map to obtain a target heat map; inputting the target heat map into the classification network to recognize static gestures. The present invention realizes the recognition of static gestures based on a deep learning neural network model, and the detection network and the segmentation network can be executed in parallel. Compared with a cascade neural network, it has a smaller number of parameters, a shorter execution time and a higher efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent devices, and in particular to a static gesture recognition method and system based on deep learning. Background Art

[0002] Aging and various diseases have caused the elderly to lose their ability to walk and move to varying degrees. Many elderly people need wheelchairs for assisted mobility. Some disabled people who cannot walk also need wheelchairs for assisted mobility. Providing the elderly and disabled with wheelchairs that are superior in performance and easy to operate can greatly improve the mobility of these two groups and help them better integrate into society. Smart wheelchairs have been widely promoted in recent years because of their strong autonomy and friendly interaction.

[0003] Smart wheelchairs are products designed for the elderly and disabled. The characteristics of users should be fully considered during design. The interaction method should give full play to the initiative of users and avoid the defects of users themselves. In order to achieve the above purpose, the interaction methods of smart wheelchairs are generally diverse, and interaction based on gesture recognition is one of the important interaction methods. For elderly people with upper limb weakness or Parkinson's syndrome, it is relatively difficult to operate the wheelchair using the joystick. The above problem can be solved well by converting different gestures into different control information. In an overly noisy environment, the effect of voice recognition will be very poor, but in a very quiet environment, patients do not want to disturb others, and using gesture recognition interaction can adapt well to the above environment. Summary of the invention

[0004] In view of this, the present invention provides a static gesture recognition method and system based on deep learning to achieve gesture recognition, and further studies on intelligent wheelchair control based on gesture recognition interaction.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A static gesture recognition method based on deep learning, the recognition method is based on a trained deep learning neural network model, the deep learning neural network model includes a first neural network sub-model and a second neural network sub-model, the first neural network sub-model includes a backbone network, and a detection network and a segmentation network connected to the backbone network and arranged in parallel, the second neural network sub-model includes a classification network;

[0007] The identification method comprises the following steps:

[0008] Inputting the image to be recognized with the static gesture into the backbone network for feature extraction to obtain a feature map of the image to be recognized;

[0009] The feature map is input into the detection network and the segmentation network respectively, the detection network is used to perform target detection on the feature map to obtain a prediction box, the segmentation network is used to perform foreground segmentation on the feature map to obtain a foreground heat map, and the foreground heat map is cropped using the prediction box to obtain a target heat map; the target is a hand area used to characterize a static gesture, and the target heat map is a heat map of the hand area;

[0010] The target heat map is input into the classification network to recognize static gestures.

[0011] Optionally, the backbone network includes a convolution layer Conv1_0, a maximum pooling layer Maxpool1-1, a residual block Block1, a residual block Block2, a residual block Block3 and a residual block Block4 which are connected in sequence; a BN+Relu module is arranged between the convolution layer Conv1_0 and the maximum pooling layer Maxpool1-1.

[0012] Optionally, the residual block Block1, the residual block Block2, the residual block Block3 and the residual block Block4 each include a convolution layer Conv1_1, a convolution layer Conv1_2, a first feature fusion module, a convolution layer Conv1_3, a convolution layer Conv1_4 and a second feature fusion module;

[0013] The input end of the convolution layer Conv1_1 is connected to the image input end; the output end of the convolution layer Conv1_1 is connected to the input end of the convolution layer Conv1_2, the output end of the convolution layer Conv1_2 is connected to an input end of a first feature fusion module, the other input end of the first feature fusion module is connected to the input end of the convolution layer Conv1_1, the output end of the first feature fusion module is connected to the input end of the convolution layer Conv1_3, the output end of the convolution layer Conv1_3 is connected to the input end of the convolution layer Conv1_4, the output end of the convolution layer Conv1_4 is connected to an input end of the second feature fusion module, the other input end of the second feature fusion module is connected to the input end of the convolution layer Conv1_3, and the output end of the second feature fusion module is connected to the image output end;

[0014] A BN+Relu module is provided between the output end of the convolution layer Conv1_1 and the input end of the convolution layer Conv1_2, and between the output end of the convolution layer Conv1_3 and the input end of the convolution layer Conv1_4;

[0015] A BN module is provided between the output end of the convolution layer Conv1_2 and one input end of the first feature fusion module, and between the output end of the convolution layer Conv1_4 and one input end of the second feature fusion module;

[0016] A Relu module is provided between the output end of the first feature fusion module and the input end of the convolution layer Conv1_3, and between the output end of the second feature fusion module and the image output end;

[0017] The residual block Block2, the residual block Block3 and the residual block Block4 each further include a convolution layer Conv1_5;

[0018] The convolution layer Conv1_5 is arranged between the input end of the convolution layer Conv1_1 and another input end of the first feature fusion module.

[0019] Optionally, the detection network includes a convolution layer Conv2_1, a deconvolution layer deConv2_1, a convolution layer Conv2_2, a deconvolution layer deConv2_2, a convolution layer Conv2_3 and a deconvolution layer deConv2_3 connected in sequence;

[0020] The output end of the deconvolution layer deConv2_3 is also connected to the input end of the convolution layer Conv2_4_1, the input end of the convolution layer Conv2_4_2 and the input end of the convolution layer Conv2_4_3 respectively;

[0021] The output end of the convolution layer Conv2_4_1 is connected to the input end of the convolution layer Conv2_5_1, the output end of the convolution layer Conv2_4_2 is connected to the input end of the convolution layer Conv2_5_2, and the output end of the convolution layer Conv2_4_3 is connected to the input end of the convolution layer Conv2_5_3; the output end of the convolution layer Conv2_5_1 is used to output the center point of the target, the output end of the convolution layer Conv2_5_2 is used to output the offset of the center point, and the output end of the convolution layer Conv2_5_3 is used to output the frame size of the target; the center point, the center point offset and the frame size are all features of the prediction box;

[0022] A BN+Relu module is provided between the convolution layer Conv2_1 and the deconvolution layer deConv2_1, between the deconvolution layer deConv2_1 and the convolution layer Conv2_2, between the convolution layer Conv2_2 and the deconvolution layer deConv2_2, between the deconvolution layer deConv2_2 and the convolution layer Conv2_3, and between the convolution layer Conv2_3 and the deconvolution layer deConv2_3;

[0023] A Relu module is provided between the convolution layer Conv2_4_1 and the convolution layer Conv2_5_1, between the convolution layer Conv2_4_2 and the convolution layer Conv2_5_2, and between the convolution layer Conv2_4_3 and the convolution layer Conv2_5_3.

[0024] Optionally, the segmentation network includes a first upsampling layer connected to the output end of the backbone network, an adaptive average pooling layer Avgpool1, an adaptive average pooling layer Avgpool2, an adaptive average pooling layer Avgpool3 and an adaptive average pooling layer Avgpool4;

[0025] The segmentation network also includes convolutional layers Conv3_1_1, Convolutional layers Conv3_1_2, Convolutional layers Conv3_1_3, and Convolutional layers Conv3_1_4, respectively connected to adaptive average pooling layers Avgpool1, adaptive average pooling layers Avgpool2, adaptive average pooling layers Avgpool3, and adaptive average pooling layers Avgpool4, and second upsampling layers, third upsampling layers, fourth upsampling layers, and fifth upsampling layers, respectively connected to convolutional layers Conv3_1_1, convolutional layers Conv3_1_2, convolutional layers Conv3_1_3, and convolutional layers Conv3_1_4;

[0026] The segmentation network also includes a convolution layer Conv3_2 connected to the first upsampling layer, the second upsampling layer, the third upsampling layer, the fourth upsampling layer and the fifth upsampling layer, and a deconvolution layer deConv3_1, a deconvolution layer deConv3_2, a deconvolution layer deConv3_3, a deconvolution layer deConv3_4 and a convolution layer Conv3_3 connected in sequence after the convolution layer Conv3_2;

[0027] Relu modules are provided between the convolution layer Conv3_1_1 and the second upsampling layer, between the convolution layer Conv3_1_2 and the third upsampling layer, between the convolution layer Conv3_1_3 and the fourth upsampling layer, and between the convolution layer Conv3_1_4 and the fifth upsampling layer;

[0028] BN+Relu modules are set between the convolution layer Conv3_2 and the deconvolution layer deConv3_1, between the deconvolution layer deConv3_1 and the deconvolution layer deConv3_2, between the deconvolution layer deConv3_2 and the deconvolution layer deConv3_3, between the deconvolution layer deConv3_3 and the deconvolution layer deConv3_4, and between the deconvolution layer deConv3_4 and the convolution layer Conv3_3.

[0029] Optionally, the classification network includes a first basic unit, a second basic unit, a third basic unit, a fourth basic unit, a fifth basic unit, a first fully connected layer, a second fully connected layer and a softmax classification layer connected in sequence;

[0030] The first basic unit, the second basic unit and the third basic unit all include a convolution layer Conv1_1 and a maximum pooling layer Maxpool1_1 connected in sequence, and a BN+Relu module arranged between the convolution layer Conv4_1 and the maximum pooling layer Maxpool4_1;

[0031] The fourth basic unit and the fifth basic unit both include a convolution layer Conv4_2, a convolution layer Conv4_3 and a maximum pooling layer Maxpool4_2 connected in sequence, and a BN+Relu module arranged between the convolution layer Conv4_2 and the convolution layer Conv4_3, and between the convolution layer Conv4_3 and the maximum pooling layer Maxpool4_2.

[0032] Optionally, the training process of the deep learning neural network model includes the following steps:

[0033] Training the first neural network sub-model to obtain the trained first neural network sub-model;

[0034] Training the second neural network sub-model to obtain a trained second neural network sub-module;

[0035] The trained first neural network submodule and the trained second neural network submodule are connected to obtain a trained deep learning neural network model.

[0036] Optionally, the loss function used to train the first neural network sub-model is:

[0037] L total =L coor +L off +λ size L size +λ seg L seg ;

[0038] Among them, L total represents the loss function used to train the first neural network sub-model, L coor represents the center point coordinate loss function, L off represents the center point offset loss function, L size represents the border size loss function, L seg represents the semantic segmentation loss function, λ size and λ seg All are correction factors.

[0039] Optionally, the loss function used to train the second neural network sub-model is:

[0040]

[0041] Among them, L class represents the loss function used to train the second neural network sub-model, y k is the true label of the gesture, is the prediction result of the gesture, and k represents the kth sample data.

[0042] A static gesture recognition system based on deep learning, the recognition system is based on a trained deep learning neural network model, the deep learning neural network model includes a first neural network sub-model and a second neural network sub-model, the first neural network sub-model includes a backbone network, and a detection network and a segmentation network connected to the backbone network and arranged in parallel, the second neural network sub-model includes a classification network;

[0043] The identification system comprises:

[0044] A feature extraction module, used for inputting an image to be identified with a static gesture into the backbone network for feature extraction to obtain a feature map of the image to be identified;

[0045] a target extraction module, for inputting the feature map into the detection network and the segmentation network respectively, performing target detection on the feature map using the detection network to obtain a prediction frame, performing foreground segmentation on the feature map using the segmentation network to obtain a foreground heat map, and cropping the foreground heat map using the prediction frame to obtain a target heat map; the target is a hand area for characterizing a static gesture, and the target heat map is a heat map of the hand area;

[0046] The gesture recognition module is used to input the target heat map into the classification network to recognize static gestures.

[0047] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0048] The present invention discloses a static gesture recognition method and system based on deep learning, the recognition method comprises the following steps: inputting an image to be recognized with a static gesture into the backbone network for feature extraction to obtain a feature map of the image to be recognized; inputting the feature map into the detection network and the segmentation network respectively, using the detection network to perform target detection on the feature map to obtain a prediction frame, using the segmentation network to perform foreground segmentation on the feature map to obtain a foreground heat map, and using the prediction frame to crop the foreground heat map to obtain a target heat map; inputting the target heat map into the classification network to recognize static gestures. The present invention realizes the recognition of static gestures based on a deep learning neural network model, and the detection network and the segmentation network can be executed in parallel. Compared with a cascade neural network, it has a smaller number of parameters, a shorter execution time and a higher efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technical personnel in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 A schematic diagram of a static gesture recognition method based on deep learning provided by an embodiment of the present invention;

[0051] Figure 2 A structural diagram of a backbone network provided by an embodiment of the present invention;

[0052] Figure 3 A structural diagram of a residual block Block1 provided in an embodiment of the present invention;

[0053] Figure 4 A structural diagram of residual blocks Block2-Block4 provided in an embodiment of the present invention;

[0054] Figure 5 A structural diagram of a detection network provided by an embodiment of the present invention;

[0055] Figure 6 A structural diagram of a segmentation network provided by an embodiment of the present invention;

[0056] Figure 7 A structural diagram of a classification network provided by an embodiment of the present invention;

[0057] Figure 8 A structural diagram of a first basic unit, a second basic unit and a third basic unit provided in an embodiment of the present invention;

[0058] Fig. 9 A structural diagram of a fourth basic unit and a fifth basic unit provided in an embodiment of the present invention;

[0059] Fig.10 A center point coordinate heat map provided by an embodiment of the present invention;

[0060] Fig.11 An example diagram of a prediction box including a true value box provided by an embodiment of the present invention;

[0061] Fig.12 An example diagram of a truth box including a prediction box provided for an embodiment of the present invention;

[0062] Fig.13 An example diagram of an embodiment of the present invention in which a prediction frame and a true value frame are interlaced;

[0063] Fig.14 The present invention provides a flowchart of a smart wheelchair interactive control method. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technical personnel in this field without creative work are within the scope of protection of the present invention.

[0065] The purpose of the present invention is to provide a static gesture recognition method and system based on deep learning to realize gesture recognition, and further to study the control of intelligent wheelchairs based on gesture recognition interaction.

[0066] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] There are still some problems in the current research on gesture recognition. The gesture recognition method based on deep learning has emerged in recent years. It uses a simple classification network for gesture recognition, which has poor adaptability to changes in skin color and background environment. The algorithm based on color threshold segmentation of hands and background and then classification has poor recognition effect when the background contains parts with similar colors to the hands. The gesture recognition parameter scale based on cascade neural networks is large and takes a long time. In summary, the current gesture recognition method based on deep learning has the problems of being easily affected by complex backgrounds, insufficient accuracy, huge network parameters, and low efficiency. Based on the above problems, the deep learning neural network model of the embodiment of the present invention includes a first neural network sub-model and a second neural network sub-model, the first neural network sub-model includes a backbone network, and a detection network and a segmentation network connected to the backbone network and arranged in parallel, and the second neural network sub-model includes a classification network. Through the detection network and the segmentation network arranged in parallel, the hand positioning and the semantic segmentation of the foreground and the background are performed in parallel, and the two processes share a backbone network. Compared with the cascade neural network, it has a smaller parameter amount, a shorter execution time and a higher efficiency, and compared with the simple classification network, it has a higher recognition accuracy.

[0068] Example 1

[0069] like Figure 1 As shown, Embodiment 1 of the present invention provides a static gesture recognition method based on deep learning, wherein the recognition method is based on a trained deep learning neural network model, wherein the deep learning neural network model includes a first neural network sub-model and a second neural network sub-model, wherein the first neural network sub-model includes a backbone network, and a detection network and a segmentation network connected to the backbone network and arranged in parallel, and the second neural network sub-model includes a classification network; Figure 1 As shown, the gesture recognition of the present invention is divided into two stages. The first stage completes the operation of hand positioning and semantic segmentation of the hand and the background, and the network used is the parallel neural network proposed in the present invention. The second stage further processes the output obtained in the first stage to complete the classification and recognition of the gesture.

[0070] The identification method comprises the following steps:

[0071] Step 101: input an image to be recognized with a static gesture into the backbone network for feature extraction to obtain a feature map of the image to be recognized.

[0072] Step 102, the feature map is input into the detection network and the segmentation network respectively, the detection network is used to perform target detection on the feature map to obtain a prediction box, the segmentation network is used to perform foreground segmentation on the feature map to obtain a foreground heat map, and the prediction box is used to crop the foreground heat map to obtain a target heat map; the target is the hand area used to characterize static gestures, and the target heat map is the heat map of the hand area. That is, the obtained feature map enters the detection network and the segmentation network respectively, and the two networks are executed in parallel. The detection network obtains the center point coordinates (x, y) of the hand and the two side lengths (h, w) of the prediction box, and the segmentation network obtains a heat map containing only the hand. The purpose of the segmentation network is to exclude the influence of the background and skin color on the subsequent classification, while the purpose of the detection network is to locate the hand more accurately, and exclude the influence of the arm posture, hand scale and spatial position of the hand on the subsequent classification.

[0073] Step 103: input the target heat map into the classification network to perform static gesture recognition.

[0074] The backbone network in Example 1 of the present invention adopts the structure of the residual block (ResBlock) in the residual neural network (ResNet), and increases the number of batch normalization layers (BN, batch normalization) therein, thereby accelerating the convergence of the network and facilitating the training of the network.

[0075] like Figure 2 As shown, the backbone network includes a convolutional layer Conv1_0, a maximum pooling layer Maxpool1-1, a residual block Block1, a residual block Block2, a residual block Block3 and a residual block Block4 which are connected in sequence; a BN+Relu module is arranged between the convolutional layer Conv1_0 and the maximum pooling layer Maxpool1-1.

[0076] like Figure 3 and 4As shown, the residual block Block1, the residual block Block2, the residual block Block3 and the residual block Block4 all include a convolution layer Conv1_1, a convolution layer Conv1_2, a first feature fusion module, a convolution layer Conv1_3, a convolution layer Conv1_4 and a second feature fusion module; the input end of the convolution layer Conv1_1 is connected to the image input end; the output end of the convolution layer Conv1_1 is connected to the input end of the convolution layer Conv1_2, the output end of the convolution layer Conv1_2 is connected to an input end of the first feature fusion module, the other input end of the first feature fusion module is connected to the input end of the convolution layer Conv1_1, the output end of the first feature fusion module is connected to the input end of the convolution layer Conv1_3, the output end of the convolution layer Conv1_3 is connected to the input end of the convolution layer Conv1_4, and the output end of the convolution layer Conv1_4 is connected to an input end of the second feature fusion module An input end of the first feature fusion module is connected, another input end of the second feature fusion module is connected to the input end of the convolution layer Conv1_3, and the output end of the second feature fusion module is connected to the image output end; a BN+Relu module is arranged between the output end of the convolution layer Conv1_1 and the input end of the convolution layer Conv1_2, and between the output end of the convolution layer Conv1_3 and the input end of the convolution layer Conv1_4; a BN module is arranged between the output end of the convolution layer Conv1_2 and an input end of the first feature fusion module, and between the output end of the convolution layer Conv1_4 and an input end of the second feature fusion module; a Relu module is arranged between the output end of the first feature fusion module and the input end of the convolution layer Conv1_3, and between the output end of the second feature fusion module and the image output end; the residual block Block2, the residual block Block3 and the residual block Block4 all also include a convolution layer Conv1_5. The convolution layer Conv1_5 is arranged between the input end of the convolution layer Conv1_1 and another input end of the first feature fusion module.

[0077] Figure 3-4 In the figure, Conv represents the convolution layer. The convolution kernel size of Conv1_0 is 7*7, the padding is 3, and the stride is 2. After the input passes through Conv1_0, the BN and ReLu operations are performed through the BN+ReLu model. Then the maximum pooling layer Maxpool1_1 is input. The pooling kernel size of this layer is 3*3, the padding is 1, and the stride is 2. The pooled data is input into the subsequent 4 ResBlocks to get the output. The image size of the input backbone network is 512*512, the number of channels is 3, and the feature size of the final output is 16*16, and the number of channels is 512.

[0078] in, Figure 3 This is the structural diagram of the staggered block Block1. Figure 3 In the figure, the convolution kernel size of Conv1_1-Conv1_4 is 3*3, the stride is 1, and the padding is 1.

[0079] Figure 4 This is the structural diagram of the staggered blocks Block2-Block4. Figure 4 In the figure, the convolution kernel size of Conv1_1 is 3*3, the padding is 1, and the stride is 2; the convolution kernel size of Conv1_2-Conv1_4 is 3*3, the padding is 1, and the stride is 1. The convolution kernel size of Conv1_5 is 1*1, the padding is 0, and the stride is 2.

[0080] As shown in 5, the detection network includes a convolution layer Conv2_1, a deconvolution layer deConv2_1, a convolution layer Conv2_2, a deconvolution layer deConv2_2, a convolution layer Conv2_3 and a deconvolution layer deConv2_3 connected in sequence; the output end of the deconvolution layer deConv2_3 is also connected to the input end of the convolution layer Conv2_4_1, the input end of the convolution layer Conv2_4_2 and the input end of the convolution layer Conv2_4_3 respectively; The output end of the convolution layer Conv2_4_1 is connected to the input end of the convolution layer Conv2_5_1, the output end of the convolution layer Conv2_4_2 is connected to the input end of the convolution layer Conv2_5_2, and the output end of the convolution layer Conv2_4_3 is connected to the input end of the convolution layer Conv2_5_3; the output end of the convolution layer Conv2_5_1 is used to output the center point of the target, the output end of the convolution layer Conv2_5_2 is used to output the offset of the center point, and the output end of the convolution layer Conv2_5_3 is used to output the border size of the target; the center point, the center point offset and the border size are all features of the prediction box; between the convolution layer Conv2_1 and the deconvolution layer deConv2_1, between the deconvolution layer deConv2_1 and the convolution layer Conv2_2, between the convolution layer Conv2_2 and the deconvolution layer deConv2_2, and between the deconvolution layer deConv2 BN+Relu modules are arranged between the convolution layer Conv2_2 and the convolution layer Conv2_3, and between the convolution layer Conv2_3 and the deconvolution layer deConv2_3; Relu modules are arranged between the convolution layer Conv2_4_1 and the convolution layer Conv2_5_1, between the convolution layer Conv2_4_2 and the convolution layer Conv2_5_2, and between the convolution layer Conv2_4_3 and the convolution layer Conv2_5_3.

[0081] Figure 5The convolution kernel size of Conv2_1-Conv2_3 is 3*3, the padding is 1, and the step size is 1. deConv stands for deconvolution. The convolution kernel size of deConv is 4*4, the internal padding is 1, and the step size is 2. Each convolution and deconvolution layer is followed by a BN+Relu module. After three convolutions and deconvolutions, the intermediate features are obtained. Figure 1 The size is 128*128 and the number of channels is 64.

[0082] Intermediate Features Figure 1 Enter three processing flows respectively, and the three flows are also completed in parallel. The convolution kernel size of Conv2_4_1-Conv2_4_3 is 3*3, the padding is 1, and the step size is 1. The convolution kernel size of Conv2_5_1-Conv2_5-3 is 1*1, the padding is 1, and the step size is 1. Intermediate features Figure 1 After three processes, the final prediction box is obtained, thus determining the position of the hand.

[0083] like Figure 6As shown, the segmentation network includes a first upsampling layer connected to the output end of the backbone network, an adaptive average pooling layer Avgpool1, an adaptive average pooling layer Avgpool2, an adaptive average pooling layer Avgpool3 and an adaptive average pooling layer Avgpool4; the segmentation network also includes an adaptive average pooling layer Avgpool1, an adaptive average pooling layer Avgpool2, an adaptive average pooling layer Avgpool3 and an adaptive average pooling layer Avgpool4 respectively connected to the adaptive average pooling layer Avgpool1, the adaptive average pooling layer Avgpool2, the adaptive average pooling layer Avgpool3 and the adaptive average pooling layer Avgpool4 The convolution layer Conv3_1_1, the convolution layer Conv3_1_2, the convolution layer Conv3_1_3 and the convolution layer Conv3_1_4, and the second upsampling layer, the third upsampling layer, the fourth upsampling layer and the fifth upsampling layer connected to the convolution layer Conv3_1_1, the convolution layer Conv3_1_2, the convolution layer Conv3_1_3 and the convolution layer Conv3_1_4 respectively; the segmentation network also includes the first upsampling layer, the second upsampling layer, the third upsampling layer, the fourth upsampling layer and the fifth upsampling layer connected to the convolution layer Conv3_1_1, the convolution layer Conv3_1_2, the convolution layer Conv3_1_3 and the convolution layer Conv3_1_4 respectively. The convolution layer Conv3_2 connected to the upsampling layer, and the deconvolution layer deConv3_1, deconvolution layer deConv3_2, deconvolution layer deConv3_3, deconvolution layer deConv3_4 and convolution layer Conv3_3 connected in sequence after the convolution layer Conv3_2; between the convolution layer Conv3_1_1 and the second upsampling layer, between the convolution layer Conv3_1_2 and the third upsampling layer, between the convolution layer Conv3_1_3 and the fourth upsampling layer, and between the convolution layer Conv3_1 Relu modules are set between the convolution layer Conv3_2 and the deconvolution layer deConv3_1, between the deconvolution layer deConv3_1 and the deconvolution layer deConv3_2, between the deconvolution layer deConv3_2 and the deconvolution layer deConv3_3, between the deconvolution layer deConv3_3 and the deconvolution layer deConv3_4, and between the deconvolution layer deConv3_4 and the convolution layer Conv3_3.

[0084] The feature maps obtained by the backbone network pass through four adaptive average pooling layers (Avgpool1-Avgpool4). The number of channels of the feature maps obtained by Avgpool1-Avgpool4 is 512, and the sizes are 1*1, 2*2, 3*3, and 6*6 respectively.

[0085] The feature maps with sizes of 1*1, 2*2, 3*3, and 6*6 are passed through Convv3_1_1-Convv3_1_4 respectively. The number of convolution kernels of the above four convolutional layers is 1*1, the step size is 1, the padding is 0, and the number of input channels and output channels is 512. The purpose is to fuse the features of each scale.

[0086] The results obtained from the above four convolutional layers and the feature maps of the backbone network are upsampled by the bilinear interpolation method, and all four results are upsampled to feature maps of 32*32 with 512 channels.

[0087] The five up-sampled results are connected at the first position of the feature, i.e., feature connection, to obtain a feature map with a size of 32*32 and a channel number of 5*512.

[0088] The concatenated result is passed through Convv3_2 to obtain the intermediate features Figure 2 , the number of convolution kernels in this convolution layer is 1*1, the stride is 1, the padding is 0, the input dimension is 5*512, and the output dimension is 512. Its purpose is to fuse features of different scales and compress the number of channels of the feature map.

[0089] feature Figure 2 After three deconvolution layers (deConvv3_1-deConvv3_4), the number of convolution kernels in the above four deconvolution layers is 4*4, the internal padding is 1, and the step size is 2. After four deconvolution layers, a feature map with a size of 512*512 and 32 channels is obtained.

[0090] The feature map with 32 channels is processed by Convv3_3, where the convolution kernel is 1*1, the step size is 1, and the padding is 0. The final output is a heat map with a size of 512*512 and a channel number of 1, which is the result of semantic segmentation of the hand and the background. The heat map is a binary image with a pixel value of 0 for the hand and a pixel value of 225 for the background. The purpose of semantic segmentation and outputting a binary heat map is to eliminate the interference of the background and the skin color of the hand.

[0091] The purpose of setting the first neural network sub-model in the present invention is to eliminate uncertain factors and reduce the burden of the subsequent classification network. The classification network can use a simple and lightweight convolutional neural network, and only use a small-scale data set to train it, and good results can be achieved.

[0092] like Figure 7 As shown, the classification network includes a first basic unit, a second basic unit, a third basic unit, a fourth basic unit, a fifth basic unit, a first fully connected layer, a second fully connected layer and a softmax classification layer which are connected in sequence.

[0093] Figure 7 The execution steps of the classification network in include:

[0094] Adjust the size of the feature map output by the first neural network sub-model to 224*224;

[0095] Expand the adjusted feature map to 3 channels. Assign the value of the first channel to the other two channels.

[0096] like Figure 8 As shown, the first basic unit, the second basic unit and the third basic unit all include a convolution layer Conv1_1 and a maximum pooling layer Maxpool1_1 connected in sequence, and a BN+Relu module arranged between the convolution layer Conv4_1 and the maximum pooling layer Maxpool4_1.

[0097] like Fig. 9 As shown, the fourth basic unit and the fifth basic unit both include a convolution layer Conv4_2, a convolution layer Conv4_3 and a maximum pooling layer Maxpool4_2 connected in sequence, and a BN+Relu module arranged between the convolution layer Conv4_2 and the convolution layer Conv4_3, and between the convolution layer Conv4_3 and the maximum pooling layer Maxpool4_2.

[0098] Figure 8 and 9 In the 5 basic units, the convolution kernel size of each convolution layer is 3*3, padded with 1, and the step size is 1. The kernel size of each maximum pooling layer is 2*2, padded with 0, and the step size is 2. After five basic units, the feature vector size is 7*7*512. The data passes through two fully connected layers and a softmax classification layer, and the output result is a feature vector of 1*number of categories. The position of the element with the largest value in the feature vector is the corresponding recognition result.

[0099] The training process of the deep learning neural network model includes the following steps: training the first neural network sub-model to obtain the trained first neural network sub-model; training the second neural network sub-model to obtain the trained second neural network sub-module; connecting the trained first neural network sub-module and the trained second neural network sub-module to obtain the trained deep learning neural network model.

[0100] The specific process is:

[0101] (1) Initialize the weights of the first neural network sub-model. The backbone network is initialized using ImageNet, and the detection network and segmentation network are initialized using random initialization methods.

[0102] (2) Use a large-scale data set to train the first neural network sub-model. The data flow is: input → first neural network sub-model → output. The second neural network sub-model is not involved in this process.

[0103] (3) After training is completed, save and lock the weights of the first neural network sub-model.

[0104] (4) The correspondence between the user's customized gestures and operations, and the collection of user gesture data sets.

[0105] (5) Initialize the weights of the second neural network sub-model using random initialization

[0106] (6) Use the collected user gesture data set to train the second neural network sub-model, and the data flow is: input → first neural network sub-model → second neural network sub-model → output. During this process, the weight of the first neural network sub-model is locked and not updated.

[0107] (7) After training is completed, save and lock the second-stage network weights.

[0108] (8) The trained model can be used to recognize user static gestures.

[0109] The format of the dataset is described as follows:

[0110] The hand detection dataset uses annotation boxes when annotating. Convert the format to the format required for training the network as follows:

[0111] Center point coordinate heat map: The original image is a 3-channel RGB image with a width of W and a height of H. The generated heat map is a single-channel image with a width of W / 4 and a height of H / 4. First, the image size is compressed to 1 / 4 of the original size. If a coordinate in the compressed image is the center point of the object, a Gaussian circle with a radius of R is generated at the coordinate, and the pixel values ​​of other areas except the Gaussian circle are set to zero.

[0112] like Fig.10 As shown, let the center coordinates of the true value box in the original image be (x0, y0), then the center of the Gaussian circle is in:

[0113]

[0114] The radius of the Gaussian circle is related to the border of the ground truth and is determined as follows:

[0115] When the overlap degree (IOU, calculated as follows) between the true value box and the predicted box is 0.7, the true value box and the predicted box may have three situations: the predicted box contains the true value box, the true box contains the predicted box, and the true box and the predicted box are intertwined, as shown in the following figure.

[0116]

[0117] Where Sinter is the area of ​​the intersection of the prediction box and the true value box, SGT is the area of ​​the true value box, and Spre is the area of ​​the prediction box.

[0118] like Fig.11 As shown, the realization represents the true value box and the dotted line represents the predicted box. When the predicted box contains the true value box, a circle is drawn with the two corner points of the true value box as the center and tangent to the predicted box, then:

[0119]

[0120]

[0121] Where a=4*Sinter, b=2*Sinter*(h+w), c=(Sinter-1)*(h*w).

[0122] like Fig.12 As shown, the realization represents the true value box and the dotted line represents the predicted box. When the true value box contains the predicted box, a circle is drawn with the two corner points of the true value box as the center and intersects with the predicted box, then:

[0123]

[0124]

[0125] Wherein a=4, b=-2(h+w), c=(1-Sinter)*(h*w).

[0126] like Fig.13 As shown, the realization represents the true value box and the dotted line represents the predicted box. When the true value box and the predicted box intersect, draw a circle with the two corner points of the true value box as the center, one intersecting with the predicted box and the other tangent to the predicted box, then:

[0127]

[0128]

[0129] Wherein a=1, b=-(h+w), c=(1-Sinter)*h*w / (1+Sinter).

[0130] Take the minimum value among r1, r2, and r3 as the radius r of the Gaussian circle.

[0131] The coordinate value at the center of the Gaussian circle is 1, and the pixel values ​​of other points in the circle depend on the distance between the point and the center of the Gaussian circle:

[0132]

[0133] Where x, y represent the coordinates of a point in the heat map, Yx, y represents the pixel value of the point, and σ is the standard deviation, which is related to the size of the target border. The larger the size, the larger the value.

[0134] Center point offset: Because the pixel value coordinates must be integers, the center point coordinates are rounded down, which results in errors. The center point offset can be added to correct the error. The center point offset is recorded as in

[0135] Length and width of the truth box: To facilitate unified data processing, the length and width of the truth box are also expressed as 1 / 4 of the original data, recorded as: in

[0136] Semantic segmentation dataset: The annotation method of the semantic dataset is as follows: the value of the hand pixel is set to 1 and the background part is set to 0.

[0137] Optionally, the loss function used to train the first neural network sub-model is:

[0138] L total =L coor +L off +λ size L size +λ seg L seg ;

[0139] Among them, L total represents the loss function used to train the first neural network sub-model, L coor represents the center point coordinate loss function, L off represents the center point offset loss function, L size represents the border size loss function, Lseg represents the semantic segmentation loss function, and λsize and λseg are correction coefficients.

[0140] The center point coordinate loss function uses the focal loss function, as follows:

[0141]

[0142] Where N is the number of center points in the heat map, is the predicted value of the pixel value at (x, y) in the heat map, and Yx,y is the true value of the pixel value at (x, y) in the heat map. α and β are correction coefficients, and their values ​​are determined according to the actual situation. The purpose of using this loss function is to deal with the imbalance of positive and negative samples, reduce the weight of easy-to-classify samples, and tolerate some smaller deviations.

[0143] The center point offset loss function is calculated using L1 loss, as follows:

[0144]

[0145] in, is the predicted value of the center point offset

[0146] The border size loss function is calculated using L1 loss as follows:

[0147]

[0148] in are the predicted values ​​of width and height respectively

[0149] Semantic segmentation loss function:

[0150] The loss value of semantic segmentation is the average loss value of all pixels in the image. The pixel loss value is calculated using focalloss:

[0151]

[0152]

[0153] Where Lpixel is the loss value of the pixel, p is the true value of a pixel, is the predicted value, and α is the correction coefficient, whose value depends on the actual situation. The purpose of using focal loss is to deal with the imbalance of positive and negative samples.

[0154] Optionally, the loss function used to train the second neural network sub-model is:

[0155]

[0156] Among them, L class represents the loss function used to train the second neural network sub-model, y k is the true label of the gesture, is the prediction result of the gesture, and k represents the kth sample data.

[0157] Example 2

[0158] Embodiment 2 of the present invention provides a static gesture recognition system based on deep learning, wherein the recognition system is based on a trained deep learning neural network model, wherein the deep learning neural network model includes a first neural network sub-model and a second neural network sub-model, wherein the first neural network sub-model includes a backbone network, and a detection network and a segmentation network connected to the backbone network and arranged in parallel, and the second neural network sub-model includes a classification network;

[0159] The identification system comprises:

[0160] A feature extraction module, used for inputting an image to be identified with a static gesture into the backbone network for feature extraction to obtain a feature map of the image to be identified;

[0161] a target extraction module, for inputting the feature map into the detection network and the segmentation network respectively, performing target detection on the feature map using the detection network to obtain a prediction frame, performing foreground segmentation on the feature map using the segmentation network to obtain a foreground heat map, and cropping the foreground heat map using the prediction frame to obtain a target heat map; the target is a hand area for characterizing a static gesture, and the target heat map is a heat map of the hand area;

[0162] The gesture recognition module is used to input the target heat map into the classification network to recognize static gestures.

[0163] Example 3

[0164] Embodiment 3 of the present invention provides an intelligent wheelchair interactive control method based on the recognition method in Embodiment 1, such as Fig.14 As shown, the interactive control method includes the following steps:

[0165] The RGB camera collects relevant images as images to be identified;

[0166] Preprocess the image to be recognized;

[0167] Input the image to be recognized into the deep learning neural network model in Example 1, and use the gesture recognition method in Example 1 to obtain the gesture recognition result;

[0168] Based on the results of gesture recognition, the intelligent wheelchair is controlled. This control includes controlling the drive motor to control the wheelchair to move forward, backward, turn left, turn right, switch different interaction modes, display different interfaces, and other operations.

[0169] Among them, the preprocessing steps exemplarily include two parts: image scaling and normalization. First, the image is scaled to a 512×512 square image. Each pixel of the image is normalized. In the formula, input represents the output image pixel value, input represents the input image pixel value; mean(input) represents the pixel mean of the input image. std represents the standard deviation of the input image pixels. After standardization, the image pixels are adjusted to the interval [-1,1]. The purpose of preprocessing is to facilitate the subsequent operation of the neural network.

[0170]

[0171] The present invention realizes the recognition of static gestures based on a deep learning neural network model, and the detection network and the segmentation network can be executed in parallel. Compared with a cascade neural network, it has a smaller number of parameters, a shorter execution time and a higher efficiency.

[0172] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0173] This article uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A static gesture recognition method based on deep learning, characterized in that: The recognition method is based on a trained deep learning neural network model, the deep learning neural network model includes a first neural network sub-model and a second neural network sub-model, the first neural network sub-model includes a backbone network, and a detection network and a segmentation network connected to the backbone network and arranged in parallel, and the second neural network sub-model includes a classification network; The identification method comprises the following steps: Inputting the image to be recognized with the static gesture into the backbone network for feature extraction to obtain a feature map of the image to be recognized; The feature map is input into the detection network and the segmentation network respectively, the detection network is used to perform target detection on the feature map to obtain a prediction box, the segmentation network is used to perform foreground segmentation on the feature map to obtain a foreground heat map, and the foreground heat map is cropped using the prediction box to obtain a target heat map; the target is a hand area used to characterize a static gesture, and the target heat map is a heat map of the hand area; The target heat map is input into the classification network to recognize static gestures.

2. The static gesture recognition method based on deep learning according to claim 1, characterized in that: The backbone network includes a convolution layer Conv1_0, a maximum pooling layer Maxpool1-1, a residual block Block1, a residual block Block2, a residual block Block3 and a residual block Block4 which are connected in sequence; a BN+Relu module is arranged between the convolution layer Conv1_0 and the maximum pooling layer Maxpool1-1.

3. The static gesture recognition method based on deep learning according to claim 2 is characterized in that: The residual block Block1, the residual block Block2, the residual block Block3 and the residual block Block4 each include a convolution layer Conv1_1, a convolution layer Conv1_2, a first feature fusion module, a convolution layer Conv1_3, a convolution layer Conv1_4 and a second feature fusion module; The input end of the convolution layer Conv1_1 is connected to the image input end; the output end of the convolution layer Conv1_1 is connected to the input end of the convolution layer Conv1_2, the output end of the convolution layer Conv1_2 is connected to an input end of a first feature fusion module, the other input end of the first feature fusion module is connected to the input end of the convolution layer Conv1_1, the output end of the first feature fusion module is connected to the input end of the convolution layer Conv1_3, the output end of the convolution layer Conv1_3 is connected to the input end of the convolution layer Conv1_4, the output end of the convolution layer Conv1_4 is connected to an input end of the second feature fusion module, the other input end of the second feature fusion module is connected to the input end of the convolution layer Conv1_3, and the output end of the second feature fusion module is connected to the image output end; A BN+Relu module is provided between the output end of the convolution layer Conv1_1 and the input end of the convolution layer Conv1_2, and between the output end of the convolution layer Conv1_3 and the input end of the convolution layer Conv1_4; A BN module is provided between the output end of the convolution layer Conv1_2 and one input end of the first feature fusion module, and between the output end of the convolution layer Conv1_4 and one input end of the second feature fusion module; A Relu module is provided between the output end of the first feature fusion module and the input end of the convolution layer Conv1_3, and between the output end of the second feature fusion module and the image output end; The residual block Block2, the residual block Block3 and the residual block Block4 each further include a convolution layer Conv1_5; The convolution layer Conv1_5 is arranged between the input end of the convolution layer Conv1_1 and another input end of the first feature fusion module.

4. The static gesture recognition method based on deep learning according to claim 1, characterized in that: The detection network includes a convolution layer Conv2_1, a deconvolution layer deConv2_1, a convolution layer Conv2_2, a deconvolution layer deConv2_2, a convolution layer Conv2_3 and a deconvolution layer deConv2_3 connected in sequence; The output end of the deconvolution layer deConv2_3 is also connected to the input end of the convolution layer Conv2_4_1, the input end of the convolution layer Conv2_4_2 and the input end of the convolution layer Conv2_4_3 respectively; The output end of the convolution layer Conv2_4_1 is connected to the input end of the convolution layer Conv2_5_1, the output end of the convolution layer Conv2_4_2 is connected to the input end of the convolution layer Conv2_5_2, and the output end of the convolution layer Conv2_4_3 is connected to the input end of the convolution layer Conv2_5_3; the output end of the convolution layer Conv2_5_1 is used to output the center point of the target, the output end of the convolution layer Conv2_5_2 is used to output the offset of the center point, and the output end of the convolution layer Conv2_5_3 is used to output the border size of the target; the center point, the center point offset and the border size are all features of the prediction box; A BN+Relu module is provided between the convolution layer Conv2_1 and the deconvolution layer deConv2_1, between the deconvolution layer deConv2_1 and the convolution layer Conv2_2, between the convolution layer Conv2_2 and the deconvolution layer deConv2_2, between the deconvolution layer deConv2_2 and the convolution layer Conv2_3, and between the convolution layer Conv2_3 and the deconvolution layer deConv2_3; A Relu module is provided between the convolution layer Conv2_4_1 and the convolution layer Conv2_5_1, between the convolution layer Conv2_4_2 and the convolution layer Conv2_5_2, and between the convolution layer Conv2_4_3 and the convolution layer Conv2_5_3.

5. The static gesture recognition method based on deep learning according to claim 1, characterized in that: The segmentation network includes a first upsampling layer connected to the output end of the backbone network, an adaptive average pooling layer Avgpool1, an adaptive average pooling layer Avgpool2, an adaptive average pooling layer Avgpool3 and an adaptive average pooling layer Avgpool4; The segmentation network also includes convolutional layers Conv3_1_1, Convolutional layers Conv3_1_2, Convolutional layers Conv3_1_3, and Convolutional layers Conv3_1_4, respectively connected to adaptive average pooling layers Avgpool1, adaptive average pooling layers Avgpool2, adaptive average pooling layers Avgpool3, and adaptive average pooling layers Avgpool4, and second upsampling layers, third upsampling layers, fourth upsampling layers, and fifth upsampling layers, respectively connected to convolutional layers Conv3_1_1, convolutional layers Conv3_1_2, convolutional layers Conv3_1_3, and convolutional layers Conv3_1_4; The segmentation network also includes a convolution layer Conv3_2 connected to the first upsampling layer, the second upsampling layer, the third upsampling layer, the fourth upsampling layer and the fifth upsampling layer, and a deconvolution layer deConv3_1, a deconvolution layer deConv3_2, a deconvolution layer deConv3_3, a deconvolution layer deConv3_4 and a convolution layer Conv3_3 connected in sequence after the convolution layer Conv3_2; Relu modules are provided between the convolution layer Conv3_1_1 and the second upsampling layer, between the convolution layer Conv3_1_2 and the third upsampling layer, between the convolution layer Conv3_1_3 and the fourth upsampling layer, and between the convolution layer Conv3_1_4 and the fifth upsampling layer; BN+Relu modules are set between the convolution layer Conv3_2 and the deconvolution layer deConv3_1, between the deconvolution layer deConv3_1 and the deconvolution layer deConv3_2, between the deconvolution layer deConv3_2 and the deconvolution layer deConv3_3, between the deconvolution layer deConv3_3 and the deconvolution layer deConv3_4, and between the deconvolution layer deConv3_4 and the convolution layer Conv3_3.

6. The static gesture recognition method based on deep learning according to claim 1, characterized in that: The classification network includes a first basic unit, a second basic unit, a third basic unit, a fourth basic unit, a fifth basic unit, a first fully connected layer, a second fully connected layer and a softmax classification layer connected in sequence; The first basic unit, the second basic unit and the third basic unit all include a convolution layer Conv1_1 and a maximum pooling layer Maxpool1_1 connected in sequence, and a BN+Relu module arranged between the convolution layer Conv4_1 and the maximum pooling layer Maxpool4_1; The fourth basic unit and the fifth basic unit both include a convolution layer Conv4_2, a convolution layer Conv4_3 and a maximum pooling layer Maxpool4_2 connected in sequence, and a BN+Relu module arranged between the convolution layer Conv4_2 and the convolution layer Conv4_3, and between the convolution layer Conv4_3 and the maximum pooling layer Maxpool4_2.

7. The static gesture recognition method based on deep learning according to claim 1, characterized in that: The training process of the deep learning neural network model includes the following steps: Training the first neural network sub-model to obtain the trained first neural network sub-model; Training the second neural network sub-model to obtain a trained second neural network sub-module; The trained first neural network submodule and the trained second neural network submodule are connected to obtain a trained deep learning neural network model.

8. The static gesture recognition method based on deep learning according to claim 7, characterized in that: The loss function used to train the first neural network sub-model is: L total =L coor +L off +λ size L size +λ seg L seg ; Among them, L total represents the loss function used to train the first neural network sub-model, L coor represents the center point coordinate loss function, L off represents the center point offset loss function, L size represents the border size loss function, L seg represents the semantic segmentation loss function, λ size and λ seg All are correction factors.

9. The static gesture recognition method based on deep learning according to claim 7, characterized in that: The loss function used to train the second neural network sub-model is: Among them, L class represents the loss function used to train the second neural network sub-model, y k is the true label of the gesture, is the prediction result of the gesture, and k represents the kth sample data.

10. A static gesture recognition system based on deep learning, characterized in that: The recognition system is based on a trained deep learning neural network model, the deep learning neural network model includes a first neural network sub-model and a second neural network sub-model, the first neural network sub-model includes a backbone network, and a detection network and a segmentation network connected to the backbone network and arranged in parallel, and the second neural network sub-model includes a classification network; The identification system comprises: A feature extraction module, used for inputting an image to be identified with a static gesture into the backbone network for feature extraction to obtain a feature map of the image to be identified; a target extraction module, for inputting the feature map into the detection network and the segmentation network respectively, performing target detection on the feature map using the detection network to obtain a prediction frame, performing foreground segmentation on the feature map using the segmentation network to obtain a foreground heat map, and cropping the foreground heat map using the prediction frame to obtain a target heat map; the target is a hand area for characterizing a static gesture, and the target heat map is a heat map of the hand area; The gesture recognition module is used to input the target heat map into the classification network to recognize static gestures.

Images