A static gesture recognition method based on attention mechanism

Abstract

The present invention discloses a static gesture recognition method based on an attention mechanism. Gesture images are collected and size-normalized; the original gesture images are input into the first layer of a reference network; the first-order feature vectors are respectively input into a gesture detail attention module and the second layer of a reference network, the second-order feature vectors and the output of the gesture detail attention module are fused, and input into the first channel attention module; the output of the first channel attention module is input into the third layer of the reference network; the first-order feature vectors and the output of the channel attention module are input into a gesture main attention module; the output of the gesture main attention module and the third-order feature vectors are fused and input into the second channel attention module; the second channel attention module is input into the remaining reference network and a softmax classifier to obtain the classification result of the gesture image. The present invention is designed for the characteristics of static gesture recognition, and can solve the problem of low model accuracy under the condition of a small number of training sets.

Description

Technical Field

[0001] The present invention relates to a method for extracting gesture images, and in particular to a static gesture recognition method based on an attention mechanism. Background Art

[0002] With the maturity of neural network technology, gesture recognition based on computer vision is becoming a trend. In practical applications, how to recognize gestures and the required accuracy also increase the difficulty of real-time gesture recognition. Although gesture recognition technology has made great progress, it still faces many challenges in real environments. For example, many factors such as lighting and distance will affect the performance of gesture recognition. In addition, there are relatively few publicly available static sign language data sets, which may require the training of models with a small number of training sets. At present, gesture recognition methods based on deep learning have gradually become mainstream. Generally, convolutional neural networks are used to automatically extract gesture features through convolutional layers and fuse them through convolutional layers. In this special case, the benefits of simply increasing the depth or width of the network are not particularly obvious, and overfitting may even occur, resulting in poor accuracy performance. Summary of the invention

[0003] In view of the problem of low accuracy of gesture feature extraction during network training under the condition of few training sets, the present invention proposes a simple and effective feature optimization module, which can specifically solve the problem of low model accuracy under the condition of few training sets. The present invention uses a universal static image processing mode and can apply the feature optimization module on any image processing benchmark network, which strengthens the gesture features, protects the integrity of the information, simplifies the goal and difficulty of network learning, and improves the recognition accuracy.

[0004] The technical solution of the present invention comprises the following steps:

[0005] 1) Collect images of various gestures to obtain original gesture images, and perform size normalization on the original gesture images;

[0006] 2) Input the size-normalized image into the first layer of the benchmark network to obtain the first-order feature vector;

[0007] 3) Input the first-order feature vector into the gesture detail attention module to obtain the gesture detail feature vector; input the first-order feature vector into the second layer of the reference network to obtain the second-order feature vector; fuse the gesture detail feature vector and the second-order feature vector to obtain the second-order feature vector after gesture detail calibration;

[0008] 4) Inputting the second-order feature vector after gesture detail calibration into the first channel attention module to obtain the second-order feature vector after gesture detail calibration and channel calibration;

[0009] 5) Inputting the first-order feature vector and the output of the first channel attention module into the gesture main attention module to obtain the gesture main feature vector;

[0010] 6) Input the output of the channel attention module into the third layer of the reference network to obtain the third-order feature vector; fuse the third-order feature vector with the gesture main feature vector and input it into the second channel attention module;

[0011] 7) The output of the second channel attention module is sequentially input into the remaining baseline network, the average pooling layer and the softmax classifier to obtain the classification result of the gesture image.

[0012] The reference network described in the present invention is any convolutional neural network.

[0013] The gesture detail attention module includes three convolutional layers connected in sequence, and an upsampling layer is connected between the first convolutional layer and the second convolutional layer; the convolution kernel size of the first convolutional layer is 1×1, and the convolution kernel sizes of the second and third convolutional layers are 3×3.

[0014] The channel attention module includes a global average pooling layer, a first fully connected layer and a second fully connected layer connected in sequence. The output of the second fully connected layer is multiplied by the input of the channel attention module in the channel dimension to obtain a feature vector as the output of the channel attention module.

[0015] The gesture body attention module is mainly composed of a first branch of the gesture body, a second branch of the gesture body and a gating module. The input of the first branch of the gesture body is a first-order feature vector, the input of the second branch of the gesture body is a second-order feature vector after gesture details and channel calibration, and the input of the gating module is the output of the first branch of the gesture body and the second branch of the gesture body.

[0016] In terms of topology, the gesture subject attention module consists of the gesture subject first branch and the gesture subject second branch in parallel, and these two branches are connected to the gating module in sequence:

[0017] The first branch of the gesture body is mainly composed of a convolution kernel with a convolution kernel size of 1×1, a convolution kernel size of 4×4 average pooling layer, a convolution kernel size of 3×3, a convolution kernel size of 4×4 average pooling layer, a convolution kernel size of 3×3, an upsampling layer and a convolution kernel size of 3×3 convolution layer connected in sequence;

[0018] The second branch of the gesture body is mainly composed of a convolution kernel with a convolution kernel size of 1×1, an average pooling layer with a convolution kernel size of 2×2, a convolution layer with a convolution kernel size of 3×3, a convolution kernel size of 4×4, a convolution layer with a convolution kernel size of 3×3, an upsampling layer and a convolution kernel size of 3×3, which are connected in sequence.

[0019] The gating module is mainly composed of two groups of average pooling layers with a convolution kernel size of 2×2 and two fully connected layers in parallel. The two groups of average pooling layers pool the outputs of the first branch of the gesture body and the second branch of the gesture body respectively to obtain two groups of feature vectors with a size of 16×16. The two groups of feature vectors are added in the height and width dimensions and flattened into a one-dimensional feature vector with a length of 256. The feature vectors are respectively input into two fully connected layers to obtain two one-dimensional feature vectors with a length of 16. The feature vectors are then input into two other fully connected layers to obtain two one-dimensional feature vectors with a length of 1. Finally, the two feature vectors are multiplied with the outputs of the first branch of the gesture body and the second branch of the gesture body, and then added in the height and width dimensions. The feature vector with a size of 32×32 is the feature vector of the gesture body.

[0020] Beneficial effects of the present invention:

[0021] The method of the present invention is designed based on the characteristics of static gesture recognition, and introduces a lightweight attention mechanism to enhance the expressiveness of necessary features in static sign language and suppress unnecessary features, achieving good performance improvement while maintaining low overhead. It can specifically alleviate the problem of low model training accuracy under the condition of a small number of training sets.

[0022] The gesture detail attention module proposed in the present invention innovatively adopts an upsampling method to amplify the gesture details, and then uses a small convolution kernel that is sensitive to small features to extract features. This method can effectively extract the detailed features of the gesture.

[0023] The gesture subject attention module proposed in the present invention adopts a dual-branch structure, which can extract gesture subject features at different scales, improving the robustness of the module. In addition, the original gating module can be autonomously learned by the network during the training process and allocate the weights between the dual branches, thus enhancing the stability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1.1 Part of the gesture images of the sign language data set of the present invention;

[0025] Figure 1.2 The gesture image obtained by randomly cropping a small range used in the present invention;

[0026] Figure 2 It is a flow chart of the technical solution of the present invention;

[0027] Figure 3 is a flowchart of the gesture detail attention module in the present invention;

[0028] Figure 4 Flow chart of the channel attention module in the present invention;

[0029] Figure 5 is a general flow chart of the gesture subject attention module in the present invention;

[0030] Figure 6 A flowchart of the first branch of the gesture body and the second branch of the gesture body in the present invention;

[0031] Figure 7 It is a gating module flow chart of the gesture subject attention module in the present invention;

[0032] Figure 8 The feature vector extracted by the gesture detail attention module in the present invention;

[0033] Fig. 9 It is the feature vector extracted by the gesture subject attention module in the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0035] The specific embodiments of the present invention are as follows:

[0036] The dataset of this invention adopts the ASL (American sign language) open source sign language dataset, some of which are as follows Figure 1.1 As shown in the figure, it contains gesture images under different angles, different lighting, different sizes and different background environments, including 28 gesture categories and non-gesture categories, a total of 29 classification categories.

[0037] like Figure 2 As shown, the implementation process includes the following steps

[0038] 1) The original image is normalized to obtain a gesture image. The size of the original gesture image is normalized to a three-channel RGB image of 3×256×256. 3×256×256 is used as the input size of the neural network. The three-channel RGB image is then standardized to map the three-channel RGB image from integers between 0 and 255 to floating-point numbers between 0 and 1.

[0039] During the training phase, a small range of random cropping is used, that is, the given image is randomly cropped into different sizes and aspect ratios, and then the cropped image is scaled to 256×256. The size of the random crop is selected to be 0.9-1 of the original image, and the aspect ratio is 0.9-1.1 of the original image. The cropped result is as follows Figure 1.2 shown.

[0040] Random cropping can deal with the semi-occlusion of some gesture images in the dataset and reduce the model's dependence on training samples.

[0041] 2) Input the original gesture image into the first layer of the baseline network to obtain the first-order feature vector.

[0042] 3) Input the first-order feature vector into the gesture detail attention module to obtain the gesture detail feature vector.

[0043] like Figure 3 As shown in the figure, the gesture detail attention module is mainly composed of a convolution layer with a convolution kernel size of 1×1, an upsampling layer and two convolution layers with a convolution kernel size of 3×3 connected in sequence. The input feature vector size is 128×128. It is first processed by a convolution layer with a convolution kernel size of 1×1 to reduce the number of channels to 1. This step can reduce the parameters required for subsequent convolution operations. Then the upsampling layer is used to double the height and width of the feature vector of the merged channel to 256×256. The detail features in the upsampled feature vector are also amplified accordingly. Then two consecutive small convolutions with a size of 3×3 and a relatively small receptive field are used for convolution, which can be sensitive to small features and better extract detail features. The output feature vector of size 64×64 is the gesture detail feature vector.

[0044] 4) Input the first-order feature vector into the second layer of the baseline network to obtain the second-order feature vector.

[0045] 5) Multiply the second-order feature vector and the gesture detail feature vector in the height and width dimensions to obtain the second-order feature vector after gesture detail calibration.

[0046] 6) The second-order feature vector after gesture detail calibration is input into the channel attention module to obtain the second-order feature vector after gesture detail and channel calibration.

[0047] Specific as Figure 4 As shown in the figure, the channel attention module is mainly composed of a global average pooling layer and two fully connected layers connected in sequence. The graphics in the lower left corner and the lower right corner of the figure represent the input and output feature vectors respectively, and the squares with different grayscales in the figure represent one-dimensional feature vectors.

[0048] The second-order feature vector after gesture detail calibration is used as input to aggregate the spatial information of the feature vector through the global average pooling layer, and then the feature vector is converted into a one-dimensional vector of channel length. The aggregation of the global average pooling layer follows the following formula:

[0049]

[0050] Where: z c represents the one-dimensional feature vector obtained after feature compression; F sq (*) represents the feature compression function; H represents the height of the feature vector; W represents the width of the feature vector; u c (i, j) represents the point at position i and j on the cth channel.

[0051] In order to allow the network to filter out effective and important features and increase the nonlinearity of the vector, the one-dimensional feature vector is put into a fully connected layer, and the output length is compressed to 1 / 8 of the original length. After passing through another fully connected layer, the length is restored to the original number of channels to obtain a one-dimensional vector containing the feature weights of each channel. Figure 4 The two arrowed segments are fused at the confluence of . The specific form of fusion is that the one-dimensional feature vector and the second-order feature vector after gesture detail calibration are multiplied in the channel dimension, and the obtained feature vector is the second-order feature vector after gesture detail and channel calibration.

[0052] 7) Inputting the second-order feature vector after gesture details and channel calibration into the third layer of the reference network to obtain the third-order feature vector;

[0053] 8) Inputting the first-order feature vector and the second-order feature vector after gesture details and channel calibration into the gesture subject attention module to obtain the gesture subject feature vector;

[0054] like Figure 5 As shown, the gesture body attention module is mainly composed of the first branch of the gesture body, the second branch of the gesture body and the gating module. The input of the first branch of the gesture body is the first-order feature vector, the input of the second branch of the gesture body is the second-order feature vector after gesture details and channel calibration, and the input of the gating module is the output of the first branch of the gesture body and the second branch of the gesture body.

[0055] like Figure 6As shown in the figure, the topological structures of the first branch of the gesture body and the second branch of the gesture body are similar, both of which are composed of a convolution kernel with a convolution kernel size of 1×1, an average pooling layer, a convolution kernel size of 3×3, an average pooling layer, a convolution kernel size of 3×3, an upsampling layer and a convolution kernel size of 3×3 connected in sequence. The only difference in the structure of the two branches is that the convolution kernel size of the first average pooling layer is different. The convolution kernel size of the first branch of the gesture body is 4×4, and the convolution kernel size of the second branch of the gesture body is 2×2. After the input feature vector is processed by the 1×1 convolution layer on the channel dimension, the parameter cost in the subsequent processing can be controlled. After being processed by the average pooling layer to abandon the detailed features and retain the main features of the gesture, this step can convert the inputs of different sizes of branches 1 and 2 into the same 32×32, and then use the convolution layer with a convolution kernel size of 3×3 to perform convolution to process the spatial information of the reduced image. Then repeat the convolution + sampling again to reduce the size of the feature vector to 8×8; use upsampling to upsample the 8×8 image that has been convolved to 32×32. The upsampled feature vector is also convolved through the convolution layer. Compared with the deconvolution with the same upsampling effect, the advantage of the present invention is that the deconvolution may produce an unstable checkerboard effect, which has a great impact on the accuracy of feature extraction. And a relatively shallow network layer is conducive to maintaining the flow of gradients. The specific implementation process is as follows: Figure 8 As shown in FIG. 1 , both the first branch of the gesture body and the second branch of the gesture body obtain feature vectors with an output size of 32×32.

[0056] The gating module is mainly composed of two groups of sub-branches in parallel, each of which consists of an average pooling layer with a convolution kernel size of 2×2 and two fully connected layers. Figure 7 As shown, the two groups of sub-branches receive the outputs of the first branch of the gesture body and the second branch of the gesture body respectively, and Figure 7 The details of the vector changes during the processing are also shown. The average pooling layers of the two groups pool the outputs of the first branch of the gesture body and the second branch of the gesture body respectively, and obtain two groups of feature vectors of size 16×16, which are expressed as follows:

[0057] x1=F avg (X1)

[0058] x2=F avg (X2)

[0059] In the formula, X1 is the feature vector output by the first branch of the gesture body, and X2 is the feature vector output by the second branch of the gesture body. Both x1 and x2 are feature vectors after pooling. Compared with not using pooling, this can reduce the number of parameters consumed by subsequent operations.

[0060] The outputs of these two branches carry information of different scales, and the idea of ​​gates is used to integrate the information of these two branches. Therefore, by adding x1 and x2 in the height and width dimensions and flattening the added feature map, a feature y is obtained to achieve the guiding feature of channel adaptation, which can be expressed as:

[0061] y=Flatten(x1+x2)

[0062] Input the feature vector y into two fully connected layers respectively, and obtain two one-dimensional feature vectors with a length of 16 respectively. Then input this feature vector into the other two fully connected layers, and obtain two one-dimensional feature vectors with a length of 1 respectively, which can be expressed as:

[0063] a=Fc1(Fc39y)

[0064] b=Fc2(Fc4(y))

[0065] In the formula, a and b are the guiding features of the first branch and the second branch of the gesture body respectively; Fc1(*), Fc2(*), Fc3(*), and Fc4(*) represent four different fully connected layers that perform linear transformation on the feature vector y. The reason for using four fully connected layers instead of two is to increase the nonlinearity of the features and avoid the degradation of performance due to too fast dimensional compression.

[0066] Finally, after multiplying the two feature vectors a and b with the outputs of the first branch and the second branch of the gesture body respectively, they are added in the height and width dimensions. The resulting feature vector of size 32×32 is the feature vector of the gesture body, which is expressed as follows:

[0067] U=a·X1+b·X2

[0068] U in the formula is the gesture subject feature vector, which is the output of the gesture subject attention module.

[0069] 9) Fusing the gesture subject feature vector and the third-order feature vector to obtain the third-order feature vector calibrated by the gesture subject;

[0070] 10) The third-order feature vector after the gesture body calibration is input into the second channel attention module, and then the output of the second channel attention module is input into the remaining benchmark network and connected to a softmax classifier to obtain the classification result of the gesture image.

[0071] The extraction effect of the gesture detail attention module is as follows Figure 8 As shown, the important detail feature of thumb placement can be extracted, and the edge features of the gesture can also be extracted to a certain extent.

[0072] The extraction effect of the gesture subject attention module is as follows Fig. 9 As shown, the shielding of complex backgrounds is quite effective, and the position of the gesture subject is accurately calibrated, which is of great help to gesture recognition, achieving the effect of suppressing invalid features and strengthening effective features.

[0073] As shown in Table 1, the error rates of the feature optimization module proposed in the present invention are applied to different benchmark networks. It can be found that the classification error rate of the network using the feature optimization module of the present invention is significantly lower than that of all benchmark networks, indicating that the method of the present invention can generalize various models under the condition of few training sets well, can more richly describe the details and main features of gestures, and greatly improve the accuracy of image classification.

[0074] Table 1 Test results of the present invention in the actual reasoning process

[0075]

Claims

1. A static gesture recognition method based on attention mechanism, characterized in that: The steps include: 1) Collect images of various gestures to obtain original gesture images, and perform size normalization on the original gesture images; 2) Input the size-normalized image into the first layer of the benchmark network to obtain the first-order feature vector; 3) Input the first-order feature vector into the gesture detail attention module to obtain the gesture detail feature vector; input the first-order feature vector into the second layer of the reference network to obtain the second-order feature vector; fuse the gesture detail feature vector and the second-order feature vector to obtain the second-order feature vector after gesture detail calibration; 4) Inputting the second-order feature vector after gesture detail calibration into the first channel attention module to obtain the second-order feature vector after gesture detail calibration and channel calibration; 5) Inputting the first-order feature vector and the output of the first channel attention module into the gesture main attention module to obtain the gesture main feature vector; 6) Input the output of the channel attention module into the third layer of the reference network to obtain the third-order feature vector; fuse the third-order feature vector with the gesture main feature vector and input it into the second channel attention module; 7) The output of the second channel attention module is sequentially input into the remaining baseline network, the average pooling layer and the softmax classifier to obtain the classification result of the gesture image.

2. The static gesture recognition method based on attention mechanism according to claim 1, characterized in that: The gesture detail attention module includes three convolutional layers connected in sequence; an upsampling layer is connected between the first convolutional layer and the second convolutional layer; the convolution kernel size of the first convolutional layer is 1×1, and the convolution kernel sizes of the second and third convolutional layers are 3×3.

3. The static gesture recognition method based on attention mechanism according to claim 1, characterized in that: The channel attention module includes a global average pooling layer, a first fully connected layer and a second fully connected layer connected in sequence. The output of the second fully connected layer is multiplied by the input of the channel attention module in the channel dimension to obtain a feature vector as the output of the channel attention module.

4. The static gesture recognition method based on attention mechanism according to claim 1, characterized in that: The gesture subject attention module includes a gesture subject first branch, a gesture subject second branch and a gating module; the gesture subject first branch and the gesture subject second branch are arranged in parallel, and the outputs of the two branches are both input into the gating module; The first branch of the gesture body includes four convolutional layers connected in sequence. The first average pooling layer is connected between the first and second convolutional layers, the second average pooling layer is connected between the second and third convolutional layers, and an upsampling layer is connected between the third and fourth convolutional layers. The convolution kernel size of the first average pooling layer of the second branch of the gesture body is different from that of the first branch of the gesture body, and the rest of the structure is the same. The gating module includes two groups of parallel sub-branches, each of which includes an average pooling layer and two fully connected layers connected in sequence.

5. The static gesture recognition method based on attention mechanism according to claim 4, characterized in that: The outputs of the first branch of the gesture body and the second branch of the gesture body are respectively input into the average pooling layers of the two groups of sub-branches, and the output feature maps of the two average pooling layers are flattened after adding the outputs in the height and width dimensions. The flattened feature maps are respectively input into the two fully connected layers of the two groups of sub-branches to obtain two one-dimensional feature vectors respectively; The two one-dimensional feature vectors output by the two groups of sub-branches are multiplied by the outputs of the second branch of the gesture body and the first branch of the gesture body to obtain two feature vectors, and then the two feature vectors are added in the height and width dimensions to obtain the feature vector of the gesture body.

6. The static gesture recognition method based on attention mechanism according to claim 4, characterized in that: The convolution kernel size of the first convolution layer is 1×1, and the convolution kernel size of the second, third and fourth convolution layers is 3×3; the convolution kernel size of the first average pooling layer in the second branch of the gesture body is 4×4, and the convolution kernel size of the first average pooling layer in the second branch of the gesture body is 2×2.

Images