A gesture image recognition method of an improved inception-v3 network

Abstract

The application discloses a gesture image recognition method of an improved Inception-v3 network. The method comprises the following steps: obtaining an original gesture image dataset; improving the Inception-v3 network as a whole to build an improved network; inputting the original gesture image dataset into the improved network and an Upsampling network for processing, inputting an original feature vector group into a convolution normalization network for processing, performing feature fusion to obtain a fused feature vector group, and then inputting the fused feature vector group into a classification network for processing to output classification results of various gesture images. According to the parallel splicing mode, the application constructs multiple improved Inception-v3 network structures, can combine the residual and batch normalization modes, simplifies the learning target and difficulty, and avoids overfitting. Furthermore, by using the CBAM attention mechanism, the application can effectively extract image features and improve the recognition accuracy.

Description

Technical Field

[0001] This invention relates to an image recognition method, specifically to a gesture image recognition method using an improved Inception-v3 network. Background Technology

[0002] With the continuous maturation of machine learning technology, gesture recognition based on computer vision is currently extremely popular. Gesture recognition technology has made significant progress, and deep learning-based gesture recognition methods are gradually becoming mainstream. Currently, most mainstream neural network models are constantly increasing their depth to improve accuracy. This leads to problems such as excessively large parameter counts, network model degradation, and overfitting. Summary of the Invention

[0003] To address the problems in the background technology, specifically the issue of excessive model parameters and potential overfitting during Inception-v3 network training leading to low gesture recognition accuracy, this invention provides an effective gesture image recognition method that meets the demand for high-precision gesture recognition with a limited number of model parameters. This invention utilizes the structure of the Inception-v3 network, redesigning the Inception module to reduce the number and difficulty of learning parameters. It incorporates residual connections to protect information integrity and prevent network degradation. An attention mechanism is introduced to focus the model on useful information while downplaying useless information, thus preventing overfitting to some extent. Furthermore, upsampling and feature fusion with low-level features are performed within the model; the fused features are more discriminative than the original input features, further improving the model's accuracy.

[0004] The technical solution adopted in this invention is:

[0005] The method of the present invention includes the following steps:

[0006] 1) Collect various gesture images, establish an original image dataset, and perform size normalization and grayscale processing on the original image dataset to obtain the original gesture image dataset;

[0007] 2) Improve the Inception-v3 network as a whole and build the improved stage_1 network, Inception1_module_1 network, Inception1_module_2 network, Inception_reshape network, Inception2_module_1 network, Inception2_module_2 network and Inception2_module_3 network; each improved network is directly obtained by improving the Inception-v3 network as a whole.

[0008] 3) Input the original gesture image dataset into the stage_1 network for processing, and output the original feature vector set;

[0009] 4) Input the original feature vector set into the Inception1_module_1 network for processing, and output the first feature vector set; input the original feature vector set into the convolutional normalization network for processing, and output the second feature vector set;

[0010] 5) Input the first feature vector set into the Inception1_module_2 network for processing, and output the third feature vector set;

[0011] 6) Input the third feature vector group into the Inception_reshape network for processing, and output the fourth feature vector group;

[0012] 7) Input the fourth feature vector group into the Inception2_module_1 network for processing, and output the fifth feature vector group;

[0013] 8) Input the fifth feature vector group into the Inception2_module_2 network for processing, and output the sixth feature vector group; input the sixth feature vector group into the Inception2_module_3 network, which is exactly the same as the Inception2_module_2 network, for processing, and output the seventh feature vector group;

[0014] 9) Input the seventh feature vector group into the Upsampling network for processing, and output the eighth feature vector group;

[0015] 10) Perform feature fusion processing on the second feature vector group and the eighth feature vector group in step 4) to obtain a fused feature vector group; input the fused feature vector group into the classification network for processing, and output the classification results of various gesture images corresponding to step 1).

[0016] In step 3), the original gesture image dataset is input into the stage_1 network for processing. For each original gesture image in the original gesture image dataset, the following operations are performed:

[0017] The structure of the stage_1 network specifically includes a first third-order ordinary convolutional module, a first CBAM attention mechanism module, a second third-order ordinary convolutional module, a third third-order ordinary convolutional module, a first max pooling layer, a first first-order ordinary convolutional module, a fourth third-order ordinary convolutional module, and a second max pooling layer.

[0018] For each original gesture image in the original gesture image dataset, the following operations are performed:

[0019] The input original gesture image is fed into the first third-order ordinary convolutional module to obtain low-level feature vectors. The output of the low-level feature vectors is then fed into the first CBAM attention mechanism module, where the output is added to the low-level feature vectors. The result of the addition is then processed sequentially through the second third-order ordinary convolutional module, the third third-order ordinary convolutional module, the first max pooling layer, the first first-order ordinary convolutional module, the fourth third-order ordinary convolutional module, and the second max pooling layer before being output to obtain the original feature vectors.

[0020] The original feature vectors are combined to form the original feature vector group.

[0021] In step 4):

[0022] a) Input the original feature vector set into the Inception1_module_1 network for processing:

[0023] The Inception1_module_1 network structure specifically includes a second first-order ordinary convolutional module, a third first-order ordinary convolutional module, a fourth first-order ordinary convolutional module, a first average pooling layer, a third max pooling layer, a first third-order separable convolutional module, a first fifth-order separable convolutional module, a fifth first-order ordinary convolutional module, a sixth first-order ordinary convolutional module, a first concatenation layer, a first batch normalization layer, and a first ReLU activation function; the batch normalization layer is a normalization layer.

[0024] For each original feature vector in the original feature vector group, perform the following operation:

[0025] The original feature vector is simultaneously fed into the second, third, and fourth first-order ordinary convolutional modules, the first average pooling layer, and the third max pooling layer for processing. The output of the third first-order ordinary convolutional module is then fed into the first third-order separable convolutional module for processing. The output of the fourth first-order ordinary convolutional module is then fed into the first fifth-order separable convolutional module for processing. The output of the first average pooling layer is then fed into the fifth first-order ordinary convolutional module for processing. The output of the third max pooling layer is then fed into the sixth first-order ordinary convolutional module for processing. The outputs of the second, third, and sixth first-order ordinary convolutional modules are then fed into the first concatenation layer for concatenation. The concatenated result is then processed sequentially through the first batch normalization (BN) layer and the first ReLU activation function to obtain the first feature vector.

[0026] Each first feature vector forms the first feature vector group; after the original feature vector group is processed by the Inception1_module_1 network, the number of channels is doubled to obtain the first feature vector group.

[0027] b) Input the original feature vector set into the convolutional normalization network for processing:

[0028] The structure of the convolutional normalization network specifically includes a second-order separable convolutional module, a first spatial convolutional layer, a first convolutional layer with a kernel size of 1×1, and a second BN layer;

[0029] For each original feature vector in the original feature vector group, perform the following operation:

[0030] The input original feature vector is sequentially fed into a second-order separable convolutional module, a first spatial convolutional layer, a first convolutional layer with a kernel size of 1×1, and a second BN layer for processing and output to obtain the second feature vector;

[0031] Each second eigenvector forms a second eigenvector group.

[0032] In step 5), the first feature vector group is input into the Inception1_module_2 network for processing.

[0033] The Inception1_module_2 network structure specifically includes a seventh-order ordinary convolutional module, an eighth-order ordinary convolutional module, a ninth-order ordinary convolutional module, a second average pooling layer, a fourth max pooling layer, a second third-order separable convolutional module, a second fifth-order separable convolutional module, an eleventh-order ordinary convolutional module, an eleventh-order ordinary convolutional module, a second splicing layer, a third BN layer, and a second ReLU activation function.

[0034] For each first feature vector in the first feature vector group, the following operation is performed:

[0035] The first feature vector is simultaneously fed into the 7th-11th-order ordinary convolutional module, the 8th-11th-order ordinary convolutional module, the 9th-11th-order ordinary convolutional module, the second average pooling layer, and the fourth max pooling layer for processing. The output of the 8th-11th-order ordinary convolutional module is then fed into the 2nd-3rd-order separable convolutional module for processing. The output of the 9th-11th-order ordinary convolutional module is then fed into the 2nd-5th-order separable convolutional module for processing. The output of the second average pooling layer is then fed into the 11th-order ordinary convolutional module for processing. The outputs of the 7th-11th-order ordinary convolutional module, the 2nd-3rd-order separable convolutional module, the 2nd-5th-order separable convolutional module, the 11th-order ordinary convolutional module, and the 11th-order ordinary convolutional module are then fed into the second concatenation layer for concatenation. The concatenation result is added to the first feature vector. The result is then processed sequentially through the third BN layer and the second ReLU activation function to obtain the third feature vector.

[0036] Each third feature vector forms a third feature vector group; the first feature vector group is processed by the Inception1_module_2 network and combined with residual connections to obtain the third feature vector group.

[0037] In step 6), the third feature vector group is input into the Inception_reshape network for processing.

[0038] The Inception_reshape network structure specifically includes a 12th-order ordinary convolutional module, a third average pooling layer, a fifth max pooling layer, a third-order separable convolutional module, a 13th-order ordinary convolutional module, a 14th-order ordinary convolutional module, a third splicing layer, a fourth BN layer, and a third ReLU activation function.

[0039] For each third eigenvector in the third eigenvector group, perform the following operation:

[0040] The input third feature vector is simultaneously fed into the twelfth-order ordinary convolutional module, the third average pooling layer, and the fifth max pooling layer for processing. The output of the twelfth-order ordinary convolutional module is then fed into the third-order separable convolutional module for processing. The output of the third average pooling layer is then fed into the thirteenth-order ordinary convolutional module for processing. The output of the fifth max pooling layer is then fed into the fourteenth-order ordinary convolutional module for processing. The outputs of the third-order separable convolutional module, the thirteenth-order ordinary convolutional module, and the fourteenth-order ordinary convolutional module are then fed into the third concatenation layer for parallel concatenation. The concatenated result is then processed sequentially through the fourth BN layer and the third ReLU activation function before being output to obtain the fourth feature vector.

[0041] Each fourth feature vector forms a fourth feature vector group; the third feature vector group is processed by the Inception_reshape network, and its height and width dimensions are halved but the number of channels remains unchanged, thus obtaining the fourth feature vector group.

[0042] In step 7), the fourth feature vector group is input into the Inception2_module_1 network for processing.

[0043] The Inception2_module_1 network structure specifically includes the following modules: the fifteenth-order ordinary convolutional module, the sixteenth-order ordinary convolutional module, the seventeenth-order ordinary convolutional module, the fourth average pooling layer, the sixth max pooling layer, the fourth third-order separable convolutional module, the fifth third-order separable convolutional module, the eighteenth-order ordinary convolutional module, the nineteenth-order ordinary convolutional module, the sixth third-order separable convolutional module, the fourth concatenation layer, the fifth batch normalization (BN) layer, and the fourth ReLU activation function.

[0044] For each fourth eigenvector in the fourth eigenvector group, perform the following operation:

[0045] The input fourth feature vector is simultaneously fed into the fifteenth-order ordinary convolutional module, the sixteenth-order ordinary convolutional module, the seventeenth-order ordinary convolutional module, the fourth average pooling layer, and the sixth max pooling layer for processing. The output of the sixteenth-order ordinary convolutional module is then fed into the fourth-order third-order separable convolutional module for processing. The output of the seventeenth-order ordinary convolutional module is then fed into the fifth-order third-order separable convolutional module and the sixth-order third-order separable convolutional module for processing. The output of the fourth average pooling layer is then fed into the eighteenth-order ordinary convolutional module for processing. The output of the sixth max pooling layer is then fed into the nineteenth-order ordinary convolutional module for processing. The outputs of the fifteenth-order ordinary convolutional module, the fourth-order third-order separable convolutional module, the sixth-order third-order separable convolutional module, the eighteenth-order ordinary convolutional module, and the nineteenth-order ordinary convolutional module are all fed into the fourth concatenation layer for concatenation. The concatenated result is then processed by the fifth BN layer and the fourth ReLU activation function and output to obtain the fifth feature vector.

[0046] Each fifth feature vector is combined to form the fifth feature vector group; the fourth feature vector group is processed by the Inception2_module_1 network and the number of channels is doubled to obtain the fifth feature vector group.

[0047] In step 8), the fifth feature vector group is input into the Inception2_module_2 network for processing, and the sixth feature vector group is input into the Inception2_module_3 network for processing.

[0048] The structures of Inception2_module_2 and Inception2_module_3 networks both include the following modules: a 21st-order ordinary convolutional module, a 21st-order ordinary convolutional module, a 22nd-order ordinary convolutional module, a 5th average pooling layer, a 7th max pooling layer, a 7th-order separable convolutional module, an 8th-order separable convolutional module, a 23rd-order ordinary convolutional module, a 24th-order ordinary convolutional module, a 9th-order separable convolutional module, a 5th concatenation layer, a 6th batch normalization (BN) layer, and a 5th ReLU activation function.

[0049] For each fifth eigenvector in the fifth eigenvector group, perform the following operation:

[0050] The input fifth feature vector is simultaneously fed into the 21st-order ordinary convolutional module, the 21st-order ordinary convolutional module, the 22nd-order ordinary convolutional module, the 5th average pooling layer, and the 7th max pooling layer for processing. The output of the 22nd-order ordinary convolutional module is then fed into the 7th-order separable convolutional module. The output of the 22nd-order ordinary convolutional module is then fed into the 8th-order and 9th-order separable convolutional modules for processing. The output of the 5th average pooling layer is then fed into the 23rd-order ordinary convolutional module for processing. The output of the 7th max pooling layer is then fed into the 24th-order ordinary convolutional module for processing. The outputs of the 21st-order ordinary convolutional module, the 7th-order separable convolutional module, the 9th-order separable convolutional module, the 23rd-order ordinary convolutional module, and the 24th-order ordinary convolutional module are fed into the 5th concatenation layer for parallel concatenation. The concatenation result is added to the fifth feature vector. The result is then processed by the 6th BN layer and the 5th ReLU activation function before being output to obtain the sixth feature vector.

[0051] Each of the sixth feature vectors forms the sixth feature vector group; the fifth feature vector group is processed by the Inception2_module_2 network and combined with residual connections to obtain the sixth feature vector group.

[0052] For each sixth feature vector in the sixth feature vector group, it is input into the Inception2_module_3 network for processing and output as the seventh feature vector;

[0053] Each of the seventh feature vectors forms the seventh feature vector group; the sixth feature vector group is processed by the Inception2_module_2 network and its dimensions are doubled to obtain the seventh feature vector group.

[0054] Therefore, this invention constructs multiple Inception network structures for this well in a parallel splicing manner, which can combine residual and batch regularization methods to simplify the learning objectives and difficulty and avoid overfitting; and by using the CBAM attention mechanism, it can effectively extract image features and improve recognition accuracy.

[0055] In step 9), the seventh feature vector group is input into the Upsampling network for processing.

[0056] The structure of the Upsampling network mainly consists of an upsampling layer, a second spatial convolutional layer, a second convolutional layer with a kernel size of 1×1, and a seventh BN layer connected in sequence.

[0057] For each of the seventh eigenvectors in the seventh eigenvector group, perform the following operation:

[0058] The input seventh feature vector is processed sequentially through an upsampling layer, a second spatial convolutional layer, a second convolutional layer with a kernel size of 1×1, and a seventh BN layer, and then output to obtain the eighth feature vector;

[0059] Each of the eighth feature vectors forms the eighth feature vector group; the seventh feature vector group is processed by the Inception2_module_2 network, and its dimensions are doubled but the number of channels remains unchanged, thus obtaining the eighth feature vector group.

[0060] In step 10), the fused feature vector group obtained by fusing the second feature vector group and the eighth feature vector group is input into the classification network for processing.

[0061] The classification network includes a sixth ReLU activation function, a second CBAM attention mechanism module, a first global average pooling layer, a first fully connected layer, a second fully connected layer, and a softmax classifier;

[0062] For each fused feature vector in the fused feature vector group, the following operations are performed:

[0063] The input fused feature vector is processed sequentially through the sixth ReLU activation function, the second CBAM attention mechanism module, the first global average pooling layer, the first fully connected layer, and the second fully connected layer. The number of channels in the processed output is mapped to the number of gesture classification categories in various gesture images. That is, if the gesture categories in the input gesture images all belong to the 28 gesture categories and 1 non-gesture category in the ASL open-source sign language dataset, for a total of 29 classification categories, then the number of channels in the processed output is mapped to 29. Finally, the processed output is input into the softmax classifier to identify and obtain the gesture classification result of the gesture image corresponding to each fused feature vector.

[0064] The CBAM attention mechanism module includes a channel attention mechanism module and a spatial attention mechanism module; the inputs of the CBAM attention mechanism module are sequentially input into the channel attention mechanism module and the spatial attention mechanism module for processing and output.

[0065] The channel attention mechanism module specifically includes a global max pooling layer, a second global average pooling layer, a third fully connected layer, a fourth fully connected layer, and a first sigmoid function. The inputs of the channel attention mechanism module are simultaneously fed into the global max pooling layer and the second global average pooling layer for processing. The output of the global max pooling layer is then processed by the third and fourth fully connected layers to output the first output. The output of the second global average pooling layer is then fed into the third and fourth fully connected layers for processing to output the second output. The first and second outputs are added together, and the sum is then processed by the first sigmoid function and output. The output result is the channel attention weight of the channel attention mechanism module.

[0066] The third fully connected layer employs a bottleneck mechanism, using a third convolutional layer with a kernel size of 1×1 to reduce the original number of channels of the input to the third fully connected layer, while the fourth fully connected layer restores the channel value of the input to the fourth fully connected layer to the original channel value.

[0067] The spatial attention mechanism module specifically includes global average pooling and global max pooling along the channel dimension, a fourth convolutional layer, and a second sigmoid function. The input of the spatial attention mechanism module is subjected to global average pooling and global max pooling along the channel dimension, and then stacked. The stacked result is then processed by the fourth convolutional layer and the second sigmoid function and output. The output result is the spatial attention weight of the spatial attention mechanism module.

[0068] The kernel size of the fourth convolutional layer in the spatial attention mechanism module is 7×7, and the output of the fourth convolutional layer after processing has a channel dimension of 1.

[0069] Because the weights of different channels and spaces are different, the input of the CBAM attention mechanism module is first processed by the channel attention mechanism module and then by the spatial attention mechanism module, so that the neural network focuses on useful information and downplays useless information. This also prevents the module from overfitting to some extent.

[0070] The first-order and third-order ordinary convolutional modules have the same structure, mainly consisting of convolutional layers, batch normalization (BN) layers, and ReLU activation functions connected in sequence. The input from either the first-order or third-order ordinary convolutional module is sequentially fed into the convolutional layer, BN layer, and ReLU activation function for processing and output. The kernel size of the convolutional layer in the first-order ordinary convolutional module is 1×1, and the kernel size of the convolutional layer in the third-order ordinary convolutional module is 3×3.

[0071] The three-order and five-order separable convolutional modules have the same structure, mainly consisting of a spatial convolutional layer, a separable convolutional layer, a BN layer, and a ReLU activation function connected in sequence. The input from either the three-order or five-order separable convolutional module is sequentially input into the spatial convolutional layer, the separable convolutional layer, the BN layer, and the ReLU activation function for processing and output. The kernel size of the separable convolutional layer in the three-order separable convolutional module is 3×3, and the kernel size of the separable convolutional layer in the five-order separable convolutional module is 5×5.

[0072] The beneficial effects of this invention are:

[0073] 1. This invention employs the CBAM attention mechanism, upsampling, and low-level feature vectors for feature fusion, which protects the integrity of information, simplifies the learning objectives and difficulty, improves the discriminative ability of feature vectors, makes more effective use of parameters, improves the accuracy of the model, and prevents overfitting of the model to a certain extent.

[0074] 2. This invention proposes an improved Inception-v3 network Inception series modules, which greatly reduces the number of model parameters, simplifies the learning objectives and difficulty, and improves recognition accuracy. At the same time, the new Inception series modules also contain a number of residual connections to protect the integrity of information and effectively shorten the gradient propagation path during training, thus avoiding overfitting to a certain extent. Attached Figure Description

[0075] Figure 1 This invention provides the original gesture image obtained by converting the original image to grayscale.

[0076] Figure 2 This is a schematic diagram of the ordinary convolution module conv2d_bn in this invention;

[0077] Figure 3 This is a schematic diagram of the separable_conv2d_bn module in this invention;

[0078] Figure 4 This is a schematic diagram of the channel attention mechanism module in this invention;

[0079] Figure 5 This is a schematic diagram of the spatial attention mechanism module in this invention;

[0080] Figure 6 This is a schematic diagram of the CBAM attention mechanism module in this invention;

[0081] Figure 7 This is a schematic diagram of the Upsampling network in this invention;

[0082] Figure 8 This is a schematic diagram of the stage_1 network in this invention;

[0083] Figure 9 This is a schematic diagram of the Inception1_module_1 network in this invention;

[0084] Figure 10 This is a schematic diagram of the Inception1_module_2 network in this invention;

[0085] Figure 11 This is a schematic diagram of the Inception_reshape network in this invention;

[0086] Figure 12 This is a schematic diagram of the Inception2_module_1 network in this invention;

[0087] Figure 13 This is a schematic diagram of the Inception2_module_2 and Inception2_module_3 networks in this invention;

[0088] Figure 14 This is a flowchart of the technical solution of the present invention;

[0089] Figure 15 This is a bar chart comparing the network parameters of this invention with those of the Inception-v3 network;

[0090] Figure 16 This is a graph comparing the network accuracy of this invention with that of the Inception-v3 network. Detailed Implementation

[0091] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0092] Specific embodiments of the present invention are as follows:

[0093] This invention utilizes the ASL (American Sign Language) open-source sign language dataset. The original image dataset contains gesture images from different angles, under different lighting conditions, at different sizes, and in different background environments, such as... Figure 1 As shown, there are 28 gesture categories and non-gesture categories, totaling 29 categories.

[0094] For the Inception series networks of the improved Inception-v3 network of this invention, i.e., excluding the stage_1 network, the processing is set according to the following formula:

[0095] C z (x) = relu·bn·conv(x)

[0096] SC z (x)=relu·bn·conv·sconv(x)

[0097] b 1×1 (x)=C1(x)

[0098] b 3×3 (x) = SC3·C1(x)

[0099] b 5×5 (x) = SC5·C1(x)

[0100] db 3×3 (x) = SC3·SC3·C1(x)

[0101] mp 3×3 (x)=C1·maxpooling 3×3 (x)

[0102] ap 3×3 (x)=C1·avgpooling 3×3 (x)

[0103] Where x represents the input; C z (x) represents a z-order ordinary convolutional module, where z represents the dimension of the convolutional layer. That is, when z = 1, the kernel size of the ordinary convolutional module is 1×1; · represents the combination operation; relu represents the Relu activation function; bn represents the BN (Batch Normalization) layer, i.e., the normalization layer; conv(x) represents the convolutional layer; SC z (x) denotes a z-order separable convolutional module; conv denotes a convolutional layer that takes the output of the previous step as input; sconv(x) denotes a separable convolutional layer; b 1×1 (x) represents a network consisting of a first-order ordinary convolutional module; C1(x) represents a first-order ordinary convolutional module; b3×3 (x) represents a network consisting of a first-order ordinary convolutional module and a separable convolutional module with a kernel size of 3×3; SC3 represents a third-order separable convolutional module that takes the output of the previous step as input; b 5×5 (x) represents a network consisting of a first-order ordinary convolutional module and a separable convolutional module with a kernel size of 5×5; SC5 represents a fifth-order separable convolutional module that takes the output of the previous step as input; db 3×3 (x) represents a network consisting of a first-order ordinary convolutional module and two separable convolutional modules with a kernel size of 3×3; mp 3×3 (x) represents a network consisting of a first-order ordinary convolutional module and a max-pooling layer with a kernel size of 3×3; C1 represents a first-order ordinary convolutional module that takes the output of the previous step as input; maxpooling 3×3 (x) represents a max-pooling layer with a kernel size of 3×3; ap 3×3 (x) represents a network consisting of a first-order ordinary convolutional module and an average pooling layer with a kernel size of 3×3; avgpooling 3×3 (x) represents an average pooling layer with a convolution kernel size of 3×3.

[0104] The formula for the Inception1_module_1 network is:

[0105]

[0106] in Indicates splicing.

[0107] The formula for the Inception1_module_2 network is:

[0108]

[0109] in This indicates a residual connection.

[0110] The formula for the Inception_reshape network is:

[0111]

[0112] The formula for the Inception2_module_1 network is:

[0113]

[0114] The formula for the Inception2_module_2 network is:

[0115]

[0116] The formula for the Inception2_module_3 network is:

[0117]

[0118] like Figure 2 As shown, the simplified expression for the ordinary convolutional module conv2d_bn designed in this invention is:

[0119] conv2d_bn(filters=x i kernel-size = x o strides = s, padding = p)

[0120] Where filters is the number of filters in the convolutional layer of the conv2d_bn module, kernel-size is the size of the filters, strides is the stride of the filters, and padding is whether to pad the input feature vector. When p is 'same', it means that the input feature vector will be padded with 0 values. When p is 'valid', it means that the input feature vector will not be padded.

[0121] like Figure 3 As shown, the simplified expression for the separable_conv2d_bn separable convolution module designed in this invention is:

[0122] separable_conv2d_bn(filters=x i ,

[0123] kernel-size=x o strides = s, padding = p)

[0124] Here, `filters` represents the number of Conv2D convolutional layer filters in the `separable_conv2d_bn` module, `kernel-size` is the size of the filters in `DepthwiseConv2D`, `strides` is the stride of the filters in `DepthwiseConv2D`, and `padding` indicates whether to pad the input feature vector in the Conv2D and `DepthwiseConv2D` convolutional layers. When `p` is 'same', it means that the input feature vector will be padded with a padding value of 0. When `p` is 'valid', it means that the input feature vector will not be padded. The filter size in the Conv2D convolutional layer is 1, and the stride is 1. The function of the Conv2D convolutional layer is to correlate the number of channels in the output features of the `DepthwiseConv2D` convolutional layer.

[0125] The simplified expressions for the AveragePooling2D and MaxPooling2D layers in this invention are as follows:

[0126] AveragePooling2D(pool_size=x, strides=s, padding=p)

[0127] MaxPooling2D(pool_size=x, strides=s, padding=p)

[0128] Where pool_size is the pooling window size of the pooling layer, strides is the pooling stride, and padding is whether to pad the input feature vector. When p is 'same', it means that the input feature vector will be padded with 0 values. When p is 'valid', it means that the input feature vector will not be padded.

[0129] The simplified expression for the Inception series network of the improved Inception-v3 designed in this invention is as follows:

[0130] module(Inception1_module_c,input=x i output = x o )

[0131] module(Inception_reshape,input=x i output = x o )

[0132] module(Inception2_module_c,input=x i output = x o )

[0133] In this context, Inception1_module_c, Inception_reshape, and Inception2_module_c represent the structure names, where c = 1, 2, or 3. The input parameter represents the height, width, and number of channels of the feature vectors input to the network, and the output parameter represents the height, width, and number of channels of the feature vectors output to the network.

[0134] like Figure 14 As shown, the specific implementation process includes the following steps:

[0135] 1) Collect various gesture images to establish an original image dataset. Normalize the size and perform grayscale processing on the original image dataset to obtain the original gesture image dataset. Normalize the feature size of the read-in original image dataset to a single-channel grayscale image dataset of 256×256×1. Use 256×256×1 as the input size of the network. Then, standardize the single-channel grayscale image dataset by mapping the pixels of the single-channel grayscale image dataset from integers between 0 and 255 to floating-point numbers between 0 and 1 to obtain the original gesture image dataset.

[0136] 2) Improve the Inception-v3 network as a whole and build the improved stage_1 network, Inception1_module_1 network, Inception1_module_2 network, Inception_reshape network, Inception2_module_1 network, Inception2_module_2 network and Inception2_module_3 network; each improved network is directly obtained by improving the Inception-v3 network as a whole.

[0137] 3) Input the original gesture image dataset into the stage_1 network for processing, such as... Figure 8 As shown, low-order features of the original gesture image are initially extracted, and the number of channels is expanded to obtain the original feature vector group with a feature size of 25×25×128.

[0138] The purpose of using multi-layer ordinary convolutional modules is to gradually reduce the height and width of the feature vectors and increase the number of channels, thereby increasing the non-linear expressive power of the network.

[0139] 4) Input the original feature vector set into the Inception1_module_1 network for processing, such as... Figure 9 As shown, feature extraction is performed on the original feature vector group to obtain the first feature vector group of multi-scale feature representation, with a feature size of 25×25×256.

[0140] The network expression for Inception1_module_1 is:

[0141] module(Inception1_module_1,

[0142] input=25×25×128, output=25×25×256)

[0143] The original feature vector set is input into a convolutional normalization network for processing to obtain the second feature vector set.

[0144] 5) Input the first feature vector group into the Inception1_module_2 network for processing, such as... Figure 10 As shown, the third feature vector group is obtained, with a feature size of 25×25×256.

[0145] The network expression for Inception1_module_2 is:

[0146] module(Inception1_module_2,

[0147] input=25×25×256, output=25×25×256)

[0148] 6) Input the third feature vector group into the Inception_reshape network for processing, such as... Figure 11 As shown, the size of the third feature vector group is reduced by shrinking its height and width to half of their original values, resulting in the fourth feature vector group with a feature size of 12×12×256.

[0149] The Inception_reshape network expression is:

[0150] module(Inception_shape,

[0151] input=25×25×256, output=12×12×256)

[0152] 7) Input the fourth feature vector group into the Inception2_module_1 network for processing, such as... Figure 12 As shown, by increasing the number of channels in the fourth feature vector group, a fifth feature vector group is obtained, with a feature size of 12×12×512.

[0153] The network expression for Inception2_module_1 is:

[0154] module(Inception2_module_1,

[0155] input=12×12×256, output=12×12×512)

[0156] The Inception2_module_1 network differs from the Inception1 series networks in that, in order to reduce the number of parameters, it adds nonlinear factors, replacing the 5×5 separable convolutional module with two consecutive 3×3 separable convolutional modules. The resulting fifth feature vector group has the same size as the original separable convolutional module, and the proportion of the two consecutive 3×3 separable convolutional modules is increased.

[0157] 8) Input the fifth feature vector group into the Inception2_module_2 network for processing to obtain the sixth feature vector group with a feature size of 12×12×512.

[0158] The network expression for Inception2_module_2 is:

[0159] module(Inception2_module_2,

[0160] input=12×12×512, output=12×12×512)

[0161] The sixth feature vector group is input into the Inception2_module_3 network, which is identical to the Inception2_module_2 network, for processing, such as... Figure 13 As shown, the seventh feature vector group is obtained, with a feature size of 12×12×512.

[0162] The network expression for Inception2_module_3 is:

[0163] module(Inception2_module_3,

[0164] input=12×12×512, output=12×12×512)

[0165] The Inception2_module_3 network aims to increase the depth of the network model and improve its accuracy.

[0166] 9) Input the seventh feature vector group into the Upsampling network for processing, such as... Figure 7 As shown, the eighth feature vector group is obtained, with a feature size of 24×24×512.

[0167] 10) The second and eighth feature vector groups are fused to obtain a fused feature vector group; this fused feature vector group is then input into the classification network for processing to obtain classification results for various gesture images. The CBAM attention mechanism module in the stage_1 network and the classification network is as follows: Figure 4 , Figure 5 and Figure 6 As shown.

[0168] The reason for fusing low-level and high-level feature vectors is that low-level features have higher resolution and contain more location and detail information, but due to fewer convolutions, their semantics are lower and they contain more noise. High-level features have stronger semantic information, but their resolution is very low and their ability to perceive details is poor. Fusing the two into a single feature vector that is more discriminative than the input features can improve the model's accuracy.

[0169] like Figure 15 As shown, the Inception-v3 network has 21,862,205 parameters, while the network of this invention has only 1,647,637 parameters. The number of parameters in the network of this invention is only about 0.075 times that of the Inception-v3 network. Figure 16 As shown, the validation set accuracy of the network of this invention is not significantly different from that of the Inception-v3 network, and both eventually stabilize at 1.00. However, the validation set accuracy curve of the network of this invention is smoother and more stable than that of the Inception-v3 network, and the convergence speed of the validation set of the network of this invention is also better than that of the Inception-v3 network.

[0170] It is evident that the improved Inception-v3 network obtained by this invention significantly reduces the number of parameters in the Inception-v3 network. This invention, combined with the CBAM attention mechanism module, assigns different weights to different channels and spatial dimensions, allowing the machine to focus on useful information while downplaying useless information. This improves parameter utilization and, to some extent, prevents overfitting. Simultaneously, this invention also incorporates upsampling to fuse high-level and low-level features into a single feature that is more discriminative than the input features, thus improving the accuracy of image classification.

Claims

1. An improved gesture image recognition method using the Inception-v3 network, characterized in that: Includes the following steps: 1) Collect various gesture images, establish an original image dataset, and perform size normalization and grayscale processing on the original image dataset to obtain the original gesture image dataset; 2) Improve the Inception-v3 network as a whole and build the improved stage_1 network, Inception1_module_1 network, Inception1_module_2 network, Inception_reshape network, Inception2_module_1 network, Inception2_module_2 network and Inception2_module_3 network; 3) Input the original gesture image dataset into the stage_1 network for processing, and output the original feature vector set; The structure of the stage_1 network specifically includes a first third-order ordinary convolutional module, a first CBAM attention mechanism module, a second third-order ordinary convolutional module, a third third-order ordinary convolutional module, a first max pooling layer, a first first-order ordinary convolutional module, a fourth third-order ordinary convolutional module, and a second max pooling layer. The original gesture image is input into the first third-order ordinary convolutional module. The first third-order ordinary convolutional module and the first CBAM attention mechanism module are connected in series. The input and output of the first CBAM attention mechanism module are residually connected. The output of the residual connection is used as the input of the second third-order ordinary convolutional module. The second third-order ordinary convolutional module, the third third-order ordinary convolutional module, the first max pooling layer, the first first-order ordinary convolutional module, the fourth third-order ordinary convolutional module, and the second max pooling layer are connected in series. The output of the second max pooling layer is the original feature vector. 4) Input the original feature vector set into the Inception1_module_1 network for processing, and output the first feature vector set; input the original feature vector set into the convolutional normalization network for processing, and output the second feature vector set. The Inception1_module_1 network is configured according to the following formula: in, This is the output of the Inception1_module_1 network; Indicates input; Indicates combined operations; Represents the ReLU activation function; This represents the BN (Batch Normalization) layer, i.e., the normalization layer; This represents a network consisting of a single first-order ordinary convolutional module. This indicates a system consisting of a first-order ordinary convolutional module and a convolutional kernel of size 3. A network consisting of 3 separable convolutional modules; This indicates a system consisting of a first-order ordinary convolutional module and a convolutional kernel of size 5. A network consisting of 5 separable convolutional modules; This indicates a system consisting of a first-order ordinary convolutional module and a convolutional kernel of size 3. A network consisting of 3 max-pooling layers; This indicates a system consisting of a first-order ordinary convolutional module and a convolutional kernel of size 3. A network consisting of 3 average pooling layers; Indicates splicing; 5) Input the first feature vector set into the Inception1_module_2 network for processing, and output the third feature vector set; The Inception1_module_2 network is configured according to the following formula: in, This is the output of the Inception1_module_2 network; Indicates residual connection; 6) Input the third feature vector group into the Inception_reshape network for processing, and output the fourth feature vector group; The Inception_reshape network is configured according to the following formula: in, This is the output of the Inception_reshape network; This indicates a system consisting of a first-order ordinary convolutional module and a convolutional kernel of size 3. A network consisting of 3 separable convolutional modules; 7) Input the fourth feature vector group into the Inception2_module_1 network for processing, and output the fifth feature vector group; The Inception2_module_1 network is configured according to the following formula: in, This is the output of the Inception2_module_1 network; This indicates a system consisting of a first-order ordinary convolutional module and two convolutional kernels of size 3. A network consisting of 3 separable convolutional modules; 8) Input the fifth feature vector group into the Inception2_module_2 network for processing, and output the sixth feature vector group; input the sixth feature vector group into the Inception2_module_3 network, which is exactly the same as the Inception2_module_2 network, for processing, and output the seventh feature vector group; The Inception2_module_2 and Inception2_module_3 networks are configured according to the following formula: in, and These are the outputs of the Inception2_module_2 and Inception2_module_3 networks, respectively. 9) Input the seventh feature vector group into the Upsampling network for processing, and output the eighth feature vector group; 10) Perform feature fusion processing on the second feature vector group and the eighth feature vector group in step 4) to obtain a fused feature vector group; input the fused feature vector group into the classification network for processing, and output the classification results of various gesture images corresponding to step 1).

2. The gesture image recognition method of the improved Inception-v3 network according to claim 1, characterized in that: In step 3), the original gesture image dataset is input into the stage_1 network for processing. For each original gesture image in the original gesture image dataset, the following operations are performed: The structure of the stage_1 network specifically includes a first third-order ordinary convolutional module, a first CBAM attention mechanism module, a second third-order ordinary convolutional module, a third third-order ordinary convolutional module, a first max pooling layer, a first first-order ordinary convolutional module, a fourth third-order ordinary convolutional module, and a second max pooling layer. The original gesture image is input into the first third-order ordinary convolutional module. The first third-order ordinary convolutional module and the first CBAM attention mechanism module are connected in series. The input and output of the first CBAM attention mechanism module are residually connected. The output of the residual connection is used as the input of the second third-order ordinary convolutional module. The second third-order ordinary convolutional module, the third third-order ordinary convolutional module, the first max pooling layer, the first first-order ordinary convolutional module, the fourth third-order ordinary convolutional module, and the second max pooling layer are connected in series. The output of the second max pooling layer is the original feature vector. The original feature vectors obtained from each original gesture image form the original feature vector group.

3. The gesture image recognition method of the improved Inception-v3 network according to claim 1, characterized in that: In step 4): a) Input the original feature vector set into the Inception1_module_1 network for processing: The Inception1_module_1 network structure specifically includes a second first-order ordinary convolutional module, a third first-order ordinary convolutional module, a fourth first-order ordinary convolutional module, a first average pooling layer, a third max pooling layer, a first third-order separable convolutional module, a first fifth-order separable convolutional module, a fifth first-order ordinary convolutional module, a sixth first-order ordinary convolutional module, a first concatenation layer, a first batch normalization layer, and a first ReLU activation function. The original feature vector is simultaneously input into the second first-order ordinary convolutional module, the third first-order ordinary convolutional module, the fourth first-order ordinary convolutional module, the first average pooling layer, and the third max pooling layer. The output of the third first-order ordinary convolutional module... The output of the first third-order separable convolutional module is used as the input to the first fifth-order separable convolutional module. The output of the first average pooling layer is used as the input to the fifth first-order ordinary convolutional module. The output of the third max pooling layer is used as the input to the sixth first-order ordinary convolutional module. The outputs of the second first-order ordinary convolutional module, the first third-order separable convolutional module, the first fifth-order separable convolutional module, the fifth first-order ordinary convolutional module, and the sixth first-order ordinary convolutional module are all input to the first concatenation layer for concatenation. The concatenation result is then concatenated with the first BN layer and the first ReLU activation function in sequence. The output of the first ReLU activation function is the first feature vector. Each first eigenvector forms a first eigenvector group; b) Input the original feature vector set into the convolutional normalization network for processing: The structure of the convolutional normalization network specifically includes a second-order separable convolutional module, a first spatial convolutional layer, a first convolutional layer with a kernel size of 1×1, and a second BN layer; For each original feature vector in the original feature vector group, perform the following operation: The input original feature vector is sequentially fed into a second-order separable convolutional module, a first spatial convolutional layer, a first convolutional layer with a kernel size of 1×1, and a second BN layer for processing and output to obtain the second feature vector; Each second eigenvector forms a second eigenvector group.

4. The gesture image recognition method of the improved Inception-v3 network according to claim 1, characterized in that: In step 5), the first feature vector group is input into the Inception1_module_2 network for processing. The Inception1_module_2 network structure specifically includes a 7th-order ordinary convolutional module, an 8th-order ordinary convolutional module, a 9th-order ordinary convolutional module, a second average pooling layer, a fourth max pooling layer, a second-order separable convolutional module, a second-order and third-order separable convolutional module, a second-order and fifth-order separable convolutional module, an 11th-order ordinary convolutional module, an 11th-order ordinary convolutional module, a second concatenation layer, a third batch normalization (BN) layer, and a second ReLU activation function. The first feature vector is simultaneously input into the 7th-order ordinary convolutional module, the 8th-order ordinary convolutional module, the 9th-order ordinary convolutional module, the second average pooling layer, the fourth max pooling layer, and the 9th-order ordinary convolutional module. The output of the block is used as the input of the second fifth-order separable convolutional module, the output of the second average pooling layer is used as the input of the eleventh-order ordinary convolutional module, the output of the fourth max pooling layer is used as the input of the eleventh-order ordinary convolutional module, and the outputs of the seventh-order ordinary convolutional module, the second third-order separable convolutional module, the second fifth-order separable convolutional module, the eleventh-order ordinary convolutional module, and the eleventh-order ordinary convolutional module are all input into the second concatenation layer for concatenation. The concatenation result is residually connected to the input of the first feature vector. The residual connection result is concatenated with the third BN layer and the second ReLU activation function in sequence. The output of the second ReLU activation function is the third feature vector. Each third eigenvector forms a third eigenvector group.

5. The gesture image recognition method of the improved Inception-v3 network according to claim 1, characterized in that: In step 6), the third feature vector group is input into the Inception_reshape network for processing. The Inception_reshape network structure specifically includes a 12th-order ordinary convolutional module, a third average pooling layer, a fifth max pooling layer, a third-order separable convolutional module, a 13th-order ordinary convolutional module, a 14th-order ordinary convolutional module, a third concatenation layer, a fourth batch normalization (BN) layer, and a third ReLU activation function. The third feature vector is simultaneously input to the 12th-order ordinary convolutional module, the third average pooling layer, and the fifth max pooling layer. The output of the 12th-order ordinary convolutional module serves as the input to the third-order separable convolutional module, the output of the third average pooling layer serves as the input to the 13th-order ordinary convolutional module, and the output of the fifth max pooling layer serves as the input to the 14th-order ordinary convolutional module. The outputs of the third-order separable convolutional module, the 13th-order ordinary convolutional module, and the 14th-order ordinary convolutional module are all input to the third concatenation layer for concatenation. The concatenated result is then concatenated with the fourth BN layer and the third ReLU activation function. The output of the third ReLU activation function serves as the fourth feature vector. Each of the fourth eigenvectors forms a fourth eigenvector group.

6. The gesture image recognition method of the improved Inception-v3 network according to claim 1, characterized in that: In step 7), the fourth feature vector group is input into the Inception2_module_1 network for processing. The Inception2_module_1 network structure specifically includes a 15th-order ordinary convolutional module, a 16th-order ordinary convolutional module, a 17th-order ordinary convolutional module, a fourth average pooling layer, a sixth max pooling layer, a fourth third-order separable convolutional module, a fifth third-order separable convolutional module, an 18th-order ordinary convolutional module, a 19th-order ordinary convolutional module, a sixth third-order separable convolutional module, a fourth concatenation layer, a fifth batch normalization (BN) layer, and a fourth ReLU activation function. The fourth feature vector is simultaneously input into the 15th-order ordinary convolutional module, the 16th-order ordinary convolutional module, the 17th-order ordinary convolutional module, the fourth average pooling layer, and the sixth max pooling layer for processing. The 16th-order ordinary convolutional module... The output of the first three-stage separable convolutional module is used as the input of the fourth three-stage separable convolutional module. The output of the seventeenth three-stage ordinary convolutional module is concatenated with the fifth three-stage separable convolutional module and the sixth three-stage separable convolutional module in sequence. The output of the fourth average pooling layer is used as the input of the eighteenth three-stage ordinary convolutional module. The output of the sixth max pooling layer is used as the input of the nineteenth three-stage ordinary convolutional module. The outputs of the fifteenth three-stage ordinary convolutional module, the fourth three-stage separable convolutional module, the sixth three-stage separable convolutional module, the eighteenth three-stage ordinary convolutional module, and the nineteenth three-stage ordinary convolutional module are all input into the fourth concatenation layer for concatenation. The concatenation result is then concatenated with the fifth batch normalization layer and the fourth ReLU activation function in sequence. The output of the fourth ReLU activation function is the fifth feature vector. Each of the fifth eigenvectors forms a fifth eigenvector group.

7. The gesture image recognition method of the improved Inception-v3 network according to claim 1, characterized in that: In step 8), the fifth feature vector group is input into the Inception2_module_2 network for processing, and the sixth feature vector group is input into the Inception2_module_3 network for processing. The structures of both Inception2_module_2 and Inception2_module_3 networks specifically include a 21st-order ordinary convolutional module, a 21st-order ordinary convolutional module, a 22nd-order ordinary convolutional module, a fifth average pooling layer, a seventh max pooling layer, a seventh third-order separable convolutional module, an eighth third-order separable convolutional module, a 23rd-order ordinary convolutional module, a 24th-order ordinary convolutional module, a ninth third-order separable convolutional module, a fifth concatenation layer, a sixth batch normalization (BN) layer, and a fifth ReLU activation function. The fifth feature vector is simultaneously input into the 21st-order ordinary convolutional module, the 21st-order ordinary convolutional module, the 22nd-order ordinary convolutional module, the fifth average pooling layer, the seventh max pooling layer, and the 22nd-order ordinary convolutional module. The output of the module serves as the input to the seventh-order third-order separable convolutional module. The output of the twenty-second-order first-order ordinary convolutional module is then concatenated with the eighth-order third-order separable convolutional module and the ninth-order third-order separable convolutional module in sequence. The output of the fifth average pooling layer serves as the input to the twenty-third-order first-order ordinary convolutional module. The output of the seventh max pooling layer serves as the input to the twenty-fourth-order first-order ordinary convolutional module. The outputs of the twenty-first-order ordinary convolutional module, the seventh-order third-order separable convolutional module, the ninth-order third-order separable convolutional module, the twenty-third-order first-order ordinary convolutional module, and the twenty-fourth-order first-order ordinary convolutional module are all input into the fifth concatenation layer for concatenation. The concatenation result is residually connected with the fifth feature vector. The residual connection result is then concatenated with the sixth BN layer and the fifth ReLU activation function in sequence. The output of the fifth ReLU activation function is the sixth feature vector. Each of the sixth eigenvectors forms a group of sixth eigenvectors; For each sixth feature vector in the sixth feature vector group, it is input into the Inception2_module_3 network for processing and output as the seventh feature vector; Each of the seventh eigenvectors forms the seventh eigenvector group.

8. The gesture image recognition method of the improved Inception-v3 network according to claim 1, characterized in that: In step 9), the seventh feature vector group is input into the Upsampling network for processing. The structure of the Upsampling network consists of an upsampling layer, a second spatial convolutional layer, a second convolutional layer with a kernel size of 1×1, and a seventh BN layer connected in sequence. For each of the seventh eigenvectors in the seventh eigenvector group, perform the following operation: The input seventh feature vector is processed sequentially through an upsampling layer, a second spatial convolutional layer, a second convolutional layer with a kernel size of 1×1, and a seventh BN layer, and then output to obtain the eighth feature vector; Each of the eighth eigenvectors forms the eighth eigenvector group.

9. The gesture image recognition method of the improved Inception-v3 network according to claim 1, characterized in that: In step 10), the fused feature vector group obtained by fusing the second feature vector group and the eighth feature vector group is input into the classification network for processing. The classification network includes a sixth ReLU activation function, a second CBAM attention mechanism module, a first global average pooling layer, a first fully connected layer, a second fully connected layer, and a softmax classifier; For each fused feature vector in the fused feature vector group, the following operations are performed: The input fused feature vector is processed sequentially through the sixth ReLU activation function, the second CBAM attention mechanism module, the first global average pooling layer, the first fully connected layer, and the second fully connected layer. The number of channels in the processed output is mapped to the number of gesture classification categories in various gesture images. Finally, the processed output is input into a softmax classifier to identify and obtain the gesture classification result of the gesture image corresponding to each fused feature vector.

10. A gesture image recognition method using an improved Inception-v3 network according to any one of claims 2-8, characterized in that: The first-order and third-order ordinary convolutional modules have the same structure, consisting of convolutional layers, batch normalization (BN) layers, and ReLU activation functions connected sequentially. The input from either the first-order or third-order ordinary convolutional module is sequentially fed into the convolutional layer, BN layer, and ReLU activation function for processing and output. The kernel size of the convolutional layer in the first-order ordinary convolutional module is 1×1, and the kernel size of the convolutional layer in the third-order ordinary convolutional module is 3×3. The three-order and five-order separable convolutional modules have the same structure, consisting of a spatial convolutional layer, a separable convolutional layer, a BN layer, and a ReLU activation function connected in sequence. The input from either the three-order or five-order separable convolutional module is sequentially input into the spatial convolutional layer, the separable convolutional layer, the BN layer, and the ReLU activation function for processing and output. The kernel size of the separable convolutional layer in the three-order separable convolutional module is 3×3, and the kernel size of the separable convolutional layer in the five-order separable convolutional module is 5×5.

Images