A gesture image feature extraction method based on FractalNet network

Abstract

This invention discloses an improved gesture image feature extraction method based on the FractalNet network. The method involves: acquiring and normalizing the size of the gesture image; inputting the original gesture image into an R-FractalNet1 network structure to obtain the original feature vector; inputting the original feature vector and the original gesture image into a spatial feature pyramid structure for multi-scale features; inputting the original feature vector into an R-FractalNet2 network structure to obtain a low-order feature vector; concatenating the multi-scale features and the low-order feature vector channels and inputting them into an R-FractalNet3 network structure to output a high-dimensional feature vector; and passing the high-dimensional feature vector through a global average pooling layer, a fully connected layer, and a softmax classifier to obtain the classification result. This invention incorporates numerous residual modules, protecting information integrity, simplifying the learning objective and difficulty, effectively shortening the gradient propagation path, avoiding overfitting, and improving recognition accuracy.

Description

Technical Field

[0001] This invention relates to a method for extracting gesture images, specifically a gesture image feature extraction method based on an improved FractalNet network. Background Technology

[0002] With the maturity of neural network technology, gesture recognition based on computer vision is becoming increasingly popular. However, in practical applications, the challenges of recognizing gestures and the required accuracy have increased the difficulty of real-time gesture recognition. Despite significant progress in gesture recognition technology, many challenges remain in real-world environments, such as the impact of lighting and distance on performance. Currently, deep learning-based gesture recognition methods are becoming mainstream. These typically use convolutional neural networks (CNNs) to automatically extract depth features such as contours and skin color from gestures through convolutional layers, and then fuse these features. However, features extracted by a single CNN tend to have a limited scale and poor accuracy. Summary of the Invention

[0003] To address the issues of overfitting and low accuracy in gesture feature extraction during the training of the FractalNet network, this invention aims to propose an effective gesture image feature extraction method that can meet the requirements of high-precision gesture recognition in complex backgrounds. This invention utilizes the structure of FractalNet, combined with residual modules to preserve information integrity, simplify the learning objective and difficulty, optimize gradient propagation in FractalNet, appropriately alleviate the accuracy degradation caused by overfitting, and incorporates a spatial feature pyramid structure into the model for feature extraction at multiple scales, generating multi-scale feature representations and further improving recognition accuracy.

[0004] The technical solution of the present invention includes the following steps:

[0005] 1) Acquire images of various gestures to obtain raw images, and perform size normalization processing on the raw images to obtain raw gesture images;

[0006] 2) Input the original gesture image into the R-FractalNet1 network structure to obtain the original feature vector;

[0007] 3) Input the original feature vector and the original gesture image into the spatial feature pyramid structure to extract features from the multi-scale image and obtain multi-scale features;

[0008] 4) Input the original feature vector into the R-FractalNet2 network structure to obtain the low-order feature vector after channel expansion;

[0009] 5) The multi-scale features output by the spatial feature pyramid structure and the low-order feature vectors output by the R-FractalNet2 network structure are concatenated in the channel dimension and then input into the R-FractalNet3 network structure to fuse features in the channel and output high-dimensional feature vectors.

[0010] 6) After passing the high-dimensional feature vector through a global average pooling layer, it is then passed through a fully connected layer to map the number of channels to 29. Finally, it is input into a softmax classifier to obtain the classification result of the gesture image.

[0011] The R-FractalNet1, R-FractalNet2, and R-FractalNet3 network structures described in this invention are all obtained by improving upon the FractalNet structure.

[0012] The R-FractalNet3 network structure is mainly composed of three fourth-order convolutional modules connected in sequence. The R-FractalNet1 and R-FractalNet2 network structures have the same topology, both mainly composed of a fourth-order convolutional module and a max pooling layer connected in sequence.

[0013] The fourth-order convolutional module consists of two third-order convolutional modules, one fourth-order single convolutional module, and an additive layer. The two third-order convolutional modules are connected consecutively to form a fourth-order continuous convolutional block. The input of the fourth-order convolutional module is processed by the fourth-order continuous convolutional block and the fourth-order single convolutional module, respectively. The results are added by the additive layer to obtain the output of the fourth-order convolutional module, which is then input into the max pooling layer.

[0014] The third-order convolutional module consists of two second-order convolutional modules, one third-order single convolutional module, and an additive layer. The two second-order convolutional modules are connected consecutively to form a third-order continuous convolutional block. The input of the third-order convolutional module is processed by the third-order continuous convolutional block and the third-order single convolutional module, respectively. The results are then added by the additive layer to obtain the output of the third-order convolutional module.

[0015] The second-order convolution module consists of two first-order convolution modules, one second-order single convolution module, and an additive layer. The two first-order convolution modules are connected in parallel and then superimposed with the residual module to form a second-order continuous convolution residual block. The input of the second-order convolution module is processed by the second-order continuous convolution residual block and the second-order single convolution module, respectively, and the results are added by the additive layer to obtain the output of the second-order convolution module.

[0016] Parallel superposition of residual modules means that the input of the original module is also input into the residual module at the same time, and the output of the residual module is added to the output of the original module as a common output.

[0017] The structures of the fourth-order single convolutional module, the third-order single convolutional module, and the second-order single convolutional module are the same as those of the first-order convolutional module. Specifically, the first-order convolutional module consists of a convolutional layer, a normalization layer, and a ReLU activation function connected in sequence.

[0018] Therefore, this invention constructs various Net network structures according to the recursive definition of continuous fractal modules, which can combine fractal and residual methods to simplify the learning objectives and difficulty, effectively shorten the gradient propagation path, and avoid overfitting; moreover, the network contains multiple sub-paths of different lengths, which can effectively extract image features and improve recognition accuracy.

[0019] The aforementioned fourth-order convolution module Configure the processing according to the following formula:

[0020]

[0021]

[0022]

[0023] Where z represents the input feature vector. This indicates a sequential combination of operations. This indicates an addition operation. Represents the ReLU activation function. This represents a Batch Normalization layer, i.e., a normalization layer. This indicates the corresponding order, i.e., the number of columns or the width of the network structure. hour, , This represents a c-order convolutional module. =1-4; This refers to the operation of parallel superimposed residual modules.

[0024] The residual module is selected as either residual mapping or identity mapping. The residual module formed by the parallel superposition of two consecutive first-order convolutional modules directly connected to the input of the network structure is a residual mapping, while the remaining residual modules are identity mappings. The residual mapping is mainly composed of operations that expand the channel dimension of the image and reduce the size dimension of the image in sequence, while the identity mapping directly outputs the input without any operations.

[0025] In step 3), the spatial feature pyramid is divided into three branches: the low-order feature branch is used as the first branch, the detail feature branch is used as the second branch, and the edge feature extraction branch is used as the third branch. The original gesture image is processed by the first branch to obtain a feature vector that is reduced by 4 times in height and width dimensions. The original feature vector is processed by the second and third branches to obtain feature vectors that are reduced by 2 times in height and width dimensions respectively.

[0026] The first branch mainly consists of a convolutional layer with a kernel size of 1×1, an average pooling layer with a kernel size of 4×4 and a stride of 4, and a ReLU activation function connected in sequence. The original gesture image first passes through a convolutional layer with a kernel size of 1×1 to process the channel information and obtain a low-order feature vector after channel dimension transformation. The low-order feature vector is then processed by two average pooling layers to obtain a feature vector that is reduced by a factor of 4 in both height and width dimensions.

[0027] The second branch is mainly composed of a convolutional layer with a kernel size of 1×1 and a max pooling layer with a kernel size of 2×2 connected in sequence. The original feature vector is first processed by the 1×1 convolutional layer to process the channel dimension, and then processed by the max pooling layer to output a feature vector that is reduced by a factor of 2 in both height and width dimensions.

[0028] The third branch mainly consists of a downsampling layer using the nearst mode, an upsampling layer using the nearst mode, and a max pooling layer with a convolutional kernel size of 2×2 connected in sequence. The original feature vector is first passed through the max pooling layer to compress it into a feature vector with a height and width dimension of 1 / 2. This feature vector is then passed through the upsampling layer to restore the height and width of the feature vector. Finally, it is pooled by the 2×2 max pooling layer, and the output is a feature vector with a height and width dimension reduced by half.

[0029] This invention contains a large number of residual modules, which protect information integrity, simplify learning objectives and difficulty, effectively shorten gradient propagation paths, avoid overfitting, and improve recognition accuracy. The spatial feature pyramid structure adopted in this invention can extract features of images at different scales, identify the main and detailed features of gestures, and improve recognition accuracy.

[0030] The beneficial effects of this invention are:

[0031] This invention employs a network model incorporating residual modules, FractalNet. The numerous residual modules in FractalNet protect the integrity of information, simplify the learning objective and difficulty, and effectively shorten the gradient propagation path during training, thus avoiding overfitting to some extent.

[0032] This invention proposes a spatial feature pyramid structure. The first branch simply reduces the scale of the original image without depthwise convolution, resulting in a feature vector containing a large number of semantic features. The second branch can extract and enhance the details in the gesture, and the third branch can accurately extract the outer contour of the gesture, identifying the main part of the gesture. The outputs of the three branches are then concatenated with the low-order feature vector in the channel dimension. By combining different receptive fields and feature representation methods, it is possible to extract features at different scales to varying degrees, generating multi-scale feature representations and further improving recognition accuracy. Attached Figure Description

[0033] Figure 1 These are partial hand gesture images from the sign language dataset of this invention;

[0034] Figure 2 The gesture image is obtained after random cropping within a small range for use in this invention.

[0035] Figure 3 This is a schematic diagram of the basic CBR module used in this invention.

[0036] Figure 4 This is a schematic diagram of the residual connection structure used in this invention;

[0037] Figure 5 This is a schematic diagram of the FractalNet module that incorporates a residual module used in this invention;

[0038] Figure 6 This is a flowchart of the technical solution of the present invention;

[0039] Figure 7 This is the first branch of the spatial feature pyramid structure in this invention;

[0040] Figure 188 shows the feature map extracted from the first branch of the spatial feature pyramid structure in this invention;

[0041] Figure 9 This is the second branch of the spatial feature pyramid structure in this invention;

[0042] Figure 10 This is the feature map extracted from the second branch of the spatial feature pyramid structure in this invention;

[0043] Figure 11 This is the third branch of the spatial feature pyramid structure in this invention;

[0044] Figure 12 This is the feature map extracted from the third branch of the spatial feature pyramid structure in this invention.

[0045] Figure 13 These are the test results for Network 1, Network 2, and Network 3 in this invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] Specific embodiments of the present invention are as follows:

[0048] The dataset used in this invention is the ASL (American Sign Language) open-source sign language dataset, some of which is shown below. Figure 1 As shown, it includes gesture images from different angles, under different lighting conditions, different sizes, and different background environments, comprising 28 gesture categories and non-gesture categories, for a total of 29 classification categories.

[0049] In a specific implementation of this invention, each convolutional structure in the R-FractalNet structure is replaced with a residual module, and the fourth-order convolutional module... The expression for the network is:

[0050]

[0051]

[0052]

[0053] Where z represents the input feature vector. This indicates a sequential combination of operations. This indicates an addition operation. Represents the ReLU activation function. This represents a Batch Normalization layer, i.e., a normalization layer. This indicates the corresponding order, i.e., the number of columns or the width of the network structure. hour, , This represents a c-order convolutional module. =1-4; This refers to the operation of parallel superimposed residual modules.

[0054] when The situation at that time is as follows Figure 3 As shown on the left, and simplified to a single CBR module, as follows: Figure 3 Right side. When The situation at that time is as follows Figure 5 As shown.

[0055] Pick After obtaining the result, adding a max pooling layer yields the depth. The R-FractalNet module is illustrated in Figure 5.

[0056] If depth Then it will indicate Figure 5 The R-FractalNet modules can be overlapped twice.

[0057] The R-FractalNet module used in this invention is simplified to the following expression:

[0058]

[0059] In the expression, R-FratcalNet_x represents the structure name, the input parameter is the height and width dimensions of the input feature vector of the structure, the output parameter is the height and width dimensions of the output feature vector of the structure, the c parameter is the width of the structure, and the b parameter is the depth of the network.

[0060] The residual module is selected as either residual mapping or identity mapping. Among them, the residual module formed by the parallel superposition of two consecutive first-order convolutional modules directly connected to the input of the network structure is a residual mapping, and the remaining residual modules are identity mappings. The residual mapping is mainly composed of operations that expand the channel dimension of the image and reduce the size dimension of the image in sequence, while the identity mapping directly outputs the input without any operation.

[0061] In residual mapping, the input features of the residual module are extracted to obtain a low-order first feature vector through residual mapping. At the same time, the input features of the residual module are extracted to a higher-dimensional second feature vector through parallel convolution modules. The first feature vector and the second feature vector are added in the height and width dimensions to obtain the output feature vector in the residual module.

[0062] In the identity mapping, the input features of the identity module are directly used as the first feature vector of the identity mapping. At the same time, the input features of the identity module are processed by the convolution module to output the second feature vector. The first feature vector and the second feature vector are added in the height and width dimensions to obtain the output feature vector in the identity module.

[0063] like Figure 6 As shown, the implementation process includes the following steps:

[0064] 1) Normalize the size of the original image to obtain the gesture image. Normalize the size of the original gesture image into a 3×256×256 three-channel RGB image. Use 3×256×256 as the input size of the neural network. Then normalize the three-channel RGB image, mapping the integers between 0 and 255 in the three-channel RGB image to floating-point numbers between 0 and 1.

[0065] During the training phase, a small-scale random cropping is employed. This involves randomly cropping a given image to different sizes and aspect ratios, then scaling the cropped image to a size of 256×256. The chosen random cropping size is 0.9-1 times the original image size, and the aspect ratio is 0.9-1.1. The cropped result is shown below. Figure 2 As shown.

[0066] Random cropping can address the situation where some gesture images in the dataset are partially occluded, thus reducing the model's dependence on training samples.

[0067] 2) The gesture image is input into the R-FractalNet1 module, where low-level features of the gesture image are initially extracted and the number of channels is expanded. The feature size of the low-level gesture image obtained after processing is 8×128×128.

[0068] like Figure 5 As shown, the module expression used is:

[0069]

[0070]

[0071] 3) Input the original feature vector and gesture image into the spatial feature pyramid structure to extract features from the multi-scale image and obtain multi-scale feature representation.

[0072] The spatial feature pyramid is divided into three branches:

[0073] ① First-order feature branch:

[0074] like Figure 7 As shown, the first branch mainly consists of a convolutional layer with a kernel size of 1×1, an average pooling layer with a kernel size of 4×4 and a stride of 4, and a ReLU activation function. The original gesture image of size 256×256 first passes through the convolutional layer with a kernel size of 1×1 to process the channel information, obtaining a low-order feature vector after channel dimension transformation. The low-order feature vector is then processed by two average pooling layers to obtain a feature vector of size 64×64.

[0075] A convolutional layer with a 1×1 kernel adds a non-linear element to the original input image, enhancing its feature representation. A 4×4 average pooling layer primarily aims to compress the feature map size without excessively processing image details, thus preserving a large amount of semantic features in the feature vector. Figure 8 .

[0076] ② Section Feature Branches:

[0077] like Figure 9As shown, the second branch mainly consists of a convolutional layer with a 1×1 kernel and a max-pooling layer with a 2×2 kernel. The 128×128 feature map is first processed by the 1×1 convolutional layer to reduce its channel dimension, and then processed by the max-pooling layer, outputting a 64×64 feature vector. Figure 10 Compared to average pooling and max pooling layers with a kernel size of 4×4, this structure is better able to notice and preserve the details in gestures. For example... Figure 10 Compared to a max pooling layer with a kernel size of 2×2, this structure does not produce too much noise, reducing the impact of subsequent processing on feature processing and extraction.

[0078] ① Edge feature extraction branch:

[0079] like Figure 11 As shown, the third branch mainly consists of a downsampling layer using the nearest mode, an upsampling layer using the nearest mode, and a max-pooling layer with a kernel size of 2×2. The 128×128 feature map is first passed through the max-pooling layer, compressing the feature dimension to a 64×64 feature vector. This feature vector is then passed through the upsampling layer to restore it to a 128×128 feature vector, and finally pooled by a 2×2 max-pooling layer, outputting a 64×64 feature vector. Figure 12 This branch eliminates the blurring of edges in the gesture feature map, identifies the main part of the gesture, and improves the accuracy of gesture recognition in complex backgrounds and when the gesture is relatively blurry.

[0080] 4) Input the original feature vector into R-FractalNet2 to obtain low-order feature vectors for channel expansion and spatial information processing.

[0081] The module expressions used are:

[0082]

[0083]

[0084] 5) The output of the spatial feature pyramid and the low-order feature vector are concatenated in the channel dimension using a concatenation layer and then input into R-FractalNet3 to fuse features in the channel dimension and output high-dimensional features.

[0085] In step 5):

[0086] The four sets of multi-scale feature vectors containing different spatial feature information, namely the first feature branch, the second feature branch, the third feature branch in step 3), and the output vector in step 4), are concatenated along the channel dimension to form a feature vector with a multi-scale spatial structure.

[0087] The module expressions used are:

[0088]

[0089]

[0090] The R-FractalNet3 module can effectively integrate different spatial features along the channel dimension, combining the extracted spatial features to expand the different semantic features contained in the feature vector along the channel dimension.

[0091] 6) After passing the high-dimensional feature vector through a global average pooling layer, it is then passed through a fully connected layer to map the number of channels to 29. Finally, it is input into a softmax classifier to obtain the classification result of the gesture image.

[0092] like Figure 13 As shown, Network 1 is a FractalNet network that does not use residual modules and spatial feature pyramids; Network 2 is a FractalNet network that uses residual modules but does not use spatial feature pyramids; and Network 3 is the FractalNet network proposed in this invention that combines residual modules and spatial feature pyramid structures. All three networks were trained on a training set containing 6206 images and tested on a test set containing 78996 images. Network 1 had 38 errors in the test results, Network 2 had 27 errors in the test results, and Network 3 had 11 errors in the test results.

[0093] It is evident that the residual module incorporated in this invention plays a role in avoiding overfitting. The spatial feature pyramid structure of this invention extracts feature vectors containing different semantic features from the image and identifies the main and detailed features of the gesture. Then, the R-FractalNet module is used to effectively fuse different features scattered across the channel dimension. Finally, the integrated processing output is the classification result, which greatly improves the accuracy of image classification.

Claims

1. A gesture image feature extraction method based on an improved FractalNet network, characterized in that, The process includes the following steps: 1) Acquiring original images of various gestures and normalizing the size of the original images to obtain original gesture images; 2) Inputting the original gesture images into an R-FractalNet1 network structure to obtain original feature vectors; 3) Inputting the original feature vectors and the original gesture images into a spatial feature pyramid structure to extract multi-scale features; 4) Inputting the original feature vectors into an R-FractalNet2 network structure to obtain low-order feature vectors after channel expansion; 5) Concatenating the multi-scale features output by the spatial feature pyramid structure and the low-order feature vectors output by the R-FractalNet2 network structure along the channel dimension and then inputting them into an R-FractalNet3 network structure to fuse features along the channels and output high-dimensional feature vectors; 6) Passing the high-dimensional feature vectors through a global average pooling layer and then through a fully connected layer, and finally inputting them into a softmax classifier to obtain the classification result of the gesture images. The topology of the R-FractalNet3 network structure consists of three fourth-order convolutional modules connected in sequence. The topologies of the R-FractalNet1 and R-FractalNet2 networks are the same, each consisting of a fourth-order convolutional module and a max pooling layer connected in sequence. The fourth-order convolutional module includes two third-order convolutional modules, one fourth-order single convolutional module, and an additive layer. The two third-order convolutional modules are connected consecutively to form a fourth-order continuous convolutional block. The input of the fourth-order convolutional module is processed by the fourth-order continuous convolutional block and the fourth-order single convolutional module, respectively. The results are added by the additive layer to obtain the output of the fourth-order convolutional module, which is then input to the max pooling layer. The third-order convolutional module includes two second-order convolutional modules, one third-order single convolutional module, and an additive layer. The two second-order convolutional modules are connected consecutively to form a third-order continuous convolutional block. The input of the third-order convolutional module is processed by the third-order continuous convolutional block and the third-order single convolutional module, respectively. The results are then added by the additive layer to obtain the output of the third-order convolutional module. The second-order convolution module includes two first-order convolution modules, one second-order single convolution module, and an additive layer. The two first-order convolution modules are connected in parallel and then superimposed with a residual module to form a second-order continuous convolution residual block. The input of the second-order convolution module is processed by the second-order continuous convolution residual block and the second-order single convolution module, respectively, and then the results are added by the additive layer to obtain the output of the second-order convolution module. The structures of the fourth-order single convolution module, the third-order single convolution module, and the second-order single convolution module are the same as those of the first-order convolution module. Specifically, the first-order convolution module is composed of a convolutional layer, a normalization layer, and a ReLU activation function connected in sequence. Furthermore, in step 3), the spatial feature pyramid structure is divided into three branches: the low-order feature branch is used as the first branch, the detail feature branch is used as the second branch, and the edge feature extraction branch is used as the third branch. The original gesture image is processed by the first branch to obtain a feature vector that is reduced by a factor of 4 in the height and width dimensions. The original feature vector is processed by the second branch and the third branch to obtain feature vectors that are reduced by a factor of 2 in their respective height and width dimensions. The first branch consists of a convolutional layer, an average pooling layer, and a ReLU activation function connected in sequence. The second branch consists of a convolutional layer with a kernel size of 1×1 and a max pooling layer with a kernel size of 2×2 connected in sequence; The third branch consists of a downsampling layer using the nearst mode, an upsampling layer using the nearst mode, and a max pooling layer connected in sequence.

2. The gesture image feature extraction method based on the improved FractalNet network according to claim 1, characterized in that: The fourth-order convolution module is configured and processed according to the following formula: Where z represents the input feature vector. This indicates a sequential combination of operations. This indicates an addition operation. Represents the ReLU activation function. This represents a Batch Normalization layer, i.e., a normalization layer. Indicates the corresponding order. This represents a c-order convolutional module. =2,3; This refers to the operation of parallel superimposed residual modules.

3. The gesture image feature extraction method based on the improved FractalNet network according to claim 1, characterized in that: The residual module is selected as either a residual mapping or an identity mapping. The residual module formed by the parallel superposition of two consecutive first-order convolutional modules directly connected to the input of the network structure is a residual mapping, while the remaining residual modules are identity mappings. The residual mapping consists of operations that expand the channel dimension of the image and reduce the size dimension of the image, while the identity mapping directly outputs the input without any operations.

Images