Gesture recognition method and device based on DCViT model

Abstract

The invention discloses a gesture recognition method and device based on a DCViT model, and the method comprises the steps: obtaining a gesture image data set of a known corresponding gesture type, carrying out the preprocessing of each gesture image, and carrying out the dividing according to a proportion, and obtaining a training set; the method comprises the following steps: establishing a DCViT model by taking a Vision Transform model as a basic framework; wherein a deformable convolution module is added to the front end of the Vision Transform model and is used for carrying out feature extraction on an input image of the DCViT model, and then a feature graph obtained through extraction is input into the Vision Transform model; training the DCViT model by using training set data to obtain a gesture recognition classifier; and identifying and classifying the gesture image to be identified by using the gesture identification classifier. According to the invention, gestures in the gesture image can be identified and classified more accurately and efficiently.

Description

Technical Field

[0001] The present invention belongs to the technical field of data recognition, and in particular relates to a gesture recognition method and device based on a DCViT model. Background Art

[0002] The main approaches to gesture recognition involve acquiring data from sensor-equipped gloves or visual sensors.

[0003] Although the gesture recognition system based on sensor gloves can provide accurate gesture capture, it also has some obvious disadvantages: (1) Data gloves can be relatively expensive, especially those advanced gloves that integrate high-precision sensors, which may limit their popularity in cost-sensitive applications. (2) Gloves often restrict movement and may cause adverse skin reactions in people with sensitive skin, and the different body sizes of users may also cause discomfort. (3) Glove-based systems may limit the user's natural interaction method because the user needs to wear gloves at all times, which may affect the naturalness and intuitiveness of the interaction. The gesture recognition method based on visual sensors can overcome the above problems. It only needs a camera to capture the gestures made by humans, and no other equipment is required, which greatly reduces the cost of use. In addition, it does not require any equipment to be worn, and is more universal.

[0004] Although gesture recognition methods based on visual sensors have achieved great success, there are still some challenging problems. The main reasons are as follows: Since the movements of human hands are complex and changeable, with a high degree of freedom, it is difficult to accurately recognize gestures from images. Lighting changes, background clutter, occlusion, and fast and changeable hand movements will affect the accuracy and operation speed of gesture recognition.

[0005] Currently, most existing gesture recognition technologies are based on hand key points, such as the patent publication number CN114898457A "A dynamic gesture recognition method and system based on hand key points and transformer" and the patent publication number "A two-handed gesture interaction method, device, electronic device and storage medium". Such gesture recognition methods are too dependent on hand key points, so that they cannot accurately detect key points for complex gestures, thus reducing recognition accuracy. Summary of the invention

[0006] The present invention provides a gesture recognition method and device based on a DCViT model, which can more accurately and efficiently recognize and classify gestures in a gesture image.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions: A gesture recognition method based on a DCViT model, comprising: Step 1, obtaining a gesture image dataset of known corresponding gesture types, preprocessing each gesture image therein and dividing it in proportion to obtain a training set; Step 2, establishing a DCViT model with the Vision Transformer model as the basic architecture; wherein, a deformable convolution module is added to the front end of the Vision Transformer model to extract features from the input image of the DCViT model, and then the extracted feature map is input into the Vision Transformer model; Step 3, using the training set data to train the DCViT model to obtain a gesture recognition classifier; Step 4: Use a gesture recognition classifier to classify the gesture image to be recognized.

[0008] Furthermore, step 1 also includes proportionally dividing the gesture image set obtained from the preprocessing to obtain a test set, and step 3 also uses the test set to assist in training the DCViT model to obtain a gesture recognition classifier that meets a preset classification accuracy.

[0009] Furthermore, the deformable convolution module includes an offset prediction layer and a deformable convolution layer; the offset prediction layer uses a convolutional network to predict the offset of each position on the input image; the deformable convolution layer: for each position on the input image, the shape of the standard convolution kernel is adjusted using the corresponding offset to obtain the deformed convolution kernel at the position, and then the deformed convolution kernel is used to perform a convolution operation on the pixels at the position.

[0010] Furthermore, the Vision Transformer model includes a Patch Embedding module, a position encoding module, a Transformer encoder module and a classification module; The Patch Embedding module is used to: split the input feature map into small blocks of fixed size, and flatten each small block into a vector; The position encoding module is used to: encode the position of each small block obtained by segmentation in the feature map before segmentation; The Transformer encoder module is composed of multiple stacked Transformer encoder layers, which processes small flattened vectors on each position encoding; The classification module is used to convert the output of the Transformer encoder module into a classification result.

[0011] Furthermore, the Transformer encoder module consists of a normalization layer, a multi-head attention mechanism, a residual connection and a linear layer.

[0012] Furthermore, the classification module is implemented by an MLP with one hidden layer during pre-training and by a linear layer during fine-tuning.

[0013] A gesture recognition device based on a DCViT model, comprising: A preprocessing module is used to: obtain a gesture image data set of known corresponding gesture types, preprocess each gesture image therein and divide it into training sets in proportion; The DCViT model module is built on the Vision Transformer model as the basic architecture. A deformable convolution module is added to the front end of the Vision Transformer model to extract features from the input image of the DCViT model, and then the extracted feature map is input to the Vision Transformer model. A model training module, used to: train the DCViT model using training set data to obtain a gesture recognition classifier; The gesture recognition and classification module is used to: use a gesture recognition classifier to recognize and classify gesture images to be recognized.

[0014] Furthermore, the deformable convolution module includes an offset prediction layer and a deformable convolution layer; the offset prediction layer uses a convolutional network to predict the offset of each position on the input image; the deformable convolution layer: for each position on the input image, the shape of the standard convolution kernel is adjusted using the corresponding offset to obtain the deformed convolution kernel at the position, and then the deformed convolution kernel is used to perform a convolution operation on the pixels at the position.

[0015] Furthermore, the Vision Transformer model includes a Patch Embedding module, a position encoding module, a Transformer encoder module and a classification module; The Patch Embedding module is used to: split the input feature map into small blocks of fixed size, and flatten each small block into a vector; The position encoding module is used to: encode the position of each small block obtained by segmentation in the feature map before segmentation; The Transformer encoder module is composed of multiple stacked Transformer encoder layers, which processes small flattened vectors on each position encoding; The classification module is used to convert the output of the Transformer encoder module into a classification result.

[0016] Beneficial Effects The gesture recognition method and device based on the DCViT model of the present invention can realize high-precision recognition of gestures in gesture images of various resolutions by constructing and training an image deep learning classification model based on the DCViT model. Specifically, For high-resolution images, if the image is directly input into the Vision Transformer model, since the model directly divides the entire image into multiple patches, and each patch contains a limited amount of information, this may lead to the loss of some detail information, and then cause a certain error in the gesture category of the final output. Therefore, the present invention adds a Deformable Convolution module at the front end of the Vision Transformer model. On the one hand, it is robust to some disturbances of the input image (such as noise, small deformation, etc.), and on the other hand, by introducing an offset, it can be adjusted during the training process to adapt to the specific shape and content of the input feature map. The convolution kernel obtained in this way can dynamically adapt to the deformation of the object and effectively solve the information loss problem of the Vision Transformer model, greatly improving the accuracy of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a method for classifying gesture images according to the present invention; Figure 2 is a flowchart of a method for classifying gesture images according to an embodiment of the present invention; Figure 3 It is a schematic diagram of the DCViT model structure of the present invention; Figure 4 It is a further detailed structural diagram of the DCViT model structure of the present invention. DETAILED DESCRIPTION

[0018] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process to further explain the technical solution of the present invention.

[0019] Example 1 This embodiment provides a gesture recognition method based on the DCViT model. Figure 1 As shown, including: Step 1: obtain a gesture image dataset of known corresponding gesture types, preprocess each gesture image, and divide it into training set data, test set data, and validation set data in a ratio of 6:2:2.

[0020] The hand gesture image dataset of the embodiment of the present invention comes from the ASL dataset, the NUS hand gesture dataset and the ASL with digits dataset. The ASL dataset contains 65,774 images, covering 24 different gestures from letters A to Z, of which the letter W has the largest number of samples, which is 6,221, and the letter F has the least number of samples, which is 5,235. During model training, the images with real label data are divided into training set and validation set, and 39,464 images are selected as training set, 13,155 images as test set, and 13,155 images as validation set.

[0021] The NUS hand gesture dataset is a more complex dataset that includes images of 10 hand gestures taken in natural backgrounds in and around the National University of Singapore. The dataset consists of gestures made by 40 subjects of different ethnicities, ranging in age from 22 to 56 years old, each of whom showed 10 gestures and performed each gesture 5 times. In addition, the dataset also includes a set of background images that do not contain any gestures. All images are in RGB format and provided in .jpeg format. During model training, 1200 images were selected as training sets, 400 as test sets, and 400 as validation sets.

[0022] The ASL with digits dataset is a dataset that focuses on digit gestures in American Sign Language (ASL). This dataset was created by Arda Mavi to help create machine learning algorithms that can recognize signed digits. The dataset contains 10 different ASL digit gestures demonstrated by 218 volunteers with their right hands, each with a corresponding color photo with a white background to ensure that the hand is centered in the photo and there is enough white light illumination. The collected images are now in a single-channel grayscale format with a size of 64x64 pixels, and their pixel values ​​have been scaled from the range of 0-255 to the range of 0-1 for fast processing by the CNN algorithm without losing any data. The dataset provides files in two formats: The dataset has been publicly released on the Kaggle dataset webpage for use in research projects to promote future work in the field of sign language recognition. During model training, 1509 images were selected as training sets, 503 as test sets, and 503 as validation sets.

[0023] See also Figure 2In the data preprocessing step, the original image is sampled into 64×64 pixels using a random sliding window sampling method. 60% of the samples in the sample library are used as training samples, 20% as validation samples, and 20% as test samples. In order to increase the diversity and variability of the samples, the model inputs a large amount of labeled training data into the model through normalization, random sampling, and data augmentation.

[0024] Step 2: Establish a DCViT model based on the Vision Transformer model as the basic architecture; wherein, add a deformable convolution module at the front end of the Vision Transformer model to extract features of the input image of the DCViT model, and then input the extracted feature map into the Vision Transformer model.

[0025] See also Figure 3 and Figure 4 The present invention adds a deformable convolution module at the front end of the Vision Transformer model as a feature extraction module of the DCViT model, including an offset prediction layer and a deformable convolution layer. The offset prediction layer uses a convolutional network to predict the offset of each position on the input image. The deformable convolution layer: for each position on the input image, the shape of the standard convolution kernel is adjusted using the corresponding offset to obtain the deformed convolution kernel at the position, and then the deformed convolution kernel is used to perform a convolution operation on the pixels at the position. Finally, the feature map obtained by the deformable convolution layer is input into the Vision Transformer model for gesture recognition and classification.

[0026] The Vision Transformer model includes a Patch Embedding module, a position encoding module, a Transformer encoder module and a classification module; The Patch Embedding module is used to: split the input feature map into small blocks of fixed size (e.g., 16x16 pixels), and flatten each small block into a vector; The position encoding module is used to: encode the position of each small block obtained by segmentation in the feature map before segmentation; The Transformer encoder module is the core part of ViT, which is composed of multiple stacked Transformer encoder layers and processes the small flattened vectors on the position encoding; The classification module includes a classification head, which is used to convert the output of the Transformer encoder module into a classification result.

[0027] In this embodiment, the number of input channels of the deformable convolution module is 3, the number of output channels is 3, the size of the convolution kernel is 3, the padding is 1, the step size is 1, the bias is True, and the modulation is True. The patch size of the Vision Transformer model is 16, the embedding dimension is 768, the number of multi-head attention heads is 12, the number of Transformer encoder layers is 12, the multi-layer perceptron dimension is 3072, and the number of categories is adjusted according to different data sets.

[0028] like Figure 4 As shown in the figure, the number of channels of the gesture image after preprocessing is 3, the size is 64×64, and the batchsize is set to 8; the preprocessed gesture image is used as the input of the deformable convolution module, first passes through the offset prediction layer, and the output after the offset prediction layer is (8×18×64×64); then enters the deformable convolution layer, and the output after the deformable convolution layer is (8×3×64×64). Then the extracted feature map is used as the input of the Vision Transformer, first passes through the Patch Embedding module, and after a series of segmentation and flattening operations, the dimension changes to (8×196×768); then passes through the position encoding module, and the dimension changes to (8×197×768); after position encoding, it is used as the input of the Transformer encoder, and the dimension becomes (8×768) after the operation of the Transformer encoder, and then input into the classification module to obtain a vector with a dimension of (8×num_classes); finally, the classification result is output.

[0029] The advantage of using a pre-trained Vision Transformer in the present invention is that its self-attention mechanism can capture the complex connections between image patches, thereby enhancing its ability to cope with highly similar gestures. Then, the present invention adds a deformable convolution module to its front end, and adjusts the local features of the convolution kernel samples by introducing a learnable offset, thereby improving the model's adaptability to changes in target shape. It helps the visual transformer reduce the computational load while maintaining the information integrity of each channel. By reorganizing the channels to fit the batch dimension and dividing them into multiple sub-feature groups, the optimal distribution of spatial semantic features within each sub-feature group is achieved.

[0030] Step 3: Use the training set data to train the DCViT model to obtain a gesture recognition classifier.

[0031] The computing environment of the embodiment of the present invention is based on NVIDIA 4090 GPU and pytorch deep learning platform. The DCViT model is trained on a computer with 128-core CPU and 192GB memory.

[0032] Reference Figure 2 In the model training phase, all training set data are input into the DCViT model, and the validation set data are used to judge the operation results of the DCViT model under different parameter values. The maximum number of training cycles is set to 200. The batch input size of each training step is 8, that is, 8 samples are input each time to fit the DCViT model. The initial learning rate is 0.001. The Adam optimization method is used to update all parameters of the DCViT model. The above iteration stops when the error is less than the given threshold or the maximum number of iterations is met.

[0033] Step 4: Use a gesture recognition classifier to classify the gesture image to be recognized.

[0034] In the classification stage, the trained DCViT model is used to classify the data to be classified to generate preliminary category prediction data. The prediction data is then input into the classification head (fully connected layer), and finally, the softmax activation function is applied to the output of the classification head to obtain the probability of each gesture category.

[0035] Example 2 This embodiment provides a gesture recognition device based on a DCViT model, including: A preprocessing module is used to: obtain a gesture image data set of known corresponding gesture types, preprocess each gesture image therein and divide it into training sets in proportion; The DCViT model module is built on the Vision Transformer model as the basic architecture. A deformable convolution module is added to the front end of the Vision Transformer model to extract features from the input image of the DCViT model, and then the extracted feature map is input to the Vision Transformer model. A model training module, used to: train the DCViT model using training set data to obtain a gesture recognition classifier; The gesture recognition and classification module is used to: use a gesture recognition classifier to recognize and classify gesture images to be recognized.

[0036] The specific implementation methods and working principles of the above modules of the gesture recognition device of this embodiment are the same as those described in Embodiment 1.

[0037] The above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A gesture recognition method based on the DCViT model, characterized in that: include: Step 1, obtaining a gesture image dataset of known corresponding gesture types, preprocessing each gesture image therein and dividing it in proportion to obtain a training set; Step 2, establishing a DCViT model with the Vision Transformer model as the basic architecture; wherein, a deformable convolution module is added to the front end of the Vision Transformer model to extract features from the input image of the DCViT model, and then the extracted feature map is input into the Vision Transformer model; Step 3, using the training set data to train the DCViT model to obtain a gesture recognition classifier; Step 4: Use a gesture recognition classifier to classify the gesture image to be recognized.

2. The gesture recognition method based on the DCViT model according to claim 1, characterized in that: Step 1 also includes dividing the gesture image set obtained by preprocessing in proportion to obtain a test set, and step 3 also uses the test set to assist in training the DCViT model to obtain a gesture recognition classifier that meets a preset classification accuracy.

3. The gesture recognition method based on the DCViT model according to claim 1, characterized in that: The deformable convolution module includes an offset prediction layer and a deformable convolution layer; the offset prediction layer uses a convolutional network to predict the offset of each position on the input image; the deformable convolution layer: for each position on the input image, the shape of the standard convolution kernel is adjusted using the corresponding offset to obtain the deformed convolution kernel at the position, and then the deformed convolution kernel is used to perform a convolution operation on the pixels at the position.

4. The gesture recognition method based on the DCViT model according to claim 1, characterized in that: The VisionTransformer model includes a Patch Embedding module, a position encoding module, a Transformer encoder module and a classification module; The Patch Embedding module is used to: split the input feature map into small blocks of fixed size, and flatten each small block into a vector; The position encoding module is used to: encode the position of each small block obtained by segmentation in the feature map before segmentation; The Transformer encoder module is composed of multiple stacked Transformer encoder layers, which processes small flattened vectors on each position encoding; The classification module is used to convert the output of the Transformer encoder module into a classification result.

5. The gesture recognition method based on the DCViT model according to claim 4, characterized in that: The Transformer encoder module consists of a normalization layer, a multi-head attention mechanism, a residual connection, and a linear layer.

6. The gesture recognition method based on the DCViT model according to claim 4, characterized in that: The classification module is implemented by an MLP with one hidden layer during pre-training and by a linear layer during fine-tuning.

7. A gesture recognition device based on the DCViT model, characterized in that: include: A preprocessing module is used to: obtain a gesture image data set of known corresponding gesture types, preprocess each gesture image therein and divide it into training sets in proportion; The DCViT model module is built on the Vision Transformer model as the basic architecture. A deformable convolution module is added to the front end of the Vision Transformer model to extract features from the input image of the DCViT model, and then the extracted feature map is input to the Vision Transformer model. A model training module, used to: train the DCViT model using training set data to obtain a gesture recognition classifier; The gesture recognition and classification module is used to: use a gesture recognition classifier to recognize and classify gesture images to be recognized.

8. The gesture recognition device based on the DCViT model according to claim 7, characterized in that: The deformable convolution module includes an offset prediction layer and a deformable convolution layer; the offset prediction layer uses a convolutional network to predict the offset of each position on the input image; the deformable convolution layer: for each position on the input image, the shape of the standard convolution kernel is adjusted using the corresponding offset to obtain the deformed convolution kernel at the position, and then the deformed convolution kernel is used to perform a convolution operation on the pixels at the position.

9. The gesture recognition device based on the DCViT model according to claim 7, characterized in that: The VisionTransformer model includes a Patch Embedding module, a position encoding module, a Transformer encoder module and a classification module; The Patch Embedding module is used to: split the input feature map into small blocks of fixed size, and flatten each small block into a vector; The position encoding module is used to: encode the position of each small block obtained by segmentation in the feature map before segmentation; The Transformer encoder module is composed of multiple stacked Transformer encoder layers, which processes small flattened vectors on each position encoding; The classification module is used to convert the output of the Transformer encoder module into a classification result.

Images