A gesture recognition method and device based on DCViT model

Abstract

The present invention discloses a gesture recognition method and device based on the DCViT model. The method comprises: obtaining a dataset of gesture images with known corresponding gesture types, preprocessing each gesture image, and dividing it proportionally to obtain a training set; establishing a DCViT model based on the Vision Transformer model; adding a deformable convolution module to the front end of the Vision Transformer model to extract features from the DCViT model's input image, and then inputting the extracted feature map into the Vision Transformer model; training the DCViT model using the training set data to obtain a gesture recognition classifier; and using the gesture recognition classifier to identify and classify the gesture images to be identified. The present invention can more accurately and efficiently identify and classify gestures in gesture images.

Description

Technical Field

[0001] The present invention belongs to the technical field of data recognition, and in particular relates to a gesture recognition method and device based on a DCViT model. Background Art

[0002] The main approaches to gesture recognition involve acquiring data from sensor-equipped gloves or vision sensors.

[0003] Although gesture recognition systems based on sensor gloves can provide accurate gesture capture, they also have some obvious disadvantages: (1) Data gloves can be relatively expensive, especially those advanced gloves that integrate high-precision sensors, which may limit their popularity in cost-sensitive applications. (2) Gloves often restrict movement and may cause adverse skin reactions in people with sensitive skin, and the different body sizes of users may also cause discomfort. (3) Glove-based systems may limit the user's natural interaction method because the user needs to wear gloves at all times, which may affect the naturalness and intuitiveness of the interaction. Gesture recognition methods based on visual sensors can overcome the above problems. It only needs a camera to capture the gestures made by humans and does not require other equipment, which greatly reduces the cost of use. In addition, it does not require any equipment to be worn, making it more universal.

[0004] Despite the tremendous success of gesture recognition methods based on visual sensors, several challenging issues remain. The main reasons are as follows: The complex and varied movements of the human hand, with their high degree of freedom, make accurate gesture recognition from images a challenge. Lighting variations, background clutter, occlusions, and the rapid and variable nature of hand movements all affect gesture recognition accuracy and computational speed.

[0005] Currently, most existing gesture recognition technologies rely on hand key points, such as patent publication number CN114898457A, "A Dynamic Gesture Recognition Method and System Based on Hand Key Points and Transformer," and patent publication number , "A Bimanual Gesture Interaction Method, Apparatus, Electronic Device, and Storage Medium." These gesture recognition methods rely too heavily on hand key points, making it difficult to accurately detect key points for complex gestures, reducing recognition accuracy. Summary of the Invention

[0006] The present invention provides a gesture recognition method and device based on the DCViT model, which can more accurately and efficiently recognize and classify gestures in gesture images.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] A gesture recognition method based on the DCViT model, comprising:

[0009] Step 1: Obtain a dataset of gesture images with known corresponding gesture types, pre-process each gesture image, and divide it into training sets according to proportions;

[0010] Step 2: Build a DCViT model based on the Vision Transformer model. Add a deformable convolution module to the front end of the Vision Transformer model to extract features from the DCViT model’s input image, and then input the extracted feature map into the Vision Transformer model.

[0011] Step 3: Use the training set data to train the DCViT model to obtain a gesture recognition classifier;

[0012] Step 4: Use a gesture recognition classifier to identify and classify the gesture image to be recognized.

[0013] Furthermore, step 1 also includes proportionally dividing the gesture image set obtained from the preprocessing to obtain a test set, and step 3 also uses the test set to assist in training the DCViT model to obtain a gesture recognition classifier that meets a preset classification accuracy.

[0014] Furthermore, the deformable convolution module includes an offset prediction layer and a deformable convolution layer; the offset prediction layer uses a convolutional network to predict the offset of each position on the input image; the deformable convolution layer: for each position on the input image, the shape of the standard convolution kernel is adjusted using the corresponding offset to obtain a deformed convolution kernel at that position, and then the deformed convolution kernel is used to perform a convolution operation on the pixel at that position.

[0015] Furthermore, the Vision Transformer model includes a Patch Embedding module, a position encoding module, a Transformer encoder module and a classification module;

[0016] The Patch Embedding module is used to: split the input feature map into small patches of fixed size and flatten each patch into a vector;

[0017] The position encoding module is used to encode the position of each small block obtained by segmentation in the feature map before segmentation;

[0018] The Transformer encoder module is composed of multiple stacked Transformer encoder layers, which processes the small flattened vectors at each position encoding;

[0019] The classification module is used to convert the output of the Transformer encoder module into a classification result.

[0020] Furthermore, the Transformer encoder module consists of a normalization layer, a multi-head attention mechanism, a residual connection and a linear layer.

[0021] Furthermore, the classification module is implemented by an MLP with one hidden layer during pre-training and by a linear layer during fine-tuning.

[0022] A gesture recognition device based on a DCViT model, comprising:

[0023] A preprocessing module is used to obtain a data set of gesture images of known corresponding gesture types, preprocess each gesture image and divide it into training sets according to proportion;

[0024] The DCViT model module is built on the Vision Transformer model. A deformable convolution module is added to the front end of the Vision Transformer model to extract features from the DCViT model's input image. The extracted feature maps are then fed into the Vision Transformer model.

[0025] A model training module is used to train the DCViT model using training set data to obtain a gesture recognition classifier;

[0026] The gesture recognition and classification module is used to: use a gesture recognition classifier to recognize and classify gesture images to be recognized.

[0027] Furthermore, the deformable convolution module includes an offset prediction layer and a deformable convolution layer; the offset prediction layer uses a convolutional network to predict the offset of each position on the input image; the deformable convolution layer: for each position on the input image, the shape of the standard convolution kernel is adjusted using the corresponding offset to obtain a deformed convolution kernel at that position, and then the deformed convolution kernel is used to perform a convolution operation on the pixel at that position.

[0028] Furthermore, the Vision Transformer model includes a Patch Embedding module, a position encoding module, a Transformer encoder module and a classification module;

[0029] The Patch Embedding module is used to: split the input feature map into small patches of fixed size and flatten each patch into a vector;

[0030] The position encoding module is used to encode the position of each small block obtained by segmentation in the feature map before segmentation;

[0031] The Transformer encoder module is composed of multiple stacked Transformer encoder layers, which processes the small flattened vectors at each position encoding;

[0032] The classification module is used to convert the output of the Transformer encoder module into a classification result.

[0033] Beneficial effects

[0034] The gesture recognition method and device based on the DCViT model of the present invention can realize high-precision recognition of gestures in gesture images of various resolutions by constructing and training an image deep learning classification model based on the DCViT model. Specifically,

[0035] For high-resolution images, if the image is directly input into the Vision Transformer model, the model directly divides the entire image into multiple patches, and each patch contains limited information. This may lead to the loss of some detailed information, which in turn causes certain errors in the gesture category output. Therefore, the present invention adds a Deformable Convolution module to the front end of the Vision Transformer model. On the one hand, it is robust to some perturbations of the input image (such as noise, small deformations, etc.). On the other hand, by introducing an offset, it can be adjusted during the training process to adapt to the specific shape and content of the input feature map. The resulting convolution kernel can dynamically adapt to the deformation of the object and effectively solve the information loss problem of the Vision Transformer model, greatly improving the accuracy of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a method for classifying gesture images according to the present invention;

[0037] Figure 2 is a flowchart of a method for classifying gesture images according to an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the DCViT model structure of the present invention;

[0039] Figure 4 It is a further detailed structural diagram of the DCViT model structure of the present invention. DETAILED DESCRIPTION

[0040] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, provides a detailed implementation method and a specific operation process, and further explains the technical solution of the present invention.

[0041] Example 1

[0042] This embodiment provides a gesture recognition method based on the DCViT model. Figure 1 As shown, including:

[0043] Step 1: Obtain a gesture image dataset with known corresponding gesture types, preprocess each gesture image, and divide it into training set data, test set data, and validation set data in a ratio of 6:2:2.

[0044] The gesture image datasets used in this embodiment of the present invention are from the ASL dataset, the NUS hand gesture dataset, and the ASL with digits dataset. The ASL dataset contains 65,774 images, covering 24 different gestures for the letters A to Z. The letter W has the largest number of samples, at 6,221, while the letter F has the smallest number, at 5,235. During model training, images with real-world labeled data were divided into training and validation sets. 39,464 images were selected as the training set, 13,155 as the test set, and 13,155 as the validation set.

[0045] The NUS hand gesture dataset is a more complex dataset, consisting of images of 10 hand gestures captured in natural settings both inside and outside the National University of Singapore. The dataset consists of gestures performed by 40 subjects of diverse ethnicity, ranging in age from 22 to 56. Each subject demonstrated 10 hand gestures five times. The dataset also includes a set of background images without any hand gestures. All images are in RGB format and provided in .jpeg format. During model training, 1,200 images were selected as the training set, 400 as the test set, and 400 as the validation set.

[0046] The ASL with digits dataset focuses on digit gestures in American Sign Language (ASL). Created by Arda Mavi, the dataset aims to facilitate the development of machine learning algorithms capable of recognizing signed digits. The dataset contains 10 different ASL digit gestures performed by 218 volunteers using their right hands. Each gesture is accompanied by a corresponding color photograph with a white background, ensuring the hand is centered in the photograph and adequately illuminated by white light. The collected images are in a single-channel grayscale format of 64x64 pixels, with pixel values ​​scaled from 0-255 to 0-1 to facilitate fast processing by CNN algorithms without data loss. The dataset is available in two file formats: The dataset has been publicly released on the Kaggle datasets page for research projects to facilitate future work in sign language recognition. During model training, 1,509 images were selected as the training set, 503 as the test set, and 503 as the validation set.

[0047] See also Figure 2 During data preprocessing, the original images were sampled to 64×64 pixels using a random sliding window. 60% of the samples in the sample library were used as training samples, 20% as validation samples, and 20% as test samples. To increase sample diversity and variability, the model fed a large amount of labeled training data through normalization, random sampling, and data augmentation.

[0048] Step 2: Establish a DCViT model based on the Vision Transformer model. Add a deformable convolution module to the front end of the Vision Transformer model to extract features from the input image of the DCViT model, and then input the extracted feature map into the Vision Transformer model.

[0049] See also Figure 3 and Figure 4 The present invention adds a deformable convolution module at the front end of the Vision Transformer model as a feature extraction module of the DCViT model, which includes an offset prediction layer and a deformable convolution layer. The offset prediction layer uses a convolutional network to predict the offset of each position on the input image. The deformable convolution layer: for each position on the input image, the shape of the standard convolution kernel is adjusted using the corresponding offset to obtain a deformed convolution kernel at that position, and then the deformed convolution kernel is used to perform a convolution operation on the pixels at that position. Finally, the feature map obtained by the deformable convolution layer is input into the Vision Transformer model for gesture recognition and classification.

[0050] The Vision Transformer model includes a Patch Embedding module, a Position Encoding module, a Transformer Encoder module, and a Classification module;

[0051] The Patch Embedding module is used to: split the input feature map into small patches of fixed size (e.g., 16x16 pixels) and flatten each patch into a vector;

[0052] The position encoding module is used to encode the position of each small block obtained by segmentation in the feature map before segmentation;

[0053] The Transformer encoder module is the core part of ViT, which is composed of multiple stacked Transformer encoder layers and processes the small flattened vectors at each position encoding;

[0054] The classification module includes a classification head, which is used to convert the output of the Transformer encoder module into a classification result.

[0055] In this example, the deformable convolution module has 3 input channels, 3 output channels, a kernel size of 3, padding of 1, stride of 1, bias set to True, and modulation set to True. The Vision Transformer model has a patch size of 16, an embedding dimension of 768, 12 multi-head attention heads, 12 Transformer encoder layers, and a multilayer perceptron dimension of 3072. The number of categories is adjusted based on the dataset.

[0056] like Figure 4 As shown in the figure, the preprocessed gesture image has 3 channels, a size of 64×64, and a batch size of 8. The preprocessed gesture image serves as the input to the deformable convolution module, first passing through the offset prediction layer, with the output being (8×18×64×64). It then enters the deformable convolution layer, with the output being (8×3×64×64). The extracted feature map then serves as the input to the Vision Transformer. It first passes through the Patch Embedding module, where it undergoes a series of splitting and flattening operations, reducing its dimension to (8×196×768). It then passes through the positional encoding module, reducing its dimension to (8×197×768). After positional encoding, it serves as the input to the Transformer encoder, where its dimension becomes (8×768). It is then input to the classification module, resulting in a vector of dimension (8×num_classes). The classification result is finally output.

[0057] The advantage of using a pre-trained Vision Transformer in this paper lies in its self-attention mechanism, which can capture the complex connections between image patches, thereby enhancing its ability to cope with highly similar gestures. The present invention then adds a deformable convolution module to its front end, introducing a learnable offset to adjust the local features of the convolution kernel samples, thereby improving the model's adaptability to changes in target shape. This helps the visual transformer reduce the computational load while maintaining the information integrity of each channel. By reorganizing the channels to fit the batch dimension and dividing them into multiple sub-feature groups, an optimal distribution of spatial semantic features is achieved within each sub-feature group.

[0058] Step 3: Use the training set data to train the DCViT model to obtain a gesture recognition classifier.

[0059] The computing environment of the present invention is based on an NVIDIA 4090 GPU and the PyTorch deep learning platform. The DCViT model was trained on a computer with a 128-core CPU and 192GB of memory.

[0060] Reference Figure 2 During the model training phase, all training data was fed into the DCViT model, and validation data was used to evaluate the DCViT model's performance under different parameter values. The maximum number of training cycles was set to 200. The batch size for each training step was 8, meaning that 8 samples were fed into the DCViT model at a time. The initial learning rate was 0.001. All parameters of the DCViT model were updated using the Adam optimization method. The iterations terminated when the error fell below a given threshold or when the maximum number of iterations was met.

[0061] Step 4: Use a gesture recognition classifier to identify and classify the gesture image to be recognized.

[0062] In the classification phase, the trained DCViT model is used to classify the data to be classified to generate preliminary category prediction data. This prediction data is then input into the classification head (fully connected layer). Finally, the softmax activation function is applied to the output of the classification head to obtain the probability of each gesture category.

[0063] Example 2

[0064] This embodiment provides a gesture recognition device based on the DCViT model, including:

[0065] A preprocessing module is used to obtain a data set of gesture images of known corresponding gesture types, preprocess each gesture image and divide it into training sets according to proportion;

[0066] The DCViT model module is built on the Vision Transformer model. A deformable convolution module is added to the front end of the Vision Transformer model to extract features from the DCViT model's input image. The extracted feature maps are then fed into the Vision Transformer model.

[0067] A model training module is used to train the DCViT model using training set data to obtain a gesture recognition classifier;

[0068] The gesture recognition and classification module is used to: use a gesture recognition classifier to recognize and classify gesture images to be recognized.

[0069] The specific implementation and working principles of the above modules of the gesture recognition device of this embodiment are the same as those described in Example 1.

[0070] The above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed by the present invention, or perform equivalent replacements on some of the technical features thereof. Such modifications, changes or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A gesture recognition method based on the DCViT model, characterized in that: include: Step 1: Obtain a dataset of gesture images with known corresponding gesture types, pre-process each gesture image, and divide it into training sets according to proportions; Step 2: Build a DCViT model based on the Vision Transformer model. Add a deformable convolution module to the front end of the Vision Transformer model to extract features from the DCViT model’s input image, and then input the extracted feature map into the Vision Transformer model. Step 3: Use the training set data to train the DCViT model to obtain a gesture recognition classifier; Step 4: Use a gesture recognition classifier to identify and classify the gesture image to be identified; The deformable convolution module includes an offset prediction layer and a deformable convolution layer; the offset prediction layer uses a convolutional network to predict the offset of each position on the input image; the deformable convolution layer: for each position on the input image, the shape of the standard convolution kernel is adjusted using the corresponding offset to obtain a deformed convolution kernel at that position, and then the deformed convolution kernel is used to perform a convolution operation on the pixel at that position; The Vision Transformer model includes a Patch Embedding module, a Position Encoding module, a Transformer Encoder module, and a Classification module; The Patch Embedding module is used to: split the input feature map into small patches of fixed size and flatten each patch into a vector; The position encoding module is used to encode the position of each small block obtained by segmentation in the feature map before segmentation; The Transformer encoder module is composed of multiple stacked Transformer encoder layers, which processes the small flattened vectors at each position encoding; The classification module is used to convert the output of the Transformer encoder module into a classification result.

2. The gesture recognition method based on the DCViT model according to claim 1, characterized in that: Step 1 also includes proportionally dividing the gesture image set obtained from the preprocessing to obtain a test set. Step 3 also uses the test set to assist in training the DCViT model to obtain a gesture recognition classifier that meets a preset classification accuracy.

3. The gesture recognition method based on the DCViT model according to claim 1, characterized in that: The Transformer encoder module consists of a normalization layer, a multi-head attention mechanism, a residual connection, and a linear layer.

4. The gesture recognition method based on the DCViT model according to claim 1, characterized in that: The classification module is implemented by an MLP with one hidden layer during pre-training and by a linear layer during fine-tuning.

5. A gesture recognition device based on the DCViT model, characterized in that: include: A preprocessing module is used to obtain a data set of gesture images of known corresponding gesture types, preprocess each gesture image and divide it into training sets according to proportion; The DCViT model module is built on the Vision Transformer model. A deformable convolution module is added to the front end of the Vision Transformer model to extract features from the DCViT model's input image. The extracted feature maps are then fed into the Vision Transformer model. A model training module is used to train the DCViT model using training set data to obtain a gesture recognition classifier; The gesture recognition and classification module is used to: use a gesture recognition classifier to recognize and classify gesture images to be recognized; The deformable convolution module includes an offset prediction layer and a deformable convolution layer; the offset prediction layer uses a convolutional network to predict the offset of each position on the input image; the deformable convolution layer: for each position on the input image, the shape of the standard convolution kernel is adjusted using the corresponding offset to obtain a deformed convolution kernel at that position, and then the deformed convolution kernel is used to perform a convolution operation on the pixel at that position; The Vision Transformer model includes a Patch Embedding module, a Position Encoding module, a Transformer Encoder module, and a Classification module; The Patch Embedding module is used to: split the input feature map into small patches of fixed size and flatten each patch into a vector; The position encoding module is used to encode the position of each small block obtained by segmentation in the feature map before segmentation; The Transformer encoder module is composed of multiple stacked Transformer encoder layers, which processes the small flattened vectors at each position encoding; The classification module is used to convert the output of the Transformer encoder module into a classification result.

Images