A millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network

Abstract

The application discloses a millimeter wave radar gesture recognition method based on an adaptive coding VisionTransformer network, which comprises the following steps: collecting a gesture to be classified by using a millimeter wave radar; extracting features from the collected gesture echo data, and synthesizing RGB gesture feature maps according to three channels according to the extracted micro-motion Doppler features and pitch azimuth angle features; expanding the RGB gesture feature maps by using a denoising diffusion implicit model; mixing the fitted images obtained by expansion and the RGB gesture feature maps to form a data set, and dividing the data set into a training data set and a verification data set; constructing an adaptive coding VisionTransformer network model; training the adaptive coding VisionTransformer network model by using the training data set; and inputting the verification data set into the trained adaptive coding VisionTransformer network model for classification. The application adopts the adaptive coding VisionTransformer network model, has low calculation complexity, high recognition accuracy, and is easy to deploy in a low-power state.

Description

Technical Field

[0001] This invention relates to the field of millimeter-wave radar gesture recognition technology, and in particular to a millimeter-wave radar gesture recognition method based on an adaptive coding VisionTransformer network. Background Technology

[0002] Because FMCW radar has relatively mature technology for extracting micro-motion Doppler features and elevation / azimuth angles of moving objects from echo signals, it provides an additional option for gesture recognition. FMCW radar gesture recognition has many traditional methods, which require manually separating feature sets dependent on domain knowledge from the data after feature extraction, and then using appropriate machine learning models, such as SVM (Support Vector Machine), k-nearest neighbors, and random forest, to classify these feature vectors into different categories to achieve gesture recognition.

[0003] However, with the rapid development of deep learning, various derived models have provided more solutions for gesture classification. Deep learning models have also reduced the complexity of gesture feature extraction. Some scholars have used three-channel synthesized gesture feature data for CNN gesture classification, achieving very good results. However, due to the inherent complexity of gestures, using only CNN models is insufficient for gestures with similar features or indistinct features. Therefore, some scholars have proposed using multi-channel gesture feature recognition that integrates channel attention and spatial attention, such as the CNN-enhanced multi-channel synthesized gesture feature recognition based on channel and spatial attention mechanisms proposed by Du et al. (see: Chuan Du et al, “Enhanced Multi-Channel Feature Synthesis for Hand Gesture Recognition Basedon CNN With a Channel and Spatial Attention Mechanism”, IEEE Access, 2020, 8:144610). However, current CNN classification methods all have the following problems.

[0004] First, the whole-image attention approach loses local features of the image, and its performance is not ideal when dealing with images with subtle differences.

[0005] Secondly, the computational requirements of existing CNN algorithms are too high, making them unsuitable for deployments with low computational power.

[0006] Third, it does not take into account the correlation between the information between the pixels in the image itself, and directly using convolution makes the relationship between pixels ignored. Summary of the Invention

[0007] To address the technical problems existing in the background art, this invention proposes a millimeter-wave radar gesture recognition method based on an adaptive coding VisionTransformer network.

[0008] This invention proposes a millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network, comprising:

[0009] Millimeter-wave radar is used to collect the gestures to be classified;

[0010] Feature extraction was performed on the collected gesture echo data to obtain micro-motion Doppler features and pitch azimuth features. The micro-motion Doppler features and pitch azimuth features were then combined into an RGB gesture feature map using three channels.

[0011] The RGB gesture feature map is augmented using a denoising diffusion implicit model to obtain a fitted image;

[0012] The fitted image and RGB gesture feature map are combined to form a dataset, which is then divided into a training dataset and a validation dataset.

[0013] Construct an adaptive encoding Vision Transformer network model;

[0014] The adaptive encoding Vision Transformer network model was trained using the training dataset;

[0015] The validation dataset is input into the trained adaptive encoding Vision Transformer network model for classification, and the classification results are obtained.

[0016] Furthermore, the RGB gesture feature map is augmented using a denoising diffusion implicit model, specifically including:

[0017] A diffusion process and noise prediction model for RGB gesture feature maps are established; the noise prediction model adopts a U-Net neural network structure.

[0018] The RGB gesture feature map is input into the noise prediction model according to the diffusion process, and the noise prediction model is trained according to the diffusion process to obtain a trained denoising diffusion implicit model.

[0019] The RGB gesture feature map is augmented using a trained denoising diffusion implicit model.

[0020] Furthermore, the implicit model of denoising diffusion includes a forward diffusion process and a reverse denoising generation process;

[0021] The forward diffusion process involves continuously adding Gaussian noise to the RGB gesture feature map x0, transforming the original data distribution into a normal distribution, thus obtaining a pure noise image x. t ;

[0022] In the reverse denoising process, a noise prediction model is used to transform the pure noise image x t The original data distribution is quickly restored from the normal distribution to obtain the fitted image.

[0023] Furthermore, the noise prediction model includes a generator and a discriminator; specifically, the RGB gesture feature map is input into the noise prediction model according to the diffusion process, and the noise prediction model is trained according to the diffusion process to obtain a trained denoising diffusion implicit model, which specifically includes:

[0024] Diffusion noise is applied to the RGB gesture feature map to obtain the image after diffusion noise and the variance of its noise components;

[0025] The image after diffusion and noise addition, along with the variance of its noise components, is input into the generator to obtain the predicted image.

[0026] The RGB gesture feature map and the predicted image are input into the discriminator to obtain the quality information of the predicted image;

[0027] Based on the image quality information of the prediction results, the generator and discriminator are optimized to obtain the trained generator and discriminator;

[0028] The trained generator and discriminator are updated into the fast denoising prediction model to obtain the trained denoising diffusion implicit model.

[0029] Furthermore, the quality information of the predicted image includes the kernel starting distance (KID) between the predicted image and the feature map corresponding to the RGB gesture feature map.

[0030] Furthermore, the discriminator includes an input layer, a resize layer, an InceptionV3 module, and a computation module.

[0031] Furthermore, the adaptive coding Vision Transformer network model includes: an image segmentation module, a patch coding module, a Transformer Encoder feature extraction module, and an MLP classification module;

[0032] The image segmentation module is used to divide the image into multiple image patch blocks; the patch encoding module is used to encode each image patch block; each encoded image patch block includes the position information of the image patch block and the information on the relationship between the pixels in the image patch block; the Transformer Encoder feature extraction module is used to extract the feature vector of the encoded image patch block; and the MLP classification module is used to classify based on the extracted feature vector.

[0033] Furthermore, the MLP classification module includes a third normalization layer, a Dropout layer, and three fully connected layers.

[0034] Furthermore, the Transformer Encoder feature extraction module includes eight encoder blocks connected end to end. Each Transformer encoder block includes a first normalization layer, a second normalization layer, a multi-head attention layer, and a multi-layer perceptron.

[0035] The input of the first normalized layer is used as the input of the encoder block. The output of the first normalized layer is connected to the input of the multi-head attention layer, and the input of the first normalized layer is connected to the output of the multi-head attention layer in a skip connection, and is also connected to the input of the second normalized layer. The output of the second normalized layer is connected to the input of the multilayer perceptron, and the output of the second normalized layer is connected to the output of the multilayer perceptron in a skip connection to serve as the input of the encoder block.

[0036] Furthermore, classifying the validation dataset into the trained adaptive encoding Vision Transformer network model also includes:

[0037] The t-SNE algorithm is used to display the classification results.

[0038] This invention proposes a millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network. This method utilizes millimeter-wave radar to collect gestures to be classified, and extracts micro-doppler features and elevation / azimuth features from the collected gesture echo data. These micro-doppler and elevation / azimuth features are then synthesized into an RGB gesture feature map using three channels. The RGB gesture feature map is then expanded using a denoising diffusion implicit model, eliminating the need for subsequent time-consuming and unstable data acquisition processes based on a subset of the data. Furthermore, compared to traditional classification algorithms such as Support Vector Machine (SVM), k-nearest neighbors (k-NN), random forest, and convolutional neural networks, this invention employs an adaptive coding ViT network model, completely abandoning convolutional feature extraction and manual feature vector extraction. This enhances the method's ability to perceive local information, reducing computational complexity, improving recognition accuracy, and facilitating deployment under low-computing-power conditions. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network, as proposed in one embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the structure of the U-net neural network in DDIM according to one embodiment of the present invention.

[0041] Figure 3 This is a diagram illustrating the process of generating an image using DDIM in one embodiment of the present invention.

[0042] Figure 4 This is a curve showing the change of KID during the DDIM image generation process in one embodiment of the present invention.

[0043] Figure 5 This is the original image of a gesture feature diagram in one embodiment of the present invention.

[0044] Figure 6 This is an image of a gesture feature map in one embodiment of the present invention, when the original image is divided into multiple patches.

[0045] Figure 7 This is a schematic diagram of the structure of an adaptive coding Vision Transformer network model in one embodiment of the present invention.

[0046] Figure 8 This is a curve showing the change of the loss function during the training process of an adaptive coding Vision Transformer network model in one embodiment of the present invention.

[0047] Figure 9 This is a confusion matrix obtained by the trained adaptive coding Vision Transformer network model after recognizing the dataset in one embodiment of the present invention.

[0048] Figure 10 This is a diagram showing the original data and the data distribution plotted using the t-SNE algorithm after training an adaptive coding Vision Transformer network model, according to one embodiment of the present invention.

[0049] Figure 11 This is a schematic diagram of a CNN gesture recognition model based on channel and spatial attention mechanisms in the existing technology.

[0050] Figure 12 This is a curve showing the change in loss value during the training process of a CNN model based on channel and spatial attention mechanisms in the existing technology.

[0051] Figure 13 This is the confusion matrix obtained by a CNN model based on trained channel and spatial attention mechanisms after recognizing the dataset in the prior art.

[0052] Figure 14 The data distribution is plotted using the t-SNE algorithm for the original data and the data after training with a CNN model based on channel and spatial attention mechanisms in the existing technology.

[0053] Figure 15 These are 10 types of gesture feature map samples. Detailed Implementation

[0054] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0055] Reference Figure 1 This invention proposes a millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network, comprising:

[0056] S1. Use millimeter-wave radar to collect the gestures to be classified;

[0057] S2. Extract features from the collected gesture echo data to obtain micro-motion Doppler features and pitch azimuth features, and synthesize the micro-motion Doppler features and pitch azimuth features into an RGB gesture feature map using three channels;

[0058] S3. The RGB gesture feature map is expanded using denoising diffusion implicit models (DDIM) to obtain a fitted image;

[0059] S4. Combine the fitted image and RGB gesture feature map to form a dataset, and divide the dataset into a training dataset and a validation dataset;

[0060] S5. Construct an adaptive coding Vision Transformer network model;

[0061] S6. Train the adaptive coding Vision Transformer network model using the training dataset;

[0062] S7. Input the validation dataset into the trained adaptive encoding Vision Transformer network model for classification and obtain the classification results.

[0063] This invention utilizes millimeter-wave radar to collect gestures to be classified, and extracts features from the collected gesture echo data to obtain micro-motion Doppler features and elevation azimuth features. These micro-motion Doppler features and elevation azimuth features are then synthesized into an RGB gesture feature map using three channels. This RGB gesture feature map is then expanded using a denoising diffusion implicit model, eliminating the need for subsequent time-consuming and unstable data acquisition processes based on a subset of the data. Furthermore, compared to traditional classification algorithms such as Support Vector Machine (SVM), k-nearest neighbor (k-NN), random forest, and convolutional neural networks, this invention employs an adaptive coding ViT network model, completely abandoning convolutional feature extraction and manual feature vector extraction. This enhances its ability to perceive local information, reducing computational complexity, improving recognition accuracy, and facilitating deployment under low-computing-power conditions.

[0064] In this embodiment, in S1, the hand gestures to be classified are collected using millimeter-wave radar, specifically including:

[0065] The IWR1642 millimeter-wave radar was used to collect data on the gestures to be identified in multiple rounds.

[0066] In a further embodiment, the gestures to be classified are collected using millimeter-wave radar, specifically including:

[0067] The hand gestures to be classified are collected using millimeter-wave radar emitted by a radar system with dual transmitters and four receivers.

[0068] The radar system's receivers are Rx1, Rx2, Rx3, and Rx4. Rx1 and Rx2 are placed vertically, and due to the wavelength distance between them, the elevation angle measurement degree of freedom is achieved. Rx2, Rx3, and Rx4 are arranged in parallel at half-wavelength intervals, achieving the azimuth angle measurement degree of freedom. Through the special arrangement of the transmitting and receiving antennas, the receiving antenna has degrees of freedom in both azimuth and elevation angles, enabling the measurement of the azimuth and elevation angles of hand gestures at specific times. This application employs Short-Time Fast Fourier Transform (FFT) to obtain the spectrum of the hand gestures for micro-Doppler feature analysis. The FFT window size and the time step of the non-overlapping samples are set to 256 ms and 1 ms, respectively.

[0069] In S2, after synthesizing the micro-motion Doppler features and pitch azimuth features into an RGB gesture feature map using three channels, it also includes classifying the RGB gesture feature map to facilitate subsequent DDIM training.

[0070] In this embodiment, in S3, the RGB gesture feature map is expanded using a denoising diffusion implicit model, specifically including:

[0071] S31. Establish a diffusion process and noise prediction model for the RGB gesture feature map; wherein, the noise prediction model adopts a U-Net neural network structure, such as... Figure 2 As shown;

[0072] S32. Input the RGB gesture feature map into the noise prediction model according to the diffusion process, and train the noise prediction model according to the diffusion process to obtain a trained denoising diffusion implicit model.

[0073] S33. Expand the RGB gesture feature map using the trained denoising diffusion implicit model.

[0074] In the process of expanding the RGB gesture feature map, the denoising diffusion implicit model uses a non-Markov chain process and U-Net to generate nearly identical fitted images on the basis of the original dataset to expand the dataset, thereby increasing the training amount and alleviating the problem of large intra-class sample variance caused by the difference in the user's gesture actions, so that the subsequent network can learn gesture features in various situations.

[0075] In this embodiment, in S3, the denoising diffusion implicit model includes a forward diffusion process and a reverse denoising generation process;

[0076] The forward diffusion process involves continuously adding Gaussian noise to the RGB gesture feature map x0, transforming the original data distribution into a normal distribution, thus obtaining a pure noise image x. t ;

[0077] In the reverse denoising process, a noise prediction model is used to transform the pure noise image x t The original data distribution is quickly restored from the normal distribution to obtain the fitted image.

[0078] Specifically, in the inverse denoising generation process, the recursive model is as follows:

[0079]

[0080] In the formula, ε θ Let represent the noise prediction model, where σ is a real vector used as an index, and α represents the total number of noise-adding steps. t :=1-β t ,β t The parameter values ​​for adding noise at time t are set manually, 0 < β1 < ... < β t-1 <β t <...<β T <1,x t This represents a given noisy observation.

[0081] Among them, the noise prediction model is T represents the total number of noise-adding steps.

[0082] Specifically, the U-Net neural network consists of four downsampling layers and four upsampling layers, with skip connections added between layers with the same resolution. Based on this, the diffusion model can be viewed as a bottleneck-free denoising autoencoder. Furthermore, the U-Net neural network structure introduces a residual structure to facilitate network propagation and channel expansion / reduction, thereby preventing gradient vanishing and gradient explosion.

[0083] Specifically, during the training of the denoising diffusion implicit model, Adam W is set as the optimization algorithm for the entire network, and the learning rate δ is set to 10. -3 Numerical stability constant ε r Set to 10 -4 The parameter value β for adding noise at time t t Set to The trigonometric function values ​​are calculated on the input image, which is 64×64. Finally, the Tensorflow framework is used to automatically calculate the gradients of each layer and backpropagate to update the learnable parameters.

[0084] In a further embodiment, the noise prediction model includes a generator and a discriminator; the RGB gesture feature map is input into the noise prediction model according to the diffusion process, and the noise prediction model is trained according to the diffusion process to obtain a trained denoising diffusion implicit model, specifically including:

[0085] Diffusion noise is applied to the RGB gesture feature map to obtain the image after diffusion noise and the variance of its noise components;

[0086] The image after diffusion and noise addition, along with the variance of its noise components, is input into the generator to obtain the predicted image.

[0087] The RGB gesture feature map and the predicted image are input into the discriminator to obtain the quality information of the predicted image;

[0088] The generator and discriminator are optimized based on the image quality information of the prediction results to obtain a trained generator and discriminator.

[0089] The trained generator and discriminator are updated into the fast denoising prediction model to obtain the trained denoising diffusion implicit model.

[0090] This setup improves the generator's generation capability and the discriminator's discrimination capability, enabling the trained denoising diffusion implicit model to generate nearly identical fitted images to expand the dataset based on the original dataset.

[0091] In order to enable the denoising diffusion implicit model to generate nearly identical fitted images to expand the dataset based on the original dataset, in a further embodiment, the quality information of the predicted image includes the kernel starting distance (KID) between the predicted image and the feature map corresponding to the RGB gesture feature map.

[0092] Specifically, for a noise map and a generated image of batch size (64 here), the polynomial kernels relative to themselves and each other are calculated using the polynomial kernel formula. Then, the average kernel of each polynomial kernel with respect to batch size is taken and added together to obtain the final KID value.

[0093] It's important to know that KID, as an image evaluation standard, should ideally be as small as possible. In the best-case scenario, KID=0 will result in an image that is almost identical to the original image.

[0094] Specifically, in S324, during the process of optimizing the generator and discriminator based on the image quality information of the predicted result image, the generation of the predicted result image is stopped when the KID of the predicted result image drops to a certain level.

[0095] In a further embodiment, the discriminator includes an input layer, a resize layer, an InceptionV3 module, and an image quality evaluation module. The resize layer resizes the image output from the input layer; the InceptionV3 module extracts features from the resized image; and the calculation module performs KID calculation based on the extracted features to obtain the quality information of the predicted image.

[0096] Specifically, the resize layer resizes the image output from the input layer to 75×75 pixels to fit the InceptionV3 module. Specifically, the InceptionV3 module preloads weights from ImageNet during feature extraction.

[0097] In one specific embodiment, in S4, the dataset contains 1600 images, and the ratio of the training set to the validation set is 3:1.

[0098] like Figure 7 As shown, in this embodiment, the adaptive coding Vision Transformer network model includes: an image segmentation module, a patch coding module, a Transformer Encoder feature extraction module, and an MLP classification module.

[0099] The image segmentation module is used to divide an image into multiple image patch blocks, such as... Figure 5 and Figure 6As shown; the patch encoding module is used to encode each image patch block; each encoded image patch block includes the position information of the image patch block and the information on the relationship between the pixels in the image patch block; the Transformer Encoder feature extraction module is used to extract the feature vector of the encoded image patch block, and the MLP classification module is used to classify according to the extracted feature vector.

[0100] With this configuration, the adaptive coding Vision Transformer network model directly enhances its ability to perceive local information through adaptive coding, thereby reducing the computational complexity of this invention, improving recognition accuracy, and making it easy to deploy under low computing power conditions.

[0101] Specifically, the position information of the image patch can be directly encoded using embedding. The encoding of the relationships between pixels within the patch is learned and output by a multilayer perceptron, which learns these relationships after the patch is flattened. Unlike linear encoding, this method significantly enhances the adaptive capability of the encoding. Specifically, the final encoding of the image patch is obtained by adding and fusing the patch's position encoding and pixel encoding.

[0102] like Figure 7 As shown, in a further embodiment, the Transformer Encoder feature extraction module includes eight encoder blocks connected end to end. Each Transformer encoder block includes a first normalization layer, a second normalization layer, a multi-head attention layer, and a multilayer perceptron.

[0103] Specifically, the input of the encoder block is used as the input of the first normalization layer. The output of the first normalization layer is connected to the input of the multi-head attention layer, and the input of the first normalization layer is connected to the output of the multi-head attention layer in a skip connection, and is also connected to the input of the second normalization layer. The output of the second normalization layer is connected to the input of the multilayer perceptron, and the output of the second normalization layer is connected to the output of the multilayer perceptron in a skip connection, serving as the input of the encoder block to prevent gradient explosion or gradient vanishing.

[0104] Therefore, compared with traditional classification algorithms such as SVM, k-nearest neighbor, random forest, and convolutional neural networks, the adaptive coding ViT network completely abandons the methods of extracting features through convolution and manually extracting feature vectors. It directly enhances its ability to perceive local information through adaptive coding and multi-head attention, thereby reducing the computational complexity of this invention, improving the recognition accuracy, and making it easy to deploy under low computing power conditions.

[0105] In a further embodiment, the multilayer perceptron includes two fully connected layers. Specifically, the first fully connected layer of the multilayer perceptron has 128 units and the activation function is GELU; the second fully connected layer has 64 units and the activation function is GELU.

[0106] In a further embodiment, the MLP classifier includes a third normalization layer, a Dropout layer, and three fully connected layers.

[0107] Specifically, the output of the Transformer encoder serves as the input to the third normalization layer. The output of the third normalization layer is flattened and fed into the Dropout layer. The output of the Dropout layer is then fed into the three fully connected layers. In these three fully connected layers, the first fully connected layer has 2048 units with GELU activation; the second fully connected layer has 1024 units with GELU activation; and the third fully connected layer has 10 units to classify 10 types of gesture data.

[0108] In this embodiment, in S6, the adaptive encoding Vision Transformer network model is trained using the training dataset, specifically including:

[0109] The training dataset is input into the constructed adaptive coding Vision Transformer network model, and the adaptive coding Vision Transformer network model is trained until convergence using cross-entropy as the loss function, resulting in the trained adaptive coding Vision Transformer network model.

[0110] Specifically, the loss function optimization method for the adaptive encoding Vision Transformer network model is AdamW, with a learning rate of 0.001 and a weight decay coefficient of 0.0001. Gradient calculations are automatically performed using Tensorflow to learn the various parameters in the network.

[0111] In this embodiment, in S7, the validation dataset is input into the trained adaptive encoding VisionTransformer network model for classification, specifically including:

[0112] The validation dataset is input into the trained adaptive encoding Vision Transformer network model for classification, and the classification results are obtained.

[0113] The t-SNE algorithm is used to display the classification results for easy observation.

[0114] The effects of this invention can be further illustrated by the following simulations:

[0115] To verify the superiority of this invention in millimeter-wave radar gesture recognition, a series of experiments were conducted to compare this invention with traditional methods. Since the amount of data before and after DDIM augmentation differs greatly, the verification results are highly random, so the data volume difference was not compared. Only the recognition results of the subsequent adaptive coding Vision Transformer (ViT) network model and CNN network were compared.

[0116] 1. Simulation conditions

[0117] The parameters of the test hardware platform are shown in Table 1, the parameters of the software platform are shown in Table 2, and the parameters of the radar platform are shown in Table 3.

[0118] Table 1 Hardware Platform Parameters

[0119] Memory 16 GB Graphics card model NVIDIA GeForce GTX 1660 Ti Graphics card memory 6GB

[0120] Table 2 Software Platform Parameters

[0121] compiler PyCharm 2021.3.1 CUDA version 11.2.0 Tensorflow version 2.10 Keras version 2.10

[0122] Table 3 Hard Radar Platform Parameters

[0123] Chirping repeat time 400μs Chirp count per frame 128 Distance resolution ~4cm Speed ​​resolution 0.032m / s

[0124] 2. Simulation Method

[0125] (1) Existing CNN gesture recognition networks based on channel and spatial attention mechanisms recognize and classify gesture feature maps;

[0126] (2) The method described in this invention is a millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network.

[0127] 3. Simulation content and simulation results

[0128] The radar used in this application is the IWR1642 radar operating at 77 GHz, with an average output power of 12 dBm. This radar can detect radial velocities ranging from 2.6 cm / s to 2.6 m / s, and has an antenna beamwidth of 120 degrees, making it suitable for gesture measurement. Hand movements are measured within the main lobe of the radar antenna, with an average distance of approximately 30 cm from the main lobe to the radar.

[0129] The 10 gestures to be recognized in this application are (1) swiping from left to right, (2) swiping from right to left, (3) swiping from bottom left to top right, (4) swiping from bottom right to top left, (5) swiping forward, (6) swiping backward, (7) swaying left and right, (8) double-tapping with fingers, (9) clenching and opening a fist, and (10) snapping fingers. The corresponding gesture feature diagrams are shown below. Figure 15 .

[0130] Simulation Experiment 1 uses a CNN gesture recognition network based on channel and spatial attention mechanisms to classify gesture feature maps. The confusion matrix of the classification results is shown below. Figure 13 As shown, the classification accuracy of each gesture can be obtained from the confusion matrix, and the overall classification accuracy is 91.00%.

[0131] Simulation Experiment 2 uses the adaptive coding ViT network model from this invention to recognize and classify gesture feature maps. The confusion matrix of the classification result is as follows: Figure 9 As shown, the classification accuracy of each gesture can be obtained from the confusion matrix, with an overall classification accuracy of 97.25%.

[0132] The FFT window size and the time step for non-overlapping samples were set to 256ms and 1ms, respectively. During image generation, the learning rate for the DDIM denoising diffusion implicit model was set to 0.001, the batch size to 64, and the original data was repeated 40 times. In each training epoch, a validation set of equal size as the training set was randomly selected for validation. The KID (Knowledge Identifier) ​​was calculated between the generated feature images and the feature images in the validation set. The KID was calculated and displayed with each iteration using Keras' callback method, facilitating monitoring of image quality changes throughout the process and allowing training to be interrupted when image quality reached a certain level. The total number of training iterations was set to 70, and the training time for the entire dataset was approximately 1 hour. The image generation process is as follows: Figure 3 As shown, the decrease in KID is as follows Figure 4 As shown, a smaller KID indicates higher image quality. Finally, nearly 1500 gesture feature maps were generated using the trained network weights.

[0133] Compared with existing CNN-enhanced multi-channel synthetic gesture feature recognition based on channel and spatial attention mechanisms, this invention not only significantly improves the overall accuracy, but also greatly improves the classification accuracy of gesture feature maps with indistinct or similar features. This verifies that the invention can extract more comprehensive and discriminative features that reflect the properties of gesture feature images, which is beneficial to improving the classification accuracy of gesture feature images.

[0134] After dimensionality reduction and visualization of the original data and training results using the t-SNE algorithm, the hand gesture recognition network based on channel and spatial attention mechanisms is used to classify the hand gesture feature maps. The results are as follows: Figure 14 As shown, the results of recognizing and classifying gesture feature maps using the adaptive coding ViT network model in this invention are as follows: Figure 10As shown, the model in this invention has better generalization ability and stronger robustness, which is beneficial for recognizing and classifying data that has never been seen before. In contrast, the CNN gesture recognition network based on channel and spatial attention mechanisms has too concentrated results in recognizing and classifying gesture feature maps.

[0135] Compare again Figure 8 and Figure 12 It can be seen that the present invention also has excellent stability of gradient descent during training, which can reduce human intervention during training.

[0136] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network, characterized in that, include: Millimeter-wave radar is used to collect the gestures to be classified; Feature extraction was performed on the collected gesture echo data to obtain micro-motion Doppler features and pitch azimuth features. The micro-motion Doppler features and pitch azimuth features were then combined into an RGB gesture feature map using three channels. The RGB gesture feature map is augmented using a denoising diffusion implicit model to obtain a fitted image; The fitted image and RGB gesture feature map are combined to form a dataset, which is then divided into a training dataset and a validation dataset. An adaptive encoding Vision Transformer network model is constructed, comprising: an image segmentation module, a patch encoding module, a Transformer Encoder feature extraction module, and an MLP classification module. The image segmentation module divides the image into multiple image patches. The patch encoding module encodes each image patch. Each encoded image patch includes its position information and the relationships between pixels within the patch. The position information is encoded using embedding, and the relationships between pixels are learned and output by a multilayer perceptron. The Transformer Encoder feature extraction module extracts feature vectors from the encoded image patches, and the MLP classification module performs classification based on the extracted feature vectors. The adaptive encoding Vision Transformer network model was trained using the training dataset; The validation dataset is input into the trained adaptive encoding Vision Transformer network model for classification, and the classification results are obtained.

2. The millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network according to claim 1, characterized in that, The RGB gesture feature map is augmented using a denoising diffusion implicit model, specifically including: A diffusion process and noise prediction model for RGB gesture feature maps are established; the noise prediction model adopts a U-Net neural network structure. The RGB gesture feature map is input into the noise prediction model according to the diffusion process, and the noise prediction model is trained according to the diffusion process to obtain a trained denoising diffusion implicit model. The RGB gesture feature map is augmented using a trained denoising diffusion implicit model.

3. The millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network according to claim 2, characterized in that, The implicit model of denoising diffusion includes a forward diffusion process and a reverse denoising generation process; The forward diffusion process involves continuously processing the RGB gesture feature map. Adding Gaussian noise transforms the original data distribution into a normal distribution, resulting in a purely noisy image. ; In the reverse denoising process, a noise prediction model is used to transform the pure noise image. The original data distribution is quickly restored from the normal distribution to obtain the fitted image.

4. The millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network according to claim 2, characterized in that, The noise prediction model includes a generator and a discriminator. Specifically, the RGB gesture feature map is input into the noise prediction model according to the diffusion process, and the model is trained according to this process to obtain a trained denoising diffusion implicit model, which includes: Diffusion noise is applied to the RGB gesture feature map to obtain the image after diffusion noise and the variance of its noise components; The image after diffusion and noise addition, along with the variance of its noise components, is input into the generator to obtain the predicted image. The RGB gesture feature map and the predicted image are input into the discriminator to obtain the quality information of the predicted image; Based on the image quality information of the prediction results, the generator and discriminator are optimized to obtain a trained generator and discriminator; The trained generator and discriminator are updated into the fast denoising prediction model to obtain the trained denoising diffusion implicit model.

5. The millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network according to claim 4, characterized in that, The quality information of the predicted image includes the kernel starting distance (KID) between the predicted image and the feature map corresponding to the RGB gesture feature map.

6. The millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network according to claim 5, characterized in that, The discriminator includes an input layer, a resize layer, an InceptionV3 module, and a computation module.

7. The millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network according to claim 1, characterized in that, The MLP classification module includes a third normalization layer, a Dropout layer, and three fully connected layers.

8. The millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network according to claim 1, characterized in that, The Transformer Encoder feature extraction module includes eight encoder blocks connected end to end. Each Transformer encoder block includes a first normalization layer, a second normalization layer, a multi-head attention layer, and a multilayer perceptron. The input of the first normalized layer is used as the input of the encoder block. The output of the first normalized layer is connected to the input of the multi-head attention layer, and the input of the first normalized layer is connected to the output of the multi-head attention layer in a skip connection, and is also connected to the input of the second normalized layer. The output of the second normalized layer is connected to the input of the multilayer perceptron, and the output of the second normalized layer is connected to the output of the multilayer perceptron in a skip connection to serve as the input of the encoder block.

9. The millimeter-wave radar gesture recognition method based on an adaptive coding Vision Transformer network according to claim 1, characterized in that, The validation dataset is input into the trained adaptive encoding Vision Transformer network model for classification, and this also includes: The t-SNE algorithm is used to display the classification results.

Images