A millimeter wave radar dynamic gesture recognition method applied to an interference environment

Abstract

This invention discloses a method for dynamic gesture recognition using millimeter-wave radar in interference environments. The method includes: reconstructing the raw echo data from the millimeter-wave radar into a three-dimensional data block and filtering out static interference using the mean cancellation method; extracting the distance and Doppler information of the dynamic gesture using a two-dimensional fast Fourier transform to obtain the distance-time map and Doppler-time map of the dynamic gesture, and improving the echo signal-to-noise ratio through coherent accumulation; constructing a CNN_Block module and using the two-dimensional convolution in the CNN_Block module to extract local features of the dynamic gesture; constructing a Transformer module; stacking multiple Transformer modules to fully extract deeper and more effective features of the dynamic gesture; using the Transformer module to enhance global attention to gesture features; using the multi-head attention mechanism in the Transformer module to suppress random dynamic interference in the environment; and constructing an FC_Block using a global average pooling layer and a fully connected layer to output a gesture label. This invention achieves gesture recognition in interference environments and improves gesture recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the field of gesture recognition and millimeter-wave radar, and relates to radar signal processing and gesture recognition technology, specifically to a dynamic gesture recognition method for millimeter-wave radar applied in interference environments. Background Technology

[0002] Dynamic gesture control, as a non-contact human-computer interaction method, has broad application prospects in various fields such as the Industrial Internet of Things, smart homes, automotive driving, remote surgery, VR games, and sign language translation. The prerequisite for dynamic gesture control is the accurate recognition of various predetermined gestures. Currently, common sensors used for gesture recognition include optical cameras and electromyography (EMG) sensors. However, these sensors have many limitations. Optical sensors require good lighting conditions and are prone to exposing personal privacy. EMG sensors need to be worn on the user's arm, resulting in a poor user experience, and the EMG signals vary from user to user, lacking universality.

[0003] In recent years, with the development of integrated radio frequency front-ends for millimeter-wave radar, its application potential in gesture recognition has attracted much attention from researchers. Millimeter-wave radar is unaffected by lighting conditions, is small in size, consumes little power, and has high range and angular resolution, enabling it to acquire more subtle gesture information. The most typical example is Google's Project Soli. At the 2016 Google I / O conference, Google initially demonstrated near-range gesture recognition using 60GHz millimeter-wave radar. Subsequently, many companies and universities have conducted research on millimeter-wave radar gesture recognition methods.

[0004] Due to the prevalence of deep learning, artificial neural networks are used as classifiers for millimeter-wave radar gesture recognition. Based on the representation of the input data of the neural network, artificial neural network classifiers used for millimeter-wave radar gesture recognition can be divided into the following categories:

[0005] (1) One-dimensional Neural Network: This method directly uses the raw radar echo as the network input for classification and recognition. For example, the paper "J.Zhu,H.Chen and W.Ye,"A Hybrid CNN–LSTM Network for the Classification of Human Activities Based on Micro-Doppler Radar,"IEEE Access,vol.8,pp.24713-24720,2020,doi:10.1109 / ACCESS.2020.2971064," proposes a deep learning network composed of a one-dimensional convolutional neural network and a long short-term memory (LSTM). The spectrogram is regarded as a one-dimensional time series with multiple channels and fed into the network. The one-dimensional convolution is used to extract the temporal features of adjacent frames in the time dimension and save the temporal information of the spectrum. Then, LSTM is used to process the global temporal information. The paper "W. Ye, H. Chen and B. Li, 'Using an End-to-End Convolutional Network on RadarSignal for Human Activity Classification, IEEE Sensors Journal, vol. 19, no. 24, pp. 12244-12252, 15 Dec. 15, 2019, doi: 10.1109 / JSEN.2019.2938997'" proposes an end-to-end deep learning network that takes a one-dimensional radar signal as input and uses two one-dimensional convolutional layers instead of a short-time Fourier transform to obtain the radar signal representation. The biggest advantage of this type of method is its small parameter count and ability to achieve a certain level of classification performance. However, the one-dimensional data representation cannot be filtered out for interference by radar signal processing algorithms, thus limiting its application scenarios.

[0006] (2) Two-dimensional neural network: The input of a two-dimensional neural network is two-dimensional image data. Radar signal processing algorithms are needed to process the echo data into a single-channel grayscale image or three-channel image data. Commonly used radar signal processing algorithms include Fast Fourier Transform (FFT) and Multiple Signal Classification (MUSIC). The literature “S. Skaria, A. Al-Hourani, M. Lech and RJ Evans, Hand-Gesture Recognition Using Two-Antenna Doppler Radar With DeepConvolutional Neural Networks,” IEEE Sensors Journal, vol.19, no.8, pp.3041-3048, 15 April 15, 2019, doi:10.1109 / JSEN.2019.2892073., uses Short-Time Discrete Fourier Transform to convert the original time-domain signal into a spectrum. Gesture representation is obtained by combining time-domain and frequency-domain information. Then, a three-layer deep convolutional neural network is constructed to extract and classify different gesture features. The paper "T. Sakamoto, X. Gao, E. Yavari, A. Rahman, O. Boric-Lubecke and VMLubecke, 'Hand Gesture Recognition Using a Radar EchoI–Q Plot and a Convolutional Neural Network,' IEEE Sensors Letters, vol.2, no.3, pp.1-4, Sept.2018, Artno.7000904, doi:10.1109 / LSENS.2018.2866371" proposes a gesture recognition technique using a convolutional neural network. This method converts radar echoes into low-resolution two-dimensional images, and then inputs these images into a two-dimensional convolutional neural network for classification and recognition. This type of method has good recognition performance, but it is mostly applied in laboratory environments, and its generalization ability in interference environments needs to be tested.

[0007] (3) Three-dimensional neural network: The input of a three-dimensional neural network is three-dimensional data similar to video, and it has also been applied in the field of dynamic gesture recognition of millimeter-wave radar. When using a three-dimensional neural network for classification training, radar echoes are often processed into three-dimensional data containing spatial and temporal information, such as range-Doppler, range-azimuth, range-elevation, etc. The paper "L.Gan,Y.Liu,Y.Li,R.Zhang,L.Huang and C.Shi,"Gesture Recognition System Using 24GHz FMCW Radar Sensor Realized on Real-Time Edge Computing Platform,"IEEE Sensors Journal,2022,vol.22,8904-8914.doi:10.1109 / JSEN.2022.3163449." proposes a radar gesture recognition system based on a real-time edge computing platform. It utilizes 24GHz radar to acquire the echo data of gestures, then extracts the distance-Doppler information of the gestures, and inputs it into a 3DCNN-LSTM for gesture classification, ultimately achieving a recognition accuracy of 95.9%. The paper "X. Shen, H. Zheng, X. Feng and J. Hu," ML-HGR-Net: AMeta-Learning Network for FMCW Radar Based Hand Gesture Recognition," IEEE Sensors Journal, vol. 22, no. 11, pp. 10808-10817, 1 June 1, 2022, doi: 10.1109 / JSEN.2022.3169231," addresses the problem of gesture recognition with limited training data. It proposes a meta-learning network based on distance-Doppler features, using a 3D convolutional neural network framework to perform classification tasks with relatively little training data. While this type of method can extract more gesture features using 3D convolution, the large number of parameters in 3D convolutional networks makes it difficult to apply to IoT scenarios.

[0008] Among the deep learning classification networks mentioned above, one-dimensional neural networks have fewer parameters than two-dimensional and three-dimensional neural networks. However, one-dimensional data loses phase information, resulting in incomplete feature extraction. Three-dimensional convolutions, compared to the other two types, can extract the most diverse features and offer better feature extraction, but they have the largest number of parameters and poor portability. Two-dimensional input data can be filtered out for interference using radar signal processing algorithms, and it can extract as many features as possible, with a moderate number of parameters, making it suitable for IoT scenarios.

[0009] However, in practical applications, random dynamic interference often exists, such as people walking, waving, or throwing objects between the radar and the gesture. The echoes of these interferences are coupled with the echoes of the valid gestures, and it is difficult to filter them out using conventional radar signal processing algorithms, which will significantly reduce the accuracy of gesture recognition.

[0010] Patent application number "2021106380061" provides a technical solution for "Dynamic Gesture Recognition Method for Millimeter-Wave Radar under Random Interference Scenarios." This solution first captures gesture information using millimeter-wave radar, analyzes the original radar echo sequence, and constructs input data blocks according to the temporal structure of the original radar echo. Second, a convolutional neural network module is built using a wrapper based on a temporal distribution layer, one-dimensional convolutional and global pooling layers, and an Inception V3 network structure. The one-dimensional temporal convolutional neural network of the convolutional neural network module extracts feature information from the radar echo. An inter-frame self-attention mechanism is used to assign weights to the extracted features, obtaining the inter-frame correlation of the sequence and suppressing random interference. Finally, a global average pooling layer and a fully connected layer are used to nonlinearly transform the previously extracted features and map them to a label set, outputting the recognition result. This solution uses millimeter-wave radar original echo data and a one-dimensional convolutional neural network, respectively. Although the number of network parameters is reduced, the lack of phase information in the one-dimensional data leads to incomplete feature extraction. Furthermore, the large amount of raw radar echo data makes the model less portable on embedded devices. Summary of the Invention

[0011] Purpose of the invention: In order to overcome the shortcomings of the existing technology, a dynamic gesture recognition method for millimeter-wave radar applied in interference environment is provided, which realizes gesture recognition in interference environment and improves gesture recognition accuracy.

[0012] Technical Solution: To achieve the above objectives, this invention provides a method for dynamic gesture recognition using millimeter-wave radar in interference environments, comprising the following steps:

[0013] S1: Reassemble the raw echo data of the millimeter-wave radar into three-dimensional data blocks, and use the mean cancellation method to filter out static interference;

[0014] S2: Two-dimensional fast Fourier transform is used to extract the distance and Doppler information of dynamic gestures, and the range-time map (RTM) and doppler-time map (DTM) of dynamic gestures are obtained. The echo signal-to-noise ratio is improved by coherent accumulation.

[0015] S3: Input the distance-time map and Doppler-time map of the dynamic gesture into the constructed CNN_Block module, and use the two-dimensional convolution in the CNN_Block module to extract the local features of the dynamic gesture;

[0016] S4: Construct the Transformer module using layer normalization, multi-head attention mechanism and MLP;

[0017] S5: By stacking multiple Transformer modules, deeper gesture features of dynamic gestures can be fully extracted;

[0018] S6: Enhance global attention to gesture features using the Transformer module;

[0019] S7: Utilize the multi-head attention mechanism in the Transformer module to suppress random dynamic disturbances in the environment;

[0020] S8: Use a global average pooling layer and a fully connected layer to construct an FC_Block, and output gesture labels through the FC_Block.

[0021] Furthermore, in step S1, since the original radar echo data acquired by the millimeter-wave radar acquisition device is only displayed as a one-dimensional data sequence, which is inconvenient for radar signal processing operations, the one-dimensional data is reorganized into a three-dimensional data block, which adopts the Chirps×Samples×Frames format.

[0022] Furthermore, the method for filtering static interference using the mean cancellation method in step S1 involves performing two-pulse cancellation on the 1D-FFT result, specifically expressed as follows:

[0023] Y(N)=X(N)-X(N-1),N=2,3,4,...

[0024] Where X(N) represents the Nth pulse.

[0025] Further, the method for obtaining the range-time map (RTM) and the doppler-time map (DTM) in step S2 is as follows: perform a one-dimensional FFT on the range dimension and the doppler dimension of each frame of data in the three-dimensional data block to obtain the range-Doppler spectrum, sum the spectrum in the doppler dimension and the range dimension respectively to improve the echo signal-to-noise ratio, and obtain the range information and doppler information of a frame of data. Stack the range information and the doppler information on the time dimension respectively to obtain the RTM and DTM.

[0026] Furthermore, in step S3, a CNN_Block module is constructed using two-dimensional convolutional layers and pooling layers. The specific construction process is as follows: a convolutional neural network with two branches is constructed using convolutional layers and pooling layers. Each convolutional layer is followed by a batch normalization layer and a nonlinear layer. Each branch includes 3 convolutional layers and 2 pooling layers. The size of the convolutional kernel is 3×3, and the number of convolutional kernels increases in increments of 64, 128, and 256. The size of the pooling kernel is 2×2.

[0027] Furthermore, the method for obtaining local features of dynamic gestures in step S3 is as follows: inputting RTM and DTM into the CNN_Block module, where RTM and DTM ∈ R C×H×W The feature maps of the two branches are obtained separately. After the feature extraction is completed, the feature maps of the two branches are concatenated according to the channel dimension to obtain the fused feature as the local feature of the dynamic gesture.

[0028] Furthermore, in step S4, a Transformer module is constructed using layer normalized network (LN), multi-head attention mechanism (MHA), and multilayer perceptron (MLP), with LN preceding both MHA and MLP. The construction process of the Transformer module is as follows:

[0029] A1: The mean and variance of the two-dimensional input data are calculated using the layer normalization function. The calculation formula is as follows:

[0030]

[0031] Where X is the input sample, LN(X) is the layer normalization value, E(X) is the sample mean, σ[X] is the standard deviation of the sample, α and β are learnable variables, and ε is a very small quantity used to prevent overflow caused by the standard deviation being 0.

[0032] A2: A multi-head attention mechanism is used to assign different weights to each feature sequence, allowing the model to focus on important features and suppress interfering features; the calculation expression is as follows:

[0033]

[0034] Where Q, K, and V are the query vector, key vector, and value vector, respectively, and W... Q W K and W V These are the weight matrices obtained by Q, K, and V during training, respectively, and W... O To calculate the output matrix obtained from attention, d k For attention score scaling factor, head i Indicates the number of attention heads, i∈(1,2,...,n);

[0035] A3: An MLP layer is constructed using two fully connected layers and an activation function to integrate information from the multi-head attention output, thereby enhancing the model's ability to fit the data.

[0036] Furthermore, step S5 consists of multiple Transformer modules from step S4. Each Transformer module can extract global features. The purpose of using multiple Transformer modules is to extract gesture features at a deeper level, as specifically reflected in steps A1, A2, and A3.

[0037] The multi-head attention mechanism assigns different weights to effective (gesture) features and interfering features, thereby paying attention to different features to different degrees and suppressing interfering features. Steps S5 to S7 can all be reflected in step A2.

[0038] Furthermore, the method for constructing FC_Block in step S8 is as follows: using one layer of global flat pooling and two fully connected layers. The global flat pooling layer is used to reduce the dimensionality of the data, thereby reducing the number of model parameters, and two fully connected layers are used to map the feature vectors to the label space.

[0039] This invention provides a method for dynamic gesture recognition using millimeter-wave radar in interference environments. The method reconstructs the original millimeter-wave radar echo into a three-dimensional data block and uses a mean-cancellation method to filter out static interference. Then, a two-dimensional fast Fourier transform is used to extract the distance and Doppler information of the dynamic gesture, obtaining the distance-time map and Doppler-time map of the dynamic gesture. Coherence accumulation is then used to improve the echo signal-to-noise ratio. Finally, a CNN-Transformer network model is built for dynamic gesture recognition. The CNN extracts local features of the gesture, and multiple Transformer modules are stacked to fully extract deeper and more effective features. Using a Transformer network enhances the global focus on gesture features, and the multi-head attention mechanism in the Transformer network effectively suppresses random dynamic interference in the environment.

[0040] The above solution can be summarized into the following three steps:

[0041] (1) Radar signal processing: First, the radar echo signal is reconstructed into a three-dimensional data block according to the format of Chirps×Samples×Frames, and the static interference in the environment is filtered out by the mean cancellation method. Then, the distance and Doppler information of the dynamic gesture are extracted by the two-dimensional FFT algorithm. The obtained RTM and DTM are coherently accumulated to improve the echo signal-to-noise ratio.

[0042] (2) CNN_Block: Use two identical convolutional modules to extract distance-time and Doppler-time features respectively, and then concatenate them to obtain fused features.

[0043] (3) Transformer network: By stacking multiple Transformer modules, more effective features at a deeper level can be extracted. On the one hand, the Transformer network can enhance the global attention to gesture features, and on the other hand, the multi-head attention mechanism in the Transformer network can effectively suppress random dynamic interference in the environment.

[0044] Beneficial effects: Compared with the prior art, this invention uses a CNN-Transformer network model for dynamic gesture recognition. It utilizes CNN to extract local features of gestures and stacks multiple Transformer modules to fully extract deeper and more effective features. Using a Transformer network can enhance the global attention to gesture features. On the other hand, the multi-head attention mechanism in the Transformer network can effectively suppress random dynamic interference in the environment, realize gesture recognition in interference environments, improve gesture recognition accuracy, and solve the problem of effectively distinguishing gesture signals from interference signals in interference environments. Attached Figure Description

[0045] Figure 1 This is a diagram of the overall structure of the CNN-Transformer network in this invention;

[0046] Figure 2 This is a flowchart of radar signal processing;

[0047] Figure 3 This is a diagram of the CNN_Block architecture;

[0048] Figure 4 This is a diagram of the Transformer Encoder module structure;

[0049] Figure 5 This is a diagram showing the gesture categories and interference in this embodiment;

[0050] Figure 6 This is a comparison chart before and after the means cancel each other out in this embodiment;

[0051] Figure 7 These are the RTM and DTM graphs from the gesture dataset in this embodiment;

[0052] Figure 8 This is a graph showing the model loss and accuracy at different learning rates in this embodiment;

[0053] Figure 9This is the CNN-Transformer confusion matrix diagram in this embodiment. Detailed Implementation

[0054] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0055] This invention provides a method for dynamic gesture recognition using millimeter-wave radar in interference environments, such as... Figure 1 As shown, it includes the following steps:

[0056] S1: Reassemble the raw echo data of the millimeter-wave radar into three-dimensional data blocks, and use the mean cancellation method to filter out static interference;

[0057] S2: Two-dimensional fast Fourier transform is used to extract the distance and Doppler information of dynamic gestures, and the range-time map (RTM) and doppler-time map (DTM) of dynamic gestures are obtained. The echo signal-to-noise ratio is improved by coherent accumulation.

[0058] S3: Input the distance-time map and Doppler-time map of the dynamic gesture into the constructed CNN_Block module, and use the two-dimensional convolution in the CNN_Block module to extract the local features of the dynamic gesture;

[0059] S4: Construct the Transformer module using layer normalization, multi-head attention mechanism and MLP;

[0060] S5: By stacking multiple Transformer modules, deeper gesture features of dynamic gestures can be fully extracted;

[0061] S6: Enhance global attention to gesture features using the Transformer module;

[0062] S7: Utilize the multi-head attention mechanism in the Transformer module to suppress random dynamic disturbances in the environment;

[0063] S8: Use a global average pooling layer and a fully connected layer to construct an FC_Block, and output gesture labels through the FC_Block.

[0064] In this invention, the CNN_Block module, the Transformer module, and the FC_Block constitute a CNN-Transformer network model. Steps S3 to S8 are the process of performing dynamic gesture recognition through the CNN-Transformer network model.

[0065] like Figure 2 As shown, in step S1, since the original radar echo data acquired by the millimeter-wave radar acquisition device is only displayed as a one-dimensional data sequence, it is inconvenient to perform radar signal processing operations. Therefore, the one-dimensional data is reorganized into a three-dimensional data block. The three-dimensional data block adopts the Chirps×Samples×Frames format and is implemented using 2D-FFT.

[0066] The mean cancellation method for filtering static interference involves performing two-pulse cancellation on the 1D-FFT results. The specific expression is as follows:

[0067] Y(N)=X(N)-X(N-1),N=2,3,4,...

[0068] Where X(N) represents the Nth pulse.

[0069] The method for obtaining the Range-Time Map (RTM) and Doppler-Time Map (DTM) in step S2 is as follows: Perform a one-dimensional FFT on the range dimension and Doppler dimension of each frame of data in the three-dimensional data block to obtain the range-Doppler spectrum. Summate the spectrum in the Doppler dimension and the range dimension respectively to improve the echo signal-to-noise ratio and obtain the range information and Doppler information of a frame of data. Stack the range information and Doppler information on the time dimension to obtain the RTM and DTM.

[0070] like Figure 3 As shown, in step S3, a CNN_Block module is constructed using two-dimensional convolutional layers and pooling layers. The specific construction process is as follows: a convolutional neural network with two branches is constructed using convolutional layers and pooling layers. Each convolutional layer is followed by a batch normalization layer and a non-linear layer. Each branch includes 3 convolutional layers and 2 pooling layers. The size of the convolutional kernel is 3×3, and the number of convolutional kernels increases in increments of 64, 128, and 256. The size of the pooling kernel is 2×2.

[0071] The method for obtaining local features of dynamic gestures is as follows: input RTM and DTM into the CNN_Block module, where RTM and DTM ∈ R. C×H×W The feature maps of the two branches are obtained separately. After the feature extraction is completed, the feature maps of the two branches are concatenated according to the channel dimension to obtain the fused feature as the local feature of the dynamic gesture.

[0072] like Figure 4 As shown, in step S4, a Transformer module is constructed using Layer Normalization (LN), Multi-Head Attention (MHA), and Multilayer Perceptron (MLP). Both MHA and MLP are preceded by LN. The construction process of the Transformer module is as follows:

[0073] A1: The mean and variance of the two-dimensional input data are calculated using the layer normalization function. The calculation formula is as follows:

[0074]

[0075] Where X is the input sample, LN(X) is the layer normalization value, E(X) is the sample mean, σ[X] is the standard deviation of the sample, α and β are learnable variables, and ε is a very small quantity used to prevent overflow caused by the standard deviation being 0.

[0076] A2: A multi-head attention mechanism is used to assign different weights to each feature sequence, allowing the model to focus on important features and suppress interfering features; the calculation expression is as follows:

[0077]

[0078] Where Q, K, and V are the query vector, key vector, and value vector, respectively, and W... Q W K and W V These are the weight matrices obtained by Q, K, and V during training, respectively, and W... O To calculate the output matrix obtained from attention, d k For attention score scaling factor, head i Indicates the number of attention heads, i∈(1,2,...,n);

[0079] A3: An MLP layer is constructed using two fully connected layers and an activation function to integrate information from the multi-head attention output, thereby enhancing the model's ability to fit the data.

[0080] The FC_Block is constructed in step S8 by using one layer of global flat pooling and two fully connected layers. Global flat pooling is used to reduce the dimensionality of the data, decreasing the number of model parameters, and two fully connected layers are used to map the feature vectors to the label space.

[0081] Based on the above technical solution, in order to verify the effectiveness of the method of the present invention, this embodiment conducts experimental verification, as follows:

[0082] An AWR1642B00ST-ODS millimeter-wave radar, paired with a DCA1000 high-speed data acquisition card, was used to collect radar echoes of hand gestures and construct a gesture echo dataset. The radar echoes were transmitted to a PC via the data acquisition card, and data processing was performed using MATLAB 2021 software. A CNN-Transformer network, based on the PyTorch 1.13 deep learning framework, was trained on an Intel i7-10700K processor and an NVIDIA GTX 3090 graphics card.

[0083] The specific experimental procedure is as follows:

[0084] Step 1: Initialize radar parameters

[0085] The radar parameter configuration is shown in Table 1.

[0086] Table 1 Radar parameter configuration

[0087]

[0088]

[0089] Step 2: Define the dynamic gesture dataset

[0090] This experiment designed six types of gestures, such as Figure 5 As shown, (a)-(f) represent, in order, a V-shape, a high-five, waving left and right, pressing up and down, clockwise rotation, and counterclockwise rotation. The dataset consisted of 9 participants performing specified hand gestures, with 50 data files collected for each gesture, totaling 2700 data files (9 participants × 6 categories × 50 files). Random interference, such as waving and projectile movements, was introduced during data collection. Random dynamic interference included... Figure 5 As shown in (g)-(h). There are 150 data files for each type of interference, for a total of 300 echo data files containing interference, meaning that the interference data accounts for 10% of the total dataset.

[0091] Figure 5 This is a comparison graph showing the cancellation of the mean values ​​of the V-shaped hand gesture. The horizontal axis represents time (s), and the vertical axis represents distance (m). The trajectory of the V-shaped hand gesture is first moving closer to the radar and then moving away from it. Figure 6 As can be seen, the radial distance between the gesture and the radar decreases when the gesture is closer to the radar, and increases when the gesture is farther away from the radar. The red arrows in the diagram represent static interference. The purpose of using mean cancellation is to filter out static interference in the environment, such as walls and tables. Figure 6 (a) It can be seen that the table in the data acquisition environment is closest to the radar and has a stronger reflection intensity, which is represented by the bottom horizontal line. Mean cancellation can effectively filter out static interference (such as...). Figure 6 (b) is shown.

[0092] After signal processing and data preprocessing, 2700 RTM and DTM images without static interference and 300 RTM and DTM images with random dynamic interference can be obtained. Figure 7 For interference-free RTM and DTM and interference-containing RTM (I_RTM) and DTM (I_DTM). Figure 7 (a)-(f) correspond to respectively Figure 5 (a)-(f), such as Figure 7 (a) (RTM) is a V-shaped hand gesture, which means first approaching the radar and then moving away from the radar; Figure 7 (b) (RTM) is a high-five gesture, characterized by an almost constant radial distance between the gesture and the radar. Random interference is circled in red in the diagram. For example, Figure 7 (b)(I_RTM) The short line circled in red represents a random interference gesture (waving).

[0093] Step 3: Model Initialization

[0094] The initialization parameters of the CNN_Block module are shown in Table 2. Here, k represents the number of convolutional kernels, Batch is the number of batches of input data, and Channel, Height, and Width are the number of channels, height, and width of the feature map, respectively.

[0095] Table 2 Initialization parameters for the CNN_Block module

[0096]

[0097] Since CNN_Block consists of two identical convolutional modules, the final output of CNN_Block is two four-dimensional tensors with shape (64,256,12,12). After flattening them, they are concatenated according to the channel dimension to obtain a tensor with shape (64,512,144).

[0098] For the Transformer Encoder module, we stack 8 Transformer Encoders and set the number of heads for the multi-head attention mechanism to 8. To prevent model overfitting, this invention uses the Dropout function after multi-head attention, randomly discarding a portion of the weights at a rate of 20%.

[0099] Step 5: Network Training and Validation

[0100] In this embodiment, the dataset is divided into training, validation, and test sets in a 6:2:2 ratio. To improve network convergence, all feature maps are uniformly sized to 64×64 before being input into the network, and then standardized and normalized. The preprocessed RTM and DTM are input into the CNN-Transformer network for training, using the cross-entropy function to define the loss function. The Adam optimizer is used for parameter optimization, and early stopping is employed to monitor the accuracy on the validation set, ending model training when the generalization performance deteriorates.

[0101] To minimize the risk of the CNN-Transformer network getting stuck in local minima during training, we examined the impact of different learning rates ξ on model training. We set the learning rate as the only variable, keeping other parameters constant, and set the learning rates to 1e-3, 1e-4, 3e-3, and 3e-4 respectively. Figure 8 The figures show the validation loss function and accuracy curves of the proposed model under different learning rates, where (a) is the validation set loss curve and (b) is the validation set accuracy curve. It can be seen that a high accuracy is achieved when the learning rate is 1e-4, and the model's validation loss and accuracy are the most stable, with fast convergence. Therefore, in the following experiments and analyses, this invention selects a learning rate of 1e-4.

[0102] To verify the online recognition accuracy of the neural network constructed in this invention, two sets of data that were not used in training were selected as test sets (including two sets of data with and without interference) to analyze the model's recognition performance for each gesture. Figure 9 The confusion matrices of the CNN-Transformer network on both noise-free and noise-filled datasets are presented. Figure 9 (a) It can be seen that in an interference-free environment, the method of the present invention can achieve a recognition accuracy of more than 98% for each gesture, especially for clapping, waving left and right, and drawing a clockwise circle, all of which have a recognition accuracy of 100%.

Claims

1. A method for dynamic gesture recognition using millimeter-wave radar in interference environments, characterized in that, Includes the following steps: S1: Reassemble the raw echo data of the millimeter-wave radar into three-dimensional data blocks, and use the mean cancellation method to filter out static interference; S2: Two-dimensional fast Fourier transform is used to extract the distance and Doppler information of dynamic gestures, and the distance-time map (RTM) and Doppler-time map (DTM) of dynamic gestures are obtained. The echo signal-to-noise ratio is improved by coherent accumulation. S3: Input the distance-time map and Doppler-time map of the dynamic gesture into the constructed CNN_Block module, and use the two-dimensional convolution in the CNN_Block module to extract the local features of the dynamic gesture; S4: Construct the Transformer module using layer normalization, multi-head attention mechanism and MLP; S5: Extract gesture features of dynamic gestures by stacking multiple Transformer modules; S6: Enhance global attention to gesture features using the Transformer module; S7: Utilize the multi-head attention mechanism in the Transformer module to suppress random dynamic disturbances in the environment; S8: Construct FC_Block using a global average pooling layer and a fully connected layer, and output gesture labels through FC_Block.

2. The method for dynamic gesture recognition of millimeter-wave radar in interference environments according to claim 1, characterized in that, In step S1, the three-dimensional data blocks are in Chirps×Samples×Frames format.

3. The method for dynamic gesture recognition of millimeter-wave radar in interference environments according to claim 1, characterized in that, The method for filtering static interference using the mean cancellation method in step S1 involves performing two-pulse cancellation on the 1D-FFT result. The specific expression is as follows: Y(N)=X(N)-X(N-1),N=2,3,4,... Where X(N) represents the Nth pulse.

4. The method for dynamic gesture recognition of millimeter-wave radar in an interference environment according to claim 1, characterized in that, The method for obtaining the range-time map (RTM) and the doppler-time map (DTM) in step S2 is as follows: perform a one-dimensional FFT on the range dimension and the doppler dimension of each frame of data in the three-dimensional data block to obtain the range-Doppler spectrum. Summate the spectrum in the doppler dimension and the range dimension respectively to improve the echo signal-to-noise ratio and obtain the range information and doppler information of a frame of data. Stack the range information and the doppler information on the time dimension to obtain the RTM and DTM.

5. The method for dynamic gesture recognition of millimeter-wave radar in interference environments according to claim 1, characterized in that, In step S3, a CNN_Block module is constructed using two-dimensional convolutional layers and pooling layers. The specific construction process is as follows: a convolutional neural network with two branches is constructed using convolutional layers and pooling layers, followed by a batch normalization layer and a nonlinear layer after each convolutional layer.

6. The method for dynamic gesture recognition of millimeter-wave radar in an interference environment according to claim 5, characterized in that, Each branch of the CNN_Block module includes 3 convolutional layers and 2 pooling layers, with each convolutional kernel being 3×3 in size. The number of convolutional kernels increases in increments of 64, 128, and 256, while the pooling kernel is 2×2 in size.

7. The method for dynamic gesture recognition of millimeter-wave radar in an interference environment according to claim 5, characterized in that, The method for obtaining local features of dynamic gestures in step S3 is as follows: inputting RTM and DTM into the CNN_Block module, where RTM and DTM ∈ R C×H×W The feature maps of the two branches are obtained separately. After the feature extraction is completed, the feature maps of the two branches are concatenated according to the channel dimension to obtain the fused feature as the local feature of the dynamic gesture.

8. The method for dynamic gesture recognition of millimeter-wave radar in interference environments according to claim 1, characterized in that, In step S4, a Transformer module is constructed using Layer Normalized Network (LN), Multi-Head Attention (MHA), and Multilayer Perceptron (MLP), with LN preceding both MHA and MLP. The construction process of the Transformer module is as follows: A1: The mean and variance of the two-dimensional input data are calculated using the layer normalization function. The calculation formula is as follows: Where X is the input sample, LN(X) is the layer normalization value, E(X) is the sample mean, σ[X] is the standard deviation of the sample, α and β are learnable variables, and ε is a very small quantity used to prevent overflow caused by the standard deviation being 0. A2: Employ a multi-head attention mechanism to assign different weights to each feature sequence, enabling the model to focus on important features and suppress interfering features; A3: An MLP layer is constructed using two fully connected layers and an activation function to integrate information from the multi-head attention output, thereby enhancing the model's ability to fit the data.

9. The method for dynamic gesture recognition of millimeter-wave radar in an interference environment according to claim 1, characterized in that, The method for constructing FC_Block in step S8 is as follows: use one layer of global flat pooling and two fully connected layers.

10. A method for dynamic gesture recognition of millimeter-wave radar in an interference environment according to claim 8, characterized in that, The calculation expression for assigning different weights to each feature sequence using the multi-head attention mechanism in step A2 is as follows: Where Q, K, and V are the query vector, key vector, and value vector, respectively, and W... Q W K and W V These are the weight matrices obtained by Q, K, and V during training, respectively, and W... O To calculate the output matrix obtained from attention, d k For attention score scaling factor, head i This represents the number of attention heads, i∈(1,2,...,n).

Images