Millimeter wave radar cross-domain gesture recognition method based on self-attention mechanism

Abstract

The invention discloses a millimeter wave radar cross-domain gesture recognition method based on a self-attention mechanism. The method comprises the following steps: acquiring multi-user and multi-scene gesture action data in an interference environment through a millimeter-wave radar, extracting distance, Doppler and angle information by utilizing an acquired echo signal, and generating RAI; a diversified training set is constructed, and the generalization ability of the model is improved; designing a neural network model fusing a space self-attention mechanism and a time self-attention mechanism, extracting gesture features of each frame by using the space self-attention mechanism, capturing motion dynamic characteristics in combination with the time self-attention mechanism, and realizing full mining of the gesture features; and converting the gesture action in the user or scene which is not used for training into an RAI input model to finish accurate recognition. The method can effectively cope with human activity interference, has high robustness and cross-domain adaptability, and is suitable for gesture recognition tasks in various complex environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless signal gesture recognition, and in particular relates to a millimeter wave radar cross-domain gesture recognition method based on a self-attention mechanism. Background Art

[0002] As an important way of human-computer interaction, gesture recognition technology provides users with a more natural and intuitive interactive experience, and can effectively improve the convenience of operation between machines and devices. In application scenarios such as smart homes, users can control home appliances through gestures, such as adjusting lighting brightness, switching TV channels, adjusting air conditioning temperature, and even interacting with virtual reality devices, thereby achieving a more intelligent lifestyle. Existing gesture recognition methods mainly rely on two technologies, camera-based methods and wearable device-based solutions, but the use of these technologies has the following defects:

[0003] Camera-based gesture recognition methods are heavily dependent on lighting conditions. In situations of insufficient light or large changes in ambient light sources, recognition accuracy may drop significantly, resulting in unstable recognition performance.

[0004] The camera may capture other private information of the user when collecting gesture data, especially when used at home or in public environments, which poses a potential risk of leaking the user's personal privacy.

[0005] The gesture recognition method based on wearable devices requires users to wear specific devices, which increases the user's operating burden. Especially when used for a long time, the inconvenience of wearing may affect their acceptance and frequency of use.

[0006] Wearable devices usually require users to wear them for a long time, and there are problems such as wearing discomfort, large weight and size of the device, which affects their comfort and convenience of long-term use.

[0007] Traditional gesture recognition technology based on cameras or wearable devices may not be applicable in some special scenarios. For example, when the user cannot conveniently wear the device or when the ambient lighting is complex, the recognition effect may be greatly reduced.

[0008] In recent years, gesture recognition technology based on millimeter-wave radar has gradually become a research hotspot due to its high resolution, strong anti-interference and privacy protection advantages. Millimeter-wave radar gesture recognition can provide high-precision recognition results in complex environments by acquiring reflected signal data and analyzing the dynamic characteristics of gestures. However, existing millimeter-wave radar gesture recognition algorithms still face the following challenges: existing models usually only perform well in specific environments or users. Once they are migrated to different users and environments, the algorithm performance will drop significantly. This is because the propagation of millimeter-wave radar signals will be affected by environmental factors (such as obstacles, interference sources, etc.) and the movement habits of different users, resulting in differences in the radar features of the same gesture in different domains. At the same time, most of the existing denoising methods for millimeter-wave radars only consider low-frequency noise, and it is difficult to distinguish human activity noise with a similar frequency to gesture movements.

[0009] Therefore, how to analyze and extract effective gesture features in millimeter-wave radar signals with limited real training data, avoid interference from human activities, and overcome the inconsistency of data features between different domains has become the key to improving the generalization ability of millimeter-wave radar gesture recognition.

[0010] Existing research focuses on improving the accuracy and robustness of the model through deep learning methods, but has not fully considered how to improve cross-domain adaptability and robustness in dynamic and complex environments, especially under the interference of human activities. Therefore, designing a millimeter-wave radar gesture recognition algorithm that can effectively extract domain-independent gesture features and maintain good performance in different environments and users has become an urgent problem to be solved. Summary of the invention

[0011] The purpose of the present invention is to provide a millimeter-wave radar cross-domain gesture recognition method based on a self-attention mechanism, which aims to efficiently recognize gestures of different users and scenarios through millimeter-wave radar echo data in the presence of human activity interference. This method, combined with a deep learning model, can accurately and quickly identify the user's gesture type, and has strong cross-domain adaptability and anti-interference from human activities. Compared with traditional gesture recognition methods based on cameras or wearable devices, the present invention can overcome problems such as ambient light changes, privacy leaks, and inconvenience in wearing, has lower computational complexity, and is suitable for long-term stable use.

[0012] In order to solve the deficiencies in the prior art, a first aspect of the present invention provides a millimeter wave radar cross-domain gesture recognition method based on a self-attention mechanism, the method comprising the following steps:

[0013] Collect multiple gestures performed by multiple users in different scenarios to obtain echo data scanned by millimeter-wave radar, which also contains information about human activity interference;

[0014] Based on the echo data, extract information including distance, Doppler, angle, etc. to build a training data set for gesture recognition;

[0015] Construct a neural network model, use the self-attention mechanism for both spatial and temporal dimensions, and train it using the collected training set to obtain a gesture recognition model;

[0016] The gesture data not covered by the training set is input into the optimized gesture recognition model to achieve accurate gesture recognition.

[0017] Furthermore, the echo data processing method includes: first acquiring the original echo signal scanned by the millimeter wave radar; then, extracting gesture distance information, Doppler information and angle information based on the echo signal; denoising through a constant false alarm rate algorithm, and superimposing along the Doppler dimension to obtain the required RAI, which is used as a training set.

[0018] Furthermore, the structure of the deep learning model includes: an input layer, multiple convolutional layers, a pooling layer, a normalization layer, a fully connected layer, a long short-term memory layer, an attention mechanism layer, and an output layer. This network structure can effectively extract the spatiotemporal features of gestures, and process time series information through the long short-term memory layer to improve recognition accuracy.

[0019] Furthermore, a deep learning model is trained based on the collected training set to obtain a gesture recognition model.

[0020] Furthermore, data of users or scenes different from those in the training set are used as test data, and the test data is input into the trained model to calculate the category probability distribution of the target gesture, and the sum of the probabilities of all categories is 1; the gesture category with the highest probability value is selected as the final recognition result.

[0021] The second aspect of the present invention also provides a millimeter wave radar cross-domain gesture recognition method based on a self-attention mechanism, the system comprising:

[0022] The data acquisition module is used to obtain millimeter-wave radar data of multiple gestures of multiple users in various scenarios, including interference from human activities;

[0023] The data processing module is used to extract features and denoise the gesture action millimeter wave radar data to obtain RAI;

[0024] The model training module is used to build a neural network model, use the self-attention mechanism for both spatial and temporal dimensions, and train it using the collected training set to obtain a gesture recognition model;

[0025] The gesture recognition module is used to input gesture information of users or scenes not covered by the training set into the gesture recognition model, so as to achieve accurate recognition of gesture categories.

[0026] Furthermore, the method for the data processing module to obtain gesture data includes: obtaining echo data of the millimeter wave radar, extracting distance, Doppler and angle features of the gesture based on these data, and generating a training data set after denoising;

[0027] The structure of the deep learning model includes: an input layer, a convolutional layer, a pooling layer, a normalization layer, a fully connected layer, a self-attention layer, a long short-term memory layer and an output layer.

[0028] Furthermore, a deep learning model is trained based on the collected training set to obtain a gesture recognition model.

[0029] Furthermore, the step of the gesture recognition module identifying the gesture type includes: calculating the probabilities of all categories of gestures based on the trained gesture recognition model, and selecting the gesture category with the highest probability as the final recognition result.

[0030] The above technical solution of the present invention has the following beneficial technical effects:

[0031] Strong cross-domain adaptability: The gesture recognition method of the present invention can adapt to gesture recognition tasks for different users and in different scenarios, solving the recognition difficulties of existing methods in complex environments with interference from human activities.

[0032] High recognition accuracy: By combining the convolutional neural network (CNN) and LSTM model in deep learning, the present invention can efficiently extract spatiotemporal features and process time series data, thereby achieving high-precision gesture recognition.

[0033] Privacy protection and long-term use: Compared with the traditional camera-based gesture recognition method, the present invention does not require a camera or wearable device, avoids the risk of privacy leakage, and is suitable for long-term use, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flowchart of a method for cross-domain gesture recognition based on millimeter-wave radar self-attention according to an embodiment of the present invention;

[0035] Figure 2 is a flow chart of collecting raw millimeter-wave radar data and processing it into RAI according to an embodiment of the present invention;

[0036] Figure 3 is a structural diagram of a gesture recognition neural network model according to an embodiment of the present invention;

[0037] Figure 4 is a structural diagram of a spatial frame module according to an embodiment of the present invention;

[0038] Figure 5is a structural diagram of a time series module according to an embodiment of the present invention;

[0039] Figure 6 The millimeter-wave radar cross-domain gesture recognition system structure based on human activity interference according to an embodiment of the present invention;

[0040] Figure 7 is an example diagram of a confusion matrix using a new user test according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is specifically described below in conjunction with embodiments and with reference to the accompanying drawings. It should be understood that the following text description is only an example of the specific implementation of the present invention and does not strictly limit the scope of protection of the specific request of the present invention.

[0042] Figure 1 is a flow chart of a method for cross-domain gesture recognition based on self-attention of a millimeter-wave radar according to an embodiment of the present invention; Figure 1 As shown, operation S101 to operation S104 are included.

[0043] In operation S101, millimeter wave radar data of various gestures of multiple users in various scenarios including interference from human activities are obtained;

[0044] In operation S102, feature extraction and denoising are performed on the gesture action millimeter wave radar data to obtain RAI

[0045] In operation S103, a neural network model is constructed, a self-attention mechanism is used for both spatial and temporal dimensions, and the model is trained using the collected training set, thereby obtaining a gesture recognition model;

[0046] In operation S104, gesture information of users or scenes not covered by the training set is input into the gesture recognition model, so as to achieve accurate recognition of gesture categories.

[0047] Through the method provided by the present invention, a millimeter wave radar device is used to detect the gestures of a target user, and the acquired millimeter wave radar gesture information is passed through a constructed neural network model to achieve efficient and accurate gesture classification.

[0048] Figure 2 is a flow chart of collecting raw millimeter-wave radar data and processing it into RAI according to an embodiment of the present invention; Figure 2 As shown, collecting millimeter-wave radar gesture action information and processing RAI includes steps S201 to S204.

[0049] In operation S201, a millimeter wave radar is used to collect a variety of gesture action data of multiple users in various scenarios including interference from human activities;

[0050] In operation S202, a three-dimensional Fourier transform is used to perform feature extraction on the gesture action millimeter wave radar data to obtain a three-dimensional matrix of range-Doppler-angle;

[0051] In operation S203, after filtering out static interference by setting a Doppler threshold, the three-dimensional matrix is ​​accumulated along the Doppler dimension to obtain RAI;

[0052] In operation S204, a constant false alarm rate denoising algorithm is used to remove low-frequency noise in the environment to obtain the RAI that is finally used to input the model.

[0053] Through the method provided by the present invention, the distance, speed and angle information of the gesture action are extracted by a three-dimensional Doppler algorithm, and then the interference of static objects and low-frequency noise in the environment is removed by setting a threshold and a denoising algorithm, and only information close to the frequency of the gesture action is retained. The final RAI is used as the input of the model.

[0054] Figure 3 is a structural diagram of a gesture recognition neural network model according to an embodiment of the present invention. Figure 3 The process of obtaining gesture recognition results is described in detail.

[0055] The RAI obtained by preprocessing the collected millimeter-wave radar gesture information represents the distance and angle information of the gesture at a certain moment. A complete gesture is completed within a period of time, so a gesture is composed of multiple consecutive RAIs. Therefore, in order to accurately recognize gestures, it is necessary not only to extract features from each RAI, but also to extract the temporal features of multiple consecutive RAIs.

[0056] The gesture recognition neural network model of the embodiment of the invention is as follows Figure 3 As shown in the figure, the processed multiple RAIs representing gestures are input into the neural network model for model training. The model is divided into two parts, namely the spatial frame module and the time series module. The spatial frame module learns a single RAI and extracts the spatial position information of the gesture in the single RAI. The time series module models the time dimension and learns the changes of gestures in the time dimension.

[0057] Specifically, the spatial frame module is used to extract the gesture features of each RAI and map the features of each RAI into a feature vector, which reduces the amount of data and speeds up the training of the model.

[0058] The time series module connects the feature vectors obtained after multiple consecutive RAIs pass through the spatial frame module. By learning the temporal movement patterns of gestures, it can distinguish between gestures and human activity interference, and extract domain-independent gesture features, thereby achieving high-accuracy cross-domain gesture recognition under human activity interference.

[0059] Figure 4 is a structural diagram of a spatial frame module according to an embodiment of the present invention. Figure 4 Detailed description of the process for a single RAI.

[0060] Specifically, the spatial frame module contains three convolutional layers, three normalization layers, three activation layers, three pooling layers, one self-attention layer, and one fully connected layer. The training steps of the spatial frame module are as follows:

[0061] Step 1: Since the spatial frame module only extracts features from a single RAI, a complete RAI sequence of gesture action is split into individual RAI sequences.

[0062] Step 2: Each RAI is passed through three convolution layers, normalization layers, activation layers, and pooling layers to obtain the feature map of the RAI.

[0063] Step 3: In order to pay more attention to the spatial information of the moving object, the feature map is passed through the self-attention layer to highlight the spatial features of the moving object containing the gesture. The attention weight generated by the self-attention layer can be expressed as:

[0064]

[0065] Where l = w l l+b l represents the linear transformation of the target position, x i =w l x i +b l represents the linear transformation of other positions, and M = H × W is the total number of pixels in the feature map. The attention weights of the parameters are then fused with the original feature map to obtain the spatial features of the moving object.

[0066] Step 4: Pass the obtained spatial features through a fully connected layer to convert the two-dimensional matrix into a one-dimensional vector and reduce the amount of feature data to reduce the memory requirements for running.

[0067] Figure 5 is a structural diagram of a time series module according to an embodiment of the present invention. Figure 5 The process of processing the feature vector of the complete gesture action after passing through the spatial frame module is introduced in detail.

[0068] Specifically, the time series module contains a position embedding layer, a multi-head self-attention layer, and a long short-term memory layer. The training steps of the time series module are as follows:

[0069] Step 1: Connect the feature vectors of a complete gesture action to form a two-dimensional matrix and input it into the position embedding layer to obtain the features of the complete gesture action containing time series information.

[0070] Step 2: Input the feature into the multi-head self-attention layer to let the model learn the temporal motion trajectory of the gesture and extract the domain-independent gesture features. The attention weight generated by the multi-head self-attention layer can be expressed as:

[0071]

[0072] Similarly, the attention weights are weighted with the features after position embedding, so that the model pays more attention to the characteristics of gesture actions and reduces the interference of domain features.

[0073] Step 3: In order to better distinguish gestures from human activity interference, the extracted domain-independent gesture features are input into the long short-term memory layer to distinguish the movement patterns of gestures and human activity interference. Finally, the last frame of data is used to classify the gesture category.

[0074] Figure 6 4 is a schematic diagram of the structure of a millimeter-wave radar cross-domain gesture recognition system under interference from human activities according to an embodiment of the present invention.

[0075] like Figure 6 As shown, the system includes a data acquisition module, a data processing module, a model training module and a gesture recognition module.

[0076] The data acquisition module S601 is used to obtain millimeter wave radar data of various gestures of multiple users in various scenarios including interference from human activities;

[0077] The data processing module S602 is used to extract features and remove noise from the gesture action millimeter wave radar data to obtain RAI;

[0078] The model training module S603 is used to construct a neural network model and train it using a training set to obtain a gesture recognition model;

[0079] The gesture recognition module S604 is used to input gesture information of users or scenes not covered by the training set into the gesture recognition model, so as to achieve accurate recognition of gesture categories.

[0080] Figure 7 is an example diagram of a confusion matrix using a new user test according to an embodiment of the present invention.

[0081] Combine the following Figure 7 The effectiveness of the above system is further illustrated. In order to verify the robustness of the present invention for different domains and its anti-interference ability to human activities, the present invention collected human activity interference from 7 users, 3 scenarios, 3 different conditions, and 6 different gestures (push forward, pull back, right swipe with the right hand, left swipe with the left hand, draw a circle clockwise, and draw an inverted triangle), totaling 8694 samples. The experiment used the Ti AWR1843BOOST millimeter-wave radar with a starting frequency of 77GHz and a bandwidth of 4GHz. When collecting data, the collection object was about 2 meters away from the device, the device was 117 cm from the ground, and human activity interference occurred on both sides or behind the collection object. Without considering cross-domain and only considering human activity interference, the present invention can achieve a recognition accuracy of 99.6%. The data is divided according to different users and scenarios. When considering cross-domain situations, the average recognition accuracy of the present invention for new users and new scenarios reached 92.79% and 94.62% respectively.

[0082] The implementation method of the present invention provides a millimeter-wave radar cross-domain gesture recognition method and system based on a self-attention mechanism. The gesture recognition method extracts the spatial features of the gesture action in each frame, and then learns the movement rules of the gesture action in the entire time dimension. After the gesture recognition model is trained by this method, the data from new users or new scenes are input into the gesture recognition model to quickly and accurately obtain the gesture action category.

Claims

1. A millimeter wave radar cross-domain gesture recognition method based on self-attention mechanism, characterized in that: include: S1: Collect multiple gesture information of multiple users in various scenarios including human activity interference, and obtain echo data scanned by millimeter-wave radar; S2: Based on the echo data, extract information including distance, Doppler, angle, etc., generate a range angle image (RAI), and construct a training set; S3: Build a neural network model, use the self-attention mechanism for both spatial and temporal dimensions, and train it using the collected training set to obtain a gesture recognition model; S4: Input gesture information of users or scenes not covered by the training set into the gesture recognition model obtained previously, so as to achieve accurate recognition of gesture categories.

2. According to the millimeter wave radar cross-domain gesture recognition method based on self-attention mechanism according to claim 1, the step of constructing a training set comprises: Collect the original echo information of various gestures of multiple users in various scenarios; Obtain distance, Doppler and angle three-dimensional information based on echo information; The RAI is obtained by denoising with the constant false alarm rate algorithm and superimposing along the Doppler dimension.

3. According to the millimeter wave radar cross-domain gesture recognition method based on self-attention mechanism according to claim 1, the structure of the neural network model comprises: Input layer, first convolution layer, first normalization layer, first activation layer, first pooling layer, second convolution layer, second normalization layer, second activation layer, second pooling layer, third convolution layer, third normalization layer, third activation layer, third pooling layer, first self-attention layer, first fully connected layer, position embedding layer, second self-attention layer, long short-term memory layer, second fully connected layer and output layer. The first self-attention layer pays attention to the spatial dimension, while the second self-attention layer pays attention to the temporal dimension.

4. According to the millimeter wave radar cross-domain gesture recognition method based on self-attention mechanism according to claim 1, the step of identifying the gesture category comprises: According to the gesture recognition model, the possible probability of each gesture category is obtained, and the sum of the gesture probabilities of all categories is 1; The gesture category with the highest probability is selected as the gesture category for final classification.

Images