End-to-end millimeter wave radar gesture recognition method based on LT-C-GRU

Abstract

The invention relates to the technical field of gesture recognition, in particular to an LT-C-GRU-based end-to-end millimeter-wave radar gesture recognition method, which comprises the following steps: acquiring millimeter-wave radar data of different gestures; processing data of different dimensions of the millimeter wave radar by using distance transformation and Doppler transformation of the LT network; and sequentially inputting the output value of the LT network into the stacked CNN network, the GRU network and the two fully connected layers, and outputting a gesture prediction result. According to the method, the LT network is used for carrying out learnable transformation on millimeter wave radar signals, then the CNN-GRU network is used for extracting information of features and time sequences, and finally, a recognition result is output through a full connection layer.

Description

Technical Field

[0001] The present invention relates to the technical field of gesture recognition, and in particular to an end-to-end millimeter-wave radar gesture recognition method based on LT-C-GRU. Background Art

[0002] FMCW millimeter-wave radar was previously mainly used in high-end automobiles and military vehicles due to its large size and high cost. With the development of low-cost millimeter-wave radar, it may be used in various fields; such as vital signs monitoring, gesture recognition, fall detection and human activity recognition.

[0003] Dynamic gesture control is a contactless human-computer interaction method, which is widely used in smart home, car driving, remote surgery, sign language translation and other fields; the premise of dynamic gesture control is to accurately recognize the predetermined gesture; some current sensors, such as optical cameras, can also be used for gesture recognition, however, optical sensors have many limitations; optical sensors must work under good lighting conditions and are prone to expose personal privacy; with the development of integrated RF front-ends, the application potential of millimeter-wave radar in gesture recognition has attracted the attention of many researchers in recent years.

[0004] Radar gesture recognition usually requires some degree of preprocessing. For example, the publication number is CN118172799A, which uses millimeter-wave radar equipment to collect pedestrian gait data in different scenarios and preprocess the data. The gait point cloud data obtained by the millimeter-wave radar, point cloud processing is to transform the parameters in the radar signal into coordinates, which involves the conversion of polar coordinates to rectangular coordinates, and also needs to consider the position and direction of the radar signal. For example, DFT (discrete Fourier transform) is used. However, the use of traditional DFT has some inherent computational costs, and generally requires some normalization operations to be performed before network injection. The paper Gesture recognition with a low power fmcw radar and a deep convolutional neural network uses DFT to process radar signals, but the ideal radar target detection network is able to use non-standardized data in the time domain as input. How to find an effective FMCW data preprocessing method is an urgent problem to be solved; in addition, building a radar target detection network to improve gesture recognition accuracy is also an urgent problem to be solved. Summary of the invention

[0005] In view of the shortcomings of the existing methods, the present invention uses the LT network to perform learnable transformation on the millimeter-wave radar signal, then extracts the features and time series information through the CNN-GRU network, and finally outputs the recognition result through the fully connected layer.

[0006] The technical solution adopted by the present invention is: an end-to-end millimeter wave radar gesture recognition method based on LT-C-GRU comprises the following steps:

[0007] Step 1: Collect FMCW data of different gestures; use the distance transformation and Doppler transformation of the LT network to process FMCW data of different dimensions;

[0008] As a preferred embodiment of the present invention, different gestures include: pinching the thumb and index finger, pinching the thumb and little finger, sliding to the right, sliding to the left, rubbing forward, rubbing backward, rotating to the left, rotating to the right, clenching a fist, sliding to the left, sliding to the right and zooming out.

[0009] As a preferred implementation manner of the present invention, the data of different dimensions include: distance and Doppler value.

[0010] As a preferred implementation of the present invention, the LT network includes: a distance transformation layer, a first LeakyReLU operation, a swap layer, a Doppler transformation layer, a second LeakyReLU operation and a normalization operation.

[0011] As a preferred embodiment of the present invention, the outputs of the range transform layer and the Doppler transform layer of the LT network are:

[0012]

[0013] Among them, ω(k,m) represents the DFT weight value of the k-th sequence in the next layer and the m-th sequence in the previous layer, and output[k] represents the DFT output value of the k-th sequence in the next layer.

[0014] As a preferred implementation of the present invention, the formula for the DFT weight value is:

[0015]

[0016] Where N represents the number of DFT points.

[0017] As a preferred embodiment of the present invention, the formula of the LeakyReLU operation is:

[0018]

[0019] Where z and b are the input vector and bias respectively.

[0020] Step 2: Input the output value of the LT network into the stacked CNN network, GRU network and two fully connected layers in sequence, and output the gesture prediction result;

[0021] As a preferred embodiment of the present invention, the stacked CNN network includes three CNN network cascades, and the CNN network includes a convolution layer, a batch normalization layer, an activation layer and a pooling layer; the convolution kernel size is 3*3, the ReLu activation function is used, and the pooling kernel size is 2*2.

[0022] As a preferred embodiment of the present invention, an end-to-end millimeter-wave radar gesture recognition system based on LT-C-GRU includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement an end-to-end millimeter-wave radar gesture recognition method based on LT-C-GRU.

[0023] As a preferred embodiment of the present invention, a computer readable medium stores a computer program code, and when the computer program code is executed by a processor, an end-to-end millimeter wave radar gesture recognition method based on LT-C-GRU is implemented.

[0024] Beneficial effects of the present invention:

[0025] 1. Construct an LT network for FMCW data processing. The output value of the LT network does not need to be preprocessed into a point cloud, nor does it need to be processed by traditional DFT. Instead, it is driven by data through learnable transformations and the deep learning mechanism of the LT network automatically learns transformations suitable for gesture action tasks;

[0026] 2. Construct a CNN-GRU neural network to recognize gestures, use CNN to fully extract gesture features, and then use the GRU network to associate the action timing features, achieving an accuracy of 96.81% in a public gesture recognition dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a network block diagram of human gesture recognition performed by the LT-CNN-GRU network of the present invention;

[0028] Figure 2 is a structural diagram of the LT network of the present invention;

[0029] Figure 3 It is a transformation diagram of two complex-valued linear layers of the LT network of the present invention;

[0030] Figure 4 It is a CNN network structure diagram constructed by the present invention;

[0031] Figure 5 This is a GRU network structure diagram used in the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0033] like Figure 1 As shown, the end-to-end millimeter-wave radar gesture recognition method based on LT-C-GRU includes the following steps:

[0034] Step 1: Collect FMCW data of different gestures; use LT network to process the distance and Doppler parameters in the millimeter wave radar data;

[0035] The different gestures include: pinching the thumb and index finger, pinching the thumb and pinky finger, sliding right, sliding left, rubbing forward, rubbing backward, rotating left, rotating right, making a fist, sliding left, sliding right and zooming out, a total of 12.

[0036] The construction of LT network (Learnable Transformation) includes:

[0037] Step 11: construct a complex-valued linear layer based on range transform and Doppler transform;

[0038] Since the original FMCW signal is complex-valued, the original millimeter-wave radar signal is directly used as input using a complex-valued linear layered method. DFT is usually performed on the three dimensions of the radar signal, namely, distance R, Doppler value D, and angle A. However, there are two main problems that limit the DFT of all three dimensions. The first is computational complexity. Gesture recognition control requires real-time classification results, which is difficult to achieve on edge devices without GPU acceleration. The other is that multiple receiving antennas are required for arrival angle estimation, which is not always supported by the hardware. Therefore, the RDAT model is only suitable for MIMO radars with rich computing resources, where T represents transformation. Doppler information and distance information are relatively important information in motion recognition, and using RT or DT alone will lose some important information. Therefore, while using RDT to effectively utilize Doppler information and distance information, complexity is also taken into account. Therefore, the LT network is constructed using RDT, which is mainly composed of two complex-valued linear layers, the first layer is distance transformation, and the second layer is Doppler transformation.

[0039] like Figure 2 As shown, it consists of two complex-valued linear layers and an exchange layer. The first layer is a distance transformation layer, which then passes through an exchange layer to exchange the first and second dimensions of radar data, and finally is input into the Doppler transformation layer.

[0040] Step 12, initialize the weights of the LT network; in both layers of transformation, DFT is used as prior knowledge to initialize the LT network. DFT is a linear transformation from the time domain to the frequency domain. In the neural network, the time-frequency linear transformation can be represented by a complex-valued linear layer, and the weights are adjusted through end-to-end training using the neural network to adapt to different tasks; the distance transformation layer transforms the fast time (sample) dimension of the original millimeter-wave radar data; the Doppler transformation layer transforms the output result of the distance transformation layer in the slow time (chirp) dimension; the distance transformation and Doppler transformation processes are as follows Figure 3 As shown, the output of the two complex-valued linear layers can be expressed as:

[0041]

[0042] Among them, ω(k,m) represents the DFT weight value of the k-th sequence in the next layer and the m-th sequence in the previous layer, and output[k] represents the DFT output value of the k-th sequence in the next layer.

[0043] The formula for the DFT weight value is:

[0044]

[0045] Wherein, N represents the number of DFT points; for example, it is 256 in the distance transform layer.

[0046] The output value of the complex-valued linear layer corresponds to the standard DFT. Although the weights of the complex-valued linear layer are initialized to DFT, after end-to-end training, the transform is no longer a Fourier transform.

[0047] Step 13. Considering that DFT output values ​​usually produce larger values, instance normalization operation is used before inputting into CNN, and nonlinear LeakyReLU activation function is added after the distance transform layer and the Doppler transform layer; LeakyReLU is used to retain the negative values ​​of DFT and provide nonlinearity, retain the phase of the activation value, and help obtain more gesture features.

[0048] The formula of LeakyReLU is:

[0049]

[0050] Among them, z and b are the input vector and bias respectively; z is the complex-valued linear layer output output[k].

[0051] Step 2: Construct a convolutional neural network;

[0052] Step 21, build a CNN network, such as Figure 4It is a structural diagram of the CNN module constructed by the present invention. The CNN network includes: a convolution layer, a batch normalization layer, an activation layer and a pooling layer; the convolution kernel size is set to 3*3, ReLu is selected as the activation function, and the pooling kernel size is set to 2*2;

[0053] Step 22: Stack and connect three CNN networks to gradually extract the previous level features.

[0054] Step 3: Build a GRU network. GRU is a recurrent neural network that can process time series data well, such as Figure 5 This is the GRU network structure diagram.

[0055] Specifically include:

[0056] Step 31: Construct reset gate and update gate; for a given time step t, assume that the input is a small batch X t ∈R n*d , where n is the number of samples and d is the input feature dimension; the hidden state of the previous time step is H t-1 ∈R n*h , h is the number of hidden units; reset gate R t ∈R n*h and update gate Z t ∈R n*h The mathematical expression is as follows:

[0057] R t =σ(X t W xr +H t-1 W hr +b r ) (4)

[0058] Z t =σ(X t W xz +H t-1 W hz +b z ) (5)

[0059] Among them, W xr ,W xz ∈R d*h is the weight parameter corresponding to the input variable, W hr ,W hz ∈R h*h is the weight parameter corresponding to the hidden unit, b r ,b z ∈R 1*h is the bias parameter, which uses the + function to convert the input value to the interval (0,1).

[0060] Step 32: Calculate the candidate hidden state at time step t

[0061]

[0062] Among them, W xh ∈R d*h and W hh ∈R h*h is the weight parameter, b h ∈R 1*h is the bias term, ⊙ is Element-wise product operator; tanh nonlinear activation function is used to ensure that the values ​​in the candidate hidden states remain in the interval (-1,1).

[0063] In the candidate hidden state In the example above, the reset gate coefficient R is used. t Controls which information in the hidden state of the previous step is used when processing input information; R t New input information X when approaching 0 t It is dominant, indicating that the information contained in the input of the current step is less relevant to the previous information; R t When it is close to 1, the newly input information is highly correlated with the previous long-term information, and comprehensive consideration is needed to generate the information processed in the current step.

[0064] Step 33: Calculate the hidden state output at time step t; combine with update gate Z t The effect of the previous hidden state H t-1 and the new candidate hidden state of this step To calculate the final output, the formula is:

[0065]

[0066] Whenever gate Z is updated t When it is close to 1, the GRU model tends to keep only the old state; at this time, the output X from this step t The information of is essentially ignored, effectively skipping time step t in the dependency chain; on the contrary, when Z t When it is close to 0, the new hidden state H t It will be close to the candidate hidden state This solves the vanishing gradient problem and better captures dependencies in sequences with long time step distances.

[0067] Step 34: Output the implicit information of the last time step of GRU as the summary information of the sequence, and input it into the fully connected layer, and output the classification result through the fully connected layer.

[0068] Experimental process:

[0069] The dataset uses the human gesture dataset collected by Peijun Zhao's team at the University of Edinburgh. The radar data is collected using the Texas Instruments IWR6843ISK millimeter-wave sensor, MMWAVEICBOOST, and DCA 1000EVM. The sensor is mounted on a tripod at a height of approximately 110 cm. The raw millimeter-wave radar data is continuously streamed and stored on a PC.

[0070] This dataset collects data from eight users. The users performed 12 different gesture activities in front of the radar, recorded the start time of each action, and then extracted the data corresponding to each action from the original data file based on the start time. The length is 1 second, and 30 samples are collected for each person for each action; the ratio of training set, validation set, and test set is 15:5:10.

[0071] In order to verify the effectiveness of the present invention, a comparison is made with the traditional DFT processing method and the existing deep learning model.

[0072]

[0073] It can be seen from the table that the present invention has the best accuracy and does not require preprocessing of radar data. It is an end-to-end neural network gesture recognition model.

[0074] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. An end-to-end millimeter-wave radar gesture recognition method based on LT-C-GRU, characterized in that: The following steps are involved: Step 1: Collect millimeter-wave radar data of different gestures; use the distance transformation and Doppler transformation of the LT network to process the millimeter-wave radar data of different dimensions; Step 2: Input the output value of the LT network into the stacked CNN network, GRU network and two fully connected layers in sequence, and output the gesture prediction result.

2. The end-to-end millimeter wave radar gesture recognition method based on LT-C-GRU according to claim 1 is characterized in that: The LT network includes: a distance transform layer, a first LeakyReLU operation, a swap layer, a Doppler transform layer, a second LeakyReLU operation, and a normalization operation.

3. The end-to-end millimeter wave radar gesture recognition method based on LT-C-GRU according to claim 2 is characterized in that: The outputs of the distance transform layer and Doppler transform layer of the LT network are: Among them, ω(k,m) represents the DFT weight value of the k-th sequence in the next layer and the m-th sequence in the previous layer, and output[k] represents the DFT output value of the k-th sequence in the next layer.

4. The end-to-end millimeter wave radar gesture recognition method based on LT-C-GRU according to claim 3 is characterized in that: The formula for the DFT weight value is: Where N represents the number of DFT points.

5. The end-to-end millimeter wave radar gesture recognition method based on LT-C-GRU according to claim 4 is characterized in that: The formula for the LeakyReLU operation is: Where z and b are the input vector and bias respectively.

6. The end-to-end millimeter wave radar gesture recognition method based on LT-C-GRU according to claim 1 is characterized in that: The stacked CNN network consists of three CNN networks in cascade. The CNN network includes a convolution layer, a batch normalization layer, an activation layer, and a pooling layer. The convolution kernel size is 3*3, the ReLu activation function is used, and the pooling kernel size is 2*2.

7. The end-to-end millimeter wave radar gesture recognition method based on LT-C-GRU according to claim 1 is characterized in that: The different gestures include: thumb and index finger pinch, thumb and pinky pinch, swipe right, swipe left, rub forward, rub backward, rotate left, rotate right, make a fist, swipe left, swipe right, and zoom out.

8. The end-to-end millimeter wave radar gesture recognition method based on LT-C-GRU according to claim 1 is characterized in that: Different dimensions of data include: distance and Doppler value.

9. The end-to-end millimeter-wave radar gesture recognition system based on LT-C-GRU is characterized by: include: a memory for storing instructions executable by a processor; A processor, configured to execute instructions to implement the end-to-end millimeter-wave radar gesture recognition method based on LT-C-GRU as described in any one of claims 1-8.

10. A computer readable medium storing computer program code, characterized in that: When the computer program code is executed by a processor, the computer program code implements the end-to-end millimeter wave radar gesture recognition method based on LT-C-GRU as described in any one of claims 1 to 8.

Images