An end-to-end hybrid gesture intent recognition method based on MIMO radar

Abstract

The application discloses an end-to-end mixed gesture intention recognition method based on MIMO radar and belongs to the technical field of human-computer interaction. The application acquires signals by using a MIMO radar, and generates a distance-angle feature map sequence in real time through a Capon high-resolution angle estimation algorithm; the sequence is input into a multi-task neural network, the network comprises a shared main stem and multiple parallel task branch heads, and can decode spatial coordinates and discrete instruction intentions contained in a gesture in parallel at each time step. A streaming CNN-RNN architecture is adopted, and the spatial pointing task and the instruction recognition task are considered in combination with the low-delay requirement and the timing of the instruction recognition task; an intelligent intention arbitration module which is trained together with the main model is integrated, and the main intention of a user is dynamically judged frame by frame in a data-driven manner. The application solves the problems of high delay, fixed time scale and unnatural switching when mixed gesture intentions are processed in the prior art, and realizes low-delay, high-precision and natural human-computer interaction.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to an end-to-end hybrid gesture intent recognition method based on MIMO radar. Background Technology

[0002] With the development of extended reality (XR), smart cockpits, and IoT devices, human-computer interaction is evolving towards a more natural and multimodal approach. Air gestures, as an intuitive input method, have enormous application potential. When users interact with air gestures, their intentions are inherently mixed, potentially including spatial pointing (such as moving a cursor in a virtual interface) and discrete commands (such as specific gestures for confirmation or returning). The former requires the system to have low-latency frame-by-frame response capabilities, while the latter requires the system to recognize complete action patterns within a short period.

[0003] Existing gesture recognition systems typically handle these tasks separately or use fixed-length data windows for analysis. The drawbacks of fixed windows are: 1) they cannot achieve the lowest possible latency response, resulting in a poor user experience for tasks requiring immediate feedback, such as spatial pointing; 2) the window length is difficult to choose—too short and it fails to capture complete commands, too long and it adds unnecessary latency. Furthermore, designing independent models for different tasks leads to the system switching between multiple models, resulting in high latency and wasted resources, which also contradicts the original intention of natural interaction.

[0004] Therefore, the industry urgently needs an intelligent perception method that can process continuous gesture data streams and understand users' multiple potential intentions end-to-end at different time scales based on dynamically changing intentions. Summary of the Invention

[0005] This invention aims to solve the problems of existing gesture recognition systems being unable to efficiently handle mixed intentions, having high latency, and exhibiting unnatural interactions, and provides a hybrid gesture intention recognition method based on MIMO radar and a unified multi-task neural network.

[0006] To solve the above-mentioned technical problems, the method proposed in this invention includes the following steps: an end-to-end hybrid gesture intent recognition method based on MIMO radar, the method comprising:

[0007] Step 1: Acquire a continuous multi-channel radar sample gesture signal data stream using a MIMO millimeter-wave radar with multiple transmit and receive antennas;

[0008] Step 2: Process each frame of data in the data stream in real time to generate a continuous distance-angle feature map sequence and label it;

[0009] Step 3: Input the distance-angle feature map sequence into a multi-task neural network model based on a convolutional recurrent neural network, with a shared backbone network and multiple parallel task branches, and train it;

[0010] Step 4: Use the trained multi-task neural network model to recognize gesture intentions in real time.

[0011] Furthermore, the specific method for step 2 is as follows:

[0012] Step 2.1: Perform a one-dimensional FFT on the signal of each channel to obtain the distance-channel complex matrix;

[0013] Step 2.2: For each distance cell, first construct the spatial covariance matrix based on the complex values ​​of each channel. ;

[0014] ;

[0015] in, It is a column vector composed of all channel signals. Expressing expectations;

[0016] Step 2.3: Calculate the results at different angles using the following formula. Spatial spectral power :

[0017] ;

[0018] in, It is the corresponding angle The guide vector is generated, and this process is performed in real time for each frame of data to generate a high-resolution distance-angle feature map.

[0019] Furthermore, the annotation method in step 2 is as follows:

[0020] Labeling content: For each frame in the data stream, two sets of labels are provided:

[0021] Coordinate labels: The actual 3D spatial coordinates of the hand at that moment in the frame;

[0022] Instruction label: If the frame belongs to a discrete instruction gesture, the label is "instruction"; otherwise, the label is "no instruction or background".

[0023] Furthermore, the multi-task neural network model in step 3 sequentially includes: a backbone network, a coordinate regression head, an action command head, and an intent arbitration module. The backbone network sequentially includes a 2D-CNN feature extractor and a recurrent neural network. The 2D-CNN feature extractor is a two-dimensional convolutional neural network, whose input is a distance-angle feature map and output is a compact feature vector. The recurrent neural network is a gated recurrent unit or a long short-term memory network, with the input being a compact feature vector. The output of the recurrent neural network is input to the coordinate regression head, the action command head, and the intent arbitration module, respectively. The coordinate regression head is a fully connected layer used to output coordinates, and its loss function is... The action command header consists of a fully connected layer connected to a Softmax function, outputting the probability distribution of the command. The loss function is the cross-entropy loss function. ;

[0024] When training a multi-task neural network model, a masked multi-task loss function is used. ;

[0025] in, , These are hyperparameters for balancing tasks. It is a binary mask, which is only valid if the truth label of the current frame is a specific instruction. The value is 1 only if the condition is met; otherwise, it is 0.

[0026] The intent arbitration module is a multilayer perceptron containing several fully connected layers, and its output is a probability distribution vector. ,in ; This represents the confidence level that the user's current intent is "spatial orientation". This indicates the confidence level that the user's intent at the current moment is a "discrete instruction"; the supervision signal during training is the labeling of the data.

[0027] Furthermore, in step 4, the final identification result is made based on the probability distribution output by the arbitration module during the actual detection.

[0028] if Greater than or equal to The system will output the coordinates retrieved from the coordinate header;

[0029] Otherwise, the classification result of the instruction header is accepted.

[0030] This invention's data-driven approach automatically learns the mapping from dynamic features of gestures to the final intent, offering exceptional flexibility and robustness. Employing a streaming CNN-RNN architecture, it balances the low-latency requirements of spatial pointing tasks with the temporal requirements of instruction recognition tasks. It integrates an intelligent intent arbitration module trained alongside the main model, dynamically determining the user's primary intent frame-by-frame in a data-driven manner. This invention addresses the issues of high latency, fixed timescales, and unnatural transitions inherent in existing technologies for processing mixed gesture intents, achieving low-latency, high-precision natural human-computer interaction. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the architecture of a hybrid gesture intent recognition system provided in an embodiment of the present invention.

[0032] Figure 2 This is a flowchart of a method provided in an embodiment of the present invention.

[0033] Figure 3 This is a detailed structural diagram of an end-to-end multi-task neural network in an embodiment of the present invention, including an intent arbitration module.

[0034] Figure 4 This is a raw radar signal image of one channel acquired by the present invention.

[0035] Figure 5 This is a schematic diagram of the result of performing a Fourier transform on the signal diagram of one channel according to the present invention.

[0036] Figure 6 This is a schematic diagram of the spatial covariance matrix calculated after Fourier transform of all channels in this invention.

[0037] Figure 7 This is a schematic diagram illustrating the process of generating a distance-angle feature map according to the present invention. Detailed Implementation

[0038] The technical solution of the present invention will now be described in more detail with reference to the accompanying drawings.

[0039] Reference Figure 1 The system 100 of this embodiment includes: a MIMO millimeter-wave radar sensor 101, a signal preprocessor 102 (which may be a DSP), and a high-performance computing unit 103 (such as a SoC with an NPU or GPU).

[0040] MIMO radar sensor 101: Employs a linear array radar with 2 transmit (Tx) and 4 receive (Rx) antenna elements. Through time division multiplexing and other methods, an equivalent array containing 2×4=8 virtual channels can be formed, thereby achieving high angular resolution.

[0041] Reference Figure 2 The specific process for the invention method is as follows:

[0042] Step S1: High-resolution feature construction;

[0043] Multiple-input multiple-output (MIMO) millimeter-wave radar is used to acquire multi-channel raw signals, such as... Figure 4 As shown; the distance-dimensional FFT result obtained by performing a distance-dimensional FFT on a single channel signal is as follows. Figure 5 As shown, the expectation is calculated by performing an FFT on the distance dimension of all channels, resulting in the following: Figure 6 The spatial covariance matrix is ​​shown. A high-resolution angle estimation algorithm (such as the Capon algorithm) is used to generate a range-angle feature map in real time for each frame of data, forming a continuous feature map sequence stream, such as... Figure 7 As shown.

[0044] Steps S2 and S3: Streaming multi-task model inference and parallel decoding;

[0045] The real-time generated distance-angle feature map sequence is sent frame by frame. Figure 3 The CNN-RNN multi-task neural network is shown. This multi-task model performs a forward propagation at each time step (i.e., after processing each frame of feature maps) and simultaneously outputs data using its parallel task-specific headers:

[0046] Spatial coordinates: the two-dimensional spatial position of the hand in the current frame.

[0047] Action command category probability: The probability distribution of the current frame belonging to a certain discrete command.

[0048] Step S4: Intelligent intent arbitration and output;

[0049] Through a learned arbitration module, the current main intent is automatically inferred at each time step based on the model's internal state, and the most relevant decoding results are selectively output.

Claims

1. An end-to-end hybrid gesture intent recognition method based on MIMO radar, characterized in that, The method includes: Step 1: Acquire a continuous multi-channel radar sample gesture signal data stream using a MIMO millimeter-wave radar with multiple transmit and receive antennas; Step 2: Process each frame of data in the data stream in real time to generate a continuous distance-angle feature map sequence and label it; Step 2.1: Perform a one-dimensional FFT on the signal of each channel to obtain the distance-channel complex matrix; Step 2.2: For each distance cell, first construct the spatial covariance matrix based on the complex values ​​of each channel. ; ； in, It is a column vector composed of all channel signals. Expressing expectations; Step 2.3: Calculate the results at different angles using the following formula. Spatial spectral power : ； in, It is the corresponding angle The guide vector is generated, and this process is performed in real time for each frame of data to generate a high-resolution distance-angle feature map; Step 3: Input the distance-angle feature map sequence into a multi-task neural network model based on a convolutional recurrent neural network, with a shared backbone network and multiple parallel task branches, and train it; The multi-task neural network model sequentially includes: a backbone network, a coordinate regression head, an action command head, and an intent arbitration module. The backbone network sequentially includes a 2D-CNN feature extractor and a recurrent neural network. The 2D-CNN feature extractor is a two-dimensional convolutional neural network, whose input is a distance-angle feature map and output is a compact feature vector. The recurrent neural network is a gated recurrent unit or a long short-term memory network, with the input being a compact feature vector. The output of the recurrent neural network is input to the coordinate regression head, the action command head, and the intent arbitration module, respectively. The coordinate regression head is a fully connected layer used to output coordinates, with a loss function of [missing information]. The action command header consists of a fully connected layer connected to a Softmax function, outputting the probability distribution of the command. The loss function is the cross-entropy loss function. ; When training a multi-task neural network model, a masked multi-task loss function is used. ; in, , These are hyperparameters for balancing tasks. It is a binary mask, which is only valid if the truth label of the current frame is a specific instruction. The value is 1 only if the condition is met; otherwise, it is 0. The intent arbitration module is a multilayer perceptron containing several fully connected layers, and its output is a probability distribution vector. ,in ; This represents the confidence level that the user's intent at the current moment is "spatial orientation". This indicates the confidence level that the user's intent at the current moment is a "discrete instruction"; the supervision signal during training is the data labeling. Step 4: Use the trained multi-task neural network model to recognize gesture intentions in real time.

2. The end-to-end hybrid gesture intent recognition method based on MIMO radar as described in claim 1, characterized in that, The annotation method in step 2 is as follows: Labeling content: For each frame in the data stream, two sets of labels are provided: Coordinate labels: The actual 3D spatial coordinates of the hand at that moment in the frame; Instruction label: If the frame belongs to a discrete instruction gesture, the label is "instruction"; otherwise, the label is "no instruction or background".

3. The end-to-end hybrid gesture intent recognition method based on MIMO radar as described in claim 1, characterized in that, In step 4, the final identification result is made based on the probability distribution output by the arbitration module during the actual detection. if Greater than or equal to The system will output the coordinates retrieved from the coordinate header; Otherwise, the classification result of the instruction header is accepted.

Images