Millimeter wave radar gesture recognition method and system based on gating collaborative fusion

Abstract

The invention discloses a millimeter wave radar gesture recognition method and system based on gating collaborative fusion, and belongs to the technical field of artificial intelligence and man-machine interaction. The method comprises the following steps: firstly, acquiring and preprocessing original data through a millimeter wave radar sensor; secondly, extracting a global spatial feature map (such as a two-dimensional point cloud map) capable of representing the overall motion track of the gesture and a plurality of local dynamic feature maps (such as a distance-time map, a Doppler-time map and the like) capable of representing dynamic information of the gesture in parallel; and then, the extracted multi-modal features are processed through the gated collaborative fusion network designed by the invention. The core of the method is that the global spatial feature map is processed through a first neural network and a global average pooling layer to generate a static global feature vector; dynamically modulating a local feature sequence processed by a second neural network through a gating unit by using the static global feature vector; and finally, fusing all modulated sequences, inputting the fused sequences into a time sequence processing module for analysis, and outputting a final gesture category. According to the method, through an innovative gating fusion mechanism, dynamic cooperation of multi-modal features is realized, the problem that a traditional fusion method cannot adapt to different gesture feature changes is effectively solved, and the accuracy and robustness of gesture recognition are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, and in particular to a multimodal gesture recognition method and system based on millimeter-wave radar data and deep learning. Background Technology

[0002] With the increasing prevalence of smart homes, in-vehicle entertainment, and virtual reality, the demand for contactless human-computer interaction technologies is growing rapidly. Camera-based visual gesture recognition is susceptible to issues such as lighting conditions, background complexity, and privacy breaches. Millimeter-wave radar, as an alternative, offers advantages such as all-weather operation, protection of user privacy, and the ability to penetrate non-metallic obstacles, demonstrating significant potential in the field of gesture recognition.

[0003] However, existing gesture recognition methods based on millimeter-wave radar typically employ simple feature stitching or fixed-weighted averaging when fusing multiple radar features, failing to effectively utilize the complementary and synergistic relationships between different feature modalities. For example, they cannot dynamically adjust the focus on detailed features such as distance, speed, and angle based on the overall spatial shape of the gesture, resulting in insufficient recognition accuracy and robustness when distinguishing similar gestures.

[0004] Therefore, there is an urgent need to propose a more efficient multimodal feature fusion mechanism to improve the performance of millimeter-wave radar gesture recognition systems. Summary of the Invention

[0005] The present invention aims to solve the problems existing in the above-mentioned background technology and provide a millimeter-wave radar gesture recognition method and system based on gated collaborative fusion, which can dynamically fuse multi-modal radar features and significantly improve the accuracy and robustness of gesture recognition.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A millimeter-wave radar gesture recognition method based on gated collaborative fusion includes the following steps:

[0008] Data acquisition and preprocessing steps: Raw ADC data is captured by a millimeter-wave radar sensor, and static clutter interference is eliminated by applying a moving target indication (MTI) filter to obtain preprocessed radar data.

[0009] Multimodal feature extraction step: From the preprocessed radar data, feature maps of at least two different modalities are extracted in parallel, and the feature maps include:

[0010] At least one global spatial feature map is used to characterize the overall motion trajectory of the gesture. In a preferred embodiment of the present invention, the feature map is a two-dimensional point cloud trajectory map.

[0011] At least one local dynamic feature map is used to characterize the dynamic information of the gesture changing over time in a specific physical dimension. In a preferred embodiment of the present invention, the feature map is selected from one or more of the following: distance-time map (RTM), Doppler-time map (DTM), azimuth-time map (ATM), and pitch-time map (ETM).

[0012] The gated collaborative fusion step, which is the core of this invention, includes:

[0013] a) Global feature vector generation: The global spatial feature map is input into the first neural network (commander branch), and global average pooling (GAP) is performed on the output features of the network to generate a static global feature vector representing the overall spatial shape of the gesture.

[0014] b) Local feature sequence generation: The at least one local dynamic feature map is input into the corresponding second neural network (auxiliary branch), and the output high-level feature map is reshaped into a local feature sequence.

[0015] c) Gated modulation: The static global feature vector is input to the gating unit to generate a corresponding gating signal for each local feature sequence; the gating signal is multiplied element-wise with the corresponding local feature sequence to obtain the modulated feature sequence.

[0016] Gating signal generation: This involves generating the characteristic F of the command branch. pc A gated vector g is generated by passing through a linear layer and a sigmoid activation function. i :

[0017] g i =σ(W g,i ·GAP(F pc )+b g,i )

[0018] Where GAP is global average pooling and σ is the sigmoid function.

[0019] Feature modulation: The generated gated signal g i Multiply element-wise onto the corresponding auxiliary features:

[0020] F′ dense,i =F dense,i ⊙g i

[0021] Where ⊙ represents element-wise multiplication (Hadamard product).

[0022] d) Final feature fusion: All gated and modulated feature sequences are spliced ​​along the feature dimension to form the final fused feature sequence.

[0023] Temporal modeling and classification steps: The final fused feature sequence is input into a temporal processing module containing a recurrent neural network (RNN) and a classifier to output the final gesture recognition category result. In a preferred embodiment of the present invention, the recurrent neural network is a bidirectional long short-term memory network (Bi-LSTM), followed by a self-attention mechanism layer to optimize the sequence representation.

[0024] The present invention also provides a gesture recognition system, which includes a module configured to perform any of the above method steps.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] An innovative gating fusion mechanism: Through a “commander-assistant” architecture, it dynamically and selectively modulates other detailed features using the global shape information provided by point clouds, achieving more intelligent and efficient multimodal information fusion and effectively improving the ability to distinguish between similar and complex gestures.

[0027] Hierarchical feature processing: The strategy of using CNN to extract local spatiotemporal patterns and then using LSTM to model high-level feature sequences realizes the division of labor and deep abstraction of radar data, giving full play to the advantages of different network structures.

[0028] Higher accuracy and robustness: By synergistically utilizing and dynamically weighting multimodal features, the method of this invention can achieve higher recognition accuracy than traditional fusion methods and has better robustness to noise and interference. Attached Figure Description

[0029] Figure 1 The overall flowchart of gesture recognition provided for an example of the present invention.

[0030] Figure 2 This is a flowchart of the preprocessing process for RTM data in an example of the present invention.

[0031] Figure 3 This is a flowchart of the preprocessing of DTM data in an example of the present invention.

[0032] Figure 4 This is a flowchart illustrating the preprocessing of ATM and ETM data in an example of the present invention.

[0033] Figure 5 This is a diagram illustrating the overall architecture of the gated collaborative fusion network in an example of the present invention. Detailed Implementation

[0034] like Figure 1As shown, this embodiment provides a detailed gesture recognition method based on gated collaborative fusion for millimeter-wave radar gesture recognition. Its overall process includes two main stages: 1) multimodal radar feature engineering; 2) a deep learning model based on gated collaborative fusion. To make the objectives, technical solutions, and advantages of this invention clearer, the preferred embodiments of this invention will be described in detail below with reference to the accompanying drawings.

[0035] This embodiment provides a detailed gesture recognition method based on millimeter-wave radar data and a gated fusion network.

[0036] 1. Data Acquisition and Preprocessing

[0037] The raw data captured by the radar sensor is a series of ADC sample values. For a gesture sequence, it can be represented as a five-dimensional complex data cube. Where N f N is the number of frames. l N is the number of loops / chirps in each frame. tx and N rx These are the number of transmitting and receiving antennas, N and N, respectively. s This is the number of ADC sampling points per Chirp.

[0038] For subsequent processing, the data from the physical antenna is reshaped to form a virtual antenna array data. Where the number of virtual antennas N va =N tx ×N rx In this embodiment, the radar used includes two transmitting antennas and four receiving antennas, forming a virtual antenna array with eight elements.

[0039] To eliminate constant clutter interference caused by static objects, a moving target indication (MTI) filter was applied. This is a first-order high-pass filter, and its mathematical expression is:

[0040] S′(t i ) = S va (t i )-α·S va (t i-1 )

[0041] Wherein, S′(t) i S is the output signal at time point i after MTI filtering. va (t i The signal data of the current i-th Chirp, S va (t i-1The signal data of the previous (i-1) Chirp, where α is the filter coefficient.

[0042] 2. Multimodal feature engineering

[0043] Four dense feature maps and one sparse point cloud map are extracted in parallel from the preprocessed data.

[0044] 2.1 Dense Feature Map Generation and Processing

[0045] Reference for generating distance-time graphs (RTM) Figure 2 First, the signal is decomposed using a one-dimensional discrete wavelet transform (DWT), and then denoising is achieved through a soft thresholding function. The soft thresholding function can be expressed as:

[0046]

[0047] Subsequently, to suppress spectral leakage, the time-domain signal is multiplied by a Hanning window function. The mathematical expression for this window function is:

[0048]

[0049] Finally, the windowed signal of each chirp is subjected to a Fast Fourier Transform (FFT) along the fast time axis, and the results of all chirps are arranged in chronological order to obtain the RTM. This process can be represented as:

[0050]

[0051] Reference for generating Doppler-time maps (DTM) Figure 3 First, the target dominant distance index r at each time point is determined using a dynamic peak detection algorithm. peak (t l ), and extract the phase history φ(t) at that distance. l ).

[0052] The phase history is subjected to a short-time Fourier transform (STFT), and its amplitude spectrum constitutes a DTM. This process can be expressed as:

[0053]

[0054] To suppress clutter in the DTM, a filtering method based on singular value decomposition (SVD) is applied. This method performs SVD on the feature map matrix, sets the k largest singular values ​​corresponding to static clutter to zero, and then reconstructs the matrix. The reconstruction formula is as follows:

[0055] X=U∑V *

[0056] Reference for generating angle-time maps (ATM / ETM) Figure 4 First, for the virtual antenna snapshot vector at the peak distance at each time point, the Multiple Signal Classification (MUSIC) algorithm is applied to estimate the angle of arrival. The MUSIC pseudospectrum is defined as:

[0057]

[0058] The angular spectrum energy at all time points is integrated along the elevation and azimuth dimensions respectively to obtain the azimuth-time map (ATM) and elevation-time map (ETM). At the same time, SVD filtering is also applied to suppress clutter.

[0059] Feature map post-processing

[0060] For RTM and DTM: An enhancement method is applied that includes median filtering, quantile thresholding, and small-size connected component removal.

[0061] For ATM: A filtering method combining smoothing along the time axis and an adaptive spatial threshold calculated independently for each column is applied. This adaptive threshold can be expressed as:

[0062] T col =μ col +α·σ col

[0063] For ETM: A trajectory tracking and reconstruction method based on Kalman filtering is applied. The core of this method is to utilize a Kalman filter incorporating an adaptive observation noise mechanism to optimally estimate the position and amplitude of the pitch angle peak. The state update equation is:

[0064]

[0065] Its observation noise covariance R k Based on the signal amplitude m norm,k Make adaptive adjustments.

[0066] 2.2 Generation of Two-Dimensional Point Cloud Trajectories

[0067] This method uses a "triple-filter centroid tracking method based on the strongest peak point of the distance-Doppler graph" to generate point cloud trajectories.

[0068] First layer: Spatial clustering filtering: Detect N peak points with the strongest energy on the RDM, and then retain only the core point clusters that are close enough to the "leader" peak through distance-gated clustering.

[0069] The second layer: Angle validity filtering: Calculate the arrival angle of each point in the core point cluster and filter out unreliable points whose angles exceed the preset range.

[0070] The third layer: Temporal smoothing filtering: This layer performs exponential smoothing updates on the centroid positions calculated from consecutive frames, and includes a frame loss tolerance mechanism. The smoothing update formula is:

[0071] pos new =α·c curr +(1-α)·po sprev

[0072] 2.3 Standardization Processing

[0073] All post-processed feature maps are mapped to the [0,1] interval through value normalization and scaled to a uniform size (e.g., 224x224) through size normalization.

[0074] 3. Gated collaborative converged network

[0075] Gated collaborative converged networks such as Figure 5 As shown, the above 5 feature maps are packaged into a tensor of (Batch, 5, 224, 224) and input into the gated collaborative fusion network designed in this invention.

[0076] Parallel feature extraction: The input tensor is split into 5 independent streams, each fed into an independent MobileNetV2 backbone network for feature extraction, resulting in high-level feature maps F. pc and F dense,i

[0077] Gating and Fusion:

[0078] Commander Branch (Point Cloud Branch) Output Feature Map F pc A global shape vector is obtained by passing through a global average pooling (GAP) layer.

[0079] The shape vector is input into four independent gating units, generating a gating signal g for each auxiliary branch. i This process can be represented as:

[0080] g i =σ(W g,i ·GAP(F pc )+b g,i )

[0081] Each gate signal and its corresponding auxiliary branch feature sequence F dense,i Element-wise multiplication yields the modulated sequence. Finally, the four modulated auxiliary sequences are concatenated along the feature dimension to form the final fused feature sequence F. fused .

[0082] Temporal modeling and classification:

[0083] The fused feature sequence F fusedIt is fed into a bidirectional long short-term memory network (Bi-LSTM) to learn temporal dependencies.

[0084] The output sequence of Bi-LSTM is weighted and summed using a self-attention mechanism to generate a fixed-length context vector c. The self-attention weights are calculated as follows:

[0085]

[0086] Finally, the context vector c is passed through a fully connected classifier, and the probability of each gesture category is output through the Softmax function. This process can be represented as:

[0087] P(y|X)=Softmax(W c ·c+b c ).

Claims

1. A method for gesture recognition based on gated co-attention fusion for millimeter wave radar, characterized in that, The method comprises the following steps: Step 1: capturing raw echo signals by a millimeter wave radar sensor, and pre-processing the raw echo signals to suppress static clutter; Step 2: extracting at least one global spatial feature map and one or more local dynamic feature maps from the pre-processed signals obtained in step 1 in parallel, wherein the global spatial feature map is used to represent the overall motion trajectory of a gesture, and the local dynamic feature map is used to represent the dynamic information of the gesture changing over time in a specific physical dimension; Step 3: building a gated collaborative fusion network, inputting the feature maps extracted in step 2 into the network, and processing the network in the following way: a) passing the global spatial feature map through a first neural network backbone and a global average pooling layer to generate a static global feature vector; b) passing the one or more local dynamic feature maps through corresponding second neural network backbones respectively, and reshaping the outputs to local feature sequences; c) using the static global feature vector generated in step a) to perform element-wise gated modulation on each local feature sequence generated in step b) through a gating unit; d) concatenating all the gated modulated local feature sequences to form a final fusion feature sequence; Step 4: inputting the final fusion feature sequence obtained in step 3 into a time series modeling and classification module to obtain the final gesture recognition category result. The method according to claim 1, wherein in step 2, the generation of the local dynamic feature map specifically comprises: a) generating a range-time map (RTM) by performing fast Fourier transform (FFT) on the pre-processed signals; b) generating a Doppler-time map (DTM) by performing short-time Fourier transform (STFT) on the phase history of a specific range gate in the pre-processed signals; c) generating an azimuth-time map (ATM) and an elevation-time map (ETM) by applying multiple signal classification (MUSIC) algorithm to the virtual antenna array data in the pre-processed signals. 2.The method of claim 1, wherein, In step 2, the global spatial feature map is a two-dimensional point cloud trajectory map, and the generation method specifically comprises sequentially performing spatial clustering filtering, angle validity filtering and time series smoothing filtering. 3.The method of claim 1, wherein, In step 3, the specific way of the gating unit to perform gated modulation on the local feature sequence is: a) inputting the static global feature vector into a network comprising a linear layer and a Sigmoid activation function to generate a gating signal; b) extending the gating signal in the time dimension to the same length as the local feature sequence; c) element-wise multiplying the extended gating signal and the local feature sequence.

4. The method of claim 1, wherein, In step 4, the time series modeling and classification module comprises a bidirectional long short-term memory network (Bi-LSTM) and a self-attention mechanism layer. 5.A millimeter wave radar gesture recognition system based on gated coattentive fusion, characterized in that, It comprises: a pre-processing module for performing the operation of step 1 of claim 1; a feature extraction module for performing the operation of step 2 of claim 1; a gated fusion module for performing the operation of step 3 of claim 1; a classification module for performing the operations recited in step 4 of claim 1.

Images