A gesture recognition method, apparatus, medium and device based on millimeter wave radar

Abstract

The application discloses a gesture recognition method and device based on a millimeter wave radar, a medium and equipment, relates to the field of millimeter wave radar point cloud feature processing and deep learning gesture recognition, and comprises the following steps: generating enhanced point cloud data containing a space domain, a speed domain and an echo intensity domain based on intermediate frequency echo data of a historical human gesture action acquired through a millimeter wave radar; fusing and projecting the space domain and the echo intensity domain in the enhanced point cloud data with the speed domain respectively to construct a three-channel point cloud feature image sequence; performing time sequence slicing processing on the three-channel point cloud feature image sequence to generate time sequence feature samples with a fixed length; training a time sequence neural network constructed through the time sequence feature samples, inputting time sequence features of a gesture to be tested into the trained time sequence neural network, and acquiring a final gesture recognition result.

Description

Technical Field

[0001] This invention relates to the field of millimeter-wave radar point cloud feature processing and deep learning gesture recognition, and particularly to a gesture recognition method, apparatus, medium and device based on millimeter-wave radar. Background Technology

[0002] Human-computer interaction technology is continuously evolving towards contactless, intelligent, and ubiquitous applications. Gesture interaction, with its natural operation and ease of use, has been widely applied in smart home control, in-vehicle cockpit operation, and public smart terminal interaction. While visual camera sensing solutions are susceptible to ambient light interference and pose privacy risks, millimeter-wave radar possesses inherent advantages such as stable all-weather operation, strong environmental anti-interference capabilities, good privacy protection, and contactless sensing, making it the preferred sensing medium for gesture interaction in various scenarios. The TI IWR6843 millimeter-wave radar can directly output raw point cloud data containing spatial coordinates, echo intensity, and motion speed, accurately capturing subtle gesture motion features and providing high-quality raw data support for gesture recognition tasks based on millimeter-wave radar in multiple scenarios.

[0003] However, most existing millimeter-wave radar gesture recognition methods use dual-channel feature maps that fuse spatial coordinates and velocity. Due to the limited angular resolution of the radar, for short-range (small-amplitude) cross-drawing, the fluctuation range is numerically at the same noise level as the slight tremor caused by the natural micro-tremor or back-and-forth movement of the hand when clicking. The "trajectory projection" of these two actions on the two-dimensional plane almost completely overlaps, resulting in a high confusion rate of similar gestures, which limits the gesture recognition accuracy of millimeter-wave radar in multiple scenarios. Summary of the Invention

[0004] This invention provides a gesture recognition method, apparatus, medium, and device based on millimeter-wave radar to solve the aforementioned problems in the prior art, namely, how to improve the accuracy of gesture recognition across different scenarios in the prior art. This invention provides a gesture recognition method based on millimeter-wave radar, which includes: Based on mid-frequency echo data of historical human hand gestures acquired by millimeter-wave radar, enhanced point cloud data containing spatial, velocity, and echo intensity domains is generated. The spatial domain and echo intensity domain of the enhanced point cloud data are fused and projected with the velocity domain to construct a three-channel point cloud feature image sequence; the three-channel point cloud feature image sequence is then processed by temporal slicing to generate fixed-length temporal feature samples. The constructed temporal neural network is trained using temporal feature samples. The three-channel temporal features of the gesture to be tested are input into the trained temporal neural network, and the low-level spatial contour features of the gesture are extracted from the three-channel temporal features frame by frame to determine the temporal spatial features. The high-level semantic and detailed motion features of the gesture are extracted from the temporal spatial features to determine the deep spatial features. The deep spatial features are analyzed and fused by forward and reverse temporal features to determine the global temporal features. The global temporal features are mapped to a multi-dimensional gesture category space to obtain the corresponding action category probabilities. The category corresponding to the maximum action category probability is taken as the final gesture recognition result.

[0005] Optionally, the acquisition of the enhanced point cloud data specifically includes: The acquired intermediate frequency echo data is filtered to obtain the filtered intermediate frequency echo data; A fast Fourier transform is performed on the filtered intermediate frequency echo data to generate a range-Doppler matrix containing target distance and velocity information; Based on the amplitude information of the extracted distance-Doppler matrix, candidate detection points are generated. The DBSCAN density clustering algorithm is used to cluster the candidate detection points to generate an enhanced point cloud containing time, space, velocity and intensity information.

[0006] Optionally, the step of performing temporal slicing on the three-channel point cloud feature image sequence to generate fixed-length temporal feature samples specifically includes: A sliding window strategy is used to perform temporal slicing on the three-channel point cloud feature image sequence. The window length is set to 20 frames and the sliding step is 10 frames. Starting from the beginning position of the three-channel point cloud feature image sequence, 20 consecutive frames of feature images are extracted as an independent subsequence until all valid windows that can be extracted are traversed. Finally, a fixed-length temporal feature sample with the format (20,32,32,3) is generated.

[0007] Optionally, the temporal neural network specifically includes: a low-level feature extraction module, a high-level semantic feature extraction module, a temporal dependency learning module, and a fully connected classification output module; wherein, the low-level feature extraction module extracts the low-level spatial contour features of the gesture from the three-channel temporal features frame by frame to determine the temporal spatial features; the high-level semantic feature extraction module extracts the high-level semantic and detailed motion features of the gesture from the temporal spatial features to determine the deep spatial features; the temporal dependency learning module performs forward and backward temporal feature analysis on the deep spatial features and fuses them to determine the global temporal features; the fully connected classification output module maps the global temporal features to a multi-dimensional gesture category space, obtains the corresponding action category probabilities, and takes the category corresponding to the maximum action category probability as the final gesture recognition result.

[0008] Optionally, the temporal feature samples may be subjected to data augmentation processing before training the constructed temporal neural network using the temporal feature samples.

[0009] Optionally, during the training of the temporal neural network, the model is compiled using the Adam optimizer with a learning rate of 0.002 and a loss function of classification cross-entropy.

[0010] Optionally, the millimeter-wave radar is a TI IWR6843 millimeter-wave radar, which is configured with two transmitting antennas and four receiving antennas.

[0011] This invention provides a gesture recognition device based on millimeter-wave radar to realize the above-mentioned gesture recognition method based on millimeter-wave radar.

[0012] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described gesture recognition method based on millimeter-wave radar.

[0013] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described gesture recognition method based on millimeter-wave radar.

[0014] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a gesture recognition method based on millimeter-wave radar. This method generates enhanced point cloud data containing spatial, velocity, and echo intensity domains based on acquired mid-frequency echo data of human gesture movements. Then, the spatial, echo intensity, and velocity domains of the enhanced point cloud data are fused and projected pairwise to construct temporal feature samples. Combined with a lightweight temporal neural network, multi-dimensional gesture features are fully explored, improving the accuracy of cross-scene gesture recognition based on millimeter-wave radar from the input level. Furthermore, by employing a Time-Distributed CNN+Bidirectional LSTM architecture to construct a lightweight temporal neural network, temporal sharing of convolutional parameters is achieved. The spatial contour features and detailed motion features of gestures are extracted level by level through two-level convolutional networks (bottom and top layers), and the sequential correlation and temporal change patterns of continuous gesture movements are captured bidirectionally. While ensuring feature extraction capabilities, the number of model parameters is significantly compressed, thereby improving the accuracy of cross-scene gesture recognition. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0016] Figure 1A flowchart illustrating a gesture recognition method based on millimeter-wave radar, provided as an embodiment of the present invention; Figure 2 The diagram illustrates nine hand gestures provided in embodiments of the present invention. Figure 2 (a) is waving inwards. Figure 2 (b) means waving outwards. Figure 2 (c) represents clockwise rotation. Figure 2 (d) represents counterclockwise rotation. Figure 2 (e) represents a click. Figure 2 (f) is for drawing an X. Figure 2 (g) means to slash to the right. Figure 2 (h) is a left stroke. Figure 2 (i) represents waving left and right;

[0017] Figure 3 This is a schematic diagram illustrating the generation of the original four-domain features and three-channel feature maps of the 6843 radar provided in an embodiment of the present invention; Figure 4 The structure diagram of a temporal neural network based on Time-Distributed CNN+Bidirectional LSTM provided in the embodiments of the present invention; Figure 5 The confusion matrix diagram of the four-domain three-channel input method based on millimeter-wave radar gesture recognition provided in the embodiments of the present invention; Figure 6 A comparison curve of recognition accuracy between three-channel input and traditional dual-channel input provided in an embodiment of the present invention; Figure 7 A schematic diagram of a computer device for a gesture recognition method based on millimeter-wave radar provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0020] Example 1 Figure 1 This is a flowchart of a gesture recognition method based on millimeter-wave radar provided in an embodiment of the present invention, as shown below. Figure 1 As shown in the figure, this embodiment illustrates a gesture recognition method based on millimeter-wave radar, including: S1: Based on the mid-frequency echo data of historical human hand gestures acquired by millimeter-wave radar, generate enhanced point cloud data containing spatial domain, velocity domain and echo intensity domain.

[0021] For example, the present invention can use the TI IWR6843 millimeter-wave radar to collect mid-frequency echo data of human hand gestures, and use algorithms such as filtering, multidimensional fast Fourier transform, angle estimation, CFAR detection, and clustering to extract multidimensional features such as time, space, velocity, and intensity of the gesture point cloud to form an enhanced point cloud (T, X, Y, V, I).

[0022] For example, the TI IWR6843 millimeter-wave radar uses a configuration of two transmitting antennas (Tx0 and Tx1) and four receiving antennas (Rx0 to Rx3). Its core acquisition parameters are: starting frequency 60 GHz, frequency modulation slope 166.022 MHz / μs, ADC sampling points 256, sampling rate 12500 ksps, idle time 7.00 μs, Tx start-up time 1.00 μs, ADC start-up time 3.00 μs, ramp end time 24.00 μs, receive gain 30 dB, each frame contains 128 chirp cycles, single-action acquisition is 50 frames, and frame period is 40 ms. The process of obtaining enhanced point clouds from raw data follows a step-by-step data flow: signal preprocessing → feature mapping → target selection → coordinate calculation → point cloud optimization. Each step is interconnected and progressively builds upon the previous one. The specific steps are as follows:

[0023] (0) In the filtering process, static clutter filtering is performed on the original intermediate frequency echo data. By suppressing the fixed interference signals in the environment, the signal-to-noise ratio of the moving gesture target is effectively improved. The filtered intermediate frequency data is output and transmitted to the two-dimensional fast Fourier transform stage, laying a high-quality data foundation for subsequent feature extraction. (2) In the two-dimensional fast Fourier transform stage, the filtered intermediate frequency data is subjected to range-to-fast Fourier transform along the ADC sampling dimension and Doppler-to-fast Fourier transform along the Chirp cycle dimension to complete the feature mapping between the range domain and the Doppler domain, generating a range-Doppler matrix containing target range and velocity information. This matrix transmits the amplitude information to the CFAR detection stage for target energy screening on the one hand, and retains the phase information of multiple antennas to transmit to the angle estimation stage for azimuth calculation on the other hand. (3) In the CFAR detection process, the two-dimensional unit average constant false alarm rate (CFAR) detection algorithm is adopted. The number of protection units, the number of training units and the detection threshold are set. By comparing the energy difference between the target unit and the surrounding training units, the effective target candidate points that meet the energy threshold requirements are screened out, and the background noise interference is accurately eliminated. (4) In the angle estimation stage, the azimuth information of the target is calculated based on the signal phase difference between multiple receiving antennas. Combined with the radial distance obtained by range FFT, the X and Y coordinates of the target in the plane are obtained through triangulation, and the precise spatial positioning of the target is completed. (5) In the clustering optimization stage, the DBSCAN density clustering algorithm is used to cluster the detected target candidate points, remove isolated noise points and invalid clusters, retain valid target clusters with spatial continuity, and finally integrate the four core features of the target: timestamp, planar spatial coordinates X / Y, motion velocity V, and echo intensity I, to generate an enhanced point cloud (T,X,Y,V,I) containing time, space, velocity, and intensity dimensions.

[0024] S2: The spatial domain and echo intensity domain of the enhanced point cloud data are fused and projected with the velocity domain to construct a three-channel point cloud feature image sequence; the three-channel point cloud feature image sequence is processed by temporal slicing to generate fixed-length temporal feature samples.

[0025] For example, the generation process of the three-channel feature map is as follows: extract four types of four-domain basic features from the original radar point cloud: X coordinate, Y coordinate, echo intensity I, and motion velocity V. Normalize X and V, Y and V, and I and V respectively. The specific process is as follows: (1) Set the radar lateral sensing fixed range Longitudinal sensing fixed range The X and Y spatial coordinates are linearly normalized to the [0,1] interval using a globally fixed range. The normalization formula is as follows: (2) The echo intensity I adopts a single-frame adaptive normalization strategy, which normalizes it to [0,1] based on the maximum intensity of all valid point clouds in the current frame. The formula is as follows: (3) The motion speed V is not subjected to pixel-level normalization operation alone. Instead, it serves as the benchmark pairing dimension for constructing the three-channel feature maps of X and V, Y and V, and I and V, thereby realizing the four-domain coupling relationship between spatial, intensity features and motion speed features, and fully preserving the dynamic motion features of the gesture.

[0026] (4) Map the normalized features to a 32×32 discrete grid. The formula for calculating the grid row and column indices is: (5) The calculated row and column indices are clamped to a range of 0 to 31 effective pixels to prevent invalid indices from being generated due to coordinate out-of-bounds errors. Pixel assignment adopts a point-to-point direct assignment method. When a single radar point cloud is mapped to a unique pixel position in the grid, the X, Y, and echo intensity feature values ​​corresponding to that point are directly assigned to the current pixel. Pixels without point cloud projection coverage are set to 0 by default.

[0027] For example, the time-series sample generation and data augmentation process can include the following three core steps: (1) In the construction of time series samples, based on a sequence of 50 three-channel feature images, a sliding window strategy is used to complete the time series slicing operation. The window length is set to 20 frames and the sliding step is 10 frames. Starting from the beginning of the original sequence, 20 consecutive feature images are extracted as an independent subsequence until all the valid windows that can be extracted are traversed. Finally, a fixed-duration time series sample with the format (20,32,32,3) is generated to ensure that each sample can completely cover the core time series change features of the gesture action. (2) In the original data augmentation stage, augmentation operations are performed only on the training set samples. Specifically, there are two types of augmentation algorithms: one is Gaussian noise injection, which superimposes Gaussian noise with a mean of 0 and a standard deviation of 0.01 on the original three-channel feature map to simulate the signal disturbances that exist in the actual acquisition scenario; the other is random amplitude scaling, which randomly scales the pixel values ​​of the original feature map by 0.9 to 1.1 times, introducing amplitude changes while preserving the feature distribution pattern. No augmentation processing is performed on the validation set samples to ensure the objectivity and authenticity of the validation results. (3) In the dataset partitioning stage, all time series samples are divided into training set and validation set in a 6:4 ratio using stratified sampling. During the partitioning process, the proportion of various gesture action samples in the training set and validation set is strictly kept consistent with that in the original dataset to avoid model generalization bias due to uneven class distribution and to ensure that the training process can fully learn the feature rules of various gestures.

[0028] S3: Train the constructed temporal neural network using temporal feature samples. Input the three-channel temporal features of the gesture to be tested into the trained temporal neural network. Extract the low-level spatial contour features of the gesture from the three-channel temporal features frame by frame to determine the temporal spatial features. Extract the high-level semantic and detailed motion features of the gesture from the temporal spatial features to determine the deep spatial features. Perform forward and reverse temporal feature analysis on the deep spatial features and fuse them to determine the global temporal features. Map the global temporal features to a multi-dimensional gesture category space to obtain the corresponding action category probabilities. Take the category corresponding to the maximum action category probability as the final gesture recognition result.

[0029] For example, such as Figure 2 As shown, this invention constructs a temporal recognition architecture combining Time-Distributed CNN and Bidirectional LSTM, with the input layer fixed to adapt to three-channel temporal features (20, 32, 32, 3). The Time-Distributed CNN extracts spatial features frame by frame through two convolutional-pooling units containing 16 and 32 3×3 convolutional kernels, respectively. After flattening and Dropout (0.5) to suppress overfitting, a 32-neuron Bidirectional LSTM layer (with L2 regularization) learns bidirectional temporal dependencies, and then Dropout (0.3) further regularizes the data. Finally, a fully connected layer with L2 regularization is used to activate the output action category probability with Softmax, achieving high-precision gesture action classification.

[0030] Specifically, a time-distributed convolutional neural network-bidirectional long short-term memory (LSTM) architecture adapted to three-channel input is constructed, including the following: (1) Time-Distributed CNN Low-Level Feature Extraction Module: The input is a three-channel temporal gesture feature (batch, 20, 32, 32, 3). Relying on the Time-Distributed structure, the convolution parameters are shared frame by frame. The first layer is configured with 16 3×3 convolutional kernels (same padding, ReLU activation, L2 regularization) to extract the low-level spatial contour features of the gesture. The output dimension is (20, 32, 32, 16). Then, a 2×2 max pooling layer is used for downsampling to compress the feature map size, and the output dimension is (20, 16, 16, 16) temporal spatial features.

[0031] (2) Time-Distributed CNN high-level semantic feature extraction module: The input is (20,16,16,16) temporal features. The Time-Distributed structure is used for frame-by-frame convolution. The second layer is configured with 32 3×3 convolution kernels (same padding, ReLU activation, L2 regularization) to mine high-level semantic and detailed motion features of gestures. The output dimension is (20,16,16,32). Then, a 2×2 max pooling layer is used for further downsampling, and the output dimension is (20,8,8,32) deep spatial features.

[0032] (3) Feature Flattening and First Regularization Module: The input is (20,8,8,32) temporal features. The Time-Distributed flattening layer maps the 8×8×32 two-dimensional features of each frame into a one-dimensional vector (2048 dimensions per frame). The overall output is a one-dimensional temporal feature with dimensions (20,2048). A Dropout layer with a Dropout rate of 0.3 is then applied to randomly discard some neuron weights to suppress overfitting of high-dimensional features and output the regularized one-dimensional temporal feature.

[0033] (4) Bidirectional LSTM Temporal Dependency Learning Module and Second Regularization Module: The input is a (20, 2048) one-dimensional temporal feature, which is fed into a Bidirectional LSTM layer to simultaneously traverse forward and backward through 20 frames of temporal features, capturing the sequential correlation and temporal change patterns of continuous gesture actions in both directions. A Dropout layer with a Dropout rate of 0.5 is then applied to further reduce overfitting in the temporal dimension and output a more generalized global temporal feature.

[0034] (5) Fully connected classification output module: The input is the global temporal features after secondary regularization. The features are mapped to the 9-dimensional gesture category space through a fully connected layer with L2 regularization constraints. The probability distribution of the 9 types of gesture actions is normalized by the Softmax activation function. The category corresponding to the maximum probability is taken as the final gesture recognition result. The model input layer is fixed to adapt to the (batch, 20, 32, 32, 3) three-channel temporal feature format.

[0035] For example, the Time-Distributed CNN+Bidirectional LSTM network uses the Adam optimizer with a learning rate of 0.002, a loss function of classification cross-entropy, a batch size of 16, and a maximum training epoch of 50. The training process incorporates a triple callback mechanism of ModelCheckpoint, EarlyStopping, and ReduceLROnPlateau, with an early stopping patience value of 8 epochs, a learning rate decay coefficient of 0.5, and a minimum learning rate limit of 0.00001. The model checks the validation set accuracy in real time through ModelCheckpoint, automatically saves the model weights with the best accuracy during the iteration process, and outputs the accuracy and loss change curves at the end of training. The resulting model is lightweight, converges smoothly, and can be directly deployed on embedded terminals.

[0036] like Figure 3 The diagram shown illustrates the generation of four-domain and three-channel feature maps for the 6843 radar according to an embodiment of the present invention. Figure 3 (a) is a visualization of the fusion of the original point cloud in a single frame, showing the spatial and energy distribution of the gesture target; Figure 3(b) is a schematic diagram of the three-channel feature map splitting, from left to right: horizontal coordinate-velocity fusion channel (XOV), vertical coordinate-velocity fusion channel (YOV), and echo intensity-velocity fusion channel (IOV). Figure 3 (c) is a typical action single frame multi-channel comparison image, showing the characteristics of each channel and the effect of three-channel fusion (RGB Merged).

[0037] Figure 4 This is a schematic diagram of the temporally distributed convolutional neural network-bidirectional long short-term memory network architecture provided in an embodiment of the present invention. In this architecture, the Input Layer is the input layer, Conv2D is a two-dimensional convolutional layer, MaxPooling is a maximum pooling layer, Flattening Modules are flattening modules, Bidirectional LSTM Module is a bidirectional long short-term memory network module, Dropout is a random deactivation layer, and Classification Module is a classification output module. Together, they constitute a complete network structure for spatiotemporal feature extraction and gesture classification. Figure 5 This is a gesture recognition confusion matrix diagram of the four-domain, three-channel input method provided in this embodiment of the invention. The True Label represents the actual label, the horizontal axis represents the model-predicted label, and the values ​​within the matrix represent the recognition accuracy for the corresponding category, used to visually present the classification and recognition effects of the nine gesture categories.

[0038] Figure 6 This is a graph comparing the recognition accuracy of a three-channel input method with that of a traditional two-channel input method, as provided in an embodiment of the present invention. The vertical axis represents recognition accuracy, and the horizontal axis represents the number of training rounds. The two curves correspond to the validation set accuracy of the three-channel scheme and the traditional two-channel scheme, respectively, and are used to compare the recognition performance differences between the two input methods.

[0039] A controlled experiment was used to verify the advantages of the three-channel input: the experimental group had a four-domain three-channel input (xov+yov+iov), while the control group had a traditional two-channel input (xov+yov). Evaluation metrics included overall validation set accuracy, macro-average precision, weighted average precision, and macro-average F1 score.

[0040] Experimental results show that all indicators of the four-domain three-channel input are significantly higher than those of the traditional dual-channel input, and the recognition accuracy is improved by 5% to 10% compared with the dual-channel input. The introduction of echo intensity i-domain features effectively improves the ability to express gesture features, and there are no obvious misjudgments in the recognition of subtle gestures across scenes.

[0041] In this embodiment, the 6843 radar collects 200 sets of each gesture action, resulting in a total of 1800 raw bin files for the nine original actions. After enhancement and window slicing, the total sample size is sufficient for each frame, the training curve converges smoothly, and the confusion matrix shows no obvious misclassification, fully meeting the non-contact high-precision gesture recognition requirements for various scenarios such as smart homes, in-vehicle interaction, and general human-computer interaction.

[0042] Example 2 For example, this embodiment builds a multi-scene acquisition system for a 6843 mmW radar by designing gesture actions and radar parameters. The specific process is as follows:

[0043] (1) Nine gestures to be recognized are identified: waving inward, waving outward, rotating clockwise, rotating counterclockwise, clicking, drawing an X, swiping right, swiping left, and waving left and right. These gestures cover mainstream human-computer interaction scenarios and are universal;

[0044] (2) A general gesture acquisition platform was built using the TI IWR6843ISK millimeter-wave radar, which can be flexibly deployed in three scenarios: smart home control panel, vehicle cockpit, and general human-computer interaction terminal. The radar installation height is 0.8~1.2m, the acquisition distance is 0.3~1m, and the working frequency band is 76~81GHz. (3) Configure radar acquisition parameters: The radar uses two transmitting antennas (Tx0 and Tx1) and four receiving antennas (Rx0~Rx3). The core acquisition parameters are: starting frequency 60GHz, frequency modulation slope 166.022MHz / μs, ADC sampling points 256, sampling rate 12500ksps, idle time 7.00μs, Tx start time 1.00μs, ADC start time 3.00μs, ramp end time 24.00μs, receiving gain 30dB, each frame contains 128 chirp cycles, 50 frames are acquired per action, and the frame period is 40ms. Radar point calibration and data transmission link debugging are completed to ensure that the raw point cloud data is stably output to the computer.

[0045] For example, the point cloud data acquisition and four-domain three-channel feature map construction process may specifically include: the tester stands in front of the acquisition radar and collects nine types of actions: waving inwards, waving outwards, rotating clockwise, rotating counterclockwise, clicking, drawing an X, swiping right, swiping left, and waving left and right. The 6843 radar outputs raw point cloud data in real time, extracting four-dimensional features frame by frame: horizontal azimuth coordinate x, vertical distance coordinate y, echo intensity i, and motion speed v; where x is the horizontal coordinate in the radar coordinate system (perpendicular to the radar radial direction, corresponding to the azimuth direction), and y is the vertical depth coordinate in the radar coordinate system (along the radar beam radial direction, corresponding to the distance direction); the four-domain feature dimension of a single frame point cloud is (N,4), where N is the number of point clouds in a single frame. Two-dimensional grid projection is performed on each of the four-domain features, with a uniform grid size of 32×32. The process may include:

[0046] The x-domain features and velocity v-domain features are weighted and fused, and projected to generate a 32×32 feature map; the y-domain features and velocity v-domain features are weighted and fused, and projected to generate a 32×32 feature map; the echo intensity i-domain features and motion velocity v-domain features are weighted and fused, and projected to generate a 32×32 feature map. These three single-channel feature maps are stacked along the channel dimension to generate a (32,32,3) three-channel radar feature image, which serves as the standard single-frame input to the neural network.

[0047] The time series data preprocessing process may include: (1) The three-channel feature maps of consecutive frames are cut by a sliding window with a window length of 20 frames and a step size of 10 frames to generate time-series samples of (20,32,32,3); (2) Stratified sampling was used to divide the training set and validation set into a 6:4 ratio to ensure that the proportion of each type of gesture sample was consistent. The training set was augmented by adding Gaussian noise and amplitude scaling, while the validation set was not augmented and only standardized.

[0048] The above describes one or more embodiments of the gesture recognition method based on millimeter-wave radar provided in this specification. Based on the same idea, this specification also provides a corresponding gesture recognition device based on millimeter-wave radar to execute the gesture recognition method based on millimeter-wave radar provided above.

[0049] Specific limitations regarding the millimeter-wave radar-based gesture recognition device can be found in the above description of the limitations of the millimeter-wave radar-based gesture recognition method, and will not be repeated here. Each module in the aforementioned millimeter-wave radar-based gesture recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0050] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described millimeter-wave radar-based gesture recognition method.

[0051] The present invention also provides Figure 7 The schematic diagram of the computer device shown is as follows: Figure 7 As shown, at the hardware level, the computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the millimeter-wave radar-based gesture recognition method provided in the above embodiment.

[0052] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0053] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

Claims

1. A gesture recognition method based on millimeter-wave radar, characterized in that, include: Based on mid-frequency echo data of historical human hand gestures acquired by millimeter-wave radar, enhanced point cloud data containing spatial, velocity, and echo intensity domains is generated. The spatial domain and echo intensity domain of the enhanced point cloud data are fused and projected with the velocity domain to construct a three-channel point cloud feature image sequence; the three-channel point cloud feature image sequence is then processed by temporal slicing to generate fixed-length temporal feature samples. The constructed temporal neural network is trained using temporal feature samples. The three-channel temporal features of the gesture to be tested are input into the trained temporal neural network. The underlying spatial contour features of the gesture are extracted frame by frame from the three-channel temporal features to determine the temporal spatial features. Deep spatial features are determined by extracting high-level semantics and detailed motion features of gestures from temporal spatial features; The deep spatial features are analyzed and fused in both forward and reverse temporal order to determine the global temporal features. The global temporal features are then mapped to a multi-dimensional gesture category space to obtain the corresponding action category probabilities. The category with the highest action category probability is taken as the final gesture recognition result.

2. The gesture recognition method based on millimeter-wave radar as described in claim 1, characterized in that, The acquisition of the enhanced point cloud data specifically includes: The acquired intermediate frequency echo data is filtered to obtain the filtered intermediate frequency echo data; A fast Fourier transform is performed on the filtered intermediate frequency echo data to generate a range-Doppler matrix containing target distance and velocity information; Based on the amplitude information of the extracted distance-Doppler matrix, candidate detection points are generated. The DBSCAN density clustering algorithm is used to cluster the candidate detection points to generate an enhanced point cloud containing time, space, velocity and intensity information.

3. The gesture recognition method based on millimeter-wave radar as described in claim 1, characterized in that, The step of performing temporal slicing on the three-channel point cloud feature image sequence to generate fixed-length temporal feature samples specifically includes: A sliding window strategy is used to perform temporal slicing on the three-channel point cloud feature image sequence. The window length is set to 20 frames and the sliding step is 10 frames. Starting from the beginning position of the three-channel point cloud feature image sequence, 20 consecutive frames of feature images are extracted as an independent subsequence until all valid windows that can be extracted are traversed. Finally, a fixed-length temporal feature sample with the format (20,32,32,3) is generated.

4. The gesture recognition method based on millimeter-wave radar as described in claim 1, characterized in that, The temporal neural network specifically includes: a low-level feature extraction module, a high-level semantic feature extraction module, a temporal dependency learning module, and a fully connected classification output module. Specifically, the low-level feature extraction module extracts the low-level spatial contour features of the gesture from the three-channel temporal features frame by frame to determine the temporal spatial features. The high-level semantic feature extraction module extracts the high-level semantic and detailed motion features of the gesture from the temporal spatial features to determine the deep spatial features. The temporal dependency learning module performs forward and reverse temporal feature analysis on the deep spatial features and fuses them to determine the global temporal features. The fully connected classification output module maps the global temporal features to a multi-dimensional gesture category space, obtains the corresponding action category probabilities, and uses the category corresponding to the maximum action category probability as the final gesture recognition result.

5. The gesture recognition method based on millimeter-wave radar as described in claim 1, characterized in that, Before training the constructed temporal neural network using temporal feature samples, the temporal feature samples are subjected to data augmentation processing.

6. The gesture recognition method based on millimeter-wave radar as described in claim 1, characterized in that, During training, the temporal neural network was compiled using the Adam optimizer with a learning rate of 0.002 and a loss function of classification cross-entropy.

7. The gesture recognition method based on millimeter-wave radar as described in claim 1, characterized in that, The millimeter-wave radar is a TI IWR6843 millimeter-wave radar, which uses a configuration of two transmitting antennas and four receiving antennas.

8. A gesture recognition device based on millimeter-wave radar, characterized in that, Implement the gesture recognition method based on millimeter-wave radar as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the gesture recognition method based on millimeter-wave radar as described in any one of claims 1-7.

10. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the gesture recognition method based on millimeter-wave radar as described in any one of claims 1-7.

Images