Computer-implemented method for training a millimeter wave radar gesture recognition device

Abstract

A computer-implemented method for training a millimeter wave radar gesture recognition device. The method includes obtaining raw inertial measurement unit (IMU) data and raw millimeter wave (mmWave) data, extracting IMU features and mmWave features from the raw IMU data and the raw mmWave data, respectively, training a diffusion model using the IMU features and the mmWave features, converting an IMU dataset using the trained diffusion model to obtain mmWave heatmaps, and training the millimeter wave radar gesture recognition device using the obtained mmWave heatmaps. The method allows for accurate conversion of IMU features to mmWave features, thereby enhancing gesture recognition capabilities.

Description

Technical Field

[0001] This invention relates to gesture recognition systems and methods, particularly millimeter-wave-based gesture recognition systems and methods. Background Technology

[0002] Millimeter-wave-based gesture recognition systems utilize millimeter-wave radar sensors to detect and recognize hand gestures. This technology has several key advantages. First, compared to other types of recognition devices, millimeter-wave radar offers high-resolution gesture detection, capturing even minute hand movements. Furthermore, unlike optical sensors, millimeter-wave radar is not limited by lighting conditions and can operate normally in harsh environments such as darkness, rain, and fog. Finally, millimeter-wave radar does not capture images, thus offering advantages in privacy protection.

[0003] However, despite the many advantages of millimeter-wave gesture recognition systems, existing millimeter-wave gesture recognition devices still have some shortcomings. First, in complex real-world scenarios, millimeter-wave radar is easily affected by reflections and interference from other objects, impacting recognition accuracy. Second, achieving real-time gesture recognition requires efficient algorithms and powerful computing capabilities, which may be difficult to implement on resource-constrained devices. Third, current feature extraction and classification algorithms still need further optimization when handling diverse gestures to improve generalization ability and recognition accuracy.

[0004] References

[0005] The following references mentioned in this article are indicated by numbers in parentheses. For all purposes, the full contents of these references are hereby included by way of citation.

[0006] [1]Ismail Khalfaoui-Hassani, Thomas Pellegrini, and Timothée Masque-lier. 2023. Dilated convolution with learnable spacings. In ICLR.

[0007] [2]Bo Li,Kaitao Xue,Bin Liu,and Yu-Kun Lai.2023.BBDM:Image-to-imagetranslation with Brownian bridge diffusion models.In IEEE / CVF CVPR.

[0008] Purpose of the invention

[0009] According to a first aspect of the present invention, a computer-implemented method for training a millimeter-wave radar gesture recognition device is provided. The method includes: acquiring raw inertial measurement unit (IMU) data and raw millimeter-wave (mmWave) data; extracting IMU features and millimeter-wave features from the raw IMU data and the raw millimeter-wave data, respectively; training a diffusion model using the IMU features and the millimeter-wave features; converting the IMU dataset into a millimeter-wave heatmap using the trained diffusion model; and training the millimeter-wave radar gesture recognition device using the obtained millimeter-wave heatmap.

[0010] Preferably, the above method includes a step of preprocessing the raw inertial measurement unit data and raw millimeter-wave data to reduce noise before the step of extracting inertial measurement unit features and millimeter-wave features.

[0011] Preferably, the step of extracting millimeter-wave features includes performing a Fourier transform on the original millimeter-wave data to obtain a range-Doppler map, and converting the range-Doppler maps of all frames in which the user performs the gesture into a two-dimensional time-velocity feature map, thereby obtaining the millimeter-wave features.

[0012] More preferably, the above-mentioned Fourier transform includes range fast Fourier transform and Doppler fast Fourier transform.

[0013] In one exemplary embodiment, the step of extracting inertial measurement unit (IMU) features includes: decomposing acceleration data into multiple levels, calculating approximation coefficients and detail coefficients for each level, reconstructing the IMU signal using the approximation coefficients and detail coefficients, and obtaining a spectrum of multiple directions from the IMU signal to obtain the IMU features.

[0014] Preferably, the step of decomposing the acceleration data into multiple levels is performed using the maximum overlap discrete wavelet transform.

[0015] Preferably, the step of obtaining the spectrum in multiple directions is performed using a short-time Fourier transform.

[0016] In another exemplary implementation, the diffusion model includes an inertial fusion module and a diffusion-based translation module.

[0017] Preferably, the inertial fusion module includes a learning dilated convolutional neural network (LDCNN) and a gating module.

[0018] Preferably, the diffusion-based translation module is based on Brownian bridge diffusion.

[0019] In another exemplary embodiment, the steps of training the diffusion model include: obtaining a fixed mapping from the inertial measurement unit dataset to the millimeter-wave heatmap through a forward process; and eliminating noise and drift through a reverse process.

[0020] Preferably, the steps of training the diffusion model include jointly training the forward process, the backward process, and the inertial fusion module.

[0021] According to a second aspect of the present invention, a method for performing millimeter-wave radar gesture recognition is provided, comprising: recognizing a user's gesture using millimeter-wave radar to capture millimeter-wave gesture data; and recognizing the captured millimeter-wave gesture data using a millimeter-wave radar gesture recognition device trained according to the method of the first aspect described above.

[0022] Preferably, the millimeter-wave radar is configured in the millimeter-wave radar gesture recognition device.

[0023] According to a third aspect of the invention, a system for training a millimeter-wave radar gesture recognition device is provided. The system includes one or more processors and a memory storing one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for performing or facilitating the execution of the computer-implemented method of the first aspect described above.

[0024] According to a fourth aspect of the invention, a non-transitory computer-readable medium is provided, on which instructions are stored; when executed by one or more processors, these instructions cause the one or more processors to perform the computer implementation method of the first aspect described above.

[0025] In another aspect of the invention, a method for converting IMU data into mmWave data for gesture recognition is provided, wherein a cross-modal learning layer is deployed in a first device and a translation layer is deployed in a second device, each layer performing data processing and conversion. The method includes: a) preprocessing raw IMU data and mmWave data in the first device to reduce noise and improve data quality; b) generating IMU features from the preprocessed IMU data and mmWave features from the preprocessed mmWave data using a signal processing algorithm; c) training a diffusion model using the generated IMU and mmWave features to achieve IMU-to-mmWave conversion; d) converting the IMU data into mmWave data using the trained diffusion model to generate an mmWave heatmap; e) training a gesture recognition network using the generated mmWave heatmap in the second device; and f) capturing real-time gesture data and performing real-time gesture recognition using a device embedded with mmWave radar.

[0026] Preferably, the IMU features are generated using maximum overlap discrete wavelet transform (MODWT) and short time Fourier transform (STFT), and the mmWave features are generated using range FFT and Doppler FFT to generate frequency-time maps.

[0027] Preferably, the diffusion model includes an inertial fusion module in the first device, which uses a learnable dilated convolutional neural network (LDCNN) to extract features from the IMU spectrogram. The extracted features are processed by a gating mechanism to selectively fuse information from the three orthogonal axes of the IMU data.

[0028] More preferably, the gating mechanism uses a fully connected fusion layer to integrate features derived from LDCNN and assign different weights to these features, thereby enhancing the sensitivity and specificity of gesture recognition.

[0029] In one specific implementation, the diffusion model is a Brownian bridge diffusion model, which includes a two-sided frame that effectively bridges the data mode gap between IMU and mmWave data.

[0030] Exemplary embodiments of the present invention therefore propose a method for uniquely integrating cross-modal deep diffusion model IMU and mmWave data, allowing for precise conversion of IMU features into mmWave features, thereby enhancing gesture recognition capabilities. In this method, an advanced inertial fusion module is implemented, employing LDCNN and gating mechanisms to provide flexible and adaptive feature extraction for IMU features. Furthermore, a bridging diffusion translation method based on a Brownian bridge diffusion model is introduced, specifically designed to bridge the modal gap between the IMU spectrogram and the mmWave heatmap, improving the fidelity and accuracy of signal conversion.

[0031] Other features and aspects of the invention will become apparent from the following detailed description and accompanying drawings. Any feature described herein that relates to one aspect or embodiment may be combined, as needed, with any other feature described herein that relates to any other aspect or embodiment. Attached Figure Description

[0032] Embodiments of the present invention will now be described by way of example and in conjunction with the accompanying drawings, wherein:

[0033] Figure 1 The present invention illustrates a system and method steps for training a millimeter-wave radar gesture recognition device according to an embodiment of the present invention, and steps for applying the trained millimeter-wave radar gesture recognition device.

[0034] Figure 2 The structure of an IMU-to-radar (I2R) conversion according to an embodiment of the present invention is shown.

[0035] Figure 3 An example of an information processing system is shown that can be used in embodiments of the present invention for training one or more methods of a millimeter-wave radar gesture recognition device.

[0036] Before explaining any embodiments of the invention in detail, it should be understood that the application of the invention is not limited to the implementation details and component arrangements shown in the following description or drawings. The invention can be implemented or performed in various other ways. Furthermore, it should be understood that the wording and terminology used in this specification are for descriptive purposes only and should not be considered limiting conditions. Detailed Implementation

[0037] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Figure 1 An overview of a cross-modal gesture recognition framework (hereinafter referred to as iRadar) according to an embodiment of the present invention is shown, which includes three layers: a cross-modal learning layer 20, a conversion layer 22, and an application layer 24. The cross-modal learning layer 20 and the conversion layer 22 can reside in the same information processing system (e.g., in the following...). Figure 3 The example information processing system to be described can also be located in different devices. The application layer 24 can be configured in the same information processing system as the cross-modal learning layer 20 and / or the conversion layer 22, or in another information processing system (e.g., a user's smart personal portable electronic device). Both the cross-modal learning layer 20 and the conversion layer 22 are used for training purposes, including training the diffusion model 28 and the gesture recognition network 30.

[0038] In the cross-modal learning layer 20, a deep diffusion model is designed to convert IMU features into mmWave features. First, the raw IMU and mmWave data are preprocessed (26) to reduce noise. Then, the proposed signal processing algorithm is used to generate IMU and mmWave features from each dataset, i.e., feature extraction step 32. Finally, the aforementioned diffusion model 28 is trained to perform the IMU-to-mmWave conversion. In the conversion layer 22, service providers can choose to use publicly available or proprietary IMU datasets. The data from these datasets are then processed using a trained IMU-to-radar diffusion model (I2R, e.g., ...). Figure 2 The data (as shown) is converted into an mmWave heatmap. After conversion, the resulting mmWave heatmap is used to train the gesture recognition network 30. In the application layer 24, users can easily use smart devices embedded with mmWave radar to directly capture mmWave gesture data and perform real-time gesture recognition for various applications.

[0039] The following will be about Figure 1 The components and steps of the cross-modal gesture recognition framework are explained in detail.

[0040] First, let's look at training the diffusion model 28. Training the diffusion model 28 requires IMU features and mmWave features extracted from the raw IMU data and mmWave data, respectively. Let's start with the mmWave features. The raw mmWave data is obtained by mixing intermediate frequency (IF) signals, which are ultimately used to sense gestures. Besides the user's gesture information, the IF signal also contains a significant amount of static noise generated by static objects such as walls, tables, and chairs. For each frame of the user's gesture action, the average of all IF signals is used as the static noise vector, and this static noise vector is subtracted from each IF signal to obtain the denoised data. Then, Range Fast Fourier Transform (Range FFT) and Doppler Fast Fourier Transform (Doppler FFT) are used to obtain the range-Doppler map (RDM) for each frame, which reflects the range and velocity information of the user performing the gesture in the current frame. Finally, to obtain the velocity change information during the gesture execution, the RDM of all frames of the user's gesture action is converted into a two-dimensional time-velocity feature map using the following equation:

[0041]

[0042] Where N r and N d These are the number of range FFTs and Doppler FFTs, respectively, B j It is a range partitioned index, RDM (n,i,j) This represents the values ​​corresponding to Doppler partition i and range partition j in the nth frame of the RDM. The Doppler FFT responds to changes in phase difference, which are proportional to the frequency. Therefore, the normalized frequency can be derived by normalizing the extracted velocity. The two-dimensional time-velocity feature map described above corresponds to the extracted millimeter-wave features.

[0043] On the other hand, let's look at the IMU features, which are obtained by generating an IMU spectrogram. In iRadar, to isolate gesture-related signals, the acceleration data along the X, Y, and Z axes is decomposed into four levels using Maximum Overlap Discrete Wavelet Transform (MODWT). Specifically, the IMU signal undergoes this decomposition process to obtain approximation coefficients α and detail coefficients β. The formula for calculating the approximation coefficients α(0,k) of level 0 is:

[0044]

[0045] Where N is the length of the input signal x(n), and ψ(0,nk) represents the wavelet function value at time (nk) in level 0. Subsequently, the approximation coefficients α and detail coefficients β are calculated iteratively using the following formula:

[0046]

[0047] Among them W h (j,n) and W l (j, n) represent the wavelet functions of the high-pass and low-pass filters at level j, respectively. Preferably, the Symlets 7 wavelet (sym7) is selected, and the original signal is decomposed into four levels. Subsequently, a threshold is applied to the detail coefficients β at each level to eliminate ambient clutter. Finally, the IMU signal is reconstructed as follows:

[0048]

[0049] in The approximate wavelet function at level J is represented by ψ(j,n), and the detail wavelet function at level j is represented by ψ(j,n). Finally, the three processed IMU signals are used to generate three spectrograms through short-time Fourier transform (STFT), obtained from three directions respectively. These spectrograms are equivalent to the extracted IMU features, which are used as inputs to the inertial fusion module.

[0050] Next, we will introduce the IMU-to-radar (I2R) diffusion model28, which is designed to convert IMU spectrograms into mmWave heatmaps. Figure 2 The structure of I2R is shown, which includes an inertial fusion module 34 and a diffusion-based translation module 36.

[0051] IMU data contains rich information on signal frequency and motion intensity. Furthermore, the spectral shape of IMU data can vary significantly between different hand gestures. Therefore, effectively modeling and extracting features from IMU spectrograms poses a major challenge. To address this, we propose an inertial fusion module incorporating a learned dilated convolutional neural network (LDCNN)—a novel convolutional method for extracting features from IMU spectrograms. Additionally, a gating mechanism assigns different weights to the features extracted by LDCNN.

[0052] In LDCNN, the positions of non-zero elements in the convolutional kernel are learned via gradient methods. However, since the positions in the kernel are integer values, this poses a challenge in terms of differentiability. To overcome this problem, one embodiment of the present invention uses an interpolation method. The main motivation for LDCNN is to explore the potential to augment the fixed grid imposed by standard DConv by learning intervals independent of the input. Unlike the grid-like arrangement of kernel elements in standard and dilated convolutions, LDCNN (in...) Figure 2 The LDConv symbol allows for a flexible number of nuclear elements[1]:

[0053]

[0054] Where O(σ) is the output feature map, I(σ) is the input feature map, and K(σ) is the input feature map. ′) is the convolution kernel, S is the neighborhood around pixel σ, and L(σ) is a learnable dilation rate function that is updated via backpropagation.

[0055] Following the LDCNN stage, the extracted features are input into gating module 38. The purpose of this module is to selectively fuse information, using a gating mechanism to filter and combine relevant features from the three orthogonal axes of the IMU data. For the IMU spectrogram features extracted by LDCNN, gating module 38 operates as follows:

[0056] G(s 1 ,s 2 ,s 3 ) = F(LDCNN(s) 1 ,s 2 ,s 3 );η),

[0057] Where s 1 ,s 2 ,s 3 η is the input IMU spectrogram, G(·) represents the gated feature output, F(·) represents a fully connected fusion layer used to integrate features from LDCNN, and η contains the learned parameters.

[0058] Now let’s look at the diffusion-based translation module 36. Existing diffusion models typically treat the translation task as a conditional generation process, which does not effectively bridge the gap between different modal domains. To address these shortcomings, one embodiment of the present invention uses a novel method based on Brownian Bridge Diffusion (BBDM) [2]. This method includes a bilateral framework specifically designed to bridge the gap between IMU data and mmWave data. By taking into account the correlation and unique features of the IMU spectrogram and the mmWave heatmap, BBDM effectively captures the subtle mapping relationship between the two. Figure 2 The process of proposing the translation method is demonstrated, including the forward and backward processes.

[0059] The forward process describes the diffusion of the IMU spectrogram to the mmWave heatmap, see [link to relevant documentation]. Figure 2 Box 40 illustrates this process, which begins with the IMU spectrogram, gradually incorporating noise and drift, and transitioning to an mmWave heatmap. The IMU spectrogram consists of a set of inputs s = {s...} 1 ,s 2 ,s 3 Let (s, h) represent the paired training data of the IMU spectrogram and mmWave heatmap. The ground truth mmWave heatmap is conditionally input into h as its destination. Assume s and h are approximately independent and normally distributed, i.e. Given an initial state s0 and intermediate states s tGiven the target state h, the forward diffusion process of the Brownian bridge can be defined as follows:

[0060]

[0061] Where T is the total number of steps in the diffusion process, δ t It's the variance. For training and inference purposes, the forward transition probability q needs to be derived. BB (x t |x t-1 ,h):

[0062]

[0063] When the diffusion process reaches its destination, i.e., t = T, we can obtain m. T =1. The forward diffusion process defines a fixed mapping from the IMU spectrogram to the mmWave heatmap.

[0064] On the other hand, the back-diffusion process, also shown in box 40, can be used to infer the initial IMU spectrogram that may lead to the observed mmWave heatmap, minimizing the difference by utilizing UNet to learn the mapping function between the IMU spectrogram and its corresponding mmWave heatmap. It acts as the inverse of the forward diffusion process. Starting from the mmWave heatmap, the back-diffusion process gradually eliminates noise and drift through reverse operations, restoring the distribution towards the IMU spectrogram. Unlike existing diffusion models, the Brownian bridge process directly draws from the conditional input s T =h starts. The reverse process is designed based on s t Predictions t-1 :

[0065]

[0066] Where μ0(s) t ,t) is the predicted mean of the noise. This is the noise variance at each step. Mean μ0(s) t ,t) needs to be learned through a neural network based on the parameter θ according to the maximum likelihood criterion.

[0067] In a preferred embodiment, the inertial fusion module can use the input IMU spectrogram as the source domain and the mmWave heatmap as the target domain to train the diffusion model. Furthermore, a U-Net neural network is implemented and used in the backward propagation process. The U-Net architecture consists of four encoders and four decoders. Each decoder consists of a convolutional block containing two convolutional layers, using ReLU as the activation function and max pooling for pooling operations. Notably, the inertial fusion module, the forward propagation process, and the backward propagation process are closely interconnected and trained together to optimize overall performance.

[0068] The trained gesture recognition network can then be used in the application layer of iRadar. For example, a user's smart device, which may include millimeter-wave radar, can connect to the gesture recognition network. When the user performs an activity, the smart device uses the millimeter-wave radar to recognize the user's gestures to capture millimeter-wave gesture data, and then uses the gesture recognition network to recognize the captured millimeter-wave gesture data. The smart device can therefore be viewed as a millimeter-wave radar gesture recognition device.

[0069] As can be seen, the exemplary embodiments of the present invention described above propose a novel cross-modal gesture recognition framework called iRadar, which synthesizes radar signals for corresponding gestures using inertial measurement unit (IMU) data. We propose a method to generate radar signals from millimeter-wave (mmWave) radar when the same gesture is performed, utilizing IMU signals commonly found in modern wearable devices. Furthermore, we develop a diffusion-based IMU-to-radar translation model that accurately converts IMU data into mmWave data. This proposed method can be used in smart device systems to enhance mmWave radar-based gesture recognition, significantly advancing intuitive human-computer interaction technology. Compared to existing millimeter-wave-based gesture recognition schemes, iRadar achieves higher efficiency and better user-friendliness.

[0070] Figure 3An example of an information processing system 1100 is shown, which can be used in embodiments of the present invention for training one or more methods for a millimeter-wave radar gesture recognition device. The information processing system 1100 typically includes suitable components required to receive, store, and execute appropriate computer instructions, commands, and / or code. The main components of the information processing system 1100 include a processor 1102 and a memory 1104. The processor 1102 may include one or more CPUs, MCUs, GPUs, logic circuits, Raspberry Pi chips, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or any other digital or analog circuits for interpreting and / or executing program instructions and / or processing signals and / or information and / or data. The memory 1104 may include one or more volatile memories (e.g., RAM, DRAM, SRAM, etc.), one or more non-volatile memories (e.g., ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM, etc.), or any combination thereof. Appropriate computer instructions, commands, codes, information, and / or data may be stored in the memory 1104. Computer instructions for performing or facilitating the execution of embodiments of the methods of the present invention may be stored in memory 1104. Processor 1102 and memory 1104 may be integrated or separate (and operatively connected). Optionally, information processing system 1100 further includes one or more input devices 1106. Examples of such input devices 1106 include: keyboard, mouse, stylus, image scanner, microphone, haptic / touch input device (e.g., touchscreen), image / video input device (e.g., camera), etc. Optionally, information processing system 1100 further includes one or more output devices 1108. Examples of such output devices 1108 include: display (e.g., monitor, screen, projector, etc.), speaker, headphones, printer, additive manufacturing machine (e.g., 3D printer), etc. The display may include a liquid crystal display, LED / OLED display, or other suitable display, and may or may not be touch-sensitive. Information processing system 1100 may further include one or more disk drives 1112, which may include one or more of the following: solid-state drives, hard disk drives, optical drives, flash drives, tape drives, etc. A suitable operating system can be installed in the information processing system 1100, for example, on the disk drive 1112 or in the memory 1104. The processor 1102 is operable on the memory 1104 and the disk drive 1112. Optionally, the information processing system 1100 also includes a communication device 1110 for establishing one or more communication links (not shown) with one or more other computing devices (e.g., servers, personal computers, terminals, tablets, telephones, watches, Internet of Things devices, or other wireless computing devices).Communication device 1110 may include one or more of the following: modem, network interface card (NIC), integrated network interface, NFC transceiver, ZigBee transceiver, Wi-Fi transceiver, Bluetooth transceiver, radio frequency transceiver, cellular (2G, 3G, 4G, 5G, 5G and above, or similar) transceiver, optical port, infrared port, USB connection, or other wired or wireless communication interface. The transceiver may be implemented by one or more devices (integrated transmitter and receiver, separate transmitter and receiver, etc.). The communication link may be wired or wireless and used for communicating commands, instructions, information, and / or data. In one example, processor 1102, memory 1104 (optionally input device 1206, output device 1108, communication device 1110, and disk drive 1112 (if any)) are directly or indirectly interconnected via a bus, peripheral interconnect (PCI), such as PCI Express, Universal Serial Bus (USB), fiber optic bus, or other similar bus structures. In one embodiment, at least some of these components may be wirelessly connected, for example, via a network such as the Internet or a cloud computing network. Those skilled in the art will understand that... Figure 3 The information processing system 1100 shown is only an example. In other embodiments, the information processing system 1100 may have different configurations (e.g., including additional components, having fewer components, etc.).

[0071] While not strictly necessary, one or more embodiments depicted in the figures can be implemented as an application programming interface (API) or a series of libraries for use by developers, and can also be included in another software application, such as a terminal or computer operating system or a portable computing device operating system. In one or more embodiments, because program modules include routines, programs, objects, components, and data files that help perform specific functions, those skilled in the art will understand that the functionality of a software application can be distributed across multiple routines, objects, and / or components to achieve the same functionality required herein.

[0072] It should also be understood that if the methods and systems of the present invention are implemented entirely or partially by a computing system, any suitable computing system architecture can be used. This will include stand-alone computers, network computers, dedicated or non-dedicated hardware devices. When the terms "computing system" and "computing device" are used, these terms are intended to include (but are not limited to) any arrangement of suitable computer or information processing hardware capable of implementing the described functions.

[0073] Those skilled in the art will understand that various changes and / or modifications can be made to the described and / or illustrated embodiments of the invention to provide other embodiments of the invention. Therefore, the described and / or illustrated embodiments of the invention should be considered illustrative in all respects and not restrictive. Optional features of certain embodiments of the invention are provided in the abstract and description. Certain embodiments of the invention may include one or more of these optional features (some of which are not specifically illustrated in the drawings). Certain embodiments of the invention may lack one or more of these optional features (some of which are not specifically illustrated in the drawings).

Claims

1. A computer implementation method for training a millimeter-wave radar gesture recognition device, comprising: Acquire raw inertial measurement unit data and raw millimeter-wave data; Inertial measurement unit features and millimeter-wave features are extracted from the original inertial measurement unit data and the original millimeter-wave data, respectively; The diffusion model is trained using the inertial measurement unit features and the millimeter wave features; The trained diffusion model was used to convert the inertial measurement unit dataset into millimeter-wave thermal maps; and The obtained millimeter-wave heatmaps were used to train the millimeter-wave radar gesture recognition device.

2. The method according to claim 1, further comprising: Prior to the step of extracting inertial measurement unit features and millimeter-wave features, a step of preprocessing the raw inertial measurement unit data and the raw millimeter-wave data to reduce noise is performed.

3. The method according to claim 1, wherein the step of extracting the millimeter-wave features includes: Perform a Fourier transform on the raw millimeter-wave data to obtain a range-Doppler plot; The range-Doppler maps of all frames in which the user performs the gesture are converted into two-dimensional time-velocity feature maps to obtain the millimeter-wave features.

4. The method according to claim 3, wherein the Fourier transform includes range fast Fourier transform and Doppler fast Fourier transform.

5. The method according to claim 1, wherein the step of extracting the features of the inertial measurement unit includes: Decompose the acceleration data into multiple levels; Calculate the approximation coefficient and detail coefficient for each of the levels; The inertial measurement unit signal is reconstructed using the approximation coefficients and the detail coefficients; and The inertial measurement unit (IMU) signal is used to obtain spectrum diagrams in multiple directions, thereby obtaining the characteristics of the IMU.

6. The method of claim 5, wherein the step of decomposing the acceleration data into multiple levels is performed using maximum overlap discrete wavelet transform.

7. The method of claim 5, wherein the step of obtaining the spectrum of the plurality of directions is performed using a short-time Fourier transform.

8. The method according to claim 1, wherein the diffusion model includes an inertial fusion module and a diffusion-based translation module.

9. The method according to claim 8, wherein the inertial fusion module comprises a learned dilated convolutional neural network and a gating module.

10. The method of claim 8, wherein the diffusion-based translation module is based on Brownian bridge diffusion.

11. The method of claim 10, wherein the step of training the diffusion model comprises: A fixed mapping from the inertial measurement unit dataset to the millimeter-wave thermal image is obtained through a forward process; Noise and drift are eliminated through a reverse process.

12. The method of claim 11, wherein the step of training the diffusion model includes jointly training the forward process, the backward process, and the inertial fusion module.

13. A method for performing millimeter-wave radar gesture recognition, comprising: Millimeter-wave radar is used to recognize user gestures in order to capture millimeter-wave gesture data; as well as The captured millimeter-wave gesture data is recognized using a millimeter-wave radar gesture recognition device trained according to the method described in claim 1.

14. The method of claim 3, wherein the millimeter-wave radar is configured in the millimeter-wave radar gesture recognition device.

15. A system for training a millimeter-wave radar gesture recognition device, comprising: One or more processors; as well as Memory; Store one or more programs configured to be executed by the one or more processors; The one or more programs include instructions for performing or facilitating the performance of the computer-implemented method of claim 1.

16. A non-transitory computer-readable medium having instructions stored thereon; which, when executed by one or more processors, cause the one or more processors to perform the computer-implemented method of claim 1.

Images