Signal simulation system and method for millimeter wave data enhancement

Abstract

The invention discloses a millimeter wave data enhancement-oriented signal simulation system, and the system comprises a human body reflection modeling module which is used for obtaining a fine-grained three-dimensional human body SMPL model for describing a body type from a human body motion database, and then carrying out the preprocessing of the model; the signal reflection modeling module is used for performing signal simulation according to the propagation principle of millimeter wave signals, constructing chirp signals transmitted by the millimeter wave radar, reflecting the chirp signals through the surface of a human body, returning the chirp signals to a receiving antenna of the millimeter wave radar, and finally mixing the chirp signals with the transmitted signals in a frequency mixer to form intermediate frequency signals; the perception feature extraction module is used for performing basic signal processing operation on the intermediate frequency signal to obtain perception features for neural network training; and the neural network training module is used for training a corresponding network by using the micro-Doppler features and the three-dimensional point cloud features so as to realize two perceptual applications of behavior recognition and skeleton tracking. The invention provides a signal simulation technology for millimeter wave data enhancement, provides data support for a millimeter wave sensing scene needing a large amount of training data, and effectively reduces the burden of training data collection.

Description

Technical Field

[0001] The present invention relates to a signal simulation technology for millimeter-wave data enhancement, belonging to the field of wireless communication technology. Background Art

[0002] Non-contact sensing technology has received extensive attention because it can achieve ubiquitous sensing in rich environments such as smart homes and smart healthcare without the need for users to carry wearable devices, reducing the user's burden. Vision-based sensing has the advantage of high accuracy and has been widely used in all aspects of life, but it may compromise user privacy. In contrast, sensing through wireless signals is generally considered a more balanced approach that can simultaneously focus on accuracy and privacy security.

[0003] Wireless sensing based on millimeter-wave radar has made great progress in the past decade. It emits frequency-modulated continuous waves (FMCWs) in the millimeter-wave band (usually in the frequency range of 30 GHz to 300 GHz) for target detection and has three attractive advantages. First, the high frequency brings sub-millimeter displacement sensitivity, which means that millimeter-wave radar can very precisely detect and measure the tiny movements of target objects. In addition, the high-frequency characteristics allow antenna arrays with high angular resolution to be placed at a lower spatial cost, enabling millimeter-wave radar to be deployed in environments with limited space. Second, the wider bandwidth used by millimeter-wave radar provides higher range resolution, enabling accurate differentiation of the distances between different targets. Third, FMCW radar has a strong spatial signal extraction ability. By using frequency-modulated continuous wave technology, millimeter-wave radar can precisely analyze the spatial position and motion state of the target, thereby achieving high-precision detection and tracking of the target. With the above characteristics, millimeter-wave radar has been widely applied to many fine-grained sensing applications.

[0004] In recent years, the rapid development of deep learning has demonstrated its powerful feature learning ability. More and more millimeter-wave sensing works have achieved very good results with the help of neural networks, especially in the fields of human skeleton tracking and behavior recognition, where a large amount of millimeter-wave data usually needs to be collected for training neural networks. However, an important problem they generally have is the scarcity of training datasets, high data acquisition costs, and the need to consume a large amount of human, material, and financial resources. For applications such as pose recognition, it is usually necessary to deploy expensive motion capture cameras to obtain real human skeletons as labels to supervise the training of neural networks. Compared with vision-based datasets, the scarcity of millimeter-wave data greatly limits the ability of deep learning and also brings difficulties in data collection for researchers. Therefore, how to reduce the data burden is an important research topic. Summary of the Invention

[0005] The technical problem to be solved by the present invention is how to provide data support for millimeter-wave sensing scenarios that require a large amount of training data and reduce the burden on researchers for data collection.

[0006] The present invention realizes the solution to the above technical problem through the following technical means: A signal simulation system for millimeter-wave data enhancement, comprising:

[0007] A human body reflection modeling module, which is used to obtain a fine-grained three-dimensional human SMPL model that can accurately describe the body shape from a human body motion database, and then perform preprocessing operations on the three-dimensional human SMPL model so that the human body model can truly reflect the reflection of millimeter-wave signals on the human body surface;

[0008] A signal reflection modeling module, which is used to simulate signals according to the propagation principle of millimeter-wave signals, construct a chirp signal transmitted by a millimeter-wave radar, reflect from the human body surface of the human body model established by the human body reflection modeling module, return to the receiving antenna of the millimeter-wave radar, and finally mix with the transmitted signal in a mixer to become an intermediate-frequency signal;

[0009] A perception feature extraction module, which is used to perform basic signal processing operations on the intermediate-frequency signal output by the signal reflection modeling module to obtain perception features for neural network training: the micro-Doppler spectrum for behavior recognition network training and the three-dimensional human body point cloud for human body skeleton tracking network training;

[0010] A neural network training module, which is used to train the corresponding network using micro-Doppler features and three-dimensional point cloud features, so as to realize two perception applications.

[0011] Furthermore, the human body reflection modeling module includes a personalized human body model acquisition unit and a human body model preprocessing unit,

[0012] Among them, in the personalized human body model acquisition unit, a fine-grained SMPL model is used for signal synthesis, and the pose parameters of the human body model and the body shape parameters obtained from the video of the real user in the standard pose are superimposed and input into the SMPL model together;

[0013] In the human body model preprocessing unit, it performs two preprocessing operations: the first step is occlusion point removal, and the second step is downsampling, and part of the sample points are selected from the original human body model surface vertices according to a certain proportion.

[0014] Furthermore, in the personalized human body model acquisition unit, the HPR occlusion point removal algorithm is used to identify and remove the occluded points through two steps of point cloud inversion and convex hull construction, so as to delete the human body vertices that are invisible from the radar perspective; in MATLAB, downsampling is realized through the downsample function.

[0015] Further, the signal reflection modeling module includes a signal attenuation factor calculation unit and an intermediate frequency signal calculation unit;

[0016] The signal attenuation factor calculation unit is used to calculate the signal attenuation factor. During the signal simulation process, it is necessary to consider the signal attenuation caused by environmental impacts, which is represented by a signal attenuation factor. This signal attenuation factor is related to the propagation distance and antenna gain;

[0017] The intermediate frequency signal calculation unit is used to calculate the intermediate frequency signal, which is calculated according to the signal propagation characteristics.

[0018] Further, the millimeter-wave radar emits a chirp signal. The starting frequency of the chirp signal is fc, the ending frequency is fc + B, the bandwidth is B, and Tc is the duration of the chirp. Then the frequency of the signal at time t is expressed as f(t) = f c + B * t / T c , and the millimeter-wave radar emits multiple chirp signals to form a frame. The transmitted signal is expressed as

[0019]

[0020] where A T is the amplitude of the transmitted signal, is a constant. The signal emitted by the transmitting antenna is reflected on the target surface and received by the receiving antenna. The received signal is expressed as

[0021]

[0022] where α is the signal attenuation factor, which is used to describe the attenuation degree of the signal intensity during the propagation process. τ is the round-trip time between the reflection point and the radar, expressed as τ = (d t + d r ) / c, where d t and d r are the distances between the reflection point and the transmitting antenna and the receiving antenna respectively, and c is the speed of light. Subsequently, the transmitted signal and the received signal are mixed in a mixer to obtain the intermediate frequency signal, expressed as

[0023]

[0024] Through the above formula, the signal of each reflection point under each transmit-receive antenna pair is obtained.

[0025] Further, the attenuation degree of the signal is affected by the antenna gain, propagation distance, and radar cross-section area. The signal attenuation factor α is expressed as

[0026]

[0027] where λ is the wavelength of the signal, d is the distance between the centroid of the triangular patch and the radar, G Tx and G Rx represent the gains of the transmitting and receiving antennas, which reflect the radiation effect of the antenna in a specific direction. According to the settings of the actually used millimeter-wave radar, σ represents the radar cross-section area, which reflects the reflection intensity of the target surface. The radar cross-section area is measured by the area of the triangular patch and the angle θ between the normal of the triangular patch and the line connecting the radar and the centroid of the triangular patch.

[0028] Furthermore, the perception feature extraction module includes a micro-Doppler feature extraction unit and a 3D point cloud feature extraction unit;

[0029] In the micro-Doppler feature extraction unit, perform a fast Fourier transform on multiple consecutive chirp signals, and then extract and concatenate the instantaneous velocity frame by frame to obtain the micro-Doppler spectrum;

[0030] In the 3D point cloud feature extraction unit, perform a distance Fourier transform on the intermediate-frequency signal to separate the reflection points at different distances, and then perform a two-dimensional angular Fourier transform to obtain the azimuth angle and elevation angle of each reflection point. According to the distance, azimuth angle, and elevation angle of the reflection point relative to the radar, calculate the coordinates of the reflection point in the three-dimensional space.

[0031] Furthermore, the neural network training module includes a behavior recognition network unit and a skeleton tracking network unit;

[0032] In the behavior recognition network unit, a lightweight network based on VGG16 is used, only the first two convolutional layers in each sub-module of VGG16 are retained, and a fully connected layer is added at the end of the network;

[0033] In the skeleton tracking network unit, the input is a tensor with dimensions of batch size * frame length * number of point cloud features. The sample sequences are sorted according to their frame lengths to ensure that sequences with similar frame lengths are grouped together for batch processing. According to the maximum frame length within the batch, zero-padding is applied to each batch. The point cloud features include the x, y, z coordinates, distance, velocity, and intensity of each point cloud. Each frame contains 128 points, and the point cloud features of each frame are integrated into a one-dimensional feature vector of 128×6. The CNN block is used to extract spatial features. This block consists of three convolutional layers and pooling layers. ReLU is used as the activation function. The kernel size of all convolutional layers is set to 3, and the stride is 1. Considering the sparsity of point cloud data, average pooling is used to reduce the feature dimension, and the kernel size and stride of all pooling layers are set to 2. Then, the output of the CNN block is fed into the multi-head attention block, which precisely locates the key feature parts through three attention heads. To make full use of the temporal information of the sample sequences, an LSTM block is introduced to extract temporal features. The output of the LSTM block is fed into the MLP block, and finally, a tensor of length 66 is obtained, which represents the x, y, and z coordinates of 22 skeleton points. This output provides a coarse-grained skeleton tracking result. The STN further aggregates the coarse-grained skeleton with the original point cloud information and processes it through another MLP block to finally obtain a fine-grained skeleton tracking result.

[0034] The present invention also provides a signal simulation method for millimeter-wave data enhancement, including:

[0035] Step S1: Obtain a fine-grained three-dimensional human SMPL model that can accurately describe the body shape from a human motion database, and then perform preprocessing operations on the three-dimensional human SMPL model so that the human model can truly reflect the reflection of millimeter-wave signals on the human surface;

[0036] Step S2: Perform signal simulation according to the propagation principle of millimeter-wave signals, construct a chirp signal transmitted by a millimeter-wave radar, reflect off the human surface of the human model established in Step S1, return to the receiving antenna of the millimeter-wave radar, and finally mix with the transmitted signal in a mixer to become an intermediate-frequency signal;

[0037] Step S3: Use basic signal processing operations on the intermediate-frequency signal output in Step S2 to obtain the perceptual features for neural network training: the micro-Doppler spectrum for behavior recognition network training and the three-dimensional human point cloud for human skeleton tracking network training;

[0038] Step S4: Use the micro-Doppler features and three-dimensional point cloud features to train the corresponding networks, thereby realizing two perceptual applications.

[0039] Further, Step S1 specifically includes:

[0040] Signal synthesis is performed using a fine-grained SMPL model. The pose parameters of the human body model and the body shape parameters obtained from the video of the real user's standard posture are superimposed and input into the SMPL model together. Two preprocessing operations are carried out: the first is occlusion point removal, and the second is downsampling. A part of the sample points are selected from the original surface vertices of the human body model according to a certain proportion.

[0041] The advantages of the present invention are as follows:

[0042] 1. The present invention proposes a signal simulation technology for millimeter-wave data enhancement, which provides data support for millimeter-wave sensing scenarios that require a large amount of training data and effectively reduces the burden of training data collection.

[0043] 2. The present invention proposes a millimeter-wave signal simulation method that uses a fine-grained SMPL model for signal synthesis to more accurately depict the human body shape. In addition, by deeply exploring the signal propagation principle and adding a signal attenuation factor to the signal synthesis, a more realistic millimeter-wave signal can be synthesized.

[0044] 3. The present invention proposes a method of superimposing the body shape information of a real human body and the pose information of a human body model, enabling the human body model to better depict a real human body. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a module diagram of a signal simulation system for millimeter-wave data enhancement according to an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of the preprocessing of the human body model in the human body reflection modeling module according to an embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of a Chirp signal in the signal reflection modeling module according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0049] Embodiment 1

[0050] The present invention trains a neural network through the sensing characteristics of the simulated millimeter-wave signal and supports two common sensing applications: behavior recognition and human skeleton tracking. Please refer to Figure 1As shown in the figure, a signal simulation system for millimeter-wave data enhancement according to the present invention mainly includes four modules: a human body reflection modeling module, a signal reflection modeling module, a perception feature extraction module, and a neural network training module, as Figure 1 shown.

[0051] The human body reflection modeling module is used to obtain a fine-grained three-dimensional human body model from a human body motion database that can accurately describe the body shape, and then perform preprocessing operations on the three-dimensional human body model so that the human body model can truly reflect the reflection of millimeter-wave signals on the human body surface.

[0052] The human body reflection modeling module includes a personalized human body model acquisition unit and a human body model preprocessing unit.

[0053] Among them, in the personalized human body model acquisition unit, in order to achieve a high degree of fidelity of the human body model, fit the real human body as accurately as possible, and ensure the accuracy of the synthesized signal, in this embodiment, a fine-grained SMPL model is used for signal synthesis. In order to make the body shape of the human body model closer to that of the real user, the pose parameters of the human body model and the body shape parameters obtained from the video of the real user in the standard pose are superimposed and input into the SMPL model together to construct a user personalized human body model, and the SMPL model will have the body shape of the real user.

[0054] In the human body model preprocessing unit, in order to endow the personalized human body model with the ability to reflect real human body signals, two preprocessing operations are performed: the first step is occlusion point removal. The present invention uses the Hidden Point Remove (HPR) occlusion point removal algorithm to identify and remove occluded points through two steps of point cloud inversion and convex hull construction, thereby deleting the human body vertices that are invisible from the radar perspective; the second step is downsampling. A part of the sample points are selected from the original human body model surface vertices according to a certain ratio to reduce the data sampling rate. For example, in MATLAB, it can be achieved through the downsample function. y = downsample(x,n), where x is the original human body model surface vertices, and n is the downsampling factor, indicating that one sample is retained from every n original samples, and the rest of the samples are discarded. This can further dilute the number of human body surface vertices and reduce the overhead of the synthesized signal. The schematic diagram of the human body model preprocessing is as Figure 2 shown.

[0055] The signal reflection modeling module is used to perform signal simulation according to the propagation principle of millimeter-wave signals, construct a frequency-modulated continuous pulse (chirp signal) emitted by a millimeter-wave radar, reflect through the human body surface of the human body model established by the human body reflection modeling module, return to the receiving antenna of the millimeter-wave radar, and finally mix with the transmitted signal in the mixer to become an intermediate frequency signal in a complete process.

[0056] The signal reflection modeling module includes a signal attenuation factor calculation unit and an intermediate frequency signal calculation unit.

[0057] The schematic diagram of the chirp signal transmitted by millimeter wave is as Figure 3 shown. The signal attenuation factor calculation unit is used to calculate the signal attenuation factor. During the signal simulation process, it is necessary to consider the signal attenuation caused by environmental impacts, which can be represented by a signal attenuation factor. This signal attenuation factor is related to the propagation distance and antenna gain.

[0058] The intermediate frequency signal calculation unit is used to calculate the intermediate frequency signal, and the intermediate frequency signal is calculated according to the signal propagation characteristics.

[0059] The perception feature extraction module is used to perform basic signal processing operations on the intermediate frequency signal output by the signal reflection modeling module to obtain the perception features for neural network training: the micro-Doppler spectrum for behavior recognition network training and the human three-dimensional point cloud for human skeleton tracking network training.

[0060] The perception feature extraction module includes a micro-Doppler feature extraction unit and a three-dimensional point cloud feature extraction unit.

[0061] In the micro-Doppler feature extraction unit, the fast Fourier transform (FFT), i.e., Doppler FFT, is performed on multiple consecutive chirp signals to obtain the instantaneous velocity of the target. Then, the instantaneous velocities are extracted and concatenated frame by frame to obtain the micro-Doppler spectrum. Many existing works use the micro-Doppler spectrum for activity recognition.

[0062] In the three-dimensional point cloud feature extraction unit, range FFT is performed on the intermediate frequency signal to separate the reflection points at different ranges. Then, we perform two-dimensional angular FFT to obtain the azimuth angle and elevation angle of each reflection point. According to the range, azimuth angle, and elevation angle of the reflection point relative to the radar, the coordinates of the reflection point in three-dimensional space can be calculated. Since the three-dimensional point cloud contains rich human spatial information, many works use the three-dimensional point cloud for pose tracking.

[0063] The neural network training module is used to train the corresponding networks using the micro-Doppler features and three-dimensional point cloud features, so as to realize two perception applications.

[0064] The neural network training module includes a behavior recognition network unit and a skeleton tracking network unit.

[0065] In the behavior recognition network unit, since the micro-Doppler spectrum can be regarded as a two-dimensional image, the CNN module is a good choice for feature extraction and classification. VGG16 has excellent activity recognition ability, but the excessive number of its convolutional layers may lead to overfitting in this task. Therefore, this unit uses a lightweight network based on VGG16, which only retains the first two convolutional layers in each sub-module of VGG16 and adds a fully connected layer at the end of the network. This lightweight network not only maintains the ability to distinguish different activities but also avoids the overfitting problem caused by too many network layers.

[0066] In the skeleton tracking network unit, the input is a tensor with dimensions batch size * frame length * number of point cloud features. Since the sample sequences in different datasets may have different frame lengths, we sort the sample sequences according to their frame lengths to ensure that sequences with similar frame lengths are grouped together for batch processing. According to the maximum frame length within the batch, zero-padding is applied to each batch. The point cloud features include the x, y, z coordinates, distance, speed, and intensity of each point cloud. Each frame contains 128 points, and the point cloud features of each frame are integrated into a one-dimensional feature vector of 128×6. The CNN block is used to extract spatial features. This block consists of three convolutional layers and pooling layers, and ReLU is used as the activation function. The kernel size of all convolutional layers is set to 3, and the stride is 1. Considering the sparsity of point cloud data, average pooling is used to reduce the feature dimension, and the kernel size and stride of all pooling layers are set to 2. Then the output of the CNN block is fed into the multi-head attention block, which precisely locates the key feature parts through three attention heads. To make full use of the temporal information of the sample sequence, an LSTM block is introduced to extract temporal features. The output of the LSTM block is fed into the MLP block, and finally a tensor of length 66 is obtained, which represents the x, y, and z coordinates of 22 skeleton points. This output provides a coarse-grained skeleton tracking result. To achieve more precise skeleton tracking, the STN further aggregates the coarse-grained skeleton with the original point cloud information and processes it through another MLP block to finally obtain a fine-grained skeleton tracking result.

[0067] As a further optimized technical solution of the present invention, in order to enable the perceptual features of the analog signal to be well used for the training of the neural network, it is necessary to ensure that the analog signal is as close as possible to the real signal. The present invention deeply analyzes the signal propagation process and simulates the complete process of the analog signal from transmission, reflection to reception. The following further elaborates on the implementation details of calculating the intermediate frequency signal by the signal reflection modeling module.

[0068] The millimeter-wave radar is a frequency-modulated continuous-wave radar that emits chirp signals. A chirp signal is a sine wave whose frequency changes linearly with time, as Figure 3As shown. Assume that the starting frequency of the chirp signal is fc, the ending frequency is fc + B, the bandwidth is B, and Tc is the duration of the chirp. Then the frequency of the signal at time t can be expressed as f(t) = f c + B * t / T c . The millimeter-wave radar emits multiple chirp signals to form a frame. The transmitted signal can be expressed as

[0069]

[0070] where A T is the amplitude of the transmitted signal, is a constant. The signal emitted by the transmitting antenna is reflected on the target surface and received by the receiving antenna. Essentially, the transmitted signal is a time-delayed version of the received signal. The received signal is expressed as

[0071]

[0072] where α is the signal attenuation factor, which is used to describe the attenuation degree of the signal intensity during the propagation process. τ is the round-trip time between the reflection point and the radar, and can be expressed as τ = (d t + d r ) / c, where d t and d r are the distances between the reflection point and the transmitting antenna and the receiving antenna respectively, and c is the speed of light. Subsequently, the transmitted signal and the received signal are mixed in the mixer to obtain the intermediate-frequency signal, expressed as

[0073]

[0074] Through the above formula, we can obtain the signal of each reflection point under each transmit-receive antenna pair.

[0075] The attenuation degree of the signal is affected by the antenna gain, propagation distance, and radar cross-section area. The signal attenuation factor α is expressed as

[0076]

[0077] where λ is the wavelength of the signal, and d is the distance between the centroid of the triangular patch and the radar. G Tx and G Rx represent the gains of the transmitting and receiving antennas, which reflect the radiation effect of the antennas in a specific direction and can be set according to the actually used millimeter-wave radar. σ represents the radar cross-section area, which reflects the reflection intensity of the target surface. The radar cross-section area is measured by the area of the triangular patch and the angle θ between the normal of the triangular patch and the line connecting the radar and the centroid of the triangular patch.

[0078] The main innovation of the present invention lies in the ability to train a neural network through the perception characteristics of simulated millimeter-wave signals, thereby supporting two common perception applications - behavior recognition and skeleton tracking.

[0079] Embodiment 2

[0080] The present invention also provides a signal simulation method for millimeter-wave data augmentation, including:

[0081] Step S1: Obtain a fine-grained three-dimensional human SMPL model that can accurately describe the body shape from a human motion database, and then perform preprocessing operations on the three-dimensional human SMPL model so that the human model can truly reflect the reflection of millimeter-wave signals on the human surface.

[0082] Step S2: Perform signal simulation according to the propagation principle of millimeter-wave signals, construct a chirp signal transmitted by a millimeter-wave radar, reflect through the human surface of the human model established in Step S1, return to the receiving antenna of the millimeter-wave radar, and finally mix with the transmitted signal in a mixer to become an intermediate-frequency signal.

[0083] Step S3: Use basic signal processing operations on the intermediate-frequency signal output in Step S2 to obtain perception features for neural network training: the micro-Doppler spectrum for behavior recognition network training and the three-dimensional human point cloud for human skeleton tracking network training.

[0084] Step S4: Use the micro-Doppler features and three-dimensional point cloud features to train the corresponding networks, thereby realizing two perception applications.

[0085] Step S1 specifically includes:

[0086] Use a fine-grained SMPL model for signal synthesis, superimpose the pose parameters of the human model and the body shape parameters obtained from the video of a real user in the standard pose, and input them into the SMPL model together; perform two preprocessing operations: the first is occlusion point removal, use the HPR occlusion point removal algorithm, and identify and remove the occluded points through two steps of point cloud inversion and convex hull construction, so as to delete the human vertices that are not visible from the radar perspective; the second is downsampling, select some sample points from the original human model surface vertices according to a certain proportion, and in MATLAB, realize downsampling through the downsample function.

[0087] In Step S2:

[0088] The signal attenuation factor calculation unit is used to calculate the signal attenuation factor. During the signal simulation process, it is necessary to consider the signal attenuation caused by environmental impacts, which can be represented by a signal attenuation factor. This signal attenuation factor is related to the propagation distance and antenna gain. Calculate the intermediate-frequency signal according to the signal propagation characteristics.

[0089] In step S3:

[0090] Perception feature extraction includes micro - Doppler feature extraction and 3D point cloud feature extraction.

[0091] Micro - Doppler feature extraction is to perform a fast Fourier transform (FFT), namely Doppler FFT, on multiple consecutive chirp signals to obtain the instantaneous velocity of the target. Then, the instantaneous velocities are extracted and concatenated frame by frame to obtain the micro - Doppler spectrum. Many existing works use the micro - Doppler spectrum for activity recognition.

[0092] 3D point cloud feature extraction is to perform range FFT on the intermediate - frequency signal to separate the reflection points at different ranges. Then, we perform 2D angular FFT to obtain the azimuth and elevation angles of each reflection point. According to the range, azimuth, and elevation angles of the reflection points relative to the radar, the coordinates of the reflection points in 3D space can be calculated. Since the 3D point cloud contains rich human body spatial information, many works use the 3D point cloud for valuation tracking.

[0093] In step S4: Neural network training includes behavior recognition and skeleton tracking.

[0094] In behavior recognition, since the micro - Doppler spectrum can be regarded as a 2D image, the CNN module is a good choice for feature extraction and classification. VGG16 has excellent activity recognition ability, but the excessive number of its convolutional layers may lead to overfitting for this task. Therefore, a lightweight network based on VGG16 is used for behavior recognition. It only retains the first two convolutional layers in each sub - module of VGG16 and adds a fully - connected layer at the end of the network. This lightweight network not only maintains the ability to distinguish different activities but also avoids the overfitting problem caused by too many network layers.

[0095] In skeleton tracking, the input is a tensor with dimensions batch size * frame length * number of point cloud features. Since sample sequences in different datasets may have different frame lengths, we sort the sample sequences according to their frame lengths to ensure that sequences with similar frame lengths are grouped together for batch processing. Zero-padding is applied to each batch according to the maximum frame length within the batch. The point cloud features include the x, y, z coordinates, distance, velocity, and intensity of each point cloud. Each frame contains 128 points, and the point cloud features of each frame are integrated into a one-dimensional feature vector of 128×6. The CNN block is used to extract spatial features. This block consists of three convolutional layers and pooling layers, with ReLU used as the activation function. The kernel size of all convolutional layers is set to 3, and the stride is 1. Considering the sparsity of point cloud data, average pooling is used to reduce the feature dimension, and the kernel size and stride of all pooling layers are set to 2. Then the output of the CNN block is fed into the multi-head attention block, which precisely locates the key feature parts through three attention heads. To make full use of the temporal information of the sample sequence, an LSTM block is introduced to extract temporal features. The output of the LSTM block is fed into the MLP block, and finally a tensor of length 66 is obtained, which represents the x, y, and z coordinates of 22 skeleton points. This output provides a coarse-grained skeleton tracking result. To achieve more accurate skeleton tracking, the STN further aggregates the coarse-grained skeleton with the original point cloud information and processes it through another MLP block to finally obtain a fine-grained skeleton tracking result.

[0096] The implementation details of step S2 are illustrated by the following example.

[0097] The millimeter-wave radar is a frequency-modulated continuous-wave radar that emits chirp signals. A chirp signal is a sine wave whose frequency varies linearly with time, as Figure 3 shown. Assuming that the starting frequency of the chirp signal is fc, the ending frequency is fc + B, the bandwidth is B, and Tc is the duration of the chirp, then the frequency of the signal at time t can be expressed as f(t) = f c + B * t / T c . The millimeter-wave radar emits multiple chirp signals to form a frame. The transmitted signal can be expressed as

[0098]

[0099] where A T is the amplitude of the transmitted signal, is a constant. The signal transmitted by the transmitting antenna is reflected on the target surface and received by the receiving antenna. Essentially, the transmitted signal is a time-delayed version of the received signal. The received signal is expressed as

[0100]

[0101] Among them, α is the signal attenuation factor, which is used to describe the attenuation degree of the signal strength during the propagation process. τ is the round-trip time between the reflection point and the radar, which can be expressed as τ = (d t + d r ) / c, where d t and d r are the distances between the reflection point and the transmitting antenna and the receiving antenna respectively, and c is the speed of light. Subsequently, the transmitted signal and the received signal are mixed in the mixer to obtain an intermediate frequency signal, which is expressed as

[0102]

[0103] Through the above formula, we can obtain the signals of each reflection point under each transmit-receive antenna pair.

[0104] The attenuation degree of the signal is affected by the antenna gain, propagation distance, and radar cross-section area. The signal attenuation factor α is expressed as

[0105]

[0106] where λ is the wavelength of the signal, and d is the distance between the centroid of the triangular patch and the radar. G Tx and G Rx represent the gains of the transmitting and receiving antennas, which reflect the radiation effect of the antennas in a specific direction and can be set according to the actually used millimeter-wave radar. σ represents the radar cross-section area, which reflects the reflection intensity of the target surface. The radar cross-section area is measured by the area of the triangular patch and the angle θ between the normal of the triangular patch and the line connecting the radar and the centroid of the triangular patch.

[0107] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0108] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A signal simulation system for millimeter wave data enhancement, characterized in that: include: The human body reflection modeling module is used to obtain a fine-grained three-dimensional human body SMPL model that can accurately describe the body shape from the human body motion database, and then preprocess the three-dimensional human body SMPL model so that the human body model can truly reflect the reflection of the human body surface to the millimeter wave signal; The signal reflection modeling module is used to simulate the signal according to the propagation principle of the millimeter wave signal, construct the chirp signal emitted by the millimeter wave radar, reflect the human body surface of the human body model established by the human body reflection modeling module, return to the receiving antenna of the millimeter wave radar, and finally mix it with the transmission signal in the mixer to become an intermediate frequency signal; The perceptual feature extraction module is used to use basic signal processing operations on the intermediate frequency signal output by the signal reflection modeling module to obtain perceptual features for neural network training: micro-Doppler spectrum for behavior recognition network training and human body 3D point cloud for human skeleton tracking network training; The neural network training module is used to train the corresponding network using micro-Doppler features and 3D point cloud features, thereby realizing two perception applications: behavior recognition and skeleton tracking.

2. A signal simulation system for millimeter wave data enhancement as claimed in claim 1, characterized in that: The human body reflection modeling module includes a personalized human body model acquisition unit and a human body model preprocessing unit. In the personalized human body model acquisition unit, a fine-grained SMPL model is used for signal synthesis, and the posture parameters of the human body model and the body shape parameters obtained from the video of the real user in the standard posture are superimposed together and input into the SMPL model; In the human body model preprocessing unit, two steps of preprocessing operations are performed: the first step is to remove occluded points, and the second step is downsampling, selecting some sample points from the original human body model surface vertices according to a certain proportion.

3. A signal simulation system for millimeter wave data enhancement as claimed in claim 2, characterized in that: In the personalized human body model acquisition unit, the HPR occlusion point elimination algorithm is used to identify and eliminate occluded points through two steps: point cloud inversion and convex hull construction, thereby deleting human body vertices that are invisible from the radar perspective; in MATLAB, downsampling is achieved through the downsample function.

4. The signal simulation system for millimeter wave data enhancement according to claim 1, characterized in that: The signal reflection modeling module includes a signal attenuation factor calculation unit and an intermediate frequency signal calculation unit; The signal attenuation factor calculation unit is used to calculate the signal attenuation factor. In the process of signal simulation, the signal attenuation caused by environmental influences needs to be considered and represented by a signal attenuation factor. The signal attenuation factor is related to the propagation distance and the antenna gain. The intermediate frequency signal calculation unit is used to calculate the intermediate frequency signal, and obtains the intermediate frequency signal according to the signal propagation characteristics.

5. A signal simulation system for millimeter wave data enhancement as claimed in claim 4, characterized in that: The millimeter-wave radar transmits a chirp signal. The starting frequency of the chirp signal is fc, the ending frequency is fc+B, the bandwidth is B, and Tc is the duration of the chirp. Then the frequency of the signal at time t is expressed as f(t)=f c +B*t / T c , the millimeter-wave radar transmits multiple chirp signals to form a frame, and the transmitted signal is expressed as Among them A T is the amplitude of the transmitted signal, is a constant. The signal emitted by the transmitting antenna is reflected on the target surface and received by the receiving antenna. The received signal is expressed as Among them, α is the signal attenuation factor, which is used to describe the attenuation degree of signal strength during propagation, and τ is the round-trip time between the reflection point and the radar, expressed as τ = (d t +d r ) / c, where d t and d r are the distances between the reflection point and the transmitting antenna and the receiving antenna, respectively, and c is the speed of light. Subsequently, the transmitting signal and the receiving signal are mixed in the mixer to obtain the intermediate frequency signal, which is expressed as Through the above formula, the signal of each reflection point under each transmitting-receiving antenna pair is obtained.

6. A signal simulation system for millimeter wave data enhancement as claimed in claim 5, characterized in that: The degree of signal attenuation is affected by antenna gain, propagation distance and radar cross-sectional area. The signal attenuation factor α is expressed as Where λ is the wavelength of the signal, d is the distance between the centroid of the triangle patch and the radar, G Tx and G Rx It represents the gain of the transmitting and receiving antennas, reflecting the radiation effect of the antenna in a specific direction. According to the actual millimeter-wave radar settings used, σ represents the radar cross-sectional area, reflecting the reflection intensity of the target surface. The radar cross-sectional area is measured by the area of ​​the triangular patch and the angle θ between the normal of the triangular patch and the line connecting the radar and the centroid of the triangular patch.

7. The signal simulation system for millimeter wave data enhancement according to claim 1, characterized in that: The perception feature extraction module includes a micro-Doppler feature extraction unit and a three-dimensional point cloud feature extraction unit; In the micro-Doppler feature extraction unit, a fast Fourier transform is performed on multiple continuous chirp signals, and then the instantaneous velocity is extracted and connected frame by frame to obtain the micro-Doppler spectrum; In the three-dimensional point cloud feature extraction unit, the intermediate frequency signal is subjected to distance Fourier transform to separate reflection points at different distances. Then, a two-dimensional angle Fourier transform is performed to obtain the azimuth and elevation of each reflection point. The coordinates of the reflection point in three-dimensional space are calculated based on the distance, azimuth and elevation of the reflection point relative to the radar.

8. The signal simulation system for millimeter wave data enhancement according to claim 1, characterized in that: The neural network training module includes a behavior recognition network unit and a skeleton tracking network unit; In the action recognition network unit, a lightweight network based on VGG16 is used, retaining only the first two convolutional layers in each submodule of VGG16, and adding a fully connected layer at the end of the network; In the skeleton tracking network unit, the input is a tensor of dimensions batch size * frame length and * number of point cloud features. The sample sequences are sorted according to their frame lengths to ensure that sequences with similar frame lengths are grouped together for batch processing. Zero padding is applied to each batch based on the maximum frame length within the batch. The point cloud features include the x, y, z coordinates, distance, speed, and intensity of each point cloud. Each frame contains 128 points. The point cloud features of each frame are integrated into a 128×6 one-dimensional feature vector. The CNN block is used to extract spatial features. The block consists of three convolutional layers and a pooling layer. ReLU is used as the activation function. The kernel size of all convolutional layers is set to 3 and the stride is 1. Considering the sparseness of point cloud data The average pooling is used to reduce the feature dimension, and the kernel size and stride of all pooling layers are set to 2. Then the output of the CNN block is fed into the multi-head attention block, which accurately locates the key feature parts through three attention heads. In order to make full use of the temporal information of the sample sequence, the LSTM block is introduced to extract the temporal features. The output of the LSTM block is fed into the MLP block, and finally a tensor of length 66 is obtained, which represents the x, y and z coordinates of 22 skeleton points. This output provides a coarse-grained skeleton tracking result. STN further aggregates the coarse-grained skeleton with the original point cloud information and processes it through another MLP block to finally obtain a fine-grained skeleton tracking result.

9. A signal simulation method for millimeter wave data enhancement, characterized in that: include: Step S1, obtaining a fine-grained three-dimensional human SMPL model that can accurately describe the body shape from a human motion database, and then performing a preprocessing operation on the three-dimensional human SMPL model so that the human body model can truly reflect the reflection of the human body surface to the millimeter wave signal; Step S2, perform signal simulation according to the propagation principle of millimeter wave signals, construct a chirp signal emitted by the millimeter wave radar, reflect the human body surface of the human body model established in step S1, return to the receiving antenna of the millimeter wave radar, and finally mix it with the transmission signal in the mixer to become an intermediate frequency signal; Step S3, using basic signal processing operations on the intermediate frequency signal outputted from step S2 to obtain perceptual features for neural network training: micro-Doppler spectrum for behavior recognition network training and human body three-dimensional point cloud for human skeleton tracking network training; Step S4: Use micro-Doppler features and three-dimensional point cloud features to train the corresponding network, thereby realizing two perception applications.

10. The signal simulation method for millimeter wave data enhancement according to claim 9, characterized in that: Step S1 specifically includes: A fine-grained SMPL model is used for signal synthesis. The posture parameters of the human body model and the body shape parameters obtained from the video of the real user in the standard posture are superimposed together and input into the SMPL model. Two preprocessing operations are performed: the first step is to remove occlusion points, and the second step is downsampling, which selects some sample points from the original human body model surface vertices according to a certain proportion.

Images