Millimeter wave human body grid reconstruction method based on cross-modal knowledge migration

Abstract

The invention discloses a millimeter wave human body grid reconstruction method based on cross-modal knowledge migration. The method comprises the steps that a data set is acquired, and the data set reflects the corresponding relation between a depth map and parameterized human body model parameters and the corresponding relation between a thermodynamic diagram extracted based on millimeter wave data and the parameterized human body model parameters; knowledge distillation training is conducted on a 3D human body grid model through the data set, a teacher model learns mapping from the depth map to parameters of a parameterized human body model with the depth map as input, and a student model learns advanced feature representation of the teacher model with the thermodynamic diagram as input; and performing human body grid reconstruction on the target by using the trained student model. According to the method, the generalization ability of the model is improved, the dependence on large-scale annotation data is reduced, the cost is reduced, and the deployment speed is increased.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a millimeter wave human body mesh reconstruction method based on cross-modal knowledge transfer. Background Art

[0002] Human body mesh reconstruction is an important technology in the field of computer vision. It reconstructs the 3D model of the human body by processing images or other sensor data. This technology is widely used in virtual reality, animation production, medical imaging and other fields.

[0003] In recent years, millimeter-wave-based human mesh reconstruction technology has made significant progress. Millimeter-wave human sensing technology uses the characteristics of millimeter waves to detect and track the human body. Due to the penetrating ability of millimeter waves, it can effectively sense the presence and dynamics of the human body in various environmental conditions, including smoke, fog and other visual obstructions.

[0004] In the prior art, mmMesh is a system that uses commercial portable millimeter wave devices to achieve real-time 3D human mesh estimation. The system designs a novel deep learning framework that can dynamically locate moving targets and capture the target's body shape and posture by analyzing the 3D point cloud generated by the millimeter wave signal reflected from the human body. mmMesh also proposes a point cloud alignment method that can accurately correspond 3D points to corresponding body parts, overcoming the influence of environmental noise and multipath effects. Through a dynamic anchor selection strategy, the system groups 3D point clouds into subsets corresponding to different body parts, thereby learning local structures and associations. In addition, mmMesh integrates the SMPL model as an additional constraint, using 86 parameters to represent the entire 3D human mesh.

[0005] To address the scarcity of millimeter-wave human motion perception datasets, SynMotion proposed a novel method for synthesizing millimeter-wave signals. The system uses existing vision-based human motion datasets to synthesize corresponding millimeter-wave perception signals, thereby inheriting the label information (bone coordinates and action names) of the visual dataset. SynMotion implements zero-shot activity recognition, that is, the classifier is trained only with synthetic data, but is able to read real millimeter-wave data for recognition. In addition, it also supports body skeleton tracking for few-shot / zero-shot learning.

[0006] mmGPE is a wireless signal enhancement framework designed to improve the generalization ability of millimeter-wave human posture estimation. The system uses a physical simulator to generate millimeter-wave Frequency-Modulated Continuous-Wave (FMCW) signals, which solves the problem of scarce training data. To bridge the gap between the signals generated by the physical simulator and the real-world signals, researchers developed a deep learning-based signal refiner. By integrating physical signal simulation and deep learning techniques, mmGPE is able to generate high-quality real signal data.

[0007] RF Genesis (RFGen) proposes a simulation-based approach to data synthesis and generative diffusion models, designed to improve the generalization capabilities of mmWave perception systems. The approach uses a custom ray tracing simulator to simulate Radio Frequency (RF) propagation and interaction with objects / environment. RFGen generates a large number of 3D scenes using a set of diffusion models and converts visual scene representations into corresponding mmWave perception data guided by application-specific "cues". By combining physics-based ray tracing with black-box diffusion models, RFGen enables accurate, scalable, and interpretable visual-to-RF data synthesis.

[0008] After analysis, although the existing millimeter-wave human body mesh reconstruction technology has made significant progress, it still faces some important challenges in practical applications. The first problem is the insufficient generalization performance, especially when dealing with unseen activity types or new user data. Existing models often perform well on training data sets, but when encountering new scenes with different distributions from the training set, the reconstruction accuracy will drop significantly. This limitation of generalization ability seriously restricts the scope of application of the system in practical environments. Secondly, there are inherent feature differences between synthetic data and real data. Although using synthetic data for training can alleviate the data scarcity problem to a certain extent, due to the limitations of the simulation process, synthetic data cannot fully capture the complexity of the real world. This results in the model's performance in real scenes often not being as good as that on synthetic data. Feature differences not only affect the accuracy of the model, but may also cause the system to produce unexpected behavior in practical applications. Another significant problem is the resolution difference between different millimeter-wave radar hardware. Since the antenna layout and angular resolution of commercial millimeter-wave radars vary, a model trained on one hardware configuration may be difficult to directly migrate to another configuration. This hardware dependency limits the versatility of the model and increases the complexity and cost of deploying and maintaining the system on different devices. Summary of the invention

[0009] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a millimeter wave human body mesh reconstruction method based on cross-modal knowledge transfer. The method comprises the following steps:

[0010] Acquire a data set, the data set reflecting the correspondence between the depth map and the parameters of the parameterized human body model, and the correspondence between the thermal map extracted based on the millimeter wave data and the parameters of the parameterized human body model;

[0011] Using the data set to perform knowledge distillation training on a 3D human body mesh model, wherein a teacher model uses the depth map as input to learn a mapping from the depth map to parameters of a parameterized human body model, and a student model uses the heat map as input to learn a high-level feature representation of the teacher model;

[0012] The trained student model is used to reconstruct the human body mesh of the target.

[0013] Compared with the prior art, the advantage of the present invention is that the millimeter wave human body mesh reconstruction method based on cross-modal knowledge transfer is provided, and a novel cross-modal knowledge transfer strategy is designed, which can effectively align the feature space of synthetic data and real data. By using the teacher model trained on the high-resolution depth map to guide the learning process of the student model, the "training-test mismatch" problem can be reduced and the performance of the model in unseen scenes can be improved. In addition, in order to solve the problem of resolution difference caused by different radar hardware, a flexible feature extraction and alignment mechanism is designed so that the model can adapt to different hardware configurations, thereby enhancing the versatility and portability of the system. In addition, in order to improve the data efficiency of the system, by making full use of the combination of synthetic data and a small amount of real data, the dependence on large-scale annotated data is reduced, making the system easier to quickly deploy and adapt in new application scenarios. Through innovative cross-modal knowledge transfer methods and comprehensive technical improvements, the present invention significantly improves the generalization ability, accuracy and practicality of millimeter wave human body mesh reconstruction, which is conducive to promoting a wider range of practical applications.

[0014] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0016] Figure 1 is a flow chart of a millimeter wave human body mesh reconstruction method based on cross-modal knowledge transfer according to an embodiment of the present invention;

[0017] Figure 2 is a schematic diagram of a human body grid according to an embodiment of the present invention;

[0018] Figure 3 is a schematic diagram of a depth image according to an embodiment of the present invention

[0019] Figure 4 is a schematic diagram of an IWR1843BOOST virtual antenna array according to an embodiment of the present invention;

[0020] Figure 5 It is a schematic diagram of the overall structure of a 3D human body mesh network model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0022] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0023] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0024] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0025] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0026] Cross-modal knowledge transfer involves applying knowledge learned in one modality (such as images) to another modality (such as millimeter waves). This paper aims to optimize the information fusion of millimeter wave human body mesh reconstruction and depth images to improve the accuracy and efficiency of human body mesh reconstruction. Figure 1 As shown, the millimeter wave human body mesh reconstruction method based on cross-modal knowledge transfer provided includes the following steps:

[0027] Step S110 , obtaining a data set, where the data set reflects the corresponding relationship between the depth map and the SMPL parameters, and the corresponding relationship between the heat map extracted based on the millimeter wave data and the SMPL parameters.

[0028] In order to train the human network model later, we first prepare the data to build the dataset. A two-pronged strategy is adopted in this stage. On the one hand, a large-scale human mesh dataset is used to synthesize millimeter wave signal data to provide rich training samples, which helps to capture the diversity of human posture and shape. On the other hand, a small amount of real millimeter wave data is collected, and RGB images are simultaneously collected as true values. This method of combining synthetic data and real data not only solves the problem of data scarcity, but also lays the foundation for the subsequent cross-modal knowledge transfer process.

[0029] In one embodiment, the process of constructing a dataset includes the following steps:

[0030] Step S111 , using a known human mesh data set to select a motion sequence of a 3D human mesh model to synthesize a millimeter wave signal and a depth image.

[0031] For example, to generate synthetic millimeter wave data, we first select a 3D human mesh model action sequence from a large-scale human mesh dataset (such as the AMASS dataset). The AMASS dataset contains a variety of action sequences that complete different postures at different distances and angles, ensuring the diversity of synthetic millimeter wave data. After selecting a certain action sequence, for each frame, there is a human mesh containing 6890 three-dimensional vertex coordinates.

[0032] Next, use the ray tracing algorithm to simulate the signal transmission and reception process of the millimeter wave radar. Assume that the radar is located at the origin, sends a directional signal and receives the signal reflected from the human body model. The radar's transmission model can be expressed as:

[0033]

[0034] Among them, p i is the position of the i-th vertex in the mesh model, r i is the reflectivity of the vertex, α is the attenuation coefficient, p is the radar position, and N is the number of vertices.

[0035] Finally, the corresponding millimeter-wave signal data can be generated according to the radar parameters (such as the number and layout of antennas).

[0036] The depth map is a form of expression that projects the 3D human body mesh information onto a 2D plane, which retains important spatial information of the human body model. For example, synthesizing a depth image includes setting the virtual camera position and resolution, projecting the 3D mesh vertices onto a 2D image, and calculating the depth value.

[0037] 1) Set the virtual camera position and resolution.

[0038] Before generating the depth image, first determine the placement of the virtual camera so that the depth information required for the depth image can be calculated. For example, the position of the virtual camera is set at the three-dimensional coordinate (0, -2, 0) meters. The generated depth map resolution is uniformly set to 64x64 to meet the needs of subsequent network input.

[0039] 2) Project the 3D mesh vertices to a 2D image.

[0040] For example, first use the HPR algorithm to estimate all visible human mesh vertices from the camera's perspective. Then, project the visible 3D mesh vertices to the 2D image plane using a projection method. Specifically, for each 3D mesh vertex, calculate its coordinates projected to the 2D image:

[0041] x i =min(W, max(1, round(x*scale))), (2)

[0042] y i =min(H, max(1, round(z*scale))) (3)

[0043] Where W and H are the width and height of the depth image, respectively, scale is the scaling ratio, x and z are the x and z axis coordinates of the 3D mesh vertex, and x i ,y i are the x and y coordinates projected onto the 2D image.

[0044] 3) Calculate the depth value.

[0045] For each pixel of the image, calculate its corresponding depth value. The depth value is the distance from the camera to the corresponding vertex of the point in 3D space. After projection, each pixel value in the depth map represents the y-axis coordinate value of the original 3D model vertex to the camera.

[0046] Figure 2 is a human body grid. Figure 3 yes Figure 2 The corresponding depth map.

[0047] Step S112, collecting millimeter wave signals and performing preprocessing, including collecting original millimeter wave signals, performing range-dimensional Fourier transform and static clutter elimination, etc.

[0048] 1) Obtaining the original millimeter wave signal

[0049] For a target at a distance d from the radar, the IF signal generated by the mixer of the transmit and receive signals of each Tx-Rx pair can be simply defined as:

[0050]

[0051] Where α represents the amplitude of the signal. The frequency and phase of the IF signal are And φ0=2πf0τ, where τ is the round-trip delay, f0 is the starting frequency of the chirp, and c is the speed of light. is the effective sweep bandwidth B and chirp duration T c The calculated frequency slope, finally N a Tx-Rx pairs can obtain N s ×N c ×N a The IF complex real signal matrix M, where N s Indicates the number of sampling points for each chirp wave, N c Indicates the number of chirp cycles in a frame. For example, N s =128,N c =128,N a =12.

[0052] 2) Distance Dimensional Fourier Transform

[0053] For a single target, the frequency f of the IF signal remains unchanged, and the distance to the target can be calculated as In the case of multiple reflection sources, the signal matrix M is first subjected to windowed FFT (i.e., Range-FFT) in the first dimension to obtain a matrix of size Matrix M R ,in is the length of Range-FFT. For example, if the window function is Hanning window,

[0054] 3) Static clutter elimination

[0055] Static clutter elimination is an important step in radar signal processing, which aims to remove the clutter generated by the reflection of stationary objects from the received signal in order to detect dynamic targets more accurately. Specifically, the matrix M R Averaging in the second dimension yields the static clutter matrix The matrix M R and the static clutter matrix After subtraction, we get the Range-FFT matrix after removing static clutter.

[0056] Step S113, extracting thermal map features based on the preprocessed millimeter wave signal.

[0057] The extracted thermogram features may include thermogram features generated by moving body parts or thermogram features generated by static parts, etc.

[0058] In one embodiment, the thermal map features generated by the moving body parts are extracted, including range-Doppler thermal maps and range-azimuth-elevation thermal maps.

[0059] 1) Range-Doppler heat map generation

[0060] Assume that the radar c By sending a chirp wave every second and observing the phase difference ω between adjacent chirps, the speed can be estimated Specifically, for the matrix After applying windowed FFT (i.e., Doppler-FFT) in the second dimension, multiple reflection sources moving at different speeds can be identified by performing Doppler-FFT on the chirp phase, and finally a magnitude of The matrix M RD ,in is the length of Doppler-FFT. RD After summing in the third dimension, we get the magnitude Distance-Doppler heat map For example, the window function is the Hanning window,

[0061] 2) Range-azimuth-elevation heat map generation

[0062] By using multiple antennas simultaneously and comparing the phase difference between adjacent receiving antennas, the angle of arrival (AoA) of the target can be estimated. Due to the spatial position differences between the receivers, the signal received by each receiver has a slight phase difference, which depends on the relative position and AoA of the receiving antennas. If there are multiple antennas in each direction, the azimuth and elevation angles can be calculated, expressed as θ a and θ e , or θ when referring to these two angles (a,e) Assume that there are N azimuth and elevation angles. a ×N e linearly spaced receiving antennas in different directions θ (a，e)m If there are M targets on the network, each target can be regarded as a signal source, and the receiving antenna array will receive a set of weighted sum signals, represented by x:

[0063]

[0064] Among them, there are M data sources, s(θ (a，e)m ) is the signal arrival angle θ (a，e)m The steering vector of the phase difference between the receivers, α mrepresents the amplitude, and n is the noise. AoA estimation can be modeled as estimating θ for each object m based on a set of receive antenna data x. (a，e) Estimate s(θ) of object m in equation (5) (a，e)m ) is to use the correlation between the receiver data x and the steering vector of the candidate angle. For example, the Beamforming method is used to calculate a a The weight set of virtual receiving antennas (including azimuth and elevation angles) And for all possible angles θ (a，e)m ∈Θ, where θ a ∈[-π,π],θ e ∈[-π,π]. When a list of weights is applied to the receiver data x, the signal from direction θ will obtain constructive interference. By searching all possible angles θ (a，e) , a power spectrum of size Θ can be obtained, where high power in the spectrum indicates the presence of a data source in that direction. For example, using the IWR1843BOOST millimeter-wave radar from Texas Instruments, which has a three-transmit and four-receive antenna array, the virtual antenna layout is as follows Figure 4 As shown in the figure, the interval between adjacent antennas is half a wavelength (i.e., λ / 2). As the input of MVDR beamforming and calculating the power spectrum, we can finally get The distance-azimuth-elevation heat map of a and N e Represents the length of the steering vector of azimuth and elevation respectively. In order to reduce the computational complexity of the human body mesh reconstruction network, the distance-azimuth-elevation heat map is summed in the second and third dimensions to obtain a value of The distance-azimuth heat map and size are For example, N a and N e Both are set to 64.

[0065] Step S120, using the acquired data set to perform knowledge distillation training on the 3D human body mesh model, wherein the teacher model uses the depth map as input and the student model uses the heat map as input.

[0066] 1) Overall structure of 3D human body mesh model

[0067] In terms of the human body mesh model architecture, a multi-stream network structure is used to process different types of millimeter wave spectrum data. This design allows the model to simultaneously consider multiple information such as range-Doppler spectrum, range-azimuth spectrum, and range-elevation spectrum, thereby fully capturing the spatial and motion characteristics of the human body. In addition, temporal modeling mechanisms such as long short-term memory networks (LSTMs) are introduced to better capture the temporal dependencies between consecutive frames. The model ultimately predicts the SMPL parameters related to the human body mesh, and then inputs the SMPL parameters into the SMPL (parametric human body model) model to generate the final 3D human body mesh.

[0068] Figure 5 The overall structure of the human mesh model, which mainly includes a teacher model and a student model. The teacher model takes the depth map as input and the SMPL parameters as output, and the student model takes the millimeter wave data as input and the SMPL parameters as output. Figure 5 The SMPL model in [1] is constructed by decomposing the human body into shape parameters and pose parameters To estimate the 3D human mesh. Shape parameters can be used to control the variation in height, weight, and body proportions of individuals. Pose parameters are used to control the pose of the 3D human mesh. The 3D human mesh, for example, includes the 3D position information of 6890 vertices. In addition, the SMPL model also outputs the 3D position information of 24 joints.

[0069] In the human mesh model training phase, the training data includes: 1) the actual collected millimeter wave data and the corresponding SMPL parameters obtained from the collected RGB images; and 2) the millimeter wave data synthesized by the action sequence of the AMASS dataset and the corresponding depth image. The corresponding SMPL parameters can be obtained from the RGB image through tools such as VIBE (https: / / github.com / mkocabas / VIBE).

[0070] 2) Teacher Model

[0071] The training goal of the teacher model is to learn the mapping from the depth image to the SMPL model parameters. For example, the teacher model contains an anchor feature extractor and a layer of FCL (fully connected layer). The anchor feature extractor includes an encoder and a layer of LSTM units. The encoder is used to extract the spatial features of the image, and the LSTM is used to learn to capture the dependencies in time series.

[0072] In one embodiment, the encoder includes five residual blocks (ResBlock) and a fully connected layer, each residual block includes two convolutional layers and a skip connection. These convolutional layers gradually extract high-level features in the depth map by gradually increasing the number of channels and downsampling with a step size of 2. The kernel size of the five residual blocks is 3×3, and the number of channels is 8, 16, 32, 64 and 128 respectively. After the two-dimensional output of the residual block is flattened to one dimension, the fully connected layer maps it to a 128-dimensional feature space.

[0073] The input of LSTM is the 128-dimensional feature vector output by the encoder, and the output is a 196-dimensional feature vector. The number of layers of LSTM is three layers: input layer, hidden layer, and output layer. After the output of LSTM passes through the last fully connected layer, it outputs a total dimension of 154 parameters, including 10 SMPL model human body shape parameters and 144 human body posture parameters. After the output is input into the neutral SMPL (neutral SMPL model has no gender information) model, the estimated 3D positions of 6890 human body mesh vertices and 3D position information of 24 joint points can be obtained.

[0074] In one embodiment, the loss function of the teacher model training includes two parts, one for the SMPL model parameters and the other for the positions of the human joints and vertices. For example, for the 10 parameters of the SMPL model that control the human body shape, the human body shape parameters predicted by the network are calculated. The MSE between the true human body shape parameter β is used as the loss to be optimized. For the remaining 144 parameters that control human body posture, the three-dimensional vector of each joint point based on Rodrigues is first converted into a rotation matrix represented by a 3×3 size matrix. The rotation matrix predicted by the network is calculated The MSE with the true rotation matrix θ is used as the loss to be optimized. Therefore, the loss function of the SMPL model parameters is Includes two items:

[0075]

[0076] For the loss of human joint points and vertex positions, calculate the predicted 3D position coordinates of the 24 joint points The three-dimensional position coordinates of the 24 real joints The MAE of is used as the loss to be optimized. Similarly, the three-dimensional position coordinates of the predicted 6890 vertices are The three-dimensional coordinates of the 6890 vertices The MAE is used as the loss to be optimized. In addition, the speed is calculated by predicting the joint point positions between two adjacent frames. And the actual joint speed between two adjacent frames Calculate MAE as the loss to be optimized. This loss function can make the joint points between two consecutive frames generated by the network smoother. Therefore, the loss function of this part Contains three items:

[0077]

[0078] Where T is the number of frames of input data.

[0079] The total loss function of the teacher model is:

[0080]

[0081] 3) Student Model

[0082] The training of the student model aims to enable the student model to learn the high-level feature representation of the teacher model through a knowledge distillation process to process mmWave data and perform human body mesh reconstruction.

[0083] Specifically, after extracting the heat map features, for example, three two-dimensional heat maps can be obtained, namely, the range-Doppler heat map, the range-azimuth heat map, and the range-elevation heat map. These three heat maps are used as inputs of the student model to predict the SMPL parameters of the human body mesh.

[0084] The student model uses a network structure similar to that of the teacher model to process multiple heat maps processed from millimeter wave data. In one embodiment, the student model includes a real feature extractor and a layer of FCL. The real feature extractor structure includes 3 encoders and an LSTM unit. The structure of each encoder is consistent and processes different heat maps respectively. The encoder includes 5 ResBlocks and a layer of FCL. The kernel size of the 5 residual blocks is 3×3, and the number of filters is 4, 8, 16, 32 and 64 respectively. After the input data passes through the residual block, it will be flattened into a one-dimensional vector, and the one-dimensional vector will be input into the fully connected layer. Finally, the fully connected layer will output a feature vector of size 128. After inputting the three kinds of heat maps into their respective encoders to obtain feature vectors, the three feature vectors obtained are spliced ​​into a one-dimensional vector and input into the LSTM to learn the activity information of the current frame and the previous frame. For example, the LSTM includes three layers such as the input layer, the hidden layer and the output layer, and the output is 256 dimensions. After obtaining the features output by LSTM, the first 196 dimensions of the 256-dimensional features are input into FCL to obtain the pose vector and shape vector of the human body mesh, and the last 64 dimensions are input into another FCL to obtain the translation vector. Furthermore, after inputting the pose vector, shape vector, and translation vector of the human body mesh into the SMPL model, the estimated 3D positions of 6890 human body mesh vertices and 3D position information of 24 joints can be obtained.

[0085] In one embodiment, the loss function during student model training includes three parts, namely, the loss of SMPL model parameters, the loss of human joint points and vertex positions, and the consistency loss between the features extracted by the student model and the features extracted by the teacher model.

[0086] For the first loss, the student model adds a translation vector loss calculation based on the teacher model. Calculate the translation vector predicted by the network The MSE between the actual translation vector α and the original translation vector α is used as the loss to be optimized. Therefore, the loss function of the SMPL model parameters is Contains three items:

[0087]

[0088] For the loss of human joint points and vertex positions, the student model is consistent with the teacher model, expressed as:

[0089]

[0090] Where T is the number of frames of input data.

[0091] For the third loss, calculate the features extracted by the student model The features extracted by the teacher model t The MSE between them is used as the loss to be optimized It is expressed as:

[0092]

[0093] The loss function of the student model is:

[0094]

[0095] In the process of knowledge distillation training, the optimization goal is to minimize the set distillation loss function. There are many types of distillation loss functions, for example, using KL divergence or cross entropy loss to measure the difference between the teacher model and the student model.

[0096] It should be understood that when describing the loss functions of the teacher model and the student model, for comparison, parameters with the same meaning are represented by the same letters unless otherwise specified by the context. In addition, the specific structure of the model can be set as needed, for example, the number of residual blocks, convolution kernel size, number of convolution layers, input and output dimensions, etc. are set differently, and the present invention is not limited to this.

[0097] In summary, the present invention adopts knowledge distillation to train the human mesh model, including two key components: a teacher model and a student model. The teacher model is trained using a high-resolution depth map obtained from a 3D mesh projection, and learns the mapping relationship from the depth map to the mesh features, so that the teacher model can capture high-quality human shape and posture information. The student model directly processes millimeter wave data, including synthetic data and real data. Through a carefully designed knowledge distillation strategy, the features of the student model are aligned with the features of the teacher model, so that the student model can learn a richer and more accurate feature representation. Moreover, during the training process, the designed loss function comprehensively considers multiple aspects, including the reconstruction error of the SMPL parameters, the error of the 3D joint point position, the error of the mesh vertex position, and the feature alignment error in the knowledge distillation process. Through this multi-objective optimization strategy, while ensuring the reconstruction accuracy, the generalization ability of the model is effectively improved.

[0098] Step S130, applying the trained student model to human body mesh reconstruction.

[0099] After the knowledge distillation training is completed, the student model can learn richer and more accurate feature representations. Compared with the teacher model, the student model is usually smaller in size. Using the trained student model for human network reconstruction can improve computational efficiency and the generalization performance of the student model while ensuring the reconstruction accuracy.

[0100] It should be noted that the model training process of the present invention can be performed offline on a server or in the cloud, and real-time human body mesh reconstruction can be achieved by embedding the trained student model into an electronic device. The electronic device can be a terminal device or a server, and the terminal device includes any terminal device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sale (POS), a car computer, an intelligent wearable device (smart watch, virtual reality glasses, virtual reality helmet, etc.). The server includes but is not limited to an application server or a Web server, and can be an independent server or a cluster server or a cloud server, etc.

[0101] In summary, compared with the prior art, the present invention has the following advantages:

[0102] 1) The present invention effectively reduces the feature differences between synthetic data and real data through the cross-modal knowledge transfer training strategy of knowledge distillation, improves the performance of the model in unseen activities and user scenarios, and improves the generalization performance.

[0103] 2) The present invention utilizes the high-resolution feature representation learned by the teacher model to help the student model overcome the resolution differences between different radar models, thereby improving the adaptability of the model.

[0104] 3) Through the multi-stream network structure and cross-modal knowledge transfer, the student model of the present invention can learn richer and more robust feature representations, which is conducive to improving the accuracy of human body network reconstruction.

[0105] 4) The present invention can make full use of large-scale synthetic data and a small amount of real data, reduce the dependence on a large amount of labeled real data, and improve data efficiency.

[0106] 5) The present invention applies a cross-modal knowledge transfer strategy, which not only solves the bottleneck problem of the existing technology, but also gives the technology a unique competitive advantage. By combining large-scale synthetic data with a small amount of real data, as well as a multi-stream network structure and a knowledge distillation framework, while improving the generalization ability of the model, it also reduces the dependence on large-scale annotated data, further reducing costs and speeding up deployment.

[0107] 6) With the rapid development of technologies such as artificial intelligence and the Internet of Things, the demand for high-precision, non-contact human body detection technology is also growing continuously, especially in the fields of intelligent security, health care, etc., where the demand for efficient and accurate human body mesh reconstruction technology is particularly urgent. The market prospect of this invention is very broad and is expected to achieve rapid growth in the next few years, with high practicality and application value.

[0108] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0109] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0110] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0111] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0112] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0113] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0114] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0115] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0116] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A millimeter wave human body mesh reconstruction method based on cross-modal knowledge transfer, comprising the following steps: Acquire a data set, the data set reflecting the correspondence between the depth map and the parameters of the parameterized human body model, and the correspondence between the thermal map extracted based on the millimeter wave data and the parameters of the parameterized human body model; Using the data set to perform knowledge distillation training on a 3D human body mesh model, wherein a teacher model uses the depth map as input to learn a mapping from the depth map to parameters of a parameterized human body model, and a student model uses the heat map as input to learn a high-level feature representation of the teacher model; The trained student model is used to reconstruct the human body mesh of the target.

2. The method according to claim 1, characterized in that The teacher model includes a first feature extractor and a first fully connected layer. The first feature extractor includes an encoder and a layer of long short-term memory units. The encoder of the teacher model is used to extract spatial features of the depth map. The student model includes a second feature extractor, a second fully connected layer and a third fully connected layer. The second feature extractor includes multiple encoders and a layer of long short-term memory units. Each encoder of the student model processes a heat map. The output of the long short-term memory unit of the teacher model is passed to the first fully connected layer to obtain human body shape parameters and human body posture parameters, and then after neutral parameterization of the human body model, the estimated three-dimensional position information of the human body mesh vertices and the three-dimensional position information of the joints are obtained. For the output of the length memory unit of the student model, one part is passed to the second fully connected layer to obtain the posture vector and shape vector of the human body mesh, and the other part is passed to the third fully connected layer to obtain the translation vector. After the posture vector, shape vector and translation vector of the human body mesh are input into the neutral parameterized human body model, the three-dimensional position information of the vertices of the human body mesh and the three-dimensional position information of the joints are estimated.

3. The method according to claim 1, characterized in that During the knowledge distillation training process, the loss function of the teacher model is set to: in: in, is the loss function used by the teacher model to neutrally parameterize the parameters of the human body model, is the loss function of the teacher model regarding the position of human joints and vertices, are the human morphological parameters predicted by the teacher model β is the true human body shape parameter β, is the rotation matrix predicted by the teacher model, θ is the true rotation matrix, is the three-dimensional position coordinate of the joint point predicted by the teacher model, is the real three-dimensional position coordinate of the joint point, are the three-dimensional position coordinates of the vertex predicted by the teacher model, is the three-dimensional position coordinate of the real vertex, is the speed calculated based on the joint point positions between two adjacent frames predicted by the teacher model, is the actual joint speed between two adjacent frames T is the number of frames of input data, and t is the frame index.

4. The method according to claim 1, characterized in that: During the knowledge distillation training process, the loss function of the student model is set to: in: in, is the loss function used by the student model to neutrally parameterize the parameters of the human body model, is the loss of the student model regarding the position of human joints and vertices. is the consistency loss between the features extracted by the student model and the features extracted by the teacher model, are the human morphological parameters predicted by the student model β is the true human body shape parameter, is the rotation matrix predicted by the student model, θ is the true rotation matrix, is the three-dimensional position coordinate of the joint point predicted by the student model, is the real three-dimensional position coordinate of the joint point are the three-dimensional position coordinates of the vertex predicted by the student model, is the three-dimensional position coordinate of the real vertex, is the speed calculated based on the joint point positions between two adjacent frames predicted by the student model, is the actual joint speed between two adjacent frames T is the number of frames of input data, is the translation vector predicted by the student model, α is the true translation vector, is the feature extracted by the student model l t is the feature l extracted by the teacher model t , t is the frame index.

5. The method according to claim 2, characterized in that: The student model includes three encoders, which correspond to the range-Doppler heat map, the range-azimuth heat map and the range-pitch angle heat map generated by the moving body parts respectively.

6. The method according to claim 5, characterized in that Each encoder of the student model contains 5 residual blocks and a fully connected layer. The convolution kernel sizes of the 5 residual blocks are all 3×3, and the number of filters are 4, 8, 16, 32 and 64 respectively. The input data will be flattened into a one-dimensional vector after passing through the residual block, and then the one-dimensional vector is passed to the fully connected layer.

7. The method according to claim 1, characterized in that The data set is obtained according to the following steps: Using a known human mesh data set, selecting an action sequence of the 3D human mesh model, and synthesizing a millimeter wave signal and a depth map; Collect the original millimeter wave signal and perform distance-dimensional Fourier transform and static clutter elimination to obtain the preprocessed millimeter wave signal; The thermal map is extracted based on the preprocessed millimeter wave signal.

8. The method according to claim 7, characterized in that The method of selecting the action sequence of the 3D human mesh model by using a known human mesh data set and synthesizing the millimeter wave signal and the depth map includes: Selecting an action sequence of the 3D human mesh model from a known human mesh data set, wherein after selecting an action sequence, for each frame, there is 3D human mesh information including a plurality of three-dimensional vertex coordinates; Using a ray tracing algorithm to simulate the signal transmission and reception process of the millimeter wave radar, and generating millimeter wave signal data corresponding to the action sequence according to radar parameters as the synthetic millimeter wave signal; The 3D human body mesh information is projected onto a 2D plane by setting the virtual camera position and resolution, and for each pixel, a corresponding depth value is calculated to obtain the depth map.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images