A multi-level human mesh reconstruction method and system based on millimeter waves

Abstract

The application discloses a multi-level human body grid reconstruction method and system based on millimeter waves. The method comprises the following steps: for a target human body, collecting reflection signals of a millimeter wave radar and performing filtering pretreatment to obtain pretreated millimeter wave radar signals; based on the millimeter wave radar signals, extracting multi-level two-dimensional heat maps, including a heat map generated by a moving body part, a heat map generated by a trunk part and a heat map generated by a static body part; inputting the multi-level two-dimensional heat maps into a trained human body 3D grid estimation model to obtain a prediction value of a human body network, including a posture vector, a shape vector and a translation vector of a human body grid; and inputting the prediction value of the human body grid into a neutral parameterized human body model to estimate skeleton and grid information of the target human body. The application can comprehensively capture features of each part of the target body, especially enhances the recognition ability of static parts, and improves the reconstruction accuracy and integrity of each part of the human body.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly, to a multi-level human body mesh reconstruction method and system based on millimeter waves. BACKGROUND

[0002] The human body mesh reconstruction task refers to the process of converting the surface shape of a human body into a three-dimensional mesh model using sensor data or image data, which has wide application in human body reconstruction, human-computer interaction and many other downstream tasks.

[0003] With the continuous progress and popularization of camera technology, such as the application of high-definition cameras, depth cameras, etc., more abundant and accurate image data is provided for visual-based human body mesh reconstruction. The visual-based human body mesh reconstruction scheme is to capture the image or video sequence of the human body using the camera, and use computer vision technology to reconstruct the three-dimensional model of the human body. Although the visual-based human body mesh reconstruction method has certain advantages, such as easy to implement, low cost, etc., it also has some disadvantages, such as high requirements for environmental conditions such as light, background, etc., easy to be disturbed by external environment, and using camera to obtain human body image may involve personal privacy issues, which requires reasonable privacy protection measures.

[0004] At present, millimeter wave radar technology has been widely used in human body perception and monitoring field, which has strong penetration, high resolution and other advantages, and has important application prospect in medical rehabilitation, security monitoring, intelligent transportation and other fields. In the aspect of human body mesh reconstruction, millimeter wave radar can realize accurate reconstruction of human body posture and action by capturing and analyzing the reflection signals of different parts of the human body. For example, the deep learning framework mmMesh is used to reconstruct dynamic 3D human body mesh from millimeter wave signals. In order to reduce the influence of environmental noise and multipath effect, mmMesh uses attention mechanism to distinguish point clouds, and the points reflected by the main body often have higher quality. Then a rough estimate of shape and pose can be generated. In order to correctly align the points and make the human body mesh more accurate, mmMesh dynamically selects some "virtual positions" near the main body as anchor points, each of which is associated with a part of the human body. According to the anchor point position, mmMesh dynamically groups the 3D point cloud into several subsets, each of which corresponds to a different body part. At the same time, local structure and association can be learned. Finally, mmMesh uses the SMPL (parametric model of human body) model as an additional constraint, allowing the entire 3D human body mesh to be represented with 86 parameters, and does not need to calculate the positions of thousands of vertices.

[0005] In the prior art, a 3D pose tracking system m3Track performs convolution operations on range-Doppler profiles based on different Doppler responses and motion velocities in order to detect all users and their corresponding ranges, while separating all users from different angles using MVDR. Given the range and angle, m3Track extracts the millimeter-wave profile of each user according to a 3-cylinder model containing head-neck, chest-arm and legs. Each profile contains two range-angle profiles (spatial features) and a range-Doppler feature (temporal feature). In addition, m3Track designs a dual-stream deep learning architecture that takes spatial and temporal features as input to extract global body shape features and local body motion features. By connecting these features, a regression model is applied to predict the 3D coordinates of the skeletal joints and reconstruct the 3D body pose. For tracking poses in a multi-user scenario, it is necessary to accurately map the reconstructed pose to the real-world 3D space. m3Track uses the K-means algorithm to divide the point cloud into multiple clusters. Each cluster center corresponds to a pose joint, and the optimal mapping relationship is obtained by searching for the minimum mapping error. Considering that the 3D coordinates calculated above are in the polar coordinate system, m3Track further designs an extended Kalman filter related to coordinates to calculate the position of the user in the Cartesian coordinate system. With the position trajectory, m3Track realizes continuous 3D pose tracking in a real-world multi-user scenario.

[0006] Through analysis, the existing human body grid reconstruction scheme based on millimeter wave mainly focuses on dynamic motion, and the processing of static body parts is insufficient, which may lead to sparse or missing reflection signals of static parts in static scenes, especially when some parts of the human body remain stationary (for example, only the upper body moves while sitting or standing, and the lower body is stationary). In static scenes, the energy reflected by static targets is often mixed with the energy reflected by surrounding static objects, making it easy for the signal processing link to mistakenly filter out the reflection signals of static targets as clutter. Therefore, due to the weak or missing reflection signals of static parts, traditional methods often have difficulty accurately capturing the reflection signals of these parts, thereby affecting the accuracy of reconstructing static body parts.

[0007] In summary, the millimeter wave radar has the following problems when performing human body grid reconstruction: the reflection signals of static body parts are sparse or missing, making it difficult for traditional millimeter wave radar systems to accurately detect the reflection signals of static targets, thereby affecting the accurate reconstruction of each part of the human body; in static scenes, the energy reflected by static targets is often mixed with the energy reflected by surrounding static objects, which makes it easy for the signal processing link to mistakenly filter out the reflection signals of static targets as clutter. SUMMARY

[0008] The object of the present application is to overcome the defects of the prior art described above, and to provide a multi-level human body grid reconstruction method and system based on millimeter waves.

[0009] According to a first aspect of the present application, a multi-level human body grid reconstruction method based on millimeter waves is provided. The method comprises the following steps:

[0010] For a target human body, the reflection signal of a millimeter wave radar is collected and preprocessed by filtering to obtain a preprocessed millimeter wave radar signal;

[0011] Based on the millimeter wave radar signal, a multi-level two-dimensional heat map is extracted, including a heat map generated by a moving body part, a heat map generated by a trunk part, and a heat map generated by a static body part;

[0012] The multi-level two-dimensional heat map is input into a trained human body 3D grid estimation model to obtain a predicted value of the human body network, including a posture vector, a shape vector and a translation vector of the human body grid;

[0013] The predicted value of the human body grid is input into a neutral parameterized human body model to estimate the skeleton and grid information of the target human body.

[0014] According to a second aspect of the present application, a multi-level human body grid reconstruction system based on millimeter waves is provided. The system comprises:

[0015] The signal acquisition module is used for collecting the reflection signal of a millimeter wave radar for a target human body and pre-processing by filtering to obtain a preprocessed millimeter wave radar signal;

[0016] The multi-level feature extraction module is used for extracting a multi-level two-dimensional heat map based on the millimeter wave radar signal, including a heat map generated by a moving body part, a heat map generated by a trunk part, and a heat map generated by a static body part;

[0017] The human body grid reconstruction module is used for inputting the multi-level two-dimensional heat map into a trained human body 3D grid estimation model to obtain a predicted value of the human body network, including a posture vector, a shape vector and a translation vector of the human body grid; and further inputting the predicted value of the human body grid into a neutral parameterized human body model to estimate the skeleton and grid information of the target human body.

[0018] Compared with the prior art, the advantages of the present application are that, in order to realize accurate reconstruction of the 3D mesh of each part of the human body and obtain rich millimeter wave reflection information of each part of the target body as the target representation, the multi-level human body mesh reconstruction method based on millimeter waves provided by the present application designs a multi-level feature extraction method, for the moving body parts, the Doppler effect and angle change caused by the movement are used to capture the spatial and temporal features of the target movement; for the trunk part, the physiological signals caused by the slight movement, such as the chest movement caused by breathing and heartbeat, are used as auxiliary information to extract features, through analyzing the slight signal changes caused by the chest movement, the reflection features of the trunk part can be more accurately extracted, thereby enhancing the detection and tracking ability of these parts; for other static body parts, the reflection information about these body parts is extracted from the static clutter. By combining these different levels of feature extraction methods, the features of each part of the target body can be comprehensively captured, thereby realizing accurate reconstruction of the 3D mesh of the human body.

[0019] Other features and advantages of the present application will become apparent from the following detailed description of illustrative embodiments thereof, which proceeds with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0021] Figure 1 is a flow chart of a multi-level human body mesh reconstruction method based on millimeter waves according to an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of an IWR1843BOOST virtual antenna array according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a human body 3D mesh estimation model according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of an encoder structure of a human body 3D mesh estimation model according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions, and numerical values of the components and steps set forth in these embodiments are not limiting to the scope of the present application unless otherwise specifically stated.

[0026] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the application or its applications or uses.

[0027] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art(s) can not be discussed in detail in this document, but should be considered as part of the description.

[0028] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Thus, other examples of the exemplary embodiments can have different values.

[0029] It should be noted that like reference numerals and letters refer to like items throughout the attached drawings, and thus once an item is defined in one drawing, it is not necessary that it be further discussed in the remaining drawings.

[0030] The multi-level human body mesh reconstruction scheme based on millimeter waves provided in the present application comprehensively captures the features of each part of the target body by extracting and combining multi-level features, and realizes accurate reconstruction of the 3D mesh of the human body. Referring to Figure 1 As shown, the multi-level human body mesh reconstruction method based on millimeter waves provided includes the following steps:

[0031] Step S110, for the target human body, the reflection signal of the millimeter wave radar is collected and pre-processed to obtain the pre-processed millimeter wave radar signal.

[0032] Specifically, the process of collecting and pre-processing the reflection signal of the millimeter wave radar includes:

[0033] Step S111, obtaining the millimeter wave original signal.

[0034] For a target with a radar distance of d, the intermediate frequency (IF) signal generated by the mixer of the transmission and reception signal of each Tx-Rx pair can be simply defined as:

[0035]

[0036] Wherein, a represents the amplitude of the signal. The frequency and phase of the IF signal are and φ0=2πf0τ, τ is the round-trip delay, f0 is the starting frequency of the chirp, and c is the speed of light. is the frequency slope calculated from the effective sweep bandwidth B and the chirp duration T c The final N a Tx-Rx pairs can obtain an IF complex real signal matrix M of N s ×N c ×N a , N s represents the number of sampling points of each chirp, and N crepresents the number of chirp cycles within a frame. For example, N s = 128, N c = 128, N a = 12.

[0037] Step S112, distance dimension Fourier transform.

[0038] For a single target, the frequency f of the IF signal remains unchanged, and the distance of the target can be calculated as In the case of multiple reflection sources, after the signal matrix M is first subjected to windowed FFT in the first dimension (i.e., Range-FFT, Range Fourier Frequency Transform), a Range-FFT matrix of size Matrix M R , where is the length of Range-FFT. For example, the window function is a Hanning window,

[0039] Step S113, static clutter removal.

[0040] Static clutter removal (Static Clutter Removal) aims to remove the clutter generated by the reflection of stationary objects from the received signal, so as to more accurately detect dynamic targets. For example, for the matrix M R , the static clutter matrix is obtained by averaging in the second dimension After the matrix M R is subtracted from the static clutter matrix , the Range-FFT matrix after removing the static clutter is obtained

[0041] Step S120, based on the preprocessed millimeter wave radar signal, extracting a multi-level two-dimensional heat map, including a heat map generated by a moving body part, a heat map generated by a trunk part, and a heat map generated by a static body part.

[0042] In this step, the multi-level heat map features of the target are extracted from the preprocessed millimeter wave signal, including the heat map features generated by the moving body part, the heat map features generated by the trunk part, and the heat map features generated by other static body parts, specifically including:

[0043] Step S121, extracting heat map features generated by a moving body part.

[0044] 1) Range-Doppler heat map generation

[0045] Assuming that the radar sends a chirp wave every T c seconds, and observes the phase difference ω between adjacent chirps, the velocity can be estimated as Specifically, for the matrix After applying a windowed FFT (Doppler Fourier Frequency Transform, Doppler-FFT) to the second dimension, multiple reflection sources moving at different speeds can be identified by performing a Doppler-FFT on the chirped phase. Ultimately, a value of [value missing] can be obtained. Matrix M RD ,in Let M be the length of the Doppler-FFT. For matrix M... RD After summing in the third dimension, the size can be obtained. Distance - Doppler Heat Map For example, the window function is the Hanning window.

[0046] 2) Generation of distance-azimuth-elevation heatmap.

[0047] The angle of arrival (AoA) of a target can be estimated by using multiple antennas simultaneously and comparing the phase difference between adjacent receiving antennas. Due to the spatial differences between the receivers, the signals received by each receiver have a slight phase difference, depending on the relative positions of the receiving antennas and the AoA. If multiple antennas are present in each direction, the azimuth and elevation angles can be calculated, denoted as θ. a and θ e Or, when referring to these two angles, they are represented as θ. (a，e) Assume there are N directions in the azimuth and elevation directions. a ×N e A linearly spaced receiving antenna, in different directions θ (a，e)m If there are M targets, each target can be considered a signal source. The receiving antenna array will receive a weighted sum of signals, represented as:

[0048]

[0049] Where M represents M data sources, s(θ) (a，e)m ) represents the signal arrival angle θ (a，e)m The steering vector of the phase difference between receivers, α m Let x represent amplitude and n represent noise. AoA estimation can be modeled as estimating θ for each object m based on a set of received antenna data x. (a，e) Value. Estimate the s(θ) of object m. (a，e)mThe method is to use the correlation between the receiver data x and the steering vector of the candidate angle. In an embodiment, the method using beamforming calculates a weight set for N a virtual receiving antennas (including azimuth and elevation angles) in the antenna array and for all possible angles θ (a，e)m ∈ Θ, where θ a ∈ [-π, π], θ e ∈ [-π, π]. When a column of weights is applied to the receiver data x, the signal from the direction θ will obtain constructive interference. By searching all possible angles θ (a,e) , a power spectrum of size Θ can be obtained, where high power in the spectrum indicates the presence of a data source in that direction. In an embodiment, a Texas Instruments IWR1843BOOST millimeter wave radar is used, which has a three-transmit-four-receive antenna array, and the virtual antenna layout is shown as Figure 2 , the spacing between adjacent antennas is half a wavelength (i.e. λ / 2). In addition, the matrix is used as the input of the MVDR beamforming and the power spectrum is calculated, and finally a distance-azimuth-elevation heat map of size is obtained, where N a and N e represent the lengths of the steering vectors of the azimuth and elevation angles, respectively. In order to reduce the computational complexity of the human body grid reconstruction network, the distance-azimuth-elevation heat map is summed in the second and third dimensions respectively, and a distance-azimuth heat map of size and a distance-elevation heat map of size are obtained. For example, N a and N e are both set to 64.

[0050] Step S122, extracting the heat map features generated by the torso part.

[0051] 1) Distance-micro-motion Doppler heat map generation

[0052] For the torso part, the physiological signals caused by slight movements, such as chest movements caused by breathing and heartbeat, are used to analyze the slight signal changes caused by chest movements to obtain a distance-micro-motion Doppler heat map.

[0053] Specifically, a plurality of frames of static clutter matrices are continuously collected and these data frames are stacked one by one to obtain a matrix M of size sc , where N f is the number of data frames. For example, N f = 64.

[0054] By performing inter-frame static clutter (i.e. subtracting the matrix M sc from the matrix M , the matrix M sc is obtained, which is free of static clutter.

[0055] Next, the matrix M is obtained by applying a windowed FFT (i.e. micro-Doppler FFT) in the first dimension. Finally, a matrix M of size N RMMD x N RMMD x N is obtained, where N is the length of the micro-Doppler FFT. By performing summation in the third dimension, a range-micro-Doppler heat map of size N is obtained. For example, the window function is a Hanning window,

[0056] 2) Range-azimuth-elevation heat map generation

[0057] Similarly to the extraction of heat map features generated by moving body parts, the range-micro-Doppler heat map M is taken as the input of MVDR (Minimum Variance Distortionless Response) beamforming. Finally, a range-azimuth-elevation heat map of size N is obtained, where N a and N e represent the length of the steering vectors for azimuth and elevation, respectively. Again, to reduce the computational complexity of the human mesh reconstruction network, summation is performed in the second and third dimensions of the range-azimuth-elevation heat map, resulting in a range-azimuth heat map of size N and a range-elevation heat map of size N . For example, N a and N e are both set to 64.

[0058] Step S123, extraction of heat map features generated by other static body parts

[0059] Specifically, the range-azimuth-elevation heat map generated by other static body parts can be extracted from the zero-Doppler region (static clutter ). For example, taking the static clutter as the input of MVDR beamforming, a range-azimuth-elevation heat map of size N is obtained, where N a and N e represent the length of the steering vectors for azimuth and elevation, respectively.denote the length of the steering vector for azimuth and elevation, respectively. To reduce the computational complexity of the human mesh reconstruction network, the distance-azimuth-elevation heatmaps are summed up in the second and third dimensions, respectively, to obtain a distance-azimuth heatmap of size and a distance-elevation heatmap of size For example, N a and N e are both set to 64.

[0060] In step S130, the multi-level two-dimensional heatmaps are input into the trained human 3D mesh estimation model to obtain the predicted value of the human network, and then the neutral parameterized human model is used to estimate the skeleton and mesh information of the target human body.

[0061] The extracted multi-level two-dimensional heatmaps are used as the human 3D mesh estimation model to estimate the skeleton and mesh information of the target human body, thereby accurately reconstructing each part of the target whole body.

[0062] The human 3D mesh estimation model can use various types of neural network models, such as a deep learning model combining LSTM and CNN.

[0063] In one embodiment, the input of the human 3D mesh estimation model is the 8 two-dimensional heatmaps obtained after multi-level heat map feature extraction, which are:

[0064] 1) heat map generated by moving body parts

[0065] distance-Doppler heat map, distance-azimuth heat map, and distance-elevation heat map.

[0066] 2) heat map generated by the torso

[0067] distance-micro-Doppler heat map, distance-azimuth heat map, and distance-elevation heat map.

[0068] 3) heat map generated by other static body parts

[0069] distance-azimuth heat map, and distance-elevation heat map.

[0070] In one embodiment, the human body 3D mesh estimation network model structure includes eight encoders and one Long-Short Term Memory (LSTM) unit, such as... Figure 3 As shown. Each encoder has the same structure. For example, an encoder contains 5 residual blocks and one fully-connected layer (FCL), as follows. Figure 4 As shown, the kernel size of each of the five residual blocks is 3×3, and the number of filters is 16, 32, 64, 128, and 256, respectively. The input data is flattened into a one-dimensional vector after passing through the residual blocks, and this one-dimensional vector is then input into the fully connected layer. Finally, the fully connected layer outputs a 128-dimensional feature vector.

[0071] Eight heatmaps are input into their respective encoders to obtain feature vectors. These eight feature vectors are then concatenated into a single one-dimensional vector, which is then fed into an LSTM to learn activity information from the current and previous frames. For example, the LSTM consists of three layers: an input layer, a hidden layer, and an output layer. The hidden states are 512-dimensional, and the output is 128-dimensional. After obtaining the features from the LSTM output, they are input into an FCL (Flexible Closure) to obtain the pose, shape, and translation vectors of the human mesh.

[0072] Finally, the predicted values ​​output by FCL are input into a neutral SMPL model (which does not contain gender information) to output the skeleton and mesh of the subject. The SMPL model is a parametric human body model that decomposes the human body into shape parameters. and posture parameters This is used to estimate a 3D human body mesh. Shape parameters can be used to control variations in height, weight, and body proportions. Pose parameters are used to control the pose of the 3D human body mesh. The 3D human body network includes the 3D position information of 6890 vertices. In addition, the SMPL model will also output the 3D position information of 24 joints.

[0073] In one embodiment, a human 3D mesh estimation model is trained with the optimization objective of minimizing a set overall loss function. For example, the overall loss function comprises two parts, used for the SMPL model parameters and the positions of human joints and vertices, respectively.

[0074] Specifically, for the loss used in the SMPL model parameters, the human morphology parameters predicted by the network are calculated for the 10 parameters controlling human morphology in the SMPL model. The Mean Square Error (MSE) between the predicted SMPL parameters and the real body shape parameters β is taken as the loss to be optimized. For the remaining 72 parameters that control the body pose, the 3D vector of each joint based on Rodrigues representation is first converted into a rotation matrix represented by a 3x3 size matrix. The rotation matrix predicted by the network is calculated The MSE between the predicted rotation matrix and the real rotation matrix θ is taken as the loss to be optimized. For the translation vector, the translation vector predicted by the network is calculated The MSE between the predicted translation vector and the real translation vector α is taken as the loss to be optimized. Therefore, the loss function of the SMPL model parameters contains three items:

[0075]

[0076] After inputting the predicted SMPL parameters into the SMPL model, the 3D coordinates of the 24 joints of the human body and the 3D coordinates of the 6890 vertices can be obtained. The 3D coordinates of the predicted 24 joints are calculated The Mean Absolute Error (MAE) between the predicted 3D coordinates of the 24 joints and the real 3D coordinates of the 24 joints x is taken as the loss to be optimized. Similarly, the 3D coordinates of the predicted 6890 vertices are calculated The MAE between the predicted 3D coordinates of the 6890 vertices and the real 3D coordinates of the 6890 vertices v is taken as the loss to be optimized. In addition, the joint position calculation speed between the predicted adjacent two frames is calculated and the real joint speed between the adjacent two frames The MAE is taken as the loss to be optimized. This loss function can make the joints generated by the network between the two consecutive frames more smooth. Therefore, the loss function for the body joint and vertex position contains three items:

[0077]

[0078] where T is the number of frames of the input data.

[0079] The overall loss function can be set as the sum or weighted sum of the loss function of the SMPL model parameters and the loss function of the body joint and vertex position . For example, the overall loss function is set as:

[0080]

[0081] The total dimension of the output of the human body 3D mesh estimation model is 157, including 3 three-dimensional positions of translation, 10 human body shape parameters of the SMPL model, and 144 human body posture parameters. After inputting the output into the SMPL model, the estimated 6890 human body mesh vertex three-dimensional positions and the three-dimensional position information of 24 joint nodes can be finally obtained.

[0082] After the human body 3D mesh estimation model is trained, it can be used for actual human body mesh reconstruction. The model application process is basically similar to the training process, and will not be described here again. In the model application process, in order to facilitate user observation and analysis, the reconstructed 3D human body mesh data can also be visualized and displayed.

[0083] It should be understood that the specific structure of the human body 3D mesh estimation model can be constructed according to actual needs, for example, setting different numbers of encoders, different numbers of residual blocks, or different sizes of convolution kernels, etc. The present application does not limit this.

[0084] It should be noted that the training process of the model involved in the present application can be performed offline on a server or in the cloud. The trained model can be embedded into an electronic device to realize real-time human body mesh reconstruction. The electronic device can be a terminal device or a server, and the terminal device includes any terminal device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sale (POS), a vehicle-mounted computer, a smart wearable device (a smart watch, virtual reality glasses, a virtual reality helmet, etc.). The server includes but is not limited to an application server or a Web server, and can be a standalone server or a cluster server or a cloud server, etc.

[0085] Correspondingly, the present application also provides a multi-level human body mesh reconstruction system based on millimeter waves, which is used to realize one or more aspects of the above method. For example, the system includes: a signal acquisition module, configured to acquire and filter the reflection signal of the millimeter wave radar to obtain a preprocessed millimeter wave radar signal; a multi-level feature extraction module, configured to extract multi-level two-dimensional heat maps based on the millimeter wave radar signal, including a heat map generated by a moving body part, a heat map generated by a trunk part, and a heat map generated by a static body part; a human body mesh reconstruction module, configured to input the multi-level two-dimensional heat maps into a trained human body 3D mesh estimation model to obtain a prediction value of a human body network, including a posture vector, a shape vector, and a translation vector of a human body mesh; and further input the prediction value of the human body mesh into a neutral parameterized human body model to estimate the skeleton and mesh information of the target human body. Further, the system can also include a result display module, configured to visualize and display the reconstructed 3D human body mesh data to facilitate user observation and analysis.

[0086] In summary, compared with the prior art, the present application has the following advantages:

[0087] 1) The traditional method is easy to misjudge the reflection signal of the static target as environmental noise and filter it out when processing the static target, which affects the detection and tracking accuracy of the target trunk. The present application can more accurately extract the reflection features of the trunk part by using the physiological signals caused by the micro-movement, such as the chest movement caused by breathing, heartbeat, etc., as auxiliary information, thereby enhancing the recognition ability of the static target (such as the trunk).

[0088] 2) The present application adopts a multi-level feature extraction method. The moving body parts are analyzed, and the spatial and temporal features of the target movement are captured by using Doppler effect and angle change information. For the trunk part, the physiological signals caused by the micro-movement, such as the chest movement caused by breathing, heartbeat, etc., are used as auxiliary information to extract features. By analyzing the micro-signal changes caused by the chest movement, the reflection features of the trunk part can be more accurately extracted, thereby enhancing the detection and tracking ability of these parts. For other static body parts, the reflection information about the body part is extracted from the static clutter. This multi-level feature extraction method can fully utilize different information sources, thereby fully capturing the features of each part of the target body and providing more accurate and complete data support for human body grid reconstruction.

[0089] 3) The multi-level feature extraction method proposed in the present application fully captures the features of each part of the target body, realizes the accurate reconstruction of the human body 3D grid, solves the shortcomings of the traditional method in processing static body parts, especially improves the perception ability of the reflection signal of the static target, can more accurately understand the motion state and posture of the human body, and can not only be applied to human body grid reconstruction in commercial millimeter wave radar, but also meet the market demand for high-precision, high-stability human body grid reconstruction technology in multiple fields, such as medical rehabilitation, security monitoring, human posture recognition, etc.

[0090] The present application can be a system, a method and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions loaded thereon for causing a processor to implement various aspects of the present application.

[0091] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0092] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0093] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0094] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0095] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0096] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0097] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0098] Embodiments of the present application have been described above, and the description is intended to be illustrative, and not restrictive, of the disclosed embodiments. Many modifications and variations of the described embodiments are possible, and all such modifications and variations are intended to be within the scope of the described embodiments. The description used herein is intended to best explain the principles of the embodiments, the practical application, and the best mode of the present application, to make this disclosure understood in the art. The scope of the present application is defined by the appended claims.

Claims

1. A method for multi-level human mesh reconstruction based on millimeter wave, comprising the following steps: collecting reflection signals of a millimeter wave radar and performing filtering pretreatment for the reflection signals to obtain pretreated millimeter wave radar signals for a target human body; extracting multi-level two-dimensional heat maps based on the millimeter wave radar signals, including a heat map generated by a moving body part, a heat map generated by a trunk part, and a heat map generated by a static body part; inputting the multi-level two-dimensional heat maps into a trained human 3D mesh estimation model to obtain predicted values of a human mesh, including a posture vector, a shape vector, and a translation vector of the human mesh; inputting the predicted values of the human mesh into a neutral parameterized human model to estimate skeleton and mesh information of the target human body; wherein the human 3D mesh estimation model comprises a plurality of encoders and a long short-term memory unit, each encoder corresponding to a multi-level two-dimensional heat map and being used to extract a feature vector from the corresponding multi-level two-dimensional heat map, and then the feature vectors extracted by the plurality of encoders are spliced into a one-dimensional vector and input into the long short-term memory unit to learn activity information of a current frame and a previous frame, and output features of the long short-term memory unit are transmitted to a first fully connected layer to obtain the predicted values of the human mesh; wherein collecting reflection signals of a millimeter wave radar and performing filtering pretreatment comprises the following steps: For a target with a radar distance of d, the original millimeter wave signal is collected to obtain an intermediate frequency complex real signal matrix M of N s ×N c ×N a , where N s represents the number of sampling points of each chirp, N c represents the number of chirp cycles in a frame, and N a represents the number of transmission-reception pairs. The distance dimension Fourier transform is performed on the signal matrix M to obtain a matrix M of size R where is the length of the distance dimension Fourier transform; The matrix M R Averaging over the second dimension gives the static clutter matrix Subtracting the static clutter matrix R and the static clutter matrix gives the range dimension Fourier transform matrix with static clutter removed wherein the multi-level two-dimensional heat maps include a range-Doppler heat map generated by a moving body part, a range-azimuth heat map generated by a moving body part, a range-pitch heat map generated by a moving body part, a range-micro-Doppler heat map generated by a trunk part, a range-azimuth heat map generated by a trunk part, a range-pitch heat map generated by a trunk part, a range-azimuth heat map generated by a static body part, and a range-pitch heat map generated by a static body part, wherein the range-micro-Doppler heat map generated by a trunk part is obtained by analyzing a slight signal change caused by chest movement, and the slight signal contains breathing or heartbeat.

2. The method of claim 1, wherein, The plurality of encoders have the same structure, and each encoder comprises a plurality of residual blocks and a second fully connected layer, and input data is flattened into a one-dimensional vector after passing through the plurality of residual blocks and then input into the second fully connected layer.

3. The method of claim 1, wherein, The range-micro-Doppler heat map generated by a trunk part is obtained according to the following steps: Continuously collect multiple frames of static clutter matrix And superimpose the collected data frames one by one to obtain a matrix M of size N sc , where N f is the number of frames of data frames; For the matrix M sc Averaging over the first dimension gives the static clutter matrix Subtracting the static clutter matrix sc from the matrix M results in a matrix without static clutter On the first dimension, a windowed FFT is applied, resulting in a matrix M of size In the second dimension, a windowed FFT is applied, resulting in a matrix M of size RMMD where N is the length of the micro-Doppler FFT.​ For the matrix M RMMD After summing over the third dimension, one obtains a range-micro-Doppler heat map of size ​ 4. The method of claim 1, wherein, The range-azimuth heat map generated by a static body part and the range-pitch heat map generated by a static body part are obtained according to the following steps: with static clutter matrix As input to the MVDR beamforming, we obtain a range-azimuth-elevation heat map of size where N k and N e denote the lengths of the steering vectors for azimuth and elevation, respectively. After summing over the second and third dimensions of the range-azimuth-elevation heat map, respectively, we obtain a range-azimuth heat map of size and a range-elevation heat map of size . 5.A system for multi-level human mesh reconstruction based on millimeter wave, comprising: a signal collection module for collecting reflection signals of a millimeter wave radar and performing filtering pretreatment for the reflection signals to obtain pretreated millimeter wave radar signals for a target human body; a multi-level feature extraction module for extracting multi-level two-dimensional heat maps based on the millimeter wave radar signals, including a heat map generated by a moving body part, a heat map generated by a trunk part, and a heat map generated by a static body part; The human body grid reconstruction module is configured to input the multi-level two-dimensional heat map into a trained human body 3D grid estimation model to obtain a predicted value of a human body grid, including a posture vector, a shape vector and a translation vector of the human body grid; and further input the predicted value of the human body grid into a neutral parameterized human body model to estimate skeleton and grid information of a target human body. The human body 3D grid estimation model includes a plurality of encoders and a long short-term memory unit, each encoder corresponding to a multi-level two-dimensional heat map and being configured to extract a feature vector from the corresponding multi-level two-dimensional heat map, and further configured to input the feature vectors extracted by the plurality of encoders into the long short-term memory unit after splicing the feature vectors into a one-dimensional vector to learn activity information of a current frame and a previous frame, and output features of the long short-term memory unit to a first fully connected layer to obtain the predicted value of the human body grid. The method further includes the following steps of collecting and filtering the reflected signal of the millimeter wave radar: For a target with a distance of d from the radar, raw millimeter wave signals are collected to obtain an intermediate frequency complex real signal matrix M of N s × N c × N a , where N s represents the number of sampling points of each chirp, N c represents the number of chirp cycles within a frame, and N a represents the number of transmission-reception pairs. The distance dimension Fourier transform is performed on the signal matrix M to obtain a matrix of size The matrix M R wherein is the length of the distance dimension Fourier transform; The matrix M R Averaging over the second dimension gives the static clutter matrix Subtracting the static clutter matrix R and the static clutter matrix gives the range dimension Fourier transform matrix with static clutter removed The multi-level two-dimensional heat map includes a distance-Doppler heat map generated by a moving body part, a distance-azimuth angle heat map generated by the moving body part, a distance-pitch angle heat map generated by the moving body part, a distance-micro-Doppler heat map generated by a trunk part, a distance-azimuth angle heat map generated by the trunk part, a distance-pitch angle heat map generated by the trunk part, a distance-azimuth angle heat map generated by a static body part, and a distance-pitch angle heat map generated by the static body part, wherein the distance-micro-Doppler heat map generated by the trunk part is obtained by analyzing a slight signal change caused by chest movement, and the slight signal contains breathing or heartbeat.

6. A computer readable storage medium having stored thereon a computer program, wherein, The computer program is executed by a processor to implement the steps of the method according to any one of claims 1 to 4. The computer program is executed by a processor to implement the steps of the method according to any one of claims 1 to 4.

Images