Multilevel human body grid reconstruction method and system based on millimeter waves

Abstract

The invention discloses a multilevel human body grid reconstruction method and system based on millimeter waves. The method comprises the following steps: for a target human body, collecting a reflection signal of a millimeter-wave radar and carrying out filtering preprocessing to obtain a preprocessed millimeter-wave radar signal; based on the millimeter wave radar signal, extracting a multi-level two-dimensional thermodynamic diagram which comprises a thermodynamic diagram generated by a moving body part, a thermodynamic diagram generated by a trunk part and a thermodynamic diagram generated by a static body part; inputting the multi-level two-dimensional thermodynamic diagram into a trained human body 3D grid estimation model to obtain predicted values of a human body network, including a posture vector, a shape vector and a translation vector of the human body network; and inputting the predicted value of the human body grid into a neutral parameterized human body model, and estimating skeleton and grid information of a target human body. The method can comprehensively capture the characteristics of each part of the target body, especially enhances the recognition capability of static parts, and improves the reconstruction accuracy and integrity of each part of the human body.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a millimeter-wave-based multi-level human body mesh reconstruction method and system. Background Art

[0002] The human body mesh reconstruction task refers to the process of converting the surface shape of the human body into a three-dimensional mesh model using sensor data or image data. This is widely used in many downstream tasks such as human body reconstruction and human-computer interaction.

[0003] With the continuous advancement and popularization of camera technology, such as the application of high-definition cameras and depth cameras, more abundant and accurate image data are provided for vision-based human mesh reconstruction. The vision-based human mesh reconstruction scheme is to capture images or video sequences of the human body with a camera and use computer vision technology to reconstruct the three-dimensional model of the human body. Although the vision-based human mesh reconstruction method has certain advantages, such as easy implementation and low cost, it also has some disadvantages. For example, it has high requirements for environmental conditions such as lighting and background, and is easily disturbed by the external environment. In addition, the use of cameras to obtain human body images may involve personal privacy issues, and reasonable privacy protection measures need to be taken.

[0004] At present, millimeter wave radar technology has been widely used in the field of human perception and monitoring. It has the advantages of strong penetration and high resolution, and has important application prospects in medical rehabilitation, security monitoring, intelligent transportation and other fields. In terms of human body mesh reconstruction, millimeter wave radar can accurately reconstruct human posture and movement by capturing and analyzing the reflected signals from different parts of the human body. For example, the deep learning framework mmMesh is used to reconstruct dynamic 3D human body meshes from millimeter wave signals. In order to mitigate the impact of environmental noise and multipath effects, mmMesh uses an attention mechanism to distinguish point clouds, and points reflected by the subject tend to have higher quality. A rough estimate of shape and posture can then be generated. In order to correctly align points and make the human body mesh more accurate, mmMesh dynamically selects some "virtual positions" near the subject as anchors, each of which is associated with a part of the human body. According to the anchor point position, mmMesh dynamically groups the 3D point cloud into several subsets, each corresponding to a different body part. At the same time, local structures and associations can be learned. Finally, mmMesh incorporates the SMPL (human parameterized model) model as an additional constraint, allowing the entire 3D human mesh to be represented with 86 parameters and without the need to calculate the positions of thousands of vertices.

[0005] In the prior art, a 3D posture tracking system m3Track performs convolution operations on the range-Doppler profile based on different Doppler responses and motion speeds to detect all users and their corresponding distances, while using MVDR to separate all users from different angles. Given the distance and angle, m3Track extracts the millimeter wave profile of each user based on a three-cylinder model containing the head, neck, chest, arms, and legs. Each profile contains two range-angle profiles (spatial features) and one range-Doppler feature (temporal feature). In addition, m3Track designs a two-stream deep learning architecture that takes spatial features and temporal features as input to extract global body shape features and local body motion features. By connecting these features, a regression model is applied to predict the 3D coordinates of skeletal joints and reconstruct 3D body postures. For tracking postures in multi-user scenarios, the reconstructed postures need to be accurately mapped into the 3D space of the real world. m3Track uses the K-means algorithm to divide the point cloud into multiple clusters. Each cluster center corresponds to a posture joint, and the optimal mapping relationship is obtained by searching for the minimum mapping error. Considering that the 3D coordinates calculated above are in the polar coordinate system, m3Track further designs a coordinate-dependent extended Kalman filter to calculate the user's position in the Cartesian coordinate system. With the help of position trajectories, m3Track achieves continuous 3D posture tracking in real-world multi-user scenarios.

[0006] After analysis, it is found that the existing millimeter-wave-based human body mesh reconstruction scheme mainly focuses on dynamic movement and does not adequately handle static body parts. This may lead to sparse or missing reflected signals from static parts in static scenes, especially when certain parts of the human body remain stationary (for example, when sitting or standing, only the upper body moves and the lower body is stationary). In static scenes, the energy reflected by static targets is often mixed with the energy reflected by surrounding static objects, making it easy for the signal processing link to mistakenly treat the reflected signals of static targets as clutter and filter them out. Therefore, due to the weak or missing signals reflected from static parts, traditional methods often find it difficult to accurately capture the reflected signals from these parts, thus affecting the accuracy of reconstructing static body parts.

[0007] In summary, millimeter-wave radar has the following problems when reconstructing human body mesh: the reflected signals from static body parts are sparse or missing, which makes it difficult for traditional millimeter-wave radar systems to accurately detect the reflected signals from static targets, thus affecting the accurate reconstruction of various parts of the human body; in static scenes, the energy reflected from static targets is often mixed with the energy reflected from surrounding static objects, which makes it easy for the signal processing link to mistakenly regard the reflected signals from static targets as clutter and filter them out. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the prior art and provide a multi-level human body grid reconstruction method and system based on millimeter waves.

[0009] According to a first aspect of the present invention, a millimeter wave-based multi-level human body mesh reconstruction method is provided. The method comprises the following steps:

[0010] For the target human body, the reflected signal of the millimeter wave radar is collected and filtered preprocessed to obtain the preprocessed millimeter wave radar signal;

[0011] Based on the millimeter-wave radar signal, extracting a multi-level two-dimensional thermal map, including a thermal map generated by a moving body part, a thermal map generated by a trunk part, and a thermal map generated by a static body part;

[0012] Inputting the multi-level two-dimensional heat map into a trained human 3D mesh estimation model to obtain a predicted value of the human body network, including a posture vector, a shape vector, and a translation vector of the human body mesh;

[0013] The predicted value of the human body mesh is input into a neutral parameterized human body model to estimate the skeleton and mesh information of the target human body.

[0014] According to a second aspect of the present invention, a millimeter wave-based multi-level human body mesh reconstruction system is provided. The system comprises:

[0015] Signal acquisition module: used to collect the reflected signal of the millimeter-wave radar for the target human body and perform filtering preprocessing to obtain the preprocessed millimeter-wave radar signal;

[0016] A multi-level feature extraction module: used to extract a multi-level two-dimensional thermal map based on the millimeter wave radar signal, including a thermal map generated by a moving body part, a thermal map generated by a torso part, and a thermal map generated by a static body part;

[0017] Human body mesh reconstruction module: used to input the multi-level two-dimensional heat map into the trained human body 3D mesh estimation model to obtain the predicted value of the human body network, including the posture vector, shape vector and translation vector of the human body mesh; and then input the predicted value of the human body mesh into the neutral parameterized human body model to estimate the skeleton and mesh information of the target human body.

[0018] Compared with the prior art, the advantage of the present invention is that in order to accurately reconstruct the 3D grid of each part of the human body and obtain the rich millimeter wave reflection information of each part of the target body as the target representation, the multi-level human body grid reconstruction method based on millimeter waves provided by the present invention designs a multi-level feature extraction method. For moving body parts, the Doppler effect and angle change generated by the movement are used to capture the spatial and temporal characteristics of the target movement; for the trunk part, the physiological signals generated by small movements, such as chest movement caused by breathing, heartbeat, etc., are used as auxiliary information to extract features. By analyzing the small signal changes caused by chest movement, the reflection characteristics of the trunk part can be more accurately extracted, thereby enhancing the detection and tracking capabilities of these parts; for other static body parts, the reflection information about these parts of the body is extracted from the static clutter. By combining these different levels of feature extraction methods, the characteristics of each part of the target body can be fully captured, thereby realizing the accurate reconstruction of the 3D grid of the human body.

[0019] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0021] Figure 1 is a flow chart of a multi-level human body mesh reconstruction method based on millimeter waves according to an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of an IWR1843BOOST virtual antenna array according to an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of a human body 3D mesh estimation model according to an embodiment of the present invention;

[0024] Figure 4 4 is a schematic diagram of the encoder structure of a human body 3D mesh estimation model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0026] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0027] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0028] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0029] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0030] The multi-level human body mesh reconstruction scheme based on millimeter waves proposed in this invention can capture the characteristics of each part of the target body comprehensively by extracting and combining multi-level features, and realize accurate reconstruction of the human body 3D mesh. Figure 1 As shown, the provided millimeter wave-based multi-level human body mesh reconstruction method includes the following steps:

[0031] Step S110, for the target human body, collect the reflected signal of the millimeter wave radar and perform filtering preprocessing to obtain a preprocessed millimeter wave radar signal.

[0032] Specifically, the process of collecting the reflected signal of the millimeter wave radar and preprocessing it includes:

[0033] Step S111, obtaining the original millimeter wave signal.

[0034] For a target at a distance d from the radar, the intermediate frequency (IF) signal generated by the mixer of the transmit and receive signals of each Tx-Rx pair can be simply defined as:

[0035]

[0036] Where α represents the amplitude of the signal. The frequency and phase of the IF signal are and φ 0 =2πf 0 τ, τ is the round trip delay, f 0 is the starting frequency of the chirp, and c is the speed of light. is the effective sweep bandwidth B and chirp duration T c The calculated frequency slope. Finally, N a Tx-Rx pairs can obtain N s ×N c ×N a The IF complex real signal matrix M, N sIndicates the number of sampling points for each chirp wave, N c Indicates the number of chirp cycles in a frame. For example, set N s =128, N c =128, N a =12.

[0037] Step S112: distance-dimensional Fourier transform.

[0038] For a single target, the frequency f of the IF signal remains unchanged, and the distance to the target can be calculated as In the case of multiple reflection sources, the signal matrix M is first subjected to a windowed FFT (i.e., Range-FFT, RangeFourier Frequency Transform) in the first dimension to obtain a value of Matrix M R ,in is the length of Range-FFT. For example, if the window function is Hanning window,

[0039] Step S113, eliminating static clutter.

[0040] Static clutter removal aims to remove the clutter generated by the reflection of stationary objects from the received signal in order to detect dynamic targets more accurately. R Averaging in the second dimension yields the static clutter matrix The matrix M R and the static clutter matrix After subtraction, the Range-FFT matrix after removing static clutter is obtained

[0041] Step S120, based on the preprocessed millimeter-wave radar signal, extracting a multi-level two-dimensional thermal map, including a thermal map generated by moving body parts, a thermal map generated by torso parts, and a thermal map generated by static body parts.

[0042] In this step, multi-level thermal map features of the target are extracted from the preprocessed millimeter wave signal, including thermal map features generated by moving body parts, thermal map features generated by torso parts, and thermal map features generated by other static body parts, including:

[0043] Step S121, extracting the thermal map features generated by the moving body parts.

[0044] 1) Range-Doppler heat map generation

[0045] Assume that the radar cBy sending a chirp wave every second and observing the phase difference ω between adjacent chirps, the speed can be estimated Specifically, for the matrix After applying windowed FFT (Doppler Fourier Frequency Transform, Doppler-FFT) in the second dimension, multiple reflection sources moving at different speeds can be identified by performing Doppler-FFT on the chirp phase. Finally, a value of The matrix M RD ,in is the length of Doppler-FFT. RD After summing in the third dimension, we get the magnitude Distance-Doppler heat map For example, the window function is the Hanning window,

[0046] 2) Generation of range-azimuth-elevation heat map.

[0047] By using multiple antennas simultaneously and comparing the phase difference between adjacent receiving antennas, the angle of arrival (AoA) of the target can be estimated. Due to the spatial position differences between the receivers, the signal received by each receiver has a slight phase difference, which depends on the relative position and AoA of the receiving antennas. If there are multiple antennas in each direction, the azimuth and elevation angles can be calculated, expressed as θ a and θ e , or θ when referring to these two angles (a，e) Assume that there are N azimuth and elevation angles. a ×N e linearly spaced receiving antennas in different directions θ (a，e)m If there are M targets on the network, each target can be regarded as a signal source, and the receiving antenna array will receive a set of weighted sum signals, expressed as:

[0048]

[0049] Among them, M means that it contains M data sources, s(θ (a，e)m ) represents the signal arrival angle θ (a，e)m The steering vector of the phase difference between the receivers, α m represents the amplitude, and n is the noise. AoA estimation can be modeled as estimating θ for each object m based on a set of receive antenna data x. (a，e) Estimate the value of s(θ (a，e)m) method uses the correlation between the receiver data x and the steering vector of the candidate angle. In one embodiment, a beamforming method is used to calculate a beamforming vector for the N antenna arrays. a The weight set of virtual receiving antennas (including azimuth and elevation angles) And for all possible angles θ (a，e)m ∈Θ, where θ a ∈[-π,π],θ e ∈[-π,π] When a list of weights is applied to the receiver data x, the signal from direction θ will obtain constructive interference. By searching all possible angles θ (a,e) , a power spectrum of size Θ can be obtained, where high power in the spectrum indicates that there is a data source in that direction. In one embodiment, the IWR1843BOOST millimeter wave radar of Texas Instruments is used, which has a three-transmit and four-receive antenna array. The virtual antenna layout is as follows: Figure 2 As shown in the figure, the interval between adjacent antennas is half a wavelength (i.e., λ / 2). As the input of MVDR beamforming and calculating the power spectrum, we can finally get The distance-azimuth-elevation heat map of a and N e Represents the length of the steering vector of azimuth and elevation respectively. In order to reduce the computational complexity of the human body mesh reconstruction network, the distance-azimuth-elevation heat map is summed in the second and third dimensions to obtain a value of The distance-azimuth heat map and size are For example, N a and N e Both are set to 64.

[0050] Step S122, extracting the thermal map features generated by the torso.

[0051] 1) Range-micromotion Doppler heat map generation

[0052] For the torso, the physiological signals generated by tiny movements, such as chest movement caused by breathing and heartbeat, are used to analyze the tiny signal changes caused by chest movement to obtain the distance-micro-motion Doppler thermal map.

[0053] Specifically, by continuously acquiring multiple frames of static clutter matrices And stack these data frames one by one to get a size of The matrix M sc , where N f is the number of data frames. For example, N f =64.

[0054] By performing inter-frame static noise (i.e., sc Averaging in the first dimension yields the static clutter matrix The matrix M sc and the static clutter matrix Subtract) to get the matrix after removing static clutter

[0055] Next, the matrix After applying the windowed FFT (i.e., micro-Doppler FFT) on the first dimension, we can finally obtain a value of The matrix M RMMD ,in is the length of the micro-Doppler FFT. RMMD After summing in the third dimension, we get the magnitude Distance-Micro Motion Doppler Heat Map For example, the window function is the Hanning window,

[0056] 2) Range-azimuth-elevation heat map generation

[0057] Similar to the extraction of thermal map features generated by moving body parts, the distance-micro-motion Doppler thermal map As the input of MVDR (minimum mean square distortionless response) beamforming, we can finally get a size of The distance-azimuth-elevation heat map of a and N e Represent the length of the steering vector of azimuth and elevation respectively. In order to reduce the computational complexity of the human body mesh reconstruction network, the distance-azimuth-elevation heat map is summed in the second and third dimensions to obtain a value of The distance-azimuth heat map and size are For example, N a and N e Both are set to 64.

[0058] Step S123: extracting the thermal map features generated by other static body parts

[0059] Specifically, the zero Doppler region (static clutter) ) to extract the range-azimuth-elevation heatmaps generated by other static body parts. As the input of MVDR beamforming, we can finally get The distance-azimuth-elevation heat map of a and N eRepresent the length of the steering vector of azimuth and elevation respectively. In order to reduce the computational complexity of the human body mesh reconstruction network, the distance-azimuth-elevation heat map is summed in the second and third dimensions to obtain a value of The distance-azimuth heat map and size are For example, N a and N e Both are set to 64.

[0060] Step S130, input the multi-level two-dimensional heat map into the trained human 3D mesh estimation model to obtain the predicted value of the human body network, and then use the neutral parameterized human body model to estimate the skeleton and mesh information of the target human body.

[0061] The extracted multi-level two-dimensional heat map is used as a human 3D mesh estimation model to estimate the skeleton and bone information of the target human body and achieve accurate reconstruction of all parts of the target body.

[0062] The human body 3D mesh estimation model can adopt various types of neural network models, such as a deep learning model that combines LSTM and CNN.

[0063] In one embodiment, the input of the human body 3D mesh estimation model is 8 types of 2D heat maps obtained after multi-level heat map feature extraction, which are:

[0064] 1) Heat map generated by moving body parts

[0065] Distance-Doppler heat map, The distance-azimuth heat map and Distance-pitch angle heat map.

[0066] 2) Heat map generated by the trunk

[0067] Distance-micro-Doppler heat map, The distance-azimuth heat map and Distance-pitch angle heat map.

[0068] 3) Heat maps generated by other static body parts

[0069] The distance-azimuth heat map and Distance-pitch angle heat map.

[0070] In one embodiment, the human body 3D mesh estimation network model structure includes 8 encoders and a long short-term memory (LSTM) unit, such as Figure 3 As shown in Figure 2, the structure of each encoder is the same. For example, the encoder contains 5 residual blocks and a fully connected layer (Fully-ConnectedLayer, FCL), as shown in Figure 2. Figure 4 As shown in the figure. The kernel size of the five residual blocks is 3×3, and the number of filters is 16, 32, 64, 128, and 256 respectively. After the input data passes through the residual block, it will be flattened into a one-dimensional vector, and the one-dimensional vector will be input into the fully connected layer. Finally, the fully connected layer outputs a feature vector of size 128.

[0071] The eight heat maps are input into their respective encoders. After obtaining the feature vectors, the eight feature vectors are concatenated into a one-dimensional vector and input into LSTM to learn the activity information of the current frame and the previous frame. For example, LSTM contains three layers: input layer, hidden layer, and output layer. The number of hidden states is 512 dimensions and the output is 128 dimensions. After obtaining the features of LSTM output, they are input into FCL to obtain the posture vector, shape vector, and translation vector of the human body mesh.

[0072] Finally, the predicted values ​​output by FCL are input into the neutral SMPL model (there is no gender information in the neutral SMPL model) to output the skeleton and mesh of the subject. The SMPL model is a parametric human body model that decomposes the human body into shape parameters. and pose parameters to estimate the 3D human mesh. Shape parameters can be used to control the variation in height, weight, and body proportions of individuals. Pose parameters are used to control the pose of the 3D human mesh. The 3D human mesh includes the 3D position information of 6890 vertices. In addition, the SMPL model will also output the 3D position information of 24 joints.

[0073] In one embodiment, the human body 3D mesh estimation model is trained with minimization of a set overall loss function as the optimization goal. For example, the overall loss function includes two parts, one for the SMPL model parameters and the other for the human body joint points and vertex positions.

[0074] Specifically, for the loss of SMPL model parameters, the 10 parameters of the SMPL model that control human body shape are used to calculate the human body shape parameters predicted by the network. The mean square error (MSE) between the true human shape parameter β is used as the loss to be optimized. For the remaining 72 parameters that control human posture, the three-dimensional vector of each joint point based on Rodrigues is first converted into a rotation matrix represented by a 3×3 size matrix. The rotation matrix predicted by the network is calculated The MSE with the true rotation matrix θ is used as the loss to be optimized. For the translation vector, calculate the translation vector predicted by the network The MSE between the actual translation vector α and the original translation vector α is used as the loss to be optimized. Therefore, the loss function of the SMPL model parameters is Contains three items:

[0075]

[0076] After inputting the predicted SMPL parameters into the SMPL model, the 3D positions of the 24 joints of the human body and the 3D coordinates of 6890 vertices can be obtained. Calculate the predicted 3D position coordinates of the 24 joints The mean absolute error (MAE) of the actual 24 joint point 3D position coordinates x is used as the loss to be optimized. Similarly, the predicted 3D position coordinates of the 6890 vertices are The MAE of the three-dimensional position coordinates v of the real 6890 vertices is used as the loss to be optimized. In addition, the speed is calculated by predicting the joint point positions between two adjacent frames. And the actual joint speed between two adjacent frames Calculate MAE as the loss to be optimized. This loss function can make the joint points between two consecutive frames generated by the network smoother. Therefore, the loss function used for the joint points and vertex positions of the human body is Contains three items:

[0077]

[0078] Where T is the number of frames of input data.

[0079] The overall loss function can be set as the loss function of the SMPL model parameters And the loss function of the human joint point and vertex position The sum, or weighted sum. For example, the overall loss function is set to:

[0080]

[0081] The total dimension of the human 3D mesh estimation model output is 157, including 3 translated 3D positions, 10 SMPL model human shape parameters, and 144 human posture parameters. After inputting this output into the SMPL model, the estimated 3D positions of 6890 human mesh vertices and 3D position information of 24 joint points can be obtained.

[0082] After the human body 3D mesh estimation model is trained, it can be used for actual human body mesh reconstruction. The model application process is basically similar to the training process and will not be repeated here. During the model application process, in order to facilitate user observation and analysis, the reconstructed 3D human body mesh data can also be visualized.

[0083] It should be understood that the specific structure of the human body 3D mesh estimation model can be constructed according to actual needs, such as setting different numbers of encoders, numbers of residual blocks or different convolution kernel sizes, etc. The present invention is not limited to this.

[0084] It should be noted that the training process of the model involved in the present invention can be performed offline on a server or in the cloud, and the trained model can be embedded in an electronic device to achieve real-time human body mesh reconstruction. The electronic device can be a terminal device or a server, and the terminal device includes any terminal device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sale (POS), a car computer, an intelligent wearable device (smart watch, virtual reality glasses, virtual reality helmet, etc.). The server includes but is not limited to an application server or a Web server, and can be an independent server or a cluster server or a cloud server, etc.

[0085] Correspondingly, the present invention also provides a millimeter-wave-based multi-level human body mesh reconstruction system for implementing one or more aspects of the above method. For example, the system includes: a signal acquisition module for collecting the reflected signal of the millimeter-wave radar for the target human body and performing filtering preprocessing to obtain the preprocessed millimeter-wave radar signal; a multi-level feature extraction module for extracting a multi-level two-dimensional thermal map based on the millimeter-wave radar signal, including a thermal map generated by a moving body part, a thermal map generated by a torso part, and a thermal map generated by a static body part; a human body mesh reconstruction module for inputting the multi-level two-dimensional thermal map into a trained human body 3D mesh estimation model to obtain a predicted value of the human body network, including a posture vector, a shape vector, and a translation vector of the human body mesh; and then inputting the predicted value of the human body mesh into a neutral parameterized human body model to estimate the skeleton and mesh information of the target human body. Further, it can also include a result display module for visually displaying the reconstructed 3D human body mesh data to facilitate user observation and analysis.

[0086] In summary, compared with the prior art, the present invention has the following advantages:

[0087] 1) When dealing with static targets, traditional methods tend to misjudge their reflected signals as environmental noise and filter them out, which affects the accuracy of detecting and tracking the target torso. The present invention uses physiological signals generated by small movements, such as chest movement caused by breathing and heartbeat, as auxiliary information to more accurately extract the reflection characteristics of the torso, thereby enhancing the recognition ability of static targets (such as the torso).

[0088] 2) The present invention adopts a multi-level feature extraction method. The moving body parts are analyzed, and the spatial and temporal characteristics of the target movement are captured using information such as the Doppler effect and angle changes. For the torso parts, physiological signals generated by small movements, such as chest movement caused by breathing and heartbeat, are used as auxiliary information to extract features. By analyzing the small signal changes caused by chest movement, the reflection characteristics of the torso parts can be more accurately extracted, thereby enhancing the detection and tracking capabilities of these parts. For other static body parts, the reflection information about these body parts is extracted from the static clutter. This multi-level feature extraction method can make full use of different information sources to comprehensively capture the characteristics of each part of the target body, providing more accurate and complete data support for human body mesh reconstruction.

[0089] 3) The multi-level feature extraction method proposed in the present invention realizes the accurate reconstruction of the human body 3D mesh by comprehensively capturing the features of each part of the target body, which solves the shortcomings of traditional methods in dealing with static body parts, especially improves the perception ability of static target reflection signals, and can more accurately understand the movement state and posture of the human body. It can not only be used for human body mesh reconstruction in commercial millimeter-wave radar, but also meet the market demand for high-precision and high-stability human body mesh reconstruction technology in multiple fields, such as medical rehabilitation, security monitoring, human posture recognition and other fields.

[0090] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0091] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0092] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0093] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0094] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0095] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0096] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0097] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0098] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A multi-level human body mesh reconstruction method based on millimeter waves, comprising the following steps: For the target human body, the reflected signal of the millimeter wave radar is collected and filtered preprocessed to obtain the preprocessed millimeter wave radar signal; Based on the millimeter-wave radar signal, extracting a multi-level two-dimensional thermal map, including a thermal map generated by a moving body part, a thermal map generated by a trunk part, and a thermal map generated by a static body part; Inputting the multi-level two-dimensional heat map into a trained human 3D mesh estimation model to obtain a predicted value of the human body network, including a posture vector, a shape vector, and a translation vector of the human body mesh; The predicted value of the human body mesh is input into a neutral parameterized human body model to estimate the skeleton and mesh information of the target human body.

2. The method according to claim 1, characterized in that The human body 3D mesh estimation model includes multiple encoders and long short-term memory units, each encoder corresponds to a multi-level two-dimensional heat map, and is used to extract feature vectors from the corresponding multi-level two-dimensional heat map. The feature vectors extracted by the multiple encoders are then spliced ​​into a one-dimensional vector and input into the long short-term memory unit to learn the activity information of the current frame and the previous frame. The output features of the long short-term memory unit are passed to the first fully connected layer to obtain the predicted value of the human body network.

3. The method according to claim 2, characterized in that The multiple encoders have the same structure, each encoder includes multiple residual blocks and a second fully connected layer, and the input data is flattened into a one-dimensional vector after passing through the multiple residual blocks, and then input into the second fully connected layer.

4. The method according to claim 1, characterized in that The overall loss function for training the human 3D mesh estimation model is set to: in: in, is the loss function of the neutral parameterized human body model parameters, is the predicted human body morphology parameter, β is the real human body morphology parameter, is the predicted rotation matrix, θ is the actual rotation matrix, is the predicted translation vector, α is the actual translation vector α, is the predicted three-dimensional position coordinate of the joint point, is the real three-dimensional position coordinate of the joint point, is the three-dimensional position coordinate of the predicted vertex, is the three-dimensional position coordinate of the real vertex, It is the speed calculated based on the predicted joint point position between two adjacent frames. is the actual joint velocity between two adjacent frames, t is the frame index, and T is the number of frames.

5. The method according to claim 1, characterized in that Collecting the reflected signal of the millimeter wave radar and performing filtering preprocessing includes the following steps: For a target with a distance d from the radar, collect the original millimeter wave signal and get N s ×N c ×N a The intermediate frequency complex real signal matrix M, where N s Indicates the number of sampling points for each chirp wave, N c Indicates the number of chirp cycles in a frame, N a Indicates the number of transmit-receive pairs; Perform distance-dimensional Fourier transform on the signal matrix M and obtain a size of Matrix M R ,in is the length of the distance-dimensional Fourier transform; For the matrix M R Averaging in the second dimension yields the static clutter matrix The matrix M R and the static clutter matrix After subtraction, the distance-dimensional Fourier transform matrix after removing static clutter is obtained:

6. The method according to claim 5, characterized in that The multi-level two-dimensional heat map includes a distance-Doppler heat map generated by moving body parts, a distance-azimuth heat map generated by moving body parts, a distance-pitch angle heat map generated by moving body parts, a distance-micro-motion Doppler heat map generated by torso parts, a distance-azimuth heat map generated by torso parts, a distance-pitch angle heat map generated by torso parts, a distance-azimuth heat map generated by static body parts, and a distance-pitch angle heat map generated by static body parts, wherein the distance-micro-motion Doppler heat map generated by torso parts is obtained by analyzing the tiny signal changes caused by chest movement, and the tiny signal includes breathing or heartbeat.

7. The method according to claim 6, characterized in that The distance-micro-motion Doppler thermal map generated by the torso is obtained according to the following steps: Continuously collect multiple frames of static clutter matrix The collected data frames are superimposed one by one to obtain a size of The matrix M sc , where N f is the frame number of the data frame; For the matrix M sc Averaging in the first dimension gives the static clutter matrix The matrix M sc and the static clutter matrix Subtract to get the matrix after removing static clutter Pair Matrix After applying the windowed FFT on the first dimension, we get The matrix M RMMD ,in is the length of the micro-Doppler FFT; For the matrix M RMMD After summing in the third dimension, we get Distance-Micro Motion Doppler Heat Map 8. The method according to claim 6, characterized in that The distance-azimuth heat map generated by the static body part and the distance-elevation heat map generated by the static body part are obtained according to the following steps: Static clutter As the input of MVDR beamforming, we get The distance-azimuth-elevation heat map of a and N e The lengths of the steering vectors representing the azimuth and elevation angles, respectively; After summing the range-azimuth-elevation heat map in the second and third dimensions, we get The distance-azimuth heat map and size are Distance-pitch angle heat map.

9. A millimeter-wave-based multi-level human body mesh reconstruction system, comprising: Signal acquisition module: used to collect the reflected signal of the millimeter-wave radar for the target human body and perform filtering preprocessing to obtain the preprocessed millimeter-wave radar signal; A multi-level feature extraction module: used to extract a multi-level two-dimensional thermal map based on the millimeter wave radar signal, including a thermal map generated by a moving body part, a thermal map generated by a torso part, and a thermal map generated by a static body part; Human body mesh reconstruction module: used to input the multi-level two-dimensional heat map into the trained human body 3D mesh estimation model to obtain the predicted value of the human body network, including the posture vector, shape vector and translation vector of the human body mesh; and then input the predicted value of the human body mesh into the neutral parameterized human body model to estimate the skeleton and mesh information of the target human body.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images