A method and system for constructing a human body 3D mesh model by fusing millimeter waves and images

Abstract

The application discloses a kind of millimeter wave and image fusion human body 3D grid model construction method and system, utilize commercial millimeter wave radar AWR1443BOOST and DCA1000EVM as millimeter wave signal transceiver, image acquisition is carried out, by fusing two kinds of signals, the position of human key point is extracted and the outline of human body is estimated, finally, the three-dimensional grid model of human body is regressed, the application has no requirement to illumination condition, has certain robustness to extreme weather and other conditions leading to image method failure.

Description

Technical Field

[0001] This invention belongs to the field of intelligent sensing technology, specifically relating to a method and system for constructing a 3D human body mesh model by fusing millimeter waves and images. Background Technology

[0002] The estimation and construction of three-dimensional human body networks have many practical applications. For example, by capturing different user postures, more realistic AR services can be obtained in fields such as entertainment and healthcare; by acquiring human body shape or body characteristics (such as height, obesity, gait, etc.), identity-based personnel management systems can be used to authorize and authenticate users; by recognizing continuous human movements, factory managers and supervisors can easily determine whether the operations of workers on a production line meet standards.

[0003] Currently, there is considerable work on generating 3D human body meshes, but the vast majority are based on image processing, i.e., computer vision. This method suffers from severe image clarity issues in low light or bright light, leading to mesh generation failure. Following this, researchers began to explore the possibilities of wireless signal-based methods, resulting in numerous studies on reconstructing human body meshes using radio frequency (RF) signals. RF signals are unaffected by weather or lighting conditions and generate strong reflected echoes after passing through the human body, enabling the capture of human features while overcoming the limitations of cameras. However, the complexity of wireless signal acquisition hardware and the stability of the methods remain significant challenges hindering the widespread application of this technology. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for constructing a 3D human body mesh model by fusing millimeter waves and images, which addresses the shortcomings of the prior art. It uses the fusion of wireless radio frequency signals and image information to quickly reconstruct a 3D human body mesh model, and solves the technical problems of being unable to perform 3D modeling of human movements or having poor modeling quality when the image quality is poor, as well as the instability of modeling by pure wireless signal methods.

[0005] The present invention adopts the following technical solution:

[0006] A method for constructing a 3D human body mesh model by fusing millimeter waves and images, comprising the following steps:

[0007] S1. Convert the reflected signal of the FMCW frequency modulated continuous wave signal into an intermediate frequency signal and collect the RGB image;

[0008] S2. Perform static clutter cancellation and phase calibration on the intermediate frequency signal obtained in step S1;

[0009] S3. Perform three fast Fourier transforms on the signal processed in step S2 to obtain the human body 3D point cloud corresponding to the intermediate frequency signal. Perform MaskIMG operation on the RGB image obtained in step S1 to obtain image information.

[0010] S4. Input the 3D point cloud and image information obtained in step S3 into a deep neural network to generate a human body 3D mesh, render the human body 3D mesh onto the original image, obtain an image containing the projection of the human body 3D mesh, and complete the 3D modeling of the human body mesh.

[0011] Specifically, in step S1, an FMCW frequency-modulated continuous wave signal is transmitted using a commercial millimeter-wave radar, and images are collected using a monocular RGB / IR camera.

[0012] Specifically, in step S2, the phase-shifted calibrated signal for:

[0013] =AS

[0014] Where S is the original signal and A is the phase shift vector.

[0015] Furthermore, the phase shift vector A is:

[0016] A=[1 ... ]

[0017] in, The phase shift introduced for the m-th virtual antenna, The natural base, It is the imaginary unit.

[0018] Specifically, in step S3, the phase-calibrated signal... Range-FFT and Doppler-FFT are performed sequentially. The average value of the Doppler-FFT heatmap is subtracted from all signals. The 128 points with the highest intensity are selected and 3D-FFT is performed to obtain the 3D point cloud.

[0019] Specifically, in step S4, the deep neural network includes:

[0020] The feature extraction module processes millimeter-wave signals and image signals separately, and fuses the features of the millimeter-wave signals and image signals;

[0021] The joint point recognition and contour estimation module takes the features fused by the feature extraction module as input and outputs a heat map of human joint points and a black and white human contour map.

[0022] The mesh generation module inputs the 2D joints from the joint recognition and contour estimation module and the fused features from the feature extraction module into PoseNet, outputting 72 parameters representing joint rotation. Then, it inputs the contour estimation map from the joint recognition and contour estimation module and the fused features from the feature extraction module into ShapeNet to obtain 10 parameters representing the shape. The shape parameters are then input into SMPL to generate a 3D model of the human body mesh.

[0023] Furthermore, a local attention method is used to process millimeter-wave signals. For a millimeter-wave signal X with feature dimension D, let [Q,K,V]=X Q, K, and V are the Query vector, Key vector, and Value vector, respectively. The feature transformation matrix is ​​used, and the attention weights are calculated by dividing the inner product of the Query and Key by . The result is that k SA operations are expanded in parallel and connected to their corresponding outputs. The study employed location embedding for different heads, associating each head with certain parts of the human body, and obtaining regional features through training.

[0024] Furthermore, output :

[0025]

[0026] Extend the k SA operations in parallel and connect them to the corresponding outputs:

[0027]

[0028] in, This is a weight matrix, where Q, K, and V are the Query vector, Key vector, and Value vector, respectively. This represents the characteristic dimension of millimeter-wave signals.

[0029] Furthermore, the keypoint recognition and contour estimation module is trained using L2 loss and cross-entropy loss.

[0030] Secondly, embodiments of the present invention provide a 3D human body mesh model construction system that fuses millimeter waves and images, comprising:

[0031] The acquisition module converts the reflected signal of the FMCW frequency-modulated continuous wave signal into an intermediate frequency signal and collects RGB images;

[0032] The preprocessing module performs static clutter cancellation and phase calibration on the intermediate frequency signal obtained by the acquisition module;

[0033] The training module performs three fast Fourier transforms on the signal processed by the preprocessing module to obtain the human body 3D point cloud corresponding to the intermediate frequency signal, and performs MaskIMG operation on the RGB image obtained by the acquisition module to obtain image information.

[0034] The output module inputs the 3D point cloud and image information obtained from the training module into the deep neural network to generate a 3D human body mesh, renders the 3D human body mesh onto the original image, obtains an image containing the projection of the 3D human body mesh, and completes the 3D modeling of the human body mesh.

[0035] Compared with the prior art, the present invention has at least the following beneficial effects:

[0036] This invention discloses a method for constructing a 3D human body mesh model by fusing millimeter wave signals and images. It is based on dual-branch signal processing of millimeter wave signals and images. The signals are meticulously cropped, filtered, and corrected. Initial phase compensation and static clutter elimination are performed on the obtained millimeter wave signals, and a 3D human body point cloud is obtained through three FFT transformations. The image signal is then blurred. The two signals are input into a deep neural network for feature fusion, key points and human contours are extracted, and a 3D human body mesh model is finally generated. This invention utilizes millimeter wave signals and image fusion for 3D human body mesh modeling. Due to the involvement of radio frequency signals, this invention can ensure normal operation under adverse weather or extreme lighting conditions. Furthermore, the presence of image signals ensures the accuracy and stability of the method.

[0037] Furthermore, the system setup of this invention uses a commercial millimeter-wave radar to transmit FMCW frequency-modulated continuous wave signals and a monocular RGB / IR camera to collect images. This combination overcomes the limitations of purely image-based methods in poor lighting conditions and also suppresses the instability of purely wireless signals in environments with multipath propagation.

[0038] Furthermore, addressing the phase deviation between different antennas caused by the hardware characteristics of millimeter-wave radar, this invention experimentally measures this error and compensates for it in subsequent millimeter-wave signal processing, achieving phase calibration of the signal. The calibrated 3D point cloud, compared to the uncalibrated 3D point cloud, more accurately reflects the actual target location.

[0039] Furthermore, targets at different distances from the radar reflect intermediate frequency signals of varying frequencies. By performing Range-FFT on the signal, the different peak values ​​represent targets at different distances. For targets at the same distance from the radar but with different velocities, the phase shift of their signals differs according to the Doppler frequency shift principle. After performing Doppler-FFT processing on the Range-FFT signal again, targets at the same radial distance but with different velocities are distinguished, resulting in a Doppler heatmap. Subtracting the average value of the Doppler-FFT heatmap from all signals achieves static clutter cancellation. The top 128 points (experimental values) with the strongest signal intensity in the heatmap are related to human movement, and these points are selected as effective information points for subsequent processing. The millimeter-wave radar's antenna array is specially designed according to its signal wavelength. Targets at different azimuths exhibit a fixed phase shift pattern in the signals acquired by different receiving antennas. Performing 3D-FFT processing on the Doppler-FFT signals from different receiving antennas yields the angle data of the detected targets. Through the above processing, a 3D point cloud related to human movement is obtained.

[0040] Furthermore, this invention employs data-driven deep learning network technology and divides the model into three parts:

[0041] 1) Feature Extraction Module: This module is used to process millimeter-wave signals and image signals, extract their data features, and fuse the features of millimeter-wave signals and image signals to achieve feature complementarity;

[0042] 2) Joint point recognition and contour estimation module: This module takes the features fused by the feature extraction module as input and outputs a heat map of human joint points and a black and white human contour map. By providing auxiliary supervision to these two outputs, it helps the subsequent model to infer various parameters.

[0043] 3) Mesh Generation Module: This module uses SMPL (Multi-person Skinned Linear Model) as the output module for the human body 3D mesh model. SMPL has advantages such as fewer parameters and avoidance of distortion. It uses only 72 joint rotation parameters and 10 shape parameters to construct the human body 3D model. The former is obtained by training PoseNet with the fused features of the 2D joints from the joint recognition and contour estimation module and the feature extraction module, while the latter is obtained by training ShapeNet with the fused features of the contour estimation map from the joint recognition and contour estimation module and the feature extraction module.

[0044] Furthermore, a novel local attention method was used in the feature extraction module to process the millimeter-wave point cloud signal. The feature transformation matrix was used to extract the Query, Key, and Value vectors of the point cloud, and the local similarity of the point cloud was calculated by combining mathematical transformations such as dot product with position embedding, and then local features were further extracted. These local features are beneficial for obtaining finer-grained information from sparse point clouds and for fusing it with global features.

[0045] Furthermore, in the keypoint recognition and contour estimation modules, keypoints are represented using 2D heatmaps, thus avoiding direct prediction of keypoint coordinates by the model, reducing the introduction of nonlinear results, and facilitating gradient descent. L2 loss is used in heatmap supervision, which allows for rapid gradient descent when the loss is high and further fine-tuning when the loss is low. The contour estimation part transforms the model prediction into a pixel binary classification problem, using cross-entropy loss, which helps improve the model's classification accuracy.

[0046] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0047] In summary, this invention further processes millimeter-wave signals, fuses image information, and further constructs a novel and effective multimodal fusion network model, achieving high-precision restoration of the human body's three-dimensional mesh. This overcomes the limitations of pure image or pure wireless solutions and expands the use of radio frequency signals and the methods of feature fusion.

[0048] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0049] Figure 1 Real-life photos showing the setup of the equipment and experimental production site;

[0050] Figure 2 Experimental proof of phase shift and proof of the necessity of phase compensation, wherein (a) is the inherent phase shift between radar antennas, (b) is the antenna phase shift after two power-ups of the radar, (c) is the 3D point cloud before signal calibration, and (d) is the 3D point cloud after signal calibration.

[0051] Figure 3 This is a schematic diagram of the millimeter-wave feature extraction module;

[0052] Figure 4 This is a schematic diagram of the feature fusion module;

[0053] Figure 5 An overview of the neural network design;

[0054] Figure 6To illustrate the system's error under experimental data and to compare it with other models;

[0055] Figure 7 This refers to the experimental error in the ablation experiment;

[0056] Figure 8 This section describes the principles of millimeter-wave radar and virtual radar arrays, where (a) represents the transmit and receive signals and modulated intermediate frequency signals of multi-target FMCW, and (b) represents the virtual antenna array of millimeter-wave radar.

[0057] Figure 9 The image shows the experimental results under poor lighting conditions.

[0058] Figure 10 The image shows the experimental results under strong light conditions.

[0059] Figure 11 The experimental results are shown in the blurred image.

[0060] Figure 12 A qualitative analysis chart of experimental results under different experimental sites. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0063] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0064] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" relationship.

[0065] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0066] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0067] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0068] This invention provides a method for constructing a 3D human body mesh model by fusing millimeter-wave and image data. It utilizes commercial millimeter-wave radars AWR1443BOOST and DCA1000EVM as millimeter-wave transceivers, and an RMONCAM G600 as a receiver for RGB and infrared images (this camera automatically switches to infrared images at night while acquiring RGB images during the day). Phase shift compensation and clutter cancellation are applied to the millimeter-wave signals. The images are then blurred and input into a neural network. The two signals are fused, features are extracted, the positions of key points on the human body are extracted, and the human body contour is estimated. Finally, a 3D human body mesh model is regressed. This method exhibits high robustness.

[0069] This invention discloses a method for constructing a 3D human body mesh model through millimeter-wave and image fusion, comprising the following steps:

[0070] S1. Use a commercial millimeter-wave radar to transmit an FMCW frequency-modulated continuous wave millimeter-wave signal, collect the raw signal and convert it into an intermediate frequency signal; use a monocular camera to collect RGB images (collect infrared images at night).

[0071] Please see Figure 1 The millimeter-wave radar and camera are located at a distance from the ground. Human activity space is The edge of the site is a distance from the millimeter-wave radar. .

[0072] S2. Perform static clutter cancellation and phase calibration on the obtained intermediate frequency signal;

[0073] Please see Figure 2 , Figure 2 In the diagram, (a) represents the phase shift of the millimeter-wave signal acquired by different receiving antennas. Figure 2 (b) indicates that the phase bias is relatively stable when the device is restarted and powered off. Figure 2 (c) and (d) in the figure represent a person 4 meters away from the millimeter-wave radar. The comparison of the reflected signal before and after calibration shows that the point cloud after calibration matches the true value, which proves the necessity of phase calibration.

[0074] Let the phase shift introduced by the m-th virtual antenna be denoted as... The resulting phase shift vector is expressed as:

[0075] A=[1 ... ]

[0076] The phase-shift calibrated signal for:

[0077] =AS

[0078] Where S is the original signal.

[0079] S3. Perform three Fast Fourier Transforms (Range-FFT, Doppler-FFT, and 3D-FFT) on the signal processed in step S2 to obtain the human body 3D point cloud corresponding to the intermediate frequency signal; perform MaskIMG operations (grayscale conversion, color inversion, etc.) on the image to perform data augmentation.

[0080] right Range-FFT and Doppler-FFT are performed sequentially. The average value of the Doppler-FFT heatmap is subtracted from all signals to eliminate static noise from the environment. The 128 points with the highest intensity are selected and 3D-FFT is performed to obtain the 3D point cloud.

[0081] The MaskIMG algorithm is used to process RGB images as follows:

[0082] enter: Original RGB image

[0083] Output: Post-processed images

[0084] =RandomGrayscale( , )

[0085] dice = Random()

[0086] if dice<

[0087] =GaussianBlur()

[0088] else if dice <

[0089] =ColorJitter()

[0090] else if dice <

[0091] =ColorInvert()

[0092] Among them, GaussianBlur() represents Gaussian blurring of the input image (Gaussian kernel size is 7*7); ColorJitter() represents changing the brightness, contrast and saturation of the image; ColorInvert() is the color inversion operation. , , , Set them to 0.8, 0.4, 0.5, and 0.6 respectively.

[0093] S4. Input the 3D point cloud and image information obtained in step S3 into the designed deep neural network to generate a human body 3D mesh, and render it onto the original image to obtain an image containing the projection of the human body 3D mesh.

[0094] A deep neural network consists of three modules: a feature extraction module, a key point recognition and contour estimation module, and a mesh generation module.

[0095] (1) Feature extraction module:

[0096] First, the millimeter-wave signal and the image signal are processed separately. Then, the features of the two are fused and input into the next stage of the network. Specifically, a local feature extraction method—local attention—is used for the millimeter-wave signal. This method is based on a multi-head attention mechanism. For a millimeter-wave signal X with feature dimension D, let...

[0097] [Q,K,V]=X

[0098] Attention weight is calculated by dividing the query by all keys. We obtain that self-supervision is represented as:

[0099]

[0100] Expand k SA operations (or heads) in parallel and connect them to the corresponding output:

[0101]

[0102] Location embedding was used for different heads, associating each head with certain parts of the human body, and regional features were obtained through training.

[0103] (2) Joint recognition and contour estimation module

[0104] The features fused in the previous stage are used as input to output a heatmap of human joints and a black-and-white outline of the human figure. The two tasks share weights and are trained using L2 loss and cross-entropy loss, respectively.

[0105] (3) Mesh generation module:

[0106] The 2D joints and fused features from the previous stage are input into PoseNet, which outputs 72 parameters representing joint rotation. Then, the contour estimation map and features are input into ShapeNet, resulting in 10 parameters representing the shape. Finally, these parameters are input into SMPL to generate a 3D model of the human body mesh.

[0107] In another embodiment of the present invention, a human body 3D mesh model construction system for millimeter wave and image fusion is provided. The system can be used to implement the above-mentioned human body 3D mesh model construction method for millimeter wave and image fusion. Specifically, the human body 3D mesh model construction system for millimeter wave and image fusion includes an acquisition module, a preprocessing module, a training module and an output module.

[0108] The acquisition module converts the reflected signal of the FMCW frequency-modulated continuous wave signal into an intermediate frequency signal and collects RGB images.

[0109] The preprocessing module performs static clutter cancellation and phase calibration on the intermediate frequency signal obtained by the acquisition module;

[0110] The training module performs three fast Fourier transforms on the signal processed by the preprocessing module to obtain the human body 3D point cloud corresponding to the intermediate frequency signal, and performs MaskIMG operation on the RGB image obtained by the acquisition module to obtain image information.

[0111] The output module inputs the 3D point cloud and image information obtained from the training module into the deep neural network to generate a 3D human body mesh, renders the 3D human body mesh onto the original image, obtains an image containing the projection of the 3D human body mesh, and completes the 3D modeling of the human body mesh.

[0112] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used in the operation of a method for constructing a 3D human body mesh model by fusion of millimeter waves and images, including:

[0113] The reflected signal of the FMCW frequency-modulated continuous wave signal is converted into an intermediate frequency signal, and RGB images are collected. Static clutter cancellation and phase calibration are performed on the intermediate frequency signal. The processed signal is subjected to three fast Fourier transforms to obtain the human body 3D point cloud corresponding to the intermediate frequency signal. The RGB image is then processed by MaskIMG to obtain image information. The 3D point cloud and image information are input into a deep neural network to generate a human body 3D mesh. The human body 3D mesh is rendered onto the original image to obtain an image containing the projection of the human body 3D mesh, thus completing the 3D modeling of the human body mesh.

[0114] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0115] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the method for constructing a 3D human body mesh model by fusion of millimeter waves and images in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps:

[0116] The reflected signal of the FMCW frequency-modulated continuous wave signal is converted into an intermediate frequency signal, and RGB images are collected. Static clutter cancellation and phase calibration are performed on the intermediate frequency signal. The processed signal is subjected to three fast Fourier transforms to obtain the human body 3D point cloud corresponding to the intermediate frequency signal. The RGB image is then processed by MaskIMG to obtain image information. The 3D point cloud and image information are input into a deep neural network to generate a human body 3D mesh. The human body 3D mesh is rendered onto the original image to obtain an image containing the projection of the human body 3D mesh, thus completing the 3D modeling of the human body mesh.

[0117] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0118] Please see Figure 3 , Figure 3For millimeter-wave feature extraction modules, a multi-head self-attention mechanism can be used to achieve high-quality local feature extraction, thus preparing for feature fusion.

[0119] Please see Figure 4 , Figure 4 The feature fusion module utilizes an attention mechanism to perform feature fusion and ensures that the fusion block runs through the entire feature extraction module. Through a series of dimensional transformations, high-quality feature fusion is achieved.

[0120] Please see Figure 5 , Figure 5 This is an overview of the designed deep neural network. After feature extraction, the deep neural network performs joint estimation and contour inference, and then uses PoseNet and ShapeNet to regress the parameters of the human body. The parameters are then input into SMPL (Multi-person Skinned Linear Model) to generate a complete three-dimensional human body mesh model.

[0121] This invention uses MPVE, MPJPE, and PA-MPJPE metrics for performance evaluation. MPVE is the average Euclidean distance between the predicted human mesh and the ground truth; MPJPE is the average Euclidean distance between the predicted 3D joints and the ground truth; PA-MPJPE is the MPJPE calculated after the predicted 3D pose is aligned with the ground truth through translation, rotation, and other operations.

[0122] Please see Figure 6 , Figure 6 The results are shown for 2300 samples, in cm. DecoMR and SPIN are two image-based methods. It can be seen that the method of this invention improves robustness while maintaining high performance.

[0123] Please see Figure 7 , Figure 7 The ablation experiment was conducted to test the performance of the present invention under the condition of missing some modules, further demonstrating the function of each module designed in the present invention.

[0124] This invention first uses the AWR1443BOOST commercial millimeter-wave radar and DCA1000EVM as millimeter-wave signal acquisition tools, and sets the millimeter-wave frequency between 77 and 81 GHz.

[0125] Please see Figure 1 and Figure 8 , Figure 8 This describes the echo principle of millimeter-wave signals and the principle of virtual antenna arrays.

[0126] The robustness of this invention was tested under different interferences.

[0127] Please see Figure 9 , Figure 10 and Figure 11 , Figure 9 The experiment was conducted under poor lighting conditions, demonstrating that the present invention can effectively construct a 3D human body model even in such conditions. Figure 10 The experiment was conducted under strong light, demonstrating that even with strong light interference, this invention can effectively capture and model human movements. Figure 11 The experiment was conducted when the image information was relatively blurry. It can be seen that under blurry conditions, the human body 3D model constructed by the present invention has basically the same movements as the real movements, which further proves that the present invention has good robustness.

[0128] The invention was also tested in different environments.

[0129] Please see Figure 12 , Figure 12 The qualitative experiments were conducted in various settings, including laboratories, corners, studios, corridors, and laboratories with chairs. This demonstrates that the invention can perceive human movements and perform 3D mesh modeling in different experimental environments, exhibiting strong adaptability and usability.

[0130] In summary, the present invention provides a method and system for constructing a 3D human body mesh model by fusing millimeter waves and images. It has no requirements for lighting conditions and exhibits good robustness to extreme weather and other conditions that may cause image information to fail. It overcomes the limitations of pure image solutions and expands the use of radio frequency signals and feature fusion methods, thereby achieving high-precision reconstruction of the human body's 3D mesh.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0133] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0134] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0135] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0136] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0137] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM). Computer-readable media may include only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content of the computer-readable media may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0138] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0139] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0141] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A method for constructing a 3D human body mesh model by fusing millimeter waves and images, characterized in that, Includes the following steps: S1. Acquire the reflected signal of the FMCW frequency-modulated continuous wave signal and convert it into an intermediate frequency signal, and collect RGB images; S2. Perform static clutter cancellation and phase calibration on the intermediate frequency signal obtained in step S1 to obtain a phase-calibrated signal. ; S3. The signal processed in step S2 Perform three fast Fourier transforms to obtain the human body 3D point cloud corresponding to the intermediate frequency signal. Perform MaskIMG algorithm operation on the RGB image obtained in step S1, including grayscale conversion and color inversion, to obtain image information. S4. Input the 3D point cloud and image information obtained in step S3 into a deep neural network to generate a 3D human body mesh. Render the 3D human body mesh onto the original image to obtain an image containing the projection of the 3D human body mesh, thus completing the 3D modeling of the human body mesh. The deep neural network includes: The feature extraction module processes millimeter-wave signals and image signals separately, and fuses the features of the millimeter-wave signals and image signals; The joint point recognition and contour estimation module takes the features fused by the feature extraction module as input and outputs a heat map of human joint points and a black and white human contour map. The mesh generation module inputs the 2D joints from the joint recognition and contour estimation module and the fused features from the feature extraction module into PoseNet, outputting 72 parameters representing joint rotation. Then, it inputs the contour estimation map from the joint recognition and contour estimation module and the fused features from the feature extraction module into ShapeNet to obtain 10 parameters representing the shape. The shape parameters are then input into a multi-person skinned linear model to generate a 3D model of the human body mesh. The local attention method is used to process millimeter-wave signals. For a millimeter-wave signal X with feature dimension D, let [Q,K,V]=X Q, K, and V are the Query vector, Key vector, and Value vector, respectively. The feature transformation matrix is ​​used, and the attention weights are calculated by dividing the inner product of the Query and Key by . The k SA operations are then expanded in parallel and connected to their corresponding outputs. The study employed location embedding for different heads, associating each head with certain parts of the human body, and obtaining regional features through training.

2. The method for constructing a 3D human body mesh model by fusing millimeter waves and images according to claim 1, characterized in that, In step S1, an FMCW frequency-modulated continuous wave signal is transmitted using a commercial millimeter-wave radar, and images are collected using a monocular RGB / IR camera.

3. The method for constructing a 3D human body mesh model by fusing millimeter waves and images according to claim 1, characterized in that, In step S2, the phase-calibrated signal for: =AS Where S is the original signal and A is the phase shift vector.

4. The method for constructing a 3D human body mesh model by fusing millimeter waves and images according to claim 3, characterized in that, The phase shift vector A is: A=[1 ... ] in, The phase shift introduced for the m-th virtual antenna, The base is the natural number. It is the imaginary unit.

5. The method for constructing a 3D human body mesh model by fusing millimeter waves and images according to claim 1, characterized in that, In step S3, the phase-calibrated signal... Range-FFT and Doppler-FFT are performed sequentially. The average value of the Doppler-FFT heatmap is subtracted from all signals. The 128 points with the highest intensity are selected and 3D-FFT is performed to obtain the 3D point cloud.

6. The method for constructing a 3D human body mesh model by fusing millimeter waves and images according to claim 1, characterized in that, Output : Extend the k SA operations in parallel and connect them to the corresponding outputs: in, This is a weight matrix, where Q, K, and V are the Query vector, Key vector, and Value vector, respectively. This represents the characteristic dimension of millimeter-wave signals.

7. The method for constructing a 3D human body mesh model by fusing millimeter waves and images according to claim 1, characterized in that, The key point recognition and contour estimation module is trained using L2 loss and cross-entropy loss.

8. A system for constructing a 3D human body mesh model by fusing millimeter waves and images, characterized in that, include: The acquisition module acquires the reflected signal of the FMCW frequency-modulated continuous wave signal and converts it into an intermediate frequency signal, and collects RGB images; The preprocessing module performs static clutter cancellation and phase calibration on the intermediate frequency signal obtained from the acquisition module to obtain a phase-calibrated signal. ; The training module processes the signals already processed by the preprocessing module. Three fast Fourier transforms are performed to obtain the human body 3D point cloud corresponding to the intermediate frequency signal. The RGB image obtained by the acquisition module is processed by the MaskIMG algorithm, including grayscale conversion and color inversion, to obtain image information. The output module takes the 3D point cloud and image information obtained from the training module and inputs it into the deep neural network to generate a 3D human body mesh. It then renders the 3D human body mesh onto the original image to obtain an image containing the projection of the 3D human body mesh, thus completing the 3D modeling of the human body mesh. The deep neural network includes: The feature extraction module processes millimeter-wave signals and image signals separately, and fuses the features of the millimeter-wave signals and image signals; The joint point recognition and contour estimation module takes the features fused by the feature extraction module as input and outputs a heat map of human joint points and a black and white human contour map. The mesh generation module inputs the 2D joints from the joint recognition and contour estimation module and the fused features from the feature extraction module into PoseNet, outputting 72 parameters representing joint rotation. Then, it inputs the contour estimation map from the joint recognition and contour estimation module and the fused features from the feature extraction module into ShapeNet to obtain 10 parameters representing the shape. The shape parameters are then input into a multi-person skinned linear model to generate a 3D model of the human body mesh. The local attention method is used to process millimeter-wave signals. For a millimeter-wave signal X with feature dimension D, let [Q,K,V]=X Q, K, and V are the Query vector, Key vector, and Value vector, respectively. The feature transformation matrix is ​​used, and the attention weights are calculated by dividing the inner product of the Query and Key by . The k SA operations are then expanded in parallel and connected to their corresponding outputs. The study employed location embedding for different heads, associating each head with certain parts of the human body, and obtaining regional features through training.

Images