Three-dimensional object reconstruction method based on millimeter wave radar and capable of realizing micro radio frequency modeling and super-resolution estimation

Abstract

The invention discloses an object three-dimensional reconstruction method capable of realizing micro radio frequency modeling and super-resolution estimation based on a millimeter wave radar, and the method comprises the following steps: S1, collecting a multi-view low-resolution intermediate frequency signal of a target object, and carrying out the static environment interference removal of the signal; s2, inputting the low-resolution intermediate-frequency signal into a super-resolution signal estimation network to generate a high-resolution intermediate-frequency signal; s3, modeling is carried out on a target object, and parameterization is carried out on space geometric parameters and radio frequency response parameters of a radio frequency Gaussian unit; s4, simulating attenuation behaviors of millimeter waves in transmitting, reflecting and receiving processes, and modeling attenuation propagation of a radio frequency Gaussian unit; s5, the reconstruction error serves as a loss function, and iterative optimization is conducted on the geometric parameters and the radio frequency response parameters of the radio frequency Gaussian unit; and S6, three-dimensional structure reconstruction of the object is realized, and calculation overhead caused by excessive invalid units is avoided. According to the invention, high-precision three-dimensional reconstruction of an object can be realized in low-illumination, shielding and other environments.

Description

Technical Field

[0001] The present invention relates to the technical field of radio frequency three-dimensional reconstruction, and in particular to a method for three-dimensional reconstruction of objects based on millimeter wave radar differentiable radio frequency modeling and super-resolution estimation. Background Art

[0002] With the rapid development of 3D reconstruction technology, especially in fields such as virtual reality, intelligent manufacturing, map construction, and robot navigation, 3D reconstruction has become an important component of modern technological applications. However, the current mainstream image-based 3D reconstruction methods perform well under good visual conditions, but still face many challenges in occlusion, low light, and complex environments. In particular, in scenarios such as industrial inspection and security inspection, objects are often packaged or occluded, which makes traditional visual reconstruction methods unable to effectively restore the 3D structure of objects. In this context, 3D reconstruction methods based on radio frequency signals have gradually gained attention because they can penetrate the occlusion of objects in complex environments and have stronger robustness.

[0003] Millimeter-wave radar, as an active sensing device, offers excellent penetration and high-precision distance resolution, making it an effective technology for overcoming the limitations of traditional methods. It can capture echo information from objects even through obstructing materials, plastic films, and certain non-metallic materials, making it suitable for industrial automation and security inspections. Furthermore, among current solutions, X-ray-based detection equipment is expensive, while SAR-based radar systems rely on complex mobile platforms, which is also costly. Furthermore, while both excel at two-dimensional imaging, they generally lack the ability to capture three-dimensional spatial information about objects. Therefore, achieving low-cost, efficient, and accurate three-dimensional reconstruction using millimeter-wave radar remains a critical challenge that needs to be addressed. Summary of the Invention

[0004] To overcome the shortcomings of the aforementioned existing technologies, the present invention provides a method for 3D reconstruction of objects based on millimeter-wave radar differentiable RF modeling and super-resolution estimation. By combining differentiable RF modeling with super-resolution signal estimation techniques, this method overcomes the limitations of traditional 3D reconstruction methods in complex occluded environments. Through high-precision RF modeling and super-resolution signal recovery, this method achieves high-precision 3D reconstruction of objects in low-light and occluded environments, independent of visible light information, and exhibits enhanced robustness and adaptability.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] The method for 3D reconstruction of an object based on millimeter-wave radar differentiable radio frequency modeling and super-resolution estimation includes the following steps:

[0007] S1. Use a multi-antenna commercial millimeter-wave radar to collect multi-view, low-resolution intermediate frequency signals of the target object and remove static environmental interference;

[0008] S2: Input the interference-removed low-resolution IF signal into the super-resolution signal estimation network and use a multi-head self-attention generative adversarial network model based on compressed sensing to generate a high-resolution IF signal. This achieves migration from the multi-antenna low-resolution signal domain to the high-resolution signal domain and eliminates noise interference.

[0009] S3, modeling the target object using a radio frequency Gaussian unit constructed using a three-dimensional Gaussian function, and parameterizing the spatial geometric properties and radio frequency response properties of the radio frequency Gaussian unit;

[0010] S4. Build a differentiable RF attenuation and propagation model based on the RF Gaussian unit model to simulate the attenuation behavior of millimeter waves during transmission, reflection, and reception, and model the attenuation and propagation of the RF Gaussian unit. Finally, explicitly derive the model to construct a differentiable millimeter-wave signal simulation system.

[0011] S5. Initialize the millimeter-wave signal simulation system in S4 using the high-range-resolution millimeter-wave radar parameters, input the radar pose when collecting the intermediate-frequency signal in S1, and output a simulated high-resolution intermediate-frequency signal. Adopt a gradient descent optimization algorithm, using the reconstruction error between the high-resolution intermediate-frequency signal in S2 and the simulated high-resolution intermediate-frequency signal output by the system as the loss function, and iteratively optimize the geometric parameters and RF response parameters of the RF Gaussian unit to gradually approximate the three-dimensional structure of the real object.

[0012] S6. In the process of iterating Gaussian parameters using the gradient descent optimization algorithm, a dynamic expansion algorithm based on gradient guidance is designed to automatically add or delete RF Gaussian units to achieve more accurate three-dimensional structure reconstruction of the object and avoid the computational overhead caused by too many invalid units.

[0013] In S1:

[0014] By rotating the target object and using a multi-antenna commercial millimeter-wave radar to collect its multi-view low-resolution intermediate frequency signals, the target surface is evenly covered from all angles.

[0015] In order to eliminate static environmental interference, the background environmental signal S is first collected without the target object. env , and then collect the mixed signal S containing the target and the environment when the target object exists mix , using the formula Where α∈(0,1] is the weighting coefficient of the background signal, Indicates averaging the environmental signals collected multiple times to reduce the impact of random noise.

[0016] In S2, the processed low-resolution intermediate frequency signal is input into a super-resolution signal estimation network to achieve migration from the low-resolution signal domain to the high-resolution signal domain and eliminate noise interference;

[0017] The super-resolution signal estimation network adopts a generative adversarial network architecture and uses a random mask module to preprocess the signal input. The generator module integrates a convolution module, a multi-head self-attention module, and a decoder module.

[0018] Random mask module designs a mask module based on random binary vectors to perform pseudo-random sampling on the input low-resolution time domain signal;

[0019] The convolution module uses multiple layers of channel-by-channel one-dimensional convolution blocks and channel-fused one-dimensional convolution blocks to perform intra-channel feature fusion and inter-channel feature fusion before and after the multi-head self-attention module.

[0020] The multi-head self-attention module models the dependencies between antenna signals in the channel dimension, enabling the network to effectively capture the mutual information of multi-antenna signals and improve super-resolution reconstruction.

[0021] The decoder module consists of fully connected layers and multi-layer upsampling modules to generate the output high-resolution millimeter wave signal.

[0022] During training, the network uses a reconstruction loss based on mean square error to ensure that the generated high-resolution signal is numerically consistent with the target signal, thereby ensuring reconstruction accuracy.

[0023] In the S3:

[0024] The target object is modeled by using a three-dimensional Gaussian unit; the spatial geometric parameters of the RF Gaussian unit are expressed by a three-dimensional vector μ = [μ x ,μ y ,μ z ] T It represents the center position of the Gaussian unit in three-dimensional space, and its shape and direction are described by the covariance matrix Σ. To ensure that the covariance matrix is ​​always positive, the formula Σ=RSS is used. T R T To calculate the covariance matrix, R is the three-dimensional rotation matrix that controls the spatial rotation direction of the Gaussian unit, S is the scaling matrix that defines the scale of the Gaussian unit in the direction of each coordinate axis, and the geometric weight factor of the Gaussian unit is calculated as G(x) = exp(-1 / 2(x-μ) T Σ -1 (x-μ));

[0025] The RF Gaussian unit also has RF response characteristics, which defines the absorption rate η of the RF Gaussian unit. absTo characterize the target object's ability to absorb radio frequency energy, the expression of the transmitted wave after absorption is:

[0026] s trans =(1-G p n abs )×exp(j·G p φ abs )×s in

[0027] Among them, G P is the geometric weight factor of the RF Gaussian unit at point P, φ abs is the phase shift during absorption, s in The incident wave of this Gaussian unit;

[0028] Define the backscatter coefficient η back Describes the reflection ability of the target or medium surface to the incident electromagnetic wave. The expression of the backscattered wave of the RF Gaussian unit is:

[0029] s back =(G P n back )×exp(j·G p φ back )×s abs

[0030] Among them, φ back is the phase shift during absorption, s abs Expression for the absorption wave of this Gaussian unit.

[0031] In said S4:

[0032] The propagation of RF signals is divided into the transmission phase and the reception phase. The propagation process in the reception phase is mathematically equivalent to the transmission phase, that is, the propagation of the transmission signal is regarded as the reverse propagation of the reception signal.

[0033] Specifically, each RF Gaussian unit transmits and receives signals along the signal propagation path, which undergoes geometric attenuation, absorption, and accumulation of phase shifts. The signal received by the RF Gaussian unit is expressed as:

[0034]

[0035] in, is the intermediate frequency signal received in the direction of solid angle ω, A0 is the initial intensity of the transmitted signal, is the geometric weight factor of the kth Gaussian unit in the ω direction, s abs The expression of the absorption wave of the Gaussian unit;

[0036] The signal propagation structure in the receiving phase is the same as that in the transmitting phase, except that the propagation direction is reversed. That is, the signal is transmitted from each RF Gaussian unit back to the receiving antenna. The entire propagation process maintains the differentiability of each Gaussian unit geometry and RF response parameters.

[0037] Finally, the differentiable RF attenuation propagation model is explicitly derived to construct a differentiable millimeter-wave signal simulation system.

[0038] In said S5:

[0039] Initialize the millimeter-wave signal simulation system in S4 using high-range-resolution millimeter-wave radar parameters, input the radar pose when collecting the intermediate-frequency signal in S1, and output a simulated high-resolution intermediate-frequency signal. Utilize a gradient descent optimization algorithm, using the reconstruction error between the high-resolution intermediate-frequency signal in S2 and the simulated high-resolution intermediate-frequency signal output by the system as the loss function. Combined with the gradient backpropagation path explicitly derived in the simulation system, the gradient of the loss function is transferred to the parameters of the RF Gaussian units that constitute the scene. These parameters are then optimized in the direction of gradient descent, and the process is iterated a certain number of times.

[0040] In the design of the loss function, the mean square error (MSE) is used as the main reconstruction loss, aiming to ensure the consistency of the generated signal and the actual signal in the Range spectrogram and the Range-Angle spectrogram. The optimization goal is to minimize this difference, thereby iteratively optimizing the representation of the RF Gaussian three-dimensional object.

[0041] In said S6:

[0042] A gradient-guided dynamic expansion algorithm was designed. The algorithm analyzes whether the mean gradient of the current RF Gaussian unit after projection in the distance dimension exceeds a certain threshold. If so, it determines that "under-reconstruction" or "over-reconstruction" problems have occurred in the reconstruction process, and handles them separately in combination with the scaling parameters of the RF Gaussian unit.

[0043] When the scaling parameter of the RF Gaussian unit is less than a certain threshold, the Gaussian unit is considered to be "under-reconstructed". At this time, the algorithm will automatically clone and adjust the existing RF Gaussian unit, expand it along the gradient direction, and increase the coverage of object details;

[0044] When the scaling parameter of a RF Gaussian unit is greater than a certain threshold, the Gaussian unit is considered to be "over-reconstructed". At this time, the algorithm will split and scale the oversized RF Gaussian unit to increase the description of the object's details.

[0045] This process is iterated repeatedly through a gradient-guided optimization strategy to gradually optimize the number and position of RF Gaussian units, thereby improving the overall effect of 3D reconstruction of the object without increasing excessive computational burden.

[0046] A 3D object reconstruction system based on millimeter-wave radar differentiable RF modeling and super-resolution estimation, comprising a radar signal acquisition module, a signal processing module, a signal super-resolution estimation module, and a 3D object reconstruction module based on differentiable RF modeling;

[0047] Radar signal acquisition module: This module consists of a multi-antenna commercial millimeter-wave radar, positioned directly in front of the target. It collects target reflection signals when the millimeter-wave radar is located at different positions. Furthermore, by rotating the target object, the collected signals ensure uniform coverage of the target surface from all angles.

[0048] Signal processing module: collects background environmental signals without target objects, eliminates static environmental interference from the original signal through static background removal method, and outputs the target signal after removing the static environment;

[0049] Signal super-resolution estimation module: This module includes a random mask component, a generator component, a discriminator component, and a reconstruction loss function. The random mask component performs random masking on the target signal. The generator component integrates an inter-channel attention component, which uses a multi-scale convolution module and the inter-channel attention component to recover high-resolution radar signals from the target signal while simultaneously performing noise elimination. The discriminator component evaluates the high-resolution signal using the reconstruction loss function and optimizes the generator output in combination with the structure of a generative adversarial network.

[0050] The object three-dimensional reconstruction module based on differentiable RF modeling includes an initialization component for three-dimensional Gaussian space representation, a signal propagation component based on differentiable RF modeling, a dynamic Gaussian unit expansion algorithm component, and an iterative optimization component based on gradient descent; the initialization component for three-dimensional Gaussian space representation processes the input signal, extracts the original initial point cloud from the target signal, and initializes the RF Gaussian unit; the signal propagation component based on differentiable RF modeling randomly selects a radar posture in the data set and obtains the reflected signal corresponding to the position; the dynamic Gaussian unit expansion algorithm component dynamically increases or decreases the Gaussian unit according to the mean gradient after projection in the distance dimension; the iterative optimization component based on gradient descent optimizes the parameters of the Gaussian unit through gradient by comparing the generated signal with the signal in the data set.

[0051] A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by a computing device, cause the computing device to perform the above method.

[0052] A computing device comprising:

[0053] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the above method.

[0054] Beneficial effects of the present invention:

[0055] The present invention provides a method for three-dimensional reconstruction of objects based on millimeter-wave radar differentiable radio frequency modeling and super-resolution estimation. While achieving high-precision three-dimensional reconstruction of objects, it also addresses the limitations of traditional image reconstruction methods under complex occlusion, low light, and non-ideal viewing angle conditions. By combining the penetration capability of millimeter-wave radar with generative adversarial network (GAN) technology, this solution can effectively restore the structure of packaged or obscured objects, significantly improving the reconstruction accuracy and robustness in complex environments. This method not only has a lower system cost, but also has higher adaptability, addressing the limitations of traditional image and radio frequency methods in scenarios such as logistics sorting and security inspections.

[0056] This invention utilizes a multi-antenna commercial millimeter-wave radar to collect multi-view, low-resolution signals from target objects. Compared to traditional synthetic aperture radars, this data acquisition platform offers greater flexibility in placement and eliminates the need for a rigid sampling array, simplifying the deployment of acquisition equipment. Furthermore, by rotating the target object and incorporating a static background removal algorithm, the system's overall handling and deployment costs are effectively reduced, further enhancing its practicality.

[0057] By introducing a super-resolution signal estimation network, the present invention can reduce the influence of noise interference and multipath effect in the signal propagation process to a certain extent, thereby improving the signal quality and enabling the present invention to have the adaptability in more dynamic and complex environments.

[0058] This method uses radio-frequency Gaussian units constructed based on three-dimensional Gaussian functions to model the target object. Compared to traditional point cloud-based representations, this modeling approach exhibits inherent anisotropy, spatial continuity, and differentiability, making it easy to integrate into gradient descent-based optimization algorithms and covering continuous three-dimensional spatial regions, effectively improving the accuracy of the reconstruction results.

[0059] This paper constructs a differentiable RF attenuation propagation model based on the RF Gaussian unit model. Due to its differentiability, this model can be naturally embedded in gradient descent optimization processes, further improving the accuracy and quality of 3D reconstruction results.

[0060] Based on the differentiability feature, the present invention adopts a gradient descent optimization algorithm with reconstruction error as the loss function to jointly optimize and iteratively the geometric parameters and RF response parameters of the RF Gaussian unit, so that the initially constructed three-dimensional Gaussian distribution representation gradually approaches the real structure of the target object, thereby achieving high-quality three-dimensional reconstruction.

[0061] This paper designs and utilizes a gradient-driven dynamic expansion algorithm that automatically adds or removes RF Gaussian units during the optimization process. This not only enhances the 3D model's ability to capture target details, but also effectively reduces computational resource consumption and improves overall reconstruction efficiency by eliminating redundant or invalid units.

[0062] In summary, the present invention proposes a 3D reconstruction method for objects that comprehensively considers hardware cost, occlusion robustness, and reconstruction quality, which can be widely used in logistics, security inspection and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Schematic diagram of the static environment removal effect.

[0064] Figure 2 This is a diagram of the super-resolution signal estimation network model framework.

[0065] Figure 3 Schematic diagram of RF parameterization of RF Gaussian unit.

[0066] Figure 4 Schematic diagram of directional RF signal attenuation and propagation.

[0067] Figure 5 Schematic diagram of sampling in the distance dimension.

[0068] Figure 6 Schematic diagram and gradient relationship diagram of "under-reconstruction" and "over-reconstruction".

[0069] Figure 7 This is the experimental diagram of the data collection scene and collection platform.

[0070] Figure 8 This is the 3D reconstruction effect of the object. DETAILED DESCRIPTION

[0071] The present invention will be described in further detail below with reference to the accompanying drawings.

[0072] The present invention provides a method for 3D reconstruction of objects based on millimeter-wave radar differentiable RF modeling and super-resolution estimation. A commercial millimeter-wave radar is used to collect target reflection signals at different locations directly in front of the target object. The target object is rotated and the above collection process is repeated to construct a target object reflection signal dataset containing multi-view reflection information. A static environment removal algorithm is used to extract pure reflection signals containing only the target object. The constructed signal dataset is input into a super-resolution signal estimation network to obtain a noise-reduced high-resolution target reflection signal. The radar pose information corresponding to the dataset is combined with the high-resolution reflection signal, and the radar pose is input into a 3D reconstruction module based on differentiable RF modeling. The module then outputs a simulated reflection signal corresponding to the 3D scene representation at that pose. By comparing this simulated signal with the actual collected signal, the parameters of the 3D Gaussian scene representation are iteratively optimized to achieve 3D reconstruction of the target object. This method can be widely used in logistics sorting, security inspection and other fields.

[0073] A method for 3D reconstruction of an object based on millimeter-wave radar differentiable radio frequency modeling and super-resolution estimation includes the following steps:

[0074] S1. Use a multi-antenna commercial millimeter-wave radar to collect background environmental signals S in the absence of target objects. env , and then collect the mixed signal S containing the target and the environment when the target object exists mix Using the formula Where α∈(0,1] is the weighting coefficient of the background signal, Indicates averaging the environmental signals collected multiple times.

[0075] Rotate the target object multiple times and collect signals at multiple different positions of a target object;

[0076] See also Figure 1 , respectively, the Range-Angle spectrum comparison between before (left) and after (right) static environment removal;

[0077] S2. The preprocessed low-resolution IF signal is fed into a super-resolution signal estimation network, which uses a generative adversarial network architecture with a multi-head self-attention module integrated into the generator. This module models the dependencies between antenna signals in the channel dimension, enabling the network to effectively capture the mutual information of multi-antenna signals and improve super-resolution reconstruction. During training, the network uses a reconstruction loss based on mean squared error.

[0078] See also Figure 2The overall structure of the super-resolution signal estimation network includes a random mask component, a generator component, a discriminator component, and a reconstruction loss function. The random mask component performs pseudo-random sampling on the input low-resolution signal based on a random binary vector, and converts the signal into a frequency domain input generator component. The generator component integrates a shallow convolution module, a multi-head self-attention module, a channel fusion convolution module, and a decoder module to generate high-resolution millimeter-wave signals. The discriminator component scores the generated high-resolution millimeter-wave signal by fusing the convolution module. The reconstruction loss function is used to train the generator and discriminator components in combination with the architecture of the generative adversarial network;

[0079] S3. Use three-dimensional Gaussian units to model the target object. The spatial geometric parameters of the RF Gaussian unit are obtained by the three-dimensional vector μ = [μ x ,μ y ,μ z ] T It represents the center position of the Gaussian unit in three-dimensional space. Its shape and direction are described by the covariance matrix Σ, using the formula Σ=RSS T R T To calculate the covariance matrix, R is the three-dimensional rotation matrix that controls the spatial rotation direction of the Gaussian unit, S is the scaling matrix that defines the scale of the Gaussian unit in the direction of each coordinate axis, and the geometric weight factor of the Gaussian unit is calculated as G(x) = exp(-1 / 2(x-μ) T Σ -1 (x-μ)).

[0080] The RF Gaussian unit also has RF response characteristics, which defines the absorption rate η of the RF Gaussian unit. abs To characterize the target object's ability to absorb radio frequency energy, the expression of the transmitted wave after absorption is:

[0081] s trans =(1-G p n abs )×exp(j·G p φ abs )×s in

[0082] Among them, G P is the geometric weight factor of the RF Gaussian unit at point P, φ abs is the phase shift during absorption, s in The incident wave of this Gaussian cell.

[0083] Define the backscatter coefficient η back Describes the reflection ability of the target or medium surface to the incident electromagnetic wave. The expression of the backscattered wave of the RF Gaussian unit is:

[0084] sback =(G P n back )×exp(j·G p φ back )×s abs

[0085] Among them, φ back is the phase shift during absorption, s abs Expression for the absorption wave of this Gaussian unit.

[0086] See also Figure 3 When the incident radiation energy reaches a certain RF Gaussian unit, part of the energy is absorbed by the unit and reflected along the incident direction; the remaining unabsorbed energy continues to penetrate the RF Gaussian unit and is transmitted along the original propagation direction;

[0087] S4. The propagation of RF signals is considered to be divided into two stages: the transmission stage and the reception stage. The propagation process in the reception stage is mathematically equivalent to the transmission stage, that is, the propagation of the transmitted signal is considered to be the reverse propagation of the received signal. Specifically, each RF Gaussian unit transmits and receives signals on the signal propagation path, and both undergo geometric attenuation, absorption, and accumulation of phase shifts. The signal received by the RF Gaussian unit is expressed as:

[0088]

[0089] in, is the intermediate frequency signal received in the direction of solid angle ω, A0 is the initial intensity of the transmitted signal, is the geometric weight factor of the kth Gaussian unit in the ω direction, s abs Expression for the absorption wave of this Gaussian unit.

[0090] The signal propagation during the receiving phase is identical to the transmission phase, with the propagation direction reversed, meaning the signal is transmitted from each RF Gaussian element back to the receiving antenna. The entire propagation process maintains the differentiability of the geometric and RF response parameters of each Gaussian element.

[0091] See also Figure 4 ,The signal propagation process mainly includes two stages: ,the attenuation propagation from the transmitting antenna to a certain RF Gaussian unit, and the attenuation propagation from the RF Gaussian unit to the receiving antenna;

[0092] See also Figure 5 ,Each RF Gaussian unit is uniformly sampled in the distance dimension from the antenna to ensure the overall differentiability;

[0093] S5. Explicitly derive the differentiable RF attenuation propagation model established in step S4. Use the simulation parameters of a commercial millimeter-wave radar with high range resolution to initialize the differentiable RF attenuation propagation model. Use the gradient descent optimization algorithm combined with the above-mentioned explicitly derived gradient propagation path to transmit the gradient of the loss function to the parameters of the RF Gaussian unit that constitutes the scene, and optimize the parameter in the direction of gradient descent, and iterate the process to a certain number of times. The mean square error (MSE) is used as the main reconstruction loss to ensure the consistency of the results of the generated signal and the actual signal in the Range spectrum and the Range-Angle spectrum. The optimization goal is to minimize this difference, thereby iteratively optimizing the representation of the RF Gaussian three-dimensional object;

[0094] S6. Analyze the mean gradient of the current RF Gaussian unit after projection in the distance dimension to determine whether "under-reconstruction" or "over-reconstruction" occurs during the reconstruction process. If under-reconstruction is detected, the algorithm automatically clones and adjusts the existing RF Gaussian unit, expanding it along the gradient direction; in the case of over-reconstruction, the algorithm splits and scales the oversized RF Gaussian unit.

[0095] See also Figure 6 During the reconstruction process, "under-reconstruction" or "over-reconstruction" may occur. Such anomalies usually manifest as abnormal gradient distributions in the distance dimension. In contrast, normal reconstruction results correspond to smooth gradient changes that conform to physical laws.

[0096] In one embodiment, the present invention provides a 3D object reconstruction system based on millimeter-wave radar differentiable RF modeling and super-resolution estimation. The system can be used to implement the above-mentioned 3D object reconstruction method based on millimeter-wave radar differentiable RF modeling and super-resolution estimation. Specifically, the 3D object reconstruction system based on millimeter-wave radar differentiable RF modeling and super-resolution estimation includes a radar signal acquisition module, a signal processing module, a signal super-resolution estimation module, and a 3D object reconstruction module based on differentiable RF modeling:

[0097] The radar signal acquisition module consists of a multi-antenna commercial millimeter-wave radar, which is deployed directly in front of the target. It collects target reflection signals when the millimeter-wave radar is located at different positions. In addition, by rotating the target object, the collected signals ensure that the target surface is evenly covered from all angles.

[0098] Signal processing module: collects background environmental signals without target objects, eliminates static environmental interference from the original signal through static background removal method, and outputs the target signal after removing the static environment;

[0099] Signal super-resolution estimation module: includes a random mask component, a generator component, a discriminator component, and a reconstruction loss function. The random mask component performs random masking on the target signal; the generator component integrates an inter-channel attention component, which uses a multi-scale convolution module and the inter-channel attention component to recover high-resolution radar signals from the target signal while simultaneously performing noise elimination; the discriminator component evaluates the high-resolution signal using the reconstruction loss function and optimizes the generator output in combination with the structure of a generative adversarial network.

[0100] The object 3D reconstruction module based on differentiable RF modeling includes an initialization component for 3D Gaussian space representation, a signal propagation component based on differentiable RF modeling, a dynamic Gaussian unit expansion algorithm component, and an iterative optimization component based on gradient descent. The initialization component for 3D Gaussian space representation processes the input signal, extracts the original initial point cloud from the target signal, and initializes the RF Gaussian unit; the signal propagation component based on differentiable RF modeling randomly selects a radar pose in the data set and obtains the reflected signal corresponding to the position; the dynamic Gaussian unit expansion algorithm component dynamically increases or decreases the Gaussian unit based on the mean gradient after projection in the distance dimension; the iterative optimization component based on gradient descent optimizes the parameters of the Gaussian unit by comparing the generated signal with the signal in the data set.

[0101] In one embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the object three-dimensional reconstruction method based on millimeter wave radar differentiable radio frequency modeling and super-resolution estimation, including:

[0102] A commercial millimeter-wave radar is used to collect target reflection signals at different locations directly in front of the target object. The target object is rotated and the above acquisition process is repeated to construct a target reflection signal dataset containing multi-view reflection information. A static environment removal algorithm is used to extract pure reflection signals containing only the target object. This constructed signal dataset is input into a super-resolution signal estimation network to obtain a denoised high-resolution target reflection signal. The radar pose information corresponding to the dataset is combined with the high-resolution reflection signal. The radar pose is then input into a 3D reconstruction module based on differentiable RF modeling, which outputs a simulated reflection signal corresponding to the 3D scene representation at that pose. By comparing this simulated signal with the actual collected signal, the parameters of the 3D Gaussian scene representation are iteratively optimized to ultimately achieve 3D reconstruction of the target object.

[0103] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (Non-Volatile Memory), such as at least one disk memory.

[0104] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the method for three-dimensional point cloud reconstruction based on signal multi-view fusion and domain migration in the above embodiment; the processor may load and execute the following steps:

[0105] A commercial millimeter-wave radar is used to collect target reflection signals at different locations directly in front of the target object. The target object is rotated and the above acquisition process is repeated to construct a target reflection signal dataset containing multi-view reflection information. A static environment removal algorithm is used to extract pure reflection signals containing only the target object. This constructed signal dataset is input into a super-resolution signal estimation network to obtain a denoised high-resolution target reflection signal. The radar pose information corresponding to the dataset is combined with the high-resolution reflection signal. The radar pose is then input into a 3D reconstruction module based on differentiable RF modeling, which outputs a simulated reflection signal corresponding to the 3D scene representation at that pose. By comparing this simulated signal with the actual collected signal, the parameters of the 3D Gaussian scene representation are iteratively optimized to ultimately achieve 3D reconstruction of the target object.

[0106] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0107] The present invention's method for 3D reconstruction of objects based on millimeter-wave radar differentiable radio frequency modeling and super-resolution estimation was tested in the laboratory:

[0108] See also Figure 7 , the figure shows the data acquisition scenario used in the present invention. Among them, the radar is installed on a moving platform, and the reflection signal of the target object is continuously collected during the movement of the platform. Unlike the synthetic aperture radar (SAR), the data acquisition platform of the present invention does not need to follow a strict motion trajectory, but only needs to synchronously record the real-time position information of the radar. In addition, in order to obtain multi-view information, the target object is rotated at an angle of 45° each time, and the acquisition process is repeated;

[0109] See also Figure 8 The figure focuses on the 3D reconstruction of the target object—a pistol. The figure includes the 3D point cloud reconstructed using the method of the present invention, the object's actual 3D point cloud form, and its corresponding rendering, used for comparison and verification of the reconstruction accuracy and effect.

[0110] Experimental results show that when using only simulated signals, the chamfer distance (CD) error of the 3D reconstructed point cloud is 0.5 cm when the target object is not occluded; when the target object is partially occluded, the CD error is 1.4 cm. These results demonstrate that the present invention can achieve highly robust 3D point cloud reconstruction even in the presence of occlusion.

[0111] Furthermore, using actual radar signals and combining them with a super-resolution signal estimation network, the CD error of the 3D reconstructed point cloud was 0.65cm when the target object was not obscured, and 1.8cm when the target object was obscured. These results demonstrate that the proposed super-resolution signal estimation network has excellent signal enhancement and reconstruction performance, effectively improving reconstruction quality in complex environments.

[0112] In summary, the present invention provides a method, system, medium, and device for three-dimensional reconstruction of objects based on millimeter-wave radar differentiable radio frequency modeling and super-resolution estimation. Using commercial millimeter-wave radar, high-quality three-dimensional reconstruction of objects is achieved. The method proposed in this invention still has good performance when the object is obscured, achieving high robustness in occlusion scenarios.

[0113] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0114] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0117] The above content is only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A 3D object reconstruction method based on millimeter-wave radar differentiable radio frequency modeling and super-resolution estimation, characterized in that: The following steps are involved: S1. Use a multi-antenna commercial millimeter-wave radar to collect multi-view, low-resolution intermediate frequency signals of the target object and remove static environmental interference; S2. Input the interference-removed low-resolution IF signal into the super-resolution signal estimation network and use a multi-head self-attention generative adversarial network model based on compressed sensing to generate a high-resolution IF signal, achieving migration from the multi-antenna low-resolution signal domain to the high-resolution signal domain and eliminating noise interference. S3, modeling the target object using a radio frequency Gaussian unit constructed using a three-dimensional Gaussian function, and parameterizing the spatial geometric properties and radio frequency response properties of the radio frequency Gaussian unit; S4. Build a differentiable RF attenuation and propagation model based on the RF Gaussian unit model to simulate the attenuation behavior of millimeter waves during transmission, reflection, and reception, and model the attenuation and propagation of the RF Gaussian unit. Finally, explicitly derive the model to construct a differentiable millimeter-wave signal simulation system. S5. Initialize the millimeter-wave signal simulation system in S4 using the high-range-resolution millimeter-wave radar parameters, input the radar pose when collecting the intermediate-frequency signal in S1, and output a simulated high-resolution intermediate-frequency signal. Adopt a gradient descent optimization algorithm, using the reconstruction error between the high-resolution intermediate-frequency signal in S2 and the simulated high-resolution intermediate-frequency signal output by the system as the loss function, and iteratively optimize the geometric parameters and RF response parameters of the RF Gaussian unit to gradually approximate the three-dimensional structure of the real object. S6. In the process of iterating Gaussian parameters using the gradient descent optimization algorithm, a gradient-guided dynamic expansion algorithm is designed to automatically add or delete RF Gaussian units to achieve three-dimensional structure reconstruction of the object and avoid the computational overhead caused by too many invalid units.

2. The method for 3D reconstruction of objects based on millimeter-wave radar differentiable radio frequency modeling and super-resolution estimation according to claim 1, characterized in that: In S1: By rotating the target object and using a multi-antenna commercial millimeter-wave radar to collect its multi-view low-resolution intermediate frequency signals, the target surface is evenly covered from all angles. First, the background environment signal S is collected without the target object. env , and then collect the mixed signal S containing the target and the environment when the target object exists mix , using the formula Where α∈(0,1] is the weighting coefficient of the background signal, Indicates averaging the environmental signals collected multiple times to reduce the impact of random noise.

3. The method for 3D reconstruction of objects based on millimeter-wave radar differentiable radio frequency modeling and super-resolution estimation according to claim 1, characterized in that: In S2, the processed low-resolution intermediate frequency signal is input into a super-resolution signal estimation network to achieve migration from the low-resolution signal domain to the high-resolution signal domain and eliminate noise interference; The super-resolution signal estimation network adopts a generative adversarial network architecture and uses a random mask module to preprocess the signal input. The generator module integrates a convolution module, a multi-head self-attention module, and a decoder module. Random mask module designs a mask module based on random binary vectors to perform pseudo-random sampling on the input low-resolution time domain signal; The convolution module uses multiple layers of channel-by-channel one-dimensional convolution blocks and channel-fused one-dimensional convolution blocks to perform intra-channel feature fusion and inter-channel feature fusion before and after the multi-head self-attention module. The multi-head self-attention module models the dependencies between antenna signals in the channel dimension, enabling the network to effectively capture the mutual information of multi-antenna signals and improve super-resolution reconstruction. The decoder module consists of fully connected layers and multi-layer upsampling modules to generate the output high-resolution millimeter wave signal.

4. The method for 3D reconstruction of objects based on millimeter-wave radar differentiable radio frequency modeling and super-resolution estimation according to claim 1, characterized in that: In the S3: The target object is modeled using a three-dimensional Gaussian unit; the spatial geometric parameters of the RF Gaussian unit are expressed by a three-dimensional vector μ = [μ x ,μ y ,μ z ] T It represents the center position of the Gaussian unit in three-dimensional space, and its shape and direction are described by the covariance matrix Σ, using the formula Σ=RSS T R T To calculate the covariance matrix, ensure that the covariance matrix is ​​always positive, where R is the three-dimensional rotation matrix that controls the spatial rotation direction of the Gaussian unit, S is the scaling matrix that defines the scale of the Gaussian unit in the direction of each coordinate axis, and the geometric weight factor of the Gaussian unit is calculated as G(x) = exp(-12(x-μ) T Σ -1 (x-μ)); Define the absorption rate η of the RF Gaussian unit abs To characterize the target object's ability to absorb radio frequency energy, the expression of the transmitted wave after absorption is: s trans =(1-G p n abs )×exp(j·G p φ abs )×s in Among them, G P is the geometric weight factor of the RF Gaussian unit at point P, φ abs is the phase shift during absorption, s in The incident wave of this Gaussian unit; Define the backscatter coefficient η back Describes the reflection ability of the target or medium surface to the incident electromagnetic wave. The expression of the backscattered wave of the RF Gaussian unit is: s back =(G P n back )×exp(j·G p φ back )×s abs Among them, φ back is the phase shift during absorption, s abs Expression for the absorption wave of this Gaussian unit.

5. The method for 3D reconstruction of objects based on millimeter wave radar differentiable radio frequency modeling and super-resolution estimation according to claim 1, characterized in that: In said S4: The propagation of RF signals is divided into the transmission phase and the reception phase. The propagation process in the reception phase is mathematically equivalent to the transmission phase, that is, the propagation of the transmission signal is regarded as the reverse propagation of the reception signal. Each RF Gaussian unit transmits and receives signals along the signal propagation path, which undergoes geometric attenuation, absorption, and accumulation of phase shifts. The signal received by the RF Gaussian unit is expressed as: in, is the intermediate frequency signal received in the direction of solid angle ω, A0 is the initial intensity of the transmitted signal, is the geometric weight factor of the kth Gaussian unit in the ω direction, s abs The expression of the absorption wave of the Gaussian unit; The signal propagation structure in the receiving phase is the same as that in the transmitting phase, except that the propagation direction is reversed. That is, the signal is transmitted from each RF Gaussian unit back to the receiving antenna. The entire propagation process maintains the differentiability of each Gaussian unit geometry and RF response parameters. Finally, the differentiable RF attenuation propagation model is explicitly derived to construct a differentiable millimeter-wave signal simulation system.

6. The method for 3D reconstruction of objects based on millimeter wave radar differentiable radio frequency modeling and super-resolution estimation according to claim 1, characterized in that: In said S5: Initialize the millimeter-wave signal simulation system in S4 using high-range-resolution millimeter-wave radar parameters, input the radar pose when collecting the intermediate-frequency signal in S1, and output a simulated high-resolution intermediate-frequency signal. Utilize a gradient descent optimization algorithm, using the reconstruction error between the high-resolution intermediate-frequency signal in S2 and the simulated high-resolution intermediate-frequency signal output by the system as the loss function. Combined with the gradient backpropagation path explicitly derived in the simulation system, the gradient of the loss function is transferred to the parameters of the RF Gaussian units that constitute the scene. These parameters are then optimized in the direction of gradient descent, and the process is iterated a certain number of times. In the loss function, the mean square error is used as the reconstruction loss to ensure the consistency of the generated signal and the actual signal in the Range spectrum and Range-Angle spectrum. The optimization goal is to minimize this difference, thereby iteratively optimizing the representation of the RF Gaussian three-dimensional object.

7. The method for 3D reconstruction of objects based on millimeter wave radar differentiable radio frequency modeling and super-resolution estimation according to claim 1, characterized in that: In said S6: A gradient-guided dynamic expansion algorithm is designed. This algorithm analyzes whether the mean gradient of the current RF Gaussian unit after projection in the distance dimension exceeds a certain threshold. If so, it determines that the reconstruction process has "under-reconstruction" or "over-reconstruction" problems, and handles them accordingly based on the scaling parameters of the RF Gaussian unit. When the scaling parameter of a RF Gaussian unit is less than a certain threshold, the Gaussian unit is considered "under-reconstructed". In this case, the algorithm will automatically clone and adjust the existing RF Gaussian unit, expanding it along the gradient direction to increase the coverage of object details. When the scaling parameter of a RF Gaussian unit is greater than a certain threshold, the Gaussian unit is considered to be "over-reconstructed". At this time, the algorithm will split and scale the oversized RF Gaussian unit, also increasing the description of the object's details.

8. A 3D object reconstruction system based on millimeter-wave radar differentiable radio frequency modeling and super-resolution estimation for implementing the method according to any one of claims 1 to 7, characterized in that: It includes radar signal acquisition module, signal processing module, signal super-resolution estimation module, and object 3D reconstruction module based on differentiable RF modeling; Radar signal acquisition module: This module consists of a multi-antenna commercial millimeter-wave radar, positioned directly in front of the target. It collects target reflection signals when the millimeter-wave radar is located at different positions. Furthermore, by rotating the target object, the collected signals ensure uniform coverage of the target surface from all angles. Signal processing module: collects background environmental signals without target objects, eliminates static environmental interference from the original signal through static background removal method, and outputs the target signal after removing the static environment; Signal super-resolution estimation module: This module includes a random mask component, a generator component, a discriminator component, and a reconstruction loss function. The random mask component performs random masking on the target signal. The generator component integrates an inter-channel attention component, which uses a multi-scale convolution module and the inter-channel attention component to recover high-resolution radar signals from the target signal while simultaneously performing noise elimination. The discriminator component evaluates the high-resolution signal using the reconstruction loss function and optimizes the generator output in combination with the structure of a generative adversarial network. The object three-dimensional reconstruction module based on differentiable RF modeling includes an initialization component for three-dimensional Gaussian space representation, a signal propagation component based on differentiable RF modeling, a dynamic Gaussian unit expansion algorithm component, and an iterative optimization component based on gradient descent; the initialization component for three-dimensional Gaussian space representation processes the input signal, extracts the original initial point cloud from the target signal, and initializes the RF Gaussian unit; the signal propagation component based on differentiable RF modeling randomly selects a radar posture in the data set and obtains the reflected signal corresponding to the position; the dynamic Gaussian unit expansion algorithm component dynamically increases or decreases the Gaussian unit according to the mean gradient after projection in the distance dimension; the iterative optimization component based on gradient descent optimizes the parameters of the Gaussian unit through gradient by comparing the generated signal with the signal in the data set.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 7.

10. A computing device, characterized in that include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method according to any one of claims 1 to 7.

Images