A method for densifying a millimeter wave radar sparse point cloud based on a conditional generative adversarial network

Abstract

This invention relates to a method for denserening sparse point clouds in millimeter-wave radar based on conditional generative adversarial networks (GANs), belonging to the interdisciplinary field of millimeter-wave radar perception and computer vision. The method includes radar raw echo acquisition and signal preprocessing to generate an extremely sparse 3D point cloud and heatmap tensor; constructing a 3D multi-channel conditional tensor that fuses range, Doppler, and energy information; offline training of the 3D conditional GAN; and voxel mesh post-processing to output a high-density 3D dense point cloud carrying radial velocity. This invention restores the true and complete outline of the target by fusing voxel conditional tensors with range, angle, and Doppler 3D physical information and constructing a 3D conditional GAN, ensuring that velocity motion characteristics conform to physical laws and suppressing false illusions. Lightweight post-processing yields a dense 3D point cloud with sub-voxel precision and accompanying velocity information, which can directly support rescue perception tasks such as personnel search and rescue, 3D mapping, and obstacle identification at fire and rescue sites.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of millimeter-wave radar perception and computer vision, and to a method for dense reconstruction of sparse point clouds of millimeter-wave radar using conditional generative adversarial networks in fire fighting and rescue scenarios, under harsh environments such as dense smoke and dim light. In particular, it relates to a method for denser millimeter-wave radar sparse point clouds based on conditional generative adversarial networks. Background Technology

[0002] In complex fire rescue scenarios such as fires in high-rise buildings, underground tunnels, and chemical industrial parks, the on-site environment exhibits three significant characteristics: high-density smoke obscuring the view, extremely low light conditions, and the coexistence of high temperatures and toxic gases. These harsh conditions severely limit the effectiveness of traditional visual sensors. At most fire rescue sites, infrared thermal imaging cameras have limited transmission capabilities in dense smoke environments, and smoke particles strongly scatter and absorb infrared radiation, leading to a sharp decline in image quality. Conventional visible light cameras are completely ineffective in complete darkness or smoky environments. Millimeter-wave radar (typically at 24GHz, 60GHz, or 77GHz), with its longer electromagnetic wavelength (on the order of millimeters), can effectively penetrate smoke, dust, and rain, and is unaffected by lighting conditions. It has become one of the core environmental perception sensors for equipment such as firefighting robots and search and rescue drones. However, the sparse point cloud characteristics of millimeter-wave radar severely hinder its application in the fire rescue field.

[0003] The sparsity of point clouds generated by millimeter-wave radar in practical applications is determined by both physical principles and signal processing factors. At the physical level, millimeter-wave radar has low spatial angular resolution. Furthermore, non-metallic materials (such as firefighters' protective clothing, wooden obstacles, and human tissue) have low reflectivity to millimeter-wave radar, resulting in a small radar cross-section (RCS) and weak echo signals, further exacerbating the sparsity and instability of the point cloud. At the signal processing level, the traditional millimeter-wave radar signal processing flow typically includes: range-Doppler (RD) matrix generation, constant false alarm rate (CFAR) detection, angle estimation, and point cloud clustering and filtering. Among these, the CFAR detection algorithm must set a certain detection threshold to distinguish targets in noise. In the fire-fighting and rescue environment, due to thermal noise generated by high temperatures, multipath interference caused by reflective surface echoes, and clutter caused by smoke particles (although millimeter waves can penetrate smoke, some scattered energy still exists in high-concentration smoke), the CFAR algorithm must increase the detection threshold to maintain a low false alarm rate. This results in a large number of weak target echoes with low signal-to-noise ratio (SNR) being filtered out, and the final output point cloud contains only a few high-confidence echo points.

[0004] Taking the detection of a standing human target as an example, at a distance of 30-50 meters, a high-performance 77GHz radar can typically only generate 1-3 discrete points. These point clouds cannot reflect the geometric morphological information such as the outline and posture of the human body; they can only provide a rough estimate of the target's distance, orientation, and radial velocity. When this extremely sparse point cloud data is directly used for advanced perception tasks such as 3D mapping, target recognition, and bounding box prediction, it is prone to numerous problems, including missed detections, target fragmentation (the same target being split into multiple segments), and false alarms (mistaking noise points for targets).

[0005] Currently, industry and academia have proposed various technical approaches to address the problem of sparse or super-resolution point clouds, but each approach has its own limitations in different fields.

[0006] 1) Super-resolution algorithms based on traditional signal processing, such as MUSIC (Multiple Signal Classification), ESPRIT (Rotation Invariant Subspace), and sparse reconstruction algorithms based on compressed sensing. These algorithms require raw time-domain / frequency-domain data input from all channels and have high signal-to-noise ratio requirements. Since the radar front-end has already processed the signal into a discrete point cloud and lost the original ADC data, these methods cannot be used for subsequent processing. Furthermore, these algorithms have high computational complexity, making them difficult to run in real-time on embedded platforms.

[0007] 2) Point cloud restoration methods based on geometric interpolation, such as linear interpolation, polynomial fitting, and Poisson surface reconstruction. These methods are suitable for point clouds with moderate density and small holes. However, for extremely sparse point clouds (where a target has only 1-3 points), geometric interpolation can produce large areas of undefined planes or surfaces, resulting in numerous illusory points. For example, linear interpolation between two points of a target generates a straight line instead of the target's true outline.

[0008] 3) Point cloud completion methods based on Graph Convolutional Networks (GCNs) or Transformers, such as PCN (Point Completion Network) and PointTr (Transformer on Point Cloud), directly input sparse point clouds as a point set and learn the distribution patterns of the point cloud in 3D space. In the field of point cloud completion, these methods mainly target the repair of missing parts of LiDAR or structured light scanning point clouds. However, their input point clouds usually still need to contain a certain number (e.g., several hundred) of background points and some target points. When the input point cloud is extremely sparse (e.g., 3-5 points), the topological relationships between points are extremely fragile, and the network struggles to extract effective shape features from the disordered point set. The network output tends to degenerate into a blurry, averaged spherical structure, failing to recover the specific shape of the target.

[0009] 4) Radar Imaging Super-Resolution Based on Generative Adversarial Networks (GANs): Existing work includes cases of applying GANs to super-resolution of synthetic aperture radar (SAR) images or radar range-angle heatmaps. For example, the radar's range-angle map is input into a super-resolution GAN to generate a high-resolution image. However, these methods have two core drawbacks: First, they typically only perform pixel-level upscaling of the heatmap (2D / 3D tensor), resulting in clear images but not point clouds in three-dimensional space. Second, they ignore the Doppler dimension information unique to radar, potentially leading to discontinuous or unreasonable physical motion properties in the generated images or point clouds. For example, the generated stationary target point cloud may contain noise points with velocity, or the torso and arms of a walking pedestrian may have discontinuous velocities.

[0010] 5) A multi-sensor fusion-based completion method can be implemented, incorporating LiDAR data to guide the training of radar point clouds, or fusing it with camera images for depth estimation. However, in fire and rescue scenarios, LiDAR will also fail in dense smoke (the laser beam will be strongly reflected by smoke, causing a surge in noise or penetration failure); cameras cannot image in dim / smoky conditions. Therefore, multi-sensor fusion solutions are not feasible unless a reasonable and scientifically sound sensor fusion scheme is explored. Summary of the Invention

[0011] (a) Technical problems to be solved In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a method for densifying sparse point clouds of millimeter-wave radar based on conditional generative adversarial networks. First, sparse millimeter-wave radar data is encoded into a range-Doppler-angle conditional tensor containing range, Doppler velocity and angle / energy information. Then, a high-resolution voxel occupancy probability grid is output by a generator based on 3D U-Net. Next, a 3D PatchGAN discriminator and various loss functions are used for training constraints. Finally, the voxel grid is converted into a dense point cloud with three-dimensional coordinates and radial velocity.

[0012] (II) Technical Solution To achieve the above objectives, the main technical solution adopted by this invention is a method for denser sparse point clouds of millimeter-wave radar based on conditional generative adversarial networks, comprising the following steps: S1. A linear frequency modulated continuous wave signal is transmitted at the fire-fighting and rescue site using millimeter-wave radar to collect the original time-domain echo at the fire-fighting site. The time-domain echo signal of each channel is preprocessed to generate a multi-channel RD heatmap tensor and perform data noise reduction to obtain a candidate detection unit index set. The candidate detection unit index and the phase data of the multi-antenna channel are processed by angle FTT, digital beamforming DBF, coordinate transformation and velocity assignment to obtain a sparse point cloud. S2. Establish a three-dimensional voxel grid index space. Construct range tensor, Doppler tensor, and angle tensor from the sparse point cloud falling into the three-dimensional voxel grid index space. Combine them into a three-dimensional multi-channel conditional tensor C. The conditional tensor C retains the three-dimensional physical properties of radar range, velocity, and angle, and is aligned with the three-dimensional voxel space. S3. Using the conditional tensor C as input, a 3D conditional adversarial generative network model is constructed, which includes a generator network and a discriminator network. The generator network adopts an encoding, bottleneck, and decoding architecture and retains local details through skip connections, outputting a voxel grid representing the occupancy probability of each position. The discriminator network uses a block-based discrimination mechanism to constrain the local consistency of the reconstruction results. During training, adversarial loss, positional bias loss, velocity smoothing loss, and spatial suppression loss are fused to form a weighted total loss, which respectively achieves the distinction between real and fake samples, accurate fitting of target positions, continuous constraint of motion velocity, and suppression of false reconstruction in blank areas. After the model training is completed, the input conditional tensor generates a high-resolution dense spatial voxel grid. S4. Perform soft thresholding on the voxel probability tensor output by the network, retain high-confidence target voxels and extract the voxel center as the initial point, then optimize the point coordinates through lightweight point processing network to improve the accuracy at the sub-voxel level, and eliminate duplicate and redundant points with non-maximum suppression. Finally, match velocity information for each point to generate a 3D dense point cloud that contains both spatial coordinates and motion attributes, so as to fully restore the target contour based on the original sparse points.

[0013] In step S1, the time-domain echo signal of each channel is preprocessed to generate a multi-channel RD heatmap tensor. Specifically, the millimeter-wave radar transmits frequency-modulated continuous wave (FMCW) signals through a multi-antenna array and collects the original time-domain echo signal data of the radar's multiple channels. The time-domain echo signal of each antenna channel is windowed using a Hanning window, and a range-dimensional FFT is performed on the time-domain echo signal to map the time-domain signal to the range domain, establishing a correspondence between the range cell index and the actual distance. Based on the range dimension of multiple consecutive time-domain echo signals, a Doppler-dimensional FFT is performed to obtain a range-Doppler heatmap, generating a multi-channel RD heatmap tensor, where the horizontal axis represents the distance, the vertical axis represents the Doppler frequency shift, and the pixel value is the echo intensity.

[0014] In step S1, the data denoising is specifically performed by using an adaptive low-threshold CFAR detection method to denoise the multi-channel RD heatmap tensor. A low false alarm rate is set, with the false alarm rate of CFAR detection set to 2-3 times the normal value. For each range-Doppler cell, the energy of its guard cell and reference window is calculated to determine whether it is a candidate point, thus obtaining a candidate detection cell index set. Each index corresponds to a detected target point.

[0015] In step S1, the candidate detection unit index and multi-antenna channel phase data are processed by angle FTT, digital beamforming (DBF), coordinate transformation, and velocity assignment to obtain a sparse point cloud. Specifically, for each candidate detection unit, its complex value data on all antenna channels is extracted, and angle FTT processing is performed on it to obtain the angle-spectrum correspondence. The spectral peak is found, and its azimuth and elevation angles are calculated. The Cartesian coordinates of the point in three-dimensional space are calculated based on the distance, azimuth, and elevation angles. The detected Doppler unit index is converted into radial velocity to construct a sparse point cloud, which includes spatial location, radial velocity, and echo intensity.

[0016] Specifically, step S2 involves determining the sensing range, defining the voxel resolution and voxel network size, and establishing a three-dimensional voxel mesh index space; constructing a multi-channel conditional tensor C by jointly using the distance tensor, Doppler tensor, and angle / energy tensor, and precisely aligning C with the three-dimensional voxel space.

[0017] In step S3, a 3D conditional generative adversarial network (GAN) consisting of a generator and a discriminator is constructed. The generator comprises an input layer, an encoding layer, a bottleneck layer, a decoding layer, and an output layer. The input layer uses a variant of 3D U-Net as the main input to the network. The encoding layer adopts a four-layer 3D convolution + BN + LeakyReLU structure, with each layer outputting a feature map, and the number of channels gradually increasing to 512. The bottleneck layer uses a single 3D convolution, maintaining the number of channels at 512. The decoding layer adopts a four-layer 3D transposed convolution + BN + ReLU structure, and uses skip connections to concatenate the features of the encoding layer with those of the decoding layer layer by layer, preserving local details. The output layer uses a 1x1x1 convolution kernel with the sigmoid activation function, and outputs a voxel-occupying probability tensor. The discriminator adopts a 3DPatchGAN-based structure, outputting feature maps through four 3D convolutional layers with a stride of 2. It judges the authenticity of each patch in the feature map database and takes the average output real value to represent the authenticity.

[0018] In step S3, the total loss in the loss function of the 3D conditional adversarial generative model is composed of a weighted sum of adversarial loss, L1 loss, Doppler smoothing loss, and spatial occupancy loss. The adversarial loss uses standard binary cross-entropy to guide the generator to produce voxel grids that are difficult for the discriminator to distinguish. The L1 loss is normalized based on the total number of true occupancy maps to provide the generator with sparse supervision signals so that it generates occupancy in the correct position. The Doppler smoothing loss extracts the 3D velocity field output by the additional velocity prediction branch from the generator's hidden layer and applies total variation regularization. The spatial occupancy loss constructs a spatial mask based on the sparse point cloud position and its neighborhood information, and directly penalizes voxels whose probability of generating in a highly certain blank area exceeds a threshold, thus suppressing the model from generating false targets.

[0019] Step S4 specifically involves performing soft thresholding on all voxels, retaining voxels with an occupancy probability higher than a preset threshold, using the center coordinates of the selected voxels as the initial point cloud, fine-tuning the center coordinates using the PointNet deep learning network, and removing closely spaced duplicate points using maximum suppression (NMS); combining the fine-tuned spatial coordinates with the corresponding velocity values ​​obtained from the conditional tensor C in step S2 to form a dense point cloud.

[0020] (III) Beneficial Effects The beneficial effects of this invention are: 1. This invention uses only millimeter-wave radar as the sole sensing source throughout the entire process, and can complete dense point cloud reconstruction without the need for multi-sensor fusion; millimeter-wave radar can penetrate smoke and dust and is unaffected by light, and combined with low-threshold adaptive CFAR detection, it fully preserves weak target echoes, solving the problem of traditional radar processing filtering out a large number of low signal-to-noise ratio human body and non-metallic obstacle echoes and extremely sparse point clouds, making it suitable for various complex fire rescue sites such as high-rise buildings, tunnels, and chemical industrial parks; 2. This invention transforms sparse radar information into a three-dimensional multi-channel voxel conditional tensor that fuses range, angle, and Doppler, fully preserving the physical dimension features of the radar. Relying on 3D conditional GAN, it learns the geometric distribution of the target from the global voxel space. It can restore the complete and realistic outline of the human body and obstacles with only 3 or fewer sparse radar points, without large-area invalid interpolation and average spherical distortion. 3. The generator of this invention adopts a 3D U-Net skip connection structure to preserve local geometric details, and the PatchGAN block discrimination mechanism ensures the consistency of the reconstructed local area. The total loss is fused with four types of constraint losses: adversarial loss to distinguish between real and fake voxels, L1 position loss to ensure that the target coordinates fit the original sparse points, Doppler smoothing loss to constrain the continuous and reasonable velocity of the generated point cloud, and avoid results that violate physical laws, such as static targets with moving velocity or human limb velocity being disjointed. Spatial domain suppression loss punishes false voxels in blank areas, greatly reducing illusory noise points, and balancing the accuracy of contour restoration with the physical reality of radar Doppler motion dimension. 4. This invention uses soft thresholding to filter high-confidence target voxels, then uses lightweight PointNet to optimize coordinates to improve the accuracy of sub-voxels, and combines non-maximum suppression to remove redundant and duplicate points, finally outputting a 3D dense point cloud with accurate radial velocity information; the entire signal preprocessing, 3D GAN inference, and point cloud optimization process does not involve ultra-high complexity calculations, does not require original radar ADC data, and can be computed using only conventionally processed sparse point cloud and RD heatmap tensors, and can be deployed on embedded terminals of firefighting robots and search and rescue drones to achieve real-time reconstruction, supporting upper-level rescue perception tasks such as 3D mapping, human target recognition, and dangerous boundary detection; 5. This invention constructs a multi-channel conditional tensor as network input to reconstruct a dense point cloud, simultaneously attaching spatial coordinates and motion velocity attributes. It solves the shortcomings of the original sparse point cloud, which can only roughly determine the target's location and distance, cannot identify human posture, or distinguish between stationary obstacles and moving trapped personnel. The reconstructed dense point cloud can accurately restore the target's outline shape, helping rescue equipment to accurately locate trapped personnel, identify obstacles such as walls and debris, reduce perception failures such as missed detections, target fragmentation, and false alarms, and improve the safety and efficiency of fire search and rescue. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0022] To better explain and facilitate understanding of the present invention, a detailed description of the invention is provided below with reference to the accompanying drawings and specific embodiments. The present invention proposes a method for denserening sparse point clouds in millimeter-wave radar based on conditional generative adversarial networks.

[0023] This method is deployed on embedded terminals of fire rescue robots and fire search and rescue drones, using a 77GHz MIMO millimeter-wave radar (3-transmit, 4-receive antenna array) for perception. During the inference phase, it relies solely on the millimeter-wave radar sensor, eliminating the need for visible light cameras, infrared thermal imaging, or lidar. For offline model training, a high-precision lidar is used to collect ground truth data. It is suitable for various harsh fire rescue scenarios, including high-rise buildings, underground tunnels, chemical industrial parks with dense smoke, darkness, and high temperatures. The entire method consists of four core steps, with data passed sequentially and each preceding output serving as the sole input for the subsequent one. Step 1: Acquire raw radar echoes from the fire scene and preprocess the signals to generate extremely sparse 3D point clouds and RD heatmap tensors; Step 2: Construct a three-dimensional multi-channel conditional tensor that fuses distance-Doppler-energy information; Step 3: Offline training and real-time voxel inference of 3D conditional generative adversarial network; Step 4: Post-process the voxel mesh to output a high-density 3D dense point cloud carrying radial velocity.

[0024] Step S1: Radar echo acquisition, signal preprocessing and extremely sparse point cloud generation.

[0025] 1.1 Acquisition of raw time-domain echo and preprocessing using window functions; The radar continuously transmits linear frequency modulated continuous wave (FMCW) signals, and multiple antenna channels simultaneously acquire the original time-domain echo of the fire scene; Hanning windowing is applied to the time-domain echo of each antenna to suppress range sidelobes and reduce clutter interference caused by high temperature and smoke.

[0026] 1.2 Two-layer FFT transformation to generate RD heatmap tensors; Range-dimensional FFT: Perform a one-dimensional FFT on the sampled data of a single-segment frequency-modulated pulse to map the time-domain signal to the range domain and establish a one-to-one correspondence between the range cell index and the actual detection range; Doppler-dimensional FFT: Perform a quadratic FFT on the distance dimension calculation results of 256 consecutive frequency-modulated pulses to generate a multi-channel distance-Doppler thermogram tensor. ; The total number of distance cells. The total number of Doppler velocity elements. The number of radar antenna channels is represented by the tensor pixel value, which represents the echo reflection energy of the corresponding range-velocity cell.

[0027] 1.3 Adaptive low-threshold CFAR noise reduction detection; CFAR false alarm rate Set to 2-3 times the normal parameters, and adopt a conservative sliding window detection mechanism: traverse all detection units in the RD heatmap, estimate the power of local thermal noise and smoke multipath clutter by relying on protection units and reference windows, remove only low-energy pure noise units, retain the weak echoes of human bodies and wooden obstacles that would be filtered out by traditional high-threshold CFAR, and output a candidate detection unit index set.

[0028] 1.4 Angle estimation and coordinate transformation to generate sparse point clouds; Angle FFT + Digital Beamforming (DBF): Extract complex phase data from all antenna channels of candidate detection units, zero-padding, FFT to solve for the peak angle spectrum, and calculate the target azimuth and elevation angles. ; Assigning 3D coordinates and velocity: combining distance cell indexing Azimuth Pitch angle Converting to Cartesian 3D coordinates Index the Doppler units Mapped to target radial velocity Synchronously read the unit echo intensity; The output is a sparse point cloud set containing only coarse distance and velocity information, without complete target contour geometry. ;in, For the first Echo intensity at each point For the first A sparse point, This is the set of all sparse point clouds.

[0029] Step S2: Complete construction process of 3D multi-channel RDA conditional tensor; The input for this step is the set of all sparse point clouds output from step 1. Heatmap tensor RD; outputs a 3-channel conditional tensor with uniform alignment. This serves as the input to the generator in step 3.

[0030] 2.1 Initialize the three-dimensional uniform voxel mesh space; Delineating the three-dimensional sensing boundary of the radar: : Represents the coordinate axes of the three-dimensional perception space, lateral Forward ,high ; Set voxel resolution Calculate the total number of triaxial voxels and construct an integer index space. : indicates that the entire voxel mesh consists of one The index of a three-dimensional integer array.

[0031] : Represents the three dimensions of the voxel mesh, all of which are positive integers. Representative along Voxel count along the axis, ; Representative along Voxel count along the axis, ; Representative along Voxel count along the axis, ; Establish a two-way mapping relationship between the index and the actual coordinates: voxel index Corresponding spatial center coordinates; The three-dimensional coordinates of any sparse point can be converted in reverse to obtain a unique corresponding voxel index.

[0032] 2.2 Constructing a single-channel 3D tensor independently for each channel; Channel 1: Distance Tensor ; Initial size A three-dimensional array of all zeros; traverse all sparse points and reverse map to obtain the corresponding voxel indices. , , respectively corresponding Axial direction; Will The value is assigned to the actual detection distance of that point. Blank voxels without sparse point coverage retain a value of 0. This is used to provide the network with a priori information about the target's spatial distance, constraining the reconstructed target to not deviate from the original sparse point location.

[0033] Channel 2: Doppler Tensor ; Synchronous initialization A three-dimensional array of all zeros; traverse sparse points, map to the corresponding voxel indices, and... The value is assigned to the radial velocity of the m-th sparse point. ; Set blank voxels to 0.

[0034] Independently encode the motion characteristics of the target, distinguish between stationary obstacles and moving trapped persons, and provide basic data for Doppler smoothing loss.

[0035] Channel 3: Angle / Energy Tensor ; initialization A three-dimensional array of all zeros; Sparse point basic assignment: Traverse sparse points, map to corresponding voxels, and then... The value is assigned to the inherent echo intensity of the m-th sparse point. ; For all For blank voxels with a value of 0, perform a spatial inverse mapping, converting the voxel center coordinates into RD graph distance indices. Doppler Index ; Representative and Voxel Index The corresponding original RD heatmap index is mapped from the voxel space center point to the RD plane through inverse coordinate transformation; By aggregating the energy from all antenna channels at this location, weak echo energy can be obtained. That is, the original RD heatmap in the index The pixel value at that location, a real number, represents a certain aggregation of the energy of all antenna channels on that range-velocity unit, and is overlaid on the blank voxel. ; Ultimately, this channel simultaneously contains high-confidence energy of discrete points and weak target reflection cues that were lost through CFAR filtering, thus compensating for the lack of information in extremely sparse point clouds.

[0036] 2.3 Three-channel tensor fusion and spatial alignment verification; Completely consistent in size By splicing and merging along the channel dimension, a four-dimensional multi-channel conditional tensor is generated. .

[0037] Unified verification of three-way tensor index mapping rules to ensure consistency The distance, velocity, and energy information in the voxel correspond to the same real-world cube with no coordinate misalignment. After completion, input the data into the network module in step 3.

[0038] Step S3: Training the 3D Conditional Generative Adversarial Network (3D CGAN); The input for this step is the output condition tensor C from step 2; the output is the voxel occupancy probability tensor. Each value This indicates the probability that the voxel is occupied.

[0039] The network consists of a 3D U-Net generator, a 3D PatchGAN discriminator, and a four-component weighted composite loss function, and is divided into two independent stages: offline model training and real-time inference.

[0040] 3.1 Network infrastructure; Generator (3D U-Net variant); Input layer: Directly load a 3-channel conditional tensor ; Encoding layer: 4 layers of 3D convolution + BN batch normalization + LeakyReLU, compressing spatial size layer by layer, and increasing the feature channels to 512; each layer caches shallow feature maps for skip connections to preserve local contour details; Bottleneck layer: Single-layer 3D convolutional fusion of global full-space context, maintaining 512 channels; branches independently output 3D velocity field tensors, dedicated to Doppler smoothing loss calculation; Decoding layer: 4 layers of 3D transposed convolution + BN + ReLU, recovering spatial dimensions layer by layer; by splicing shallow features of corresponding levels of the encoding layer through skip connections, the loss of contour details caused by downsampling is compensated; Output layer: 1×1×1 3D convolution compresses channels to 1D, paired with a Sigmoid activation function, outputting a voxel occupancy probability tensor. The tensor value ranges from 0 to 1, representing the confidence level that the corresponding voxel has a real target.

[0041] Discriminator (3D PatchGAN block discrimination network); The inputs are divided into two categories: the voxel occupancy probability tensor output by the generator. The actual binary voxels occupy the label tensor. The network consists of four consecutive 3D convolutional layers with a stride of 2. It extracts local features from 3D patches and does not output a single true or false score for the entire grid. Instead, it independently distinguishes each 3D patch and finally takes the average score of all patches as the output. It finely constrains the consistency of local space to avoid local contour distortion and large areas of false illusion.

[0042] 3.2 Weighted composite total loss function; Total loss Calculation formula: ; These are all hyperparameters, settings ; Combating losses Standard binary cross-entropy loss is used to build an adversarial game between the generator and the discriminator, guiding the generator to generate voxel meshes that are difficult for the discriminator to distinguish. Normalized L1 position loss : ; The total number of cells occupied by the target in the true value voxel constrains the reconstructed target to strictly fit the original sparse points, avoiding overall positional shift; Doppler smoothing loss : ; It is a three-dimensional spatial gradient operator that calculates the gradient norm 2 of the velocity field, penalizes abrupt velocity changes on the surface of the same object, ensures the continuous velocity of the human torso and limbs, and the uniform velocity of a stationary target across the entire domain, which conforms to the Doppler physical laws. The predicted velocity field tensor output by the intermediate layer of the generator; Space occupation loss Constructing a blank spatial mask based on sparse point neighborhood and radar main beam range Mark areas where no objects can be identified by the radar; mark areas where the probability of occupancy exceeds a threshold. The system punishes false voxels, suppressing illusory points and false obstacles that are generated out of thin air.

[0043] During training, dataset preparation first involves simultaneously acquiring 77GHz millimeter-wave radar and high-precision lidar data, covering various scenarios such as dense smoke, no light, tunnels, and indoor fire scenes. The same frame of radar data is batch-processed according to steps 1 and 2 to generate a conditional tensor C. Simultaneously, dense laser point clouds are mapped to a 0.5m voxel grid of the same size to generate binary ground truth voxels. All samples are divided into training, validation, and test sets in a ratio of 8:1:1, with a batch size of 4. Small translations and velocity noise perturbations are applied to the conditional tensor to achieve data augmentation.

[0044] Instantiate generator G and discriminator D; select Adam optimizer, with a 50% learning rate decay every 30 epochs; enable label smoothing and gradient clipping to prevent gradient explosion and mode collapse; set an early stopping rule: if the total loss of the validation set does not decrease after 10 consecutive epochs, terminate training and save the optimal generator weights; the maximum training iteration epoch limit is 200.

[0045] Single-epoch alternating game training; Phase 1: Freeze the generator G and update only the discriminator D. After clearing the gradients to zero, input the true ground truth voxels and the generator output pseudo voxels respectively, calculate the discriminator adversarial loss, and update only the discriminator weights during backpropagation.

[0046] Phase 2: Freeze the discriminator D and update the generator G. Zero out the gradients and perform forward inference using the conditional tensor to obtain the predicted voxels and velocity fields. Calculate the adversarial loss, L1 position loss, Doppler smoothing loss, and space occupancy loss sequentially, and sum them by weight to obtain the total loss. Backpropagation updates all network weights of the generator.

[0047] Phase 3: Batch log recording, retaining discriminator loss and generator component loss for convergence monitoring.

[0048] Step S4: Post-processing of voxel meshes to generate dense 3D point clouds with velocity information; The input for this step is the voxel probability tensor output from step 3. Output dense point cloud The point cloud fully restores the target outline, simultaneously carrying three-dimensional spatial coordinates and radial velocity attributes.

[0049] First, perform soft threshold filtering to select high-confidence voxels, and then execute the filtering formula. ,in This is the Sigmoid function, which maps input real values ​​to the interval (0, 1).

[0050] It significantly suppresses noisy voxels with an occupancy probability below 0.5, preserves and enhances high-confidence target voxels, and reduces subsequent computational load.

[0051] Extract the voxel centers as the initial coarse point set, and traverse the voxel occupancy probability tensor after soft thresholding. Only valid voxels with an occupancy probability > 0.5 are retained. Voxel values ​​with a probability below 0.5 in the original output are suppressed to be closer to 0, while those above 0.5 are retained and enhanced. This is achieved through voxel indexing, utilizing an index-coordinate mapping formula. Calculate the three-dimensional center coordinates of each effective voxel cube based on the sensing range and resolution. This forms an unoptimized initial point cloud.

[0052] Lightweight PointNet sub-voxel level coordinate fine-tuning feeds the initial coarse point set into the built-in small PointNet network, and performs regression fine-tuning of the 3D coordinates of all points, eliminating the contour jaggedness and step distortion caused by discrete voxel meshes, thereby improving the accuracy of sub-voxel level coordinates and smoothing the target shape contour.

[0053] Non-maximum suppression (NMS) eliminates redundant and duplicate points. A three-dimensional Euclidean distance threshold is set. When the spatial distance between two points is less than the threshold, only the points with higher confidence are retained, and the rest of the redundant points are deleted. This balances the density of the point cloud with the real-time computing pressure of the embedded terminal.

[0054] Matching the radial velocity, the final dense point cloud is output. The fine-tuned and deduplicated point coordinates are then mapped back to the corresponding voxel indices. The conditional tensor Doppler channel from step 2 is then read. The velocity value of the internal matching is used as , representing the The radial velocity of the final output point; combining coordinates and velocity to generate the first... A dense point , Representing the The three-dimensional spatial coordinates of the final output point are real numbers in meters. They have been further optimized by fine-tuning the regression network, and their accuracy is higher than that of direct voxel centers. All points constitute a dense cluster of points. Its point density and accuracy are superior to the ensemble of all input sparse point clouds. .

[0055] Implementation results: Human targets that were originally only 1 to 3 discrete points can be reconstructed into hundreds of points, completely restoring human posture and three-dimensional contours of obstacles; dense point clouds can be directly supplied to upper-level rescue tasks such as SLAM three-dimensional mapping of fire scene, identification of trapped personnel, and early warning of dangerous obstacles.

[0056] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for densifying sparse point clouds in millimeter-wave radar based on conditional generative adversarial networks, characterized in that, Includes the following steps: S1. By transmitting linear frequency modulated continuous wave signals at the fire fighting and rescue site using millimeter-wave radar, the original time-domain echo of the site environment is collected. The time-domain echo signal of each channel is preprocessed to generate a multi-channel RD heat map tensor and perform data noise reduction to obtain a candidate detection unit index set. Angle FFT, digital beamforming DBF, coordinate transformation and velocity assignment are performed on the candidate detection unit index and multi-antenna channel phase data to obtain sparse point clusters. S2. Establish a three-dimensional voxel grid index space. Construct range tensor, Doppler tensor, and angle tensor from the sparse point cloud falling into the three-dimensional voxel grid index space. Combine them into a three-dimensional multi-channel conditional tensor C. The conditional tensor C retains the three-dimensional physical properties of radar range, velocity, and angle, and is aligned with the three-dimensional voxel space. S3. Using the conditional tensor C as input, a 3D conditional generative adversarial network model is constructed, which includes a generator network and a discriminator network. The generator network adopts an encoding, bottleneck, and decoding architecture and retains local details through skip connections, outputting a voxel grid representing the occupancy probability of each position. The discriminator network uses a block-based discrimination mechanism to constrain the local consistency of the reconstruction results. During training, adversarial loss, positional bias loss, velocity smoothing loss, and spatial suppression loss are fused to form a weighted total loss, which respectively achieves the distinction between real and fake samples, accurate fitting of target positions, continuous constraint of motion velocity, and suppression of false reconstruction in blank areas. After the model training is completed, the conditional tensor C is input to generate a high-resolution dense spatial voxel grid. S4. Perform soft thresholding on the voxel probability tensor output by the network, retain high-confidence target voxels and extract the voxel center as the initial point, then optimize the point coordinates through lightweight point processing network to improve the accuracy at the sub-voxel level, and eliminate duplicate and redundant points with non-maximum suppression. Finally, match velocity information for each point to generate a 3D dense point cloud that contains both spatial coordinates and motion attributes, so as to fully restore the target contour based on the original sparse points.

2. The method for densifying sparse point clouds of millimeter-wave radar based on conditional generative adversarial networks as described in claim 1, characterized in that: In step S1, the time-domain echo signal of each channel is preprocessed to generate a multi-channel RD heatmap tensor. Specifically, the millimeter-wave radar transmits frequency-modulated continuous wave (FMCW) signals through a multi-antenna array and collects the original time-domain echo signal data of the radar's multiple channels. The time-domain echo signal of each antenna channel is windowed using a Hanning window, and the time-domain echo signal is subjected to range-dimensional FFT to map the time-domain signal to the range domain, establishing a correspondence between the range cell index and the actual distance. Doppler FFT is performed on the distance dimension results of multiple consecutive time-domain echo signals to obtain a distance-Doppler heatmap, generating a multi-channel RD heatmap tensor, where the horizontal axis is the distance, the vertical axis is the Doppler frequency shift, and the pixel value is the echo intensity.

3. The method for densifying sparse point clouds of millimeter-wave radar based on conditional generative adversarial networks as described in claim 2, characterized in that: In step S1, the data denoising is specifically performed by using an adaptive low-threshold CFAR detection method to denoise the multi-channel RD heatmap tensor. The false alarm rate of CFAR detection is set to 2-3 times the normal value. For each distance-Doppler cell, the energy of its guard cell and reference window is calculated to determine whether it is a candidate point, and a candidate detection cell index set is obtained. Each index corresponds to a detected target point.

4. The method for densifying sparse point clouds of millimeter-wave radar based on conditional generative adversarial networks as described in claim 1, characterized in that: In step S1, angle FFT, digital beamforming (DBF), coordinate transformation, and velocity assignment are performed on the candidate detection unit index and multi-antenna channel phase data to obtain a sparse point cloud. Specifically, for each candidate detection unit, its complex value data on all antenna channels is extracted, and angle FFT processing is performed on it to obtain the angle-spectrum correspondence. The spectral peak is found, and its azimuth and elevation angles are calculated. The Cartesian coordinates of the point in three-dimensional space are calculated based on the distance, azimuth, and elevation angles. The detected Doppler unit index is converted into radial velocity, and a sparse point cloud is constructed, which includes spatial location, radial velocity, and echo intensity.

5. The method for densifying sparse point clouds of millimeter-wave radar based on conditional generative adversarial networks as described in claim 1, characterized in that: Specifically, step S2 involves determining the sensing range, defining the voxel resolution and voxel network size, and establishing a three-dimensional voxel mesh index space; constructing a multi-channel conditional tensor C by jointly using the distance tensor, Doppler tensor, and angle tensor, and precisely aligning the conditional tensor C with the three-dimensional voxel space.

6. The method for densifying sparse point clouds of millimeter-wave radar based on conditional generative adversarial networks as described in claim 1, characterized in that: In step S3, a 3D conditional generative adversarial network (GAN) consisting of a generator and a discriminator is constructed. The generator comprises an input layer, an encoding layer, a bottleneck layer, a decoding layer, and an output layer. The input layer uses a variant of 3D U-Net as the main input to the network. The encoding layer adopts a four-layer 3D convolution + BN + LeakyReLU structure, with each layer outputting a feature map, and the number of channels gradually increasing to 512. The bottleneck layer uses a single 3D convolution, maintaining the number of channels at 512. The decoding layer adopts a four-layer 3D transposed convolution + BN + ReLU structure, and uses skip connections to concatenate the features of the encoding layer with those of the decoding layer layer by layer, preserving local details. The output layer uses a 1x1x1 convolution kernel with the sigmoid activation function, and outputs a voxel-occupying probability tensor. The discriminator adopts a 3DPatchGAN-based structure, outputting feature maps through four 3D convolutional layers with a stride of 2. It judges the authenticity of each patch in the feature map database and takes the average output real value to represent the authenticity.

7. The method for densifying sparse point clouds of millimeter-wave radar based on conditional generative adversarial networks as described in claim 1, characterized in that: In step S3, the total loss function of the 3D conditional generative adversarial network model is composed of a weighted sum of adversarial loss, L1 loss, Doppler smoothing loss, and spatial occupancy loss. The adversarial loss uses standard binary cross-entropy to guide the generator to produce voxel grids that are difficult for the discriminator to distinguish. The L1 loss is normalized based on the total number of real occupancy maps to provide the generator with sparse supervision signals so that it generates occupancy in the correct positions. The Doppler smoothing loss extracts the 3D velocity field output by the additional velocity prediction branch from the generator's hidden layer and applies total variation regularization. The spatial occupancy loss constructs a spatial mask based on the sparse point cloud position and its neighborhood information, and directly penalizes voxels whose probability of being generated in highly certain blank areas exceeds a threshold, thus suppressing the model from generating false targets.

8. The method for densifying sparse point clouds of millimeter-wave radar based on conditional generative adversarial networks as described in claim 1, characterized in that: Step S4 specifically involves performing soft thresholding on all voxels, retaining voxels with an occupancy probability higher than a preset threshold, using the center coordinates of the selected voxels as the initial point cloud, fine-tuning the center coordinates using the PointNet deep learning network, and removing closely spaced duplicate points using maximum suppression (NMS); combining the fine-tuned spatial coordinates with the corresponding velocity values ​​obtained from the conditional tensor C in step S2 to form a dense point cloud.

Images