Cross-modal translation method and system for laser radar to 4d radar based on latent diffusion bridge

Abstract

This invention discloses a cross-modal translation method and system for lidar to 4D radar based on a latent diffusion bridge. The method includes: employing a key voxel-aware variational autoencoder (VAE) to perform logarithmic normalization preprocessing and key voxel mask recognition on high-dimensional noisy 4D radar data; utilizing a lidar VAE isomorphic to the 4D radar VAE, combined with a patch-wise contrastive learning strategy, to achieve semantic and spatial alignment between lidar and 4D radar in a unified latent space; modeling the cross-modal conversion task as a Brownian bridge random diffusion process, predicting the diffusion drift term using a 3D U-Net, and achieving the conversion from lidar latent vectors without Doppler information to complete 4D radar latent vectors under dual boundary constraints. This invention not only achieves the complete generation of 4D radar data in the native polar coordinate system for the first time, but also significantly improves target detection accuracy in autonomous driving scenarios through data augmentation, providing a novel technical solution for the construction of low-cost radar perception systems.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and autonomous driving perception technology, and in particular, it is a cross-modal translation method and system for lidar to 4D radar based on a potential diffusion bridge. Background Technology

[0002] In fields such as autonomous driving, intelligent transportation, and robotic perception, the stability and comprehensiveness of environmental perception systems directly determine the operational safety of equipment. Millimeter-wave radar, with its all-weather, all-time operating capabilities, possesses unique advantages over sensing technologies like lidar. It can provide 4D measurement data including distance, azimuth, elevation, and Doppler velocity, comprehensively encoding spatial structure and motion information, providing crucial support for target detection and tracking in complex environments. With the rapid development of autonomous driving technology, the demand for high-quality radar data is increasing daily. However, the high cost of millimeter-wave radar data acquisition severely restricts the training and optimization of related perception algorithms. Therefore, enhancing radar datasets through data generation technology has become a key direction for promoting the development of radar perception technology and has significant industry application value.

[0003] Existing radar data generation technologies are mainly divided into two categories: one is the traditional radar simulation method based on physical models, which generates radar data by simulating physical processes such as electromagnetic wave propagation and scattering, but suffers from high computational complexity and difficulty in capturing the complexity of real scenes; the other is the data-driven method based on deep learning, which generates data by learning the distribution characteristics of real radar data, and has the advantages of high efficiency and strong realism, and has become the mainstream research direction.

[0004] In deep learning-based radar data synthesis techniques, early research focused primarily on simplified 2D radar representations, such as range-Doppler (RD) maps and range-azimuth (RA) maps. For example, some schemes employ generative adversarial networks (GANs) based on convolutional neural networks (CNNs) to synthesize RD maps conditionally using target bounding boxes; others use recurrent consistency adversarial training to generate azimuth-height tensors from elevation maps; and still others utilize variational autoencoders (VAEs) to generate RA maps conditionally using a target list. To obtain richer spatial information, some research has begun exploring radar data generation based on lidar input, leveraging the spatial and semantic consistency between lidar and radar to achieve data augmentation. For instance, the L2RDaS scheme, based on conditional GANs, generates dense radar cubes in a 3D Cartesian coordinate system from lidar input; the L2RGAN scheme uses cGANs to convert lidar input to a 2D Cartesian radar tensor; and other schemes use voxelized feature extraction to predict sparse radar point clouds from lidar input.

[0005] In the research of conditional generation diffusion models, existing techniques have achieved high-quality generation of images, audio, and video by injecting conditions such as labels, text, and images into denoising networks. For cross-domain translation tasks, the diffusion bridge model has been proposed. By directly learning the diffusion process between two domains and using the source domain to guide forward and backward diffusion, it effectively alleviates the domain gap problem and has achieved good results in 2D image translation tasks such as medical imaging and remote sensing images.

[0006] Despite some progress in existing technologies, many unresolved issues remain in achieving cross-modal translation of complete 4D radar data using lidar, as follows:

[0007] (1) Technical performance defects: Existing deep learning methods mostly focus on the generation of simplified 2D or 3D radar representations, which cannot retain the complete spatial and motion information of 4D radar, resulting in serious information loss in the generated data, making it difficult to support downstream high-precision perception tasks; at the same time, 4D radar tensors have characteristics such as high dimension, large dynamic range of signal power (spanning multiple orders of magnitude), and serious background noise interference, making it difficult for existing models to accurately model their complex distribution, resulting in low fidelity of the generated data.

[0008] (2) Poor cross-modal adaptability: LiDAR provides sparse 3D point clouds, while 4D radar outputs dense 4D voxel tensors. There are huge differences between the two in terms of dimension and sparsity. Existing technologies cannot achieve effective alignment of semantic and spatial information between the two. In addition, LiDAR lacks the key Doppler velocity information of 4D radar, resulting in significant differences in feature distribution between the two. It is difficult to accurately infer the key features of radar from the spatial cues of LiDAR, and the cross-modal domain gap problem is prominent.

[0009] (3) Limited application scenarios: The radar data generated by the existing solution is incomplete and has low fidelity, which cannot meet the needs of high-precision perception scenarios such as autonomous driving. The performance improvement in downstream detection tasks after dataset augmentation is limited, and it is difficult to truly solve the industry pain point of radar data scarcity. Summary of the Invention

[0010] The purpose of this invention is to address the problems of existing technologies, such as the inability to generate complete 4D radar data, poor cross-modal alignment, low data fidelity, and limited downstream application performance. It provides a cross-modal translation method and system for LiDAR to 4D radar based on a potential diffusion bridge. This method enables the generation of 4D radar tensors that retain complete spatial and motion information, guided by LiDAR data, thereby improving data generation fidelity. Simultaneously, it effectively enhances the performance of downstream target detection tasks, reduces radar data acquisition costs, and promotes the development of autonomous driving perception technology.

[0011] The technical solution to achieve the objective of this invention is as follows: On the one hand, a cross-modal translation method for lidar to 4D radar based on a potential diffusion bridge is provided. This method achieves cross-modal conversion between lidar and 4D radar through a three-stage strategy of "compression-alignment-conversion," and includes the following:

[0012] A variational autoencoder (VAE) with key voxel perception is used to perform logarithmic normalization preprocessing and key voxel mask recognition on high-dimensional noisy 4D radar data, compressing it into a low-dimensional latent space and achieving accurate reconstruction of absolute signal power.

[0013] By utilizing a lidar VAE that is isomorphic to a 4D radar VAE and combining it with a patch-wise contrastive learning strategy, semantic and spatial alignment between lidar and 4D radar is achieved in a unified latent space to bridge the modal differences between the two in terms of dimensionality, sparsity, and information dimension.

[0014] The cross-modal conversion task is modeled as a Brownian bridge random diffusion process. The diffusion drift term is predicted by 3D U-Net, and the conversion from the lidar latent vector without Doppler information to the complete 4D lidar latent vector is realized under the constraint of double boundary conditions. After decoding and background noise synthesis, the output is a full-dimensional 4D lidar tensor containing range, azimuth, elevation and Doppler velocity.

[0015] Furthermore, the method includes the following steps:

[0016] Step 1: Preprocess the radar data;

[0017] Step 2: Construct and train a key voxel-sensing 4D radar (VAE);

[0018] Step 3: Perform potential spatial alignment on the lidar-4D radar;

[0019] Step 4: Perform cross-mode conversion between lidar and 4D radar based on the potential diffusion bridge to obtain the 4D radar potential vector;

[0020] Step 5: Reconstruct the 4D radar based on the 4D radar latent vector to generate the complete 4D radar tensor.

[0021] Furthermore, step 1 involves preprocessing the radar data, specifically including:

[0022] Regarding 4D radar data:

[0023] (1) Preprocess the raw 4D radar data;

[0024] A constant false alarm rate (CFAR) detector is used to identify key target voxels on a range-azimuth-elevation spatial grid and generate a binary key voxel mask. ;

[0025] For key voxels, retain the original Doppler power vector; for background voxels, calculate the power mean on the Doppler axis.

[0026] Represent 4D radar data as triplets Where R is the mean voxel Doppler power matrix, is the key voxel Doppler tensor, where the background voxel padding is 0;

[0027] (2) For R and Perform logarithmic normalization and z-score normalization;

[0028] Regarding lidar data:

[0029] Normalize the lidar data.

[0030] Furthermore, step 2 involves constructing and training a key voxel-sensing 4D radar VAE, specifically including:

[0031] A key voxel-sensing VAE model is constructed, including an encoder (EncR) and a decoder (DecR). A 3D ResNet block model is used to model the spatial structure, and a self-attention layer is inserted to capture long-range dependencies. Preprocessed 4D radar data is stitched along the Doppler axis to form the VAE input, which is then processed by the encoder. Encode 4D radar data into low-dimensional potential vectors Its posterior distribution is N represents the posterior distribution. These are low-dimensional latent vectors. The mean vector and standard deviation vector; the decoder The key voxel mask, the normalized Doppler mean matrix and the key voxel Doppler tensor are reconstructed by three output heads respectively, and the absolute signal power value is reconstructed by inverse normalization.

[0032] The training objective of VAE is to minimize the comprehensive loss function, as shown in the following formula:

[0033]

[0034] In the formula, This is the total training loss of the 4D radar VAE. Represents the latent space variables for 4D radar Seeking expectations, This is the reconstruction loss term for the 4D radar, where β is the KL divergence weight. (P|Q) represents the KL divergence, which measures the difference between two distributions, P and Q. P and Q are only used to refer to variables and have no actual meaning. It is the posterior probability distribution of the encoder. Let I represent the standard normal prior distribution, and let I denote the identity matrix. The formula for reconstruction loss is as follows:

[0035]

[0036]

[0037]

[0038] in, The coefficients are used to balance the weights of the binary cross-entropy loss (BCE) and the L2 and L1 values ​​of the denormalized absolute power values, which are the weights of each reconstruction loss. This represents the multiplication operation used to adjust the weighting of the BCE loss. Indicates L2 loss, Indicates L1 loss, Represents the true key voxel mask. The key voxel mask representing the reconstruction. This represents the binary cross-entropy loss between a and b. It is used only to refer to variables and has no actual meaning. This represents the Doppler mean matrix of a real 4D radar. This represents the reconstructed original scale Doppler mean matrix. This represents the normalized Doppler mean matrix. Represents the reconstructed normalized Doppler mean matrix. The Doppler vector representing the true key voxel. This represents the normalized key voxel Doppler vector. This represents the original scale key voxel Doppler vector reconstructed. represents the reconstructed normalized key voxel Doppler vector, and ⊙ represents element-wise multiplication.

[0039] Furthermore, step 3, which involves aligning the potential space of the lidar and the 4D radar, includes: training a lidar VAE with the same structure as the 4D radar VAE, and combining this with a patch-wise contrastive learning strategy to achieve semantic and spatial alignment of the potential spaces of the lidar and the 4D radar.

[0040] Furthermore, step 3, which involves performing potential spatial alignment of the lidar-4D radar, specifically includes:

[0041] Step 3-1: Construct a lidar VAE with the same structure as the 4D radar VAE, including the encoder. and decoder The input channel is set to 1;

[0042] Step 3-2: After performing min-max normalization on the sparse 3D LiDAR data, the data is input into the LiDAR VAE and processed by the encoder. Generation and 4D radar potential vectors LiDAR latent vectors with completely identical dimensions and shapes Its posterior distribution is N represents the posterior distribution. These are low-dimensional latent vectors. The mean vector and standard deviation vector;

[0043] Step 3-3, the goal of LiDAR VAE training is to minimize VAE loss. The reconstruction loss is calculated using the L2 distance between the input and the reconstruction result;

[0044] Semantic and spatial alignment is achieved through a patch-wise contrastive learning strategy, specifically including: for paired LiDAR-4D radar samples, latent encoding of LiDARs at the same spatial location. With 4D radar potential coding As positive sample pairs, the encodings of different locations or different samples are used as negative sample pairs; the final training loss of the LiDAR VAE is:

[0045]

[0046] in,

[0047]

[0048] in, This represents a scalar value indicating the total training loss of the LiDAR VAE. This represents the fundamental loss scalar of the lidar VAE. To compare the loss weights, Let be the scalar representing the cross-modal contrastive learning loss value, sim(·,·) be the cosine similarity function, N represent the number of samples in the training batch, and M represent the number of latent patches for each sample. This represents the latent patch feature of the i-th lidar sample at the j-th spatial location. This represents the latent patch feature of the i-th 4D radar sample at the j-th spatial location. This represents the latent patch feature of the l-th spatial location of the k-th 4D radar sample, where τ is the temperature hyperparameter. Represents the exponential transformation, log( ) represents a logarithmic transformation.

[0049] Furthermore, step 4, which describes cross-modal conversion based on a potential diffusion bridge, specifically includes:

[0050] In the aligned latent space, the translation from lidar to 4D radar is modeled as a Brownian bridge stochastic process;

[0051] Step 4-1: For paired samples (I,R), use the pre-trained and fixed encoders of the LiDAR VAE and 4D radar VAE. and Generate respectively and Constructing a random interpolation process ,satisfy:

[0052]

[0053] Where I and R are the LiDAR and 4D radar samples, respectively, ε ~ N(0,I), and t∈[0,1]. For the variance term, realize the variance from t=1. up to t=0 The smooth transition, whose time evolution satisfies the stochastic differential equation, i.e., SDE:

[0054]

[0055] In the formula, For standard Brownian motion, This represents the change in latent encoding at time t. This indicates the global noise intensity.

[0056] Step 4-2: Implement the drift term prediction network using 3D U-Net. Furthermore, the network incorporates sinusoidal time embeddings and adapts to different diffusion time steps t through AdaLN layers; the training objective is to minimize the mean square error of the drift term prediction.

[0057]

[0058] In the formula, This represents the drift prediction loss of the diffusion bridge. This represents the joint expectation of the lidar latent vector and the 4D radar latent vector at time t. This indicates that the 3D U-shaped network model has parameters The predicted drift term below;

[0059] During the inference phase, non-Markov sampling is used to accelerate generation, given a time-step sequence. Finally, the 4D radar potential vector was obtained. The estimated value , ∈[0,1].

[0060] Furthermore, step 5, which involves reconstructing the 4D radar based on the 4D radar latent vector to generate a complete 4D radar tensor, specifically includes:

[0061] Step 5-1, use the 4D radar potential vector obtained in step 4. Input pre-trained 4D radar VAE decoder The reconstructed normalized radar data was obtained. The absolute signal power value is obtained by inverse normalization. ;

[0062] Step 5-2, Fill in key voxels Noise is generated for the background voxels while keeping the total power constant, and finally the complete 4D radar tensor is obtained.

[0063] Furthermore, in step 5-2, noise is generated on the background voxels using a Gaussian-softmax distribution.

[0064] On the other hand, a cross-modal translation system for lidar to 4D radar is provided, the system comprising:

[0065] The first module is used to preprocess radar data;

[0066] The second module is used to: construct and train a key voxel-sensing 4D radar (VAE);

[0067] The third module is used to perform potential spatial alignment of the lidar-4D radar.

[0068] The fourth module is used to achieve cross-mode conversion between lidar and 4D radar based on the potential diffusion bridge, and to obtain the potential vector of 4D radar.

[0069] The fifth module is used to realize: 4D radar reconstruction based on 4D radar latent vectors, generating a complete 4D radar tensor.

[0070] Compared with the prior art, the significant advantages of this invention are:

[0071] (1) The key technologies adopted in this invention, such as key voxel perception coding, cross-modal potential alignment and diffusion bridge translation, not only realize the complete generation of 4D radar data in the native polar coordinate system for the first time, but also significantly improve the target detection accuracy in autonomous driving scenarios through data augmentation, providing a new technical solution for the construction of low-cost radar perception systems.

[0072] (2) This invention utilizes a key voxel-aware variational autoencoder (VAE) to logarithmically normalize the 4D radar tensor, separate key target voxels from background clutter, design a multi-output head reconstruction (binary key voxel mask, Doppler mean matrix, key voxel Doppler vector), and introduce L1 loss to reconstruct the absolute signal power value. This achieves efficient compression and accurate reconstruction of high-dimensional, high-dynamic-range, and noisy 4D radar data, while also realizing accurate identification of key voxels, laying a high-quality data foundation for subsequent cross-modal translation.

[0073] (3) This invention uses a patch-based contrastive learning module to encode LiDAR data into a unified latent space that is semantically and spatially aligned with radar. By constructing positive sample pairs of latent codes for LiDAR and radar at the same spatial location and negative sample pairs of latent codes at different locations or samples, the distribution concentration is adjusted using cosine similarity and temperature parameters. This effectively bridges the differences in dimensionality, sparsity and information between sparse 3D LiDAR and dense 4D radar, and achieves semantic and spatial consistency alignment between the two modal latent spaces.

[0074] (4) This invention constructs a Brownian bridge diffusion process in an aligned latent space, models the LiDAR-to-4DRadar translation as a diffusion bridge problem with double boundary conditions, uses a 3D U-Net network to predict the drift term of the diffusion process, combines sinusoidal time embedding and AdaLN to adapt to different diffusion time steps, and adopts the DDIM accelerated sampling strategy to successfully synthesize a complete 4D radar tensor from LiDAR input lacking Doppler information, while ensuring the spatial structure and motion information integrity of the synthesized data.

[0075] (5) This invention uses a three-stage framework of “radar VAE compression - cross-modal latent space alignment - diffusion bridge translation”, combined with multi-weather scene training of K-Radar dataset and extraction of key voxel masks by CFAR algorithm, so that the synthesized 4D radar data is superior to the existing baseline model in XYZ space, RAE space and key voxel related indicators (MAE, PSNR, SSIM, IoU), achieving high-fidelity 4D radar generation effect. In scenarios where only synthetic data or synthetic data is used to enhance real data, it significantly improves the performance of downstream 3D target detection tasks, especially with good adaptability and robustness under various weather conditions.

[0076] (6) By maintaining the original polar coordinate representation of LiDAR and 4D radar, this invention avoids spatial distortion and deviation caused by coordinate transformation and voxelization. Combined with Gaussian-softmax distribution modeling of background clutter, the synthesized 4D radar data not only closely matches the real data under normal weather conditions, but also retains the spatial structure and velocity-sensitive area of ​​key targets in low visibility scenarios such as cloudy, rainy, and snowy days, thus improving the authenticity and practicality of data generation.

[0077] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0078] Figure 1 This is a flowchart illustrating the principle of a cross-modal translation method for lidar to 4D radar based on a potential diffusion bridge in one embodiment. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0080] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0081] In one embodiment, combined Figure 1 This paper presents a cross-modal translation method for lidar to 4D radar based on a potential diffusion bridge, belonging to the field of autonomous driving perception and cross-modal data generation technology. It solves the technical challenges of high data acquisition costs for millimeter-wave radar, incomplete information in existing radar generation schemes, and large gaps between cross-modal domains. This method achieves accurate generation of high-fidelity 4D radar tensors and effectively improves the performance of downstream target detection tasks. The method realizes cross-modal conversion from lidar to 4D radar through a three-stage strategy of "compression-alignment-conversion," including the following:

[0082] A variational autoencoder (VAE) with key voxel perception is used to perform logarithmic normalization preprocessing and key voxel mask recognition on high-dimensional noisy 4D radar data, compressing it into a low-dimensional latent space and achieving accurate reconstruction of absolute signal power.

[0083] By utilizing a lidar VAE that is isomorphic to a 4D radar VAE and combining it with a patch-wise contrastive learning strategy, semantic and spatial alignment between lidar and 4D radar is achieved in a unified latent space to bridge the modal differences between the two in terms of dimensionality, sparsity, and information dimension.

[0084] The cross-modal conversion task is modeled as a Brownian bridge random diffusion process. The diffusion drift term is predicted by 3D U-Net, and the conversion from the lidar latent vector without Doppler information to the complete 4D lidar latent vector is realized under the constraint of double boundary conditions. After decoding and background noise synthesis, the output is a full-dimensional 4D lidar tensor containing range, azimuth, elevation and Doppler velocity.

[0085] Furthermore, in one embodiment, the method includes the following steps:

[0086] Step 1: Preprocess the radar data;

[0087] Step 2: Construct and train a key voxel-sensing 4D radar (VAE);

[0088] Step 3: Perform potential spatial alignment on the lidar-4D radar;

[0089] Step 4: Perform cross-mode conversion between lidar and 4D radar based on the potential diffusion bridge to obtain the 4D radar potential vector;

[0090] Step 5: Reconstruct the 4D radar based on the 4D radar latent vector to generate the complete 4D radar tensor.

[0091] Preferably, in some embodiments, step 1 involves preprocessing the radar data, specifically including:

[0092] Regarding 4D radar data:

[0093] (1) Preprocess the raw 4D radar data;

[0094] A constant false alarm rate (CFAR) detector is used to identify key target voxels on a range-azimuth-elevation spatial grid and generate a binary key voxel mask. ;

[0095] For key voxels, retain the original Doppler power vector; for background voxels, calculate the power mean on the Doppler axis.

[0096] Represent 4D radar data as triplets Where R is the mean voxel Doppler power matrix, is the key voxel Doppler tensor, where the background voxel padding is 0;

[0097] (2) For R and Perform logarithmic normalization and z-score normalization;

[0098] Regarding LiDAR data:

[0099] Normalize the lidar data.

[0100] Specifically, this includes: rasterizing the LiDAR point cloud to a polar coordinate grid consistent with that of the 4D radar to ensure spatial alignment; if there is a LiDAR reflection point in the grid, its reflection intensity (p) is taken as the grid value; if it is an empty grid without a reflection point, it is uniformly marked as -1 to clearly distinguish between effective signals and blank areas.

[0101] Then, the intensity information is normalized by min-max, mapping the reflection intensity values ​​of all valid grids to a fixed range (usually [0, 1]) to eliminate the intensity value differences caused by different scenes and different targets. Empty voxels are marked as -1 to unify the data format.

[0102] Preferably, in some embodiments, step 2, which involves constructing and training a key voxel-sensing 4D radar VAE, specifically includes:

[0103] A key voxel-sensing VAE model is constructed, including an encoder (EncR) and a decoder (DecR). A 3D ResNet block model is used to model the spatial structure, and a self-attention layer is inserted to capture long-range dependencies. Preprocessed 4D radar data is stitched along the Doppler axis to form the VAE input, which is then processed by the encoder. Encode 4D radar data into low-dimensional potential vectors Its posterior distribution is N represents the posterior distribution. These are low-dimensional latent vectors. The mean vector and standard deviation vector; the decoder The key voxel mask, the normalized Doppler mean matrix and the key voxel Doppler tensor are reconstructed by three output heads respectively, and the absolute signal power value is reconstructed by inverse normalization.

[0104] The training objective of VAE is to minimize the comprehensive loss function, as shown in the following formula:

[0105]

[0106] In the formula, This is the total training loss of the 4D radar VAE. Represents the latent space variables for 4D radar Seeking expectations, This is the reconstruction loss term for the 4D radar, where β is the KL divergence weight. (P|Q) represents the KL divergence, which measures the difference between two distributions, P and Q. P and Q are only used to refer to variables and have no actual meaning. It is the posterior probability distribution of the encoder. Let I represent the standard normal prior distribution, and let I denote the identity matrix. The formula for reconstruction loss is as follows:

[0107]

[0108]

[0109]

[0110] in, The coefficients are used to balance the weights of the binary cross-entropy loss (BCE) and the L2 and L1 values ​​of the denormalized absolute power values, which are the weights of each reconstruction loss. This represents the multiplication operation used to adjust the weighting of the BCE loss. Indicates L2 loss, Indicates L1 loss, Represents the true key voxel mask. The key voxel mask representing the reconstruction. This represents the binary cross-entropy loss between a and b. It is used only to refer to variables and has no actual meaning. This represents the Doppler mean matrix of a real 4D radar. This represents the reconstructed original scale Doppler mean matrix. This represents the normalized Doppler mean matrix. Represents the reconstructed normalized Doppler mean matrix. The Doppler vector representing the true key voxel. This represents the normalized key voxel Doppler vector. This represents the original scale key voxel Doppler vector reconstructed. represents the reconstructed normalized key voxel Doppler vector, and ⊙ represents element-wise multiplication.

[0111] Preferably, in some embodiments, step 3, which involves aligning the potential space of the lidar-4D radar, includes: training a lidar VAE with the same structure as the 4D radar VAE, and combining this with a patch-wise contrastive learning strategy to achieve semantic and spatial alignment of the potential spaces of the lidar and the 4D radar.

[0112] Preferably, in some embodiments, step 3, which involves performing potential spatial alignment of the lidar-4D radar, specifically includes:

[0113] Step 3-1: Construct a lidar VAE with the same structure as the 4D radar VAE, including the encoder. and decoder The input channel is set to 1;

[0114] Step 3-2: After performing min-max normalization on the sparse 3D LiDAR data (empty voxels are marked as -1), the data is input into the LiDAR VAE and passed through the encoder. Generation and 4D radar potential vectors LiDAR latent vectors with completely identical dimensions and shapes Its posterior distribution is N represents the posterior distribution. These are low-dimensional latent vectors. The mean vector and standard deviation vector;

[0115] Step 3-3, the goal of LiDAR VAE training is also to minimize the VAE loss similar to that in Step 2. The reconstruction loss is calculated using the L2 distance between the input and the reconstruction result;

[0116] Semantic and spatial alignment is achieved through a patch-wise contrastive learning strategy, specifically including: for paired LiDAR-4D radar samples, latent encoding of LiDARs at the same spatial location. With 4D radar potential coding As positive sample pairs, the encodings of different locations or different samples are used as negative sample pairs; the final training loss of the LiDAR VAE is:

[0117]

[0118] in,

[0119]

[0120] in, This represents a scalar value indicating the total training loss of the LiDAR VAE. This represents the fundamental loss scalar of the lidar VAE. To compare the loss weights, Let be the scalar representing the cross-modal contrastive learning loss value, sim(·,·) be the cosine similarity function, N represent the number of samples in the training batch, and M represent the number of latent patches for each sample. This represents the latent patch feature of the j-th spatial location of the i-th lidar sample. This represents the latent patch feature of the i-th 4D radar sample at the j-th spatial location. This represents the latent patch feature of the l-th spatial location of the k-th 4D radar sample, where τ is the temperature hyperparameter. Represents the exponential transformation, log( ) represents a logarithmic transformation.

[0121] Preferably, in some embodiments, step 4, which involves cross-modal conversion based on a potential diffusion bridge, specifically includes:

[0122] In the aligned latent space, the translation from lidar to 4D radar is modeled as a Brownian bridge stochastic process;

[0123] Step 4-1: For paired samples (I,R), use the pre-trained and fixed encoders of the LiDAR VAE and 4D radar VAE. and Generate respectively and Constructing a random interpolation process ,satisfy:

[0124]

[0125] Where I and R are the LiDAR and 4D radar samples, respectively, ε ~ N(0,I), and t∈[0,1]. For the variance term, realize the variance from t=1. up to t=0 The smooth transition, whose time evolution satisfies the stochastic differential equation, i.e., SDE:

[0126]

[0127] In the formula, For standard Brownian motion, This represents the change in latent encoding at time t. This indicates the global noise intensity.

[0128] Step 4-2: Implement the drift term prediction network using 3D U-Net. Furthermore, the network incorporates sinusoidal time embeddings and adapts to different diffusion time steps t through AdaLN layers; the training objective is to minimize the mean square error of the drift term prediction.

[0129]

[0130] In the formula, This represents the drift prediction loss of the diffusion bridge. This represents the joint expectation of the lidar latent vector and the 4D radar latent vector at time t. This indicates that the 3D U-shaped network model has parameters The predicted drift term below;

[0131] During the inference phase, non-Markov sampling is used to accelerate generation, given a time-step sequence. Finally, the 4D radar potential vector was obtained. The estimated value , ∈[0,1].

[0132] Preferably, in some embodiments, step 5, which involves reconstructing the 4D radar based on the 4D radar latent vector to generate a complete 4D radar tensor, specifically includes:

[0133] Step 5-1, use the 4D radar potential vector obtained in step 4. Input pre-trained 4D radar VAE decoder The reconstructed normalized radar data was obtained. The absolute signal power value is obtained by inverse normalization. ;

[0134] Step 5-2, Fill in key voxels Noise is generated for the background voxels while keeping the total power constant, and finally the complete 4D radar tensor is obtained.

[0135] Preferably, in step 5-2, noise is generated on the background voxels using a Gaussian-softmax distribution.

[0136] In one embodiment, a cross-modal translation system from lidar to 4D radar is provided, the system comprising:

[0137] The first module is used to preprocess radar data;

[0138] The second module is used to: construct and train a key voxel-sensing 4D radar (VAE);

[0139] The third module is used to perform potential spatial alignment of the lidar-4D radar.

[0140] The fourth module is used to achieve cross-mode conversion between lidar and 4D radar based on the potential diffusion bridge, and to obtain the potential vector of 4D radar.

[0141] The fifth module is used to realize: 4D radar reconstruction based on 4D radar latent vectors, generating a complete 4D radar tensor.

[0142] Specific limitations regarding the cross-modal translation system from lidar to 4D radar can be found in the limitations of the cross-modal translation method from lidar to 4D radar mentioned above, and will not be repeated here. Each module in the aforementioned cross-modal translation system from lidar to 4D radar can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0143] Preferably, in some embodiments, the system includes: a data preprocessing module, a key voxel sensing 4D radar VAE module, a lidar-4D radar potential alignment module, a potential diffusion bridge translation module, and a radar data reconstruction module. The connection relationships and functions of each module are as follows:

[0144] ① Data Preprocessing Module: Connected to the key voxel-sensing 4D radar VAE module and the lidar-4D radar potential alignment module, this module receives raw 4D radar and lidar data, performs CFAR detection, re-representation, and normalization on the radar data, normalizes the lidar data, and outputs preprocessed radar triples and normalized lidar data. The hardware implementation is a processor, and the software implementation is based on the Python programming language, using the NumPy library for data processing.

[0145] ② Key Voxel Sensing 4D Radar VAE Module: Connected to the data preprocessing module, potential diffusion bridge translation module, and radar data reconstruction module, including an encoder. and decoder The encoder employs a 3D ResNet block + self-attention layer structure, while the decoder uses a 3D ResNet block + self-attention layer structure with three output heads. This structure encodes the preprocessed radar data into low-dimensional latent vectors and reconstructs the radar data from the latent vectors. The hardware implementation is a GPU (e.g., NVIDIA A800), and the software implementation is based on the PyTorch framework, using 3D convolutional layers and self-attention layers to construct the network. Key parameters: downsampling factor l=4, β=1.0. =0.5.

[0146] ③ LiDAR-4D Radar Potential Alignment Module: Connected to the data preprocessing module and the potential diffusion bridge translation module, including the LiDAR VAE (… , The contrastive learning unit (VAE) is used to calculate the InfoNCE loss between the LiDAR and radar latent codes. The LiDAR VAE structure is consistent with the radar VAE. The hardware implementation is based on a GPU, and the software implementation is based on the PyTorch framework. Key parameters: =0.3, τ=0.1.

[0147] ④ Latent Diffusion Bridge Conversion Module: Connected to the LiDAR-4D radar latent alignment module and radar data reconstruction module, it includes a 3D U-Net drift prediction network and a diffusion sampling unit. The 3D U-Net adopts an encoder-decoder structure, containing jumper connections and a temporal embedding layer. The diffusion sampling unit implements SDE-based reverse iterative sampling. The hardware implementation is based on a GPU, and the software implementation is based on the PyTorch framework. Key parameters: number of diffusion steps = 1000, σ = 1.0, δ = 2.0.

[0148] ⑤ Radar Data Reconstruction Module: Connected to the key voxel-sensing 4D radar VAE module and the potential diffusion bridge translation module, this module receives the reconstructed data output from the decoder, performs inverse normalization, generates noise for the background voxels, and outputs the complete 4D radar tensor. The hardware implementation is a processor, and the software implementation is based on the Python programming language, using the NumPy library for numerical calculations.

[0149] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements:

[0150] Step 1: Preprocess the radar data;

[0151] Step 2: Construct and train a key voxel-sensing 4D radar (VAE);

[0152] Step 3: Perform potential spatial alignment on the lidar-4D radar;

[0153] Step 4: Perform cross-mode conversion between lidar and 4D radar based on the potential diffusion bridge to obtain the 4D radar potential vector;

[0154] Step 5: Reconstruct the 4D radar based on the 4D radar latent vector to generate the complete 4D radar tensor.

[0155] For specific limitations on each step, please refer to the limitations on the cross-modal translation method from lidar to 4D radar based on potential diffusion bridges mentioned above, which will not be repeated here.

[0156] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0157] Step 1: Preprocess the radar data;

[0158] Step 2: Construct and train a key voxel-sensing 4D radar (VAE);

[0159] Step 3: Perform potential spatial alignment on the lidar-4D radar;

[0160] Step 4: Perform cross-mode conversion between lidar and 4D radar based on the potential diffusion bridge to obtain the 4D radar potential vector;

[0161] Step 5: Reconstruct the 4D radar based on the 4D radar latent vector to generate the complete 4D radar tensor.

[0162] For specific limitations on each step, please refer to the limitations on the cross-modal translation method from lidar to 4D radar based on potential diffusion bridges mentioned above, which will not be repeated here.

[0163] As a specific example, the invention will be further verified and illustrated in one embodiment.

[0164] Regarding the implementation environment, this embodiment provides the configuration information for the experiment, as follows:

[0165] (1) Dataset: The K-Radar dataset was used as the training and testing data. This dataset contains 35K pairs of 4D radar and lidar paired samples, covering various weather conditions. The 4D radar data is a tensor in the native polar coordinate system with a size of (63,192,96,32), corresponding to Doppler (-2~2 m / s), range (0~88.4 m), azimuth (-48~48°), and elevation (-16~16°). The lidar point cloud was rasterized onto the same polar coordinate grid as the radar to achieve spatial alignment. Data preprocessing: The radar data was processed using the CFAR algorithm (false alarm rate 0.1) to generate key voxel masks, with key voxels accounting for approximately 1.5%. The radar mean matrix and key voxel Doppler tensors were logarithmically normalized and z-score normalized. The lidar intensity was normalized using min-max normalization, and empty voxels were marked as -1.

[0166] (2) Hardware environment: Training and inference are based on two NVIDIA A800 GPUs, with an Intel Xeon Gold 6330 CPU, 256GB of memory, and 1TB SSD storage capacity.

[0167] (3) Software environment: The operating system is Ubuntu 20.04 LTS, the programming language is Python 3.8, the deep learning framework is PyTorch 1.13.1, and the dependent libraries include NumPy 1.24.3, SciPy 1.10.1, Matplotlib 3.7.1. The Adam optimizer is used for model training.

[0168] In addition to the configuration of the implementation environment mentioned above, the following is the specific operational process from data preparation and use to evaluation and testing:

[0169] (1) Data preparation stage: The K-Radar dataset is divided into training set, extended set and test set in a ratio of 4:2:1; CFAR detection is performed on the radar data in the training set to generate key voxel masks. Calculate the mean Doppler power R for each voxel and construct a radar triplet. For the triplet and Logarithmic normalization and z-score normalization are performed; the lidar data are normalized to min-max, and the intensity values ​​are mapped to the [0,1] interval, with empty voxels filled with -1.

[0170] (2) Model training phase:

[0171] ① Training the key voxel-sensing 4D radar VAE: The preprocessed radar triples are concatenated along the Doppler axis to form an input tensor of (D+1)×R×A×E, which is then input into the radar VAE; the batch size is set to 32, the learning rate is 0.0001, a warm-up strategy is adopted (the learning rate is linearly increased in the first 1000 steps), and the number of training rounds is 200; the loss function is the one described in step 2. , where β=1.0, =0.5; Validate every 10 rounds during training and save the model weights with the minimum reconstruction loss on the validation set.

[0172] ② Training the LiDAR-4D radar latent alignment module: Fix the weights of the radar VAE, and input the normalized LiDAR data into the LiDAR VAE; set the batch size to 32, the learning rate to 0.0001, and the number of training epochs to 150; the loss function is... ,in =0.3, τ=0.1; In contrastive learning, the latent encoding of each sample is divided into multiple patches according to spatial location. Positive sample pairs are the LiDAR and radar patch encodings at the same spatial location, and negative sample pairs are the patch encodings of different samples or different locations; After training, the weights of the LiDAR VAE are saved.

[0173] ③ Training the potential diffusion bridge model: Using pre-trained and fixed radar VAEs and lidar VAEs, the paired samples (I,R) in the training set are encoded to generate... and Construct a 3D U-Net drift prediction network, with the input being... The sinusoidal embedding of time step t outputs a drift term. The diffusion steps were set to 1000, noise scheduling was linear (σ varies linearly from 0 to 1), batch size was 32, learning rate was 0.0001, and training epochs were 250; the loss function was... The model weights are validated every 20 rounds during training, and the model weights with the best performance are saved.

[0174] (3) Cross-modal translation stage: Load all trained module weights; input the LiDAR data from the test set into the LiDAR VAE to generate latent vectors. Set the time step sequence length of the diffusion sampling to S=100, and the time steps t to be uniformly distributed from 1 to 0; from =1= We begin by predicting drift terms using 3D U-Net. The following formula is used for iterative updates to obtain... =s-1:

[0175]

[0176] Here, λ is taken from the standard normal distribution, and δ controls the noise level. Setting δ = 0 generates a non-Markovian deterministic sampler.

[0177] (4) Radar data reconstruction stage: Input the radar VAE decoder to obtain the reconstructed image. The absolute signal power value is obtained by inverse normalization. For each voxel, if Then fill The Doppler vector; if Then the generation obeys Gaussian noise, normalized by softmax and then multiplied by The total power remains unchanged; the final output is a complete 4D radar tensor. .

[0178] (5) Validation phase: First, a control group was set up, and existing mainstream schemes were selected as the baseline for comparison, including: ① L2RDaS (3D XYZ space, no Doppler information); ② Pix2PixHD RAE (3D RAE space, no Doppler information); ③ Latent Pix2PixHD (4D RAE+Doppler, latent spatial adversarial translation); ④ Latent Diffusion (4D RAE+Doppler, latent spatial conditional diffusion); Two ablation models were also set up: ⑤ w / o L1 (removing L1 reconstruction loss of radar VAE); ⑥ w / o Lcont (removing contrastive learning loss). Evaluation metrics included: MAE, PSNR, and SSIM in XYZ and RAE spaces, MAE, PSNR, and mask IoU of key voxels; APBEV and AP3D (IoU thresholds of 0.3 and 0.5) were used for downstream detection tasks.

[0179] (6) Testing phase: Based on the verification method in phase 5, the performance comparison data on the test set for the synthetic performance testing requirements are shown in Table 1 below:

[0180] Table 1 Performance Comparison Data

[0181]

[0182] As shown in Table 1, the present invention (L2RLDB) outperforms the baseline scheme and ablation model in all metrics. In the XYZ space, the PSNR reaches 33.01, the SSIM reaches 0.9092, and the MAE is as low as 0.0525; in the RAE space, the PSNR reaches 29.21, the SSIM reaches 0.7324; the key voxel mask IoU reaches 0.5452, and the Doppler vector MAE reaches 23.44, all of which are significantly better than existing schemes.

[0183] For the downstream detection performance testing, detection performance was evaluated under two training settings: one using only synthetic data for training, and the other using real data plus synthetic data augmentation for training. The results show that, when trained with only synthetic data, the synthetic data generated in this invention outperforms all baseline schemes in both APBEV and AP3D, approaching the performance of training with real data. Under the data augmentation setting, compared to training with only real data, detection performance is significantly improved, and performance gains are achieved under most weather conditions, verifying the effectiveness of the synthetic data.

[0184] For parameter sensitivity analysis, the key hyperparameters σ (noise intensity) and τ (contrast learning temperature) of the diffusion model were analyzed, and the model performance was tested when σ was 0.5, 1.0, 1.5, and 2.0, and τ was 0.05, 0.1, 0.2, and 0.3. The results show that when σ=1.0 and τ=0.1, the model achieves optimal performance in PSNR and SSIM, with minimal performance fluctuations, demonstrating the good stability and robustness of the proposed solution.

[0185] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A cross-modal translation method for lidar to 4D radar based on potential diffusion bridging, characterized in that, The method achieves cross-modal conversion between lidar and 4D radar through a three-stage strategy of "compression-alignment-conversion", including the following: A variational autoencoder (VAE) with key voxel perception is used to perform logarithmic normalization preprocessing and key voxel mask recognition on high-dimensional noisy 4D radar data, compressing it into a low-dimensional latent space and achieving accurate reconstruction of absolute signal power. By utilizing a lidar VAE that is isomorphic to a 4D radar VAE and combining it with a patch-wise contrastive learning strategy, semantic and spatial alignment between lidar and 4D radar is achieved in a unified latent space to bridge the modal differences between the two in terms of dimensionality, sparsity, and information dimension. The cross-modal conversion task is modeled as a Brownian bridge random diffusion process. The diffusion drift term is predicted by 3D U-Net, and the conversion from the lidar latent vector without Doppler information to the complete 4D lidar latent vector is realized under the constraint of double boundary conditions. After decoding and background noise synthesis, the output is a full-dimensional 4D lidar tensor containing range, azimuth, elevation and Doppler velocity.

2. The cross-modal translation method for lidar to 4D radar based on a potential diffusion bridge according to claim 1, characterized in that, The method includes the following steps: Step 1: Preprocess the radar data; Step 2: Construct and train a key voxel-sensing 4D radar (VAE); Step 3: Perform potential spatial alignment on the lidar-4D radar; Step 4: Perform cross-mode conversion between lidar and 4D radar based on the potential diffusion bridge to obtain the 4D radar potential vector; Step 5: Reconstruct the 4D radar based on the 4D radar latent vector to generate the complete 4D radar tensor.

3. The cross-modal translation method for lidar to 4D radar based on a potential diffusion bridge according to claim 2, characterized in that, Step 1 involves preprocessing the radar data, specifically including: Regarding 4D radar data: (1) Preprocess the raw 4D radar data; A constant false alarm rate (CFAR) detector is used to identify key target voxels on a range-azimuth-elevation spatial grid and generate a binary key voxel mask. ; For key voxels, retain the original Doppler power vector; for background voxels, calculate the power mean on the Doppler axis. Represent 4D radar data as triplets Where R is the mean voxel Doppler power matrix, is the key voxel Doppler tensor, where the background voxel padding is 0; (2) For R and Perform logarithmic normalization and z-score normalization; Regarding lidar data: Normalize the lidar data.

4. The cross-modal translation method for lidar to 4D radar based on a potential diffusion bridge according to claim 3, characterized in that, Step 2 involves constructing and training a key voxel-sensing 4D radar VAE, specifically including: A key voxel-sensing VAE model is constructed, including an encoder (EncR) and a decoder (DecR). A 3D ResNet block model is used to model the spatial structure, and a self-attention layer is inserted to capture long-range dependencies. Preprocessed 4D radar data is stitched along the Doppler axis to form the VAE input, which is then processed by the encoder. Encode 4D radar data into low-dimensional potential vectors Its posterior distribution is N represents the posterior distribution. These are low-dimensional latent vectors. The mean vector and standard deviation vector; the decoder The key voxel mask, the normalized Doppler mean matrix and the key voxel Doppler tensor are reconstructed by three output heads respectively, and the absolute signal power value is reconstructed by inverse normalization. The training objective of VAE is to minimize the comprehensive loss function, as shown in the following formula: In the formula, This is the total training loss of the 4D radar VAE. Represents the latent space variables for 4D radar Seeking expectations, This is the reconstruction loss term for the 4D radar, where β is the KL divergence weight. (P|Q) represents the KL divergence, which measures the difference between two distributions, P and Q. P and Q are only used to refer to variables and have no actual meaning. It is the posterior probability distribution of the encoder. Let I represent the standard normal prior distribution, and let I denote the identity matrix. The formula for reconstruction loss is as follows: in, The coefficients are used to balance the weights of the binary cross-entropy loss (BCE) and the L2 and L1 values ​​of the denormalized absolute power values, which are the weights of each reconstruction loss. This represents the multiplication operation used to adjust the weighting of the BCE loss. Indicates L2 loss, Indicates L1 loss, Represents the true key voxel mask. The key voxel mask representing the reconstruction. This represents the binary cross-entropy loss between a and b. It is used only to refer to variables and has no actual meaning. This represents the Doppler mean matrix of a real 4D radar. This represents the reconstructed original scale Doppler mean matrix. This represents the normalized Doppler mean matrix. Represents the reconstructed normalized Doppler mean matrix. The Doppler vector representing the true key voxel. This represents the normalized key voxel Doppler vector. This represents the original scale key voxel Doppler vector reconstructed. represents the reconstructed normalized key voxel Doppler vector, and ⊙ represents element-wise multiplication.

5. The cross-modal translation method for lidar to 4D radar based on a potential diffusion bridge according to claim 4, characterized in that, Step 3, which involves aligning the potential space of the lidar with that of the 4D radar, includes: training a lidar VAE with the same structure as the 4D radar VAE, and combining this with a patch-wise contrastive learning strategy to achieve semantic and spatial alignment of the potential spaces of the lidar and the 4D radar.

6. The cross-modal translation method for lidar to 4D radar based on a potential diffusion bridge according to claim 5, characterized in that, Step 3, which involves performing potential spatial alignment of the lidar-4D radar, specifically includes: Step 3-1: Construct a lidar VAE with the same structure as the 4D radar VAE, including the encoder. and decoder The input channel is set to 1; Step 3-2: After performing min-max normalization on the sparse 3D LiDAR data, the data is input into the LiDAR VAE and processed by the encoder. Generation and 4D radar potential vectors LiDAR latent vectors with completely identical dimensions and shapes Its posterior distribution is N represents the posterior distribution. These are low-dimensional latent vectors. The mean vector and standard deviation vector; Step 3-3, the goal of LiDAR VAE training is to minimize VAE loss. The reconstruction loss is calculated using the L2 distance between the input and the reconstruction result; Semantic and spatial alignment is achieved through a patch-wise contrastive learning strategy, specifically including: for paired LiDAR-4D radar samples, latent encoding of LiDARs at the same spatial location. With 4D radar potential coding As positive sample pairs, the encodings of different locations or different samples are used as negative sample pairs; the final training loss of the LiDAR VAE is: in, in, This represents a scalar value indicating the total training loss of the LiDAR VAE. This represents the fundamental loss scalar of the lidar VAE. To compare the loss weights, Let be the scalar representing the cross-modal contrastive learning loss value, sim(·,·) be the cosine similarity function, N represent the number of samples in the training batch, and M represent the number of latent patches for each sample. This represents the latent patch feature of the i-th lidar sample at the j-th spatial location. This represents the latent patch feature of the i-th 4D radar sample at the j-th spatial location. This represents the latent patch feature of the l-th spatial location of the k-th 4D radar sample, where τ is the temperature hyperparameter. Represents the exponential transformation, log( ) represents a logarithmic transformation.

7. The cross-modal translation method for lidar to 4D radar based on a potential diffusion bridge according to claim 6, characterized in that, Step 4, which describes cross-modal conversion based on a potential diffusion bridge, specifically includes: In the aligned latent space, the translation from lidar to 4D radar is modeled as a Brownian bridge stochastic process; Step 4-1: For paired samples (I,R), use the pre-trained and fixed encoders of the LiDAR VAE and 4D radar VAE. and Generate respectively and Constructing a random interpolation process ,satisfy: Where I and R are the LiDAR and 4D radar samples, respectively, ε ~ N(0,I), and t∈[0,1]. For the variance term, realize the variance from t=1. up to t=0 The smooth transition, whose time evolution satisfies the stochastic differential equation, i.e., SDE: In the formula, For standard Brownian motion, This represents the change in latent encoding at time t. This indicates the global noise intensity. Step 4-2: Implement the drift term prediction network using 3D U-Net. Furthermore, the network incorporates sinusoidal time embeddings and adapts to different diffusion time steps t through AdaLN layers; the training objective is to minimize the mean square error of the drift term prediction. In the formula, This represents the drift prediction loss of the diffusion bridge. This represents the joint expectation of the lidar latent vector and the 4D radar latent vector at time t. This indicates that the 3D U-shaped network model has parameters The predicted drift term below; During the inference phase, non-Markov sampling is used to accelerate generation, given a time-step sequence. Finally, the 4D radar potential vector was obtained. The estimated value , ∈[0,1].

8. The cross-modal translation method for lidar to 4D radar based on a potential diffusion bridge according to claim 7, characterized in that, Step 5, which describes 4D radar reconstruction based on 4D radar latent vectors to generate a complete 4D radar tensor, specifically includes: Step 5-1: The 4D radar potential vector obtained in Step 4... Input pre-trained 4D radar VAE decoder The reconstructed normalized radar data was obtained. The absolute signal power value is obtained by inverse normalization. ; Step 5-2, Fill in key voxels Noise is generated for the background voxels while keeping the total power constant, and finally the complete 4D radar tensor is obtained.

9. The cross-modal translation method for lidar to 4D radar based on a potential diffusion bridge according to claim 8, characterized in that, In step 5-2, noise is generated on the background voxels using a Gaussian-softmax distribution.

10. A cross-modal translation system for lidar to 4D radar based on the method of any one of claims 1 to 9, characterized in that, The system includes: The first module is used to preprocess radar data; The second module is used to: construct and train a key voxel-sensing 4D radar VAE; The third module is used to achieve potential spatial alignment of the lidar-4D radar. The fourth module is used to achieve cross-mode conversion between lidar and 4D radar based on the potential diffusion bridge, and to obtain the potential vector of 4D radar. The fifth module is used to realize: 4D radar reconstruction based on 4D radar latent vectors, generating a complete 4D radar tensor.

Images