Lidar point cloud generation method and system based on pre-trained image diffusion model

Abstract

This invention discloses a method and system for generating LiDAR point clouds based on a pre-trained image diffusion model. This invention enables unconditionally controlled generation of LiDAR point clouds in real-world scenes, conditionally controlled generation of LiDAR point clouds based on camera images, and LiDAR point cloud completion under zero-shot learning conditions. By introducing a dual-spatial consistency optimization strategy at both the 2D and 3D levels, the pre-trained Stable Diffusion 3 model can perceive spatial information and geometric consistency. A physically guided control network (ControlNet) is designed to endow the model with the ability to generate point clouds for specific scenes, thereby supporting the conversion task from camera images to LiDAR point clouds. Extensive experiments in a 64-line LiDAR scenario demonstrate the significant effectiveness and application potential of the proposed method in both conditional and unconditional generation tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and autonomous driving technology, and in particular, it is a method and system for generating lidar point clouds based on a pre-trained image diffusion model. Background Technology

[0002] With the widespread application of autonomous driving systems in transportation, logistics, and services, environmental perception technology has received extensive attention from academia and industry. LiDAR, as a core sensor, provides precise 3D geometric measurements and is widely used in autonomous vehicles, measurement drones, and mobile robots. However, acquiring real LiDAR data covering diverse scenarios and specific tasks is costly and the dataset size is limited, severely hindering the performance improvement of downstream perception tasks. To address this issue, synthesizing realistic LiDAR point clouds to supplement or partially replace expensive real data has become an important research direction.

[0003] Early work primarily explored the use of deep generative models for LiDAR point cloud generation. Given the irregular and sparse nature of outdoor LiDAR point clouds, researchers typically convert them into distance images, i.e., 2D projections of equidistant rectangles onto 3D point clouds. For example, Caccia et al. (Lucas Caccia, Herke Van Hoof, Aaron Courville, and Joelle Pineau. 2019. Deep generative modeling of lidar data. In 2019 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS). 5034–5040.) utilized variational autoencoders and generative adversarial networks to generate point clouds on spherical projection maps. However, VAEs are often limited by posterior collapse, resulting in blurry and low-diversity generated samples; while GANs can generate high-quality samples, training through a game between the generator and discriminator can easily lead to training instability and pattern collapse.

[0004] In recent years, diffusion models have become central to computer vision due to their simple likelihood training objectives, training stability, and ability to generate high-fidelity samples. Inspired by this, some works have begun to apply diffusion models to LiDAR data generation. For example, Zyrianov et al. (Vlas Zyrianov, Xiyue Zhu, and Shenlong Wang. 2022. Learning to generate realistic lidar point clouds. In European Conference on Computer Vision. 17–35.) modeled the generation process as a stochastic denoising process in a distance view. Nakashima et al. (Kazuto Nakashima and Ryo Kurazume. 2024. Lidar data synthesis with denoising diffusion probabilistic models. In 2024 IEEE International Conference on Robotics and Automation (ICRA). 14724–14731.) applied denoising diffusion probabilistic models (DDPMs) to distance and reflectance intensity images, and considered spatial inductive bias for generation.

[0005] However, most existing work requires redesigning the network architecture and training from scratch. The available LiDAR training data is extremely limited compared to image datasets containing billions of samples, restricting the model's scene modeling capabilities. While pre-trained image generation models excel at capturing rich 2D semantics and texture, directly applying them to LiDAR generation tasks faces a significant domain gap: pre-trained image models lack explicit absolute depth perception, struggle to maintain spatial consistency, leading to geometric discontinuities and structural distortions in 3D reconstruction. Furthermore, existing methods often fail to incorporate LiDAR-specific physical characteristics, such as vertical scanning patterns, and cannot achieve conditional generation that conforms to physical priors, resulting in suboptimal generation performance. Summary of the Invention

[0006] The purpose of this invention is to address the problems existing in the prior art by proposing a cross-modal data prediction method based on transfer learning. This method utilizes a large language model to transform text into a two-dimensional lexical mapping graph, which is then used as a blueprint to directly guide the attention calculation of the diffusion model, thereby achieving efficient and accurate spatially controllable generation without the need for training.

[0007] The technical solution to achieve the purpose of this invention is: a method for generating lidar point clouds based on a pre-trained image diffusion model, the method comprising the following steps:

[0008] Step 1: Obtain the LiDAR point cloud dataset and use the Hough voting-based mapping method to project the 3D point cloud data into a 2D distance image, which will then be used as input data for model training.

[0009] Step 2: Construct a generative model based on a pre-trained image diffusion transformer, and introduce a low-rank adaptation module, namely the LoRA module, to adapt to the distance image domain;

[0010] Step 3: Design a dual-space consistency optimization strategy to simultaneously constrain the consistency of the latent space and the consistency of the 3D geometric space during the fine-tuning of the generative model. At the same time, optimize the parameters of the generative model by constructing a composite loss function so that the generative model can learn the spatial information and geometric structure of the LiDAR.

[0011] Step 4: Construct a physically guided control network by introducing a physical perception module to encode the vertical scanning mode of the lidar sensor as a physical prior, so as to realize point cloud generation control based on camera image conditions.

[0012] Step 5: Using the fine-tuned generative model and the physics-guided control network, a distance image is sampled from Gaussian noise and generated.

[0013] Step 6: Reconstruct the 3D LiDAR point cloud from the distance image generated in Step 5 using inverse projection.

[0014] Furthermore, in step 1, the 3D point cloud data is projected into a 2D distance image using a Hough voting-based mapping method. Specifically, this includes: for point clouds in Euclidean space... Each point in Convert it to spherical coordinates The formula is as follows:

[0015]

[0016]

[0017]

[0018] In the formula, Point clouds The three-dimensional coordinates of each point in the data. This represents the distance from the lidar sensor to the corresponding point. It is the azimuth angle. The tilt angle is N; N represents the number of points in the point cloud P.

[0019] Estimating the first using the Hough voting mechanism The specific height of each lidar sensor and pitch angle And adjust the projection process according to the following formula to reduce geometric distortion:

[0020]

[0021] Then, point Rasterization to size 2D cylindrical projection and normalize the depth range to The interval; where, These are the height and width, respectively.

[0022] Furthermore, step 2 involves constructing a generative model based on a pre-trained image diffusion transformer, specifically including:

[0023] The Stable Diffusion 3 (SD3) model, based on the corrected flow framework, is used as the fundamental model. This model predicts the velocity field. To define the transmission path between noise distribution and data distribution;

[0024] LoRA parameters inserted only into the DiT backbone network Fine-tune the settings while maintaining the pre-trained weights. Freeze, velocity field is represented as :

[0025]

[0026] In the formula, for Latent noise variables at time t, As a conditional feature, It is a low-rank decomposition matrix;

[0027] The updates of latent variables follow flow-based dynamic equations:

[0028]

[0029] In the formula, For time step, for Noise latent variables at time points.

[0030] Furthermore, the composite loss function described in step 3 Defined as:

[0031]

[0032] In the formula, For conditional flow matching loss, For potential reconstruction target losses, For deep consistency loss, To preserve loss at the edge, Point alignment reward loss; These are the corresponding weighting coefficients.

[0033] Furthermore, the aforementioned , , , and Specifically, they are represented as follows:

[0034] (1) Conditional flow matching loss :

[0035]

[0036] In the formula, This represents a clean sample that participates from the real data distribution. To represent noise sampled from a standard Gaussian distribution, Indicates time step Data samples ,noise and condition information The joint expectations express The square of the norm;

[0037] (2) Potential reconstruction target loss :

[0038]

[0039] In the formula, For the generated potential representation, For the true potential representation;

[0040] (3) Depth consistency loss This is used to preserve depth variations on an object's surface through gradient matching.

[0041]

[0042] In the formula, This represents the gradient difference features between the generated image and the real image in the height direction. This indicates the gradient difference between the two along the width direction;

[0043] (4) Edge preservation loss Used to capture fine object details in pixel space:

[0044]

[0045] In the formula, and These represent the generated distance image and the ground truth distance image, respectively. This indicates an edge feature extraction operation, configured to extract high-frequency geometric structure information and gradient change features from the image; Norm, a measure of feature differences;

[0046] (5) Point alignment reward loss Used for penalizing the reconstruction of point clouds Compared with real point clouds Geometric inconsistencies between them:

[0047]

[0048] In the formula, , and These are the distance images generated respectively. and true distance image The point cloud derived from the conversion, , They represent point clouds respectively. and The first in One point.

[0049] Furthermore, the construction of the physical guidance control network described in step 4 specifically includes:

[0050] Step 4-1: Construct the physical perception module. First, generate spatial variation weights based on the vertical scanning mode of the lidar sensor as physical priors. :

[0051]

[0052]

[0053]

[0054] In the formula, Represents the row index of the image. Image height, Indicates the first The vertical elevation angle corresponding to the row; Let represent the physical prior value of the LiDAR scanning mode at the position corresponding to the i-th row and j-th column in the 2D mesh; Represents the vertical physical prior value of the LiDAR scanning mode in the i-th row of the 2D mesh; Indicates the number of columns in a 2D mesh; It is a preset nonlinear mapping function, which is configured to reflect the physical laws of the lidar point cloud distribution;

[0055] Step 4-2, using a trainable hybrid network to integrate physical priors With the learned attention map Combined, the final spatial adjustment map is generated. :

[0056]

[0057]

[0058] In the formula, Represents the Hadamard product. It is a learnable attention map that captures specific scene patterns. This refers to the output characteristics of the physical sensing module, i.e., the DiT module. These are the mixing coefficients learned from the input; This represents the vertical physical prior information of the lidar; This represents a trainable hybrid network; This represents a learnable network used to extract importance scores;

[0059] Step 4-3: Inject the output of the physical sensing module into the control layer:

[0060]

[0061] In the formula, For the control layer, This is the output of the l-th control layer;

[0062] Step 4-4: Based on the formula in Step 4-3, obtain the output sequences of all control layers in the control network. The output sequence is then fused with the corresponding features of the backbone network DiT to jointly adjust the denoising process; whereby, For the first The output of the layer control layer, This indicates the total number of control layers in the control network.

[0063] Furthermore, the control layer is specifically implemented as a two-layer MLP containing LayerNorm and SiLU activation functions.

[0064] On the other hand, a lidar point cloud generation system is provided, the system comprising:

[0065] The first module is used to: acquire the LiDAR point cloud dataset and use the Hough voting-based mapping method to project the 3D point cloud data into a 2D distance image, which is then used as input data for model training.

[0066] The second module is used to implement: constructing a generative model based on a pre-trained image diffusion transformer, and introducing a low-rank adaptation module, namely the LoRA module, to adapt to the distance image domain;

[0067] The third module is used to implement: design a dual-space consistency optimization strategy to simultaneously constrain the consistency of the latent space and the consistency of the 3D geometric space during the fine-tuning of the generative model, and optimize the parameters of the generative model by constructing a composite loss function so that the generative model learns the spatial information and geometric structure of the LiDAR.

[0068] The fourth module is used to: construct a physically guided control network, and introduce a physical perception module to encode the vertical scanning mode of the lidar sensor as a physical prior, so as to realize point cloud generation control based on camera image conditions;

[0069] The fifth module is used to: sample and generate distance images from Gaussian noise using a fine-tuned generative model and a physically guided control network;

[0070] The sixth module is used to restore the distance image generated by the fifth module into a 3D LiDAR point cloud through inverse projection.

[0071] On the other hand, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the lidar point cloud generation method based on a pre-trained image diffusion model.

[0072] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the lidar point cloud generation method based on a pre-trained image diffusion model.

[0073] Compared with the prior art, the significant advantages of this invention are:

[0074] (1) This invention proposes a new method for generating lidar point clouds based on a pre-trained image diffusion model (such as Stable Diffusion 3), which effectively alleviates the problem of scarce lidar training data by utilizing the rich generation priors in the image model.

[0075] (2) This invention introduces a dual spatial consistency optimization strategy in the fine-tuning stage, which includes depth, edge and point alignment loss, enabling the pre-trained model to perceive spatial information and geometric structure, effectively bridging the gap between 2D image semantics and 3D point cloud geometry.

[0076] (3) The present invention designs a physical guidance control network, which creatively injects the physical prior of lidar - vertical scanning mode into the generation process, so that the generated point cloud conforms to physical laws in geometric distribution.

[0077] (4) This invention not only supports unconditional high-quality point cloud generation, but also supports conditional generation based on camera images and point cloud completion tasks under zero-sample conditions, which has important application value in the fields of autonomous driving simulation and data augmentation.

[0078] (5) The proposed method is the first framework to explore and adapt the pre-trained image diffusion model to the lidar point cloud generation task, breaking the bottleneck of traditional methods being limited by the scarcity of lidar data.

[0079] (6) A dual-space consistency optimization strategy is proposed. By applying constraints to both the 2D latent space and the 3D geometric space, the problem of geometric distortion caused by directly applying the image model is effectively solved, ensuring that the generated distance image has accurate depth perception and structural semantics.

[0080] (7) A physical-guided control network was designed to encode the vertical scanning mode of the lidar sensor as a physical prior, which not only realizes the controllable generation from camera image to lidar point cloud, but also ensures that the generation result conforms to the physical characteristics of the sensor.

[0081] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0082] Figure 1 This is a flowchart of the lidar point cloud generation method based on a pre-trained image diffusion model according to one embodiment of the present invention.

[0083] Figure 2 This is a visualization of the qualitative comparison between the present invention and the prior art R2Flow in an unconditional generation task in one embodiment. Figure 2 (a) in the image represents the true point cloud. Figure 2 (b) in the image is the point cloud generated by R2Flow. Figure 2 (c) in the figure represents the point cloud generated by this method.

[0084] Figure 3 This is a comparison chart showing the performance of the present invention in an image-to-LiDAR conditional generation task in one embodiment, wherein... Figure 3In the image (a), the input camera image is shown. Figure 3 (b) in the image represents the corresponding real point cloud. Figure 3 (c) in the image represents the point cloud generated by the LiDM method. Figure 3 In the diagram, (d) represents the point cloud generated by the MLP-based perception module. Figure 3 (e) in the figure represents the point cloud generated by the physical sensing module proposed in this method.

[0085] Figure 4 This is a visualization result of zero-sample point cloud completion based on text prompts in one embodiment of the present invention, wherein... Figure 4 In the diagram, (a) represents the original point cloud. Figure 4 (b) in the diagram represents point clouds obscured from a viewing angle of 10° to 25°. Figure 4 (c) in the text indicates that the input prompt is "a car". Figure 4 In the text, (d) indicates the input prompt word "a road sign". Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0087] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0088] In one embodiment, combined Figure 1 This paper provides a method for generating lidar point clouds based on a pre-trained image diffusion model, the method comprising the following steps:

[0089] Step 1: Obtain the LiDAR point cloud dataset and convert the disordered 3D point cloud into a 2D distance image using a Hough voting-based projection method, which will then be used as input data for model training.

[0090] Step 2: Construct a generative model based on a pre-trained image diffusion transformer (DiT) and introduce a low-rank adaptation (LoRA) module to adapt to the feature distribution in the distance image domain;

[0091] Step 3: Design a dual-space consistency optimization strategy to simultaneously constrain the consistency of the latent space and the consistency of the 3D geometric space during the fine-tuning of the generative model. At the same time, optimize the parameters of the generative model by constructing a composite loss function so that the generative model can learn the spatial information and geometric structure of the LiDAR.

[0092] Step 4: Construct a Physics-Guided ControlNet, introduce a physical perception module to encode the vertical scanning mode of the lidar sensor as a physical prior, and realize point cloud generation control based on conditions and its physical characteristics.

[0093] Step 5: Using the fine-tuned generative model and the physics-guided control network, a distance image is sampled from Gaussian noise and generated.

[0094] Step 6: Reconstruct the 3D LiDAR point cloud from the distance image generated in Step 5 using inverse projection.

[0095] Furthermore, in one embodiment, step 1, which uses a Hough voting-based mapping method to project and convert 3D point cloud data into a 2D distance image, specifically includes:

[0096] To adapt to the input format of the pre-trained image model, the unstructured sparse point cloud is first projected from Euclidean space to a range image representation. For the point cloud in Euclidean space... , each of the points Convert to spherical coordinates The specific formula is as follows:

[0097]

[0098]

[0099] In the formula, Point clouds The three-dimensional coordinates of each point in the data. This represents the distance from the lidar sensor to the corresponding point. It is the azimuth angle. The angle is θ; N represents the number of points in the point cloud P.

[0100] Considering that in existing datasets (such as KITTI-360), multiple laser beams from the Velodyne lidar system do not originate from the same origin, this may lead to errors in the Cartesian to spherical coordinate transformation, thereby distorting the data distribution of the range view and degrading image quality. Therefore, this embodiment employs the Hough voting mechanism to estimate the... The specific height of each lidar sensor and pitch angle And adjust the projection process according to the following formula to reduce geometric distortion:

[0101]

[0102] Then, point Rasterization to 2D cylindrical projection ,in This represents the grid coordinates. Finally, the depth value... Normalization to The interval is used to match the input distribution of Stable Diffusion 3 (SD3).

[0103] Furthermore, in one embodiment, step 2, constructing a generative model based on a pre-trained image diffusion transformer, specifically includes:

[0104] Stable Diffusion 3 (SD3), based on the Rectified Flow framework, is used as the fundamental model. Unlike traditional Denoising Diffusion Probability Models (DDPMs) that predict noise, SD3 predicts the velocity field. This defines the transmission path between the noise distribution and the data distribution.

[0105] To achieve efficient parameter fine-tuning, this invention does not update the entire DiT backbone network, but only fine-tunes the LoRA parameters inserted into it. Maintain pre-trained weights Freeze. The velocity field is represented as:

[0106]

[0107] In the formula, For a moment The noise latent variable, As a conditional feature, It is a low-rank decomposition matrix.

[0108] The updates of latent variables follow flow-based dynamic equations:

[0109]

[0110] In the formula, For time step, for Noise latent variables at time points. In this way, the model retains the pre-trained generative priors while effectively adapting to the distance image domain.

[0111] Furthermore, in one embodiment, in the dual-space consistency optimization strategy described in step 3, the composite loss function... Defined as:

[0112]

[0113] In the formula, For conditional flow matching loss, For potential reconstruction target losses, For deep consistency loss, To preserve loss at the edge, Point alignment reward loss; These are the corresponding weighting coefficients.

[0114] The specific definitions of each loss function are as follows:

[0115] (1) Conditional flow matching loss :

[0116]

[0117] In the formula, This represents a clean sample that participates from the real data distribution. To represent noise sampled from a standard Gaussian distribution, Indicates time step Data samples ,noise and condition information The joint expectations express The square of the norm.

[0118] (2) Potential reconstruction target loss Used for direct supervision of latent spatial representation, ensuring the latent representation of the generated image. Latent representation of real images Consistency:

[0119]

[0120] In the formula, For the generated potential representation, This represents the true potential.

[0121] (3) Depth consistency loss Given the continuity of depth in the real world, gradient matching is used to maintain accurate depth variations along the object's surface. The formula is as follows:

[0122]

[0123] In the formula, This represents the gradient difference features between the generated image and the real image in the height direction. This represents the gradient difference between the two along the width direction.

[0124] (4) Edge preservation loss To address the loss of high-frequency details and blurred boundaries caused by VAE compression, a pixel-space edge preservation objective is introduced:

[0125]

[0126] In the formula, and These represent the generated distance image and the ground truth distance image, respectively. This indicates an edge feature extraction operation, configured to extract high-frequency geometric structure information and gradient change features from the image; The norm that represents the difference in features.

[0127] (5) Point alignment reward loss To punish the reconstruction of point clouds Compared with real point clouds Geometric inconsistencies between 2D and 3D models must be addressed to ensure geometric accuracy after the 2D-to-3D conversion.

[0128]

[0129] In the formula, , and These are the distance images generated respectively. and true distance image The point cloud derived from the conversion, , They represent point clouds respectively. and The first in One point.

[0130] Furthermore, in one embodiment, step 4, constructing the control network for physical guidance, specifically includes:

[0131] Step 4-1: Construct the physical sensing module. It was observed that the lidar point cloud is denser in low-lying areas (e.g., the ground) and sparser with higher uncertainty in high-lying areas (e.g., the sky). Therefore, a nonlinear curve was used to simulate the vertical prior. The specific formula is as follows:

[0132]

[0133]

[0134]

[0135] In the formula, Represents the row index of the image. Image height, Indicates the first The vertical elevation angle corresponding to the row; Let represent the physical prior value of the LiDAR scanning mode at the position corresponding to the i-th row and j-th column in the 2D mesh; Represents the vertical physical prior value of the LiDAR scanning mode in the i-th row of the 2D mesh; Indicates the number of columns in a 2D mesh; This is a pre-defined nonlinear mapping function configured to reflect the physical laws governing the distribution of lidar point clouds. Specifically, for low-elevation-angle regions (such as the ground and obstacles) with dense point cloud distribution and high confidence, higher weight values ​​are assigned to enhance monitoring; while for high-elevation-angle regions (such as the sky) with sparse point cloud distribution or uncertainties, lower weight values ​​are assigned.

[0136] Step 4-2, use a trainable hybrid network (MLP) to integrate physical priors. With learned attention weights Adaptive combination generates the final spatial adjustment map. :

[0137]

[0138]

[0139] In the formula, Represents the Hadamard product. It is a learnable attention map that captures specific scene patterns. This refers to the output characteristics of the physical sensing module, i.e., the DiT module. These are the mixing coefficients learned from the input; This represents the vertical physical prior information of the lidar; This represents a trainable hybrid network; This represents a learnable network used to extract importance scores. This mechanism allows the model to flexibly learn scene-specific patterns while leveraging physical inductive biases.

[0140] Step 4-3: Inject the output of the physical sensing module into the control layer:

[0141]

[0142] In the formula, For the control layer, This is the output of the l-th control layer.

[0143] Step 4-4: Based on the formula in Step 4-3, obtain the output sequences of all control layers in the control network. The output sequence is then fused with the corresponding features of the backbone network DiT to jointly adjust the denoising process; whereby, For the first The output of the layer control layer, This indicates the total number of control layers in the control network.

[0144] Preferably, in some embodiments, the control layer is specifically implemented as a two-layer MLP containing LayerNorm and SiLU activation functions.

[0145] In one embodiment, a lidar point cloud generation system is provided, the system comprising:

[0146] The first module is used to: acquire the LiDAR point cloud dataset and use the Hough voting-based mapping method to project the 3D point cloud data into a 2D distance image, which is then used as input data for model training.

[0147] The second module is used to implement: constructing a generative model based on a pre-trained image diffusion transformer, and introducing a low-rank adaptation module, namely the LoRA module, to adapt to the distance image domain;

[0148] The third module is used to implement: design a dual-space consistency optimization strategy to simultaneously constrain the consistency of the latent space and the consistency of the 3D geometric space during the fine-tuning of the generative model, and optimize the parameters of the generative model by constructing a composite loss function so that the generative model learns the spatial information and geometric structure of the LiDAR.

[0149] The fourth module is used to: construct a physically guided control network, and introduce a physical perception module to encode the vertical scanning mode of the lidar sensor as a physical prior, so as to realize point cloud generation control based on camera image conditions;

[0150] The fifth module is used to: sample and generate distance images from Gaussian noise using a fine-tuned generative model and a physically guided control network;

[0151] The sixth module is used to restore the distance image generated by the fifth module into a 3D LiDAR point cloud through inverse projection.

[0152] Specific limitations regarding the LiDAR point cloud generation system based on a pre-trained image diffusion model can be found in the limitations of the LiDAR point cloud generation method based on the pre-trained image diffusion model described above, and will not be repeated here. Each module in the aforementioned LiDAR point cloud generation system based on the pre-trained image diffusion model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0153] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements:

[0154] Step 1: Obtain the LiDAR point cloud dataset and use the Hough voting-based mapping method to project the 3D point cloud data into a 2D distance image, which will then be used as input data for model training.

[0155] Step 2: Construct a generative model based on a pre-trained image diffusion transformer, and introduce a low-rank adaptation module, namely the LoRA module, to adapt to the distance image domain;

[0156] Step 3: Design a dual-space consistency optimization strategy to simultaneously constrain the consistency of the latent space and the consistency of the 3D geometric space during the fine-tuning of the generative model. At the same time, optimize the parameters of the generative model by constructing a composite loss function so that the generative model can learn the spatial information and geometric structure of the LiDAR.

[0157] Step 4: Construct a physically guided control network by introducing a physical perception module to encode the vertical scanning mode of the lidar sensor as a physical prior, so as to realize point cloud generation control based on camera image conditions.

[0158] Step 5: Using the fine-tuned generative model and the physics-guided control network, a distance image is sampled from Gaussian noise and generated.

[0159] Step 6: Reconstruct the 3D LiDAR point cloud from the distance image generated in Step 5 using inverse projection.

[0160] For specific limitations on each step, please refer to the limitations on the LiDAR point cloud generation method based on the pre-trained image diffusion model mentioned above, which will not be repeated here.

[0161] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0162] Step 1: Obtain the LiDAR point cloud dataset and use the Hough voting-based mapping method to project the 3D point cloud data into a 2D distance image, which will then be used as input data for model training.

[0163] Step 2: Construct a generative model based on a pre-trained image diffusion transformer, and introduce a low-rank adaptation module, namely the LoRA module, to adapt to the distance image domain;

[0164] Step 3: Design a dual-space consistency optimization strategy to simultaneously constrain the consistency of the latent space and the consistency of the 3D geometric space during the fine-tuning of the generative model. At the same time, optimize the parameters of the generative model by constructing a composite loss function so that the generative model can learn the spatial information and geometric structure of the LiDAR.

[0165] Step 4: Construct a physically guided control network by introducing a physical perception module to encode the vertical scanning mode of the lidar sensor as a physical prior, so as to realize point cloud generation control based on camera image conditions.

[0166] Step 5: Using the fine-tuned generative model and the physics-guided control network, a distance image is sampled from Gaussian noise and generated.

[0167] Step 6: Reconstruct the 3D LiDAR point cloud from the distance image generated in Step 5 using inverse projection.

[0168] For specific limitations on each step, please refer to the limitations on the LiDAR point cloud generation method based on the pre-trained image diffusion model mentioned above, which will not be repeated here.

[0169] As a specific example, the invention will be further verified and illustrated in one embodiment.

[0170] To comprehensively verify the effectiveness and robustness of the proposed LiDAR point cloud generation method based on implicit manifold priors in complex driving scenarios, this embodiment conducted systematic quantitative comparison and qualitative analysis experiments on the challenging public dataset KITTI-360. This dataset contains 81,106 frames of 64-line LiDAR scan data across 9 sequences, covering diverse urban driving scenarios in Germany. This embodiment selected the first two sequences (30,758 frames in total) as the test set and the remaining sequences as the training set. All experiments were conducted on a computing platform equipped with an NVIDIA Tesla A800 GPU, and the Fréchet Range Distance (FRD), Jensen-Shannon Divergence (JSD), Maximum Mean Discrepancy (MMD), and volume-based metrics Fréchet Sparse Volume Distance (FSVD) and Fréchet Point-based Volume were used. Distance (FPVD) is the primary evaluation metric. Among them, JSD and MMD are used to measure the statistical similarity between generated point clouds and real point clouds, while FRD, FSVD and FPVD focus on evaluating perceptual quality.

[0171] First, this embodiment evaluates the model's performance on the unconditional lidar generation task and compares it with several state-of-the-art baseline methods, including LiDARGAN, LiDAR VAE, UltraLiDAR, Projected GAN, LiDARGen, LiDM, and the latest R2Flow. Furthermore, this invention evaluates the comprehensive performance of the proposed method across different metrics, as shown in Table 1. As can be seen from the results in Table 1, the proposed IMPEL method achieves optimal results across all five evaluation metrics. Specifically, the method significantly reduces the FRD value to 400.69, the JSD value to 0.048, and the MMD value to 2.4 × 10⁻⁻⁻⁶. 4 With an FSVD value of 22.5 and an FPVD value of 26.4, compared to the suboptimal R2Flow method, this invention improves the FRD metric by 58.42%, and improves the JSD and MMD metric by 14.29% and 46.67%, respectively. This significant improvement demonstrates that this invention, through a dual spatial consistency optimization strategy, can effectively capture and maintain the structural semantics of point clouds, making the generated point clouds closer to real-world data in terms of statistical distribution and geometric structure. Figure 2 Qualitative visualization analysis shows that existing technologies such as R2Flow often generate samples with excessive noise or sparse structure, making it difficult to accurately reproduce the geometric boundaries of objects. However, the samples generated by this invention are not only clearer in object details (such as vehicle outlines and road edges), but also maintain a high degree of consistency with the real-world reference point cloud in terms of overall scene layout, verifying the superiority of this model in capturing object structural semantics and modeling spatial relationships.

[0172] Table 1. Quantitative comparison of unconditional generation results based on 64-line data (i.e., KITTI-360).

[0173] Method Years↓ FRD↓ JSD↓ MMD↓ FSVD↓ FPVD↓ LiDAR GAN IROS 2019 2738.48 0.236 1.1×10⁻² 95.3 68.1 LiDAR VAE IROS 2019 1356.75 0.181 3.6×10⁻³ 91.6 87.1 UltraLiDAR CVPR 2023 1911.88 0.182 2.3×10⁻³ 65.1 59.9 Projected GAN NeurIPS 2021 1022.62 0.127 1.0×10⁻³ 53.6 52.3 LiDARGen ECCV 2022 1104.32 0.111 <![CDATA[4.5×10⁻ 4 ]]> 59.3 53.0 LiDM CVPR 2024 1449.42 0.153 6.1×10⁻³ 38.0 40.7 R2Flow ICRA 2025 963.65 0.056 <![CDATA[9.1×10⁻ 4 ]]> 37.9 35.1 IMPEL (This invention) / 400.69 0.048 <![CDATA[2.4×10⁻ 4 ]]> 22.5 26.4 ΔImprov. / 58.42% 14.29% 46.67% 40.63% 24.79%

[0174] Furthermore, this invention also evaluated the model's performance in cross-modal conditional generation tasks, focusing on verifying the effectiveness of Physics-Guided ControlNet in image-to-LiDAR generation. The comparison results are shown in Table 2. In the experiments, this invention was compared with LiDM, Vanilla ControlNet, and an MLP-based spatial weighting module. As can be seen from the results in Table 2, the Physics-aware method proposed in this invention achieves the best results in FRD, JSD, and MMD, with the FRD value reduced to 458.70 and the JSD value optimized to 0.053. Further analysis of Table 2 and... Figure 3It is known that although the LiDM method is specifically designed for Camera-to-LiDAR tasks, it can only capture limited semantic information (such as vehicle density and road width) and fails to faithfully reflect the geometric structure of the real scene. This is mainly due to its lack of explicit modeling of the physical characteristics of the LiDAR. In contrast, this invention introduces a physical perception module, uses vertical physical priors to encode the scanning pattern of the LiDAR, and combines it with a scene-specific attention mechanism. This enables the generation of objects with fine geometric structures, such as walls, vehicles, and road guardrails. Furthermore, these generated objects correspond spatially to the input image, demonstrating the crucial role of physical priors in enhancing the model's semantic understanding and spatial controllability. Simultaneously, this embodiment also conducted zero-sample point cloud completion experiments, such as... Figure 4 As shown, by occluding a point cloud region at a 15-degree azimuth angle and inputting text prompts such as "A Car" or "A roadsign", the results show that the present invention can effectively align text semantics with 3D spatial structure by utilizing the prior knowledge of a pre-trained image diffusion model, generating complete content that matches the prompt description, further broadening the application scenarios of the present invention.

[0175] Table 2 Comparison of Image-to-LiDAR Generation Performance

[0176] Method FRD↓ JSD↓ MMD↓ LiDM 1521.46 0.169 9.0×10⁻³ vanilla ControlNet 506.32 0.072 <![CDATA[8.1×10⁻ 4 ]]> MLP-based 483.38 0.055 <![CDATA[4.4×10⁻ 4 ]]> Physics-aware 458.70 0.053 <![CDATA[2.9×10⁻ 4 ]]>

[0177] Finally, to verify the necessity of each component in the proposed dual spatial consistency optimization strategy and its specific contribution to the final generation quality, this embodiment conducted detailed ablation experiments, setting up experimental groups for removing potential reconstruction targets (without recon), removing depth consistency targets (without depth), removing edge-aware targets (without edge), removing point-level alignment rewards (without point), and directly using the original SD3 model (vanilla SD3). The results are shown in Table 3. As can be seen from the results in Table 3, directly applying the original SD3 model yielded the worst performance (FRD of 1008.99), indicating that the original model lacks the ability to perceive spatial structure and maintain spatial consistency. Further analysis revealed that when potential reconstruction, depth consistency, and edge-aware targets were removed, the model's 3D spatial modeling accuracy decreased significantly, particularly in the JSD and MMD metrics. This highlights the importance of inter-pixel gradient coherence and edge-preserving constraints in maintaining the geometric continuity of the distance image and ensuring accurate 3D point distribution. Furthermore, removing the point-level alignment bonus (without point) caused the FRD metric to increase from 400.69 to 467.58, indicating that this module effectively captures accurate local features by refining point cloud details during the low-noise denoising stage. In summary, the complete IMPEL framework, through the synergistic effect of its modules, achieves optimal performance across all evaluation metrics, fully demonstrating the effectiveness and indispensability of the optimization strategy proposed in this invention for achieving high-fidelity LiDAR point cloud generation.

[0178] Table 3 Main Ablation Experiments

[0179] Method FRD↓ JSD↓ MMD↓ vanilla SD3 1008.99 0.113 1.4×10⁻³ w / o recon 430.41 0.057 <![CDATA[3.9×10⁻ 4 ]]> w / o depth 410.58 0.059 <![CDATA[4.2×10⁻ 4 ]]> w / o edge 422.33 0.057 <![CDATA[4.1×10⁻ 4 ]]> w / o point 467.58 0.049 <![CDATA[2.6×10⁻ 4 ]]> IMPEL 400.96 0.048 <![CDATA[2.4×10⁻ 4 ]]>

[0180] In summary, the proposed LiDAR point cloud generation method based on a pre-trained image diffusion model can achieve unconditionally controlled real-scene LiDAR point cloud generation, conditionally controlled LiDAR point cloud generation based on camera images, and LiDAR point cloud completion under zero-shot learning conditions. By introducing a dual spatial consistency optimization strategy at both the 2D and 3D levels, the pre-trained Stable Diffusion 3 model can perceive spatial information and geometric consistency. By designing a physically guided control network (ControlNet), the model is endowed with the ability to generate point clouds for specific scenes, thus supporting the conversion task from camera images to LiDAR point clouds. Extensive experiments in a 64-line LiDAR scenario demonstrate the significant effectiveness and application potential of the proposed method in both conditional and unconditional generation tasks.

[0181] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A method for generating lidar point clouds based on a pre-trained image diffusion model, characterized in that, The method includes the following steps: Step 1: Obtain the LiDAR point cloud dataset and use the Hough voting-based mapping method to project the 3D point cloud data into a 2D distance image, which will then be used as input data for model training. Step 2: Construct a generative model based on a pre-trained image diffusion transformer, and introduce a low-rank adaptation module, namely the LoRA module, to adapt to the distance image domain; Step 3: Design a dual-space consistency optimization strategy to simultaneously constrain the consistency of the latent space and the consistency of the 3D geometric space during the fine-tuning of the generative model. At the same time, optimize the parameters of the generative model by constructing a composite loss function so that the generative model can learn the spatial information and geometric structure of the LiDAR. Step 4: Construct a physically guided control network by introducing a physical perception module to encode the vertical scanning mode of the lidar sensor as a physical prior, so as to realize point cloud generation control based on camera image conditions. Step 5: Using the fine-tuned generative model and the physics-guided control network, a distance image is sampled from Gaussian noise and generated; Step 6: Reconstruct the 3D LiDAR point cloud from the distance image generated in Step 5 using inverse projection.

2. The lidar point cloud generation method based on a pre-trained image diffusion model according to claim 1, characterized in that, Step 1 uses a Hough voting-based mapping method to project and convert 3D point cloud data into a 2D distance image. Specifically, this includes: for point clouds in Euclidean space... Each point in Convert it to spherical coordinates The formula is as follows: In the formula, Point clouds The three-dimensional coordinates of each point in the data. This represents the distance from the lidar sensor to the corresponding point. It is the azimuth angle. The tilt angle is N; N represents the number of points in the point cloud P. Estimating the first using the Hough voting mechanism The specific height of each lidar sensor and pitch angle And adjust the projection process according to the following formula to reduce geometric distortion: Then, point Rasterization to size 2D cylindrical projection and normalize the depth range to The interval; where, These are the height and width, respectively.

3. The lidar point cloud generation method based on a pre-trained image diffusion model according to claim 2, characterized in that, Step 2 involves constructing a generative model based on a pre-trained image diffusion transformer, specifically including: The Stable Diffusion 3 (SD3) model, based on the corrected flow framework, is used as the fundamental model. This model predicts the velocity field. To define the transmission path between noise distribution and data distribution; LoRA parameters inserted only into the DiT backbone network Fine-tune the settings while maintaining the pre-trained weights. Freeze, velocity field is represented as : In the formula, for Latent noise variables at time 10:00 As a conditional feature, It is a low-rank decomposition matrix; The updates of latent variables follow flow-based dynamic equations: In the formula, For time step, for Noise latent variables at time points.

4. The lidar point cloud generation method based on a pre-trained image diffusion model according to claim 3, characterized in that, The composite loss function described in step 3 Defined as: In the formula, For conditional flow matching loss, For potential reconstruction target losses, For deep consistency loss, To preserve loss at the edge, Point alignment reward loss; These are the corresponding weighting coefficients.

5. The lidar point cloud generation method based on a pre-trained image diffusion model according to claim 4, characterized in that, The , , , and Specifically, they are represented as follows: (1) Conditional flow matching loss : In the formula, This represents a clean sample that participates from the real data distribution. To represent noise sampled from a standard Gaussian distribution, Indicates time step Data samples ,noise and condition information The joint expectations express The square of the norm; (2) Potential reconstruction target loss : In the formula, For the generated potential representation, For the true potential representation; (3) Depth consistency loss This is used to preserve depth variations on an object's surface through gradient matching. In the formula, This represents the gradient difference features between the generated image and the real image in the height direction. This indicates the gradient difference between the two along the width direction; (4) Edge preservation loss Used to capture fine object details in pixel space: In the formula, and These represent the generated distance image and the ground truth distance image, respectively. This indicates an edge feature extraction operation, configured to extract high-frequency geometric structure information and gradient change features from the image; Norm, a measure of feature differences; (5) Point alignment reward loss Used to punish the reconstruction of point clouds Compared with real point clouds Geometric inconsistencies between them: In the formula, , and These are the distance images generated respectively. and true distance image The point cloud derived from the conversion, , They represent point clouds respectively. and The first in One point.

6. The lidar point cloud generation method based on a pre-trained image diffusion model according to claim 5, characterized in that, Step 4, which describes constructing the control network for physical booting, specifically includes: Step 4-1: Construct the physical perception module. First, generate spatial variation weights based on the vertical scanning mode of the lidar sensor as physical priors. : In the formula, Represents the row index of the image. Image height, Indicates the first The vertical elevation angle corresponding to the row; Let represent the physical prior value of the LiDAR scanning mode at the position corresponding to the i-th row and j-th column in the 2D mesh; Represents the vertical physical prior value of the LiDAR scanning mode in the i-th row of the 2D mesh; Indicates the number of columns in a 2D mesh; It is a preset nonlinear mapping function, which is configured to reflect the physical laws of the lidar point cloud distribution; Step 4-2, using a trainable hybrid network to integrate physical priors With the learned attention map Combined, the final spatial adjustment map is generated. : In the formula, Represents the Hadamard product. It is a learnable attention map that captures specific scene patterns. This refers to the output characteristics of the physical sensing module, i.e., the DiT module. These are the mixing coefficients learned from the input; This represents the vertical physical prior information of the lidar; This represents a trainable hybrid network; This represents a learnable network used to extract importance scores; Step 4-3: Inject the output of the physical sensing module into the control layer: In the formula, For the control layer, This is the output of the l-th control layer; Step 4-4: Based on the formula in Step 4-3, obtain the output sequences of all control layers in the control network. The output sequence is then fused with the corresponding features of the backbone network DiT to jointly adjust the denoising process; whereby, For the first The output of the layer control layer, This indicates the total number of control layers in the control network.

7. The lidar point cloud generation method based on a pre-trained image diffusion model according to claim 6, characterized in that, The control layer is specifically implemented as a two-layer MLP containing LayerNorm and SiLU activation functions.

8. A lidar point cloud generation system based on the method of any one of claims 1 to 7, characterized in that, The system includes: The first module is used to: acquire the LiDAR point cloud dataset and use the Hough voting-based mapping method to project the 3D point cloud data into a 2D distance image, which is then used as input data for model training. The second module is used to implement: constructing a generative model based on a pre-trained image diffusion transformer, and introducing a low-rank adaptation module, namely the LoRA module, to adapt to the distance image domain; The third module is used to implement: design a dual-space consistency optimization strategy to simultaneously constrain the consistency of the latent space and the consistency of the 3D geometric space during the fine-tuning of the generative model, and optimize the parameters of the generative model by constructing a composite loss function so that the generative model learns the spatial information and geometric structure of the LiDAR. The fourth module is used to: construct a physically guided control network, and introduce a physical perception module to encode the vertical scanning mode of the lidar sensor as a physical prior, so as to realize point cloud generation control based on camera image conditions; The fifth module is used to: sample and generate distance images from Gaussian noise using a fine-tuned generative model and a physically guided control network; The sixth module is used to restore the distance image generated by the fifth module into a 3D LiDAR point cloud through inverse projection.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images