Multi-modal conditional lidar point cloud generation method based on three-dimensional layout guidance

Abstract

The application discloses a kind of multi-modal condition laser radar point cloud generation methods based on three-dimensional layout guide, comprising: step S1 obtains the multi-modal input information of user, input information includes text description, image data or original point cloud, and it is converted into unified 3D layout representation;Step S2 is based on 3D layout, utilizes the semantic and depth control signal of pixel alignment generation using ray projection technology, and it is generated by conditional generation model Laser radar point cloud in compliance with layout constraint;Step S3 outputs the laser radar point cloud generated to construct or expand autonomous driving simulation scene.Specifically, the method utilizes 3D layout as bridge to connect sparse user input and complex point cloud distribution;By converting multi-modal input into 3D layout containing geometric primitives, and further converting into semantic and depth distance image, ControlNet is used to fine-tune the pre-trained diffusion model, and high-quality, controllable laser radar point cloud generation based on arbitrary text, image or point cloud is realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, autonomous driving simulation, and computer vision, specifically to a method for generating point clouds for multimodal conditional LiDAR based on 3D layout guidance. This invention is applicable to the generation of training data for autonomous driving simulation, particularly the generation and editing of autonomous driving LiDAR scenes under cross-modal conditional (text / image / point cloud) control. Background Technology

[0002] With the rapid development of autonomous driving technology, extremely high demands are placed on the robustness of perception algorithms. Training and validating high-performance perception models requires massive amounts of precisely labeled LiDAR point cloud data. However, collecting real-world physical data faces multiple challenges: first, physical acquisition is costly and inefficient; second, real-world road testing presents potential safety risks; and finally, certain extreme or rare scenarios (such as traffic accidents and severe weather) are difficult to reproduce in reality, resulting in a long-tail effect in data distribution. Therefore, data synthesis technology based on generative models has become a key approach to solving these pain points.

[0003] Existing lidar point cloud generation technologies have primarily evolved from variational autoencoders (VAEs) to generative adversarial networks (GANs), and then to probabilistic diffusion models. Although diffusion model-based methods (such as LiDARGen) have achieved significant breakthroughs in generation quality, they still face the following significant technical bottlenecks in terms of controllability and versatility: First, there is a modal gap and an imbalance in control. Existing conditional generation methods typically attempt to directly encode simple user input (such as a short text description, a monocular image, or a randomly drawn sketch) into global feature vectors to guide the generation of complex 3D point clouds. However, due to the extremely low information density of user input (sparse signal) and the extremely complex distribution of the target point cloud (high-dimensional dense signal), a significant information imbalance exists between the two. This imbalance often makes it difficult for the generation model to adhere to the conditional constraints, resulting in point clouds whose object layout and geometry often differ greatly from the user's intentions.

[0004] Second, there is a lack of unified cross-modal representation. For inputs of different modalities such as text, images, or sketches, existing technologies typically require separate encoders and generative networks designed for each modality, lacking a unified intermediate representation layer. This makes the model difficult to generalize and unable to support flexible editing of the generated scene (such as adding or deleting objects in the generated scene).

[0005] Third, there is a lack of geometric and physical consistency. Traditional end-to-end generation models often treat point clouds as unordered point sets or simple distance images, ignoring the physical scanning characteristics of LiDAR (such as light occlusion and scan line distribution). Directly generated point clouds often fail to guarantee the spatial occlusion relationships between objects and the physical rationality of the geometric structure, leading to doubts about the credibility of simulation data and making it difficult to use for training downstream perception algorithms.

[0006] Therefore, there is an urgent need for a high-fidelity lidar point cloud generation method that can bridge the gap between sparse user input and dense point cloud distribution and has physical geometric consistency. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a method for generating multimodal conditional LiDAR point clouds based on 3D layout guidance. This method creatively introduces 3D layout as a universal intermediate condition representation, uses ray casting technology to transform the layout into a strong control signal for pixel alignment, and combines ControlNet and a diffusion model to achieve "simulation from scratch" of high-fidelity LiDAR point clouds from arbitrary text, images, or sketches.

[0008] To address the aforementioned technical problems, this invention provides a method for generating multimodal conditional lidar point clouds based on 3D layout guidance, comprising the following steps: Step S1: Unified layout representation of multimodal inputs.

[0009] Acquire the user's multimodal input information, which includes text descriptions, monocular image data, or single-frame point clouds; Using a parsing module for a specific modality, the multimodal input information is uniformly converted into a standardized 3D scene layout (3DLayout) representation; The 3D scene layout consists of a set of semantic primitives with semantic labels and simple geometric structures, including but not limited to cuboids, ellipsoids and planes.

[0010] Step S2: Generate control signals based on physical light projection.

[0011] Based on the 3D scene layout generated in step S1, the physical scanning mechanism of the LiDAR is simulated, and the raycasting algorithm is used for spatial sampling, thereby projecting the 3D geometric layout into pixel-aligned semantic range image and depth range image to construct control signals.

[0012] First, using the virtual lidar sensor as the origin, rays are emitted into the field of view to completely simulate the physical scanning process of a real lidar.

[0013] Then, using a raycasting algorithm, the intersection points of each ray with primitives in the 3D scene layout are calculated. These intersection points accurately record the spatial relationship between the sensor and the scene surface.

[0014] Finally, the intersection information calculated above is projected onto the imaging plane of the sensor to generate two pixel-aligned control images.

[0015] Depth-distance image: records the spatial distance information corresponding to each pixel; Semantic distance image: records the object category label to which each pixel belongs.

[0016] These two control images constitute a dense, precise pixel-by-pixel control signal.

[0017] Step S3: Conditional diffusion generation based on ControlNet.

[0018] Construct and train a generative network based on a diffusion probabilistic model; The semantic distance image and depth distance image generated in step S2 are used as control conditions and input into the conditional branch based on the ControlNet architecture; ControlNet is used to modulate the features of a pre-trained unconditional diffusion model to control the iterative denoising generation process of the LiDAR point cloud distance image. Finally, the generated point cloud distance image is back-projected into a 3D point cloud, and a high-fidelity LiDAR point cloud that meets the input conditions is output.

[0019] Furthermore, in step S1, a differentiated layout transformation strategy is adopted for inputs of different modalities: For text input: Use large language models (such as GPT-5) to understand complex scene descriptions (e.g., "intersection with only a few vehicles"), generate structured 3D modeling script code (such as BlenderPython scripts), and automatically build scene layouts in the 3D engine that include object size, position, and category.

[0020] For image input: use semantic segmentation models (such as SAM) and monocular depth estimation models (such as DepthAnything) to extract pixel-level semantic and depth information of the image, and backproject it into a pseudo-point cloud (PseudoPointCloud); then use density clustering algorithms (such as DBSCAN) to extract the 3D bounding boxes of foreground objects and construct the scene layout.

[0021] For point cloud input: Semantic segmentation is performed on the input sparse point cloud, and instance clustering is performed based on semantic categories to fit the geometric parameters of the object to reconstruct the 3D layout.

[0022] Furthermore, in step S2, the specific calculation process of the ray casting algorithm is as follows: For the coordinates in the distance image, The pixels, based on the horizontal angular resolution of the LiDAR. and vertical angular resolution Determine the corresponding ray emission angle: horizontal yaw angle. and vertical pitch angle .

[0023] After a ray intersects with a primitive, obtain the Euclidean distance between the intersection points. .

[0024] Based on this distance, the three-dimensional coordinates of this point in the sensor coordinate system are calculated using the inverse transformation formula of spherical coordinates. :

[0025] The control signal comprises two parts: a depth map (log-normalized distance values). ) and semantic graph (different categories are assigned different semantic labels).

[0026] Furthermore, in step S3, the training of the unconditional diffusion generative model comprises two stages: Phase 1 (Unconditional Base Training): Constructing a denoising score network based on the U-Net architecture Training is performed on unconditional, large-scale LiDAR datasets such as KITTI-360.

[0027] The objective function used is noise-conditioned score matching:

[0028] in, The original distance image, For noise level, It is Gaussian noise.

[0029] Phase Two (Fine-tuning of Conditions): Freeze the parameters of the base model trained in the first stage and introduce the ControlNet branch.

[0030] The ControlNet branch replicates the encoder structure of the base model and connects to the decoder of the base model through a zero-convolution layer.

[0031] The control signal generated in step S2 Type ControlNet.

[0032] The training objective is to minimize the conditional denoising error.

[0033] This stage only updates the parameters of the ControlNet branch and the zero convolutional layer to ensure that the model learns to obey layout control while retaining its ability to generate high-quality point clouds.

[0034] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method steps.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows: By creatively introducing 3D layout as a universal intermediate condition representation, and using ray casting technology to transform the layout into a strong control signal for pixel alignment, and combining ControlNet with a diffusion model, we have achieved "zero-to-zero simulation" of high-fidelity LiDAR point clouds from arbitrary text, images, or sketches. Attached Figure Description

[0036] To more clearly illustrate the embodiments and technical solutions of the present invention, the accompanying drawings used in the embodiments or technical solutions will be briefly introduced below.

[0037] Figure 1 This is an overall flowchart of the LiDARDraft method provided in this embodiment of the invention.

[0038] Figure 2This is a schematic diagram showing the effect and comparison of generating LiDAR point clouds based on images according to an embodiment of the present invention.

[0039] Figure 3 This is a schematic diagram illustrating the effect of generating LiDAR point clouds based on text and point cloud input, as provided in an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of the effect of editing point cloud scene based on modified 3D layout provided in the embodiment of the present invention.

[0041] Figure 5 This is a time-series consistency verification effect diagram provided by an embodiment of the present invention on the SemanticKITTI dataset.

[0042] Figure 6 This is a radar chart for cross-dataset performance evaluation provided in an embodiment of the present invention.

[0043] Figure 7 This is an analysis chart of the running time and memory usage efficiency of each functional module provided in the embodiments of the present invention.

[0044] Figure 8 This is a schematic diagram of an ablation experiment illustrating the influence of different control channels on the generation results provided in this embodiment of the invention; Figure 9 This is a flowchart of a method for generating point clouds for multimodal conditional lidar based on 3D layout guidance. Detailed Implementation

[0045] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0046] Reference Figure 1 , Figure 9 As shown, this embodiment of the invention provides a method for generating multimodal conditional lidar point clouds based on 3D layout guidance. The core of this method lies in solving modal differences through "unified layout representation", solving geometric consistency through "ray projection", and solving generation controllability through "ControlNet fine-tuning".

[0047] Figure 1 This demonstrates the complete process of converting multiple modal inputs, such as images, point clouds, and text, into a 3D scene layout (UnifiedLayout), generating control signals using raycasting, and inputting them into ControlNet to guide the unconditional diffusion model to generate target LiDAR point clouds.

[0048] Step S1: Unified Layout Representation of Multimodal Inputs.

[0049] In autonomous driving simulation, users may want to define scenarios in various ways. This embodiment designs a universal interface to convert heterogeneous inputs into homogeneous 3D layouts.

[0050] Text generation layout: For text input (e.g., "On a straight road, there is a car ahead and buildings on both sides"), this invention utilizes a pre-trained large language model (such as GPT-5) as the reasoning core. Through Prompt Engineering, the LLM is required to output an executable 3D modeling code (such as the Blender Python API). After execution, this code places corresponding geometric primitives in a virtual 3D space: cuboids represent vehicles and buildings, ellipsoids represent vegetation, and planes represent roads. This method leverages the powerful commonsense reasoning capabilities of the LLM to automatically complete the size, orientation, and relative positional relationships of objects in the scene.

[0051] Image generation layout: For monocular image input, this invention extracts the layout through a pipeline of "perception-clustering-reconstruction". First, a panoramic semantic segmentation map is obtained using the SegmentAnythingModel (SAM); a relative depth map is obtained using the DepthAnythingModel. Combined with camera intrinsics, pixels are back-projected into 3D space to form a pseudo-point cloud. Due to the noise in the pseudo-point cloud, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) density clustering algorithm is used to cluster the foreground object points, and the minimum oriented bounding box of each cluster is calculated, thereby generating a layout composed of 3D bounding boxes.

[0052] Figure 2 The generation effect of the method of the present invention when processing image input was compared with that of existing technologies (such as LiDARDiffusion), demonstrating the advantages of the present invention in terms of vehicle generation accuracy and consistency of open roads.

[0053] Point cloud generation layout: For sparse raw point cloud input, a 3D semantic segmentation network and clustering algorithm are directly applied to extract the pose and size of all objects of interest in the scene, reconstructing a parametric 3D layout. This allows users to edit existing point cloud data (such as moving vehicle positions) and then regenerate new point clouds.

[0054] Figure 3 The left half shows point clouds generated from text descriptions (such as "intersection" and "few vehicles") based on GPT; the right half shows diverse point cloud samples generated based on a single frame of point cloud input while maintaining layout consistency.

[0055] Step S2: Generation of high-dimensional control signals based on physical ray projection (Raycasting-based Control).

[0056] In order to accurately transfer the geometric constraints of the 3D layout to the 2D generation network, this embodiment uses raycasting technology.

[0057] The system sets the parameters of the virtual LiDAR (e.g., 64 lines, 360-degree horizontal field of view, and -24.9 to 2.0 degrees vertical field of view). Rays are emitted into space with the sensor center as the origin.

[0058] For each ray Calculate its nearest intersection with all geometric primitive meshes in the scene. .

[0059] Depth control signal: Calculate from origin to depth. Euclidean distance After normalization, a depth-range image is formed. .

[0060] Semantic control signals: Obtain the semantic labels of the hit primitives. Encoded as a semantic distance image .

[0061] Compared to traditional bird's-eye view (BEV) projection, the range image generated based on ray projection is completely pixel-aligned in topology with the original data format of LiDAR, providing the most direct and distortion-free geometric constraints.

[0062] Experiments show that if light projection is removed and BEV projection is used instead, the generated quality index FRD will deteriorate from 23.01 to 123.66, proving the necessity of this step.

[0063] Step S3: Layout-guided Generation based on ControlNet.

[0064] This embodiment uses a score-based diffusion model as the generation skeleton, with the specific architecture referring to StableDiffusion, but adapted to the characteristics of LiDAR data.

[0065] Unconditional pre-training: First, an unconditional point cloud generation model is trained on the KITTI-360 dataset (containing 69,580 samples). The input is... The distance image is obtained. The network structure adopts U-Net, which includes an encoder, a decoder, and skip connections. The training objective is to learn the prior distribution of the LiDAR point cloud so that it can recover a reasonable point cloud structure from Gaussian noise.

[0066] ControlNet tweaks: To introduce layout control, this invention does not directly modify the pre-trained U-Net parameters, but instead introduces ControlNet. ControlNet contains a "trainable copy" whose structure is completely identical to the U-Net encoder. The control signals (semantic map + depth map) generated in step S2 are input into ControlNet.

[0067] The feature maps output by ControlNet are processed through zero convolution layers—that is, layers with weights and biases initialized to 0. Convolutional layers – injected into the corresponding layers of the frozen U-Net decoder.

[0068] This design has two core advantages: Training stability: In the early stages of training, the output of the zero convolutional layer is 0, and the model is equivalent to the pre-trained unconditional model, avoiding "catastrophic forgetting"; as training progresses, the control signals gradually take effect.

[0069] Training efficiency: Only minor adjustments are needed to the parameters of the ControlNet part, resulting in a significantly faster convergence speed compared to training from scratch (high-quality results can be generated in just 5,000 iterations).

[0070] Performance evaluation and application: This invention has been extensively evaluated on the KITTI-360 and nuScenes datasets.

[0071] Generation quality: In terms of metrics such as Frechet Range Distance (FRD) and Minimum Matching Distance (MMD), This invention (LiDARDraft) significantly outperforms existing baseline methods (such as LatentDiffusion, LiDARDiffusion, and Text2LiDAR). For example, in text generation tasks, the FRD metric is reduced from 170.12 for Text2LiDAR to 23.88 (the lower the better).

[0072] Controllability and diversity: such as Figure 4 As shown, users can precisely control the distribution of objects in the generated results through text commands (such as "remove vehicles") or by directly modifying the 3D layout. At the same time, the model can automatically complete reasonable background details (such as road surface textures), achieving a balance between controllability and diversity.

[0073] Figure 4 The left side shows the input 3D layout (white boxes indicate vehicle positions), the middle and right sides show the corresponding generated point clouds (blue boxes indicate generated vehicles), and the results of scene editing by modifying the layout (such as removing vehicles) are also shown.

[0074] Operating efficiency: such as Figure 7 As shown, although the generation module (ControlNet) is the main force of GPU computing, the overall inference latency is within an acceptable range, supporting efficient data production.

[0075] Figure 7 The bar charts illustrate the runtime and GPU / CPU memory usage of modules such as depth estimation, semantic segmentation, clustering (DBSCAN), text layout generation, ray casting, and generative network (ControlNet).

[0076] Figure 8 The paper visually demonstrates the differences in geometric fidelity and structural accuracy of the generated LiDAR point clouds when using only layout, only semantic information, only depth information, and both semantic and depth information (Semantic+Depth).

[0077] Figure 5 The generation results for three consecutive time steps are shown, demonstrating that the point cloud sequence generated by this invention maintains good geometric consistency and visual realism in the time dimension.

[0078] Figure 6 The present invention demonstrates quantitative evaluation results on the KITTI-360 and nuScenes datasets for four different input conditions: semantic graph, text, image, and point cloud, in terms of metrics such as FRD, MMD, JSD, and FPD.

[0079] In summary, by constructing a unified layout representation and physically consistent control signals, this invention successfully breaks down the barriers between multimodal input and point cloud generation, providing an efficient, flexible, and high-quality data generation paradigm for autonomous driving simulation.

[0080] The scope of protection covered by this invention is not limited to the specific embodiments described above. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this invention should be included within the scope of protection of this invention.

Claims

1. A method for generating point clouds of multimodal conditional lidar based on 3D layout guidance, characterized in that, Including the following steps: Step S1: Unified layout representation of multimodal input: Obtain the user's multimodal input information, and use the parsing module for specific modalities to uniformly convert the multimodal input information into a standardized 3D scene layout representation; Step S2: Generation of control signals based on physical ray projection: Based on the 3D scene layout representation, the 3D geometric layout representation is projected into a semantic distance image and a depth distance image containing control signals; Step S3: Conditional diffusion generation based on ControlNet: The semantic distance image and depth distance image are used as control conditions and input into the conditional branch based on the ControlNet architecture; The pre-trained unconditional diffusion model is modulated using ControlNet to output a high-fidelity lidar point cloud that meets the input conditions.

2. The method for generating multimodal conditional lidar point clouds based on three-dimensional layout guidance according to claim 1, characterized in that, In step S1, the input information includes at least two of the following: text description, monocular image data, and single-frame point cloud.

3. The method for generating multimodal conditional lidar point clouds based on three-dimensional layout guidance according to claim 1, characterized in that, Different layout transformation strategies are adopted for inputs of different modalities.

4. The method for generating multimodal conditional lidar point clouds based on three-dimensional layout guidance according to claim 1, characterized in that, In step S2, a ray casting algorithm is used to obtain semantic distance images and depth distance images.

5. The method for generating multimodal conditional lidar point clouds based on three-dimensional layout guidance according to claim 1, characterized in that, The training of the unconditional diffusion model includes an unconditional base training phase and a conditional fine-tuning phase.

Images