Automatic driving simulation system based on cumulative point cloud picture structure constraint

Abstract

An automatic driving simulator based on cumulative point cloud atlas structural constraint comprises a point cloud skeleton system based on log data and a generator, point cloud data in log records are decomposed into a dynamic foreground and a static background, and dynamic updating is supported while scene geometric information is reserved by accumulatively constructing a skeleton respectively; point cloud prior (a color map / depth map), previous frame information (hidden features with noise) and a long-distance background are integrated, and multi-condition balance is realized through ControlNet to improve a generation effect; through introduction of data enhancement guided by a point cloud skeleton, overfitting caused by incomplete scanning of a foreground is relieved, a latent variable of a frame before noise reduces error accumulation in an autoregression process, and the cross-frame / cross-view consistency of a dynamic agent is ensured by ID cycle consistency loss.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and particularly relates to an automatic driving simulation system based on accumulated point cloud graph structure constraint. BACKGROUND

[0002] Existing automatic driving benchmark tests can be roughly divided into two categories: one is open-loop benchmark test which cannot provide dynamic feedback according to the decision made by an end-to-end automatic driving model, resulting in evaluation deviation. The other is to provide a system closed-loop simulation, but there is still a significant gap between the rendering and behavior and the real world. SUMMARY

[0003] The present application proposes an automatic driving simulator based on accumulated point cloud graph structure constraint to solve the problems that the prior art cannot balance scene geometry and dynamic control, overfitting and poor consistency caused by incomplete scanning, and the generated results are easy to be inconsistent when the evaluation trajectory deviates greatly from the original log. The point cloud data in the log record is decomposed into dynamic foreground and static background, and the skeleton is constructed by accumulation respectively, so as to retain the scene geometry information while supporting dynamic update. The point cloud prior (color map / depth map), previous frame information (noisy hidden features) and long-distance background are integrated, and ControlNet is used to realize multi-condition balance to improve the generation effect. The point cloud skeleton guided data enhancement is introduced to alleviate the overfitting caused by incomplete scanning of the foreground, the noise previous frame latent variable reduces the error accumulation in the autoregressive process, and the ID cycle consistency loss guarantees the consistency of the dynamic agent across frames and views.

[0004] The present application is implemented by the following technical solutions:

[0005] The present application relates to an automatic driving simulator based on accumulated point cloud graph structure constraint, comprising a point cloud skeleton system based on log data and a generator, wherein: the point cloud skeleton system decouples and accumulates scene-level point cloud into background and foreground point cloud, and obtains an instant updated point cloud skeleton by editing point cloud through the boundary box of the current time step, and provides a color map and a depth map as constraint conditions for the generation of the current frame; the generator integrates previous frame information, long-distance city background and constraint conditions from the point cloud skeleton system according to ControlNet, and generates an automatic driving simulation scene by combining graph structure constraint, noisy previous conditions and ID cycle consistency loss.

[0006] The instant updated point cloud skeleton is realized by the following manner: the point cloud skeleton system is split into background point cloud skeleton and foreground point cloud skeleton according to the off-line log data containing point cloud, image and 3D bounding box label according to dynamic and static attributes, the background point cloud skeleton is obtained by accumulating point cloud of static background point cloud outside the 3D box label, the point cloud accumulation of the same intelligent agent is realized by combining the point cloud in the 3D box with the ID of tracking different intelligent agents, and the unswept area is supplemented by the clustering template.

[0007] The background point cloud skeleton corresponds to static background elements such as roads and buildings, and the foreground point cloud skeleton corresponds to dynamic intelligent agents such as vehicles and pedestrians.

[0008] The generator comprises a point cloud skeleton input module, a data enhancement module and a loss function module, wherein: the point cloud skeleton interaction module generates new color graphs and depth graph projections into the camera view as the latest input conditions of the generator according to the real-time updated point cloud skeleton. The data enhancement module enhances the generalization ability to the trajectory offset scene by simulating the view angle change by applying random offset to the foreground point cloud. The loss function module ensures the ID consistency of the dynamic intelligent agent in the cross-frame and cross-view generation by calculating the ID cyclic consistency loss.

[0009] Technical effects

[0010] The present application takes the point cloud skeleton accumulated by off-line logs as the core condition of the diffusion model, decomposes the point cloud data in the off-line logs into dynamic foreground point cloud and static background point cloud, constructs the skeleton by point cloud accumulation respectively, and supports 3D box editing; for the problem of mutual conflict in traditional multi-condition generation, the ControlNet realizes the multi-condition balance of point cloud prior (color graph / depth graph), previous frame information (noisy hidden features) and long-distance background, and designs ID cyclic consistency loss to enhance the generation effect. Compared with the prior art, the present application realizes the precise control of scene structure and dynamic target, and significantly improves the generation effect of the automatic driving simulator through the graph structure constraint based on the point cloud skeleton and the multi-dimensional condition collaborative mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 It is a structural schematic diagram of the present application;

[0012] Figure 2 It is a flowchart of point cloud skeleton construction and utilization;

[0013] Figure 3 It is a flowchart of the generator. DETAILED DESCRIPTION

[0014] As Figure 1As shown, the embodiment relates to an automatic driving simulator based on accumulated point cloud structure constraints, including a point cloud skeleton system and a generator based on log data.

[0015] As shown, Figure 2 and Figure 3 As shown, the embodiment is based on the above-mentioned simulator automatic driving simulation method, which accumulates the point cloud skeleton based on the original point cloud and image by the point cloud skeleton system, and divides it into foreground point cloud skeleton and background point cloud skeleton through 3D box, and the depth map and color map obtained are used as the input conditions of the generator; the generator extracts the most accurate elements from different condition inputs through ControlNet, and introduces cycle loss optimization to generate the target.

[0016] The method specifically includes:

[0017] Step 1, the point cloud skeleton system decouples and accumulates the scene-level point cloud into background point cloud skeleton and foreground point cloud skeleton and projects to obtain color map and depth map, while supporting point cloud skeleton editing based on 3D box, as shown Figure 2 Specifically includes:

[0018] 1.1 Obtain multi-frame point cloud and image data from offline logs, and project the point cloud to a self-centered coordinate system to obtain color point cloud.

[0019] 1.2 Based on the 3D bounding box and ID tracking information in the offline log, the point cloud obtained in step 1.1 is segmented into dynamic foreground and static background.

[0020] The dynamic foreground includes but is not limited to vehicles, pedestrians, etc., and the static background includes but is not limited to roads, buildings, etc.

[0021] The point cloud decoupling specifically includes: , wherein: represents the complete point cloud set of the current scene in the t-th frame, represents the point cloud set located within the 3D box in the t-th frame, represents the point cloud set located outside the 3D box in the t-th frame.

[0022] 1.3 Accumulate the static background point cloud in the global coordinate system, store it after voxel downsampling, and form the background static skeleton.

[0023] 1.4 Accumulate the dynamic foreground point cloud in the local coordinate system, retain the geometric features of each vehicle, pedestrian, etc. foreground agent, and form a dynamically editable foreground skeleton.

[0024] 1.5 In the simulation process, according to the dynamic position (bounding box) output by the traffic simulator, the foreground skeleton obtained in step 1.4 is pasted into the background skeleton obtained in step 1.3, and the point cloud skeleton is updated.

[0025] 1.6 Project the updated point cloud skeleton to the camera plane to obtain a color map for providing background details and a depth map for enhancing geometric consistency based on clustered templates as the core conditions of the generative model.

[0026] Step 2, train the diffusion model to selectively extract the most accurate elements from different conditional inputs through ControlNet, filtering out irrelevant or conflicting information, such as Figure 3 as shown, specifically including:

[0027] 2.1 Use the color map and depth map generated based on the updated point cloud skeleton from step 1 as the current conditions to constrain scene geometry and details, while training the model to add a small spatial offset (X / Y axis normal distribution noise) to the foreground point cloud of step 1, avoiding model overfitting to color features, and strengthening dependence on the bounding box and depth.

[0028] 2.2 Encode the previous frame image in the log as a latent variable and add slight noise to alleviate error accumulation in the autoregressive process and enhance temporal coherence.

[0029] 2.3 Extract anchor images from the offline log at a fixed frequency, remove close-range vehicles, and inject background information through cross-attention.

[0030] 2.4 In the low-noise diffusion step, enforce the foreground region of the generated image to be consistent with the real image by calculating the ID cycle consistency loss.

[0031] The ID cycle consistency loss where: , is the reconstructed image, is the cumulative decay coefficient of the diffusion model at time step t', is the noisy latent vector at time step t', is the noise prediction network of the diffusion model, is the foreground agent mask, is the low-noise time step mask, is the real image.

[0032] 2.5 Fuse multiple conditions through ControlNet as the input of the diffusion model to finally generate multi-view images of the current frame, which are fed back to the end-to-end autonomous driving model for the next step of decision-making.

[0033] Through specific experiments, the generation performance of the autonomous driving simulator is evaluated on the nuScenes validation dataset and the Log-Replay of the nuPlan-mini dataset based on the Stable Diffusion v1.5 (SDv1.5) generation model architecture, where the nuPlan-mini dataset is fixed as “a driving scene in {city}” (the city name in the dataset is replaced, such as “Singapore”); the nuScenes dataset directly uses the original annotated scene description without additional text constraints. At the same time, the increased offset trajectory is the trajectory of “right lane change”, which is used to simulate the real scene of “autonomous driving system active decision leading to trajectory offset”.

[0034] As shown in Table 1, the generation performance of the autonomous driving simulator is evaluated on the nuScenes validation dataset and the Log-Replay of the nuPlan dataset, and the results show that the present application achieves the most advanced performance in terms of generation quality, achieving lower FID and FVD scores. FID refers to measuring the quality and diversity of generated images by comparing the distribution in the feature space of generated images and real images; FVD refers to measuring the quality and diversity of generated videos by comparing the distribution in the feature space of generated videos and real videos.

[0035] Table 1 Comparison of generation performance evaluation results

[0036] As shown in Table 1, the generation performance of the autonomous driving simulator is evaluated on the nuScenes validation dataset and the Log-Replay of the nuPlan dataset, and the results show that the present application achieves the most advanced performance in terms of generation quality, achieving lower FID and FVD scores.

[0037] To fully reflect the simulation capability of the autonomous driving model, the model is evaluated on the offset trajectory, as shown in Table 2, where: IoU is obtained by calculating the ratio of the intersection and union of the predicted results and the true label, and mIoU is the average value of the IoU of all classes, which can comprehensively evaluate the segmentation accuracy of the model on each class.

[0038] Table 2 Comparison of evaluation results on offset trajectory

[0039] As shown in Table 2, the model is evaluated on the offset trajectory, and the present application also achieves the most advanced effect in terms of segmentation accuracy on each class, with an FID score of 27.73 and an mIoU value of 61.31%.

[0040] The above specific embodiments can be partially adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, each implementation within the scope is subject to the present application.

Claims

1. An autonomous driving simulator based on accumulated point cloud graph structure constraints, characterized in that, The application relates to a point cloud skeleton system and a generator based on log data, wherein the point cloud skeleton system decouples and accumulates scene-level point clouds into background and foreground point clouds, and edits the point clouds through a current time step boundary box to obtain an instant-updated point cloud skeleton, and provides a color map and a depth map for generation of a current frame as constraint conditions; the generator integrates previous frame information, a long-distance urban background and constraint conditions from the point cloud skeleton system through a ControlNet, and generates an automatic driving simulation scene by combining a graph structure constraint, a previous condition with noise and an ID cycle consistency loss. The instant-updated point cloud skeleton is achieved by the following method: the point cloud skeleton system splits background and foreground point cloud skeletons according to dynamic and static attributes based on offline log data containing point clouds, images and 3D boundary box annotations, accumulates static background point clouds outside the 3D box to obtain the background point cloud skeleton, accumulates point clouds in the 3D box and combines the ID of different agents to realize point cloud accumulation of the same agent, and supplements the unscanned area through a clustering template. The generator comprises a point cloud skeleton input module, a data enhancement module and a loss function module, wherein the point cloud skeleton interaction module generates new color maps and depth maps projected to the camera view angle as the latest input conditions of the generator according to the instant-updated point cloud skeleton, the data enhancement module enhances the generalization ability to the trajectory offset scene by simulating the simulation view angle change by applying random offset to the foreground point cloud, and the loss function module ensures the ID consistency of the dynamic agent in the cross-frame and cross-view generation by calculating the ID cycle consistency loss.

2. The automatic driving simulator based on accumulated point cloud map structure constraint according to claim 1, characterized in that, The depth map and the color map obtained by the point cloud skeleton system based on the original point cloud and the image accumulation point cloud skeleton are divided into foreground and background point cloud skeletons through a 3D box, and are used as input conditions of the generator; 3. The automatic driving simulator based on accumulated point cloud map structure constraint according to claim 1, characterized in that, The generator extracts the most accurate elements from different condition inputs through a ControlNet, and introduces a cycle loss to optimize the generation target.

4. A driving simulation method based on the simulator according to any one of claims 1 to 3, characterized by, The point cloud skeleton system decouples and accumulates scene-level point clouds into background and foreground point cloud skeletons and projects to obtain color maps and depth maps, and supports point cloud skeleton editing based on a 3D box, and specifically comprises the following steps: 1.1 obtaining multiple frames of point cloud and image data from offline logs, and projecting the point cloud to a self-centered coordinate system to obtain a color point cloud; 5. The driving simulation method according to claim 4, characterized in that, 1.2 segmenting the point cloud obtained in step 1.1 into dynamic foreground and static background based on 3D boundary boxes and ID tracking information in the offline logs; 1.3 accumulating the static background point cloud in a global coordinate system, storing after voxel downsampling to form a background static skeleton; 1.4 accumulating the dynamic foreground point cloud in a local coordinate system, retaining the geometric features of each vehicle, pedestrian and other foreground agents to form a dynamically editable foreground skeleton; The point cloud decoupling specifically refers to: Wherein: represents a complete point cloud set of a current scene in a t-th frame, represents a point cloud set of all points located in a 3D frame in the t-th frame, represents a point cloud set of all points located outside the 3D frame in the t-th frame. 1.5 during simulation, pasting the foreground skeleton obtained in step 1.4 into the background skeleton obtained in step 1.3 according to the dynamic position, i.e. the boundary box, output by a traffic simulator, and updating the point cloud skeleton; ​ ​ 1.6 Project the updated point cloud skeleton onto the camera plane to obtain a color map for providing background details and a depth map for enhancing geometric consistency based on clustered templates as the core conditions of the generative model.

6. The driving simulation method according to claim 4, characterized by, Train the diffusion model to selectively extract the most accurate elements from different conditional inputs through ControlNet, filtering out irrelevant or conflicting information, including: 2.1 Use the color map and depth map generated based on the updated point cloud skeleton from step 1 as the current conditions to constrain scene geometry and details, while training the model with a slight spatial offset, i.e., X / Y axis normal distribution noise, added to the foreground point cloud of step 1 to avoid model overfitting to color features and strengthen dependence on bounding boxes and depth; 2.2 Encode the previous frame image recorded in the log as a latent variable and add slight noise to alleviate error accumulation in the autoregressive process and enhance temporal coherence; 2.3 Extract anchor images from the offline log at a fixed frequency, remove close-range vehicles, and inject background information through cross-attention; 2.4 In the low-noise diffusion step, enforce the foreground region of the generated image to be consistent with the real image by calculating the ID cyclic consistency loss; The ID cycle consistency loss wherein: , is the reconstructed image, is the accumulated decay coefficient of the diffusion model at time step t', is the noisy latent vector at time step t', is the noise prediction network of the diffusion model, is the foreground agent mask, is the low-noise time step mask, is the real image; 2.5 Fuse multiple conditions through ControlNet as the input of the diffusion model to finally generate multi-view images of the current frame, which are fed back to the end-to-end autonomous driving model for the next decision.

Images