Automatic driving generative model for reactive closed loop evaluation

Abstract

The invention discloses an automatic driving generation model for reactive closed-loop evaluation, the automatic driving generation model comprises a behavior controller and a generative renderer, the behavior controller is based on a classic rule system IDM, and the generative renderer is based on a diffusion model; the workflow of the automatic driving generation model is as follows: S1, an end-to-end automatic driving model takes a current frame look-around picture as input, and outputs a planned future driving route of an own vehicle; on the framework, the simulation framework provided by the invention realizes high-frequency interaction with an end-to-end driving model by utilizing the generation capability of the generative model in an autoregression mode, and generates images consistent in time; the provided time consistency module, projection-based object level control and retrieval-based scene level control can significantly improve the model generation fidelity.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving simulation technology, and in particular to an autonomous driving generation model for reactive closed-loop evaluation. Background Art

[0002] 1. End-to-end autonomous driving evaluation Unlike traditional modular autonomous driving systems, the goal of end-to-end autonomous driving is to directly predict future trajectories based on raw sensor data. As a new direction, the benchmarking of end-to-end autonomous driving models remains an open problem. Existing evaluation methods can be divided into three categories. (1) Open-loop evaluation usually measures the displacement error between the predicted trajectory and the recorded expert trajectory. (2) Closed-loop simulation uses simulators such as CARLA to evaluate planning performance in a closed-loop manner. (3) The recent work NAVSIM proposes a method between open-loop and closed-loop evaluation. It proposes a non-reactive simulation.

[0003] 2. Autonomous Driving Generation Model There has been a lot of work exploring the application of generative models in autonomous driving, such as generating realistic sensor data, with most of the work focusing on generating novel driving scenes, mainly for data augmentation in perception tasks. These works include generating driving scenes conditioned on static bird's-eye views for object detection and online mapping, generating video clips for tasks such as tracking and trajectory prediction, and using video diffusion models as world models to implicitly simulate driving scenes.

[0004] At present, there are still the following problems in autonomous driving simulation: 1. End-to-end autonomous driving evaluation (1) Open-loop evaluation has problems such as data set imbalance, heavy reliance on expert self-state, and distribution bias.

[0005] (2) From the perspective of rendering and behavior, there is a clear gap between common simulators such as CARLA and the real world. Specifically, the rendering materials are not realistic enough and the behavior of other cars is relatively rigid.

[0006] (3) Although NAVSIM is closer to closed-loop metrics than open-loop, it fails to capture the model’s planning capabilities in scenarios where agent interactions have a critical impact on planning outcomes, such as merging into traffic or changing lanes in dense traffic. Although NAVSIM uses real-world visual data, it is limited to open-loop simulations and assumes that other agents are not reactive, and therefore cannot simulate interactive driving scenarios.

[0007] 2. Autonomous Driving Generation Model Single-frame image generation lacks temporal consistency constraints, while video generation models cannot interact step-by-step with end-to-end autonomous driving models and therefore cannot be used for closed-loop interaction simulation. Summary of the invention

[0008] The purpose of the present invention is to address the shortcomings of the prior art and to propose an autonomous driving generation model for reactive closed-loop evaluation.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions: An autonomous driving generative model for reactive closed-loop evaluation, the autonomous driving generative model comprising a behavior controller and a generative renderer, wherein the behavior controller is based on a classic rule system IDM and the generative renderer is based on a diffusion model; The workflow of the autonomous driving generation model is as follows: S1: End-to-end autonomous driving model, which takes the surround image of the current frame as input and outputs the future driving route planned by the vehicle; S2: Behavior controller, which takes the driving route of the ego vehicle output by the end-to-end autonomous driving model as input, deduce the reactions of other vehicles in the scene, and output the positions of the ego vehicle and other vehicles at the next moment; S3: Generative renderer, which takes the positions of the self-vehicle and other vehicles at the next moment and map elements as input to generate the corresponding surround image; S4: The above steps S1-S3 are repeatedly executed in turn to complete the simulation of the driving process of the end-to-end autonomous driving model, and then an evaluation is given; The autonomous driving generation model is generated by autoregression, that is, the surround image l of each frame t All are based on the previous frame of the picture l t-1 Obtained through ControlNet encoding.

[0010] Preferably, the generative renderer comprises a temporal consistency module, a projection-based object-level control module and a retrieval-based scene-level control module.

[0011] Preferably, the working method of the temporal consistency module is to first use the Stable Diffusion VAE encoder to convert the previous frame image l t-1 Encoded into the latent space, which is then passed to the ControlNet encoder. Finally, the hidden features output by each layer of ControlNet are directly added to the corresponding layers of the U-Net encoder.

[0012] Preferably, the VAE encoder of Stable Diffusion is used to convert the previous frame image l t-1 Encode into the latent space and obtain the conditional latent variable z prev , in zprev Random Gaussian noise is added to the signal and the noise level is also input to the ControlNet encoder, which is expressed as:

[0013] Among them, ε represents the ControlNet encoder; is randomly sampled Gaussian noise, , n is the time step t in the diffusion model.

[0014] Preferably, during the simulation process of the projection-based object-level control module, the behavior controller provides object-level information: the vehicle three-dimensional bounding box B t and vectorized map elements M t ; The above two object-level information are used as control information for the generative renderer, and the 3D bounding box B is transformed using the camera intrinsics and extrinsics. t and vectorized map elements M t Projection from the LiDAR coordinate system into a 2D perspective image; The projected discrete coordinates are then plotted into a set of binary masks of the same size, with the input image being and ; The above two object-level control information are combined and encoded into the latent space through a convolutional network; These encoded control signals are injected into the denoising process using the ControlNet encoder. The plant-level control is expressed as: .

[0015] Preferably, the retrieval-based scene-level control module works by retrieving two frames of images with the closest physical distance from each other in the database to generate each current frame of the image; The two frames of retrieved pictures are encoded through the image encoder and input into ControlNet as the key and value in the reference attention module so that the generation of the current frame can find the correspondence of the background.

[0016] Preferably, since the spatial relationship between the current frame and the two reference frames can be explicitly calculated based on the coordinate transformation, the pixel spatial relationship information is injected into the reference attention mechanism; A discrete grid P of size (H, W, D, 4) is calculated for all images using pixel 3D position encoding, where D is the number of points sampled along the depth axis; Then, using the current frame P ego and two reference frames P ref The camera transformation matrix transforms the mesh from camera space to 3D coordinate system; Finally, in the cross attention of ControlNet, P ego As Query, it is the position encoding of the query, and P ref As the key, the position encoding can be expressed as:

[0017] This way, the current frame can find correspondences in the reference image, thereby tracking the static background and generating transformed pixels.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. In terms of framework, the simulation framework proposed in this application utilizes the generation capability of the generative model in an autoregressive manner to achieve high-frequency interaction with the end-to-end driving model and generate temporally consistent images; 2. The proposed time-consistent module, projection-based object-level control, and retrieval-based scene-level control can significantly improve the model generation fidelity. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more specifically and intuitively illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0020] Figure 1 A schematic diagram of the architecture of the autonomous driving generation model for reactive closed-loop evaluation proposed in the present invention; Figure 2 A schematic diagram of a temporal consistency module of an autonomous driving generation model for reactive closed-loop evaluation proposed in the present invention; Figure 3 A schematic diagram of the training collapse of the autonomous driving generation model for reactive closed-loop evaluation proposed in the present invention; Figure 4 A revised schematic diagram of the autonomous driving generation model for reactive closed-loop evaluation proposed in the present invention; Figure 5 Schematic diagram of projection-based object-level control for the autonomous driving generation model for reactive closed-loop evaluation proposed in the present invention; Figure 6 Schematic diagram of the retrieval-based scene-level control of the autonomous driving generation model for reactive closed-loop evaluation proposed in the present invention Figure 1 ; Figure 7 Schematic diagram of the retrieval-based scene-level control of the autonomous driving generation model for reactive closed-loop evaluation proposed in the present invention Figure 2 . DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0022] Reference Figure 1-2 , an autonomous driving generative model for reactive closed-loop evaluation, which can be used for closed-loop simulation of an end-to-end autonomous driving model. The autonomous driving generative model includes a behavior controller and a generative renderer, wherein the behavior controller is based on a classic rule system IDM, and the generative renderer is based on a diffusion model; The workflow of the autonomous driving generation model is as follows: S1: End-to-end autonomous driving model, which takes the surround image of the current frame as input and outputs the future driving route planned by the vehicle; S2: Behavior controller, which takes the driving route of the ego vehicle output by the end-to-end autonomous driving model as input, deduce the reactions of other vehicles in the scene, and output the positions of the ego vehicle and other vehicles at the next moment; S3: Generative renderer, which takes the positions of the self-vehicle and other vehicles at the next moment and map elements as input to generate the corresponding surround image; S4: The above steps S1-S3 are repeatedly executed in turn to complete the simulation of the driving process of the end-to-end autonomous driving model, and then an evaluation is given; The autonomous driving generation model is generated by autoregression, that is, the surround image l of each frame t All are based on the previous frame of the picture l t-1 Obtained through ControlNet encoding.

[0023] The generative renderer includes a temporal consistency module, a projection-based object-level control module, and a retrieval-based scene-level control module.

[0024] The temporal consistency module works by first using the Stable Diffusion VAE encoder to transform the previous frame l t-1 Encoded into the latent space, which is then passed to the ControlNet encoder. Finally, the hidden features output by each layer of ControlNet are directly added to the corresponding layers of the U-Net encoder.

[0025] Reference Figure 3 , the above will lead to the problem of cumulative error and crash during use: during training, the previous frame uses the stored real-world image; while during use, the previous frame is generated by the model itself, which is different from the real-world image. Therefore, due to the recursive nature of autoregressive generation, errors will continue to accumulate, eventually leading to generation crash.

[0026] The above problem is caused by the generative renderer's over-reliance on the quality of the previous frame. We propose to solve this problem by using Gaussian blur for noise modulation during training to allow the generative renderer to adapt to the previous frame with degraded quality.

[0027] Specifically, during the training process, the Stable Diffusion VAE encoder is used to convert the previous frame image l t-1 Encode into the latent space and obtain the conditional latent variable z prev , in z prev Random Gaussian noise is added to the signal and the noise level is also input to the ControlNet encoder, which is expressed as:

[0028] Among them, ε represents the ControlNet encoder; is randomly sampled Gaussian noise, , the noise addition strategy is similar to the training strategy of the diffusion model, and n is the time step t in the diffusion model.

[0029] In order to further avoid the accumulation of high-frequency errors, we perform Gaussian blur processing on the previous frame image. The proposed technology can effectively alleviate the degradation problem. Figure 4 .

[0030] Reference Figure 5 , based on the projection object-level control module During the simulation, the behavior controller provides object-level information: the vehicle 3D bounding box B t and vectorized map elements M t ; The above two object-level information are used as control information for the generative renderer, and the 3D bounding box B is transformed using the camera intrinsics and extrinsics. t and vectorized map elements M t Projection from the LiDAR coordinate system into a 2D perspective image; The projected discrete coordinates are then plotted into a set of binary masks of the same size, with the input image being and ; The object semantic information is integrated by assigning each category its own dedicated channel, and the above two object-level control information are combined and encoded into the latent space through a convolutional network; These encoded control signals are injected into the denoising process using the ControlNet encoder. The plant-level control is expressed as: .

[0031] Generative models are prone to hallucinations. To solve this problem, we allow the model to refer to nearby static scene information in the database, such as high-rise buildings and landmark buildings, to reduce hallucinations.

[0032] Specifically, the retrieval-based scene-level control module works by retrieving the two frames of images with the closest physical distance from each other in the database to generate each current frame of the image; The two frames of retrieved images are encoded through an image encoder such as ResNet and input into ControlNet as the key and value in the reference attention module so that the generation of the current frame can find the corresponding relationship of the background, such as Figure 6 The orange part is shown in the middle, and the blue part is the original part of ControlNet.

[0033] Since the spatial relationship between the current frame and the two reference frames can be explicitly calculated based on the coordinate transformation, the pixel spatial relationship information is injected into the reference attention mechanism; A discrete grid P of size (H, W, D, 4) is calculated for all images using pixel 3D position encoding, where D is the number of points sampled along the depth axis; Then, using the current frame P ego and two reference frames P ref The camera transformation matrix transforms the mesh from camera space to 3D coordinate system; Finally, in the cross attention of ControlNet, P ego As Query, it is the position encoding of the query, and P ref As the key, the position encoding can be expressed as:

[0034] In this way, the current frame can find the corresponding relationship in the reference image, thereby tracking the static background and generating the transformed pixels, referring to Figure 7 .

[0035] 1. In terms of framework, the simulation framework proposed in this application utilizes the generation capability of the generative model in an autoregressive manner to achieve high-frequency interaction with the end-to-end driving model and generate temporally consistent images; 2. The proposed time-consistent module, projection-based object-level control, and retrieval-based scene-level control can significantly improve the model generation fidelity.

[0036] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A generative model for autonomous driving for reactive closed-loop evaluation, characterized in that: The autonomous driving generation model includes a behavior controller and a generative renderer, wherein the behavior controller is based on the classic rule system IDM and the generative renderer is based on the diffusion model; The workflow of the autonomous driving generation model is as follows: S1: End-to-end autonomous driving model, which takes the surround image of the current frame as input and outputs the future driving route planned by the vehicle; S2: Behavior controller, which takes the driving route of the ego vehicle output by the end-to-end autonomous driving model as input, deduce the reactions of other vehicles in the scene, and output the positions of the ego vehicle and other vehicles at the next moment; S3: Generative renderer, which takes the positions of the self-vehicle and other vehicles at the next moment and map elements as input to generate the corresponding surround image; S4: The above steps S1-S3 are repeatedly executed in turn to complete the simulation of the driving process of the end-to-end autonomous driving model, and then an evaluation is given; The autonomous driving generation model is generated by autoregression, that is, the surround image l of each frame t All are based on the previous frame of the picture l t-1 Obtained through ControlNet encoding.

2. The autonomous driving generation model for reactive closed-loop evaluation according to claim 1, characterized in that: The generative renderer includes a temporal consistency module, a projection-based object-level control module, and a retrieval-based scene-level control module.

3. The autonomous driving generation model for reactive closed-loop evaluation according to claim 2, characterized in that: The working method of the temporal consistency module is to first use the Stable Diffusion VAE encoder to convert the previous frame image l t-1 Encoded into the latent space, which is then passed to the ControlNet encoder. Finally, the hidden features output by each layer of ControlNet are directly added to the corresponding layers of the U-Net encoder.

4. The autonomous driving generation model for reactive closed-loop evaluation according to claim 3, characterized in that: Use Stable Diffusion's VAE encoder to encode the previous frame l t-1 Encode into the latent space and obtain the conditional latent variable z prev , in z prev Random Gaussian noise is added to the signal and the noise level is also input to the ControlNet encoder, which is expressed as: Among them, ε represents the ControlNet encoder; is randomly sampled Gaussian noise, , n is the time step t in the diffusion model.

5. The autonomous driving generation model for reactive closed-loop evaluation according to claim 4, characterized in that: The projection-based object-level control module provides object-level information during the simulation process: the vehicle 3D bounding box B t and vectorized map elements M t ; The above two object-level information are used as control information for the generative renderer, and the 3D bounding box B is transformed using the camera intrinsics and extrinsics. t and vectorized map elements M t Projection from the LiDAR coordinate system into a 2D perspective image; The projected discrete coordinates are then plotted into a set of binary masks of the same size, with the input image being and ; The above two object-level control information are combined and encoded into the latent space through a convolutional network; These encoded control signals are injected into the denoising process using the ControlNet encoder. The plant-level control is expressed as: 。 6. The autonomous driving generation model for reactive closed-loop evaluation according to claim 5, characterized in that: The working mode of the retrieval-based scene-level control module is to generate each current frame of the picture by retrieving two frames of pictures with the closest physical distance between them in the database; The two frames of retrieved pictures are encoded through the image encoder and input into ControlNet as the key and value in the reference attention module so that the generation of the current frame can find the correspondence of the background.

7. The autonomous driving generation model for reactive closed-loop evaluation according to claim 6, characterized in that: Since the spatial relationship between the current frame and the two reference frames can be explicitly calculated based on the coordinate transformation, the pixel spatial relationship information is injected into the reference attention mechanism; A discrete grid P of size (H, W, D, 4) is calculated for all images using pixel 3D position encoding, where D is the number of points sampled along the depth axis; Then, using the current frame P ego and two reference frames P ref The camera transformation matrix transforms the mesh from camera space to 3D coordinate system; Finally, in the cross attention of ControlNet, P ego As Query, it is the position encoding of the query, and P ref As the key, the position encoding can be expressed as: This way, the current frame can find correspondences in the reference image, thereby tracking the static background and generating transformed pixels.

Images