Apparatus for generating a reaction closed loop assessment of an autonomous driving model
By combining a behavior controller and a generative renderer, along with temporal consistency and scene-level control, the problem of interactive scene simulation in end-to-end autonomous driving evaluation is solved, achieving time-consistent image generation and high-fidelity evaluation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-10
- Publication Date
- 2026-03-27
AI Technical Summary
Existing end-to-end autonomous driving evaluation methods suffer from problems such as imbalanced datasets, reliance on expert ego states, distribution bias, large discrepancies between simulators and the real world, and inability to simulate interactive driving scenarios. The generated models lack time consistency and cannot perform closed-loop interactive simulations.
The system employs a combination of a behavior controller and a generative renderer. The behavior controller is based on the classic rule system IDM, while the generative renderer is based on a diffusion model. It generates surround view images through an autoregressive approach and combines a time consistency module, object-level control based on projection, and scene-level control based on retrieval to achieve high-frequency interaction and time consistency.
It enables high-frequency interaction with the end-to-end driving model, generates time-consistent images, significantly improves the fidelity of model generation, and allows for reactive closed-loop evaluation.
Smart Images
Figure CN119623298B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of automatic driving simulation, and particularly relates to a device for generating a reactive closed-loop evaluation of an automatic driving model. BACKGROUND
[0002] 1. End-to-end autonomous driving evaluation
[0003] Unlike traditional modular autonomous driving systems, the goal of end-to-end autonomous driving is to directly predict future trajectories from raw sensor data. As a new direction, the benchmarking of end-to-end autonomous driving models is still an open problem. Existing evaluation methods can be divided into three categories. (1) Open-loop evaluation usually measures the displacement error between the predicted trajectory and the recorded expert trajectory. (2) Closed-loop simulation uses simulators such as CARLA to evaluate planning performance in a closed-loop manner. (3) The recent work NAVSIM proposes a method between open-loop and closed-loop evaluation. It proposes a non-reactive simulation.
[0004] 2. Autonomous driving generative model
[0005] There has been a lot of work exploring the application of generative models in autonomous driving, such as generating realistic sensor data, most of which focuses on generating novel driving scenarios, mainly for data augmentation in perception tasks. These works include generating static bird's eye view driving scenes as conditions for target detection and online mapping, generating video clips for tasks such as tracking and trajectory prediction, and using video diffusion models as world models to implicitly simulate driving scenarios.
[0006] Current autonomous driving simulation still has the following problems:
[0007] 1. End-to-end autonomous driving evaluation
[0008] (1) Open-loop evaluation has problems such as dataset imbalance, heavy reliance on expert self-state, and distribution shift.
[0009] (2) From the perspectives of rendering and behavior, common simulators such as CARLA have obvious gaps with the real world. Specifically, the rendering materials are not realistic enough, and the behavior of other vehicles is rather rigid.
[0010] (3) Although NAVSIM is closer to closed-loop indicators compared to open-loop, it cannot capture the planning ability of the model in scenarios where agent interaction has a key impact on planning results, such as merging into traffic flow or changing lanes in dense traffic. Although NAVSIM uses real-world visual data, it is limited to open-loop simulation and assumes that other agents are not reactive, so it cannot simulate interactive driving scenarios.
[0011] 2. Autonomous driving generative model
[0012] Single-frame image generation lacks the constraint of temporal consistency, and the video generation model cannot interact with the end-to-end autonomous driving model step by step, so it cannot be used for closed-loop interactive simulation. SUMMARY
[0013] The purpose of the present application is to solve the problems existing in the prior art, and the device for generating a reactive closed-loop evaluation of an autonomous driving model is proposed.
[0014] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:
[0015] The device for generating a reactive closed-loop evaluation of an autonomous driving model comprises a behavior controller and a generative renderer, wherein the behavior controller is based on a classic rule system IDM, and the generative renderer is based on a diffusion model.
[0016] The autonomous driving generation model workflow is as follows:
[0017] S1: The end-to-end autonomous driving model takes the current frame of the surround view picture as input and outputs the planned future driving route of the ego vehicle;
[0018] S2: The behavior controller takes the ego vehicle driving route output by the end-to-end autonomous driving model as input, deduces the reactions of other vehicles in the scene, and outputs the positions of the ego vehicle and other vehicles at the next time;
[0019] S3: The generative renderer takes the positions of the ego vehicle and other vehicles at the next time and the map elements as input to generate the corresponding surround view picture;
[0020] S4: The above steps S1-S3 are repeatedly executed in turn to complete the simulation of the driving process of the end-to-end autonomous driving model, and further give the evaluation;
[0021] The autonomous driving generation model generates in a self-recurrent manner, i.e. each frame of the surround view picture l t is obtained based on the previous frame of the picture l t-1 through ControlNet encoding.
[0022] Preferably, the generative renderer comprises a temporal consistency module, a projection-based object-level control module, and a retrieval-based scene-level control module.
[0023] Preferably, the working mode of the temporal consistency module is to first encode the previous frame of the picture l t-1 into the latent space using the VAE encoder of Stable Diffusion, then pass it to the ControlNet encoder, and finally the hidden features output by each layer of ControlNet are directly added to the corresponding layer of the U-Net encoder.
[0024] Preferably, the VAE encoder of Stable Diffusion is used to encode the last frame of picture l t-1 into latent space, obtaining conditional pre-hidden variable z prev , and add random Gaussian noise to z prev , and the noise level is also input into the ControlNet encoder, which is specifically expressed as:
[0025]
[0026] where ε represents the ControlNet encoder; is a randomly sampled Gaussian noise, , n is the time step t in the diffusion model.
[0027] Preferably, the object-level control module based on projection provides object-level information: vehicle three-dimensional bounding box B t and vectorized map element M t during the simulation process.
[0028] The above two object-level information is used as the control information of the generative renderer, and the camera intrinsic and extrinsic parameters are used to project the three-dimensional bounding box B t and the vectorized map element M t from the laser radar coordinate system to the two-dimensional perspective view.
[0029] Then, the projected discrete coordinates are drawn into a set of binary masks of the same size, and the input images are and .
[0030] The above two object-level control information is merged and encoded into the latent space through a convolutional network.
[0031] The ControlNet encoder is used to inject these encoded control signals into the denoising process, and the object-level control is expressed as:
[0032] .
[0033] Preferably, the scene-level control module based on retrieval works by retrieving the two frames of pictures closest in physical distance before and after the current frame to generate the current frame of picture.
[0034] The two retrieved pictures are encoded by the image encoder, and are input into the ControlNet as the key and value in the reference attention module, so that the current frame generation can find the corresponding relationship of the background.
[0035] Preferably, since the spatial relationship between the current frame and the two reference frames can be explicitly calculated according to the coordinate transformation, the pixel spatial relationship information is injected into the reference attention mechanism;
[0036] A discrete grid P of size (H, W, D, 4) is calculated for all images using pixel 3D position encoding, where D is the number of points sampled along the depth axis;
[0037] Then, the grid is transformed from camera space to 3D coordinate system using the camera transformation matrix of the current frame P ego and the two reference frames P ref .
[0038] Finally, in the cross-attention of ControlNet, P ego is the position encoding of Query, and P ref is the position encoding of Key, which can be expressed as:
[0039]
[0040] In this way, the current frame can find the corresponding relationship in the reference image, thereby tracking the static background and generating the converted pixels.
[0041] Compared with the prior art, the present application has the following beneficial effects:
[0042] 1. On the framework, the simulation framework proposed in the present application uses the generation ability of the generation model in an autoregressive manner to realize high-frequency interaction with the end-to-end driving model and generate time-consistent images;
[0043] 2. The time-consistent module, object-level control based on projection, and scene-level control based on retrieval proposed can significantly improve the model generation fidelity. DETAILED DESCRIPTION
[0044] In order to more specifically and intuitively illustrate the technical solutions in the embodiments of the present application or the prior art, a brief introduction will be given below to the drawings needed to be used in the embodiments or prior art description.
[0045] Figure 1 The device architecture diagram for generating the autonomous driving model for reactive closed-loop evaluation proposed in the present application;
[0046] Figure 2 The time consistency module diagram of the device for generating the autonomous driving model for reactive closed-loop evaluation proposed in the present application;
[0047] Figure 3 The training collapse diagram of the device for generating the autonomous driving model for reactive closed-loop evaluation proposed in the present application;
[0048] Figure 4 A modified schematic diagram of the device for generating a reactive closed-loop evaluation of an autonomous driving model according to the present application;
[0049] Figure 5 A projection-based object-level control schematic diagram of the device for generating a reactive closed-loop evaluation of an autonomous driving model according to the present application;
[0050] Figure 6 A retrieval-based scenario-level control schematic diagram of the device for generating a reactive closed-loop evaluation of an autonomous driving model according to the present application Figure 1 ;
[0051] Figure 7 A retrieval-based scenario-level control schematic diagram of the device for generating a reactive closed-loop evaluation of an autonomous driving model according to the present application Figure 2 . DETAILED DESCRIPTION
[0052] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments.
[0053] With reference to Figures 1-2 , the device for generating a reactive closed-loop evaluation of an autonomous driving model can be used for closed-loop simulation of an end-to-end autonomous driving model, and the device comprises a behavior controller and a generative renderer, wherein the behavior controller is based on a classic rule system IDM, and the generative renderer is based on a diffusion model.
[0054] The working process of the autonomous driving generation model is as follows:
[0055] S1: The end-to-end autonomous driving model takes a current frame of a surround picture as input and outputs a planned future driving route of a self vehicle;
[0056] S2: The behavior controller takes the driving route of the self vehicle output by the end-to-end autonomous driving model as input, deduces the reactions of other vehicles in the scene, and outputs the positions of the self vehicle and the other vehicles at the next moment;
[0057] S3: The generative renderer takes the positions of the self vehicle and the other vehicles at the next moment and map elements as input and generates a corresponding surround picture;
[0058] S4: The above steps S1-S3 are repeatedly executed in turn to complete the simulation of the driving process of the end-to-end autonomous driving model and further give an evaluation.
[0059] The autonomous driving generation model generates in a self-recurrent manner, that is, each frame of a surround picture l t is generated based on the previous frame of picture lt-1 The ControlNet encoding is obtained.
[0060] The generative renderer includes a temporal consistency module, a projection-based object-level control module, and a retrieval-based scene-level control module.
[0061] The working mode of the temporal consistency module is first to use the VAE encoder of Stable Diffusion to encode the previous frame picture l t-1 to the latent space, and then pass it to the ControlNet encoder, and finally the hidden features output by each layer of ControlNet are directly added to the corresponding layer of the U-Net encoder.
[0062] Referring to Figure 3 , the above does this to cause the problem of using time to accumulate errors and collapse: in the training process, the previous frame picture uses the stored real world picture; and in the use process, the previous frame picture is generated by the model itself, which has a gap with the real world image. Therefore, due to the recursive nature of autoregressive generation, the error will continue to accumulate, eventually leading to generation collapse.
[0063] The above problem is caused by the excessive dependence of the generative renderer on the quality of the previous frame picture, and we propose to use Gaussian blur to modulate noise during the training process, so that the generative renderer can adapt to the previous frame picture with quality loss, to solve this problem.
[0064] Specifically, in the training process, the VAE encoder of Stable Diffusion is used to encode the previous frame picture l t-1 into the latent space to obtain the conditional pre-hidden variable z prev , random Gaussian noise is added to z prev , and the noise level is also input to the ControlNet encoder, which is specifically expressed as:
[0065] Where ε represents the ControlNet encoder; is a randomly sampled Gaussian noise, , the noise addition strategy is similar to the training strategy of the diffusion model, and n is the time step t in the diffusion model.
[0066] In order to further avoid the accumulation of high-frequency errors, we perform Gaussian blur on the previous frame image, and the proposed technology can effectively alleviate the degradation problem, referring to Figure 4 .
[0067] Referring to Figure 5 , the projection-based object-level control module provides object-level information: the three-dimensional bounding box B tand vectorized map elements M t ;
[0068] The above two object-level information is taken as the control information of the generative renderer, and the camera intrinsic and extrinsic parameters are used to project the three-dimensional bounding box B t and vectorized map elements M t from the laser radar coordinate system to the two-dimensional perspective view;
[0069] Then, the projected discrete coordinates are drawn into a set of binary masks of the same size, and the input image is and ;
[0070] The object semantic information is integrated by assigning each category its own dedicated channel, and the above two object-level control information is combined and encoded into the latent space through the convolutional network;
[0071] The ControlNet encoder is used to inject these encoded control signals into the denoising process, and the object-level control is expressed as:
[0072] .
[0073] The generative model is prone to hallucination, in order to solve this problem, we let the model can reference the nearby static scene information in the database, such as high-rise buildings, landmark buildings, to reduce hallucination.
[0074] Specifically, the working mode of the scene-level control module based on retrieval is to generate the current frame of picture by retrieving the two frames of pictures closest in physical distance in the database;
[0075] The two retrieved pictures are encoded by an image encoder such as ResNet, and are input into ControlNet as the key and value in the reference attention module, so that the generation of the current frame can find the corresponding relationship of the background, such as Figure 6 The orange part is shown in the blue part of the ControlNet original part.
[0076] Since the spatial relationship between the current frame and the two reference frames can be calculated according to the coordinate transformation, the pixel spatial relationship information is injected into the reference attention mechanism;
[0077] A discrete grid P of size (H, W, D, 4) is calculated for all images using pixel three-dimensional position encoding, where D is the number of points sampled along the depth axis;
[0078] Then, the camera transformation matrix of the current frame P ego and the two reference frames P ref is used to transform the grid from the camera space to the 3D coordinate system;
[0079] Finally, in the cross-attention of ControlNet, P ego As the position encoding of Query, P ref As the position encoding of Key, P
[0080]
[0081] In this way, the current frame can find the corresponding relationship in the reference image, thereby tracking the static background and generating the converted pixels, refer to Figure 7 .
[0082] 1. On the framework, the simulation framework proposed in the application uses the generation ability of the generation model in an autoregressive manner to realize high-frequency interaction with the end-to-end driving model and generate images consistent in time;
[0083] 2. The time consistency module, the object-level control based on projection, and the scene-level control based on retrieval proposed can significantly improve the model generation fidelity.
[0084] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any skilled person in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.
Claims
1. An apparatus for generating a reaction closed loop evaluated autonomous driving model, characterized in that, The device comprises a behavior controller and a generative renderer, wherein the behavior controller is based on a classic rule system IDM, and the generative renderer is based on a diffusion model; The automatic driving generation model workflow is as follows: S1: An end-to-end automatic driving model takes the current frame of a panoramic picture as input and outputs a planned future driving route of the ego vehicle; S2: A behavior controller takes the driving route of the ego vehicle output by the end-to-end automatic driving model as input, deduces the reactions of other vehicles in the scene, and outputs the positions of the ego vehicle and other vehicles at the next moment; S3: A generative renderer takes the positions of the ego vehicle and other vehicles at the next moment and map elements as input and generates a corresponding panoramic picture; S4: The steps S1-S3 are repeatedly executed in a loop to complete the simulation of the driving process of the end-to-end automatic driving model and further give an evaluation; The automatic driving generation model generates in a self-recurrent manner, i.e., each frame of the surround view picture l t is based on the previous frame of picture l t-1 is obtained through ControlNet coding.
2. The apparatus for generating a reaction closed loop evaluated autonomous driving model of claim 1, wherein, The generative renderer comprises a temporal consistency module, a projection-based object-level control module, and a retrieval-based scene-level control module.
3. The apparatus for generating a reaction closed loop evaluated autonomous driving model of claim 2, wherein, The time consistency module works in the following way: first, the previous frame of picture l t-1 is encoded into the latent space using the VAE encoder of Stable Diffusion, then it is passed to the ControlNet encoder, and finally the hidden features output by each layer of the ControlNet are directly added to the corresponding layers of the U-Net encoder.
4. The apparatus for generating a reaction closed loop evaluated autonomous driving model of claim 3, wherein, The VAE encoder using Stable Diffusion encodes the previous frame picture l t-1 into the latent space, obtaining the conditional pre-hidden variable z prev , and adds random Gaussian noise to z prev , and the noise level is also input into the ControlNet encoder, which is specifically expressed as: ； Wherein, ε represents a ControlNet encoder; ， is a randomly sampled Gaussian noise, ， n is the time step t in the diffusion model.
5. The apparatus for generating a reaction closed loop evaluated autonomous driving model of claim 4, wherein, The projection-based object-level control module provides object-level information: vehicle three-dimensional bounding box B t and vectorized map elements M t ; The above two object-level information is taken as control information of the generative renderer, and the three-dimensional bounding box B t and vectorized map elements M t is projected into a two-dimensional perspective view from the laser radar coordinate system; Then, the projected discrete coordinates are drawn into a set of binary masks of the same size, and the input image is and ; The above two object-level control information is merged and encoded into the latent space through a convolutional network; These encoded control signals are injected into the denoising process using the ControlNet encoder, and the object-level control is expressed as: 。 6. The apparatus for generating a reaction closed loop evaluated autonomous driving model of claim 5, wherein, The working mode of the retrieval-based scene-level control module is to generate the current frame of picture by retrieving the two frames of pictures with the closest physical distance in the database; The two retrieved pictures are encoded by an image encoder, and are input into the ControlNet as the key and value in the reference attention module, so that the generation of the current frame can find the corresponding relationship of the background.
7. The apparatus for generating a reaction closed loop evaluated autonomous driving model of claim 6, wherein, Since the spatial relationship between the current frame and the two reference frames can be explicitly calculated according to the coordinate transformation, the pixel spatial relationship information is injected into the reference attention mechanism; A discrete grid P with a size of (H, W, D, 4) is calculated for all images using pixel three-dimensional position encoding, where D is the number of points sampled along the depth axis; Then, the mesh is transformed from camera space to 3D coordinate system using camera transformation matrices of the current frame P ego and two reference frames P ref . Finally, in the cross-attention of ControlNet, P ego As the position encoding of Query, i.e., the query, P ref As the position encoding of Key, i.e., the key, P ； In this way, the current frame can find the corresponding relationship in the reference image, thereby tracking the static background and generating the converted pixels.