Self-correction method and device for end-to-end model of automatic driving and medium

Abstract

The application discloses an automatic driving end-to-end model self-correction method, device and medium. The method first evaluates multi-view images through an end-to-end automatic driving model, detects failure cases and extracts intermediate output information. Then, the causes of the failure cases are analyzed as multi-modal prompts, combined with bird's eye view layout and scene description, and data requirements are generated through a visual language model. Next, improved generation model Drive-Sora is used to generate multi-view video data with spatio-temporal consistency and / or diversity. Finally, the generated data is used as new training data to iteratively update and optimize the end-to-end automatic driving model, and through performance evaluation, it enters the next round of data generation and optimization cycle. The application significantly improves the robustness and generalization ability of the automatic driving model, reduces the collision rate and planning error in the long-tail scene, and provides strong support for the development and application of automatic driving technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an automatic driving end-to-end model self-correction method, device and medium. BACKGROUND

[0002] In the field of automatic driving, end-to-end planning methods have become the mainstream standard for current automatic driving systems. This method integrates perception, prediction, decision-making, and planning into a unified framework, achieving comprehensive processing of complex driving environments. However, the data-driven approach faces a serious "long tail" problem in the application of end-to-end planning methods.

[0003] The "long tail" problem mainly refers to the fact that in the actual operation of the automatic driving system, most of the time the vehicle encounters common scenarios, but there are still a small number of scenarios that have a significant impact on the overall performance of the system, although they occur infrequently. These long-tail scenarios are often difficult to obtain sufficient training samples through conventional data collection methods due to their diversity and complexity, resulting in poor performance of the automatic driving model in these scenarios, and even safety hazards. This lack of robustness greatly limits the widespread application and commercialization process of automatic driving technology.

[0004] In the development process of related technologies, self-correction technology has gradually attracted attention. In particular, in the field of automatic driving, self-correction technology is considered an important means to improve system performance and safety. Although large language models (LLMs) and visual language models (VLMs) have strong semantic and reasoning capabilities, they can assist in error verification and correction, and to some extent, they can adjust the automatic driving output by providing external feedback, improving decision-making reliability, but these methods have obvious limitations. Specifically, they do not update the training data within the automatic driving model, and cannot implement targeted optimization based on fault cases. This means that although these models can improve the performance of the automatic driving system to some extent, they cannot fundamentally solve the "long tail" problem caused by the data-driven approach.

[0005] At the same time, in the development process of end-to-end automatic driving models, although models such as ST-P3, UniAD, VAD, etc. integrate perception, prediction, decision-making, and planning into a framework in different ways, they have attracted widespread attention, but still face many challenges in practical applications. These challenges include but are not limited to insufficient adaptability of the model to complex scenarios, performance degradation in long-tail scenarios, etc. SUMMARY

[0006] The main purpose of the present application is to provide an automatic driving end-to-end model self-correction method, device and medium, aiming to solve the "long tail" problem faced by the existing data-driven end-to-end planning model, and improve the robustness and generalization ability of the automatic driving system.

[0007] To achieve the above purpose, the present application provides the following solutions:

[0008] In a first aspect, the present application provides an automatic driving end-to-end model self-correction method, comprising the following steps:

[0009] Evaluating the multi-view image by the end-to-end automatic driving model, detecting the failure cases and extracting the intermediate output information of the perception, prediction and planning stages;

[0010] Analyzing the specific reasons of the failure cases into multi-modal prompts, combining the bird's eye view layout and scene description, and generating data requirements through a visual language model;

[0011] Based on the data requirements, generating multi-view video data with spatio-temporal consistency and / or diversity by using a Drive-Sora generation model;

[0012] Taking the generated multi-view video data as new training data, iteratively updating and optimizing the end-to-end automatic driving model;

[0013] Evaluating the performance of the updated model, analyzing the newly appeared failure cases and entering the next round of data generation and optimization cycle.

[0014] Further, the evaluation of the multi-view image by the end-to-end automatic driving model, the detection of the failure cases and the extraction of the intermediate output information of the perception, prediction and planning stages specifically include:

[0015] Inputting the multi-view image into the end-to-end automatic driving model for evaluation, focusing on analyzing the collision risk prediction results outputted by the model, and selecting the cases with faults from the original training set;

[0016] Collecting the intermediate outputs of the model, including but not limited to the perception results, prediction results and planning results, and transmitting the above intermediate outputs to the subsequent modules together with the multi-view image for later data collection;

[0017] Superimposing the abstract structured coordinates outputted by the model on the multi-view image, transmitting the visualized results from the bird's eye view to the subsequent data collection module, and realizing the comprehensive analysis of the model performance from the first person perspective and the global perspective.

[0018] Further, the analysis of the specific reasons of the failure cases into multi-modal prompts, the combination of the bird's eye view layout and scene description, and the generation of data requirements through a visual language model specifically include:

[0019] A Tailor-Agent system is constructed, which includes a plurality of functional agents based on visual language models;

[0020] The Tailor-Agent system decomposes the overall goal into executable sub-tasks according to the overall goal and requirements of the project, and correspondingly allocates them to the cause analysis agent, the requirement design agent and the multi-modal prompt collection agent;

[0021] The cause analysis agent performs in-depth analysis on the output of the end-to-end model, checks the intermediate output from aspects including foreground goal, background environment and / or weather, and determines the specific causes that may lead to planning failure;

[0022] The requirement design agent converts the cause analysis results output by the cause analysis agent into specific data requirements;

[0023] The multi-modal prompt collection agent constructs multi-modal prompts according to the generated data requirements, calculates the similarity of images in the data set using the visual language model, extracts the bird's eye view layout of related images, and reorganizes the scene description information;

[0024] Finally, the multi-modal prompts are collected and output.

[0025] Further, based on the data requirements, a Drive-Sora generation model is used to generate multi-view video data with spatio-temporal consistency and / or diversity, specifically including:

[0026] The Drive-Sora generation model receives the multi-modal prompts generated by the Tailor-Agent system as input; generates new multi-view video data through the Drive-Sora, and evaluates the quality of the generated data.

[0027] Further, the multi-view video data includes the layout structure of the foreground and the background, and the Drive-Sora generation model performs embedding processing on the foreground data to generate foreground embedding , whose formula is:

[0028]

[0029]

[0030] wherein, is a Fourier embedder for processing geometric data;

[0031] is a T5 text encoder for converting dense descriptions into text embeddings;

[0032] For multi-layer perceptron, used to fuse all input information to generate foreground embedding ;

[0033] background layout Through the VAE encoder processing, generate background embedding ;

[0034] scene description text Through T5 text encoder processing, generate text embedding .

[0035] Then integrate multimodal embedding through cross attention mechanism 、 and , the specific process includes:

[0036] The foreground embedding Spliced with the text embedding Form a complete feature representation, as the key and value Input of attention mechanism;

[0037] The background embedding As a query Input, the calculation formula of integrated embedding is as follows:

[0038]

[0039]

[0040] Where, is the initial input feature, represents the splicing operation of foreground and text embedding, represents the cross attention mechanism, is the dimension of embedding vector.

[0041] Further, the Drive-Sora generation model generates tasks through a space-time diffusion transformer and introduces ControlNet for layout guidance to ensure the consistency of the generated video with the input prompt. The calculation process of the space-time diffusion transformer is as follows:

[0042]

[0043] Where, is the trained convolutional layer, with an initial value of zero;

[0044] Refers to ControlNet-Transformer, used to process input features and background embedding The integration of the two.

[0045] Further, the Drive-Sora generation model replaces the self-attention block in the space-time diffusion transformer with a multi-view spatial attention mechanism, and the specific calculation process is as follows:

[0046]

[0047]

[0048]

[0049] wherein, represents a linear projection operation, represents a dimension reshaping operation, represents a multi-view spatial attention mechanism.

[0050] Further, the Drive-Sora generation model adopts a classifier-free guidance strategy to enhance the consistency of generated images and conditions, specifically including:

[0051] During the training process, randomly set a single condition to be empty with a probability of 5%, and set all conditions to be "empty" at the same time with a probability of 5%;

[0052] Generate the final image by combining the conditional generation result and the unconditional generation result, using the guide parameter 、 、 Adjust the balance of condition generation and sampling consistency, and the specific formula is as follows:

[0053]

[0054] wherein, the generation network depends on the foreground frame parameter , the background color parameter and the text description three conditions, represents an empty condition.

[0055] Secondly, the present application provides an automatic driving end-to-end model self-correction device, comprising:

[0056] The fault finding module is configured to evaluate the multi-view image through the end-to-end automatic driving model, identify the failure cases and extract the intermediate output information of the perception, prediction and planning stages;

[0057] The data requirement generation module is configured to analyze the specific causes of the failure cases into multi-modal prompts, combine the bird's eye view layout and scene description, and generate data generation requirements through a visual language model;

[0058] The multi-view data generation module is configured to generate high-quality multi-view video data with spatiotemporal consistency and / or diversity based on the data generation requirements and using the Drive-Sora generation model.

[0059] The model iteration and update module is configured to use the generated multi-view video data as new training data to iteratively update and optimize the end-to-end autonomous driving model.

[0060] The performance evaluation module is configured to evaluate the performance of the updated model, analyze newly emerging failure cases, and trigger the next round of data generation and optimization cycle.

[0061] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the autonomous driving end-to-end model self-correction method described above.

[0062] Through the above technical solutions, the beneficial effects of this invention are as follows: At the data generation level, this application significantly improves efficiency and diversity, no longer limited by traditional data collection methods, and provides rich and diverse data resources for model training. From the perspective of model performance, it powerfully enhances the robustness and generalization ability of autonomous driving models, enabling them to maintain stable and accurate performance in various complex driving scenarios. The effect is particularly pronounced in long-tail scenarios, significantly reducing collision rates and planning errors, and significantly improving the safety and reliability of autonomous driving. Furthermore, this invention reduces reliance on manual annotation, using a closed-loop optimization method to generate customized data based on model failure cases, continuously optimizing the performance and stability of the autonomous driving model, enabling the model to better fit the complex and ever-changing real-world driving environment, and providing solid support for the development and application of autonomous driving technology. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and the embodiments in the accompanying drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 A flowchart illustrating an embodiment of the self-correction method for an end-to-end autonomous driving model provided in this application;

[0065] Figure 2 A schematic diagram of the structure of an autonomous driving end-to-end model self-correction device provided in an embodiment of this application;

[0066] Figure 3 Fig. 1 is a structural schematic diagram of a computer device according to an embodiment of the present application;

[0067] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0068] It should be understood that the specific embodiments described herein are merely exemplary and intended to explain the present application, but not to limit the present application. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0069] The foregoing and other technical contents, features and effects of the present application will be further described in the following with reference to the accompanying drawings. Figures 1-3 The detailed description of the embodiments will be clearly presented. The structural contents mentioned in the following embodiments are all referred to the accompanying drawings.

[0070] The exemplary embodiments of the present application will be described below with reference to the accompanying drawings.

[0071] The present application proposes a self-correcting data generation engine framework-CorrectAD, which aims to improve the performance of end-to-end autonomous driving models in long-tail scenarios by continuously generating high-quality training data and dynamically optimizing model failure cases.

[0072] CorrectAD converts failure cases into data requirements through iterative optimization, generates high-quality video scenarios matching the requirements using the improved generation model Drive-Sora, and re-trains the autonomous driving model with the newly generated data, thereby solving old failure problems while improving the overall robustness and scenario adaptability of the model.

[0073] Please refer to Figure 1 In an embodiment of the present application, the method comprises the following steps:

[0074] S100, evaluating the multi-view images by the end-to-end autonomous driving model, detecting failure cases and extracting intermediate output information of the perception, prediction and planning stages;

[0075] S200, analyzing the specific reasons of the failure cases into multi-modal prompts, combining the bird's eye view layout and scene description, and generating data requirements through a visual language model;

[0076] S300, generating multi-view video data with spatio-temporal consistency and / or diversity based on the data requirements using the Drive-Sora generation model;

[0077] S400, iteratively update and optimize the end-to-end autonomous driving model with the generated multi-view video data as new training data;

[0078] S500, evaluate the performance of the updated model, analyze the newly emerged failure cases, and enter the next round of data generation and optimization cycle.

[0079] In a specific embodiment, step S100 includes the following steps:

[0080] S101, input the multi-view image into the end-to-end autonomous driving model for evaluation, focusing on analyzing the collision risk prediction results output by the model, and selecting failure cases from the original training set;

[0081] S102, collect the intermediate output of the model, including but not limited to perception results, prediction results, and planning results, and transmit the above intermediate output together with the multi-view image to the subsequent module for later data collection;

[0082] S103, superimpose the abstract structured coordinates output by the model onto the multi-view image, and transmit the visualization results from the bird's eye view to the subsequent data collection module, realizing comprehensive analysis of the model performance from the first-person perspective and the global perspective.

[0083] In a specific embodiment, step S200 includes the following steps:

[0084] S201, build a Tailor-Agent system, which includes multiple functional agents based on visual language models (VLM);

[0085] S202, the Tailor-Agent system decomposes the overall goal into executable sub-tasks according to the overall goal and requirements of the project, and assigns them to the cause analysis agent, requirement design agent, and multi-modal prompt collection agent;

[0086] S203, the cause analysis agent performs in-depth analysis on the output of the end-to-end model, checks the intermediate output from the aspects of foreground target, background environment, and / or weather, and determines the specific causes that may lead to planning failure;

[0087] S204, the requirement design agent converts the cause analysis results output by the cause analysis agent into specific data requirements;

[0088] S205, the multi-modal prompt collection agent constructs multi-modal prompts according to the generated data requirements, calculates the similarity of images in the data set using visual language models, extracts the bird's eye view layout of related images, and reorganizes the scene description information;

[0089] S206, finally collect and output multi-modal prompts.

[0090] In a specific embodiment, step S300 comprises the following steps:

[0091] S301, the Drive-Sora generation model receives the multi-modal prompt generated by the Tailor-Agent system as input;

[0092] S302, new multi-view video data is generated by the Drive-Sora, and the quality of the generated data is evaluated.

[0093] Wherein, the multi-modal prompt contains bird's eye view (BEV) layout information and scene semantic description. In the specific implementation process, the multi-modal control generation module takes the BEV layout as input, and projects these layouts into the camera view according to the camera parameters, thereby constructing the layout structure containing the foreground and the background. The foreground layout covers multiple key elements, specifically the coordinates of the bounding box , the orientation of the bounding box , the instance identification , and the dense description . The background layout is a road map represented by different color lines , the size of which is RHxWx3, and these lines will be aligned according to the pixel position of the camera image, thereby ensuring the consistency of geometric information and camera view.

[0094] The scene description is in the form of text, providing semantic information of the environment in the scene, such as weather conditions, road conditions, etc. This text information will be further encoded to generate text embedding for subsequent generation tasks. At the same time, the Drive-Sora generation module performs embedding processing on the foreground data to generate foreground embedding , the calculation formula is:

[0095]

[0096]

[0097] Wherein, is the Fourier embedder, which is used to process geometric data;

[0098] is the T5 text encoder, which is used to convert the dense description into text embedding;

[0099] is the multi-layer perceptron, which is used to fuse all input information to generate foreground embedding ;

[0100] The background layout is processed by the VAE encoder to generate background embedding ;

[0101] scene description text text embedding generated by T5 text encoder processing .

[0102] Further, the Drive-Sora generation model integrates multi-modal embeddings through cross-attention mechanisms , and , the specific process includes:

[0103] concatenate the foreground embedding and the text embedding to form an overall feature representation, which is input as the key and value of the attention mechanism ;

[0104] background embedding is input as the query , and the calculation formula of the integrated embedding is as follows:

[0105]

[0106]

[0107] wherein, is the initial input feature, represents the concatenation operation of the foreground and text embeddings, represents the cross-attention mechanism, is the dimension of the embedding vector, and is a scale factor for normalizing attention calculation.

[0108] Through the cross-attention mechanism, the model can efficiently integrate multi-modal information to ensure that the generated content meets the constraints of the multi-modal input.

[0109] After integrating the multi-modal embedding, the Drive-Sora generation model generates the task through a space-time diffusion transformer (STDiT) and introduces ControlNet for layout guidance to ensure consistency between the generated video and the input prompt. The calculation process of the space-time diffusion transformer is as follows:

[0110]

[0111] wherein, is the trained convolutional layer, with an initial value of zero;

[0112] refers to ControlNet-Transformer, which is used to process input features and background embedding ​The integration of the above.

[0113] Among the above, ControlNet provides precise control over the layout, enabling the generated video scene to better meet the given conditions. Specifically, STDiT not only incorporates the characteristics of multi-modal prompts, but also further enhances the conditional control ability of the generation process through collaboration with ControlNet.

[0114] The Drive-Sora generation model replaces the self-attention block in STDiT with a multi-view spatial attention mechanism, ensuring multi-view consistency without introducing additional training parameters. The specific calculation process is as follows: Figure 1

[0115]

[0116]

[0117]

[0118] wherein, represents a linear projection operation, represents a dimension reshaping operation, represents a multi-view spatial attention mechanism.

[0119] Through the above reshaping operation, the input multi-view image is embedded along the view dimension, obtaining the reshaped embedding After this reshaping operation, the multi-view image embedding can be integrated into a form more convenient for subsequent processing, obtaining the reshaped embedding. Then, the reshaped multi-view embedding is calculated for self-attention, so that the model can capture the spatial relationship information between different view embeddings, thereby realizing the spatial consistency of multi-view image embedding. The achievement of this spatial consistency helps the model to more accurately understand and process multi-view image data in subsequent processing, improving the perception and analysis ability of complex scenes.

[0120] The Drive-Sora generation model adopts a classifier-free guidance (CFG) strategy to enhance the consistency between generated images and conditions, which specifically includes:

[0121] During training, randomly set a single condition to empty with a probability of 5%, and set all conditions to "empty" at the same time with a probability of 5%;

[0122] Generate the final image by combining the conditional generation result and the unconditional generation result, and use the guidance parameter , , to adjust the balance between conditional generation and sampling consistency. ​

[0123] In the current task, the generation network depends on the foreground frame parameters , background color parameters and text description Three conditions, so CFG will apply the unconditional denoising results to each condition respectively, the specific formula is as follows:

[0124]

[0125] Wherein, represents the empty condition, , , The text description, background color, and foreground frame are the guide parameters corresponding to the text description, background color, and foreground frame respectively.

[0126] In the above, The results generated under different conditions are adjusted step by step: the text description guides the generation of text-related features, the background color parameter adjusts the generated background road topology, and the foreground frame parameter affects the generation of vehicles in the foreground. Through the weighted combination of these conditions, the final generation result is a comprehensive consideration of multiple factors, which can better meet the generation needs of multiple input conditions.

[0127] In step S400, the generated multi-view video data is used as new training data to iteratively update and optimize the end-to-end autonomous driving model, including:

[0128] Use the new data generated by Drive-Sora to update the end-to-end model;

[0129] Adjust the parameters and structure of the model using the information in the new data;

[0130] Improve the performance of the model to adapt to complex driving scenarios.

[0131] In step S500, the performance of the updated model is evaluated, and new failure cases are analyzed and entered into the next round of data generation and optimization cycle, including:

[0132] Evaluate the performance of the updated model and analyze its performance on different indicators;

[0133] After the evaluation is completed, according to the results, start a new iteration cycle, return to the starting stage, and repeat the above steps;

[0134] Through the cycle iteration, continuously optimize the performance of the model, so that it performs better in the task of autonomous driving.

[0135] Based on the same inventive concept, the embodiments of the present application also provide an automatic driving end-to-end model self-correction device for implementing the automatic driving end-to-end model self-correction method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more automatic driving end-to-end model self-correction device embodiments provided below can refer to the limitations of the automatic driving end-to-end model self-correction method described above, which will not be repeated here.

[0136] In an exemplary embodiment, as shown in Figure 2 An automatic driving end-to-end model self-correction device 400 is provided, which includes a fault finding module 410, a data requirement generation module 420, a multi-view data generation module 430, a model iterative update module 440, and a performance evaluation module 450.

[0137] The fault finding module 410 is configured to evaluate multi-view images by the end-to-end automatic driving model, identify failure cases and extract intermediate output information of the perception, prediction and planning stages.

[0138] The data requirement generation module 420 is configured to analyze the specific causes of the failure cases into multi-modal prompts, combine the bird's eye view layout and scene description, and generate data generation requirements through a visual language model.

[0139] The multi-view data generation module 430 is configured to generate high-quality multi-view video data with spatio-temporal consistency and / or diversity based on the data generation requirements using a Drive-Sora generation model.

[0140] The model iterative update module 440 is configured to use the generated multi-view video data as new training data to iteratively update and optimize the end-to-end automatic driving model.

[0141] The performance evaluation module 450 is configured to evaluate the performance of the updated model, analyze newly occurring failure cases and trigger the next round of data generation and optimization cycle.

[0142] In an exemplary embodiment, the specific workflow of the automatic driving end-to-end model self-correction device is as follows:

[0143] Step One: In the initial stage of model running, CorrectAD detects failure cases by running the base autonomous driving model (e.g., UniAD) on a validation set. These cases often include scenarios where the vehicle crashes, trajectory deviates, or fails to handle complex environments (such as red lights, traffic jams, etc.). By analyzing these failure cases, CorrectAD attributes them to three major categories: Foreground factors (such as pedestrian occlusion or vehicle position errors), Background factors (such as incorrect road markings or missing scene details), and Weather factors (such as insufficient lighting or low visibility).

[0144] Step Two: Using a visual language model (VLM), CorrectAD conducts in-depth analysis of these cases, generating specific failure cause descriptions and assessing the classification confidence of the failure scenarios. The system sets a confidence threshold (e.g., 0.8), and if it exceeds this threshold, it further extracts more detailed failure causes, such as "unable to recognize road markings under low-light conditions at night" or "misjudgment of the position of the vehicle in front due to fog." Through multiple rounds of step-by-step analysis, CorrectAD generates requirement specifications for each failure case.

[0145] Step Three: Based on the extracted failure causes, CorrectAD converts the failure causes into structured multi-modal data requirements through its data requirement generation module (RequirementDesigner). These requirements include scene descriptions and corresponding bird's-eye view images, which are used to guide the generation module to generate scene data that meets specific requirements. For example, if a failure case is attributed to "misidentification of traffic signals at a complex intersection," the system will generate a requirement description as "an intersection scene with multiple traffic signals and high vehicle flow" and combine it with the BEV layout to generate input for the model.

[0146] Step Four: These requirements are further processed by the Multimodal Prompt Collector, which analyzes the scene text description and image features to filter out the most relevant initial data for the requirements, thereby constructing efficient generation conditions.

[0147] Step five, CorrectAD utilizes the improved generative model Drive-Sora, which is based on STDiT and meets the requirements of conditional generation by introducing the enhanced module of ControlNet-Transformer. Specifically, Drive-Sora adds a trainable copy module to the encoder part of STDiT, which is used to process multi-modal inputs (such as BEV layout and scene description) and generate features consistent with the conditions. The output of each module is fused with the features of the main model through a zero linear layer, achieving layer-by-layer penetration of conditional control capabilities.

[0148] In addition, as specifically described above, Drive-Sora also introduces a multi-view spatial attention mechanism to ensure the consistency of generated videos between multiple perspectives, as well as a temporal attention mechanism to enhance the coherence of videos in the time dimension. During model training, the classifier-free guidance (CFG) strategy is used to further strengthen the conditional control capabilities by adjusting the guidance parameters.

[0149] In the experiment, Drive-Sora was trained in a single perspective and optimized in multiple perspectives on the nuScenes dataset. The initial weights were derived from OpenSora 1.1, the training batch size was 16, the HybridAdam optimizer was used, the learning rate was 2e-5, and after 30,000 iterations, high-quality generation was achieved.

[0150] Step six, the quality of the generated results was evaluated through ablation experiments and a series of indicators.

[0151] As shown in Table 1, FID (Frechet Inception Distance) was used to measure the visual quality difference between generated images and real images, which decreased from 25.65 to 15.08; FVD (Frechet Video Distance) was used to measure the consistency of generated videos in the time dimension, which decreased from 97.32 to 94.51; CLIP similarity score was used to evaluate the consistency of text description and generated scene, which increased from 72.28 to 86.73, achieving high spatio-temporal consistency and semantic alignment.

[0152] Table 1: Ablation experiment of Drive-Sora generated data quality

[0153]

[0154] In Table 1, NDS represents the detection score of the detection model on the generated images.

[0155] Step seven, after adding the generated new data to the original training set, CorrectAD re-trains the basic autonomous driving model.

[0156] As shown in Table 2, the experimental results show that by using the data generated by CorrectAD to train the model, the collision rate of the UniAD model on the nuScenes validation set is reduced from 0.31% to 0.19%, and the path planning error (L2 error) is reduced from 1.03 meters to 0.98 meters, which exceeds the previous method of training the model by collecting relevant data, which shows that CorrectAD significantly optimizes the autonomous driving task and can effectively reduce the path deviation and collision risk in complex scenarios.

[0157] Table 2: Path planning performance of CorrectAD on nuScenes benchmark

[0158]

[0159] In Table 2, NMP represents an end-to-end interpretable neural motion planner, SA-NMP represents a neural motion planner based on sparse attention, FF represents a local motion planner using self-supervised free space prediction, EO represents a differentiable ray casting method for self-supervised occupancy prediction, UniAD represents a planning-oriented end-to-end autonomous driving method, and AIDE represents an automatic data engine method for object detection in autonomous driving.

[0160] In order to verify the effectiveness of the two modules Tailor-Agent and Drive-Sora proposed by CorrectAD, an ablation experiment was conducted by alternately disabling each module, and the experimental results are shown in Table 3.

[0161] Table 3: Ablation experiment of two modules proposed by CorrectAD

[0162]

[0163] The experiment set up different module activation combinations: the first case is that both modules are not activated (Tailor-Agent no, Drive-Sora no), at this time the data copied and pasted from the training set and randomly distributed is used as augmented data. From the experimental data, in this case, the L2 error in different time periods (1 second, 2 seconds, 3 seconds) is 0.54 meters, 1.03 meters, and 1.71 meters, respectively, with an average of 1.09 meters; the collision rate in the corresponding time period is 0.05%, 0.18%, and 0.81%, respectively, with an average of 0.35%. This shows that the random redundancy of such data has no positive effect on the model performance.

[0164] The second case is to enable the Drive-Sora module but not the Tailor-Agent module (Tailor-Agent no, Drive-Sora yes). In this setting, the L2 error is 0.53 meters, 0.99 meters, and 1.66 meters at 1 second, 2 seconds, and 3 seconds, respectively, with an average of 1.06 meters. The collision rate is 0.10%, 0.20%, and 0.62% in the corresponding time periods, with an average of 0.31%. Compared with the first case, the data has improved, indicating that using Drive-Sora to generate diversified training data to some extent solves the problem of data random redundancy.

[0165] The third case is to enable the Tailor-Agent module but not the Drive-Sora module (Tailor-Agent yes, Drive-Sora no). At this time, the L2 error is 0.52 meters, 0.96 meters, and 1.62 meters at 1 second, 2 seconds, and 3 seconds, respectively, with an average of 1.03 meters. The collision rate is 0.08%, 0.20%, and 0.58% in the corresponding time periods, with an average of 0.29%. This shows that the Tailor-Agent module also has a certain effect on model performance improvement.

[0166] The last case is to enable both Tailor-Agent and Drive-Sora modules (Tailor-Agent yes, Drive-Sora yes). The experimental results show that the L2 error is 0.50 meters, 0.92 meters, and 1.53 meters at 1 second, 2 seconds, and 3 seconds, respectively, with an average of 0.98 meters. The collision rate is 0.02%, 0.14%, and 0.42% in the corresponding time periods, with an average of 0.19%.

[0167] Through this series of experimental results comparison, it can be clearly seen that finally, by using Drive-Sora to generate rich and diverse data customized by Tailor-Agent, CorrectAD achieves the lowest L2 error and collision rate, which fully highlights the importance of enhanced data distribution provided by Tailor-Agent and data diversity provided by Drive-Sora for improving the performance of end-to-end models.

[0168] CorrectAD, with its self-correcting generation-training-evaluation closed-loop process, achieves performance enhancement of end-to-end autonomous driving models by continuously improving the quality of generated data and optimizing the failure performance of the model. Through the generation of rich and diverse scenarios by multi-modal prompts, the model's task completion rate and generalization performance in complex dynamic environments are enhanced, making it suitable for various practical application scenarios such as unmanned vehicles, intelligent transportation systems, and simulation testing platforms.

[0169] In an example embodiment, a computer device, which can be a server or a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 3 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store automatic driving end-to-end model data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement an automatic driving end-to-end model self-correction method.

[0170] Those skilled in the art can understand that Figure 3 The structure shown in the above

[0171] In an example embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0172] In an example embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0174] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to a memory, a database or other medium used in the embodiments provided in the present application can include at least one of a non-volatile and a volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive random access memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric random access memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc.

[0175] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0176] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0177] The principles and implementation modes of the present application are described by using specific examples in the present application. The above embodiments are only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. An automatic driving end-to-end model self-correction method, characterized in that, The method comprises the following steps: evaluating the multi-view image through an end-to-end autonomous driving model, detecting failure cases and extracting intermediate output information of the perception, prediction and planning stages; analyzing the specific reasons for the failure cases into multi-modal prompts, combining the bird's eye view layout and scene description, and generating data requirements through a visual language model; generating multi-view video data with spatio-temporal consistency and / or diversity based on the data requirements using a Drive-Sora generation model; the Drive-Sora generation model first receives the multi-modal prompts of the Tailor-Agent system; embeds the foreground, background data and scene description, integrates the multi-modal embeddings through cross-attention mechanism; then uses the spatio-temporal diffusion transformer combined with ControlNet to guide the layout, ensures the multi-view consistency through multi-view spatial attention, and adjusts the condition and sampling balance through the classifier-free guidance strategy, thereby generating multi-view video data with spatio-temporal consistency and / or diversity; the Tailor-Agent system comprises a plurality of functional agents based on the visual language model; the Tailor-Agent system decomposes the overall goal into executable sub-tasks according to the overall goal and requirements of the project, and correspondingly allocates them to the cause analysis agent, requirement design agent and multi-modal prompt collection agent; the generated multi-view video data is used as new training data to iteratively update and optimize the end-to-end autonomous driving model; the performance of the updated model is evaluated, new failure cases are analyzed, and the next round of data generation and optimization cycle is entered.

2. The method of claim 1, wherein, The evaluation of the multi-view image through the end-to-end autonomous driving model, the detection of failure cases and the extraction of intermediate output information of the perception, prediction and planning stages specifically comprise: inputting the multi-view image into the end-to-end autonomous driving model for evaluation, focusing on analyzing the collision risk prediction results output by the model, and selecting cases that have failed from the original training set; collecting the intermediate output of the model, including but not limited to perception results, prediction results and planning results, and transmitting the above intermediate output to the subsequent module together with the multi-view image for later data collection; superimposing the abstract structured coordinates output by the model on the multi-view image, transmitting the visualization results from the bird's eye view to the subsequent data collection module, and realizing comprehensive analysis of the model performance from the first person perspective and the global perspective.

3. The method of claim 2, wherein, The analysis of the specific reasons for the failure cases into multi-modal prompts, the combination of the bird's eye view layout and the scene description, and the generation of data requirements through the visual language model specifically comprise: building a Tailor-Agent system, wherein the cause analysis agent performs in-depth analysis on the output of the end-to-end model, checks the intermediate output from the aspects of foreground target, background environment and / or weather, and determines the specific reasons that may cause planning failure; the requirement design agent converts the cause analysis results output by the cause analysis agent into specific data requirements; The multi-modal prompt collection agent constructs a multi-modal prompt according to the generated data requirement, calculates the similarity of images in the data set by using a visual language model, extracts the bird's eye view layout of the relevant images, and reorganizes the scene description information; Finally, the multi-modal prompt is collected and output.

4. The method of claim 3, wherein, The multi-modal prompt is generated by the Tailor-Agent system based on the data requirement, and the Drive-Sora generation model is used to generate multi-view video data with spatio-temporal consistency and / or diversity, specifically including: The Drive-Sora generation model receives the multi-modal prompt generated by the Tailor-Agent system as input; The Drive-Sora generation model generates new multi-view video data and evaluates the quality of the generated data.

5. The method of claim 4, wherein, The multi-view video data includes a layout structure of foreground and background, the Drive-Sora generation model performs embedding processing on foreground data to generate foreground embedding The formula is: wherein, is a Fourier embedder for processing geometry data, including object coordinates , object orientation , object instance information ; is a T5 text encoder for converting dense descriptions to text embeddings; a multi-layer perceptron for fusing all input information to generate a foreground embedding ; Background layout By the VAE encoder Processing, generating background embedding ; scene description text generate text embeddings by processing with a t5 text encoder ; The multi-modal embeddings are then integrated through a cross-attention mechanism , and , the specific process including: Embedding the foreground With the text embedding Concatenating to form an overall feature representation as key for the attention mechanism And value Input; background embedding as a query The input, integrated embedding, is computed as follows: wherein, denotes a linear projection operation, is an initial input feature, denotes a concatenation operation of foreground and text embeddings, denotes a cross-attention mechanism, is a dimension of embedding vectors.

6. The method of claim 5, wherein, The Drive-Sora generation model processes the generation task through a spatio-temporal diffusion transformer and introduces a ControlNet for layout guidance to ensure consistency between the generated video and the input prompt. wherein, is a trained convolutional layer with initial values set to zero; Refers to ControlNet-Transformer, used to process input features and background embeddings integration.

7. The method of claim 6, wherein, The Drive-Sora generation model replaces the self-attention block in the spatio-temporal diffusion transformer with a multi-view spatial attention mechanism, and the specific calculation process is as follows: wherein, denotes a linear projection operation, denotes a dimension reshaping operation, denotes a multi-view spatial attention mechanism.

8. The method of claim 7, wherein, The Drive-Sora generation model adopts a classifier-free guidance strategy to enhance the consistency between the generated image and the condition, specifically including: During the training process, a single condition is randomly set to empty with a probability of 5%, and all conditions are simultaneously set to "empty" with a probability of 5%; Generating the final image by combining the conditioned generation result and the unconditioned generation result using the guidance parameter , , Adjusting the balance of the conditioned generation and the sampling consistency, the specific formula is as follows: wherein the generation network depending on the foreground frame parameters , the background color parameters and the text description three conditions, denotes the empty condition.

9. An automatic driving end-to-end model self-correction device for implementing the method of any one of claims 1-8, characterized in that, Including: The fault finding module is configured to evaluate multi-view images through an end-to-end autonomous driving model, identify failure cases and extract intermediate output information in the perception, prediction and planning stages; The data requirement generation module is configured to analyze the specific causes of the failure cases into multi-modal prompts, combine the bird's eye view layout and the scene description, and generate data generation requirements through a visual language model; The multi-view data generation module is configured to generate high-quality multi-view video data with spatio-temporal consistency and / or diversity based on the data generation requirements using the Drive-Sora generation model; The model iteration update module is configured to use the generated multi-view video data as new training data to iteratively update and optimize the end-to-end autonomous driving model; The performance evaluation module is configured to evaluate the performance of the updated model, analyze new failure cases and trigger the next round of data generation and optimization cycle.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the autonomous driving end-to-end model self-correction method of any one of claims 1-8.

Images