Automatic driving end-to-end model self-correction method and device and medium

By evaluating failed cases and generating multi-view video data for iterative optimization, the performance degradation problem in long-tail scenarios is solved and the robustness and safety of the autonomous driving system is improved.

CN120564157AActive Publication Date: 2025-08-29WESTLAKE UNIV

Patent Information

Application Number
CN202511057590.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-07-30
Publication Date
2025-08-29
Estimated Expiration
2045-07-30

AI Technical Summary

Technical Problem

The existing end-to-end planning model of autonomous driving is poor in long-tail scenarios and cannot effectively solve the performance degradation and safety risks in complex scenarios, resulting in insufficient robustness and generalization capabilities of autonomous driving systems.

Method used

Through the end-to-end autonomous driving model evaluation failure cases, the visual language model is used to generate multi-modal prompts, combined with bird's-eye view layout and scene description, multi-view video data with time and space consistency and diversity are generated, and the model is iteratively updated and optimized.

Benefits of technology

It significantly improves the robustness and generalization capabilities of the autonomous driving model in long-tail scenarios, reduces the collision rate and planning errors, improves the safety and reliability of autonomous driving, and reduces the dependence on manual labeling.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120564157A_ABST
    Figure CN120564157A_ABST
Patent Text Reader

Abstract

The invention discloses an automatic driving end-to-end model self-correction method and device and a medium. The method comprises the steps that firstly, a multi-view image is evaluated through an end-to-end automatic driving model, a failure case is detected, and intermediate output information is extracted; analyzing the reason of the failure case into a multi-mode prompt, and generating a data demand through a visual language model in combination with the aerial view layout and the scene description; then, generating multi-view video data with time-space consistency and / or diversity by using an improved generative model Drive-Sora; and finally, taking the generated data as new training data, carrying out iterative updating and optimization on the end-to-end automatic driving model, and entering a next data generation and optimization cycle through performance evaluation. According to the method, the robustness and generalization ability of the automatic driving model are remarkably improved, the collision rate and the planning error in the long-tail scene are reduced, and powerful support is provided for development and application of the automatic driving technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, device, and medium for self-correction of an end-to-end model of autonomous driving. Background Art

[0002] In the field of autonomous driving, end-to-end planning has become the mainstream standard for autonomous driving systems. This approach integrates multiple processes—perception, prediction, decision-making, and planning—into a unified framework, enabling comprehensive handling of complex driving environments. However, data-driven approaches to end-to-end planning face a severe "long tail" problem.

[0003] The "long tail" problem primarily refers to the fact that in the actual operation of autonomous driving systems, while vehicles mostly encounter common scenarios, a small number of scenarios, while occurring less frequently, can significantly impact the system's overall performance. Due to the diversity and complexity of these long tail scenarios, it is often difficult to obtain sufficient training samples through conventional data collection methods. This results in poor performance of autonomous driving models in these scenarios, and may even lead to safety hazards. This lack of robustness significantly limits the widespread application and commercialization of autonomous driving technology.

[0004] As related technologies develop, self-correction technology has gradually gained attention. In the field of autonomous driving, in particular, it is considered a crucial means of improving system performance and safety. While large language models (LLMs) and visual language models (VLMs) possess powerful semantic and reasoning capabilities, assisting with error verification and correction, and to a certain extent improving decision reliability by providing external feedback to adjust autonomous driving outputs, these approaches have significant limitations. Specifically, they do not update the training data within the autonomous driving model, making it impossible to implement targeted optimizations based on failure cases. This means that while these models can improve autonomous driving system performance to a certain extent, they cannot fundamentally address the "long tail" problem inherent in data-driven approaches.

[0005] Meanwhile, in the development of end-to-end autonomous driving models, while models such as ST-P3, UniAD, and VAD have attracted widespread attention by integrating perception, prediction, decision-making, and planning into a single framework through various approaches, they still face numerous challenges in practical application. These challenges include, but are not limited to, insufficient adaptability to complex scenarios and performance degradation in long-tail scenarios. Summary of the Invention

[0006] The main purpose of this application is to provide a method, device and medium for self-correction of an end-to-end model for autonomous driving, aiming to solve the "long tail" problem faced by data-driven end-to-end planning models in the existing technology and improve the robustness and generalization ability of the autonomous driving system.

[0007] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides a method for self-correction of an end-to-end model for autonomous driving, comprising the following steps: Evaluate multi-view images through an end-to-end autonomous driving model, detect failure cases, and extract intermediate output information from the perception, prediction, and planning stages; Parse the specific reasons for failure cases into multimodal prompts, combine bird's-eye view layout and scene description, and generate data requirements through visual language models; Based on the data requirements, using the Drive-Sora generation model to generate multi-view video data with spatiotemporal consistency and / or diversity; Use the generated multi-view video data as new training data to iteratively update and optimize the end-to-end autonomous driving model; Evaluate the performance of the updated model, analyze new failure cases, and enter the next round of data generation and optimization.

[0008] Furthermore, the end-to-end autonomous driving model is used to evaluate multi-view images, detect failure cases, and extract intermediate output information from the perception, prediction, and planning stages, including: Multi-view images are fed into an end-to-end autonomous driving model for evaluation, with a focus on analyzing the collision risk predictions it outputs and filtering out failure cases from the original training set. Collect the intermediate outputs of the model, including but not limited to perception results, prediction results, and planning results, and transmit the above intermediate outputs together with multi-view images to subsequent modules for later data collection; The abstract structured coordinates output by the model are superimposed on the multi-view image, and the visualization results are transmitted to the subsequent data collection module from a bird's-eye view, enabling a comprehensive analysis of the model performance from both the first-person perspective and the global perspective.

[0009] Furthermore, the specific reasons for failure cases are parsed into multimodal prompts, combined with the bird's-eye view layout and scene description, and data requirements are generated through the visual language model, specifically including: Build a Tailor-Agent system, which includes multiple functional agents based on visual language models; The Tailor-Agent system breaks down the overall project goals into executable subtasks based on the project's overall goals and requirements, and assigns them to the cause analysis agent, requirement design agent, and multimodal prompt collection agent. The cause analysis agent performs in-depth analysis of the output of the end-to-end model, examining intermediate outputs from aspects including foreground targets, background environment, and / or weather to identify specific causes that may have led to planning failures. The demand design agent converts the cause analysis results output by the cause analysis agent into specific data requirements; The multimodal prompt collection agent constructs multimodal prompts based on the generated data requirements, calculates the similarity of images in the dataset using a visual language model, extracts the bird's-eye view layout of related images, and reconstructs the scene description information; Finally, multimodal prompts are collected and output.

[0010] Furthermore, based on the data requirements, generating multi-view video data with spatiotemporal consistency and / or diversity using the Drive-Sora generation model specifically includes: The Drive-Sora generative model receives as input the multimodal prompts generated by the Tailor-Agent system; generates new multi-view video data through Drive-Sora and evaluates the quality of the generated data.

[0011] Furthermore, the multi-view video data includes a layout structure of foreground and background, and the Drive-Sora generation model embeds the foreground data to generate a foreground embedding , the formula is:

[0012]

[0013] in, is a Fourier embedder, used to process geometric data; It is a T5 text encoder used to convert dense description Convert to text embeddings; A multi-layer perceptron that fuses all input information and generates foreground embedding ; Background Layout Through the VAE encoder Processing, generating background embedding ; Scene description text Processed by T5 text encoder to generate text embedding .

[0014] Then, we integrate multimodal embeddings through the cross-attention mechanism. 、 and The specific process includes: Embed the foreground With text embedding Concatenate to form an overall feature representation, which serves as the key to the attention mechanism Sum enter; Background embedding As a query Input, integrated embedding calculation formula is as follows:

[0015]

[0016] in, is the initial input feature, represents the concatenation operation of foreground and text embeddings, represents the cross attention mechanism, is the dimension of the embedding vector.

[0017] Furthermore, the Drive-Sora generation model processes the generation task through a spatiotemporal diffusion transformer and introduces ControlNet for layout guidance to ensure the consistency of the generated video with the input prompt. The calculation process of the spatiotemporal diffusion transformer is as follows:

[0018] in, For the convolutional layers to be trained, the initial value is set to zero; Refers to ControlNet-Transformer, which is used to process input features and background embedding integration.

[0019] Furthermore, the Drive-Sora generative model replaces the self-attention block in the spatiotemporal diffusion transformer with a multi-view spatial attention mechanism. The specific calculation process is as follows:

[0020]

[0021]

[0022] in, represents the linear projection operation, Represents a dimension reshaping operation, Represents the multi-view spatial attention mechanism.

[0023] Furthermore, the Drive-Sora generative model adopts a classifier-free guidance strategy to enhance the consistency of the generated image and the conditions, specifically including: During training, a single condition is randomly set to empty with a probability of 5%, and all conditions are set to "empty" at the same time with a probability of 5%; The final image is generated by combining the conditional generation results with the unconditional generation results, using the guided parameters 、 、 The balance between adjusting condition generation and sampling consistency is as follows:

[0024] Among them, the generation network Depends on foreground frame parameters , background color parameters and text description Three conditions, Indicates a null condition.

[0025] In a second aspect, the present application provides an end-to-end model self-correction device for autonomous driving, comprising: A fault finding module is configured to evaluate multi-view images through an end-to-end autonomous driving model, identify failure cases, and extract intermediate output information from the perception, prediction, and planning stages; The data requirement generation module is configured to parse the specific causes of failure cases into multimodal prompts, combine the bird's-eye view layout and scene description, and generate data generation requirements through the visual language model; a multi-view data generation module configured to generate high-quality multi-view video data with spatiotemporal consistency and / or diversity using a Drive-Sora generation model based on the data generation requirements; A model iterative update module is configured to use the generated multi-view video data as new training data to iteratively update and optimize the end-to-end autonomous driving model; The performance evaluation module is configured to perform performance evaluation on the updated model, analyze newly emerged failure cases and trigger the next round of data generation and optimization cycle.

[0026] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the above-mentioned end-to-end model self-correction methods for autonomous driving.

[0027] Through the above technical solutions, the beneficial effects of the present invention are as follows: at the data generation level, this application can greatly improve efficiency and diversity, is no longer limited by the limitations of traditional data collection methods, and provides rich and diverse data resources for model training. From the perspective of model performance, it greatly enhances the robustness and generalization ability of the autonomous driving model, allowing the model to maintain stable and accurate performance when facing various complex driving scenarios. Especially in long-tail scenarios, the effect is more prominent, greatly reducing the collision rate and planning error, and significantly improving the safety and reliability of autonomous driving. In addition, the present invention reduces dependence on manual labeling, uses closed-loop optimization methods to generate customized data based on model failure cases, and continuously optimizes the performance and stability of the autonomous driving model, so that the model can better fit the complex and changeable real driving environment, providing solid and powerful support for the development and application of autonomous driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and the embodiments in the drawings do not constitute any limitation to the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 A flowchart of an end-to-end model self-correction method for autonomous driving provided in one embodiment of the present application; Figure 2 A schematic diagram of the structure of an end-to-end model self-correction device for autonomous driving provided in one embodiment of the present application; Figure 3 A schematic diagram of the structure of a computer device provided in one embodiment of the present application; The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0030] It should be understood that the specific embodiments described herein are merely for explaining the present application and are not intended to limit the present application. Rather, these embodiments are provided to make the present disclosure more thorough and complete and to fully convey the scope of the present disclosure to those skilled in the art.

[0031] The above and other technical contents, features and effects of the present invention are described below with reference to the attached Figure 1-3 The detailed description of the embodiments will clearly show that the structural contents mentioned in the following embodiments are all based on the accompanying drawings.

[0032] Various exemplary embodiments of the present invention will be described below with reference to the accompanying drawings.

[0033] This paper proposes a self-correcting data generation engine framework - CorrectAD, which aims to improve the performance of end-to-end autonomous driving models in long-tail scenarios by continuously generating high-quality training data and dynamically optimizing model failure cases.

[0034] CorrectAD uses iterative optimization to convert failure cases into data requirements, uses the improved generative model Drive-Sora to generate high-quality video scenes that match the requirements, and retrains the autonomous driving model based on the newly generated data, thereby improving the overall robustness and scenario adaptability of the model while solving old failure problems.

[0035] See also Figure 1 In one embodiment of the present invention, the method comprises the following steps: S100, evaluate multi-view images through an end-to-end autonomous driving model, detect failure cases and extract intermediate output information from the perception, prediction and planning stages; S200: Analyze the specific reasons for failure cases into multimodal prompts, combine bird's-eye view layout and scene description, and generate data requirements through visual language model; S300, based on the data requirement, using the Drive-Sora generation model to generate multi-view video data with spatiotemporal consistency and / or diversity; S400, using the generated multi-view video data as new training data to iteratively update and optimize the end-to-end autonomous driving model; S500: Evaluate the performance of the updated model, analyze new failure cases, and enter the next round of data generation and optimization cycle.

[0036] In a specific embodiment, step S100 includes the following steps: S101. Input the multi-view images into the end-to-end autonomous driving model for evaluation, focusing on analyzing the collision risk prediction results outputted by the model and filtering out failure cases from the original training set. S102. Collect the intermediate outputs of the model, including but not limited to perception results, prediction results, and planning results, and transmit the intermediate outputs together with the multi-view images to subsequent modules for later data collection; S103. Superimpose the abstract structured coordinates output by the model on the multi-view image, and transmit the visualization results to the subsequent data collection module from a bird's-eye view perspective, so as to achieve a comprehensive analysis of the model performance from a first-person perspective and a global perspective.

[0037] In a specific embodiment, step S200 includes the following steps: S201, building a Tailor-Agent system, which includes multiple functional agents based on the visual language model (VLM); S202: The Tailor-Agent system breaks down the overall goal into executable subtasks based on the overall project goal and requirements, and assigns them to the cause analysis agent, the requirements design agent, and the multimodal prompt collection agent respectively; S203. The cause analysis agent performs an in-depth analysis of the output of the end-to-end model, examines the intermediate output from aspects including foreground targets, background environment, and / or weather, and determines the specific causes that may lead to planning failure. S204. The demand design agent converts the cause analysis results output by the cause analysis agent into specific data requirements. S205, the multimodal prompt collection agent constructs multimodal prompts based on the generated data requirements, calculates the similarity of images in the dataset using a visual language model, extracts the bird's-eye view layout of related images, and reconstructs scene description information; S206. Finally, collect and output multimodal prompts.

[0038] In a specific embodiment, step S300 includes the following steps: S301, the Drive-Sora generative model receives as input the multimodal prompts generated by the Tailor-Agent system; S302: Generate new multi-view video data through Drive-Sora and evaluate the quality of the generated data.

[0039] Among them, the multimodal prompts include bird's-eye view (BEV) layout information and scene semantic description. In the specific implementation process, the multimodal control generation module takes the BEV layout as input and projects these layouts into the camera view according to the camera parameters, thereby constructing a layout structure including foreground and background. The foreground layout covers multiple key elements, specifically the coordinates of the bounding box , the orientation of the bounding box , instance identifier and dense description The background layout is a road map represented by lines of different colors. , whose size is RH×W×3, and these lines will be aligned according to the pixel position of the camera image to ensure the consistency of geometric information and camera perspective.

[0040] The scene description is presented in text form, providing semantic information about the environment in the scene, such as weather conditions, road conditions, etc. This text information is further encoded to generate text embeddings , used for subsequent generation tasks. At the same time, the Drive-Sora generation module embeds the foreground data to generate the foreground embedding , the calculation formula is:

[0041]

[0042] in, is a Fourier embedder, used to process geometric data; It is a T5 text encoder used to convert dense description Convert to text embeddings; A multi-layer perceptron that fuses all input information and generates foreground embedding ; Background Layout Through the VAE encoder Processing, generating background embedding ; Scene description text Processed by T5 text encoder to generate text embedding .

[0043] Furthermore, the Drive-Sora generative model integrates multimodal embeddings through a cross-attention mechanism. 、 and The specific process includes: Embed the foreground With text embedding Concatenate to form an overall feature representation, which serves as the key to the attention mechanism Sum enter; Background embedding As a query Input, integrated embedding calculation formula is as follows:

[0044]

[0045] in, is the initial input feature, represents the concatenation operation of foreground and text embeddings, represents the cross attention mechanism, is the dimension of the embedding vector, used to normalize the scale factor of the attention calculation.

[0046] Through the cross-attention mechanism, the model can efficiently fuse multimodal information and ensure that the generated content conforms to the constraints of multimodal input.

[0047] After integrating multimodal embeddings, the Drive-Sora generative model uses the Spatiotemporal Diffusion Transformer (STDiT) to handle the generation task and introduces ControlNet for layout guidance to ensure that the generated video is consistent with the input prompt. The calculation process of the Spatiotemporal Diffusion Transformer is as follows:

[0048] in, For the convolutional layers to be trained, the initial value is set to zero; Refers to ControlNet-Transformer, which is used to process input features and background embedding integration.

[0049] In the above, ControlNet provides precise control over the layout, so that the generated video scene can better meet the given conditions. Specifically, STDiT not only combines the characteristics of multimodal prompts, but also further enhances the conditional control capabilities of the generation process through synergy with ControlNet.

[0050] The Drive-Sora generative model replaces the self-attention block in STDiT with a multi-view spatial attention mechanism, ensuring multi-view spatial attention without introducing additional training parameters. Figure 1 The specific calculation process is as follows:

[0051]

[0052]

[0053] in, represents the linear projection operation, Represents a dimension reshaping operation, Represents the multi-view spatial attention mechanism.

[0054] Through the above reshaping operation, the input multi-view image is embedded Concatenate along the view dimension to get the reshaped embedding After this reshaping operation, the multi-view image embeddings can be integrated into a form that is more convenient for subsequent processing, resulting in a reshaped embedding. Next, self-attention is calculated on the reshaped multi-view embeddings, allowing the model to capture the spatial relationship information between different view embeddings, thereby achieving spatial consistency of the multi-view image embeddings. The achievement of this spatial consistency helps the model to more accurately understand and process multi-view image data in subsequent processing, improving its perception and analysis capabilities for complex scenes.

[0055] The Drive-Sora generative model uses a classifier-free guidance (CFG) strategy to enhance the consistency of generated images with the conditions, including: During training, a single condition is randomly set to empty with a probability of 5%, and all conditions are set to "empty" at the same time with a probability of 5%; The final image is generated by combining the conditional generation results with the unconditional generation results, using the guided parameters 、 、 Regulate the balance between condition generation and sampling consistency.

[0056] In the current task, the generated network Depends on foreground frame parameters , background color parameters and text description There are three conditions, so CFG will apply the unconditional denoising results to each condition separately. The specific formula is as follows:

[0057] in, Indicates an empty condition, 、 、 They are the guidance parameters corresponding to text description, background color, and foreground frame respectively.

[0058] In the above, The results generated under different conditions are gradually adjusted: text description Guide the generation of text-related features and background color parameters Adjust the generated background road topology, while the foreground box parameters This affects the generation of vehicles in the foreground. Through the weighted combination of these conditions, the final It is a generation result that comprehensively considers multiple factors and can better meet the generation requirements of multiple input conditions.

[0059] In step S400, the generated multi-view video data is used as new training data to iteratively update and optimize the end-to-end autonomous driving model, including: Use new data generated by Drive-Sora to perform targeted updates to the end-to-end model; Use information from new data to adjust the parameters and structure of the model; Improve model performance to adapt it to complex driving scenarios.

[0060] In step S500, the updated model is evaluated for performance, new failure cases are analyzed, and the next round of data generation and optimization is entered, including: Evaluate the performance of the updated model and analyze its performance on different indicators; After the evaluation is completed, a new iteration cycle is started based on the results, returning to the starting stage and repeating the above steps; The model performance is continuously optimized through iterative cycles to make it perform better in autonomous driving tasks.

[0061] Based on the same inventive concept, embodiments of the present application also provide an autonomous driving end-to-end model self-correction device for implementing the aforementioned autonomous driving end-to-end model self-correction method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more embodiments of the autonomous driving end-to-end model self-correction device provided below can be found in the limitations of the autonomous driving end-to-end model self-correction method described above and will not be repeated here.

[0062] In an exemplary embodiment, Figure 2 The figure shows an end-to-end model self-correction device for autonomous driving, which includes a fault finding module 410, a data requirement generation module 420, a multi-view data generation module 430, a model iterative update module 440, and a performance evaluation module 450. a fault finding module 410 configured to evaluate the multi-view images through an end-to-end autonomous driving model, identify failure cases, and extract intermediate output information of the perception, prediction, and planning stages; The data requirement generation module 420 is configured to parse the specific causes of the failure cases into multimodal prompts, combine the bird's-eye view layout and scene description, and generate data generation requirements through a visual language model; a multi-view data generation module 430 configured to generate high-quality multi-view video data with spatiotemporal consistency and / or diversity using a Drive-Sora generation model based on the data generation requirements; a model iterative update module 440 configured to use the generated multi-view video data as new training data to iteratively update and optimize the end-to-end autonomous driving model; The performance evaluation module 450 is configured to perform performance evaluation on the updated model, analyze newly emerged failure cases and trigger the next round of data generation and optimization cycle.

[0063] In an exemplary embodiment, the specific workflow of the autonomous driving end-to-end model self-correction device is as follows: Step 1: During the initial model run, CorrectAD runs a baseline autonomous driving model (such as UniAD) on a validation set to detect failure cases. These typically include vehicle collisions, trajectory deviations, or failures to correctly handle complex environments (such as red lights or traffic jams) in specific scenarios. By analyzing these failures, CorrectAD attributes them to three main categories: foreground factors (such as pedestrian occlusion or incorrect vehicle positioning), background factors (such as incorrect road markings or missing scene details), and weather factors (such as insufficient lighting or low visibility).

[0064] Step 2: Use a visual language model (VLM) to conduct in-depth analysis of these cases, generate detailed descriptions of the failure reasons, and assess the classification confidence of the failure scenarios. The system sets a confidence threshold (for example, 0.8). If it exceeds this threshold, it further extracts more detailed failure reasons, such as "unable to recognize road markings in low-light conditions at night" or "misjudgment of the preceding vehicle's position due to fog." Through multiple rounds of step-by-step analysis, CorrectAD generates a requirement description for each failure case.

[0065] Step 3: Based on the extracted failure reasons, CorrectAD uses its data requirement generation module (RequirementDesigner) to convert the failure reasons into structured multimodal data requirements. These requirements, which include scenario descriptions and corresponding bird's-eye views, guide the generation module in generating scenario data that meets specific requirements. For example, if a failure case is attributed to "misidentification of traffic lights at a complex intersection," the system will generate a corresponding requirement description as "an intersection scenario with multiple traffic lights and high vehicle volume," and combine it with the BEV layout to generate input for the generation model.

[0066] Step 4: These requirements are further processed by the Multimodal Prompt Collector, which filters out the initial data most relevant to the requirements by analyzing the scene text description and image features, thereby constructing efficient generation conditions.

[0067] Step 5: CorrectAD utilizes the improved generative model Drive-Sora, based on STDiT. This model incorporates an enhanced ControlNet-Transformer module to meet the requirements of conditional generation. Specifically, Drive-Sora incorporates a trainable copy module (Trainable Copy) into the encoder portion of STDiT to process multimodal inputs (such as BEV layout and scene description) and generate features consistent with the conditions. The output of each module is fused with the features of the main model through a ZeroLinear layer, achieving layer-by-layer penetration of conditional control capabilities.

[0068] As detailed above, Drive-Sora also introduces a multi-view spatial attention mechanism to ensure consistency across multiple viewpoints in the generated video, as well as a temporal attention mechanism to enhance temporal coherence. During model training, a classifier-free guidance (CFG) strategy is used to further enhance conditional control capabilities by adjusting guidance parameters.

[0069] In the experiment, Drive-Sora performed single-view training and multi-view optimization on the nuScenes dataset. The initial weights came from OpenSora 1.1, the training batch size was 16, the HybridAdam optimizer was used, the learning rate was 2e-5, and high-quality generation was achieved after 30,000 iterations.

[0070] Step 6: The quality of the generated results is evaluated through ablation experiments and a series of indicators.

[0071] As shown in Table 1, FID (Frechet Inception Distance), used to measure the visual quality difference between generated images and real images, dropped from the initial 25.65 to 15.08; FVD (Frechet Video Distance), used to measure the consistency of generated videos in the temporal dimension, dropped from 97.32 to 94.51; and the CLIP similarity score, used to evaluate the consistency between text descriptions and generated scenes, increased from 72.28 to 86.73, achieving high spatiotemporal consistency and semantic alignment.

[0072] Table 1: Ablation experiment of Drive-Sora generated data quality

[0073] In Table 1, NDS represents the detection score of the detection model on the generated image.

[0074] Step 7: After adding the generated new data to the original training set, CorrectAD retrains the basic autonomous driving model.

[0075] As shown in Table 2, experimental results demonstrate that by using data generated by CorrectAD to train the model, the UniAD model's collision rate on the nuScenes validation set was reduced from 0.31% to 0.19%, and the path planning error (L2 error) was reduced from 1.03 meters to 0.98 meters. This surpasses previous methods that trained models by collecting relevant data. This demonstrates that CorrectAD significantly optimizes autonomous driving tasks and can effectively reduce path deviation and collision risks in complex scenarios.

[0076] Table 2: Path planning performance of CorrectAD on the nuScenes benchmark

[0077] In Table 2, NMP represents an end-to-end interpretable neural motion planner, SA-NMP represents a sparse attention-based neural motion planner, FF represents a local motion planner using self-supervised free space prediction, EO represents a differentiable ray casting method for self-supervised occupancy prediction, UniAD represents a planning-guided end-to-end autonomous driving method, and AIDE represents an automatic data engine method for object detection in autonomous driving.

[0078] In order to verify the effectiveness of the Tailor-Agent and Drive-Sora modules proposed by CorrectAD, an ablation experiment was conducted by alternately disabling each module. The experimental results are shown in Table 3.

[0079] Table 3: Ablation experiments of the two modules proposed by CorrectAD

[0080] The experiment set up different combinations of module activation: In the first case, both modules were disabled (no Tailor-Agent, no Drive-Sora). In this case, data was augmented using randomly distributed data copied and pasted from the training set. The experimental data showed that in this case, the L2 error was 0.54 meters, 1.03 meters, and 1.71 meters over different time periods (1, 2, and 3 seconds), respectively, with an average of 1.09 meters. The collision rate was 0.05%, 0.18%, and 0.81% over the corresponding time periods, with an average of 0.35%. This indicates that the random redundancy of this data has no positive impact on model performance.

[0081] The second scenario involves enabling the Drive-Sora module but not the Tailor-Agent module (without Tailor-Agent and with Drive-Sora). In this setup, using Drive-Sora to generate diverse training data, the L2 error was 0.53 meters, 0.99 meters, and 1.66 meters at 1, 2, and 3 seconds, respectively, with an average of 1.06 meters. The collision rate was 0.10%, 0.20%, and 0.62% at the corresponding time periods, with an average of 0.31%. This improvement compared to the first scenario suggests that using Drive-Sora to generate diverse training data addresses the issue of random data redundancy to a certain extent.

[0082] The third case involves enabling the Tailor-Agent module and disabling the Drive-Sora module (Tailor-Agent enabled, Drive-Sora disabled). In this case, the L2 error is 0.52m, 0.96m, and 1.62m at 1, 2, and 3 seconds, respectively, with an average of 1.03m. The collision rate is 0.08%, 0.20%, and 0.58% at the corresponding time periods, with an average of 0.29%. This demonstrates that the Tailor-Agent module also contributes to improved model performance.

[0083] The final scenario involves enabling both the Tailor-Agent and Drive-Sora modules simultaneously (with Tailor-Agent enabled and Drive-Sora disabled). Experimental results show that the L2 error dropped to 0.50m, 0.92m, and 1.53m in 1, 2, and 3 seconds, respectively, with an average of 0.98m. The collision rate in these time periods was 0.02%, 0.14%, and 0.42%, respectively, with an average of 0.19%.

[0084] Through the comparison of this series of experimental results, it can be clearly seen that, ultimately, by using Drive-Sora to generate rich and diverse data customized by Tailor-Agent, CorrectAD achieved the lowest L2 error and collision rate. This fully highlights the importance of the enhanced data distribution provided by Tailor-Agent and the data diversity provided by Drive-Sora in improving end-to-end model performance.

[0085] CorrectAD, with its self-correcting generation-training-evaluation closed-loop process, achieves end-to-end performance enhancements for autonomous driving models by continuously improving the quality of generated data and optimizing the model's failure response. By generating a rich variety of scenarios through multimodal cues, it enhances the model's task completion rate and generalization performance in complex and dynamic environments. It is suitable for a variety of real-world applications, including autonomous vehicles, intelligent transportation systems, and simulation test platforms.

[0086] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store end-to-end model data of autonomous driving. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for self-correction of an end-to-end model of autonomous driving is implemented.

[0087] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0088] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0089] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0090] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0091] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0092] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0093] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An end-to-end model self-correction method for autonomous driving, characterized by: The steps include: Evaluate multi-view images through an end-to-end autonomous driving model, detect failure cases, and extract intermediate output information from the perception, prediction, and planning stages; Parse the specific reasons for failure cases into multimodal prompts, combine bird's-eye view layout and scene description, and generate data requirements through visual language models; Based on the data requirements, using the Drive-Sora generation model to generate multi-view video data with spatiotemporal consistency and / or diversity; Use the generated multi-view video data as new training data to iteratively update and optimize the end-to-end autonomous driving model; Evaluate the performance of the updated model, analyze new failure cases, and enter the next round of data generation and optimization.

2. The method according to claim 1, characterized in that The end-to-end autonomous driving model evaluates multi-view images, detects failure cases, and extracts intermediate output information from the perception, prediction, and planning stages. Specifically, it includes: Multi-view images are fed into an end-to-end autonomous driving model for evaluation, with a focus on analyzing the collision risk predictions it outputs and filtering out failure cases from the original training set. Collect the intermediate outputs of the model, including but not limited to perception results, prediction results, and planning results, and transmit the above intermediate outputs together with multi-view images to subsequent modules for later data collection; The abstract structured coordinates output by the model are superimposed on the multi-view image, and the visualization results are transmitted to the subsequent data collection module from a bird's-eye view, enabling a comprehensive analysis of the model performance from both the first-person perspective and the global perspective.

3. The method according to claim 2, characterized in that The specific causes of failure cases are parsed into multimodal prompts, combined with the bird's-eye view layout and scene description, and the data requirements are generated through the visual language model, specifically including: Build a Tailor-Agent system, which includes multiple functional agents based on visual language models; The Tailor-Agent system breaks down the overall project goals into executable subtasks based on the project's overall goals and requirements, and assigns them to the cause analysis agent, requirement design agent, and multimodal prompt collection agent. The cause analysis agent performs in-depth analysis of the output of the end-to-end model, examining intermediate outputs from aspects including foreground targets, background environment, and / or weather to identify specific causes that may have led to planning failures. The demand design agent converts the cause analysis results output by the cause analysis agent into specific data requirements; The multimodal prompt collection agent constructs multimodal prompts based on the generated data requirements, calculates the similarity of images in the dataset using a visual language model, extracts the bird's-eye view layout of related images, and reconstructs the scene description information; Finally, multimodal prompts are collected and output.

4. The method according to claim 3, characterized in that The generating multi-view video data with spatiotemporal consistency and / or diversity using the Drive-Sora generation model based on the data requirements specifically includes: The Drive-Sora generative model receives as input the multimodal prompts generated by the Tailor-Agent system; New multi-view video data is generated through Drive-Sora and the quality of the generated data is evaluated.

5. The method according to claim 4, characterized in that The multi-view video data includes a layout structure of foreground and background, and the Drive-Sora generation model embeds the foreground data to generate a foreground embedding , the formula is: in, is a Fourier embedder used to process geometric data; It is a T5 text encoder used to convert dense description Convert to text embeddings; A multi-layer perceptron that fuses all input information and generates foreground embedding ; Background Layout Through the VAE encoder Processing, generating background embedding ; Scene description text Processed by T5 text encoder to generate text embedding ; Then, we integrate multimodal embeddings through the cross-attention mechanism. 、 and The specific process includes: Embed the foreground With text embedding Concatenate to form an overall feature representation, which serves as the key to the attention mechanism Sum enter; Background embedding As a query Input, integrated embedding calculation formula is as follows: in, is the initial input feature, represents the concatenation operation of foreground and text embeddings, represents the cross attention mechanism, is the dimension of the embedding vector.

6. The method according to claim 5, characterized in that The Drive-Sora generation model processes the generation task through the spatiotemporal diffusion transformer and introduces ControlNet for layout guidance to ensure the consistency of the generated video and the input prompt. The calculation process of the spatiotemporal diffusion transformer is as follows: in, For the convolutional layers to be trained, the initial value is set to zero; Refers to ControlNet-Transformer, which is used to process input features and background embedding integration.

7. The method according to claim 6, characterized in that The Drive-Sora generative model replaces the self-attention block in the spatiotemporal diffusion transformer with a multi-view spatial attention mechanism. The specific calculation process is as follows: in, represents the linear projection operation, Represents a dimension reshaping operation, Represents the multi-view spatial attention mechanism.

8. The method according to claim 7, characterized in that The Drive-Sora generative model uses a classifier-free guidance strategy to enhance the consistency of generated images with conditions, specifically including: During training, a single condition is randomly set to empty with a probability of 5%, and all conditions are set to "empty" at the same time with a probability of 5%; The final image is generated by combining the conditional generation results with the unconditional generation results, using the guided parameters 、 、 The balance between adjusting condition generation and sampling consistency is as follows: Among them, the generation network Depends on foreground frame parameters , background color parameters and text description Three conditions, Indicates a null condition.

9. An end-to-end model self-correction device for autonomous driving, characterized in that: include: A fault finding module is configured to evaluate multi-view images through an end-to-end autonomous driving model, identify failure cases, and extract intermediate output information from the perception, prediction, and planning stages; The data requirement generation module is configured to parse the specific causes of failure cases into multimodal prompts, combine the bird's-eye view layout and scene description, and generate data generation requirements through the visual language model; a multi-view data generation module configured to generate high-quality multi-view video data with spatiotemporal consistency and / or diversity using a Drive-Sora generation model based on the data generation requirements; A model iterative update module is configured to use the generated multi-view video data as new training data to iteratively update and optimize the end-to-end autonomous driving model; The performance evaluation module is configured to perform performance evaluation on the updated model, analyze newly emerged failure cases and trigger the next round of data generation and optimization cycle.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the autonomous driving end-to-end model self-correction method described in any one of claims 1 to 8 are implemented.

Citation Information

Patent Citations

  • Multi-modal end-to-end automatic driving method and system based on unified aerial view representation

    CN119049000A

  • Multi-view semantic recognition method based on depth map assistance

    CN119888738A

  • World model understanding-based end-to-end control method for computing base platform

    CN120057020A

  • Automatic driving large model framework based on 3D space-time perception and human-like decision-making reasoning

    CN120182938A

  • Training method and device of automatic driving neural network model and related equipment

    CN120219878A

Cited By

  • Driving scene recognition method and device and storage medium

    CN121305496A