Method for generating automatic driving scene data, electronic device, medium and product

Abstract

This application provides a method, electronic device, medium, and product for generating autonomous driving scene data, relating to the field of intelligent driving technology. The method involves acquiring performance evaluation data of an autonomous driving model; generating autonomous driving scene generation prompts based on a preset generative world model using the performance evaluation data; generating autonomous driving scene data matching the autonomous driving scene generation prompts based on the generative world model; and using the autonomous driving scene data for data-driven model parameter optimization training of the autonomous driving model. Using this application can improve the generation efficiency of long-tail problem scene data and accelerate the continuous iteration and optimization of vehicle autonomous driving system performance.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a method for generating autonomous driving scenario data, an electronic device, a medium, and a product. Background Technology

[0002] Currently, the performance iteration and safety improvement of vehicle autonomous driving systems highly rely on the core R&D model of "data-driven". For example, regarding the problem of diverse and rare "corner cases" in real-world driving scenarios that severely limit the performance improvement of vehicle autonomous driving systems, the "data closed loop" is a recognized core technology paradigm in the intelligent driving industry. It can collect real-world road data through large-scale operational fleets, identify "fault cases" and "performance bottlenecks" in autonomous driving systems through automatic identification or manual annotation, and then generate effective data for model retraining, thereby achieving continuous iteration and optimization of system performance.

[0003] However, in related technologies, the mainstream implementation schemes of vehicle autonomous driving data closed-loop systems all take real road data as the core data source. The operation of the entire process is highly dependent on real road data collected by large-scale fleets. Due to the extremely low probability of the high-risk "long tail problem" that truly threatens driving safety in natural road driving scenarios, the system has an inherent defect of scarce data in high-risk long tail problem scenarios, thus making it impossible to achieve effective iteration and performance improvement of vehicle autonomous driving systems in key risk scenarios. Summary of the Invention

[0004] The main objective of this application is to propose a method, electronic device, medium, and product for generating autonomous driving scenario data, aiming to improve the efficiency of generating long-tail problem scenario data, thereby accelerating the continuous iteration and optimization of the performance of the vehicle autonomous driving system.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for generating autonomous driving scene data, the method comprising: Obtain performance evaluation data for autonomous driving models; Based on the performance evaluation data, a preset generative world model is generated to provide prompts for autonomous driving scenarios. The generative world model generates autonomous driving scene data that matches the autonomous driving scene generation prompts; the autonomous driving scene data is used to perform data-driven model parameter optimization training on the autonomous driving model.

[0006] In some embodiments, the autonomous driving scenario generation prompt based on the performance evaluation data to generate a preset generative world model includes: Based on the performance evaluation data, the autonomous driving model is subjected to performance bottleneck diagnosis processing to obtain a structured bottleneck diagnosis report; The structured bottleneck diagnostic report is transformed into the optimization objective of the autonomous driving model; Based on the optimization objective, a preset generative world model is generated to provide prompts for autonomous driving scenarios.

[0007] In some embodiments, the process of performing performance bottleneck diagnosis on the autonomous driving model based on the performance evaluation data to obtain a structured bottleneck diagnosis report includes: Based on the performance evaluation data, a performance aggregation analysis is performed to obtain the failure characteristics of the autonomous driving model; Based on the failure characteristics, the root cause diagnosis of the autonomous driving model is performed to obtain the root cause of the failure of the autonomous driving model. The root cause of the failure is formatted to obtain a structured bottleneck diagnostic report.

[0008] In some embodiments, the autonomous driving scenario generation prompt based on the optimization objective to generate a preset generative world model includes: The optimization objective is deconstructed into a scene generation task; Based on the scenario, task definition scenario parameters are generated to obtain the autonomous driving scenario parameter combination; The combination of parameters for the autonomous driving scenario is processed by instruction compilation to obtain an autonomous driving scenario generation prompt based on a preset generative world model; the autonomous driving scenario generation prompt conforms to the input protocol of the generative world model.

[0009] In some embodiments, the method further includes: Acquire real-world driving log data; the real-world driving log data is generated by the vehicle deploying the autonomous driving model; Based on the real-world driving log data, the autonomous driving model is subjected to problem scenario mining processing to obtain real accident scenario data; The generative world model is iteratively optimized and trained based on the real accident scenario data.

[0010] In some embodiments, the iterative optimization training of the generative world model based on the real accident scenario data includes: The initial priority of the real accident scenario data is determined based on a preset multi-dimensional scenario priority evaluation index. The initial priority is dynamically adjusted based on a preset priority dynamic adjustment rule to obtain the adjusted priority; the priority dynamic adjustment rule includes at least an adjustment rule based on the computing power of the generative world model; The generative world model is iteratively optimized and trained based on the adjusted priorities and the real accident scenario data.

[0011] In some embodiments, the method further includes: The autonomous driving scenario data is mixed with the real accident scenario data to obtain a hybrid dataset; The autonomous driving model is trained to optimize its parameters based on the hybrid dataset, resulting in an optimized autonomous driving model.

[0012] To achieve the above objectives, a second aspect of this application proposes a system for generating autonomous driving scene data, the system comprising: The acquisition module is used to acquire performance evaluation data for autonomous driving models. The model prompting module is used to generate prompts for autonomous driving scenarios based on the performance evaluation data to generate a preset generative world model. The scene generation module is used to generate autonomous driving scene data that matches the autonomous driving scene generation prompts based on the generative world model; the autonomous driving scene data is used to perform data-driven model parameter optimization training on the autonomous driving model.

[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method for generating autonomous driving scene data as described in the first aspect.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for generating autonomous driving scenario data as described in the first aspect.

[0015] To achieve the above objectives, a fifth aspect of this application provides a computer program product, which includes a computer program that, when executed by a processor, implements the method for generating autonomous driving scene data as provided in the first aspect above.

[0016] The method, system, electronic device, computer-readable storage medium, and computer program product for generating autonomous driving scene data proposed in this application obtain performance evaluation data of an autonomous driving model; generate autonomous driving scene generation prompts based on the performance evaluation data using a preset generative world model; generate autonomous driving scene data that matches the autonomous driving scene generation prompts based on the generative world model; and use the autonomous driving scene data to perform data-driven model parameter optimization training on the autonomous driving model.

[0017] Compared to traditional methods that collect and process real-world road data to retrain autonomous driving models, this application's embodiment pre-sets a generative world model as the core data source. By generating autonomous driving scenario generation prompts based solely on the performance evaluation data of the autonomous driving model, it can generate matching autonomous driving scenario data based on this generative world model. This autonomous driving scenario data can then be used for data-driven model parameter optimization training of the autonomous driving model. This solves the problems of high scenario construction costs and insufficient coverage of key driving scenario data in traditional solutions, improving the efficiency of generating long-tail problem scenario data and accelerating the continuous iteration and optimization of vehicle autonomous driving system performance. Attached Figure Description

[0018] Figure 1 A flowchart illustrating the steps of the method for generating autonomous driving scene data provided in this application embodiment; Figure 2 for Figure 1 A detailed flowchart of step S102; Figure 3 A schematic diagram of the performance analysis and bottleneck diagnosis module involved in some embodiments of the method for generating autonomous driving scenario data provided in this application; Figure 4 A schematic diagram of the structure of the scene generation engine module involved in some embodiments of the method for generating autonomous driving scene data provided in this application; Figure 5 A flowchart illustrating the steps of the method for generating autonomous driving scene data provided in this application embodiment in other embodiments; Figure 6 A flowchart illustrating the steps of the method for generating autonomous driving scene data provided in this application in some other embodiments; Figure 7 A schematic diagram of the system structure involved in a complete embodiment of the method for generating autonomous driving scene data provided in the application embodiment; Figure 8A schematic diagram of the workflow involved in a complete embodiment of the method for generating autonomous driving scene data provided in this application embodiment; Figure 9 This is a schematic diagram of the structure of the autonomous driving scene data generation system provided in the embodiments of this application; Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] It should be noted that although functional modules are divided in the device / system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device / system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] First, a brief explanation of the overall concept of the method for generating autonomous driving scene data provided in the embodiments of this application will be given.

[0023] Currently, the evolution of autonomous driving systems relies heavily on a "data-driven" R&D model. Traditional rule-based or simple machine learning models cannot handle the massive, diverse, and rare "corner cases" that occur in the real world, such as irregularly placed traffic cones on construction sites, strange objects falling from vehicles ahead, and pedestrian behavior in extreme weather. To address these issues, "data closed-loop" has become an industry-recognized core technology paradigm. Its core idea is to collect real-world road data from a fleet, automatically or manually identify "fault cases" or "performance bottlenecks" in the system, and then generate targeted data for model retraining, thereby continuously improving the performance of the autonomous driving system.

[0024] However, the data closed-loop system in related technologies faces many bottlenecks, such as high data acquisition costs (maintaining a large-scale test fleet for road surveys requires huge hardware, manpower and time costs), scarcity of data for key scenarios (truly dangerous "long-tail problems" are hard to come by in natural road surveys, making it difficult for models to be effectively improved in these key scenarios), long iteration cycles (from problem discovery, data acquisition, labeling to model retraining and testing, the whole process is time-consuming and seriously affects development efficiency), and security risks (reproducing dangerous scenarios for testing on real roads or in closed test fields poses security risks), etc.

[0025] For example, traditional data closed-loop systems based on large-scale road surveys and simulations typically perform iterative optimization of autonomous driving models based on the following modules: Road Data Collection Module: Collects real-time sensor data (cameras, LiDAR, millimeter-wave radar, etc.) and vehicle status data 24 / 7 through a fleet of autonomous vehicles deployed globally; Data mining module: Utilize trigger rules (such as emergency braking, manual takeover) or automated analysis tools to mine "fault segments" that are not performing well from massive amounts of road data; Simulation testing module: These "fault fragments" are injected into the simulation environment, generalized and mutated to generate more similar scenarios in order to test the robustness of the system; Model training and evaluation module: Retrain perception, prediction, and planning models using enhanced scene data, and evaluate performance improvements in simulation and real vehicles; Closed-loop iteration: New models that meet performance standards are updated to the fleet via Over-the-Air Technology (OTA), initiating a new round of data collection and iteration.

[0026] However, the data source and scene generation of this approach heavily rely on real road survey data that has already occurred. Furthermore, although the simulation system can fine-tune the parameters of existing scenes (such as changing the weather and lighting), it is difficult to creatively and reasonably generate new, unseen but physically realistic complex interactive scenes, especially in the long tail region where data is sparse.

[0027] To address the aforementioned issues, this application proposes a method, system, electronic device, computer-readable storage medium, and computer program product for generating autonomous driving scenario data. The aim is to overcome the excessive reliance on real-world road data and efficiently, cost-effectively, and safely generate massive amounts of high-quality, highly diverse "long-tail problem" scenario data, thereby accelerating the iteration and maturity of autonomous driving systems.

[0028] This application embodiment obtains performance evaluation data of an autonomous driving model; generates autonomous driving scene generation prompts based on the performance evaluation data using a preset generative world model; generates autonomous driving scene data that matches the autonomous driving scene generation prompts based on the generative world model; and uses the autonomous driving scene data to perform data-driven model parameter optimization training on the autonomous driving model.

[0029] Compared to traditional methods of collecting and processing real road data to retrain autonomous driving models, this application embodiment pre-sets a core data source for a generative world model. By generating autonomous driving scene generation prompts for the generative world model based on the performance evaluation data of the autonomous driving model, autonomous driving scene data matching the autonomous driving scene generation prompts can be generated based on the generative world model. Thus, the autonomous driving scene data can be used for data-driven model parameter optimization training of the autonomous driving model.

[0030] Thus, this application introduces a generative world model as the system's virtual data engine to replace or greatly enhance the road data module in the traditional data loop. This eliminates the need to passively wait for road data and instead uses a trained generative world model that understands and simulates the laws of the physical world to directly obtain the required data through instructions (such as "generate a scene on a rainy night where pedestrians suddenly dart out from in front of a parked bus"). This eliminates the need for expensive and dangerous real-world data collection, directly solving the problems of high scene construction costs and insufficient coverage of key driving scenario data in traditional solutions. It can improve the generation efficiency of long-tail problem scenario data, thereby accelerating the continuous iteration and optimization of the vehicle's autonomous driving system performance.

[0031] Furthermore, this application's embodiments also employ a goal-oriented scene generation and evolution mechanism. By combining the generation process of the generative world model with the capability boundaries of the autonomous driving model, it first analyzes the failure cases of the current autonomous driving model in simulation testing, abstracting the model's weaknesses (e.g., inaccurate prediction of motorcycles crossing laterally). Then, this weakness is used as a guiding condition to drive the generative world model to generate a large number of relevant but diverse scenes (e.g., scenes of motorcycles crossing laterally at various speeds and angles). In this way, on-demand allocation and precise guidance of scene data generation can be achieved, thereby greatly improving the efficiency of data use and the speed of model iteration. This not only solves the problem of long iteration cycles in traditional solutions but also allows the model to proactively expose and repair its own defects.

[0032] Furthermore, the embodiments of this application can also perform hybrid looping and co-evolution of the autonomous driving model based on real and virtual data. That is, the iterative optimization of the autonomous driving model is not a purely virtual loop, but rather a mixture of high-quality virtual data generated by the generative world model and a small amount of key real road data for retraining the autonomous driving model. At the same time, novel and challenging cases discovered in the real world can also be used to fine-tune and improve the accuracy and realism of the generative world model itself, thereby forming a closed loop of virtual-real integration and co-evolution. In this way, it can ensure that the foundation of the autonomous driving model is always anchored in the real physical world, avoiding the model deviation problem that may be caused by purely virtual generation, while also amplifying the value of real data.

[0033] Next, the method, system, electronic device, computer-readable storage medium, and computer program product for generating autonomous driving scene data provided in this application will be specifically described through the following embodiments, and firstly, the various detailed embodiments of the method for generating autonomous driving scene data provided in this application will be described in detail.

[0034] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0035] It should be noted that the method for generating autonomous driving scenario data provided in this application relates to the field of intelligent driving technology. The method for generating autonomous driving scenario data provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be an in-vehicle terminal, or an electronic device such as a smartphone, tablet, laptop, or desktop computer that is associated with a vehicle and can communicate and interact with the vehicle via a network. The server can be a backend server terminal device of the terminal, which can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The software can be an application implementing the method for generating autonomous driving scenario data, a computer program, and a storage medium carrying the computer program. It should be understood that, based on different design needs of practical applications, the terminal, server, and software of the autonomous driving scene data generation method provided in this application embodiment may also be other forms not listed here, depending on the different feasible embodiments. The autonomous driving scene data generation method provided in this application embodiment does not specifically limit these.

[0036] Furthermore, this application can also be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: vehicle terminals, personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, personal computers (PCs), minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0037] For ease of understanding and explanation, the following text will use the method for generating autonomous driving scene data provided in the embodiments of this application on a terminal device as an example to describe the various specific embodiments of this application in detail. In some descriptions, the terminal device may be simply referred to as the terminal. The implementation of the method for generating autonomous driving scene data provided in the embodiments of this application for any of the above-described subjects can refer to the implementation process of the method for generating autonomous driving scene data described below.

[0038] Please refer to Figure 1 , Figure 1 The flowchart illustrates the steps of the method for generating autonomous driving scene data provided in this application embodiment in some embodiments. It should be understood that, although... Figure 1 The flowcharts illustrating subsequent steps show the execution order of some method steps. However, based on different design needs in practical applications, the method for generating autonomous driving scenario data provided in this application embodiment can, of course, employ a different execution order of method steps than shown in the figures. That is, Figure 1 The order of the method steps shown does not constitute a limitation on the execution logic order of the autonomous driving scene data generation method provided in the embodiments of this application. Any other method based on this method is not limited to this method. Figure 1 Reasonable changes to the sequence of steps shown should be included within the protection scope of the autonomous driving scenario data generation method provided in the embodiments of this application.

[0039] like Figure 1 As shown, in some embodiments, the method for generating autonomous driving scenario data provided in this application may include steps S101 to S103 as shown below.

[0040] Step S101: Obtain performance evaluation data of the autonomous driving model.

[0041] It should be noted that the autonomous driving model can be a model deployed on a vehicle to enable the vehicle to perform autonomous driving-related functions. This autonomous driving model can be an integrated algorithm model trained based on machine learning / deep learning. As the "brain" of the vehicle to achieve autonomous perception, judgment and driving, it can take environmental / vehicle data collected by vehicle sensors (cameras, millimeter-wave radar, lidar, positioning, etc.), high-precision maps and positioning information as input. After the model calculates, it directly outputs the vehicle's driving decisions and control commands, thereby assisting / replacing human drivers in completing driving operations.

[0042] When generating autonomous driving scenario data to improve the performance of the autonomous driving model, the terminal device first obtains the performance evaluation data of the autonomous driving model.

[0043] In some embodiments, the terminal device can use a simulation evaluation platform to evaluate the performance of the autonomous driving model in real time, thereby obtaining the performance evaluation data of the autonomous driving model.

[0044] For example, a simulation evaluation platform can be a high-concurrency, scalable cloud-based simulation system. Its core internal structure includes: a scenario library (storing a large number of standard test scenarios and hazardous scenarios), a simulation kernel (capable of running autonomous driving models at high speed and calculating their interaction results with the simulation environment), and an evaluation matrix that can be a set of well-defined performance indicators (KPIs) (such as safety indicators, comfort indicators, and traffic efficiency). Based on this, terminal devices can input the relevant test scenario sets of the autonomous driving model into the simulation evaluation platform. The platform then performs simulation evaluations on these test scenario sets, generating detailed, quantitative performance evaluation reports (including scores for each KPI and detailed logs) and outputting them. The terminal device can then use these performance evaluation reports as performance evaluation data for the autonomous driving model.

[0045] In some embodiments, the terminal device can also directly receive the relevant performance evaluation data of the autonomous driving model manually defined and generated by the user (e.g., "pedestrians cannot be detected when illuminated by the high beams of oncoming vehicles at night") through a preset human-machine interface (HMI), and then generate the performance evaluation data of the autonomous driving model (e.g., including safety index scores and corresponding evaluation data) through natural language processing (NLP) tools.

[0046] Step S102: Generate autonomous driving scenario generation prompts based on the performance evaluation data and a preset generative world model.

[0047] It should be noted that the pre-defined generative world model can be a generative world model pre-trained by the terminal device that can understand and simulate the laws governing the physical world. For example, the generative world model can be the NVIDIA Cosmos-Drive open-source model (which is post-trained based on the world base model cosmos-transfe model using the Road Data Survey (RDS) dataset).

[0048] After obtaining the performance evaluation data of the autonomous driving model, the terminal device can further generate autonomous driving scene generation prompts based on the performance evaluation data, which can guide the generative world model to perform relevant scene generation tasks (such as "generate more scenes of pedestrians suddenly running in on rainy nights").

[0049] In some embodiments, the terminal device can perform performance analysis on the autonomous driving model based on performance evaluation data to identify the model's weaknesses, and generate autonomous driving scenario generation prompts based on the identified results.

[0050] In some embodiments, the terminal device can generate autonomous driving scene generation prompts through a preset scene generation engine.

[0051] For example, a scene generation engine can be a pre-configured planning and scheduling system for the terminal device. It includes a target parser that transforms high-level, fuzzy descriptions into specific, executable generation task instructions. Thus, after identifying weaknesses in the autonomous driving model based on performance evaluation data analysis, the terminal device can input a description of these weaknesses into the scene generation engine. The engine then parses this description to generate specific instructions that the generative world model can understand and execute. In this way, the terminal device can use the instructions generated by the scene generation engine as prompts for autonomous driving scene generation within the generative world model.

[0052] Step S103: Generate autonomous driving scene data that matches the autonomous driving scene generation prompt based on the generative world model; the autonomous driving scene data is used to perform data-driven model parameter optimization training on the autonomous driving model.

[0053] It should be noted that autonomous driving scenario data can be autonomous driving simulation test scenario data for vehicles (also known as autonomous driving simulation test scenarios). Its composition usually includes static elements (road structure, traffic signs, buildings, etc.), dynamic elements (vehicles, pedestrians, bicycles, etc.), and environmental elements (lighting, rain, snow, fog, etc.). In essence, it refers to the dynamic interaction description of autonomous driving vehicles and environmental elements (roads, traffic participants, weather, etc.) under specific time and task conditions. It is the core basis for verifying the safety and robustness of autonomous driving. Internationally, standardized formats such as OpenSCENARIO and OpenDRIVE are commonly used for description.

[0054] After the terminal device generates a prompt for the autonomous driving scenario, it can input the prompt into a preset generative world model. Based on the prompt, the generative world model is guided to perform a scenario generation task, generate autonomous driving scenario data that matches the prompt, and output it.

[0055] For example, a generative world model is a multimodal artificial intelligence (AI) model (such as the open-source NVIDIA COSMOS model) that can understand and physically simulate traffic environments. After the terminal device inputs autonomous driving scenario generation prompts (such as "generate more scenarios of pedestrians suddenly running in on rainy nights") into the generative world model, the generative world model can generate massive amounts of high-fidelity and diverse virtual scene data based on the prompts. This data can be specifically used to enhance the known weaknesses of the autonomous driving model and then output. In this way, the terminal device uses this virtual scene data as autonomous driving scenario data that matches the autonomous driving scenario generation prompts.

[0056] After generating autonomous driving scenario data based on a generative world model, the terminal device can use this autonomous driving scenario data to perform data-driven model parameter optimization training on the current autonomous driving model, thereby continuously iterating and optimizing the performance of the autonomous driving model based on a data closed loop.

[0057] In some embodiments, during the process of continuously iteratively optimizing the performance of the autonomous driving model based on data closed loop, the terminal device can further obtain the performance evaluation data of the trained autonomous driving model, and continue to execute the above steps of generating autonomous driving scene generation prompts based on performance evaluation data and generating matching autonomous driving scene data based on generative world model, so as to continue to optimize and train the autonomous driving model based on the newly generated autonomous driving scene data in a new loop.

[0058] In some embodiments, the terminal device can use a driving model training platform to optimize the model parameters of the autonomous driving model, thereby obtaining a new version of the autonomous driving model after training.

[0059] For example, a driving model training platform can be a distributed AI model training infrastructure, typically including a task scheduler that manages GPU / TPU clusters and supports training frameworks such as PyTorch and TensorFlow. Its automated pipeline for training autonomous driving models includes nodes such as data loading, model building, loss calculation, backpropagation, model validation, and checkpoint saving. Based on this, terminal devices can sample training data from autonomous driving scenario data and input this training data into the driving model training platform. The platform then uses this training data to automatically train the current autonomous driving model (such as a perception, prediction, planning, or decision-making model, or an ensemble model), resulting in a new version of the autonomous driving model, which is then output.

[0060] In this embodiment, when generating autonomous driving scene data, the terminal device first acquires the performance evaluation data of the autonomous driving model. Then, based on this performance evaluation data, it generates autonomous driving scene generation prompts that guide the generative world model to perform relevant scene generation tasks. Finally, these prompts are input into a preset generative world model, which then guides the model to perform scene generation tasks, generating and outputting autonomous driving scene data that matches the prompts. In this way, the terminal device can use the autonomous driving scene data to perform data-driven model parameter optimization training on the current autonomous driving model, thereby continuously iteratively optimizing the performance of the autonomous driving model based on a data closed loop.

[0061] Compared to traditional methods of collecting and processing real road data to retrain autonomous driving models, this application embodiment pre-sets a core data source for a generative world model. By generating autonomous driving scene generation prompts for the generative world model based on the performance evaluation data of the autonomous driving model, autonomous driving scene data matching the autonomous driving scene generation prompts can be generated based on the generative world model. Thus, the autonomous driving scene data can be used for data-driven model parameter optimization training of the autonomous driving model.

[0062] Thus, this application introduces a generative world model as the system's virtual data engine to replace or greatly enhance the road data module in the traditional data loop. This eliminates the need to passively wait for road data and instead uses a trained generative world model that understands and simulates the laws of the physical world to directly obtain the required data through instructions (such as "generate a scene on a rainy night where pedestrians suddenly dart out from in front of a parked bus"). This eliminates the need for expensive and dangerous real-world data collection, directly solving the problems of high scene construction costs and insufficient coverage of key driving scenario data in traditional solutions. It can improve the generation efficiency of long-tail problem scenario data, thereby accelerating the continuous iteration and optimization of the vehicle's autonomous driving system performance.

[0063] Please refer to Figure 2 , Figure 2 for Figure 1 A detailed flowchart of step S102.

[0064] like Figure 2 As shown, in some embodiments, step S102 above: generating autonomous driving scenario generation prompts based on the performance evaluation data to generate a preset generative world model may include steps S201 to S203 as shown below.

[0065] Step S201: Based on the performance evaluation data, perform performance bottleneck diagnosis processing on the autonomous driving model to obtain a structured bottleneck diagnosis report.

[0066] When generating prompts for autonomous driving scenarios based on performance evaluation data to create a generative world model, the terminal device can first perform performance bottleneck diagnosis processing on the autonomous driving model based on the performance evaluation data, thereby obtaining a structured bottleneck diagnosis report for the autonomous driving model.

[0067] For example, a terminal device can use a data analysis and attribution platform to perform performance analysis and bottleneck diagnosis on an autonomous driving model using performance evaluation data as input and techniques such as probable cause analysis and ablation experiments. This yields a clear description of the performance bottlenecks output by the platform (identifying the problem type (e.g., "the perception module has a high rate of missing pedestrians under the halo of high beams at night") and the root cause hypothesis). The terminal device can then format the performance bottleneck descriptions output by the platform into a structured report, which can serve as a structured bottleneck diagnosis report for the autonomous driving model.

[0068] Step S202: Transform the structured bottleneck diagnosis report into the optimization objective of the autonomous driving model.

[0069] After receiving the structured bottleneck diagnostic report of the autonomous driving model, the terminal device can further transform and process the structured bottleneck diagnostic report to obtain the measurable optimization target of the autonomous driving model.

[0070] For example, after the terminal device processes performance evaluation data using technologies such as probable cause analysis and ablation experiments based on a data analysis and attribution platform to generate a clear description of performance bottlenecks, it can further transform this description of performance bottlenecks directly into specific and measurable improvement goals (such as "improving the perception module's perception results for pedestrians heavily obscured by rain at night"). In this way, the terminal device can use this improvement goal as the optimization target for the current autonomous driving model.

[0071] Step S203: Generate autonomous driving scene generation prompts based on the preset generative world model according to the optimization objective.

[0072] After obtaining the optimization target of the autonomous driving model, the terminal device can compile instructions based on the optimization target to generate autonomous driving scene generation prompts that the generative world model can understand and execute. Based on these prompts, the generative world model is guided to execute relevant scene generation tasks to generate corresponding autonomous driving scene data.

[0073] For example, the terminal device inputs the optimization goal of the current autonomous driving model into the scene generation engine mentioned above, and then, based on the scene generation engine, transforms the optimization goal into specific generation task instructions that the generative world model can understand and execute, thereby obtaining the autonomous driving scene generation prompt of the generative world model.

[0074] In some embodiments, step S201 above: performing performance bottleneck diagnosis processing on the autonomous driving model based on the performance evaluation data to obtain a structured bottleneck diagnosis report, may include the following steps: Based on the performance evaluation data, a performance aggregation analysis is performed to obtain the failure characteristics of the autonomous driving model; Based on the failure characteristics, the root cause diagnosis of the autonomous driving model is performed to obtain the root cause of the failure of the autonomous driving model. The root cause of the failure is formatted to obtain a structured bottleneck diagnostic report.

[0075] It should be noted that failure characteristics can be the sum of behavioral patterns and quantitative representations of autonomous driving models within a preset operational design domain (such as perception, prediction, planning and decision-making), where insufficient performance leads to output results deviating from design expectations and failing to meet functional safety and performance requirements. Failure characteristics can include anomaly indicators and the identification of high-frequency failure scenario types. Anomaly indicators are the quantitative dimension of failure characteristics, while the identification of high-frequency failure scenario types is the scenario distribution dimension of failure characteristics. Together, they can comprehensively depict the full-dimensional patterns of performance failure in autonomous driving models.

[0076] When a terminal device performs performance bottleneck diagnosis on an autonomous driving model based on performance evaluation data to generate a structured bottleneck diagnosis report, it can first perform performance aggregation analysis on the autonomous driving model based on the performance evaluation data to identify abnormal indicators and failure characteristics such as high-frequency failure scenarios. Then, the terminal device can further perform root cause diagnosis based on these failure characteristics, combining possible cause analysis and controlled ablation experiments to pinpoint the root cause of the autonomous driving model's failure. Finally, the terminal device can format the root cause of the failure to generate a structured bottleneck diagnosis report for the autonomous driving model.

[0077] For example, the terminal device can input the performance evaluation data (simulation evaluation report and detailed logs) of the autonomous driving model as follows: Figure 3 The performance analysis and bottleneck diagnosis module shown performs performance bottleneck diagnosis on the autonomous driving model based on the performance evaluation data, thereby obtaining a structured bottleneck diagnosis report for the autonomous driving model. This module includes a performance aggregation layer, a root cause diagnosis layer, and an optimization target generation layer. Among them: The performance aggregation layer includes a Key Performance Indicator (KPI) dashboard, which aggregates various metrics from simulation evaluation reports (perception result metrics such as Mean Average Precision (mAP), accuracy, recall, etc., and other metrics such as collision rate, takeover rate, etc.) for trend analysis and version comparison. Furthermore, the performance aggregation layer also performs failure scenario clustering for autonomous driving models based on performance evaluation data. That is, it uses unsupervised learning (such as clustering algorithms) to automatically group thousands of failure cases and quickly identify high-frequency and typical failure patterns of autonomous driving models (such as "all failures are due to pedestrians crossing at night in the rain").

[0078] The root cause diagnosis layer primarily analyzes possible causes and conducts controlled ablation experiments for verification. Specifically, for failed scenarios, it utilizes techniques such as feature visualization (e.g., CAM technology, which generates a heatmap by calculating the correlation between the feature maps of the last few layers of the model and the final prediction score, and then overlays it onto the original image; the hotter the area in the heatmap, the greater its contribution to the prediction) to explore the internal decision-making logic of the autonomous driving model (e.g., whether the perception missed a target or the predicted trajectory was incorrect). Then, in the ablation experiment analysis phase, a single variable of the scene is controlled and changed in simulation (e.g., removing an obstacle, changing weather or lighting conditions), and the model's performance is observed to see if it recovers, thus verifying the root cause hypothesis.

[0079] The optimization target generation layer is used to generate bottleneck reports: it formats the diagnostic results into a structured bottleneck diagnostic report, clearly identifying weak points and root cause hypotheses. Furthermore, the optimization target generation layer also formulates target instructions: it transforms the structured bottleneck diagnostic report into specific optimization targets that the "scene generation engine" can understand (e.g., "improve the perception module's perception results for pedestrians with severe occlusion on rainy night days").

[0080] In some embodiments, step S203 above: generating autonomous driving scene generation prompts based on the preset generative world model according to the optimization objective, may include the following steps: The optimization objective is deconstructed into a scene generation task; Based on the scenario, task definition scenario parameters are generated to obtain the autonomous driving scenario parameter combination; The combination of parameters for the autonomous driving scenario is processed by instruction compilation to obtain an autonomous driving scenario generation prompt based on a preset generative world model; the autonomous driving scenario generation prompt conforms to the input protocol of the generative world model.

[0081] After the terminal device transforms the structured bottleneck diagnostic report into the optimization objective of the autonomous driving model, and further generates autonomous driving scene generation prompts based on the generative world model based on the optimization objective, it can first deconstruct the optimization objective into one or more scene generation tasks. Then, based on the scene generation task, the scene parameter space is defined to obtain the combination of autonomous driving scene parameters. Finally, by using an index compiler that is compatible with the input protocol of the generative world model, the combination of autonomous driving scene parameters is compiled into instructions to obtain autonomous driving scene generation prompts that conform to the input protocol.

[0082] For example, the scene generation engine module used by the terminal device can be as follows: Figure 4 As shown, this module takes the optimization objective output by the performance analysis and bottleneck diagnosis module as input, and sequentially processes it through the objective deconstruction layer, the objective parameterization layer, and the specified generation and distribution layer to obtain accurate autonomous driving scene generation prompts, which are then output to the generative world model. Among these: The target deconstruction layer primarily uses a target deconstructor to break down high-level optimization targets into specific, quantifiable scene generation tasks. For example, the optimization target "improving the perception module's perception results for pedestrians heavily obscured by rain at night" is deconstructed into the scene generation task "generating a combined scene of 'nighttime + rain + obscured pedestrians'".

[0083] The target parameterization layer primarily handles the definition of scene parameters, specifically defining the scene parameter space and arranging and combining scene parameters based on the scene generation task, thereby obtaining the combination of autonomous driving scene parameters. The scene parameter space definition involves defining adjustable parameter dimensions for each task. These dimensions include: environmental dimensions (lighting conditions, such as light intensity and angle), weather (precipitation type and intensity), and visibility; road topology dimensions (road type, number of lanes, and traffic sign / signal light status); dynamic element dimensions (traffic participant types, pedestrians, non-motorized vehicles, and special objects), behavior patterns (constant speed, acceleration, cutting out, and crossing), and interaction complexity (traffic flow and density); and key event dimensions (whether it involves conflict and whether it belongs to a long-tail scenario, such as special vehicles or animals). Furthermore, the scene parameter arrangement and combination involves arranging and combining the selected parameters to maximize the coverage of that type of scenario.

[0084] The instruction compilation and distribution layer mainly uses an instruction compiler to combine parameters into prompts (autonomous driving scenario generation prompts) that can be understood and executed by the "generative world model". The file format is as follows: #clip1.json {"sunny": "The weather is sunny, located in the highway, surrounded by wire rope barriers and support poles.", "rainy": "The weather is rainy, located in the city center, surrounded by villa area."}.

[0085] In some embodiments, when defining scene parameters, if relevant dimensions are missing during target deconstruction, the target parameterization layer can automatically fill in the missing dimensions. For example, when deconstructing "enhancing the perception result of the perception module for heavily occluded pedestrians in nighttime rain" into "nighttime + rain + occluded pedestrians", due to the lack of dimensions such as road topology, the target parameterization layer can fill in the missing dimensions to obtain "nighttime + rain + road topology + occluded pedestrians", and then perform parameter combination.

[0086] In some embodiments, after compiling and distributing the generated instructions to obtain the autonomous driving scenario generation prompts, the generation instruction compilation and distribution layer can further perform quality control and verification processes. By logically verifying the generated prompt instructions, the physical rationality and effectiveness of the autonomous driving scenario generated by the generative world model based on the prompts can be ensured.

[0087] In this embodiment, the terminal device first performs performance bottleneck diagnosis on the autonomous driving model based on performance evaluation data, obtaining a structured bottleneck diagnosis report. This report is then transformed into an optimization objective for the autonomous driving model. Subsequently, based on this optimization objective, a preset generative world model is generated to provide autonomous driving scene generation prompts. Thus, by employing a goal-oriented scene generation and evolution mechanism, the process of generating autonomous driving scenes using the generative world model is combined with the capability boundaries of the autonomous driving model. First, the failure cases of the current autonomous driving model in simulation testing are analyzed to abstract the model's weaknesses (e.g., inaccurate prediction of lateral motorcycle crossings). Then, these weaknesses are used as guiding conditions to drive the generative world model to generate a large number of relevant but diverse autonomous driving scenes (e.g., lateral motorcycle crossing scenes at various speeds and angles). This enables the generation of autonomous driving scene data to have on-demand allocation and precise guidance characteristics, thereby greatly improving the efficiency of autonomous driving scene data utilization and the speed of model optimization iteration. This not only solves the problem of long iteration cycles in traditional solutions but also allows the model to proactively expose and repair its own defects.

[0088] Please refer to Figure 5 , Figure 5 The flowcharts of the method for generating autonomous driving scene data provided in this application are shown in some other embodiments.

[0089] like Figure 5 As shown, in some embodiments, the method for generating autonomous driving scenario data provided in this application may further include steps S501 to S503 as shown below.

[0090] Step S501: Obtain real-world driving log data; the real-world driving log data is generated by the vehicle deploying the autonomous driving model.

[0091] It should be noted that real-world driving log data can be generated by vehicles equipped with autonomous driving models and transmitted to terminal devices. This real-world driving log data can include sensor data, vehicle status, driving decisions, etc.

[0092] While using the generative world model to generate autonomous driving scenario data, the terminal device can also continuously iterate and optimize the generative world model. Based on this, the terminal device obtains real-world driving log data generated by vehicles that have deployed the autonomous driving model.

[0093] For example, a fleet of autonomous vehicles equipped with autonomous driving models can be formed. Each vehicle in the fleet is equipped with a sensor suite (LiDAR, cameras, millimeter-wave radar, etc.), a computing unit, and a data logger. The fleet management platform is responsible for monitoring vehicle status, task scheduling, and data transmission. Vehicles in the autonomous driving fleet receive over-the-air (OTA) updates (from iteratively optimized versions of the autonomous driving model) and / or human driver takeover commands or interventions as input. When encountering a "corner case" (such as a rare accident or dangerous scenario) that the autonomous driving model struggles to handle, a special flag is triggered, prioritizing the transmission of relevant scenario data to terminal devices (such as a cloud-based management platform). In this way, the terminal devices can access massive amounts of real-world driving logs (including sensor data, vehicle status, driving decisions, etc.) output by the autonomous driving fleet.

[0094] Step S502: Based on the real-world driving log data, perform problem scenario mining processing on the autonomous driving model to obtain real accident scenario data.

[0095] During the iterative optimization of the generative world model, after obtaining real-world driving log data, the terminal device further performs problem scenario mining processing on the autonomous driving model based on the real-world driving log data, thereby obtaining real accident scenario data with high value for optimizing the generative world model.

[0096] For example, terminal devices can process real-world driving log data through a pre-designed problem scenario mining module. This module can be a pre-designed analysis platform based on big data and artificial intelligence (AI), which may include automated anomaly detection algorithms (such as rule engines and machine learning models), expert annotation and review interfaces, and a scenario priority evaluation system. This platform can automatically filter out segments containing events such as sudden braking, aggressive steering, and manual intervention from massive amounts of logs and assign them corresponding priority tags. Terminal devices can input massive amounts of real-world driving log data from autonomous driving fleets into this analysis platform, which will then process the log data to filter out seed scenarios (screened and labeled, high-value, complex scenarios (especially real accidents or high-risk near-miss accident data)). The terminal device can then use these seed scenarios as real-world accident scenario data for subsequent optimization of the generative world model.

[0097] Step S503: Iteratively optimize and train the generative world model based on the real accident scene data.

[0098] After obtaining real-world accident scenario data, the terminal device can iteratively optimize and train the generative world model based on this data, thereby obtaining a new generative world model. This new model will serve as the core data source for generating corresponding autonomous driving scenario data based on autonomous driving scenario prompts during the next round of iterative optimization of the autonomous driving model.

[0099] In some embodiments, step S503: iteratively optimizing and training the generative world model based on the real accident scene data may include the following steps: The initial priority of the real accident scenario data is determined based on a preset multi-dimensional scenario priority evaluation index. The initial priority is dynamically adjusted based on a preset priority dynamic adjustment rule to obtain the adjusted priority; the priority dynamic adjustment rule includes at least an adjustment rule based on the computing power of the generative world model; The generative world model is iteratively optimized and trained based on the adjusted priorities and the real accident scenario data.

[0100] It should be noted that the multi-dimensional scenario priority assessment indicators include at least two of the following: risk impact level, actual occurrence frequency, correlation with technical bottlenecks, urgency of business needs, and adaptability to generative computing resources.

[0101] When terminal devices iteratively optimize and train generative world models based on real-world accident scenario data, they can use a combination of "multi-dimensional dynamic weighted evaluation + generative model computing power scheduling linkage" to add priority labels to the real-world accident scenario data. Then, based on these priority labels, the generative world model is optimized and trained. This ensures that when generating autonomous driving scenario data for iterative optimization of the autonomous driving model, the generative world model prioritizes the production of scenario data that is most critical to safety improvement, has the most prominent technical bottlenecks, and has the most urgent business needs. Specifically, the terminal device first determines the initial priority of the real-world accident scenario data based on at least two of the following indicators: risk impact level, actual occurrence frequency, correlation with technical bottlenecks, urgency of business needs, and adaptability to generative computing resources. Then, it further uses a dynamic priority adjustment rule based at least on the computing power of the generative world model to dynamically adjust this initial priority, obtaining the adjusted priority. Finally, the terminal device can iteratively optimize and train the generative world model based on the adjusted priority and the real-world accident scenario data (for example, converting the highest-priority scenario data into a generated prompt that the world model can understand and execute, and inputting this prompt into the world model for model parameter optimization training).

[0102] In some embodiments, the terminal device can use the problem scenario mining module to process real-world driving log data to filter out seed scenarios, and then use a scenario priority strategy to perform multi-dimensional dynamic weighted evaluation of each problem scenario and generative model-based computing power scheduling, thereby outputting seed scenarios with priority labels.

[0103] In some embodiments, when the multi-dimensional scenario priority assessment indicators include risk impact level, actual occurrence frequency, correlation with technical bottlenecks, urgency of business needs, and adaptability to generative computing resources, the scenario priority assessment dimension evaluation criteria may include: The assessment criteria for the risk impact level dimension include: fatal risk (such as death from a collision with a pedestrian / vehicle) > serious injury risk (such as a high-speed rear-end collision) > minor injury risk > no risk; The evaluation criteria for the actual frequency dimension include: high frequency (≥10 times / day / vehicle, such as following vehicle start-stop) > medium frequency (1-9 times / day / vehicle) > low frequency (1-10 times / week / vehicle) > very low frequency (≤1 time / month / vehicle). The evaluation criteria for the technical bottleneck correlation dimension include: strong correlation with the current model's weak links (such as high-frequency failure scenarios marked by the performance analysis module, with an average of >5 failures per month) > medium correlation (1 to 5 failures per month) > weak correlation (<1 failure per month). The evaluation criteria for the urgency of business needs include: core needs (such as mandatory high-speed autonomous navigation driving (NOA)) > key needs (such as key NOA in urban areas) > routine needs (such as scenario completion) > exploratory needs (such as niche scenarios). The evaluation criteria for the generative computing resource adaptability dimension include: high adaptability (complex simulation / multi-vehicle interaction) > medium adaptability (normal simulation) > low adaptability (simple scenario) > no adaptability (cannot be reproduced).

[0104] In some embodiments, when the multi-dimensional scenario priority assessment indicators include risk impact level, actual occurrence frequency, technical bottleneck correlation, business demand urgency and generative computing resource adaptability, the dynamic weight allocation and corresponding scoring rules of the risk impact level indicator are shown in Table 1 below.

[0105] Table 1

[0106] In addition, the dynamic weighting of the actual occurrence frequency index and the corresponding scoring rules are shown in Table 2 below.

[0107] Table 2

[0108] The dynamic weighting and corresponding scoring rules for the technical bottleneck correlation index are shown in Table 3 below.

[0109] Table 3

[0110] The dynamic weighting and corresponding scoring rules for the business demand urgency index are shown in Table 4 below.

[0111] Table 4

[0112] The dynamic weighting and corresponding scoring rules for the generative computing resource suitability index are shown in Table 5 below.

[0113] Table 5

[0114] In some embodiments, based on the dynamic weight allocation and corresponding scoring rules of the above-mentioned multi-dimensional scenario priority evaluation indicators, the terminal device can calculate the priority score of the real accident scenario data according to the total score formula shown below.

[0115] .

[0116] In some embodiments, the mapping relationship between the priority level of real accident scenario data and the computing power of the generative world model can be shown in Table 6 below.

[0117]

[0118] For example, for real accident scenario data S001: high-speed following vehicle suddenly braking (fatal + 8 times per day + model braking delay + high-speed NOA core + high adaptability), based on the above scoring rules, its risk impact level score is determined to be 3.5, the actual occurrence frequency score is 3.0, the technical bottleneck correlation score is 3.5, the business demand urgency score is 1.5, the case adaptation score is 1.0, and the final priority score is 12.5. Therefore, the priority level of scenario S001 can be determined to be P0, and the computing power label is high computing power. For the real-world accident scenario data S002: pedestrians crossing an unlit intersection on a rainy night (minor injury + 3 times per week + insufficient perception + key NOA area in urban areas + medium adaptation), based on the above scoring rules, we can determine its risk impact level score as 2.5, actual occurrence frequency score as 0.8, technical bottleneck correlation score as 3.0, business demand urgency score as 1.2, and computational case adaptation score as 0.7, with a final priority score of 8.2. Therefore, we can determine that the priority level of scenario S002 is P1, and the computing power label is medium computing power.

[0119] In some embodiments, the computing power adjustment rule based on the generative world model can be: when computing power is strained, it can be tilted to the processing priority level of P0 / P1. For example, when the computing power of the generative model is strained, the adjustment logic of "pausing the generation of P3 scene and tilting all computing power to P0 / P1" can be executed.

[0120] In some embodiments, when the terminal device dynamically adjusts the initial priority based on the priority dynamic adjustment rules, it can also combine simulation evaluation and real fleet data to update the scene priority in real time. For example, when the model's accuracy in scene P0 is ≥95%, the weight of the technical bottleneck correlation in that scene is automatically reduced, and the priority is downgraded to P1; when the frequency of scene P2 occurrences is found to be high in real fleets, the weight of the actual occurrence frequency is automatically increased, and the priority is upgraded to P0; and when the technical bottleneck is resolved (such as when the perception accuracy meets the standard), the weight of the technical bottleneck correlation in the corresponding scene is reduced, and the priority is downgraded.

[0121] Please refer to Figure 6 , Figure 6 A flowchart illustrating the steps of the method for generating autonomous driving scene data provided in this application in some other embodiments.

[0122] like Figure 6As shown, in some embodiments, the method for generating autonomous driving scenario data provided in this application may further include steps S601 and S602 as shown below.

[0123] Step S601: Mix the autonomous driving scenario data with the real accident scenario data to obtain a hybrid dataset; Step S602: Based on the hybrid dataset, drive the autonomous driving model to perform model parameter optimization training to obtain the optimized autonomous driving model.

[0124] When the terminal device iteratively optimizes and trains the autonomous driving model based on the autonomous driving scenario data output by the generative world model, and obtains real accident scenario data by mining problem scenarios based on real-world driving log data, the terminal device can also mix the autonomous driving scenario data with the real accident scenario data to form a hybrid dataset, and then sample training data based on the hybrid dataset to optimize and train the model parameters of the current autonomous driving model.

[0125] For example, when a terminal device uses a driving model training platform to optimize the parameters of the current autonomous driving model, it can mix autonomous driving scenario data with real accident scenario data to form a mixed dataset. Then, it can sample training data from this mixed dataset and input the training data into the driving model training platform. The driving model training platform can then use the training data to automatically train the current autonomous driving model, obtain a new version of the autonomous driving model after training, and output it.

[0126] In this embodiment, real-world driving log data uploaded by the autonomous driving fleet is acquired through a terminal device. Based on this real-world driving log data, the autonomous driving model undergoes problem scenario mining processing to obtain real accident scenario data. Then, the generative world model is iteratively optimized and trained based on the priority labels carried by this real accident scenario data. Furthermore, the autonomous driving scenario data and the real accident scenario data are mixed through the terminal device to obtain a hybrid dataset. This hybrid dataset is then used to drive the autonomous driving model to perform model parameter optimization training, resulting in an optimized autonomous driving model.

[0127] Thus, this embodiment, based on the hybrid circulation and co-evolution of real and virtual data, ensures that the iterative optimization of the autonomous driving model is not a purely virtual loop. Instead, it mixes high-quality virtual data generated by the world model with a small amount of key real-world road data for model retraining. Simultaneously, novel and challenging cases discovered in the real world are used to fine-tune and improve the accuracy and realism of the world model itself, forming a closed loop of virtual-real integration and co-evolution. This ensures that the foundation of the autonomous driving model remains anchored in the real physical world, avoiding the model deviation problem that might arise from purely virtual generation of scene data dependent on the world model, while simultaneously amplifying the value of relevant real-world data.

[0128] Next, a complete embodiment of the method for generating autonomous driving scene data provided in this application will be presented.

[0129] Please refer to Figure 7 , Figure 7 A schematic diagram of the system structure involved in a complete embodiment of the method for generating autonomous driving scene data provided in the application embodiment.

[0130] like Figure 7 As shown, when a terminal device applies the autonomous driving scenario data generation method provided in this application embodiment to iteratively optimize an autonomous driving model using a generative world model as the core data source, firstly, in the real scenario acquisition and problem scenario mining stage, an autonomous driving fleet serves as the terminal data acquisition and model deployment carrier. The autonomous driving fleet outputs the autonomous driving operation data generated during real road tests to the problem scenario mining module. The problem scenario mining module identifies and mines problem scenarios based on the received autonomous driving operation data, outputting seed scenarios (a portion of real accident scenario data) for iterative optimization of the generative world model, and outputting a small amount of real accident scenario data for iterative optimization of the autonomous driving model using mixed virtual scenarios. The generative world model, as the core data source carrier in a closed loop, receives the seed scenarios output by the problem scenario mining module as basic input, and simultaneously configures its own iterative optimization self-loop link, enabling it to complete self-iterative optimization based on the running results. The optimized generative world model generates massive amounts of virtual data (autonomous driving scenario data) based on the input generation instructions (autonomous driving scenario generation prompts). By fusing this massive amount of virtual data with a small amount of real accident scenario data, a dataset for training the autonomous driving model can be constructed. This mixed dataset is output to the driving model training platform for iterative optimization of the autonomous driving model.

[0131] The driving model training platform completes the training and updating of the autonomous driving model based on the received mixed dataset and outputs two types of results: the first type is the updated autonomous driving model after training, which is distributed to the autonomous driving fleet through OTA update to complete the iterative upgrade of the terminal autonomous driving model; the second type is the new version of the autonomous driving model after training, which is output to the simulation evaluation platform.

[0132] The simulation evaluation platform receives the new version of the autonomous driving model output from the driving model training platform, performs performance evaluation operations, and outputs the corresponding performance feedback data to the performance analysis and bottleneck diagnosis module after the evaluation is completed.

[0133] The performance analysis and bottleneck diagnosis module receives performance feedback data output from the simulation evaluation platform, identifies the weak links in the autonomous driving model based on the performance feedback data, and outputs the optimization targets corresponding to the weak links to the scene generation engine.

[0134] The scene generation engine is a goal-oriented engine. It receives optimization goals from the performance analysis and bottleneck diagnosis module, generates corresponding generation instructions based on the optimization goals, and outputs the generation instructions to the generative world model. This enables the generative world model to complete a new round of massive virtual data generation based on the generation instructions.

[0135] Please refer to Figure 8 , Figure 8 A schematic diagram of the workflow involved in a complete embodiment of the method for generating autonomous driving scene data provided in this application.

[0136] like Figure 8 As shown, the terminal device acquires full real-vehicle road test data and virtual simulation operation data of the autonomous driving system as the basic input for subsequent performance judgment, problem discovery and data collection.

[0137] Next, the terminal device performs the first judgment step based on the acquired full operational data, namely: determining whether a performance bottleneck or fault case has been found, and executing the corresponding branch process according to the judgment result: If the judgment result is negative, the corresponding annotation system has met the standard, and the process will proceed directly to the final mature model deployment stage. If the judgment result is yes, proceed to the subsequent bottleneck and fault diagnosis stage.

[0138] Then, the terminal device performs problem diagnosis and optimization target generation, that is, it executes the following steps in a fixed sequence: For identified performance bottlenecks or fault cases, perform bottleneck or fault diagnosis operations to locate the core link where the problem occurs; Based on the diagnostic results, perform performance analysis and root cause analysis to identify the root cause of the performance bottleneck or failure. Based on the identified root causes of the problems, targeted optimization goals are generated, providing clear guidance for subsequent scenario generation.

[0139] Next, the terminal device first inputs the generated targeted optimization target into the scene generation engine. The scene generation engine then generates corresponding scene generation instructions based on the optimization target, executes the generation instructions to the world model, and sends the generation instructions to the generative world model. The generative world model then receives the scene generation instructions and generates a massive amount of high-fidelity realistic virtual scenes as the core data source.

[0140] In addition, the terminal device can simultaneously acquire real cornercase road-collected data based on the autonomous driving fleet road test data, thereby inputting the massive high-fidelity real virtual scenes output by the generative world model and the real cornercase road-collected data into the hybrid dataset to complete the construction of the dataset for training the autonomous driving model.

[0141] Next, the terminal device trains the autonomous driving model based on the constructed hybrid dataset, resulting in an updated model. The trained model will then be evaluated in a scene library to obtain a complete performance evaluation result.

[0142] Finally, the terminal device performs a second judgment step based on the performance evaluation results of the new model: Does the evaluation result meet the standard? If the judgment result is negative, the corresponding annotation system has not met the standard. The evaluation result will be fed back to the "bottleneck or fault diagnosis" stage, and the entire process of diagnosis, root cause location, optimization target generation, scene and data generation, model training and evaluation will be re-executed to form an iterative optimization small closed loop. If the judgment result is yes, then proceed to the mature model deployment stage; after the mature model is deployed, it is put back into the initial autonomous driving fleet road test + virtual simulation test stage, starting a new round of full-process iterative optimization, and completing the full-link closed loop of autonomous driving model iterative optimization.

[0143] In this embodiment, the iterative optimization closed loop of the autonomous driving model, which uses a generative world model as the core data source, addresses the core pain points of traditional data closed loops based on road surveys and simulations, and achieves comprehensive and disruptive beneficial progress in five core dimensions: 1. Data source dimension: Completely break away from dependence on actual road sampling, and achieve a dual breakthrough in cost and supply capacity.

[0144] Traditional autonomous driving model iteration relies entirely on real road data, which has core drawbacks such as high data acquisition costs and data supply being strictly limited by the scale of road testing. This embodiment adopts a data source architecture that uses generative world models as the main source and real data as a supplement. The core benefits are: it greatly reduces the data acquisition cost of autonomous driving model iteration, while enabling the unlimited generation of training data and the generation of corresponding data according to the needs of model iteration, thus completely breaking the bottleneck of traditional solutions' strong dependence on real road data.

[0145] 2. Long-tail scenario coverage: From passive waiting to proactive creation, systematically solving the long-tail problem of autonomous driving.

[0146] Traditional technologies can only passively wait for long-tail corner case scenarios to emerge during real-world road surveys, and cannot proactively construct target scenarios. This results in extremely low coverage of long-tail scenarios, making it difficult to systematically solve the long-tail safety issues in autonomous driving. This embodiment can proactively create target long-tail scenarios through generative world models, achieving targeted coverage of target scenarios. The core beneficial effect is that it can systematically overcome the long-tail problem in the iteration of autonomous driving models, comprehensively fill the shortcomings in the scenario coverage of models, and fundamentally improve the operational safety of autonomous driving systems.

[0147] 3. Development efficiency dimension: The iteration cycle is shortened by orders of magnitude, greatly improving the response speed of technology iteration.

[0148] Traditional technologies are limited by the data collection cycle of real-world roads and the accumulation cycle of scenarios, resulting in long iteration cycles for autonomous driving models, measured in months or quarters, leading to extremely low development efficiency and an inability to quickly respond to new scenario requirements. This embodiment, relying on the rapid data generation capabilities of generative world models, shortens the model iteration cycle to the level of days or weeks. The core beneficial effect of this extremely short iteration cycle is that it significantly accelerates the maturity and deployment of autonomous driving technology, enabling rapid response to new demands from the market, regulations, and real-world driving scenarios, and comprehensively improving the iteration efficiency and market competitiveness of autonomous driving technology.

[0149] 4. Security Dimension: The entire process is a closed-loop virtual space, enabling zero-risk testing in extreme scenarios.

[0150] Traditional technologies rely on real-world road testing to complete model verification and data collection. However, real-world testing carries significant driving safety risks and is limited by safety constraints, making it impossible to test in extremely dangerous scenarios. In this embodiment, the entire model iteration process can be carried out in a virtual space. The core benefits are: absolute safety throughout the entire model iteration process, while enabling full and unconstrained testing in various extremely dangerous scenarios, completely filling the testing gap in high-risk extreme scenarios that traditional solutions cannot cover.

[0151] 5. Scene Authenticity Dimension: Breaking through existing data boundaries, taking into account both high authenticity and high scene diversity.

[0152] Traditional road survey techniques offer high realism in scenario generation, but the boundaries and types of these scenarios are entirely limited by existing road survey data, making it impossible to generate new scenarios that transcend existing data boundaries, resulting in a severe lack of scenario diversity. The scenarios generated in this embodiment not only maintain high realism but also generate entirely new scenarios that transcend existing data boundaries. Furthermore, the physical rationality of the scenarios is guaranteed by the world model. The core beneficial effect is that, while maintaining high scenario realism, it achieves greater diversity in autonomous driving training scenarios, providing comprehensive and sufficient scenario support for improving the generalization ability of autonomous driving models.

[0153] In summary, this embodiment completely solves the five core pain points of traditional data closed loops based on road sampling and simulation by reconstructing the data source architecture with generative world models as the core. These pain points include strong data dependence, difficulty in covering long tails, low iteration efficiency, high testing risk, and limited scene boundaries. It constructs a low-cost, high-coverage, fast-iterative, high-safety, and highly diverse autonomous driving model iterative optimization closed loop, providing a brand-new underlying architecture for the rapid, safe, and comprehensive iteration of autonomous driving technology.

[0154] Please refer to Figure 9 This application also provides an autonomous driving scene data generation system, which can implement the above-described autonomous driving scene data generation method.

[0155] like Figure 9 As shown, the autonomous driving scenario data generation system provided in this application embodiment may include: The acquisition module is used to acquire performance evaluation data for autonomous driving models. The model prompting module is used to generate prompts for autonomous driving scenarios based on the performance evaluation data to generate a preset generative world model. The scene generation module is used to generate autonomous driving scene data that matches the autonomous driving scene generation prompts based on the generative world model; the autonomous driving scene data is used to perform data-driven model parameter optimization training on the autonomous driving model.

[0156] In some embodiments, the model prompting module is further configured to perform performance bottleneck diagnosis processing on the autonomous driving model based on the performance evaluation data to obtain a structured bottleneck diagnosis report; convert the structured bottleneck diagnosis report into an optimization target for the autonomous driving model; and generate autonomous driving scene generation prompts based on the optimization target using a preset generative world model.

[0157] In some embodiments, the model prompting module is further configured to perform performance aggregation analysis based on the performance evaluation data to obtain the failure characteristics of the autonomous driving model; perform root cause diagnosis processing on the autonomous driving model based on the failure characteristics to obtain the root cause of the failure of the autonomous driving model; and perform formatting processing on the root cause of the failure to obtain a structured bottleneck diagnosis report.

[0158] In some embodiments, the model prompting module is further configured to deconstruct the optimization objective into a scene generation task; define scene parameters based on the scene generation task to obtain an autonomous driving scene parameter combination; perform instruction compilation processing on the autonomous driving scene parameter combination to obtain an autonomous driving scene generation prompt for a preset generative world model; the autonomous driving scene generation prompt conforms to the input protocol of the generative world model.

[0159] In some embodiments, the autonomous driving scenario data generation system provided in this application further includes: The model optimization module is used to acquire real-world driving log data; the real-world driving log data is generated by the vehicle deploying the autonomous driving model; based on the real-world driving log data, the autonomous driving model is subjected to problem scenario mining processing to obtain real accident scenario data; based on the real accident scenario data, the generative world model is iteratively optimized and trained.

[0160] In some embodiments, the model optimization module is further configured to determine the initial priority of the real accident scenario data based on a preset multi-dimensional scenario priority evaluation index; dynamically adjust the initial priority based on a preset priority dynamic adjustment rule to obtain the adjusted priority; the priority dynamic adjustment rule includes at least an adjustment rule based on the computing power of the generative world model; and iteratively optimize and train the generative world model based on the adjusted priority and the real accident scenario data.

[0161] In some embodiments, the model optimization module is further configured to mix the autonomous driving scenario data with the real accident scenario data to obtain a hybrid dataset; and drive the autonomous driving model to perform model parameter optimization training based on the hybrid dataset to obtain an optimized autonomous driving model. It should be noted that the specific implementation of the autonomous driving scene data generation system provided in this application embodiment is basically the same as the specific implementation of the autonomous driving scene data generation method described above, and will not be repeated here.

[0162] Please see Figure 10This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-mentioned method for generating autonomous driving scenario data.

[0163] In some embodiments, the electronic device may be any smart terminal such as an in-vehicle terminal, an in-vehicle hardware platform (e.g., an in-vehicle computer), a tablet computer, a smartphone, or a wearable device; or, the electronic device may be a vehicle including a memory and a processor.

[0164] like Figure 10 As shown, the electronic device provided in this application embodiment may include: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the method for generating autonomous driving scene data according to the embodiments of this application. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0165] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for generating autonomous driving scenario data.

[0166] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0167] This application also provides a computer program product, including a computer program. The steps implemented by the computer program when executed by a processor are basically the same as those in the specific embodiments of the above-described method for generating autonomous driving scenario data, and will not be repeated here.

[0168] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0169] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0170] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0172] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0173] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding factor is divided by the following factor, or that the related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0174] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0175] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating autonomous driving scenario data, characterized in that, The method includes: Obtain performance evaluation data for autonomous driving models; Based on the performance evaluation data, a preset generative world model is generated to provide prompts for autonomous driving scenarios. The generative world model generates autonomous driving scene data that matches the autonomous driving scene generation prompts; the autonomous driving scene data is used to perform data-driven model parameter optimization training on the autonomous driving model.

2. The method according to claim 1, characterized in that, The autonomous driving scenario generation prompts based on the performance evaluation data to generate a preset generative world model include: Based on the performance evaluation data, the autonomous driving model is subjected to performance bottleneck diagnosis processing to obtain a structured bottleneck diagnosis report; The structured bottleneck diagnostic report is transformed into the optimization objective of the autonomous driving model; Based on the optimization objective, a preset generative world model is generated to provide prompts for autonomous driving scenarios.

3. The method according to claim 2, characterized in that, The process of performing performance bottleneck diagnosis on the autonomous driving model based on the performance evaluation data to obtain a structured bottleneck diagnosis report includes: Based on the performance evaluation data, a performance aggregation analysis is performed to obtain the failure characteristics of the autonomous driving model; Based on the failure characteristics, the root cause diagnosis of the autonomous driving model is performed to obtain the root cause of the failure of the autonomous driving model. The root cause of the failure is formatted to obtain a structured bottleneck diagnosis report.

4. The method according to claim 2, characterized in that, The autonomous driving scene generation prompt based on the preset generative world model generated according to the optimization objective includes: The optimization objective is deconstructed into a scene generation task; Based on the scenario, task definition scenario parameters are generated to obtain the autonomous driving scenario parameter combination; The combination of parameters for the autonomous driving scenario is processed by instruction compilation to obtain an autonomous driving scenario generation prompt based on a preset generative world model; the autonomous driving scenario generation prompt conforms to the input protocol of the generative world model.

5. The method according to claim 1, characterized in that, The method further includes: Acquire real-world driving log data; the real-world driving log data is generated by the vehicle deploying the autonomous driving model; Based on the real-world driving log data, the autonomous driving model is subjected to problem scenario mining processing to obtain real accident scenario data; The generative world model is iteratively optimized and trained based on the real accident scenario data.

6. The method according to claim 5, characterized in that, The iterative optimization training of the generative world model based on the real accident scenario data includes: The initial priority of the real accident scenario data is determined based on a preset multi-dimensional scenario priority evaluation index. The initial priority is dynamically adjusted based on a preset priority dynamic adjustment rule to obtain the adjusted priority; the priority dynamic adjustment rule includes at least an adjustment rule based on the computing power of the generative world model; The generative world model is iteratively optimized and trained based on the adjusted priorities and the real-world accident scenario data.

7. The method according to claim 5, characterized in that, The method further includes: The autonomous driving scenario data is mixed with the real accident scenario data to obtain a hybrid dataset; The autonomous driving model is trained to optimize its parameters based on the hybrid dataset, resulting in an optimized autonomous driving model.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the method for generating autonomous driving scene data according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for generating autonomous driving scenario data as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method for generating autonomous driving scenario data as described in any one of claims 1 to 7.

Images