Collision scene generation and closed-loop optimization method for automatic driving based on LLM multi-agent

Abstract

This invention discloses a method for generating and optimizing autonomous driving collision scenarios based on LLM multi-agent systems, belonging to the fields of autonomous driving simulation testing and large language model multi-agent applications. It solves the problems of existing methods, such as single path annotation, low automation, non-standard sensor configuration, and disconnect between scenario generation and algorithm optimization. This method first completes multi-agent collision scenario path annotation for vehicles, non-motorized vehicles / pedestrians, and infrastructure, and constructs a dataset. Then, it builds a five-function LLM multi-agent collaborative framework. After data structuring and task scheduling, it generates standardized CARLA simulation configurations and trajectory codes. Upon execution, it generates collision scenarios and collects multi-sensor and V2X relative pose data. The data is used for training and quantitative evaluation of autonomous driving algorithms. Based on the evaluation results, a three-level closed-loop fine-tuning of simulation code, scenario paths, and parameter configurations is implemented. This invention achieves standardized multi-agent scenario generation and data acquisition, improves scenario generation efficiency and data validity, constructs an end-to-end closed-loop optimization system, significantly improves the training and iteration efficiency of autonomous driving algorithms, has strong code compatibility, and is suitable for autonomous driving algorithm research and testing.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving simulation testing, large language model multi-agent application, and autonomous driving algorithm optimization. In particular, it relates to an autonomous driving collision scene generation and closed-loop optimization method based on LLM multi-agent, which is applicable to the training, evaluation, and iterative optimization of autonomous driving perception, decision-making, and planning algorithms. It can realize the construction of an end-to-end closed-loop scene generation system from collision scene annotation to algorithm optimization. Background Technology

[0002] The development of autonomous driving systems heavily relies on massive, realistic, and diverse collision simulation scenarios for algorithm training and testing. Existing methods for generating autonomous driving collision scenarios mainly suffer from the following problems, and mainstream methods (such as rule-based scenario generation, single LLM agent scenario generation, and traditional CARLA scripting) have not effectively solved them:

[0003] 1. The scene path annotation is too simplistic. Most methods only support path annotation between vehicles and lack multi-subject path annotation for main vehicles and non-motorized vehicles / pedestrians. The scene is not closely related to the actual traffic scene and the generalization ability of the trained algorithm is poor. For example, existing scene generation-related works (such as the DeepAccident scene generation framework and the ChatScene annotation platform) and published patent CN121599036A (Chinese University of Hong Kong (Shenzhen), published on 2025.12.02) only realize collision path annotation between vehicles and do not include non-motorized vehicles and pedestrians. The scene is not realistic enough, and the original ChatScene annotation platform does not support multi-subject path annotation function.

[0004] 2. The automation level of scene generation is low. Traditional methods rely on manual writing of CARLA simulation code, which requires mastery of API usage, programming and scene design knowledge. This is a high threshold, inefficient and prone to problems such as parameter mismatch and failure to trigger collision logic. For example, the scene generation method based on autonomous driving simulation environment described in patent CN121580645A (China Telecom Transportation Technology Co., Ltd., published on 2025.11.28) still relies on manual intervention to complete the core parameter configuration of the simulation scene. Writing a single scene takes several hours and the parameter matching error rate is high.

[0005] 3. Existing LLM-based scene generation methods are mostly based on a single intelligent agent architecture, lacking specialized division of labor. The generated code is prone to problems such as missing environment configuration and unreasonable sensor deployment, and cannot directly adapt to the needs of multi-sensor data acquisition. For example, the published patent CN120930370A (Beijing University of Aeronautics and Astronautics, published on August 14, 2025) uses a single LLM to generate adversarial meta-scenes and scripts without functional division of labor, and often suffers from the problem of missing sensor configuration.

[0006] 4. Scene generation and algorithm optimization are disconnected. Existing methods can only generate simulation scenes and collect data, but have not established a closed-loop system of data collection-algorithm training and evaluation-scene parameter fine-tuning. They cannot optimize simulation scenes in reverse based on algorithm performance, resulting in low algorithm iteration efficiency. For example, the published patent CN121168294A (Jilin University, published on 2025.11.24) generates key scenes based on quantum reinforcement learning, but can only complete scene generation and data collection, without closed-loop optimization logic.

[0007] 5. The sensor configuration in the simulation scenario does not match the actual testing requirements. There is a lack of standardized multi-view RGB camera and LiDAR configuration schemes, and the collected sensor data cannot meet the training requirements of end-to-end autonomous driving algorithms. For example, the multimodal data fusion scenario generation method described in the published patent CN121483045A (Beijing Intelligent Vehicle Connectivity Industry Innovation Center Co., Ltd., published on 2026.01.09) does not specify sensor configuration standards, and the collected data cannot be directly adapted to end-to-end algorithm training.

[0008] Furthermore, the number and types of entities deployed in existing simulation scenarios are fixed, lacking a standardized deployment scheme that includes the main vehicle, conflicting vehicles, two following vehicles, and infrastructure. Moreover, it lacks real-time recording logic for the relative poses between V2X agents, failing to provide effective training data for vehicle-road cooperative algorithms. Therefore, there is an urgent need for a method for generating and optimizing autonomous driving collision scenarios that integrates manual annotation, LLM multi-agent collaboration, standardized simulation deployment, data acquisition, and closed-loop optimization to address the shortcomings of existing technologies. Summary of the Invention

[0009] (a) Technical problems to be solved

[0010] This invention aims to address the technical problems in existing autonomous driving collision scene generation methods, such as single path labeling, low automation, unclear division of labor among multiple agents, non-standard sensor configuration, and disconnect between scene generation and algorithm optimization. It provides an autonomous driving collision scene generation and closed-loop optimization method based on LLM multi-agents, which realizes closed-loop management of multi-agent collision scene path labeling, automated scene generation, standardized multi-sensor data acquisition, algorithm training and evaluation, and scene parameter fine-tuning, thereby improving the training efficiency and generalization ability of autonomous driving algorithms.

[0011] (II) Technical Solution

[0012] To achieve the above objectives, the present invention adopts the following technical solution: a method for generating and optimizing autonomous driving collision scenarios based on LLM multi-agent systems, the core steps of which are as follows:

[0013] 1. Multi-agent collision scenario path annotation: The collision scenario paths of main vehicles, non-motorized vehicles / pedestrians and infrastructure are manually annotated, covering five typical collision types, and a multi-agent path annotation dataset is constructed.

[0014] 2. Construction of LLM Multi-Agent Collaborative Framework: Five agents with clearly defined functions are instantiated through LLM to form a pipeline-style collaborative operation system, responsible for semantic parsing, task scheduling, environment configuration, path generation and code integration.

[0015] 3. Annotated data structure transformation: The semantic enhancement agent parses, completes, and transforms the annotated data into a structure, outputting standardized scenario descriptions to provide a benchmark for task scheduling.

[0016] 4. Task decomposition and scheduling: The task scheduling agent decomposes the overall task into three sub-tasks: environment configuration, path generation, and device deployment. It schedules the corresponding intelligent agents to work in parallel to avoid scenario initialization conflicts.

[0017] 5. Environment and Equipment Deployment Code Generation: The CARLA Environment Agent generates simulation environment parameter configuration and standardized sensor deployment code. The sensors adopt a standard configuration of 6 RGB cameras + 1 32-line LiDAR.

[0018] 6. Multi-subject trajectory code generation: The scene path selection agent generates multi-subject driving trajectories and collision triggering logic code based on the marked path, ensuring that conflicting subjects arrive at the collision point simultaneously.

[0019] 7. Code Integration: The code integration agent unifies, splices, and checks dependencies of the code generated by each agent, generating complete Python code that can be executed directly in the CARLA emulator.

[0020] 8. Simulation Scene Generation and Data Acquisition: Execute integrated code, standardize the deployment of main body combinations, synchronously collect multi-sensor data and V2X agent relative pose data and store them frame by frame.

[0021] 9. Algorithm Training and Performance Evaluation: The collected data is used to train autonomous driving algorithms, and the algorithm performance is evaluated using four core indicators, outputting quantitative evaluation results.

[0022] 10. Closed-loop fine-tuning optimization: Based on the evaluation results, the scene path, simulation parameters or agent logic are fine-tuned in three levels, and the feedback is sent to the corresponding steps to form a closed loop, thereby improving the algorithm iteration efficiency.

[0023] In addition, the LLM multi-agent collaborative framework supports secondary collaboration. During closed-loop fine-tuning, there is no need to rebuild the framework. New scenarios can be quickly generated by only modifying some parameters, reducing the cost of fine-tuning.

[0024] (III) Beneficial Effects

[0025] Compared with the prior art, the present invention has the following significant advantages:

[0026] 1. Achieve multi-subject path annotation with high scene realism: For the first time, non-motorized vehicles / pedestrians are included in the collision scene path annotation scope. A multi-subject annotation dataset of main vehicles, non-motorized vehicles / pedestrians and infrastructure is constructed, covering five typical collision types. The scene is close to the actual traffic scene, and the generalization ability of the trained autonomous driving algorithm is greatly improved.

[0027] 2. LLM multi-agent specialized division of labor, high generation efficiency: Construct five functionally distinct LLM multi-agent collaborative frameworks to realize pipeline-style operations of labeled data parsing, task scheduling, code generation, and code integration, replacing manual code writing, reducing the operation threshold, improving scene generation efficiency, and avoiding the problem of parameter mismatch caused by manual coding;

[0028] 3. Standardized simulation deployment and sensor configuration, with strong data validity: A standardized main deployment scheme including the master vehicle, conflicting vehicles, two following vehicles and infrastructure is formulated, as well as a standardized sensor configuration scheme of 6 RGB cameras + 1 32-line LiDAR. The sensor parameters are in line with the actual test requirements, and the relative pose data between V2X agents are recorded simultaneously, providing high-quality and structured training data for autonomous driving and vehicle-road cooperative algorithms.

[0029] 4. Establishing an end-to-end closed-loop system with high algorithm iteration efficiency: For the first time, collision scene generation, multi-sensor data acquisition, algorithm training and evaluation, and scene parameter fine-tuning are integrated into a closed-loop system. The simulation scene is optimized in reverse based on the algorithm performance evaluation results, realizing the collaborative iteration of scene and algorithm. This solves the problem of the disconnect between scene generation and algorithm optimization in existing methods, and greatly improves the iteration efficiency of autonomous driving algorithms.

[0030] 5. Supports secondary collaboration and low fine-tuning cost: The LLM multi-agent collaborative framework supports secondary collaboration. During closed-loop optimization, there is no need to regenerate the complete code. Only some parameters need to be modified to quickly generate new simulation scenarios, which reduces the time and manpower costs of scenario fine-tuning.

[0031] 6. The code can be executed independently and has strong compatibility: The generated simulation code has undergone strict dependency checks and logic assembly, and can be executed directly in the CARLA simulator without secondary modifications. It is also compatible with CARLA 0.9.12 and above, has strong compatibility, and can be widely used in autonomous driving algorithm research and development and testing scenarios. Attached Figure Description

[0032] Figure 1 is a flowchart of the overall implementation of the autonomous driving collision scene generation and closed-loop optimization method based on LLM multi-agent in this invention. It shows the complete execution steps in sequence: S1 multi-agent path labeling, S2 LLM multi-agent framework construction, S3 labeled data structure transformation, S4 task decomposition and scheduling, S5 environment and equipment code generation, S6 multi-agent trajectory code generation, S7 simulation code integration, S8 simulation scene generation and data acquisition, S9 algorithm training and performance evaluation, and S10 three-level closed-loop fine-tuning optimization. The trigger logic for the first, second, and third levels of fine-tuning is also marked. Figure 2 is a schematic diagram of the LLM multi-agent collaborative framework structure of the present invention. From top to bottom, it is divided into an input layer semantic enhancement agent, a core layer task scheduling agent, an execution layer scene path selection agent, and an output layer CARLA code integration agent. Each agent works together to complete the conversion of natural language scene description to CARLA executable code, supports closed-loop fine-tuning secondary collaboration and adapts to CARLA simulator. Figure 3 is a block diagram of the standardized deployment and multimodal data acquisition of the CARLA simulation scene of the present invention, showing the standardized sensor configuration scheme (6 RGB cameras + 32-line LiDAR), multi-subject combination (main vehicle, conflict vehicle, 2 following vehicles and infrastructure), V2X relative pose acquisition logic, and the storage format and output flow of three types of structured data: RGB image, LiDAR point cloud and pose data. Detailed Implementation

[0033] The present invention will be further described in detail below with reference to specific embodiments. The scope of protection of the present invention is not limited to the following embodiments. All modifications and substitutions based on the technical solutions of the present invention fall within the scope of protection of the present invention.

[0034] This embodiment provides a method for generating and optimizing autonomous driving collision scenarios based on LLM multi-agent systems. It instantiates a five-agent collaborative framework based on a large language model (LLM, such as GPT-4), and uses the CARLA 0.9.12 simulator as the application platform to achieve the generation of multi-agent collision scenarios, standardized data acquisition, and closed-loop optimization of the autonomous driving algorithm. The specific implementation steps are as follows:

[0035] Step S1: Path annotation for multi-object collision scene

[0036] Using a semi-automatic manual annotation method, paths in five typical key collision scenarios of autonomous driving were annotated to construct a multi-agent path annotation dataset:

[0037] S1.1 Labeled objects: Main vehicles, non-motorized vehicles (electric vehicles, bicycles), pedestrians, infrastructure (traffic lights, road signs, charging piles);

[0038] S1.2 Annotation content: The driving / movement path, initial position, speed range, relative pose constraints with other subjects, and collision triggering nodes (such as intersections and straight lane change points) of each subject.

[0039] S1.3 Annotation Tool: Based on the ChatScene annotation platform, this tool is optimized to address the shortcoming of the original version, which did not support non-motorized vehicle / pedestrian path annotation. It adds non-motorized vehicle / pedestrian path annotation functionality, exports annotation results in a structured format, adapts to the path generation logic of the CARLA road network API, and references the DeepAccident scene annotation specification to improve the accuracy and adaptability of the annotation data.

[0040] S1.4 Scene Coverage: Straight-to-straight head-on collision, straight-to-straight lane change scrape, straight-to-straight rear-end collision, straight-to-turning collision, turning-to-turning collision, with no fewer than 50 valid paths marked for each scene.

[0041] Step S2: Construction of the LLM Multi-Agent Collaborative Framework

[0042] Five functionally defined agents are instantiated using GPT-4 to build a standardized collaborative framework. These agents interact with each other via the HTTP local call protocol. The framework employs a distributed collaborative architecture to ensure independent operation and data synchronization for each agent.

[0043] S2.1 description_enhancer: Semantic enhancement agent, responsible for semantic parsing, complexity evaluation and structure transformation of labeled data;

[0044] S2.2 task_scheduler: Task scheduling agent, responsible for task decomposition, agent scheduling and cooperation logic control;

[0045] S2.3 carla_environment: CARLA environment agent, responsible for simulation environment configuration, infrastructure deployment, and sensor configuration code generation;

[0046] S2.4 route_selector_agent: Scene path selection agent, responsible for multi-subject trajectory code generation and collision logic embedding;

[0047] S2.5 carla_code_merger: CARLA code integration agent, responsible for integrating, checking, and generating complete scripts from the code generated by each agent.

[0048] Step S3: Transformation of labeled data into a structured form

[0049] The labeled dataset from step S1 is input into description_enhancer, and the agent performs the following operations:

[0050] S3.1 Semantic parsing: Identify entity types (vehicles, non-motorized vehicles, pedestrians, infrastructure), environmental attributes (preset weather, lighting, map), and interaction intentions (collision type, relative pose constraints) in the labeled data.

[0051] S3.2 Complexity Assessment: Based on the number of subjects, path complexity, and collision triggering conditions, the scene is divided into three levels: simple, medium, and complex.

[0052] S3.3 Complete Ambiguous Information: Complete missing sensor parameters, simulation duration, V2X communication constraints, etc. in the annotation according to the default standard;

[0053] S3.4 Structured Transformation: The output includes a structured scene description containing scene ID, simulator version, map name, subject list, path parameters, collision constraints, sensor parameter requirements, and V2X communication constraints, providing a clear benchmark for subsequent task scheduling.

[0054] Step S4: Task Decomposition and Scheduling

[0055] After receiving the structured scenario description, task_scheduler performs the following operations:

[0056] S4.1 Task Decomposition: The overall task is broken down into three sub-tasks: environment configuration, path generation, and device deployment. The device deployment sub-task includes infrastructure deployment and sensor configuration.

[0057] S4.2 Task Scheduling: Schedule the environment configuration + device deployment subtask to carla_environment, and the path generation subtask to route_selector_agent;

[0058] S4.3 Collaborative Control: Adopting the strategy of "environment configuration first, path generation in parallel", the scheduling instruction is first sent to carla_environment, and then sent to route_selector_agent after a 1-second delay. This ensures that the environment parameters and infrastructure deployment take effect before the path code, avoiding scenario initialization conflicts.

[0059] Step S5: Environment and Equipment Deployment Code Generation

[0060] Based on scheduling instructions, carla_environment generates Python code containing three core modules, adapted for the CARLA 0.9.12 emulator.

[0061] S5.1 Map Loading and Environmental Parameters Module: Specify the CARLA map name (e.g., Town01 city map, Town04 highway map), and set physical parameters such as weather (sunny / rainy / snowy), illumination (noon / sunset / night), and gravitational acceleration 9.8m / s².

[0062] S5.2 Infrastructure Deployment Module: Defines the type, location, and relative position of infrastructure to vehicles, such as deploying traffic lights at an intersection with a horizontal distance of 50 meters from the main vehicle;

[0063] S5.3 Sensor Configuration Module: Configures sensors for each vehicle and infrastructure, including 6 RGB cameras with a resolution of 1600×900, a front-view camera FOV of 90°, a side-view camera FOV of 120°, and a rear-view camera FOV of 100°, and a 32-line lidar with a detection range of 70 meters, a sampling frequency of 10Hz, and a vertical field of view of ±15°. The sensors are all mounted on the top / front of the vehicle and in preset positions on the infrastructure.

[0064] Step S6: Multi-subject trajectory code generation

[0065] The `route_selector_agent` function generates multi-agent driving / motion trajectories based on scheduling instructions and path information from the labeled dataset. (Python code follows)

[0066] S6.1 Path Extraction: Based on the CARLA road network API, the lane waypoint sequence is automatically extracted to ensure that the main vehicle and conflicting vehicles travel along the lane, and non-motorized vehicles / pedestrians move in the non-motorized vehicle lane / sidewalk according to the marked path;

[0067] S6.2 Collision Logic Generation: Calculate the intersection of the trajectories of the main vehicle and the conflicting entities as the collision point, and fine-tune the initial positions (e.g., the distance between the main vehicle and the conflicting vehicle is 60 meters) and speeds (e.g., the main vehicle is 30 km / h and the conflicting vehicle is 25 km / h) of each entity to ensure that both parties arrive at the collision point simultaneously.

[0068] S6.3 Trajectory Code Generation: Generates time-series trajectory code containing the position, velocity, and acceleration of each subject, embeds real-time calculation logic for relative pose between V2X agents, updates the distance, angle, and velocity difference data between each subject every 0.1 seconds, and triggers collision events at the collision point;

[0069] S6.4 Follow vehicle configuration: Generate trajectory codes for two following vehicles. The following vehicles maintain a safe distance of 7 meters from the vehicle in front and travel along the same path to simulate actual traffic flow.

[0070] Step S7: Simulation code integration

[0071] carla_code_merger receives the code snippets generated by carla_environment and route_selector_agent, and performs the following operations to complete the code integration:

[0072] S7.1 Variable standardization: Standardize variable names and function names such as "map_name", "car_speed", and "sensor_path" in the code to avoid naming conflicts;

[0073] S7.2 Logic Assembly: Concatenate code snippets in the order of "Environment Initialization → Infrastructure Deployment → Main Body Generation → Sensor Mounting → Behavior Driving → Data Acquisition → V2X Communication" to ensure smooth execution of the logic;

[0074] S7.3 Dependency Check: Automatically imports Python libraries such as CARLA, numpy, pandas, and PIL, as well as the CARLA 0.9.12 API, and adds code defining the data save path (sensor_data folder in the script's directory) and format (PNG, NPY, CSV).

[0075] S7.4 Script Generation: Generates complete Python code that can be executed directly in the CARLA emulator and saves it as the generated_scenario.py file.

[0076] Step S8: Simulation Scene Generation and Data Acquisition

[0077] Start the CARLA 0.9.12 emulator by executing the command ". / CarlaUE4.sh" in the terminal to start the server;

[0078] Execute the command "python generated_scenario.py" in another terminal to upload the integrated code to the emulator;

[0079] The simulator automatically executes code to complete environment initialization, infrastructure deployment, and the generation of the main body, including the main vehicle, the conflicting vehicle, the two following vehicles, and the infrastructure, as well as the sensor mounting, to build a dynamic and highly accurate collision simulation scenario.

[0080] During the simulation, multi-sensor data and relative pose data are recorded simultaneously:

[0081] RGB camera data: Visual images are stored in PNG format, with a frame rate of 10Hz, and the file is named "{body name}camera_frame{frame number: 06d}.png";

[0082] LiDAR data: Point cloud data is stored in NPY format, with a frame rate of 10Hz, and the file is named "{body name}lidar_frame{frame number: 06d}.npy";

[0083] Relative pose data: Stores real-time distance, angle, and velocity difference data between each V2X agent in CSV format, including fields such as frame number, timestamp, subject name, relative X / Y / Z coordinates, relative angle, and velocity difference;

[0084] All data is stored synchronously frame by frame, with each frame number corresponding to a timestamp. The data is stored in the sensor_data folder in the directory where the script is located. Each run creates an independent timestamp session directory to avoid data overwriting.

[0085] Step S9: Algorithm Training and Performance Evaluation

[0086] The sensor data and relative pose data collected in step S8 are input into the autonomous driving algorithm training and evaluation module. The module includes three sub-modules: data preprocessing, model training, and performance evaluation. The specific operations are as follows:

[0087] S9.1 Data Preprocessing: Denoise, crop, and normalize RGB image data; denoise, downsample, and transform coordinates LiDAR point cloud data; remove outliers from relative pose data; and manually label sensor data to generate training labels.

[0088] S9.2 Model Training: Divide the preprocessed dataset into training and test sets in an 8:2 ratio, and input them into the end-to-end autonomous driving algorithm model (such as the Transformer+CNN fusion model) to jointly train the perception, decision-making, and planning algorithms. Set the number of training rounds to 100 and the batch size to 32.

[0089] S9.3 Performance Evaluation: The collision success rate, perception accuracy, decision rationality, and path tracking error are used as core evaluation indicators to comprehensively evaluate the trained algorithm. The collision success rate is the number of actual collision-triggered scenarios divided by the total number of scenarios. Perception accuracy is the accuracy of target detection and recognition. Decision rationality is the degree of matching between the algorithm's decision and the human expert's decision. Path tracking error is the average distance between the algorithm's planned path and the actual driving path. The output includes quantitative evaluation results and an analysis of the algorithm's performance defects.

[0090] Step S10: Closed-loop fine-tuning and optimization

[0091] Based on the evaluation results of step S9, a three-level fine-tuning strategy is adopted to feed back the fine-tuned information to the corresponding links, forming a closed-loop optimization. The specific strategy is as follows:

[0092] First-level fine-tuning (simulation code layer): If the evaluation results show that the collision scene is not triggered, the sensor data is invalid, or the subject's motion is abnormal, it indicates that the simulation code parameters are mismatched. Fine-tune the CARLA simulation scene configuration parameters (such as the subject's initial position, velocity, and sensor installation position) and the multi-agent code generation logic, and feed the fine-tuned parameters back to step S3 to re-perform the structure conversion and code generation.

[0093] Secondary fine-tuning (scene path layer): If the evaluation results show that the algorithm performance does not reach the preset threshold and the defect is due to insufficient scene path coverage or large differences from the actual scene, fine-tune the collision scene path labeling rules (such as adding path labels for complex traffic scenes and optimizing the movement paths of non-motorized vehicles / pedestrians), supplement the labeled dataset, feed the new labeled dataset back to step S1, and re-label the path.

[0094] Level 3 fine-tuning (parameter configuration layer): If the evaluation results show local performance defects in the algorithm (such as low perception accuracy for a certain collision type), only the path parameters or sensor configuration parameters of the corresponding scene are fine-tuned (such as adjusting the detection range of the LiDAR and optimizing the camera view). Through secondary collaboration of LLM multi-agents, only route_selector_agent and carla_code_merger are called to modify the relevant code, without rebuilding the framework, and new simulation code is quickly generated.

[0095] After fine-tuning, repeat steps S3-S9 to regenerate the simulation scene, collect data, and train and evaluate the algorithm until the algorithm performance reaches the preset threshold, thus realizing an end-to-end closed loop of autonomous driving collision scene generation and algorithm optimization.

[0096] Industrial applicability

[0097] The present invention provides an autonomous driving collision scene generation and closed-loop optimization method based on LLM multi-agent, which realizes the integrated operation of multi-agent collision scene path labeling, automated scene generation, standardized data acquisition, algorithm training and evaluation, and closed-loop fine-tuning, significantly improving the generation efficiency and realism of autonomous driving simulation scenes, as well as the training and iteration efficiency of autonomous driving algorithms.

[0098] The method of this invention can be directly deployed on autonomous driving algorithm R&D and testing platforms. The generated simulation code can be directly executed in the CARLA simulator. The collected sensor data can be directly used for training autonomous driving perception, decision-making, planning and vehicle-road cooperative algorithms. The closed-loop optimization strategy can realize the collaborative iteration of scenarios and algorithms. It is suitable for autonomous driving R&D work of vehicle manufacturers, autonomous driving algorithm companies, universities and research institutions, and has significant industrial applicability and broad application prospects.

Claims

1. A method for generating and optimizing collision scenarios for autonomous driving based on LLM multi-agent systems, characterized in that, Includes the following steps: Step S1: Manually label the key collision scenario paths of autonomous driving and construct a multi-subject path labeling dataset. The labeled objects include the main vehicle, non-motorized vehicles / pedestrians and infrastructure. The labeled content includes the driving / movement path, relative pose constraints and collision triggering nodes of each subject. Step S2: Construct an LLM multi-agent collaborative framework. Instantiate five agents with defined functions through LLM: description_enhancer (semantic enhancement agent), task_scheduler (task scheduling agent), carla_environment (CARLA environment agent), route_selector_agent (scene path selection agent), and carla_code_merger (CARLA code integration agent). Each agent works collaboratively through a preset protocol. Step S3: Input the labeled dataset from step S1 into description_enhancer. This agent performs semantic parsing, scene complexity evaluation, and structure transformation on the labeled information, and outputs a structured scene description containing scene ID, subject list, path parameters, collision constraints, and simulator adaptation requirements. Step S4: The task_scheduler receives the structured scenario description, performs task decomposition and scheduling, breaks down the task into three sub-tasks: environment configuration, path generation, and device deployment, and schedules the environment configuration sub-task to carla_environment and the path generation sub-task to route_selector_agent. Step S5: carla_environment configures the environmental parameters of the CARLA simulator, such as weather, lighting, and map, according to the scheduling instructions. At the same time, it generates infrastructure deployment and multi-subject sensor configuration code, which includes multiple RGB cameras and LiDARs with different perspectives. Step S6: Based on the scheduling instructions and the path information of the labeled dataset, route_selector_agent selects the collision path and generates the driving / movement trajectory code of the main vehicle and non-motorized vehicles / pedestrians. The code includes the initial position, driving / movement direction, speed, relative pose and collision triggering logic of each subject. Step S7: carla_code_merger receives the code snippets generated by carla_environment and route_selector_agent, integrates the code, and generates complete code that can be executed directly in the CARLA simulator; Step S8: Upload the integrated code to the CARLA simulator for execution, deploy multiple subject combinations in the simulation scene, automatically build a dynamic and high-precision collision simulation scene, and simultaneously record multi-view sensor data and relative pose data between V2X agents. Step S9: Input the sensor data and relative pose data collected in step S8 into the autonomous driving algorithm training and evaluation module to complete the training and performance evaluation of the algorithm and output the evaluation results. Step S10: Based on the evaluation results of step S9, fine-tune the collision scene path labeling rules, LLM multi-agent collaborative logic, or CARLA simulation scene configuration parameters, and feed the fine-tuned information back to step S1 or step S3 to form a closed-loop system for autonomous driving collision scene generation and algorithm optimization.

2. The method according to claim 1, characterized in that: In step S1, the key collision scenarios include five types: straight-to-straight head-on collision, straight-to-straight lane change collision, straight-to-straight rear-end collision, straight-to-turning collision, and turning-to-turning collision. The path annotation adopts a manual semi-automatic annotation method, and the annotation results can be adapted to the CARLA simulator's road network API and path generation logic.

3. The method according to claim 1, characterized in that: In step S3, the semantic parsing process of description_enhancer includes identifying entity types (vehicles, non-motorized vehicles, pedestrians, infrastructure), environmental attributes, and interaction intentions (collisions, relative pose constraints) in the labeled data, and completing the fuzzy labeled information. The structured scene description also includes sensor parameter requirements, V2X communication constraints, and simulation duration parameters.

4. The method according to claim 1, characterized in that: In step S4, the task decomposition and scheduling logic of task_scheduler is as follows: if the scene requirement includes environment configuration (weather, lighting, map), then an environment configuration subtask is generated and scheduled to carla_environment; if the scene requirement includes main path or collision logic, then a path generation subtask is generated and scheduled to route_selector_agent; if the scene requirement includes sensor or infrastructure deployment, then the task is integrated into the environment configuration subtask and executed by carla_environment.

5. The method according to claim 1, characterized in that: In step S5, the environment configuration and equipment deployment code generated by carla_environment includes three main modules: map loading and environment parameter module: specifying the CARLA map name and setting physical parameters such as weather, lighting, and gravitational acceleration; infrastructure deployment module: defining the location, type, and relative pose relationship of infrastructure with vehicles; and sensor configuration module: defining the 1600×900 resolution and FOV parameters of 6 RGB cameras, and the 70-meter detection range, 10Hz sampling frequency, and installation location parameters of the 32-line LiDAR.

6. The method according to claim 1, characterized in that: In step S6, the trajectory code generated by route_selector_agent is based on the CARLA road network API and automatically extracts the lane waypoint sequence to ensure that the main vehicle travels along the lane and non-motorized vehicles / pedestrians move along the marked path.

7. The method according to claim 1, characterized in that: In step S8, the multi-entity combination includes a main vehicle, conflicting vehicles, two following vehicles, and infrastructure. The following vehicles maintain a preset safe distance from the vehicle in front and travel along the same path. The sensor data includes visual image data from RGB cameras and point cloud data from 32-line LiDAR. The relative pose data includes real-time distance, angle, and speed difference data between each V2X agent. All data is stored synchronously frame by frame, with each frame number corresponding to a timestamp.

8. The method according to claim 1, characterized in that: In step S9, the autonomous driving algorithm training and evaluation module includes three sub-modules: data preprocessing, model training, and performance evaluation. Data preprocessing performs noise reduction, annotation, and format conversion on sensor data. Model training supports end-to-end training of autonomous driving perception, decision-making, and planning algorithms. Performance evaluation uses collision success rate, perception accuracy, decision rationality, and path tracking error as core evaluation indicators.

9. The method according to claim 1, characterized in that: In step S10, the fine-tuning strategy is executed in stages according to the evaluation results: if the evaluation results show that the collision scene was not triggered or the sensor data is invalid, the CARLA simulation scene configuration parameters and multi-agent code generation logic are fine-tuned and fed back to step S3; if the evaluation results show that the algorithm performance did not reach the preset threshold and the reason is insufficient scene path coverage, the collision scene path annotation rules are fine-tuned, the annotation dataset is supplemented, and fed back to step S1; if the evaluation results show that the algorithm has local performance defects, the path parameters or sensor configuration parameters of the corresponding scene are fine-tuned and fed back to step S3.

10. The method according to any one of claims 1-9, characterized in that: The LLM multi-agent collaborative framework supports secondary collaboration. During the closed-loop optimization process, only route_selector_agent and carla_code_merger need to be called to modify the subject trajectory or environmental parameters, without having to rebuild the entire collaborative framework and generate complete code.

Images