Automatic driving simulation test scene generation method based on text traffic accident report

Abstract

The invention relates to an automatic driving simulation test scene generation method based on a text traffic accident report, and aims to apply a generative language model technology in the field of natural language processing, a code generation technology in the field of program synthesis and a fuzzy test theory in the field of software test. The problem that diversified and standardized high-quality simulation test scenes are lacked in the current automatic driving test field is solved, and the method can generate diversified scene files in the OpenSCENARIO format on the basis of natural languages, and mutates seed scenes through multiple transformation operators so as to efficiently discover defects existing in an automatic driving system.

Description

Technical Field

[0001] The present invention belongs to the field of software testing, and in particular to a simulation test method for an autonomous driving system. Generating autonomous driving simulation test scenarios based on text traffic accident reports is a new attempt. Different from previous methods for generating autonomous driving simulation test scenarios, this method uses generative language model technology in the field of natural language processing, code generation technology in the field of program synthesis, and fuzzy testing theory in the field of software testing to solve the problem of lack of diversified, standardized, high-quality simulation test scenarios in the current autonomous driving testing field. Background Art

[0002] Automated Driving Systems (ADS) aim to improve driving experience, ensure traffic safety and alleviate road congestion through automation and intelligent technologies. With the rapid development of artificial intelligence, sensor technology and computing power, ADS has become a core area of ​​extensive research and exploration in academia and industry, and is expected to improve traffic efficiency and reduce traffic accidents. However, the complexity of the external environment, such as weather conditions, road conditions and other dynamic traffic participants, makes ADS face many challenges in actual deployment. Potential threats in these complex environments may cause system failures, trigger unpredictable consequences, and even lead to serious traffic accidents. Studies have shown that defects in ADS are one of the main causes of accidents, reflecting that current technology still has obvious limitations in dealing with complex environments, and the reliability and safety of the system need to be further improved.

[0003] Current testing methods for autonomous driving systems mainly include road testing and simulation testing. Road testing is time-consuming and labor-intensive, and is usually limited to verifying specific scenarios within a closed area. Even if real scenarios that may trigger defects can be identified, it is still difficult to collect comprehensive data due to the diversity and unpredictability of the environment. Therefore, testing methods that rely on simulation platforms have become mainstream. The main goal of simulation testing is to generate critical scenarios that may cause accidents, thereby ensuring rigorous testing of ADS. By generating scenarios that are difficult to test on a large scale in a real environment, the simulation method effectively improves testing efficiency.

[0004] However, despite the many advantages of simulation testing, there are still several unresolved challenges and limitations. One important challenge is the insufficient diversity of test scenarios, as it is often difficult to represent the interactions between traffic participants. Existing methods rely on visual media such as images and videos to generate scenarios, but this approach requires a lot of storage space, computing resources, and manual annotation. Therefore, the diversity of generated scenarios is limited by the existing data in the dataset, making it difficult to effectively simulate various real-world complex situations. Another challenge is the lack of standardization of test scenarios. Although languages ​​such as OpenSCENARIO provide scenario specifications to simplify scenario generation, existing methods rarely utilize OpenSCENARIO, which makes the generated test scenarios difficult to migrate and extend between different simulation platforms. In addition, the lack of standardization also makes the automatic semantic analysis of scenarios more complicated, thereby reducing the interpretability of test results and hindering the effectiveness of defect detection and system performance optimization.

[0005] Based on the above background and analysis of related work, this patent proposes a method for generating autonomous driving simulation test scenarios based on text traffic accident reports to solve the problem that high-quality and diverse test scenarios are difficult to generate and standardized test scenarios are difficult to construct in current autonomous driving simulation tests. Text reports provide rich causal relationships, dynamic interactions, and detailed contextual information without the need for a large amount of manual annotation and computing resources, making them more efficient than visual data in large-scale scene generation. The scenarios generated by this technology are described in the OpenSCENARIO format, an ADS test standard supported by multiple simulation platforms. This technology uses carefully designed prompt words to call a large language model (LLM) to parse the accident report, thereby converting the text The report is converted into a seed scenario file. Then, the technology uses scenario decomposition, mutation, and assembly methods to generate a scenario tree starting from a single seed, in which nodes are interconnected through derivative relationships. The mutation operations used include dynamic mutation, trajectory mutation, physical property mutation, and environment mutation, which can effectively enhance the diversity of the generated scenarios. The derivative relationship based on the assembly method enables each child node to contain more scenario modules than its parent node, representing a more specific or complex scenario. Developers can identify the triggering conditions that cause unexpected behavior of ADS by comparing the system performance in the parent node and child node scenarios. This analysis not only helps developers understand the behavioral logic of ADS, but also accelerates the improvement of system performance. Summary of the invention

[0006] The problem to be solved by the present invention is that nowadays, deep learning frameworks have become the cornerstone of innovative applications in many fields. With the rapid development of these frameworks in complexity and functionality, it is particularly important to ensure their reliability and performance. Building, training, verifying and using deep learning models are the most commonly used application scenarios of deep learning frameworks. In traditional software testing, the generation of test cases is relatively direct and the construction of test predictions is more clear, which can usually be completed based on requirement specifications or design documents. However, for deep learning frameworks, it is very difficult to generate specific test cases directly from the design and requirements of highly abstract deep learning models. Secondly, due to the high complexity and context dependency of the output results of deep learning models , it is also very challenging to clearly define an expected output for each test case. In addition, deep learning models are data-driven, and the existing test input data cannot guarantee the breadth of deep learning framework tests, and lacks effective input data to trigger possible defects. In order to ensure that simulation tests can simulate real conditions more realistically and effectively help identify potential defects of ADS, it is particularly important to build diverse and realistic test scenarios. However, the existing scenario file generation methods have the following two problems: a) Insufficient test scenario diversity: When generating scenarios, in addition to simple static elements, there are also dynamic elements such as vehicles and pedestrians. Can their respective behaviors and interactions with each other be accurately restored? The quality of the test scene is very important. In order to solve the above problems, existing methods rely heavily on visual media resources. For example, test scenes are generated based on real traffic accident images or videos. This method uses datasets such as KITTI, Udacity and GTSRB. However, this method has two defects. First, this method has high requirements for resources such as computing and storage. The storage of visual media information such as pictures and videos not only takes up a lot of storage space, but their high complexity also requires a lot of computing resources for semantic parsing. This performance bottleneck will become more severe as the video becomes larger and the resolution increases. Second, this type of data is often difficult to obtain, and it takes a lot of time and manpower to conduct Manual annotation, which greatly inhibits the diversity of scenario files; b) Lack of standardization of test cases: In order to accurately describe scenarios and simplify the scenario creation process, the development of related descriptive programming languages ​​has been emphasized. However, these programming languages, even the more classic OpenSCENARIO and Scenic, have not been widely used in practice. Therefore, the scenario files generated now are often deeply coupled with the simulator and the operating platform, and have poor mobility and portability. In addition, due to different standards, it is difficult for developers to use automated methods to parse scenario semantics, and it is difficult to extract valuable information and specific causal chains from test results, which hinders ADS defect identification and performance optimization.Therefore, the present invention mainly solves the problem that it is difficult to generate high-quality and diversified test scenarios and difficult to construct standardized test scenarios in current autonomous driving simulation tests.

[0007] The technical solution of the present invention is: a method for generating an autonomous driving simulation test scenario based on a text traffic accident report, which is characterized in that a large language model is called by using carefully designed prompt words to parse the accident report, thereby converting the text report into a seed scenario file; then, the technology uses scene decomposition, mutation and assembly methods to generate a scene tree starting from a single seed, in which nodes are interconnected through derivative relationships, and the mutation operations used include dynamic mutation, trajectory mutation, physical property mutation and environmental mutation, which can effectively enhance the diversity of the generated scenes, and these scene files are all described using the international standard OpenSCENARIO, which can easily achieve cross-platform simulation. The input of this method is a text traffic accident report that describes the complete traffic accident process, and the output is a scene file tree in the OpenSCENARIO format, in which the derivative relationships can be used to test prophecy construction.

[0008] The method includes the following four modules and steps:

[0009] (1) Test scene element collection: The present invention first conducts an in-depth analysis of the semantic structure and grammatical rules of OpenSCENARIO and extracts necessary information from official documents. Since OpenSCENARIO adopts an XML-based modeling method and has a clear hierarchical structural relationship, the present invention constructs a graph database to store the attributes of scene elements and their connection relationships, providing necessary support for subsequent steps to ensure the grammatical correctness of the generated scene.

[0010] (2) Test scenario conversion: The present invention utilizes the powerful natural language analysis and generation capabilities of the large language model to parse the complex natural language descriptions contained in the text. In combination with the semantic characteristics of the text describing the test scenario, the present invention designs targeted prompt words to extract the static elements, dynamic elements contained in the scenario and their interactive behaviors with the tested objects from the text description. By using the semantic and grammatical rules stored in the graph database, the present invention converts the parsing results into a test scenario file that conforms to the OpenSCENARIO standard to ensure the structuring and standardization of the generated scenario.

[0011] (3) Test scenario variation: In order to cover a wider range of test scenarios and assist developers in analyzing the behavioral causal chain of the autonomous driving system, the present invention designs a scenario decomposition and variation method. Through the scenario decomposition technology, the complete scenario description is decomposed into scenario blocks containing complete semantics (corresponding to the XML elements in OpenSCENARIO). Subsequently, the present invention designs four types of mutation operators for different scenario elements (such as environmental elements, dynamic participants, etc.) to generate diversified test scenarios.

[0012] (4) Test scenario assembly: Based on scenario variation, the present invention inserts the newly generated scenario blocks into the scenario template in the correct semantic order to generate a semantically rich test scenario file tree with a parent-child derivation relationship. By comparing the execution results of the autonomous driving system in the parent-child chain scenario, or analyzing the execution results of the entire scenario, the triggering conditions of abnormal behavior are identified, and the behavior causal chain is further explored to provide developers with logical improvement suggestions on design, code, etc.

[0013] The present invention is characterized in that:

[0014] 1. Extract key information from text traffic accident reports based on a large language model.

[0015] 2. Able to translate OpenSCENARIO scene files based on textual accident reports.

[0016] 3. Based on the seed scene file, it can realize mutation and assembly, and complete the generation of the scene file tree with a derivative relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The overall flow chart for the implementation of the present invention.

[0018] Figure 2 It is the prompt word used to input the large model in the key step 2 of this method.

[0019] Figure 3 These are the seven relative position relationships involved in the prompt words in the key step 2 of this method. DETAILED DESCRIPTION

[0020] The present invention uses the generative language model technology in the field of natural language processing, the code generation technology in the field of program synthesis, and the fuzzy testing theory in the field of software testing to solve the problem of the lack of diversified, standardized, high-quality simulation test scenarios in the current field of autonomous driving testing.

[0021] 1. Construction of scene element database

[0022] The construction of the scene element database lays the foundation for subsequent scene transformation, variation and assembly, and provides guidance for these steps in terms of semantic and syntactic correctness. The present invention uses the web crawler tool Scrapy to obtain the official OpenSCENARIO document described in the Unified Modeling Language (UML). The document defines each element in the scene as a class, enumeration, interface or basic type, and uses UML to describe inheritance relationships, combination relationships and their attributes. The specific information of each attribute includes name, type, cardinality and description. In UML, cardinality is usually used to represent the quantitative relationship between two object types, which is extended here to the quantitative relationship between two scene elements. The description usually includes a reasonable value range of the scene attribute. Based on the acquired information, the present invention constructs a scene element database. There are two types of nodes in this database: XSDElement and XSDAttribute. XSD in OpenSCENARIO DElement is used to define the main data structure of scene description, such as weather, vehicle objects and actions. These elements constitute the core content of XML documents, describing the components and hierarchy of the scene. XSDAttribute in OpenSCENARIO provides additional information for elements, usually basic and specific attributes that cannot be further subdivided. There are three types of edges in the database: one represents the inclusion relationship between two elements, one represents the inheritance relationship between two elements, and one represents the "attributed" relationship where the elements have specific attributes. The database contains the necessary information required to generate scene files, such as the relationship between scene elements, the data type of each attribute and its value range, providing syntax support for subsequent steps.

[0023] 2. Parsing of textual car accident reports based on large language models

[0024] Large Language Model (LLM) is a natural language processing (Natural Language The present invention uses a large language model (i.e., GPT-4o) to analyze text accident reports to automatically process unstructured data and extract the necessary information required to generate OpenSCENARIO files. Compared with traditional rule-based methods, the large language model has excellent language understanding capabilities, enabling it to capture implicit patterns and contextual meanings in text. The present invention focuses on traffic participants and their behaviors that interact with autonomous vehicles in accident reports. The present invention selects text descriptions in accident reports involving autonomous vehicles collected by the California Department of Motor Vehicles for analysis. The text accident details in the report are filled in manually using natural language, providing a complete narrative of the actions of the vehicles involved before the accident, the time of the incident, and its consequences. The present invention designs a multi-round dialogue prompt (promot) to make full use of the advantages of the large language model in understanding the context of long texts and gradually extract useful information from the text report. Multi-round dialogues help to continuously refine and clarify issues, ensuring that the large language model accurately captures key details in the report, thereby improving the effectiveness and relevance of generated test scenarios.

[0025] The present invention defines traffic participants (such as vehicles and pedestrians) that interact with the autonomous vehicle in an accident as non-player characters (NPCs), and defines stationary vehicles or objects on the road as obstacles. Then, the number, type, location or behavior of these two types of objects are analyzed. To ensure the authenticity of the subsequent conversion scene, such as Figure 2 As shown, the design of the present invention prompts to convert the behavior of the NPC before the accident into an ordered sequence of events. The sequence consists of fine-grained events (such as left turn, right turn, acceleration, deceleration, etc.). Such a design helps to standardize the output of the large language model, improve the accuracy of the parsing results, and ensure that the generated scenes are more detailed and coherent, avoiding the omission of key driving operations or emergencies. The present invention supports the generation of sequences from 13 fine-grained events through permutations and combinations, comprehensively covering all behaviors of the NPC involved in existing accident reports. When determining the position of the NPC, the present invention converts its absolute position on the road into a relative position relative to the autonomous driving vehicle. As shown in Figure 3As shown, the present invention defines seven types of relative positions, which are only used to establish the initial positions of the NPC and the autonomous vehicle in the scene. The subsequent changes in the NPC positions are determined by the event sequence. If their relative positions belong to one of R5, R6 and R7, the large language model needs to further determine whether the NPC and the autonomous vehicle are in the same lane. This design aims to separate the NPC's behavior from specific road structures and adapt the behavior to different road scenarios (such as intersections, T-junctions). The prompt design logic for determining the position of static obstacles is similar to that of the NPC.

[0026] After obtaining the parsing results, the present invention generates a seed scene file based on the information in the entity database. First, the present invention establishes a scene framework to fill in detailed scene information. The framework predefines the basic structure of the scene file, including the basic elements and their labels required for scene execution (for example, map paths, entity definitions, global behaviors, initialization behaviors, and entity behaviors), but does not include specific attribute values ​​or definitions. Next, the present invention fills in the initialization attributes (such as shape, initial speed, and position) of the corresponding entity according to the type and position information of the NPC and obstacles in the parsing results. For autonomous driving vehicles, the present invention only specifies its initial position. Finally, the present invention completes the storyboard definition in the scene file according to the event sequence. The present invention defines each Event element in sequence within the Maneuver element to ensure that the behavior of each entity is consistent with the parsed scene logic.

[0027] 3. Scene Block Variation

[0028] In order to improve the diversity of generated scenes, the present invention designs a scene disassembly and scene block mutation method. The scene disassembly method aims to identify scene blocks that can be mutated and reorganized in the seed scene file. The scene template guides the subsequent scene block assembly and clarifies the insertion order and position. The identified scene block types include weather, traffic signals, obstacles, NPC definitions and NPC-related events. The first three are classified as environmental factors, and NPC definitions and events are the core elements of entities and storyboards respectively. The present invention performs a more detailed disassembly of the event sequence in the storyboard, and regards a single event as an independent scene block to generate more variable blocks. In implementation, based on the characteristics of OpenSCENARIO described by XML, different types of scene blocks are identified by element-specific tag names and name attribute values. In view of the diverse semantic features contained in different types of scene blocks, the present invention designs multiple types of mutation operators, as follows:

[0029] (1) Dynamic variation: This type of operator changes the dynamic characteristics of dynamic objects and simulates driving behavior and motion patterns. Specific operators include:

[0030] ●Target Speed ​​Mutation (TSM): TSM changes the absolute target speed set for a dynamic object, corresponding to the AbsoluteTargetSpeed ​​element. The absolute target speed is a fixed value that a dynamic object should reach under specific conditions.

[0031] ●Dynamic Transition Mutation (DTM): DTM specifies the dynamic characteristics of value transitions and defines how values ​​change over time or distance. It corresponds to the TransitionDynamics element. The mutated attributes are dynamicsShape (i.e., transition shape, which determines how values ​​change over time or distance), value (actual transition value), and dynamicsDimension (transition is based on time or distance).

[0032] ●Vehicle Performance Mutation (VPM): VPM changes the performance characteristics of the vehicle in the test scenario, modifying the values ​​of maxAcceleration (maximum acceleration), maxDeceleration (maximum deceleration), maxSpeed ​​(maximum speed), etc.

[0033] (2) Trajectory mutation: This operator modifies the NPC predefined motion trajectory and the initial position of the autonomous vehicle. In order to ensure that the mutated scene file is consistent with the event progress logic of the text report, the present invention mainly uses the waypoint mutation (WPM) operator to fine-tune the lane change event offset, corresponding to the Waypoint element. When generating the scene, if the ADS initial position is not specified in the extracted event sequence, it is set to the lane with laneId = -1 by default. In order to fully evaluate the ADS pathfinding capability, the ADS initial position is mutated to the opposite lane of the same road (laneId = 1) with a probability of 5%, involving the Position element.

[0034] (3) Physical property mutation: This type of mutation operator refers to mutations that involve changing the physical properties of an entity. The specific operators are explained as follows:

[0035] Dimension Mutaiton (DM): Width, length, and height are critical to accurately simulate vehicles, pedestrians, or obstacles. DM adjusts these dimensions to simulate entities of different sizes.

[0036] ●Vehicle Category Mutation (VCM): This operator indirectly changes the physical form of the NPC by changing its category, not only changing the size, but also the shape, number of wheels, design features and other structural attributes.

[0037] (3) Environmental variation: This type of operator modifies the external conditions that affect the behavior of the ADS. The specific operators are as follows:

[0038] ●Weather Mutation (WM): WM changes the simulated weather type and randomly generates corresponding attributes, including precipitation intensity on rainy days, snowfall intensity on snowy days, visibility on foggy days, and sun azimuth and sun altitude on sunny days. At the same time, the road friction coefficient is adjusted according to the weather. The higher the rain or snow intensity, the lower the road friction.

[0039] ●Traffic Signals Mutation (TSM): TSM changes the state of traffic signals when the NPC interacts with the autonomous vehicle, testing the ADS's ability to handle different traffic light states. The mutation attribute is the target state of the traffic signal in the TrafficSignalState element.

[0040] 4. Scene file tree generation

[0041] The present invention sequentially inserts the mutated scene blocks into the scene template to generate multiple new scene files with different semantic richness. In the present invention, the scene blocks are inserted in the order of weather, traffic lights, NPCs, events, and obstacles. Figure 1 As shown, this assembly method generates a scene file tree with nodes having a derivative relationship. The number of nodes at each level is related to the number of scene block mutations. The derivative relationship means that the child node has one more semantically rich and complex scene block than the parent node. When the parent node does not trigger the ADS unexpected behavior but the child node triggers it, this feature helps to identify the triggering conditions of the unexpected behavior.

[0042] 5. Defect Detection

[0043] The present invention focuses on the unexpected behaviors of ADS in three dimensions: safety, stability, and intelligence. The safety of ADS is first evaluated by detecting whether a collision occurs. Users can use automated scripts to record the speed, angular velocity, braking and other information of ADS in the test scenario for analysis and evaluation, and realize automatic collision detection by identifying sudden deceleration without manual judgment. Sudden deceleration refers to a sudden drop in speed in a short period of time, that is, the acceleration change rate (jerk) exceeds a certain threshold. Existing studies mostly use acceleration changes at time intervals of 5ms to 15ms to calculate jerk, which can be used for collision detection and ADS stability evaluation. In ADS testing, stability refers to the ability to perform operations stably and naturally. Force is related to passenger comfort and smooth vehicle movement. Existing research also uses yaw rate as a key stability indicator. Because stability is a long-term state rather than instantaneous fluctuations, the present invention uses a 1-second moving average window to smooth jerk and yaw rate, reduce short-term peak impact, and more accurately reflect the dynamic performance of ADS. In terms of intelligence, the present invention explores and tests the logical chain behind the ADS behavior, that is, the decision-making process and response logic under a specific scenario combination, covering environmental factors and other traffic participants' behavior management and response methods. In order to comprehensively evaluate intelligence, in addition to collision and stability issues, the present invention detects three types of unexpected behaviors: abnormal startup, failure to reach the designated target point, and misidentification of traffic signals and obstacles.

Claims

1. A method for generating a simulation test scenario for autonomous driving based on a textual accident report, characterized in that The input is a text accident report describing the complete accident process, and the output is a scene file tree in OpenScENARIO format; the method includes the following steps: (1) Test scene element collection: The present invention first conducts an in-depth analysis of the semantic structure and grammatical rules of OpenSCENARIO and extracts necessary information from official documents. Since OpenSCENARIO adopts an XML-based modeling method and has a clear hierarchical structure relationship, the present invention constructs a graph database to store the attributes of scene elements and their connection relationships, providing necessary support for subsequent steps to ensure the grammatical correctness of the generated scene. (2) Test scenario conversion: The present invention utilizes the powerful natural language analysis and generation capabilities of the large language model to parse the complex natural language descriptions contained in the text; combined with the semantic characteristics of the text describing the test scenario, the present invention designs targeted prompt words to extract the static elements and dynamic elements contained in the scenario and their interactive behaviors with the tested object from the text description; by using the semantic and grammatical rules stored in the graph database, the present invention converts the parsing results into a test scenario file that complies with the OpenSCENARIO standard to ensure the structuring and standardization of the generated scenario; (3) Test scenario variation: To cover a wider range of test scenarios and assist developers in analyzing the behavioral causal chain of the autonomous driving system, the present invention designs a scenario decomposition and variation method; Through the scene decomposition technology, the complete scene description is decomposed into scene blocks containing complete semantics (corresponding to XML elements in OpenSCENARIO); then, the present invention designs four types of mutation operators for different scene elements (such as environmental elements, dynamic participants, etc.) to generate diversified test scenes; (4) Test scene assembly: Based on the scene variation, the present invention inserts the newly generated scene blocks into the scene template in the correct semantic order to generate a semantically rich test scene file tree with a parent-child derivation relationship; By comparing the execution results of the autonomous driving system in parent-child chain scenarios, or analyzing the execution results of the entire scenario, the triggering conditions of abnormal behavior can be identified, and the behavioral causal chain can be further explored to provide developers with logical improvement suggestions such as design and code.

2. A method for generating an autonomous driving simulation test scenario based on a textual traffic accident report according to claim 1, characterized in that The accident report is parsed by invoking a large language model using carefully designed prompt words, thereby converting the text report into a seed scene file; then, the technology uses scene decomposition, mutation, and assembly methods to generate a scene tree starting from a single seed. The mutation operations used include dynamic mutation, trajectory mutation, physical property mutation, and environmental mutation. These operations can effectively enhance the diversity of the generated scenes.

3. A method for generating an autonomous driving simulation test scenario based on a textual traffic accident report according to claim 1, characterized in that There is a derivative relationship between the generated scenario files, which can be used to identify the abnormal behavior triggering conditions of the autonomous driving system.

4. A method for generating an autonomous driving simulation test scenario based on a textual traffic accident report according to claim 1, characterized in that The adapted scene file format is OpenSCENARIO, an international standard that supports multiple simulation platforms.

Images