Automatic driving automobile key test scene generation method based on large language model

Abstract

The invention belongs to the technical field of automatic driving test, and particularly relates to an automatic driving vehicle key test scene generation method based on a large language model. The method comprises the following steps: 1, converting accident video data into semantic text description by a general large model, and establishing an accident video text library; 2, performing parameter fine adjustment on the open source large model by using the accident video text library, so that the large model has key features for identifying accident data; 3, inputting the design operation domain element conditions of the tested system into the fine-tuned large model, and generating a scene type meeting the test requirements according to the characteristics of the tested system; and 4, according to the scene type and the scene key elements output by the large model, performing parameter combination on the key elements by using a combination test method to generate key scene parameters. The invention solves the problem that the existing scene generation method is difficult to comprehensively consider the natural driving accident data characteristics and the design operation domain constraint of the tested system.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving testing, and specifically relates to a method for generating key test scenarios for autonomous driving vehicles based on large language models. Background Art

[0002] With the development of technologies such as sensing, computing power, and artificial intelligence, the mass production and road use of autonomous driving vehicles have become an inevitable trend. However, as a highly complex system, autonomous driving vehicles must undergo perfect testing to ensure their road safety. The "Guidelines for the Access Management of Intelligent Connected Vehicle Manufacturers and Products (Trial)" clearly states that a perfect test scenario library for autonomous driving vehicles should be established to ensure the completeness of testing. However, most of the data collected during traditional road tests are safety scenarios. How to generate key scenario types that are likely to interfere with the autonomous driving system with high coverage has become the focus of current research. Some technologies have started research on related content, such as "A Method for Generating Autonomous Driving Test Scenarios Based on a Personalized Driver Model" CN114492157B, "A Method, System, and Device for Generating Autonomous Driving Virtual Test Scenarios" CN116127854B. However, these existing scenario generation technologies mostly start from the scenarios in natural driving data or adversarial learning data, lacking consideration of the designed operating range of the system under test. How to generate high-value key scenarios while considering the designed operating domain of the system under test has not been effectively solved yet. Summary of the Invention

[0003] The present invention provides a method for generating key test scenarios for autonomous driving vehicles based on large language models, which solves the problem that existing scenario generation methods are difficult to comprehensively consider the characteristics of natural driving accident data and the constraints of the designed operating domain of the system under test. It can set standardized test cases for third-party testing institutions according to the test requirements of the sending manufacturers, improving the credibility and effectiveness of test results.

[0004] The technical solution of the present invention is described in conjunction with the accompanying drawings as follows:

[0005] The present invention provides a method for generating key test scenarios for autonomous driving vehicles based on large language models, including the following steps:

[0006] Step 1: The general large model converts accident video data into semantic text descriptions to establish an accident video text library;

[0007] Step 2: Use the accident video text library to fine-tune the parameters of the open-source large model so that the large model can identify the key features in the accident data;

[0008] Step 3: Input the elements and conditions of the designed operating domain of the system under test into the fine-tuned large model, and generate scenario types that meet the test requirements according to the characteristics of the system under test;

[0009] Step 4: According to the scenario type and scenario key elements output by the large model, use the combinatorial testing method to combine the parameters of the key elements to generate key scenario parameters.

[0010] Furthermore, the specific method of the above Step 1 is as follows:

[0011] 11) Collect and establish an accident video database;

[0012] 12) Input the accident videos collected in Step 11 into the Gemini-2.0 large model in sequence, and convert the accident videos or videos close to collisions into verbal text descriptions.

[0013] Furthermore, the accident video database includes the Road Hazard Stimuli database, the MP-RAD database, the CCD database, and the A3D database;

[0014] Furthermore, the specific method of the above Step 12 is as follows:

[0015] 121) Construct an ontology of scene key elements; including roads, road infrastructure, target objects, the vehicle itself, weather, and time;

[0016] 122) Establish a standardized prompt template; based on the ontology of scene elements established in Step 121, consider the process of the scene occurrence and the key features of the scene to establish a prompt template for scene text conversion;

[0017] Furthermore, the prompt template for scene text conversion is as follows: This is a traffic scene video from the perspective of a driving recorder. Analyze the video frame by frame, describe the entire process of the traffic scene in language, and analyze the main reasons for possible dangers in the scene; in addition, analyze the road type, number of lanes, lane line type, traffic sign type, signal state change, building type on both sides of the road, types and corresponding quantities and behaviors of surrounding traffic participants, the position of the vehicle itself on the road, the motion state of the vehicle itself, weather type, whether it is day or night currently; output the analyzed information as an excel table, with the first column being the verbal description of the scene, the second column being the main reasons for possible dangers, and each subsequent column being the road type and lane number information mentioned respectively.

[0018] Furthermore, the specific method of the above Step 2 is as follows:

[0019] 21) Convert the accident video text library into the standardized format of model fine-tuning data;

[0020] 22) Set the learning rate of the model to 1e-5, the number of training epochs to 100, the maximum gradient norm to 1, and complete the fine-tuning of the model parameters.

[0021] Further, the specific method of Step 3 is as follows:

[0022] 31) Allocate different scenario elements into the scenario element ontology model in Step 121) according to the designed operation domain provided by the system under test to establish a richer scenario ontology model;

[0023] 32) Sequentially select different scenario element types from the scenario elements established in Step 31), including road type, number of lanes, traffic sign type, signal light type, number of target vehicles, corresponding type of target vehicle, weather type, time. Replace the scenario elements in Step 21) where the road type is a straight road, the number of lanes is two lanes, the traffic sign type is none, the signal light type is none, the number of target vehicles is 1, the target vehicle type is a sedan, the weather type is sunny, and the time is daytime. And based on the fine-tuned model, output the scenario language description and explain the reason why this scenario is a critical scenario.

[0024] Further, the specific method of Step 4 is as follows:

[0025] 41) Conduct a weight analysis on different elements in the critical scenario generated in Step 32) according to the analytic hierarchy process. The analysis basis is the reason why different scenario elements make the scenario mentioned in Step 32) a critical scenario. Select 5 to 10 types of scenario elements with the highest importance weights of scenario elements as variable parameters under the scenario. The number of variable parameters is judged according to the complexity of the scenario;

[0026] 42) Set the parameter space by statistically analyzing the variation range of the variable parameters in the natural driving process in Step 41). The discrete step size of different parameters is adjusted using an adaptive step size adjustment scheme according to the size of the parameter space and the importance degree of the parameters. Sequentially adjust the values of different parameters (starting from the position with the highest natural driving probability), and observe the maximum collision time in the entire scenario when using the standard driver model for testing before and after the test parameter adjustment. If the change in the collision time is greater than 0.1 s, then use this change value as the step size of this type of parameter. Subsequently, use the method of multi-factor combination testing to determine the combination strength, and set the combination strength to 3.

[0027] In Step 42), the situation where the parameter is sensitive or dangerous is likely to occur during the natural driving process means analyzing by sequentially adjusting the values of different parameters

[0028] The beneficial effects of the present invention are:

[0029] The present invention proposes a method for solving the generation of key test scenarios for autonomous vehicles by integrating large model technology, which solves the problem that existing scenario generation methods are difficult to comprehensively consider the characteristics of natural driving accident data and the constraints of the designed operating domain of the system under test. It can set standardized test cases for third-party testing institutions according to the test requirements of the sending manufacturers, improving the credibility and effectiveness of test results. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the embodiments. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0031] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention and not to limit the present invention. Additionally, it should be noted that for the convenience of description, only the parts related to the present invention are shown in the drawings and not all structures.

[0033] Refer to Figure 1 , this embodiment provides a method for generating key test scenarios for autonomous vehicles based on a large language model, including the following steps:

[0034] Step 1: The general large model converts accident video data into semantic text descriptions and establishes an accident video text library, specifically as follows:

[0035] 11) Collect and establish an accident video database; the accident video database includes the Road Hazard Stimuli database, the MP-RAD database, the CCD database, and the A3D database;

[0036] 12) Sequentially input the accident videos collected in step 11) into the Gemini-2.0 large model, and convert the accident videos or videos of near collisions (the "Automated Lane Keeping System ALKS" defines "near collision risk" as "a situation where an autonomous vehicle needs to achieve a braking deceleration of more than 5 m / s 2 above to possibly avoid a collision.").) into language text descriptions, specifically as follows:

[0037] 121) Construct the ontology of key elements of the scenario. The ontology of scenario elements can be constructed with reference to the six-layer scenario model, including roads (type, number of lanes), road infrastructure (type of lane lines, traffic signs, signal lights, roadside buildings), target objects (type, location, relative motion state), the vehicle itself (location in the road, motion state), weather (type), time (morning / evening).

[0038] 122) Establish a standardized prompt template. Based on the ontology of scenario elements established in step 121), consider the process of the scenario occurrence and the key features of the scenario to establish a prompt template for scenario text conversion. Specifically, the prompt template for scenario text conversion is: "This is a traffic scenario video from the perspective of a driving recorder. Analyze the video frame by frame, and use very detailed language to describe the entire process of the scenario, and analyze the main reasons for possible dangers in this scenario. In addition, analyze the road type, number of lanes, type of lane lines, type of traffic signs, changes in signal light states, types of buildings on both sides of the road, types and corresponding quantities and behaviors of surrounding traffic participants, the location of the vehicle itself in the road, the motion state of the vehicle itself, weather type, and whether it is day or night currently. Output the above information as an excel table, with the first column being the language description of the scenario, the second column being the main reasons for possible dangers, and each subsequent column being all the information such as the road type, number of lanes, etc. mentioned." The video conversion text effect is shown in Table 1:

[0039] Table 1

[0040]

[0041]

[0042]

[0043] Step 2: Use the accident video text library to fine-tune the parameters of the open-source large model so that the large model can identify the key features in the accident data, specifically as follows:

[0044] 21) Convert the accident video text library into the standardized format of the fine-tuning data for the Qwen2-1.5B-Instruct-MLX model. The format of each video text training data is as follows:

[0045] "{

[0046] "instruction":"Given the following scenario elements, generate a key test scenario and explain the reason why this scenario is a key scenario.",

[0047] "input": "The road type is a straight road, the number of lanes is two lanes, the type of traffic sign is none, the type of traffic signal is none, the number of target vehicles is 1, the type of target vehicle is a sedan, the weather type is sunny, and the time is daytime.",

[0048] "output": "During a sunny daytime, our vehicle is driving on the left road of a straight two-lane road. There are no traffic signs and traffic signals on the road. Suddenly, a sedan on the right road cuts in from the left in front of our vehicle and decelerates, resulting in a collision between our vehicle and the sedan in front. The main reason for the danger is that the sedan suddenly makes a lane-changing and decelerating maneuver, causing a collision with our vehicle.",

[0049] },”

[0050] In the formula, the scene elements after "input" are the data in the 3rd column and the last column of the scene data table generated in step 122), and the data after "output" are the data in the 1st column and the 2nd column of the scene data table generated in step 122).

[0051] 22) Set the learning rate of the model to 1e-5, the number of training epochs to 100, the maximum gradient norm to 1, and complete the fine-tuning of the model parameters.

[0052] Step 3: Input the design operation domain element conditions of the system under test into the fine-tuned large model, and generate a scene type that meets the test requirements according to the characteristics of the system under test, as follows:[[]]END]

[0053] 31) According to the design operation domain provided by the system under test, allocate different scene elements into the scene element ontology model in step 121) to establish a richer scene ontology model;

[0054] 32) Select different scene element types in turn from the scene elements established in step 31), including road type, number of lanes, type of traffic sign, type of traffic signal, number of target vehicles, corresponding type of target vehicle, weather type, time, replace the scene elements in "input" in step 21), and output the scene language description based on the fine-tuned Qwen2-1.5B-Instruct-MLX model and explain the reason why this scene is a key scene.

[0055] Among them, some scene elements of the design operation domain of the system under test are shown in Table 2:

[0056] Table 2

[0057]

[0058]

[0059] Step 4: According to the scene type and scene key elements output by the large model, use the combination test method to combine the key elements and generate key scene parameters, as follows:

[0060] 41) performing a weight analysis on different elements in the key scene generated in step 32) according to the hierarchical analysis method, the analysis basis being the reason why different scene elements are a key scene for the scene mentioned in step 32), selecting 5 to 10 scene elements with the highest importance weight as variable parameters for the scene, and determining the number of variable parameters according to the complexity of the scene;

[0061] 42) The parameter space is set by counting the range of variation of the variable parameters in step 41) during the natural driving process. The discrete step sizes of different parameters are adjusted using an adaptive step size scheme according to the size of the parameter space and the importance of the parameters. A smaller step size is used for situations where the parameters are sensitive or the natural driving process is prone to danger, and a larger step size is used for other non-critical situations. Subsequently, the combination strength is determined using a multi-factor combination test method, and the combination strength is set to 3.

[0062] Among them, the output results of some generated test scenario types are shown in Table 3:

[0063] Table 3

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] In summary, the problem that the existing scenario generation methods are difficult to comprehensively consider the characteristics of natural driving accident data and the constraints of the design and operation domain of the system under test has been solved. It can help third-party testing agencies set up standardized test cases based on the testing requirements of the test manufacturers, thereby improving the credibility and effectiveness of the test results.

[0071] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating key test scenarios for autonomous driving vehicles based on a large language model, characterized in that: The following steps are involved: Step 1: The general large model converts the accident video data into semantic text descriptions and establishes an accident video text library; Step 2: Use the accident video text library to fine-tune the parameters of the open source big model so that the big model can identify key features in the accident data; Step 3: Input the design and operation domain elements of the system under test into the fine-tuned large model, and generate scenario types that meet the test requirements based on the characteristics of the system under test; Step 4: Based on the scene type and scene key elements output by the large model, use the combination test method to combine the parameters of the key elements to generate key scene parameters.

2. The method for generating key test scenarios for an autonomous driving vehicle based on a large language model according to claim 1, characterized in that: The specific method of step one is as follows: 11) Collect and establish accident video database; 12) Input the accident videos collected in step 11) into the Gemini-2.0 model in sequence, and convert the accident videos or near-collision videos into language descriptions.

3. The method for generating key test scenarios for an autonomous driving vehicle based on a large language model according to claim 2, characterized in that: The accident video database includes a Road Hazard Stimuli database, an MP-RAD database, a CCD database and an A3D database.

4. The method for generating key test scenarios for an autonomous driving vehicle based on a large language model according to claim 2, characterized in that: The specific method of step 12) is as follows: 121) Construct the key elements of the scene ontology, including roads, road infrastructure, targets, the vehicle, weather and time; 122) Establish standardized prompt word templates; Based on the scene element ontology established in step 121), a scene text conversion prompt word template is established by considering the scene occurrence process and the scene key features.

5. The method for generating key test scenarios for an autonomous driving vehicle based on a large language model according to claim 4, characterized in that: The scene text conversion prompt word template is: This is a traffic scene video from the perspective of a driving recorder. The video is analyzed frame by frame, the entire process of the traffic scene is described in language, and the main reasons for possible dangers in the scene are analyzed; in addition, the road type, number of lanes, lane line type, traffic sign type, signal light status changes, building types on both sides of the road, types and corresponding numbers and behaviors of surrounding traffic participants, the position of the vehicle on the road, the movement status of the vehicle, weather type, and whether it is daytime or night are analyzed in the traffic scene; the analyzed information is output as an Excel table, the first column of the table is the language description of the scene, the second column is the main reasons for possible dangers, and each subsequent column is the mentioned road type and lane number information.

6. The method for generating key test scenarios for an autonomous driving vehicle based on a large language model according to claim 1, characterized in that: The specific method of step 2 is as follows: 21) Convert the accident video text library into a standardized format for model fine-tuning data; 22) Set the learning rate of the model to 1e-5, the number of training rounds to 100, the maximum gradient norm to 1, and complete the model parameter fine-tuning.

7. The method for generating key test scenarios for an autonomous driving vehicle based on a large language model according to claim 4, characterized in that: The specific method of step three is as follows: 31) Allocating different scene elements to the scene element ontology model in step 121) according to the design operation domain provided by the system under test, and establishing a richer scene ontology model; 32) Select different scene element types from the scene elements established in step 31), including road type, number of lanes, traffic sign type, signal light type, number of target vehicles, corresponding type of target vehicle, weather type, and time, and replace the scene elements in step 21) with the road type of straight road, the number of lanes of two lanes, the traffic sign type of none, the signal light type of none, the number of target vehicles of 1, the target vehicle type of sedan, the weather type of sunny day, and the time of daytime, and output the scene language description based on the fine-tuned model and explain why the scene is a key scene.

8. The method for generating key test scenarios for an autonomous driving vehicle based on a large language model according to claim 7, characterized in that: The specific method of step 4 is as follows: 41) performing a weight analysis on different elements in the key scene generated in step 32) according to the analytic hierarchy process, the analysis basis being the reason why different scene elements are a key scene for the scene mentioned in step 32), selecting 5 to 10 scene elements with the highest importance weight as variable parameters for the scene, and the number of variable parameters is determined according to the complexity of the scene; 42) The parameter space is set by statistically analyzing the range of variation of the variable parameters in step 41) during natural driving. The discrete step sizes of different parameters are adjusted using an adaptive step size scheme according to the size of the parameter space and the importance of the parameters. A smaller step size is used for situations where the parameters are sensitive or the natural driving process is prone to danger, and a larger step size is used for other non-critical situations. Subsequently, the combination strength is determined using a multi-factor combination test method, and the combination strength is set to 3.

Images