Automatic driving test scene generation framework and generation method

Abstract

The invention discloses an automatic driving test scene generation framework and generation method, and the method comprises the steps: a road network generator is used for obtaining a node file and an edge file generated by a multi-modal large language model based on prompt word engineering and scene description, inputting the node file and the edge file into a traffic simulation tool, and obtaining a road network file synthesized by the traffic simulation tool; the vehicle generator is used for acquiring a plurality of initial vehicle models generated by the multi-modal large language model based on prompt word engineering and scene description, inputting the initial vehicle models into a traffic simulation tool, and acquiring a vehicle model generated after the traffic simulation tool adds a path file for each initial vehicle model; and the scene generation module is used for integrating the road network file and the plurality of vehicle models, obtaining an automatic driving test scene, building a bridge for interactive use of the multi-modal large language model and a traffic simulation tool, and constructing a traffic simulation mechanism of the multi-modal large language model based on a prompt project. A customized, real, reliable and targeted automatic driving test scene can be generated.

Description

Technical Field

[0001] This article relates to the field of autonomous driving testing, and in particular to an autonomous driving test scenario generation framework and generation method. Background Art

[0002] In recent years, autonomous driving technology has made significant progress. Due to the high risk, low test efficiency and high test cost of open road testing, autonomous driving simulation testing has become a commonly used testing method in the industry. Autonomous driving simulation testing requires a large number of autonomous driving test scenarios. Currently, there are two main methods for generating autonomous driving test scenarios. One is to generate a simulation test scenario library based on expert experience. This method mainly relies on expert experience, is difficult to apply on a large scale, and the test results do not have theoretical reliability guarantees. The other is a data-driven method that mainly uses training data to generate realistic test scenarios. However, this method is limited by training data, has limited scenario coverage, and cannot generate specific test scenarios. There are challenges in meeting customized testing requirements. Summary of the invention

[0003] The embodiments of the present application provide an autonomous driving test scenario generation framework and generation method, construct a traffic simulation mechanism of a multimodal large language model based on prompt engineering, and can generate customized, realistic, reliable, and targeted autonomous driving test scenarios.

[0004] On the one hand, an embodiment of the present application provides an autonomous driving test scenario generation framework, including:

[0005] A road network generator is used to obtain a node file and an edge file generated by a multimodal large language model based on a prompt word project and a scene description, and input the node file and the edge file into a traffic simulation tool to obtain a road network file synthesized by the traffic simulation tool;

[0006] A vehicle generator, used to obtain a plurality of initial vehicle models generated by the multimodal large language model based on the prompt word project and the scenario description, and input each of the initial vehicle models into the traffic simulation tool, and obtain a vehicle model generated by the traffic simulation tool after adding a path file to each of the initial vehicle models, wherein each of the initial vehicle models includes a departure point, an arrival point, and a departure time;

[0007] The scenario generation module is used to integrate the road network file and the multiple vehicle models to obtain an autonomous driving test scenario.

[0008] On the other hand, an embodiment of the present application provides a method for generating an autonomous driving test scenario, comprising:

[0009] Inputting the node file and the edge file into a traffic simulation tool to obtain a road network file synthesized by the traffic simulation tool;

[0010] Inputting a plurality of initial vehicle models into the traffic simulation tool, and obtaining a vehicle model generated by the traffic simulation tool after adding a path file to each of the initial vehicle models;

[0011] Integrate the road network file and the plurality of vehicle models to generate an autonomous driving test scenario;

[0012] Among them, the node file, the edge file and multiple initial vehicle models are all generated by a multimodal large language model based on prompt word engineering and scene description; each of the initial vehicle models includes a departure point, an arrival point and a departure time.

[0013] Compared with the related art, an autonomous driving test scenario generation framework and generation method in an embodiment of the present application builds a bridge for the interactive use of a multimodal large language model and a traffic simulation tool, constructs a traffic simulation mechanism of a multimodal large language model based on prompt engineering, fully utilizes the extensive external knowledge reasoning capabilities of the multimodal large language model, and uses a traffic simulation tool to simplify the complexity of generating scenario codes, and combines specific textual scenario descriptions to generate customized, realistic, reliable, and targeted autonomous driving test scenarios, meeting diverse autonomous driving scenario testing needs, and using a multimodal large language model and a traffic simulation tool to improve the efficiency of generating autonomous driving test scenarios. Autonomous driving test scenarios can be generated quickly and efficiently, are easy to apply on a large scale, and help improve the efficiency and accuracy of autonomous driving tests.

[0014] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by implementing the present application. Other advantages of the present application can be realized and obtained by the schemes described in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0016] Figure 1 A schematic diagram of a framework for generating an autonomous driving test scenario in an embodiment of the present application;

[0017] Figure 2 This is a flow chart of a method for generating an autonomous driving test scenario according to an embodiment of the present application;

[0018] Figure 3 A schematic diagram of a system for generating an autonomous driving test scenario according to a specific example 1 of the present application;

[0019] Figure 4 A schematic diagram of a system for generating an autonomous driving test scenario for specific example 2 of this application;

[0020] Figure 5 This is an autonomous driving test scenario diagram generated based on the collision report of specific example 2 of this application. DETAILED DESCRIPTION

[0021] The present application describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present application. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0022] The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements disclosed in the present application may also be combined with any conventional features or elements to form a unique invention scheme. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme. Therefore, it should be understood that any feature shown and / or discussed in the present application may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other restrictions. In addition, various modifications and changes may be made within the scope of protection of the attached claims.

[0023] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps of the specific order described. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps of performing them in the order written, and those skilled in the art can easily understand that these sequences can be changed and still remain within the spirit and scope of the embodiments of the present application.

[0024] Safety is the main factor hindering the widespread deployment of autonomous driving vehicles at home and abroad. To this end, the industry has devoted a lot of energy and is committed to eliminating potential safety hazards by implementing strict and comprehensive testing processes. The current autonomous driving test methods are constantly improving, mainly including two test methods. One is open road testing, which can complete autonomous driving tests in complex and changeable traffic environments, but its main problems are high risk, low test efficiency, and high cost. It will bring unpredictable safety risks to other pedestrians and vehicles using the road, and there are potential social safety hazards; the other method is computer simulation testing, which avoids the safety hazards caused by open road testing, can better cover extreme situations, and has low cost. At present, autonomous driving test scenarios are mainly constructed based on expert experience or data-driven methods to complete simulation tests. Methods based on expert experience rely on experience and are difficult to apply on a large scale, while data-driven methods are difficult to generate targeted scenarios and difficult to meet specific test needs.

[0025] The present application embodiment provides an autonomous driving test scenario generation framework, such as Figure 1 As shown:

[0026] A road network generator is used to obtain a node file and an edge file generated by a multimodal large language model based on a prompt word project and a scene description, and input the node file and the edge file into a traffic simulation tool to obtain a road network file synthesized by the traffic simulation tool;

[0027] A vehicle generator, used to obtain a plurality of initial vehicle models generated by the multimodal large language model based on the prompt word project and the scenario description, and input each of the initial vehicle models into the traffic simulation tool, and obtain a vehicle model generated by the traffic simulation tool after adding a path file to each of the initial vehicle models, wherein each of the initial vehicle models includes a departure point, an arrival point, and a departure time;

[0028] The scenario generation module is used to integrate the road network file and the multiple vehicle models to obtain an autonomous driving test scenario.

[0029] In this embodiment, the autonomous driving test scenario generation framework, the multimodal large language model, and the traffic simulation tool together form an autonomous driving test scenario generation system. The road network generator and the vehicle generator are bridges between the multimodal large language model and the traffic simulation tool, and establish a connection between unrelated multimodal large language models and traffic simulation tools. After the road network generator obtains the node file (also called node file) and edge file (also called edge file) generated by the multimodal large language model, the node file and the edge file are input into the traffic simulation tool. The traffic simulation tool simulates and synthesizes the road network file based on the node file and the edge file, and then sends the synthesized road network file to the road network generator.

[0030] In this embodiment, the node file refers to a control node in the road network, such as an intersection, a starting point and an end point of each path in the road network; the edge file refers to a path file connecting any two nodes.

[0031] In this embodiment, the traffic simulation tool may be SUMO (full name: Simulation of Urban Mobility). SUMO is an open source, highly portable microscopic traffic simulation software package that can process and generate road networks of various sizes.

[0032] In this embodiment, the road network generator can connect the multimodal large language model and the netconvert tool of SUMO. The multimodal large language model can generate node files and edge files in XML format that can be recognized by the netconvert tool according to the prompt word project. Then, the road network generator transfers the node files and edge files in XML format to the netconvert tool of SUMO to generate a road network file in XML format, wherein the road network file contains real and diverse road geometry scenes.

[0033] In this embodiment, the vehicle generator can connect the multimodal large language model and the duarouter tool of SUMO. After obtaining the initial vehicle model generated by the multimodal large language model, the vehicle generator can input the initial vehicle model into the duarouter tool of SUMO. The duarouter tool adds a path file to the initial vehicle model to generate a vehicle model, wherein the vehicle model includes at least a background vehicle model and a vehicle model to be tested.

[0034] In this embodiment, the scene description is a text description of the scene, which can be obtained based on the scene generation requirements input by the user, or directly using files such as accident reports as the scene description. The multimodal large language model generates node files and edge files based on the first small sample prompt words of the prompt word project and the scene description; an initial vehicle model is generated based on the second small sample prompt words of the prompt word project and the scene description, and the initial vehicle model includes the starting point, arrival point and departure time of the vehicle model, but does not include the path from the starting point to the arrival point; the scene description may include the type of scene, the time of the scene (night or day), the width of the main road in the road network, the number of forks, the width of the forks, the starting point and end point of each road, the location of fixed buildings, the number of background vehicle models in the scene, the driving direction, starting point, arrival point, departure time, color, size, weight, number of passengers corresponding to the background vehicle model and the vehicle model to be tested, etc. The contents included in the above scene description are all exemplary descriptions, which are not limited to this application and will not be repeated one by one.

[0035] In this embodiment, the scenario generation module integrates the road network file and multiple vehicle models, and sets the multiple vehicle models in the road network file, that is, the multiple vehicle models interact in the road network to obtain an autonomous driving test scene. The types of autonomous driving test scenes may include intersections, highways, roads around schools, rural roads, roads in rainy and snowy weather, etc.

[0036] The autonomous driving test scenario generation framework of this embodiment builds a bridge for the interactive use of the multimodal large language model and the traffic simulation tool, constructs a traffic simulation mechanism of the multimodal large language model based on the prompt engineering, fully utilizes the extensive external knowledge reasoning ability of the multimodal large language model, and uses the traffic simulation tool to simplify the complexity of generating scenario codes, and generates customized, reliable, and targeted autonomous driving test scenarios in combination with specific textual scenario descriptions, meeting the diverse autonomous driving scenario testing needs, and using the multimodal large language model and traffic simulation tool to improve the efficiency of generating autonomous driving test scenarios, and can quickly and efficiently generate autonomous driving test scenarios, which is easy to apply on a large scale and helps to improve the efficiency and accuracy of autonomous driving tests.

[0037] In an exemplary embodiment, the node file and the edge file may be generated by the multimodal large language model according to a pre-designed first small sample prompt word and the scene description, wherein the first small sample prompt word includes scene constraints and format requirements of the traffic simulation tool;

[0038] The initial vehicle model may be generated by the multimodal large language model according to a pre-designed second small sample prompt word and the scene description, wherein the second small sample prompt word includes a vehicle generation requirement;

[0039] The vehicle model includes a vehicle model to be tested and a background vehicle model.

[0040] In this embodiment, the first small sample prompt word may include two parts: the constraints of the scene and the format requirements of SUMO for node files and edge files. The format requirements include generating node files and edge files in XML format. The constraints of the scene include the road length, intersection location, traffic light location, static obstacle location, speed limit sign location, etc. in the scene. The multimodal large language model can generate node files and edge files in XML format based on the input first small sample prompt word and scene description.

[0041] In this embodiment, the second small sample prompt words include vehicle generation requirements, including the maximum speed corresponding to each vehicle model, SUMO format requirements for vehicle models, etc. The multimodal large language model can generate an initial vehicle model that meets the SUMO format requirements based on the input second small sample prompt words and scene description.

[0042] In this embodiment, the first small sample prompt word and the second small sample prompt word can both be generated by a prompt word generator.

[0043] In an exemplary embodiment, the autonomous driving test scenario generation framework may also include: a retrieval enhancement generation module, which is used to obtain a scenario generation request input by a user, and search in a pre-built road network database according to the scenario generation request to obtain relevant road networks and corresponding road network text descriptions.

[0044] In this embodiment, existing road network files can be used to generate a road network database, which includes a large number of road network files. These road network files can come from an open street map database and re-collected data. The road network files are visualized as images and described in text to generate a road network text description that corresponds one-to-one with the road network images, and then the road network database is obtained. The road network database stores the correspondence between the road network files, road network images and road network text descriptions.

[0045] In this embodiment, the retrieval enhancement generation module can search in the road network database according to the scene generation request input by the user, obtain the road network image most similar to the scene generation request as the relevant road network, and obtain the road network text description corresponding to the relevant road network.

[0046] In an exemplary embodiment, the road network database may include a correspondence between a road network and a text description of the road network, and the text description of the road network may be generated by the multimodal large language model based on the road network.

[0047] In this embodiment, when generating a road network text description for a road network image, each road network image can be input into a multimodal large language model for image recognition, and a text description of the road network image is generated by the multimodal large language model. The text description is feature extracted using text-embedding-ada-002 to generate a road network text description corresponding to the road network image. The road network database can be constructed based on Chroma DB.

[0048] In an exemplary embodiment, the autonomous driving test scenario generation framework may further include: a text interpreter including a prompt word generator and an answer extractor;

[0049] The prompt word generator may be used to generate prompt words, the prompt words including the first small sample prompt words, the second small sample prompt words and the third small sample prompt words, and send the prompt words to the multimodal large language model;

[0050] The answer extractor may be used to obtain a detailed description generated by the multimodal large language model based on the scenario generation request, the road network text description and the third sample prompt word, extract information from the detailed description, generate the scenario description and send the scenario description to the multimodal large language model;

[0051] The third small sample prompt words include scene description examples.

[0052] In this embodiment, the text interpreter improves the performance of the multimodal large language model through the prompt words generated by the prompt word generator to generate a detailed scene description, that is, a detailed description.

[0053] In this embodiment, after the retrieval enhancement generation module sends the scene generation request and the road network text description to the multimodal large language model, and the prompt word generator sends the third small sample prompt word to the multimodal large language model, the multimodal large language model can generate a detailed description of the scene based on the scene generation request, the road network text description, and the third small sample prompt word, the answer extractor can obtain and process the detailed description, retain key information to obtain the scene description, and return the scene description to the multimodal large language model; the multimodal large language model can generate a node file and an edge file based on the scene description and the first small sample prompt word, and generate an initial vehicle model based on the scene description and the second small sample prompt word.

[0054] In this embodiment, the scene description example in the third small sample prompt word is a minimized scene example description, which can help the multimodal large language model understand the task of generating a detailed description; the scene description generated by the answer extractor retains the key information in the detailed description, including road direction, number of lanes, number of vehicles, etc.

[0055] In this embodiment, the scene generation request may also include descriptions of pedestrians and buildings. The detailed description generated by the multimodal large language model and the scene description generated by the answer extractor may both include these pedestrian and building description information, so that when the multimodal large language model generates a road network file based on the scene description and the first small sample prompt word, the road network file includes pedestrian and building information in addition to road information, thereby forming a scene with richer elements.

[0056] In an exemplary embodiment, the scene description may include a pre-acquired collision report.

[0057] In this embodiment, the autonomous driving test scenario generation framework may not include a retrieval enhancement generation module and an answer extractor, and the prompt word generator may not generate a third small sample prompt word. The scene description may use a pre-acquired collision report, and the collision report is input into the multimodal large language model as the scene description. The multimodal large language model may generate a node file and an edge file based on the collision report and the first small sample prompt word, and generate an initial vehicle model based on the collision report and the second small sample prompt word.

[0058] In an exemplary embodiment, the autonomous driving test scenario generation framework may also include: a performance evaluator, used to send an evaluation command to the traffic simulation tool, obtain the path error prompt information generated by the traffic simulation tool and send it to the multimodal large language model, obtain the path correction file generated by the multimodal large language model based on the prompt word engineering and the path error prompt information, and send the path correction file to the traffic simulation tool, iteratively optimize the initial path file, and generate the path file.

[0059] In this embodiment, the performance evaluator is a bridge between the multimodal large language model and the traffic simulation tool. When the traffic simulation tool is SUMO, the performance evaluator can connect the multimodal large language model and the duarouter tool of SUMO.

[0060] In this embodiment, the initial path file can correspond to the initial vehicle model one by one. For each initial vehicle model, the duarouter tool can connect the starting point and the arrival point of the initial vehicle model to generate an initial path file. Due to the structural limitations of the road network, there may not be a solvable reachable path between the starting and arrival points of the initial vehicle model generated by the multimodal large language model. The performance evaluator optimizes the initial path file, and the path file generated after optimization can conform to the actual road network information contained in the road network file, so that the vehicle model with the added path file can realize simulated interaction in the road indicated by the road network file.

[0061] In this embodiment, the path file corresponds to the vehicle model one by one, and the performance evaluator can send an evaluation command to the duarouter tool; after receiving the evaluation command, the duarouter tool can evaluate according to the road network file generated by the netconvert tool of SUMO and the initial path file generated by the duarouter tool, and determine whether there is an error in the initial path file. If there is an error, a path error prompt information is sent to the performance evaluator; after receiving the path error prompt information, the performance evaluator forwards it to the multimodal large language model; the multimodal large language model obtains the fourth small sample prompt word sent by the prompt word generator, and the fourth small sample prompt word includes the path requirement. The multimodal large language model generates a path correction file according to the path error prompt information, the scene description and the fourth small sample prompt word; the performance evaluator obtains the path correction file and forwards it to the traffic simulation tool, and the traffic simulation tool iteratively optimizes the initial path file according to the path correction file. This iterative optimization process may include one time or multiple times until the duarouter tool determines that there is no error in the path file, which means that the path file has been generated, and the duarouter tool adds the path file to the initial vehicle model, generates a vehicle model and sends it to the vehicle generator.

[0062] In this embodiment, self-improvement of the autonomous driving test scenario generation system is achieved through the performance evaluator. The performance evaluator is an intermediate bridge connecting the multimodal large language model and the traffic simulation tool. The performance evaluator, the multimodal large language model and the traffic simulation tool jointly realize the correction and optimization of the path file in the vehicle model, enhances the multimodal large language model's understanding of the scene, improves the degree of consistency between the vehicle model and the road network file, and ensures the rationality and accuracy of the generated autonomous driving test scenario.

[0063] The present application also provides a method for generating an autonomous driving test scenario, including steps S100-S300. Figure 2 As shown:

[0064] S100: Inputting the node file and the edge file into a traffic simulation tool to obtain a road network file synthesized by the traffic simulation tool;

[0065] S200: Inputting a plurality of initial vehicle models into the traffic simulation tool, and obtaining a vehicle model generated by the traffic simulation tool after adding a path file to each of the initial vehicle models;

[0066] S300: Integrate the road network file and the plurality of vehicle models to generate an autonomous driving test scenario;

[0067] In this embodiment, the node file, the edge file and the multiple initial vehicle models are all generated by a multimodal large language model based on prompt word engineering and scene description; each of the initial vehicle models includes a departure point, an arrival point and a departure time.

[0068] In this embodiment, the generation method can be executed based on the autonomous driving test scenario generation framework as described in the above embodiment.

[0069] In this embodiment, in step S100, the road network generator can execute the interaction between the multimodal large language model and the traffic simulation tool. The multimodal large language model generates a node file (also called a node file) and an edge file (also called an edge file) according to the first small sample prompt word and the scene description, and can send the node file and the edge file to the road network generator; the road network generator sends the node file and the edge file to the traffic simulation tool; the traffic simulation tool generates a road network file according to the received node file and edge file, and returns it to the road network generator; wherein the first small sample prompt word may include scene constraints and the format requirements of the traffic simulation tool, for example, the traffic simulation tool is SUMO, and SUMO requires the format of the node file and the edge file to be in XML format. After the instruction of the first small sample prompt word, the multimodal large language model can generate a node file and an edge file in XML format, so that the netconvert tool of SUMO can recognize the node file and the edge file and generate a road network file in XML format.

[0070] In this embodiment, in step S200, the vehicle generator can perform the interaction between the multimodal large language model and the traffic simulation tool, the multimodal large language model generates an initial vehicle model according to the second small sample prompt word and the scene description, and the initial vehicle model can be sent to the vehicle generator; the vehicle generator sends the initial vehicle model to the traffic simulation tool; the initial vehicle model includes a departure point, an arrival point and a departure time, and the duarouter tool of the traffic simulation tool SUMO can add a path file to the initial vehicle model, generate a vehicle model and return it to the vehicle generator; wherein the path file is a path connecting the departure point to the arrival point and conforming to the road network file, there are at least two vehicle models, including at least a background vehicle model and a vehicle model to be tested, the second small sample prompt word includes a vehicle generation requirement, and the vehicle generation requirement can also include a SUMO format requirement, so that the SUMO duarouter tool can recognize the initial vehicle model and add a path file to it.

[0071] In this embodiment, in step S300, the scene generation module can integrate the road network file and multiple vehicle models, and set the multiple vehicle models in the road network file, that is, the multiple vehicle models interact in the road network to obtain an autonomous driving test scene.

[0072] In an exemplary embodiment, before "obtaining the vehicle model generated by the traffic simulation tool after adding the path file to each of the initial vehicle models" in step S200, step S150 may be performed for each of the initial vehicle models to obtain the path file corresponding to the initial vehicle model:

[0073] S150: instructing the traffic simulation tool to interact with the multimodal large language model, performing one or more iterative optimizations on the initial path file of the initial vehicle model, and generating the path file corresponding to the initial vehicle model;

[0074] In this embodiment, the initial path file can be a straight line connecting the starting point and the arrival point of the initial vehicle model. Since the vehicle model realizes simulated automatic driving operation in the road network file, the route traveled by the vehicle model must comply with the actual route requirements in the road network file, that is, when the vehicle model travels in the road network included in the road network file, the driving route needs to comply with the road direction restrictions in the road network and cannot drive out of the known area. The initial path file needs to be iteratively optimized. In step S150, the performance evaluator can instruct the multimodal large language model and the traffic simulation tool to interact, and the duarouter tool of SUMO of the traffic simulation tool can be connected to the performance evaluator to realize the interaction between the duarouter tool and the multimodal large language model, and generate a path file corresponding to the vehicle model one by one.

[0075] In an exemplary embodiment, each iteration optimization in step S150 may include steps S151-S153:

[0076] S151: sending an evaluation command to the traffic simulation tool to obtain an analysis result generated by the traffic simulation tool according to the road network file and the pre-correction path file, wherein the analysis result is used to determine whether there is an error in the pre-correction path file, if not, executing step S152; if yes, executing step S153;

[0077] S152: instructing the traffic simulation tool to use the pre-correction path file as the path file;

[0078] S153: Receive the path error prompt information sent by the traffic simulation tool, forward the path error prompt information to the multimodal large language model, obtain the path correction file generated by the multimodal large language model based on the path error prompt information, the scene description and the pre-designed fourth small sample prompt word, and forward the path correction file to the traffic simulation tool, instruct the traffic simulation tool to optimize the pre-correction path file according to the path correction file to generate a corrected path file, and use the corrected path file as the pre-correction path file for the next iterative optimization.

[0079] In this embodiment, the pre-correction path file during the first iterative optimization is the initial path file, and the fourth small sample prompt word includes path requirements.

[0080] In this embodiment, steps S151-S153 may be executed by a performance evaluator, and the iterative optimization may be performed once or multiple times until there is no error in the pre-correction path file, which means that the path file has been generated.

[0081] In this embodiment, for example, in one of the iterations, the path file before correction is that the vehicle starts from point A and reaches point B through edges e2 and e4 in the road network file. However, when executing step S151, after the duarouter tool of SUMO receives the evaluation command sent by the performance evaluator, it uses the road network file and the path file before correction for analysis. The analysis result is "no connection between edge e2 and edge e4 is found, and the vehicle cannot reach edge e4 from edge e2". The performance evaluator receives the path error prompt information output by the duarouter tool.

[0082] In this embodiment, in order to illustrate the autonomous driving test scenario generation framework and generation method of the embodiment of the present application, a specific example 1 is used to describe in detail as follows: Figure 3 The autonomous driving test scenario generation system shown includes an autonomous driving test scenario generation framework, a multimodal large language model, a traffic simulation tool, and a road network database:

[0083] In this specific example, the autonomous driving test scenario generation framework includes: a road network generator, a vehicle generator, a scenario generation module, a retrieval enhancement generation module, a text interpreter, and a performance evaluator; wherein the text interpreter includes an answer extractor and a prompt word generator; the road network generator, the vehicle generator, and the performance evaluator are bridges between the multimodal large language model and the traffic simulation tool; the retrieval enhancement generation module connects the road network database and the multimodal large language model; and the text interpreter connects the multimodal large language model.

[0084] The process of generating an autonomous driving test scenario in this specific example and the functions of each part of the autonomous driving test scenario generation system are as follows:

[0085] Retrieval enhancement generation module: obtains the scenario generation request input by the user, searches the road network database according to the scenario generation request, obtains the relevant road network and road network text description, and forwards them to the multimodal large language model;

[0086] Prompt word generator: generates the first, second, third, and fourth small sample prompt words and sends them to the multimodal large language model, wherein the first small sample prompt words include the scene constraint conditions and the format requirements of the traffic simulation tool, the second small sample prompt words include the vehicle generation requirements, the third small sample prompt words include the scene description example, and the fourth small sample prompt words include the path requirements;

[0087] Multimodal large language model: Generates detailed descriptions based on the scenario generation request, road network file description, and the third small sample prompt word, and sends the detailed descriptions to the answer extractor;

[0088] Answer extractor: processes the detailed description, extracts key information, generates a scene description, and sends the scene description to the multimodal large language model;

[0089] Multimodal large language model: Generate node files and edge files based on the first small sample prompt words and scene description, and send them to the road network generator; Generate an initial vehicle model based on the second small sample prompt words and scene description, and send it to the vehicle generator;

[0090] Road network generator: sends node files and edge files to traffic simulation tools;

[0091] Traffic simulation tool: Generates a road network file based on the node file and edge file and returns it to the road network generator;

[0092] Vehicle Generator: Sends the initial vehicle model to the traffic simulation tool;

[0093] Traffic simulation tool: for each initial vehicle model, an initial path file is generated according to the departure and arrival points carried by the initial vehicle model;

[0094] Performance evaluator: sends evaluation commands to the traffic simulation tool;

[0095] Traffic simulation tool: after receiving the evaluation command, it makes a judgment based on the road network file and the initial path file. If there is an error in the initial path file, it sends a path error prompt message to the performance evaluator.

[0096] Performance evaluator: sends the received path error prompt information to the multimodal large language model;

[0097] Multimodal large language model: Generates a path correction file based on the path error prompt information, scene description and the fourth small sample prompt word;

[0098] Performance evaluator: obtains the path correction file and forwards it to the traffic simulation tool;

[0099] Traffic simulation artifacts: optimize and correct the initial path file according to the path correction file, obtain the corrected path file, make judgments based on the road network file and the corrected path file, and if there is an error in the corrected path file, continue to send path error prompt information to the performance evaluator for the next round of iterative optimization; if there is no error in the corrected path file, add the corrected path file as a path file to the initial vehicle model, generate a vehicle model, and send the vehicle model to the vehicle generator;

[0100] Scenario generation module: obtains the road network file from the road network generator, obtains the vehicle model from the vehicle generator, synthesizes the road network file and the vehicle model, and generates an autonomous driving test scenario.

[0101] In this specific example, for example, the scene generation request input by the user obtained by the retrieval enhancement generation module is: "Generate 5 scenes with forks in the road"; the answer extractor in the text interpreter can generate 5 scene descriptions, such as the first scene description is "the main road is divided into two at a small angle, forming a Y-shaped fork in the road", and the second scene description is "the fork in the road is divided into three forks by the two-way six-lane main road. The first fork extends from the main road to the left front, the second fork extends along the direction of the main road, and the third fork turns right from the main road."

[0102] In this embodiment, in order to further illustrate the autonomous driving test scenario generation framework and generation method of the embodiment of the present application, the following is a detailed description using a specific example 2. Figure 4 The autonomous driving test scenario generation system shown includes an autonomous driving test scenario generation framework, a multimodal large language model, and a traffic simulation tool:

[0103] In this specific example, the autonomous driving test scenario generation framework includes: a road network generator, a vehicle generator, a scenario generation module, a prompt word generator, and a performance evaluator; the road network generator, the vehicle generator, and the performance evaluator are bridges between the multimodal large language model and the traffic simulation tool; the prompt word generator connects the multimodal large language model.

[0104] The process of generating an autonomous driving test scenario in this specific example and the functions of each part of the autonomous driving test scenario generation system are as follows:

[0105] Multimodal large language model: Get collision reports from user input;

[0106] Prompt word generator: generates the first, second, and fourth small sample prompt words and sends them to the multimodal large language model, wherein the first small sample prompt words include the scene constraints and the format requirements of the traffic simulation tool, the second small sample prompt words include the vehicle generation requirements, and the fourth small sample prompt words include the path requirements;

[0107] Multimodal large language model: Use the collision report as a scene description, generate node files and edge files based on the first small sample prompt words and the collision report, and send them to the road network generator; generate an initial vehicle model based on the second small sample prompt words and the collision report, and send it to the vehicle generator;

[0108] Road network generator: sends node files and edge files to traffic simulation tools;

[0109] Traffic simulation tool: Generates a road network file based on the node file and edge file and returns it to the road network generator;

[0110] Vehicle Generator: Sends the initial vehicle model to the traffic simulation tool;

[0111] Traffic simulation tool: for each initial vehicle model, an initial path file is generated according to the departure and arrival points carried by the initial vehicle model;

[0112] Performance evaluator: sends evaluation commands to the traffic simulation tool;

[0113] Traffic simulation tool: after receiving the evaluation command, it makes a judgment based on the road network file and the initial path file. If there is an error in the initial path file, it sends a path error prompt message to the performance evaluator.

[0114] Performance evaluator: sends the received path error prompt information to the multimodal large language model;

[0115] Multimodal large language model: Generates path correction files based on path error prompt information, collision reports, and the fourth small sample prompt words;

[0116] Performance evaluator: obtains the path correction file and forwards it to the traffic simulation tool;

[0117] Traffic simulation artifacts: optimize and correct the initial path file according to the path correction file, obtain the corrected path file, make judgments based on the road network file and the corrected path file, and if there is an error in the corrected path file, continue to send path error prompt information to the performance evaluator for the next round of iterative optimization; if there is no error in the corrected path file, add the corrected path file as a path file to the initial vehicle model, generate a vehicle model, and send the vehicle model to the vehicle generator;

[0118] Scenario generation module: obtains the road network file from the road network generator, obtains the vehicle model from the vehicle generator, synthesizes the road network file and the vehicle model, and generates an autonomous driving test scenario.

[0119] In this specific example, the content of the collision report is as follows: "This three-vehicle collision occurred during the day at the intersection of two rural roads; the north-south lane is a two-lane road with no traffic control devices, including a north-south lane and a south-north lane, the road is flat, and the speed limit is 89 km / h; the east-west lane is an undivided two-way lane with a stop sign to control traffic flow, a road platform, and a speed limit of 56 km / h; vehicle 1 is a sedan, the vehicle is going straight from south to north, and vehicle 1 slows down to let vehicle 3 in front of it pass. Completed a right turn from south to east, then crossed the intersection and continued to drive north; Vehicle 2 intended to cross the intersection and drive north; Vehicle 2 also took braking measures when Vehicle 1 slowed down, but the deceleration was not enough and still hit the rear of Vehicle 1, causing Vehicle 1 to continue to move forward after being rear-ended, hitting the right rear of Vehicle 3 and knocking Vehicle 3 off the edge of the road, causing Vehicle 3 to partially stop on the dirt shoulder; Vehicle 1 partially left the lane, with the right tire stopping on the east edge of the road and the left tire stopping on the lane. "The autonomous driving test scenario generated based on this collision report is as follows Figure 5 shown.

[0120] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term "computer storage medium" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. An autonomous driving test scenario generation framework, characterized in that: include: A road network generator is used to obtain a node file and an edge file generated by a multimodal large language model based on a prompt word project and a scene description, and input the node file and the edge file into a traffic simulation tool to obtain a road network file synthesized by the traffic simulation tool; A vehicle generator, used to obtain a plurality of initial vehicle models generated by the multimodal large language model based on the prompt word project and the scenario description, and input each of the initial vehicle models into the traffic simulation tool, and obtain a vehicle model generated by the traffic simulation tool after adding a path file to each of the initial vehicle models, wherein each of the initial vehicle models includes a departure point, an arrival point, and a departure time; The scenario generation module is used to integrate the road network file and the multiple vehicle models to obtain an autonomous driving test scenario.

2. The autonomous driving test scenario generation framework according to claim 1, wherein: The node file and the edge file are generated by the multimodal large language model according to a pre-designed first small sample prompt word and the scene description, wherein the first small sample prompt word includes scene constraints and format requirements of the traffic simulation tool; The initial vehicle model is generated by the multimodal large language model according to a pre-designed second small sample prompt word and the scene description, wherein the second small sample prompt word includes a vehicle generation requirement; The vehicle model includes a vehicle model to be tested and a background vehicle model.

3. The autonomous driving test scenario generation framework according to claim 2, characterized in that: Also includes: The retrieval enhancement generation module is used to obtain a scene generation request input by a user, and perform a search in a pre-built road network database according to the scene generation request to obtain relevant road networks and corresponding road network text descriptions.

4. The autonomous driving test scenario generation framework according to claim 3, characterized in that: The road network database includes a correspondence between a road network and a text description of the road network, and the text description of the road network is generated by the multimodal large language model based on the road network.

5. The autonomous driving test scenario generation framework according to claim 3, characterized in that: Also includes: Text interpreter, including cue word generator and answer extractor; The prompt word generator is used to generate prompt words, the prompt words include the first small sample prompt words, the second small sample prompt words and the third small sample prompt words, and send the prompt words to the multimodal large language model; The answer extractor is used to obtain the detailed description generated by the multimodal large language model based on the scenario generation request, the road network text description and the third sample prompt word, extract information from the detailed description, generate the scenario description and send the scenario description to the multimodal large language model; The third small sample prompt words include scene description examples.

6. The autonomous driving test scenario generation framework according to claim 1, characterized in that: The scenario description includes a pre-acquired collision report.

7. The autonomous driving test scenario generation framework according to claim 1, characterized in that: Also includes: A performance evaluator is used to send an evaluation command to the traffic simulation tool, obtain the path error prompt information generated by the traffic simulation tool and send it to the multimodal large language model, obtain the path correction file generated by the multimodal large language model based on the prompt word project and the path error prompt information, and send the path correction file to the traffic simulation tool, iteratively optimize the initial path file, and generate the path file.

8. A method for generating an autonomous driving test scenario, characterized in that: include: Inputting the node file and the edge file into a traffic simulation tool to obtain a road network file synthesized by the traffic simulation tool; Inputting a plurality of initial vehicle models into the traffic simulation tool, and obtaining a vehicle model generated by the traffic simulation tool after adding a path file to each of the initial vehicle models; Integrate the road network file and the plurality of vehicle models to generate an autonomous driving test scenario; Among them, the node file, the edge file and multiple initial vehicle models are all generated by a multimodal large language model based on prompt word engineering and scene description; each of the initial vehicle models includes a departure point, an arrival point and a departure time.

9. The method for generating an autonomous driving test scenario according to claim 8, wherein: Before obtaining the vehicle model generated by the traffic simulation tool after adding the path file to each of the initial vehicle models, the method includes: The following operations are performed for each of the initial vehicle models: Instructing the traffic simulation tool to interact with the multimodal large language model, performing one or more iterative optimizations on the initial path file of the initial vehicle model, and generating the path file corresponding to the initial vehicle model; The initial path file is a straight line connecting the starting point and the arrival point of the initial vehicle model.

10. The method for generating an autonomous driving test scenario according to claim 9, wherein: Each iteration of the optimization includes the following steps: Sending an evaluation command to the traffic simulation tool to obtain an analysis result generated by the traffic simulation tool according to the road network file and the pre-correction path file, wherein the analysis result is used to determine whether there is an error in the pre-correction path file; If not, instructing the traffic simulation tool to use the pre-correction path file as the path file; If yes, receiving the path error prompt information sent by the traffic simulation tool, forwarding the path error prompt information to the multimodal large language model, obtaining a path correction file generated by the multimodal large language model based on the path error prompt information, the scenario description and the pre-designed fourth small sample prompt word, and forwarding the path correction file to the traffic simulation tool, instructing the traffic simulation tool to optimize the pre-correction path file according to the path correction file to generate a corrected path file, and using the corrected path file as the pre-correction path file for the next iterative optimization; The pre-correction path file during the first iterative optimization is the initial path file, and the fourth small sample prompt word includes path requirements.

Images