Multi-modal driving test scene generation method and system

Abstract

The invention discloses a multi-modal driving test scene generation method and system, and the method comprises the steps: obtaining a plurality of groups of model training samples, each group of model training samples comprising a user natural language description, a driving scene visual image, a risk point thermodynamic diagram, and a safety key scene description code; constructing a driving test scene generation framework, and training the driving test scene generation framework by using the multiple groups of model training samples to obtain a trained driving test scene generation framework; wherein the driving test scene generation framework comprises an image content coding model, a risk point mining model, a thermodynamic diagram coding model and a large language model, the risk point mining model and the thermodynamic diagram coding model are connected in series, and the output end of the image content coding model and the output end of the thermodynamic diagram coding model are connected with the input end of the large language model; and generating a multi-modal driving test scene according to the input user natural language description and the driving scene visual image by using the trained driving test scene generation architecture.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more specifically, to a method and system for generating a multimodal driving test scenario. Background Art

[0002] In autonomous driving technology, the generation of driving test scenarios is an important part of ensuring vehicle safety and system stability. At present, the generation of driving test scenarios mainly relies on manually written descriptions or model-based generation methods. In order to ensure the comprehensiveness and realism of the test scenarios, a large amount of scenario data and high-precision risk point identification are required in the generation process of driving test scenarios. For this reason, although the manually written descriptions in the prior art can provide highly customized scenarios, the manual description process is time-consuming and inefficient. Although the model-based generation method can automatically generate scenarios, it usually only relies on the user's natural language description, and it is difficult to accurately generate the details of the actual scenario, especially in the generation of safety-critical scenarios. There are great limitations. Summary of the invention

[0003] The present application provides a multimodal driving test scenario generation method and system thereof, which solves the problems in the prior art of low efficiency of manually written descriptions and significant limitations of model-based generation methods in generating safety-critical scenarios.

[0004] The specific technical solutions are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for generating a multimodal driving test scenario, comprising:

[0006] Acquire a number of driving scene visual images, and obtain multiple groups of model training samples based on the driving scene visual images and user natural language descriptions; wherein each group of model training samples includes the user natural language description, the driving scene visual image, the risk point heat map of the driving scene visual image, and the safety-critical scene description code of the driving scene visual image;

[0007] Constructing a driving test scenario generation architecture, and using multiple groups of the model training samples to train the driving test scenario generation architecture, to obtain the trained driving test scenario generation architecture; wherein the driving test scenario generation architecture includes an image content coding model, a risk point mining model, a heat map coding model and a large language model, the risk point mining model and the heat map coding model are connected in series, and the output ends of the image content coding model and the heat map coding model are respectively connected to the input end of the large language model, the inputs of the image content coding model and the risk point mining model are both the visual image of the driving scene, the output of the image content coding model is the image content feature, the output of the risk point mining model is the risk point heat map, the input of the heat map coding model is the risk point heat map, the output of the heat map coding model is the risk point location feature, the input of the large language model is the user's natural language description, the image content feature and the risk point location feature, and the output of the large language model is a safety-critical scenario description code;

[0008] The trained driving test scenario generation architecture is used to generate a multimodal driving test scenario according to the input user natural language description and the driving scenario visual image.

[0009] In some embodiments of the present application, the step of acquiring a plurality of driving scene visual images and obtaining a plurality of groups of model training samples according to the plurality of driving scene visual images and user natural language descriptions specifically includes:

[0010] Acquire a plurality of the driving scene visual images, mark the potential driving risk points in each of the driving scene visual images to obtain the risk point information, and generate the risk point heat map of the driving scene visual images according to the risk point information;

[0011] According to the user's natural language description, the driving scene visual image and the corresponding risk point information, and based on the expert-large language model, the safety-critical scene description code of the driving scene visual image is generated to obtain multiple groups of model training samples.

[0012] In some embodiments of the present application, the expert-large language model includes a trained large language model (Large Language Models, LLMs), and the safety-critical scene description code of the driving scene visual image is generated based on the expert-large language model according to the user's natural language description, the driving scene visual image and the corresponding risk point information, and specifically includes:

[0013] The user's natural language description, the driving scene visual image and the corresponding risk point information are input into the trained large language model, and an initial scene description code corresponding to the driving scene visual image is output. The initial scene description code is reviewed and modified by experts to generate the safety-critical scene description code.

[0014] In some embodiments of the present application, the plurality of driving scene visual images are driving scene images of different operating conditions, different times and different scene types.

[0015] In some embodiments of the present application, the use of multiple groups of the model training samples to train the driving test scenario generation framework to obtain the trained driving test scenario generation framework specifically includes:

[0016] Pre-training the image content coding model using the driving scene visual images in the plurality of groups of model training samples to obtain a preliminarily trained image content coding model;

[0017] Pre-training the risk point mining model using the driving scene visual images and the corresponding risk point heat maps in the multiple groups of the model training samples to obtain a risk point mining model after preliminary training;

[0018] Based on the image content coding model after preliminary training and the risk point mining model after preliminary training, the driving test scenario generation architecture is updated, and the updated driving test scenario generation architecture is end-to-end trained as a whole using multiple groups of model training samples to obtain the trained driving test scenario generation architecture.

[0019] In some embodiments of the present application, the risk point mining model is pre-trained using a supervised learning method.

[0020] In some embodiments of the present application, the image content encoding model is a Transformer-based visual image encoder.

[0021] In a second aspect, an embodiment of the present application provides a multi-modal driving test scenario generation system, including:

[0022] A sample data generation module is used to obtain a number of driving scene visual images, and obtain multiple groups of model training samples based on the driving scene visual images and user natural language descriptions; wherein each group of model training samples includes the user natural language description, the driving scene visual image, the risk point heat map of the driving scene visual image, and the safety-critical scene description code of the driving scene visual image;

[0023] An architecture construction module is used to construct a driving test scenario generation architecture, and use multiple groups of the model training samples to train the driving test scenario generation architecture to obtain the trained driving test scenario generation architecture; wherein the driving test scenario generation architecture includes an image content coding model, a risk point mining model, a heat map coding model and a large language model, the risk point mining model and the heat map coding model are connected in series, and the output ends of the image content coding model and the heat map coding model are respectively connected to the input end of the large language model, the inputs of the image content coding model and the risk point mining model are both the driving scene visual image, the output of the image content coding model is the image content feature, the output of the risk point mining model is the risk point heat map, the input of the heat map coding model is the risk point heat map, the output of the heat map coding model is the risk point location feature, the input of the large language model is the user's natural language description, the image content feature and the risk point location feature, and the output of the large language model is a safety-critical scenario description code;

[0024] The scenario generation module is used to generate a multimodal driving test scenario based on the input user natural language description and the driving scene visual image using the trained driving test scenario generation architecture.

[0025] In a third aspect, an embodiment of the present application provides a multimodal driving test scenario generation device, comprising: a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it executes the multimodal driving test scenario generation method as described in the first aspect.

[0026] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a multimodal driving test scenario as described in the first aspect is implemented.

[0027] The beneficial effects of the embodiments of the present application are as follows:

[0028] Through the driving test scenario generation architecture, multi-modal generation is performed, and the multi-scale scenario parsing model and heat map encoding network are used in combination to combine risk point information with natural language descriptions. The large language model is used to directly generate safety-critical scenarios that meet user needs, which significantly improves the automation and accuracy of the generation process. The results are more targeted and effective, with high efficiency, and can meet the needs of large-scale testing. In addition, the multi-modal driving test scenario generation method uses supervised learning methods to identify risk points and convert them into heat maps. The location information of risk points is extracted and encoded through the heat map encoding network, which effectively extracts and marks risk points in driving scenarios, further improving the accuracy of generated scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 A schematic diagram of a flow chart of a method for generating a multi-modal driving test scenario provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of generating a heat map of risk points in a multimodal driving test scenario generation method provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of a driving test scenario generation architecture provided in an embodiment of the present application;

[0033] Figure 4 A schematic block diagram of the composition of a multimodal driving test scenario generation system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The terms "including" and "having" in the embodiments of the present application and the accompanying drawings and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0036] In the prior art, the scenario generation method based on manually written descriptions needs to rely on engineers to manually write scenario descriptions according to actual needs. Although the scenario can be customized according to specific needs, the method of manually writing descriptions is not only inefficient but also prone to human errors due to the large amount of manual labor involved. Especially when the complexity of the scenario increases, it is more likely to produce omissions and errors, and cannot meet the needs of high-frequency and large-scale testing. The scenario generation method based on the model, for example, uses natural language processing technology to generate driving scenarios based on the natural language description input by the user. This method mainly focuses on the parsing of natural language and can only rely on the user's natural language description for generation. It lacks effective processing of the complexity of the actual driving scenario and cannot optimize specific risk points. It does not deal with specific risk points in complex scenarios. In addition, it is difficult to accurately generate according to the complexity and variability of the actual driving scenario by relying only on predefined rules or simple models. At the same time, the method in the prior art is difficult to directly generate test cases for safety-critical scenarios, especially in high-risk situations. It cannot guarantee that the generated scenario can cover all potential risk points. In response to the above problems, the embodiment of the present application discloses a multimodal driving test scenario generation method. The following are detailed descriptions.

[0037] Figure 1 A flow chart of a multimodal driving test scenario generation method provided in an embodiment of the present application. This embodiment can be applied to the case where a flowchart, for example, a sequential flowchart or a tree diagram is converted into a corresponding semantic text. The method can be executed by a multimodal driving test scenario generation device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal, a personal computer (PC) or a server.

[0038] It should be noted that the device for executing the multimodal driving test scene generation method provided in the embodiment of the present application can be integrated into the application software that supports the generation of multimodal driving test scenes, and the software can be installed in an electronic device. Optionally, the electronic device can be a mobile terminal or a PC, etc. The application software can be any type of software that requires the generation of multimodal driving test scenes. The specific application software will not be described one by one here, as long as the generation of multimodal driving test scenes can be achieved. It can also be a specially developed application program, which is used in the software that realizes the generation of multimodal driving test scenes, or it can be integrated in the corresponding page. The user can realize the generation of multimodal driving test scenes through the page integrated in the PC.

[0039] In an embodiment of the present application, the multimodal driving test scenario generation method includes the following steps:

[0040] Step S110: Acquire a number of driving scene visual images, and obtain multiple groups of model training samples based on the number of driving scene visual images and user natural language descriptions.

[0041] The driving scene of the embodiments of the present application can be a structured road scene or an unstructured road scene such as a scenic spot. The technical personnel in this field can flexibly set the specific application scenarios according to actual needs, and this application does not make any specific limitations on this.

[0042] In an embodiment of the present application, each group of model training samples includes a user's natural language description, a driving scene visual image, a risk point heat map corresponding to the driving scene visual image, and a safety-critical scene description code corresponding to the driving scene visual image. Specifically, the multimodal driving test scene generation method widely collects different driving scene visual images to obtain a number of driving scene visual images as model training samples, and annotates the potential driving risk points in each driving scene visual image to obtain the corresponding risk point information, and subsequently generates a corresponding risk point heat map based on the risk point information of the driving scene visual image, such as Figure 2 As shown. At the same time, for the visual images of driving scenes and their risk points, the corresponding safety-critical scene description codes are semi-automatically generated based on the user's natural language description and the expert-large language model. Therefore, after each driving scene visual image goes through the above production process, it is expanded into a four-tuple form of "user natural language description-driving scene visual image-risk point heat map-safety-critical scene description code", so that the model training samples are formed according to the aforementioned multiple groups of user natural language descriptions, driving scene visual images and their corresponding risk point heat maps, and safety-critical scene description codes.

[0043] In some specific embodiments, the expert-large language model includes a trained open source large language model (Large Language Models, LLMs). Furthermore, the semi-automatic generation process of the safety-critical scene description code of the driving scene visual image is: the user's natural language description, the driving scene visual image and its corresponding risk point information are input into the trained large language model, and the large language model performs preliminary generation of the corresponding scene description code, and outputs the initial scene description code corresponding to the driving scene visual image. After that, the initial scene description code is manually reviewed by expert technicians in related fields, and necessary modifications are made to finally generate the safety-critical scene description code, thereby efficiently generating high-quality data. This semi-automatic generation method is more efficient than the data manufacturing mode of purely manually written descriptions, and has higher accuracy and data quality than the model-based generation method.

[0044] In addition, the visual images of several driving scenes in the embodiments of the present application are driving scene images of different working conditions, different times and different scene types, so as to enhance the ability of the architecture to generate driving test scenes by collecting various driving scene images. Among them, the "working conditions" mentioned above include but are not limited to normal traffic conditions, congested traffic conditions, bad weather conditions, etc., "time" includes but is not limited to daytime, evening, night, etc., and "scene types" include but are not limited to urban roads, highways, complex intersections, etc.

[0045] Step S120: construct a driving test scenario generation framework, and use multiple groups of model training samples to train the driving test scenario generation framework to obtain a trained driving test scenario generation framework.

[0046] In the embodiments of the present application, Figure 3 As shown, the driving test scenario generation architecture 200 includes an image content encoding model 210, a risk point mining model 220, a heat map encoding model 230 and a large language model 240. The image content encoding model 210 is used for visual image information extraction. The risk point mining model 220 and the heat map encoding model 230 are connected in series for driving scenario risk point mining and location information encoding. The large language model 240 arranged downstream is used for the generation of multimodal safety-critical scenarios. Specifically, the output ends of the image content encoding model 210 and the heat map encoding model 230 are respectively connected to the input ends of the large language model 240, the inputs of the image content encoding model 210 and the risk point mining model 220 are both visual images of driving scenes, the output of the image content encoding model 210 is image content features, the output of the risk point mining model 220 is a risk point heat map, the input of the heat map encoding model 230 is a risk point heat map, the output of the heat map encoding model 230 is a risk point location feature, the input of the large language model 240 is a user's natural language description, image content features, and risk point location features, and the output of the large language model 240 is a safety-critical scene description code. Among them, the driving test scene generation architecture 200 is provided with a heat map encoding model 230, which is used to encode and extract risk point information in the driving scene, and send the risk point information (especially the location information) to the downstream node, so that the subsequent generation of safety-critical scenes in the architecture is more targeted.

[0047] In some embodiments, the image content encoding model 210 is a mainstream Transformer-based visual image encoder in the industry, which extracts the content in the current driving scene visual image and ensures the internal relative position relationship of the extracted image content features, so as to facilitate the generation of downstream safety-critical scene description codes. At the same time, in parallel with the image content encoding model 210, the visual image information is extracted, and the current driving scene visual image is sent to the series structure of "risk point mining model 220-heat map encoding model 230", and finally the risk point position feature corresponding to the current driving scene visual image is obtained, so that the mining and position information encoding of the driving scene risk point are automatically completed through the risk point mining model 220 and the heat map encoding model 230. Afterwards, the obtained image content features, risk point position features and the user's natural language description input by the user are jointly sent to the large language model 240 downstream of the architecture, so that the large language model 240 is used to generate a safety-critical scene description code for the current driving scene and satisfying the user's natural language description, which is used for the generation of multimodal driving test scenes.

[0048] In other embodiments, when performing architecture training, the multimodal driving test scene generation method first pre-trains the image content encoding model 210 and the risk point mining model 220 independently, and then performs end-to-end training on the overall architecture. Specifically, the image content encoding model 210 is pre-trained using multiple groups of driving scene visual images in the model training samples to obtain the image content encoding model 210 after preliminary training. At the same time, the risk point mining model 220 is pre-trained using multiple groups of driving scene visual images and their corresponding risk point heat maps in the model training samples to obtain the risk point mining model 220 after preliminary training. In detail, based on multiple groups of paired data of "driving scene visual image-risk point heat map" in the sample data, the risk point mining model 220 is trained using a supervised learning method so that the network can directly output the risk point heat map corresponding to the current driving scene. Afterwards, based on the initially trained image content coding model 210 and the initially trained risk point mining model 220, the driving test scenario generation architecture 200 is updated, and using multiple sets of model training samples' "driving scene visual images, user natural language descriptions - safety-critical scene description codes" paired data, the updated driving test scenario generation architecture 200 is end-to-end trained as a whole using a supervised learning method, so that each module in the architecture forms an organic whole, and the trained driving test scenario generation architecture 200 is obtained.

[0049] Step S130: using the trained driving test scenario generation framework, a multimodal driving test scenario is generated according to the input user natural language description and the driving scenario visual image.

[0050] The multimodal driving test scenario generation method in the embodiment of the present application is based on a driving test scenario generation architecture of a large language model, and comprehensively uses user natural language descriptions, a multi-scale scenario parsing model, and a heat map encoding network to directly generate safety-critical scenarios for given driving scenarios, thereby overcoming the deficiencies of existing model generation methods in risk point processing and the limitations of existing technologies in generating safety-critical scenarios, ensuring that potential risk points can be fully covered, significantly improving the pertinence and effectiveness of driving scenario generation, and using the trained driving test scenario generation architecture to directly generate multimodal driving test scenarios, thereby improving the efficiency of driving test scenario generation, overcoming the inefficiency of existing manual writing methods, and being able to meet the needs of large-scale testing.

[0051] Corresponding to the above method embodiment, the present application embodiment also provides a multi-modal driving test scenario generation system, which is used to execute the multi-modal driving test scenario generation method steps in the above embodiment. Figure 4 As shown, the multimodal driving test scenario generation system 300 includes: a sample data generation module 310 , an architecture construction module 320 and a scenario generation module 330 .

[0052] Specifically, the sample data generation module 310 is used to obtain a number of driving scene visual images, and obtain multiple groups of model training samples based on the driving scene visual images and user natural language descriptions. Each group of model training samples includes user natural language descriptions, driving scene visual images, risk point heat maps of driving scene visual images, and safety-critical scene description codes of driving scene visual images.

[0053] The architecture building module 320 is used to build a driving test scenario generation architecture, and train the driving test scenario generation architecture using multiple sets of model training samples to obtain a trained driving test scenario generation architecture. Figure 3 As shown, the driving test scenario generation architecture 200 includes an image content coding model 210, a risk point mining model 220, a heat map coding model 230 and a large language model 240. The risk point mining model 220 and the heat map coding model 230 are connected in series, and the output ends of the image content coding model 210 and the heat map coding model 230 are respectively connected to the input ends of the large language model 240. The inputs of the image content coding model 210 and the risk point mining model 220 are both visual images of driving scenes, the output of the image content coding model 210 is image content features, the output of the risk point mining model 220 is a risk point heat map, the input of the heat map coding model 230 is a risk point heat map, the output of the heat map coding model 230 is a risk point location feature, the input of the large language model 240 is a user's natural language description, image content features and risk point location features, and the output of the large language model 240 is a safety-critical scenario description code.

[0054] The scenario generation module 330 is used to generate a multimodal driving test scenario based on the input user natural language description and the driving scene visual image using the trained driving test scenario generation architecture.

[0055] It should be noted that the multimodal driving test scenario generation system provided in the embodiment of the present application is based on the same concept as the multimodal driving test scenario generation method embodiment of the present application, and the technical effect it brings is the same as the multimodal driving test scenario generation method embodiment of the present application. Therefore, the specific content can be found in the description in the multimodal driving test scenario generation method embodiment of the present application, and will not be repeated here.

[0056] The embodiment of the present application also provides a multimodal driving test scenario generation device, which includes: a processor, a memory, and a computer program stored in the memory, and when the processor executes the computer program, the steps in the above multimodal driving test scenario generation method embodiment are implemented. Alternatively, when the processor executes the computer program, the functions of each module in the above multimodal driving test scenario generation system embodiment are implemented.

[0057] In a specific embodiment, the computer program can be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to complete the embodiment of the present application. One or more modules can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the multimodal driving test scenario generation device. For example, the computer program can be divided into a sample data generation module, an architecture construction module, and a scenario generation module, and the specific functions of each module are as follows:

[0058] The sample data generation module is used to obtain a number of driving scene visual images, and obtain multiple groups of model training samples based on the number of driving scene visual images and user natural language descriptions.

[0059] The architecture building module is used to build a driving test scenario generation architecture, and use multiple groups of model training samples to train the driving test scenario generation architecture to obtain the trained driving test scenario generation architecture.

[0060] The scenario generation module is used to utilize the trained driving test scenario generation architecture to generate a multimodal driving test scenario based on the input user natural language description and driving scene visual images.

[0061] The multimodal driving test scenario generation device may be a computing device such as an on-board computer, a cloud management server, etc. Those skilled in the art may understand that the multimodal driving test scenario generation device may include but is not limited to a processor and a memory, and may also include more or fewer components, or a combination of certain components, or different components. For example, the multimodal driving test scenario generation device may also include input and output devices, network access devices, buses, etc.

[0062] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0063] The memory may be an internal storage unit of the multimodal driving test scenario generating device, for example, a hard disk or memory of the multimodal driving test scenario generating device. The memory may also be an external storage device of the multimodal driving test scenario generating device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital Card, referred to as SD card), a flash card (FlashCard), etc. equipped on the multimodal driving test scenario generating device. Further, the memory may also include both an internal storage unit of the multimodal driving test scenario generating device and an external storage device. The memory is used to store computer programs and other programs or data required by the multimodal driving test scenario generating device. The memory may also be used to temporarily store data that has been output or is to be output.

[0064] In the above embodiments, the description of each embodiment has different emphases. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0065] Those skilled in the art will appreciate that the modules and algorithm steps of the various embodiments described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but this implementation should not be considered to exceed the scope of the present invention.

[0066] In addition, the embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program can be executed by a processor to complete the multimodal driving test scenario generation method as described in the above embodiment. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), etc.

[0067] In summary, the present application discloses a multimodal driving test scenario generation method and system thereof, which performs multimodal generation through a driving test scenario generation architecture, comprehensively uses a multi-scale scenario parsing model and a heat map encoding network, combines risk point information with natural language descriptions, and uses a large language model to directly generate safety-critical scenarios that meet user needs, significantly improving the automation and accuracy of the generation process, and the results are more targeted and effective, with high efficiency, which can meet the needs of large-scale testing. In addition, the multimodal driving test scenario generation method uses a supervised learning method to identify risk points and convert them into heat maps, extracts and encodes the location information of risk points through a heat map encoding network, effectively extracts and marks risk points in driving scenarios, and further improves the accuracy of generated scenarios.

[0068] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of one embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present application.

[0069] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating a multi-modal driving test scenario, characterized in that: include: Acquire a number of driving scene visual images, and obtain multiple groups of model training samples based on the driving scene visual images and user natural language descriptions; wherein each group of model training samples includes the user natural language description, the driving scene visual image, the risk point heat map of the driving scene visual image, and the safety-critical scene description code of the driving scene visual image; Constructing a driving test scenario generation architecture, and using multiple groups of the model training samples to train the driving test scenario generation architecture, to obtain the trained driving test scenario generation architecture; wherein the driving test scenario generation architecture includes an image content coding model, a risk point mining model, a heat map coding model and a large language model, the risk point mining model and the heat map coding model are connected in series, and the output ends of the image content coding model and the heat map coding model are respectively connected to the input end of the large language model, the inputs of the image content coding model and the risk point mining model are both the visual image of the driving scene, the output of the image content coding model is the image content feature, the output of the risk point mining model is the risk point heat map, the input of the heat map coding model is the risk point heat map, the output of the heat map coding model is the risk point location feature, the input of the large language model is the user's natural language description, the image content feature and the risk point location feature, and the output of the large language model is a safety-critical scenario description code; The trained driving test scenario generation architecture is used to generate a multimodal driving test scenario according to the input user natural language description and the driving scenario visual image.

2. The method for generating a multimodal driving test scenario according to claim 1, characterized in that: The step of acquiring a plurality of driving scene visual images and obtaining a plurality of groups of model training samples according to the plurality of driving scene visual images and user natural language descriptions specifically includes: Acquire a plurality of the driving scene visual images, mark the potential driving risk points in each of the driving scene visual images to obtain the risk point information, and generate the risk point heat map of the driving scene visual images according to the risk point information; According to the user's natural language description, the driving scene visual image and the corresponding risk point information, and based on the expert-large language model, the safety-critical scene description code of the driving scene visual image is generated to obtain multiple groups of model training samples.

3. The method for generating a multi-modal driving test scenario according to claim 2, characterized in that: The expert-large language model includes a trained large language model, and the safety-critical scene description code of the driving scene visual image is generated based on the expert-large language model according to the user's natural language description, the driving scene visual image and the corresponding risk point information, and specifically includes: The user's natural language description, the driving scene visual image and the corresponding risk point information are input into the trained large language model, and an initial scene description code corresponding to the driving scene visual image is output. The initial scene description code is reviewed and modified by experts to generate the safety-critical scene description code.

4. The method for generating a multi-modal driving test scenario according to claim 2, characterized in that: The plurality of driving scene visual images are driving scene images of different working conditions, different times and different scene types.

5. The method for generating a multi-modal driving test scenario according to claim 1, characterized in that: The using multiple groups of the model training samples to train the driving test scenario generation framework to obtain the trained driving test scenario generation framework specifically includes: Pre-training the image content coding model using the driving scene visual images in the plurality of groups of model training samples to obtain a preliminarily trained image content coding model; Pre-training the risk point mining model using the driving scene visual images and the corresponding risk point heat maps in the multiple groups of the model training samples to obtain a risk point mining model after preliminary training; Based on the image content coding model after preliminary training and the risk point mining model after preliminary training, the driving test scenario generation architecture is updated, and the updated driving test scenario generation architecture is end-to-end trained as a whole using multiple groups of model training samples to obtain the trained driving test scenario generation architecture.

6. The method for generating a multi-modal driving test scenario according to claim 5, characterized in that: The risk point mining model is pre-trained using a supervised learning method.

7. The method for generating a multi-modal driving test scenario according to claim 1, characterized in that: The image content coding model is a Transformer-based visual image encoder.

8. A multi-modal driving test scenario generation system, characterized in that: include: A sample data generation module is used to obtain a number of driving scene visual images, and obtain multiple groups of model training samples based on the driving scene visual images and user natural language descriptions; wherein each group of model training samples includes the user natural language description, the driving scene visual image, the risk point heat map of the driving scene visual image, and the safety-critical scene description code of the driving scene visual image; An architecture construction module is used to construct a driving test scenario generation architecture, and use multiple groups of the model training samples to train the driving test scenario generation architecture to obtain the trained driving test scenario generation architecture; wherein the driving test scenario generation architecture includes an image content coding model, a risk point mining model, a heat map coding model and a large language model, the risk point mining model and the heat map coding model are connected in series, and the output ends of the image content coding model and the heat map coding model are respectively connected to the input end of the large language model, the inputs of the image content coding model and the risk point mining model are both the driving scene visual image, the output of the image content coding model is the image content feature, the output of the risk point mining model is the risk point heat map, the input of the heat map coding model is the risk point heat map, the output of the heat map coding model is the risk point location feature, the input of the large language model is the user's natural language description, the image content feature and the risk point location feature, and the output of the large language model is a safety-critical scenario description code; The scenario generation module is used to generate a multimodal driving test scenario based on the input user natural language description and the driving scene visual image using the trained driving test scenario generation architecture.

9. A multi-modal driving test scenario generating device, characterized in that: include: A processor, a memory, and a computer program stored in the memory, wherein when the processor executes the computer program, the method for generating a multimodal driving test scenario as described in any one of claims 1 to 7 is executed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal driving test scenario generation method as described in any one of claims 1-7 is implemented.

Images