Simulation evaluation management method and related device

Abstract

The invention provides a simulation evaluation management method and a related device in the field of automatic driving. According to the technical scheme provided by the invention, the corresponding simulation scene and evaluation criterion can be automatically matched based on the attribute information of the to-be-evaluated automatic driving software to obtain the evaluation task, so that the efficiency of generating the evaluation task can be improved. Furthermore, the invention further provides a technical scheme for managing one or more of the simulation scene, the evaluation criterion, the evaluation object and the evaluation report, so that the automation degree of the simulation evaluation management process can be improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and particularly to a simulation evaluation management method and related devices. Background Art

[0002] Before the autonomous driving software is actually applied to a vehicle, it is necessary to perform simulation evaluation on the autonomous driving software to ensure that the performance such as the safety and reliability of the autonomous driving software meets the requirements.

[0003] Currently, when performing simulation evaluation on the autonomous driving software, a person manually selects appropriate scenarios and evaluation criteria according to the version of the autonomous driving software, and generates an evaluation task based on the selected scenarios and evaluation criteria, and performs simulation evaluation on the autonomous driving software to obtain an evaluation result.

[0004] However, the efficiency of the above simulation evaluation method is relatively low. Summary of the Invention

[0005] This application provides a simulation evaluation management method and related devices, which can improve the simulation evaluation efficiency of the autonomous driving software.

[0006] In a first aspect, this application provides a simulation evaluation management system, which includes a simulation scenario library, an evaluation criterion library, an evaluation object management module, and an evaluation task management module. The evaluation object management module is used to maintain the attribute information of multiple autonomous driving software.

[0007] The evaluation task management module is used to: receive a first operation instruction, where the first operation instruction indicates to perform simulation evaluation on a first autonomous driving software among multiple autonomous driving software; generate an evaluation task package based on the first operation instruction, and the evaluation task package includes a file indicating a first simulation scenario in the simulation scenario library, a first evaluation criterion in the evaluation criterion library, and the first autonomous driving software. The attribute information of the first simulation scenario matches the attribute information of the first autonomous driving software, and the attribute information of the first evaluation criterion matches the attribute information of the first autonomous driving software.

[0008] This simulation evaluation management system can automatically match corresponding evaluation criteria and simulation scenarios based on the autonomous driving software indicated by the first operation instruction, so as to generate an evaluation task. Compared with a person manually selecting evaluation criteria and simulation scenarios for the autonomous driving software to be evaluated, it can improve the generation efficiency of the evaluation task, and thus can improve the evaluation efficiency of the autonomous driving software.

[0009] It can be understood that the name of the evaluation task management module is only an example, and as long as it can execute or implement the above two operations, it should be included in the protection scope of the evaluation task management module of this application.

[0010] It can be understood that the present application does not limit the input method of the first operation instruction. For example, the first operation instruction can be an instruction input by a user through an operation on the user interface provided by the simulation evaluation management system, or the first operation instruction can be an instruction received from other systems, devices or apparatuses.

[0011] It can be understood that the first simulation scenario can include one or more simulation scenarios, and the first evaluation criterion can include one or more evaluation criteria.

[0012] It can be understood that the evaluation task package contains files indicating the first simulation scenario in the simulation scenario library, the first evaluation criterion in the evaluation criterion library, and the first autonomous driving software. It can be: the evaluation task package contains at least three types of files. One type of these three types of files is a file indicating the first simulation scenario, another type is a file indicating the first evaluation criterion, and another type is a file indicating the first autonomous driving software.

[0013] Alternatively, the evaluation task package contains files indicating the first simulation scenario in the simulation scenario library, the first evaluation criterion in the evaluation criterion library, and the first autonomous driving software. It can be: the file indicating the first simulation scenario, the file indicating the first evaluation criterion, and the file indicating the first autonomous driving software are the same, that is, the information indicating the first simulation scenario, the information indicating the first evaluation criterion, and the information indicating the first autonomous driving software are located in the same file.

[0014] When the evaluation task package contains at least three types of files, if the first simulation scenario includes multiple simulation scenarios, in some possible implementation manners, different simulation scenarios among these multiple simulation scenarios can be indicated by different files, that is, this type of file indicating the simulation scenario can include multiple files, and these multiple files correspond to the multiple simulation scenarios one by one; in some other possible implementation manners, these multiple simulation scenarios can be indicated by one file, that is, one file indicates multiple simulation scenarios, or in other words, one file contains information describing multiple simulation scenarios; in some other possible implementation manners, different types of simulation scenarios among these multiple simulation scenarios can be indicated by different files, that is, this type of file indicating the simulation scenario can include multiple files, and these multiple files correspond to multiple types of simulation scenarios one by one, each file indicates one or more simulation scenarios of the corresponding type, or in other words, the simulation scenarios of the same type are described by one file, and the simulation scenarios of different types are described by different files.

[0015] When there are at least three types of files in the evaluation task package, if the first evaluation criterion includes multiple evaluation criteria, in some possible implementation manners, different evaluation criteria among these multiple evaluation criteria can be indicated by different files, that is, this type of file indicating the evaluation criteria can include multiple files, and these multiple files correspond one by one to the multiple evaluation criteria; in some other possible implementation manners, these multiple evaluation criteria can be indicated by one file, that is, one file indicates multiple evaluation criteria, or in other words, a file contains information describing multiple evaluation criteria; in some other possible implementation manners, different types of evaluation criteria among these multiple evaluation criteria can be indicated by different files, that is, this type of file indicating the evaluation criteria can include multiple files, and these multiple files correspond one by one to multiple types of evaluation criteria, each file indicates one or more evaluation criteria of the corresponding type, or in other words, the evaluation criteria of the same type are described by one file, and the evaluation criteria of different types are described by different files.

[0016] It can be understood that the evaluation criteria, simulation scenarios, and attribute information matching the autonomous driving software to be evaluated in the evaluation task package can be one or multiple.

[0017] In some implementation manners, the evaluation criterion library can include at least one of the following evaluation criteria: traffic rule evaluation criterion, safety evaluation criterion, or comfort evaluation criterion. Or in other words, the types of evaluation criteria can include traffic rule type, safety evaluation type, or comfort evaluation type.

[0018] In some possible implementation manners, the first simulation scenario can include multiple simulation scenarios in the simulation scenario library whose attribute information matches the attribute information of the first autonomous driving software, or even all the simulation scenarios whose attribute information matches. This can improve the accuracy and reliability of the evaluation results of the autonomous driving software.

[0019] In some possible implementation manners, the first evaluation criterion can include multiple evaluation criteria in the evaluation criterion library whose attribute information matches the attribute information of the first autonomous driving software, or even all the evaluation criteria whose attribute information matches. This can improve the accuracy and reliability of the evaluation results of the autonomous driving software.

[0020] In some possible implementation manners, the attribute information of the first autonomous driving software can include ODD attribute information.

[0021] In some possible implementation manners, the simulation scenario library can have at least one of the following functions: query the simulation scenarios in the simulation scenario library, count the information of the simulation scenarios in the simulation scenario library, create a new simulation scenario in the simulation scenario library, delete the simulation scenarios in the simulation scenario library, or modify the attribute information of the simulation scenarios in the simulation scenario library.

[0022] It can be understood that the query in this application can include the meanings of viewing and / or retrieving.

[0023] It can be understood that querying the simulation scenarios in the simulation scenario library can include the following meanings: querying the simulation scenarios containing specified attribute information, querying the attribute information of specified simulation scenarios, or querying the scenario schematic diagrams of specified simulation scenarios. Among them, after obtaining the query results, the way of outputting the query results is not limited. For example, the simulation scenario or the attribute information of the simulation scenario can be displayed through the user interface, or a file indicating the simulation scenario can be output for the user to use or download, etc.

[0024] Statistical information of the simulation scenarios in the simulation scenario library can include at least one of the following meanings: counting the total number of simulation scenarios in the simulation scenario library, counting the number of simulation scenarios with specified attribute information in the simulation scenario library, or counting the number of attribute information of the simulation scenarios in the simulation scenario library.

[0025] Creating a new simulation scenario for the simulation scenario library can include the following meanings: adding the simulation scenario files that have been created or exist on other systems or devices to the simulation scenario library, or directly inputting the attribute information of the simulation scenario to create a simulation scenario.

[0026] Through this implementation method, the simulation scenarios in the simulation scenario library can be managed flexibly according to needs, thereby improving the accuracy of the evaluation tasks generated based on this simulation scenario library, further improving the accuracy of the evaluation results, and ultimately improving the reliability of the autonomous driving software to be evaluated.

[0027] In some possible implementation methods, the evaluation criterion library can have at least one of the following functions: querying the evaluation criteria in the evaluation criterion library, counting the information of the evaluation criteria in the evaluation criterion library, creating a new evaluation criterion for the simulation scenario library, deleting the evaluation criteria in the evaluation criterion library, or modifying the attribute information of the evaluation criteria in the evaluation criterion library.

[0028] It can be understood that querying the evaluation criteria in the evaluation criterion library can include the following meanings: querying one or more evaluation criteria of a specified type, querying the attribute information of a specified evaluation criterion, querying one or more evaluation criteria with specified attribute information, or querying the evaluation criteria of a specified type and specified attribute information. Among them, after obtaining the query results, the way of outputting the query results is not limited. For example, the query results can be displayed through the user interface, or a file indicating the query results can be output for the user to use or download, etc.

[0029] The information of the evaluation criteria in the statistical evaluation criteria library may include at least one of the following meanings: the total number of evaluation criteria in the statistical evaluation criteria library, the number of evaluation criteria of a specified type, the number of evaluation criteria in the statistical evaluation criteria library whose attribute information is the specified attribute information, or the number of attribute information of the evaluation criteria in the statistical evaluation criteria library.

[0030] To create a new evaluation criterion for the evaluation criteria library may include the following meanings: adding an evaluation criterion file that has been established or exists on other systems or devices to this evaluation criteria library, or directly inputting the attribute information of the evaluation criterion to create an evaluation criterion.

[0031] Through this implementation method, the evaluation criteria in the evaluation criteria library can be flexibly managed as needed, thereby improving the accuracy of the evaluation tasks generated based on this evaluation criteria library, further improving the accuracy of the evaluation results, and ultimately improving the reliability of the evaluated autonomous driving software.

[0032] In some possible implementation methods, the autonomous driving software in the evaluation object may be associated with a vehicle manufacturer and / or a vehicle model.

[0033] Among them, the autonomous driving software being associated with a vehicle model can be understood as: the autonomous driving software belongs to the vehicles of this vehicle model, or rather, this autonomous driving software can be applied to the vehicles of this vehicle model.

[0034] The vehicle model can be further associated with a vehicle manufacturer. The vehicle model being associated with a vehicle manufacturer can be understood as: the vehicles of this vehicle model are the vehicles of this vehicle manufacturer.

[0035] The autonomous driving software being associated with a vehicle manufacturer can be understood as: the autonomous driving software belongs to the vehicles of this vehicle manufacturer, or rather, this autonomous driving software can be applied to the vehicles of this vehicle manufacturer.

[0036] The autonomous driving software being associated with a vehicle manufacturer and a vehicle model can be understood as: the autonomous driving software belongs to the vehicles of this vehicle model produced by this vehicle manufacturer, or rather, this autonomous driving software can be applied to the vehicles of this vehicle model produced by this vehicle manufacturer.

[0037] Or rather, the autonomous driving software in the evaluation object can be divided according to the vehicle manufacturer and / or the vehicle model. For example, first divide different vehicle manufacturers, then divide different vehicle models for each vehicle manufacturer, and then divide different versions of autonomous driving software based on each vehicle model.

[0038] In this management method, when selecting the autonomous driving software to be evaluated from the evaluation object, the hierarchical relationship is clear, which helps users accurately specify the autonomous driving software to be evaluated, improves the selection efficiency of the autonomous driving software to be evaluated, thereby improving the generation efficiency of the evaluation tasks, and further improving the evaluation efficiency of the autonomous driving software.

[0039] In some possible implementation manners, the evaluation object management module may be used to perform at least one of the following operations: query an evaluation object, count evaluation objects, create a new evaluation object, delete an evaluation object, or modify the attribute information of an evaluation object.

[0040] It can be understood that the name of the evaluation object management module is only an example. As long as a module can perform or implement the two operations in this implementation manner, it should be included in the protection scope of the evaluation object management module of this application.

[0041] It can be understood that querying an evaluation object may include the following meanings: querying a specified vehicle manufacturer, querying a specified vehicle model, querying a specified vehicle model of a specified vehicle manufacturer, querying the attribute information of the autonomous driving software of a specified vehicle manufacturer, querying the attribute information of the autonomous driving software of a specified vehicle model, querying the attribute information of the autonomous driving software of a specified vehicle manufacturer and a specified vehicle model, querying the autonomous driving software with specified attribute information, querying the vehicle manufacturer and / or vehicle model indicated by the autonomous driving software with specified attributes, querying the autonomous driving software with specified vehicle manufacturer and / or specified vehicle model and specified attribute information. Among them, after the query result is obtained, the way of outputting the query result is not limited. For example, the query result can be displayed through the user interface, or a file containing the query result can be output for the user to use or download, etc.

[0042] Counting evaluation objects may include at least one of the following meanings: counting the total number of autonomous driving software in the evaluation objects, counting the number of vehicle manufacturers, counting the number of vehicle models of each vehicle manufacturer, counting the number of vehicle models of a specified vehicle manufacturer, counting the number of vehicle models, counting the number of autonomous driving software of each vehicle manufacturer, counting the number of autonomous driving software of each vehicle model, counting the number of autonomous driving software of a specified vehicle manufacturer, counting the number of autonomous driving software of a specified vehicle model, counting the number of autonomous driving software with specified attribute information in the evaluation objects, or counting the number of attribute information of the autonomous driving software in the evaluation objects.

[0043] Creating a new evaluation object may include at least one of the following meanings: adding a new vehicle manufacturer, adding a new vehicle model, adding new autonomous driving software, adding the attribute information of a vehicle manufacturer, adding the attribute information of a vehicle model, adding the attribute information of autonomous driving software.

[0044] Through this implementation manner, the evaluation objects can be managed flexibly according to needs, thereby improving the accuracy of the evaluation tasks generated based on the evaluation objects, further improving the accuracy of the evaluation results, and ultimately improving the reliability of the autonomous driving software to be evaluated.

[0045] In some possible implementation manners, the simulation evaluation management system further includes an evaluation report management module, and the evaluation report management module is used to perform at least one of the following operations: querying an evaluation report, or statistically analyzing an evaluation report.

[0046] It can be understood that the name of the evaluation report management module is only an example, and any module that can execute or implement the two operations in this implementation manner should be included in the protection scope of the evaluation report management module of this application.

[0047] It can be understood that querying the evaluation report can include at least one of the following meanings: querying the evaluation report of the autonomous driving software of a specified vehicle manufacturer, querying the evaluation report of the autonomous driving software of a specified vehicle model, querying the evaluation report of the autonomous driving software of a specified vehicle model of a specified vehicle manufacturer, querying the evaluation report of the autonomous driving software with specified attribute information, querying the evaluation report of a specified evaluation task, or querying a specified evaluation report. Among them, after obtaining the query result, the way of outputting the query result is not limited. For example, the evaluation report can be displayed through the user interface, or a file containing the evaluation report can be output for the user to use or download, etc.

[0048] It can be understood that statistical analysis of the evaluation report can include at least one of the following meanings: counting the number of simulation scenarios that pass and / or do not pass the evaluation in the simulation scenarios involved in the evaluation task corresponding to the evaluation report, counting the number of evaluation criteria that pass and / or do not pass the evaluation in the evaluation criteria involved in the evaluation task corresponding to the evaluation report, counting the number of evaluation criteria that pass and / or do not pass the evaluation in the traffic rule-based evaluation criteria involved in the evaluation task corresponding to the evaluation report, counting the number of evaluation criteria that pass and / or do not pass the evaluation in the safety-based evaluation criteria involved in the evaluation task corresponding to the evaluation report, or counting the number of evaluation criteria that pass and / or do not pass the evaluation in the comfort-based evaluation criteria involved in the evaluation task corresponding to the evaluation report.

[0049] Through this implementation manner, the evaluation object can be flexibly managed according to needs, thereby improving the accuracy of the evaluation task generated based on the evaluation object, further improving the accuracy of the evaluation result, and ultimately improving the reliability of the autonomous driving software to be evaluated.

[0050] In some possible implementation manners, the evaluation task management module can also be used to perform at least one of the following operations: querying the evaluation task package, or counting the information of the evaluation task.

[0051] It can be understood that counting the evaluation task can include at least one of the following meanings: counting the number of all simulation scenarios included in the evaluation task, counting the number of simulation scenarios of a specified type in the evaluation task, counting the number of simulation scenarios with specified attribute information in the evaluation task, counting the number of all evaluation criteria included in the evaluation task, counting the number of evaluation criteria of a specified type in the evaluation task, or counting the number of evaluation criteria with specified attribute information in the evaluation task.

[0052] It can be understood that viewing the evaluation tasks can include at least one of the following meanings: viewing all evaluation tasks, viewing specified evaluation tasks, viewing evaluation tasks including specified simulation scenarios, viewing evaluation tasks including specified evaluation criteria, viewing evaluation tasks including specified autonomous driving software, viewing evaluation tasks including autonomous driving software of specified vehicle manufacturers, viewing evaluation tasks including autonomous driving software of specified vehicle models, or viewing evaluation tasks including autonomous driving software with specified attribute information.

[0053] In a second aspect, the present application provides a simulation evaluation management method, which includes: receiving a first operation instruction, where the first operation instruction indicates to perform a simulation evaluation on a first autonomous driving software; generating an evaluation task package based on the first operation instruction, where the evaluation task package includes files indicating the attribute information of a first simulation scenario, a first evaluation criterion, and the first autonomous driving software, the attribute information of the first simulation scenario matches the attribute information of the first autonomous driving software, and the attribute information of the first evaluation criterion matches the attribute information of the first autonomous driving software.

[0054] It can be understood that this method can be implemented by a cloud service, or it can also be implemented on a side server or a side device. The execution entity of this method can be called a simulation evaluation management system.

[0055] In some possible implementation manners, the method may further include: receiving a second operation instruction, where the second operation instruction is used to indicate the following operations: querying the simulation scenarios in the simulation scenario library, counting the information of the simulation scenarios in the simulation scenario library, creating a new simulation scenario in the simulation scenario library, deleting the simulation scenarios in the simulation scenario library, or modifying the attribute information of the simulation scenarios in the simulation scenario library; performing the operations indicated by the second operation instruction.

[0056] In some possible implementation manners, the method may further include: receiving a third operation instruction, where the third operation instruction is used to indicate the following operations: querying the evaluation criteria in the evaluation criterion library, counting the information of the evaluation criteria in the evaluation criterion library, creating a new evaluation criterion in the simulation scenario library, deleting the evaluation criteria in the evaluation criterion library, or modifying the attribute information of the evaluation criteria in the evaluation criterion library; performing the operations indicated by the third operation instruction.

[0057] In some possible implementation manners, the method may further include: receiving a fourth operation instruction, where the fourth operation instruction is used to indicate the following operations: querying the autonomous driving software, deleting the autonomous driving software, querying the attribute information of the autonomous driving software, counting the attribute information of the autonomous driving software, creating a new autonomous driving software, deleting the attribute information of the autonomous driving software, or modifying the attribute information of the autonomous driving software.

[0058] In some possible implementations, the method may further include: receiving a fifth operation instruction for instructing the following operations: querying an evaluation report, or statistically analyzing an evaluation report; and performing the operations instructed by the fifth operation instruction.

[0059] For relevant concepts or steps in this method, reference may be made to the relevant content in the first aspect, which will not be elaborated here.

[0060] In a third aspect, the present application provides a simulation evaluation management device. The device may include a processor and a storage medium. The storage medium stores instructions, and when the instructions are run by the processor, the method in the second aspect or any possible implementation of the second aspect is implemented.

[0061] In a fourth aspect, a simulation evaluation management device is provided, including a processor for processing data and / or information so that the method in the second aspect or any possible implementation of the second aspect is implemented.

[0062] In one design, the device may be an end-side device, or a device, module, circuit, or chip configured in the end-side device, or a device that can be used in matching with the end-side device.

[0063] In one design, the device may be a cloud-side server, or a device, module, circuit, or chip configured in the cloud-side server, or a device that can be used in matching with the cloud-side server.

[0064] Optionally, the device may further include a memory for storing programs or instructions, and the processor is used to run the programs or instructions so that the method in the second aspect or any possible implementation of the second aspect is implemented.

[0065] Optionally, the device may further include the transceiver circuit, or the input / output interface.

[0066] In a fifth aspect, a chip is provided, including a processor for running programs or instructions so that the method in the second aspect or any possible implementation of the second aspect is implemented.

[0067] Optionally, the chip may further include a memory for storing programs or instructions. Optionally, the chip may further include the transceiver circuit, or the input / output interface.

[0068] In a sixth aspect, a computer-readable storage medium is provided. The computer-readable storage medium includes instructions, and when the instructions are run by the processor, the method in the second aspect or any possible implementation of the second aspect is implemented.

[0069] In a seventh aspect, there is provided a computer program product, which includes computer program code or instructions. When the computer program code or instructions are run, they cause the method in the second aspect or any possible implementation of the second aspect to be implemented. Description of the Drawings

[0070] Figure 1 Exemplary system architecture diagram of an embodiment of the present application;

[0071] Figure 2 Exemplary composition diagram of a simulation scenario library of an embodiment of the present application;

[0072] Figure 3 Exemplary diagram of a simulation scenario of an embodiment of the present application;

[0073] Figure 4 Exemplary composition diagram of an evaluation criterion library of an embodiment of the present application;

[0074] Figure 5 Exemplary composition diagram of an evaluation object of an embodiment of the present application;

[0075] Figure 6 Exemplary flowchart of a method for generating an evaluation task of an embodiment of the present application;

[0076] Figure 7 Structural schematic diagram of a simulation evaluation management device provided by an embodiment of the present application. Detailed Embodiments

[0077] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application.

[0078] To facilitate a clear description of the technical solutions in the embodiments of the present application, in the embodiments of the present application, terms such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and roles. Those skilled in the art can understand that the terms "first" and "second" do not limit the quantity and execution order, and the terms "first" and "second" do not necessarily mean different.

[0079] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0080] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects and indicates that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item)" or its similar expression refers to any combination of these items, including any combination of single item(s) or plural item(s). For example, at least one (item) of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0081] The application scenario of an embodiment of the present application is as follows: The autonomous driving software is evaluated based on a simulation scenario and evaluation criteria.

[0082] An exemplary architecture diagram of the autonomous driving simulation evaluation system of an embodiment of the present application is as Figure 1 shown. It can be seen from Figure 1 that the autonomous driving simulation evaluation system includes three parts, namely, an evaluation management system, an automated evaluation execution system, and a system. It can be understood that the autonomous driving simulation evaluation system in this embodiment can also be referred to as an autonomous driving simulation evaluation platform.

[0083] The simulation evaluation management system may include a simulation scenario library, an evaluation criteria library, an evaluation object management module, and an evaluation task management module. In some possible implementation manners, the simulation evaluation management system may further include an evaluation report management module.

[0084] An exemplary composition diagram of the simulation scenario library is as Figure 2 shown. Figure 2 The shown simulation scenario library may include one or more functional scenarios. A functional scenario may include one or more logical scenarios, and a logical scenario may include one or more specific scenarios.

[0085] The functional scenario is a scenario at an abstract level and can describe which entities and the relationships between entities in the scenario through natural language. An example of the attributes of the functional scenario is shown in Table 1.

[0086] Table 1 Attributes of the Functional Scenario

[0087] Functional scenario name The leading vehicle makes an emergency cut-out when encountering a stationary vehicle Functional scenario description When the host vehicle is following the leading vehicle, the leading vehicle makes an emergency cut-out when encountering a stationary vehicle Functional scenario classification Traffic rule evaluation scenario, safety evaluation scenario

[0088] The logical scenario can define road structure attributes and weather attributes, and can also define vehicle behaviors and vehicle parameters for generating specific scenarios. Among them, the road structure attributes are used to guide the generation of the simulation map. An example of the attributes of the logical scenario is shown in Table 2.

[0089] Attributes of the Logical Scenario in Table 2

[0090]

[0091]

[0092] Figure 3 Among them, V1 represents the speed of the first vehicle when sorted from left to right, D1 represents the distance between the first two vehicles when sorted from left to right, D2 represents the distance between the middle two vehicles when sorted from left to right, V2 represents the speed of the second vehicle when sorted from left to right, and T represents the time required for the second vehicle to change from the current lane to another lane when sorted from left to right.

[0093] A functional scenario can include the attributes of multiple logical scenarios. For example, different road types (straight roads or curves), and also different vehicle behaviors (overtaking on the left or overtaking on the right), etc.

[0094] The specific scenario can define the specific values of vehicle parameters, which are used to characterize the specific positions, speeds of entities, and the positional relationships between entities, etc., in order to guide the generation of vehicles in the simulation scenario. An example of the attributes of the specific scenario sub-elements is shown in Table 3.

[0095] Table 3 Attributes of the Specific Scenario

[0096]

[0097] A logical scenario includes the attributes of multiple specific scenarios, such as different vehicle distances, vehicle speeds, etc.

[0098] In this embodiment, the attributes of the functional scenario, logical scenario, and specific scenario can be regarded as a process of gradually refining the simulation scenario.

[0099] It can be understood that the hierarchical structure of the above simulation scenario library and the attributes of the scenarios at each level are only examples, and do not limit the composition of the simulation scenario library in this embodiment and the scope of the attributes of the simulation scenario.

[0100] An exemplary composition diagram of the evaluation criterion library in this embodiment is as Figure 4 shown. Figure 4 The shown evaluation criterion library can include three types of evaluation criteria, namely traffic rule evaluation criteria, safety evaluation criteria, and comfort evaluation criteria.

[0101] The traffic rule evaluation criteria include traffic rule sub-items, which are used to evaluate whether the behavior of the autonomous driving algorithm meets the requirements of traffic rules. An example of the attributes of the traffic rule evaluation criteria is shown in Table 4.

[0102] Table 4 Attributes of the Traffic Rule Evaluation Criteria

[0103]

[0104] The safety evaluation criteria can include a collision sub-item and a safety distance sub-item, which are used to evaluate the behavioral safety of the autonomous driving algorithm. The collision sub-item is the basic safety evaluation criterion. The safety time sub-item is the expected collision time between the host vehicle and other vehicles driving on the road in each scenario, serving as the safety evaluation criterion in the time dimension.

[0105] The comfort evaluation criteria can be divided into lateral and longitudinal accelerations and lateral and longitudinal decelerations, which are used to evaluate whether they meet the comfort requirements of the driver and passengers.

[0106] It can be understood that the hierarchical structure of the above evaluation criterion library and the attributes of each type of evaluation criterion are only examples, and do not limit the composition of the evaluation criterion library and the scope of the attributes of the evaluation criterion in this embodiment.

[0107] In this embodiment, an example composition diagram of the evaluation object is as Figure 5 shown. Figure 5 The evaluation object shown includes a vehicle manufacturer, a vehicle model, and autonomous driving software. In this embodiment, the autonomous driving software can also be referred to as the autonomous driving software version.

[0108] Among them, the evaluation object can include one or more vehicle manufacturers. Under each vehicle manufacturer, there can be one or more vehicle models. Under each vehicle model, there can be one or more evaluated or to-be-evaluated autonomous driving software. The multiple autonomous driving software included under one vehicle model can be different autonomous driving software versions.

[0109] The vehicle manufacturer is an automobile manufacturer. The vehicle models under the vehicle manufacturer are different models of autonomous driving vehicles produced by the vehicle manufacturer. The autonomous driving software version under the vehicle model is the version of the autonomous driving software of the vehicle model.

[0110] A vehicle manufacturer can have one or more attributes. An example of the attribute of a vehicle manufacturer is the name of the vehicle manufacturer. A vehicle model can have one or more attributes. An example of the attribute of a vehicle model is the name of the vehicle model.

[0111] The autonomous driving software version is the object of autonomous driving simulation evaluation. An autonomous driving software can have one or more attributes. For example, an autonomous driving software can have attributes such as an operational design domain (ODD) attribute, a version number, or a version name.

[0112] It can be understood that an autonomous driving software can have one or more ODD attributes. Examples of the ODD attributes of the autonomous driving software are shown in Tables 5 and 6.

[0113] Table 5 Examples of ODD Attributes of Autonomous Driving Software

[0114] Road type Highway Road structure Simple road Weather condition Sunny Lighting condition Daytime Leading vehicle condition None … …

[0115] Table 6 Example 2 of ODD Attributes of Autopilot Software

[0116] Road type Highway Road structure Simple road, ramp Weather condition Sunny Lighting condition Daytime Leading vehicle condition Yes … …

[0117] It can be understood that the hierarchical structure of the above evaluation objects and the attributes of the autopilot software are only examples, and do not limit the composition of the evaluation objects and the scope of the attributes of the autopilot software in this embodiment.

[0118] In some possible implementation manners, the evaluation object management module of this embodiment can be used to implement at least one of the following operations: query evaluation objects, create new evaluation objects, count the information of evaluation objects, delete evaluation objects, or modify the attributes of evaluation objects.

[0119] For example, vehicle model information, vehicle manufacturer information, or autopilot software version information can be created.

[0120] For another example, vehicle model information, vehicle manufacturer information, or autopilot software version information can be deleted.

[0121] For yet another example, new attributes can be added to, existing attributes can be deleted from, or the values of existing attributes can be modified for vehicle models, vehicle manufacturers, or autopilot software.

[0122] For yet another example, it is possible to query which vehicle manufacturers there are, query which vehicle models are under a specified vehicle manufacturer, query which autopilot software versions are under a specified vehicle manufacturer, query which autopilot software versions are under a specified vehicle model, query which attribute information is included in a specified autopilot software, etc.

[0123] As an example, the evaluation object management module can receive an instruction indicating which operation to perform, and execute the corresponding operation according to the indication of the instruction. For example, a user can input an instruction to delete a certain object in the evaluation object through a user interface. After the evaluation object management module receives this instruction, it deletes the corresponding object in the evaluation object.

[0124] In this embodiment, the evaluation task management module can screen out simulation scenarios that meet the ODD attributes of the autopilot software to be evaluated from the simulation scenario library and screen out evaluation criteria that meet the ODD attributes from the evaluation criterion library, so as to generate an evaluation task that includes the screened simulation scenarios, the screened evaluation criteria, and the autopilot software to be evaluated.

[0125] For example, the evaluation task management module reads all the simulation scenarios in the simulation scenario library, filters out the simulation scenarios that meet the ODD attributes from these simulation scenarios, and reads all the evaluation criteria in the evaluation criterion library, and filters out the evaluation criteria that meet the ODD attributes from these evaluation criterion libraries, so as to generate an evaluation task package including the filtered simulation scenarios, the filtered evaluation criteria, and the autonomous driving software to be evaluated. The evaluation task package in this embodiment can be simply referred to as an evaluation task.

[0126] As an example, the autonomous driving software to be evaluated can be the autonomous driving software selected or indicated by the operation instructions input by the user to the evaluation task management module. For example, the evaluation task management module can display the information in the evaluation object to the user through the user interface, and the autonomous driving software selected by the user's click is the autonomous driving software to be evaluated.

[0127] After the evaluation task management module determines the autonomous driving software to be evaluated, it obtains the attribute information of the autonomous driving software, such as obtaining the ODD attribute information of the autonomous driving software.

[0128] As an example, after the user indicates the autonomous driving software to be evaluated to the evaluation task management module, the evaluation task management module starts to execute the above operation of generating an evaluation task.

[0129] As another example, after the user indicates the autonomous driving software to be evaluated to the evaluation task management module, it is also necessary to further input an instruction to confirm the start of generating an evaluation task, and the evaluation task management module starts to execute the above operation of generating an evaluation task. An example of inputting an instruction to confirm the start of generating an evaluation task is: clicking the button to confirm the start of generating an evaluation task.

[0130] Taking the ODD attribute of the autonomous driving software to be evaluated as the ODD attribute in Table 6 as an example, the flowchart of the evaluation task management module generating an evaluation task is as Figure 6 shown.

[0131] For example, the evaluation task management module filters all simulation scenarios with the value of "highway" for the "road type" attribute from the simulation scenario library according to the value of "highway" for the "road type" attribute in the ODD attribute of the autonomous driving software to be evaluated. For the sake of simplicity in subsequent descriptions, the simulation scenarios filtered here are referred to as the first alternative simulation scenarios; according to the value of "simple road or ramp" for the "road structure" attribute in the ODD attribute of the autonomous driving software to be evaluated, it filters all simulation scenarios with the value of "simple road or ramp" for the "road structure" attribute from the first alternative simulation scenarios. For the sake of simplicity in subsequent descriptions, the simulation scenarios filtered here are referred to as the second alternative simulation scenarios; according to the value of "clear" for the "weather condition" attribute in the ODD attribute of the autonomous driving software to be evaluated, it filters all simulation scenarios with the value of "clear" for the "weather condition" attribute from the second alternative simulation scenarios. For the sake of simplicity in subsequent descriptions, the simulation scenarios filtered here are referred to as the third alternative simulation scenarios; according to the value of "daytime" for the "lighting condition" attribute in the ODD attribute of the autonomous driving software to be evaluated, it filters all simulation scenarios with the value of "daytime" for the "lighting condition" attribute from the third alternative simulation scenarios. For the sake of simplicity in subsequent descriptions, the simulation scenarios filtered here are referred to as the fourth alternative simulation scenarios; according to the value of "yes" for the "lead vehicle condition" attribute in the ODD attribute of the autonomous driving software to be evaluated, it filters all simulation scenarios with the value of "yes" for the "lead vehicle condition" attribute from the fourth alternative simulation scenarios. At this time, the simulation scenarios filtered here are the simulation scenarios that constitute the evaluation task, that is, the evaluation scenarios.

[0132] For example, the evaluation task management module filters all the evaluation criteria with the value of "highway" for the "road type" attribute from the evaluation criterion library according to the value of "highway" for the "road type" attribute in the ODD attribute of the autonomous driving software to be evaluated. For the sake of simplicity in subsequent descriptions, the evaluation criteria filtered here are referred to as the first alternative evaluation criteria; according to the value of "simple road or ramp" for the "road structure" attribute in the ODD attribute of the autonomous driving software to be evaluated, it filters all the evaluation criteria with the value of "simple road or ramp" for the "road structure" attribute from the first alternative evaluation criteria. For the sake of simplicity in subsequent descriptions, the evaluation criteria filtered here are referred to as the second alternative evaluation criteria; according to the value of "sunny" for the "weather condition" attribute in the ODD attribute of the autonomous driving software to be evaluated, it filters all the evaluation criteria with the value of "sunny" for the "weather condition" attribute from the second alternative evaluation criteria. For the sake of simplicity in subsequent descriptions, the evaluation criteria filtered here are referred to as the third alternative evaluation criteria; according to the value of "daytime" for the "lighting condition" attribute in the ODD attribute of the autonomous driving software to be evaluated, it filters all the evaluation criteria with the value of "daytime" for the "lighting condition" attribute from the third alternative evaluation criteria. For the sake of simplicity in subsequent descriptions, the evaluation criteria filtered here are referred to as the fourth alternative evaluation criteria; according to the value of "yes" for the "lead vehicle condition" attribute in the ODD attribute of the autonomous driving software to be evaluated, it filters all the evaluation criteria with the value of "yes" for the "lead vehicle condition" attribute from the fourth alternative evaluation criteria. At this time, the evaluation criteria filtered here are the evaluation criteria that make up the evaluation task.

[0133] After the evaluation task management module filters out the evaluation scenarios and evaluation criteria, it can generate an evaluation task package that includes files indicating these evaluation scenarios, evaluation criteria, and the autonomous driving software.

[0134] Furthermore, the evaluation task management module can send the evaluation task package to the evaluation automation execution system.

[0135] In this embodiment, after receiving the evaluation task package, the evaluation automation execution system starts the scheduling of the simulation platform and executes the evaluation of the autonomous driving software.

[0136] In this embodiment, the simulation platform can be composed of vehicle dynamics software, virtual scene simulation software, and the autonomous driving software to be evaluated. Among them, the vehicle dynamics software and the virtual scene simulation software respectively provide a virtual vehicle and a virtual environment for evaluating the autonomous driving software in a simulation manner.

[0137] The simulation platform is scheduled by the evaluation automation execution system to evaluate the autonomous driving software. For example, the vehicle dynamics simulation software can transmit the vehicle attitude to the virtual scene simulation software under the scheduling of the evaluation automation execution system. The virtual scene simulation software transmits the vehicle information and environmental information to the autonomous driving software to be evaluated under the scheduling of the evaluation automation execution system. The autonomous driving software to be evaluated transmits the vehicle control information to the vehicle dynamics simulation software under the scheduling of the evaluation automation execution system, and continuously loops the above process, so as to realize the evaluation of the autonomous driving software.

[0138] As an example, after the evaluation task management module issues an evaluation task to the evaluation automation execution system, if the evaluation automation execution system is in an idle state, the evaluation task is executed; if the evaluation automation execution system is in a non-idle state, the evaluation task can be suspended and then started to be executed after the evaluation automation execution system becomes idle.

[0139] After the evaluation automation execution system completes the evaluation task, it can generate an evaluation report from the evaluation results and upload it to the evaluation report management module of the simulation evaluation management system.

[0140] After receiving the evaluation report, the evaluation report management module can perform report retrieval, viewing, and statistical analysis for the evaluation object, or perform statistical analysis for the collection of all evaluation objects.

[0141] As an example, after the evaluation task is generated, the evaluation task management module can generate the report items corresponding to the evaluation task, and the evaluation status is recorded as unevaluated. And the evaluation task management module also issues the report items to the evaluation automation execution system.

[0142] After the evaluation automation execution system completes the evaluation, it can record the evaluation results in the report items to generate an evaluation report and upload it to the evaluation report management module of the simulation evaluation management system, and the evaluation status can be modified to evaluated.

[0143] Through the evaluation report management module, the report can be viewed, the passing or not passing of the autonomous driving software for each evaluation criterion can be statistically analyzed according to the evaluation scenario, and the specific driving data of the autonomous driving simulation vehicle can be viewed, etc.

[0144] The evaluation report can include at least one of the following: basic information, description of evaluation items, passing statistics of evaluation, or details of evaluation scenario results. An example of the content of the evaluation report is shown in Table 7.

[0145] Table 7 Content of the Evaluation Report

[0146]

[0147] It can be understood that the content of the above evaluation report is only an example and does not limit the scope of the content of the evaluation report in this embodiment.

[0148] In some possible implementation manners, the simulation scenario library of this embodiment may have at least one of the following functions: querying a simulation scenario, creating a new simulation scenario, deleting a simulation scenario, or modifying the attributes of a simulation scenario.

[0149] For example, a functional scenario, a logical scenario, or a specific scenario can be created. Also, a functional scenario, a logical scenario, or a specific scenario can be deleted. Additionally, new attributes can be added to, existing attributes can be deleted from, or the values of existing attributes can be modified for a functional scenario, a logical scenario, or a specific scenario.

[0150] As an example, the simulation scenario library can receive an instruction indicating which operation to perform and execute the corresponding operation according to the instruction. For example, a user can input an instruction to delete a certain simulation scenario in the simulation scenario library through a user interface. After the simulation scenario library receives the instruction, it deletes the simulation scenario in the simulation scenario library.

[0151] In some possible implementation manners, the evaluation criterion library of this embodiment may have at least one of the following operations: querying an evaluation criterion, creating a new evaluation criterion, deleting an evaluation criterion, or modifying the attributes of an evaluation criterion.

[0152] For example, a traffic rule - based criterion library, a safety - based criterion library, or a comfort - based criterion library can be created. Also, a traffic rule - based criterion library, a safety - based criterion library, or a comfort - based criterion library can be deleted. Additionally, new attributes can be added to, existing attributes can be deleted from, or the values of existing attributes can be modified for a traffic rule - based criterion library, a safety - based criterion library, or a comfort - based criterion library.

[0153] As an example, the evaluation criterion library can receive an instruction indicating which operation to perform and execute the corresponding operation according to the instruction. For example, a user can input an instruction to delete a certain evaluation criterion in the evaluation criterion library through a user interface. After the evaluation criterion library receives the instruction, it deletes the evaluation criterion in the evaluation criterion library.

[0154] In some possible implementation manners, the evaluation report management module can also perform at least one of the following operations: retrieving an evaluation report, viewing an evaluation report, performing statistical analysis on a certain evaluation object, and performing statistical analysis on the collection of all evaluation objects.

[0155] As an example, after the evaluation task management module generates an evaluation task, it can notify the evaluation report management module that a new evaluation task has been issued, and the status of the evaluation report of this evaluation task in the evaluation report management module can be displayed as pending submission.

[0156] As an example, after the evaluation automation execution system finishes an evaluation task, it can submit an evaluation report to the evaluation report management module. The status of the evaluation report of this evaluation task in the evaluation report management module can be displayed as submitted, and statistics and queries can be performed.

[0157] It can be understood that one or more of the simulation evaluation management system, the evaluation automation execution system, and the simulation system in this embodiment can be deployed on the same computing device, or can be separately deployed on different computing devices.

[0158] In some possible implementation manners, the simulation evaluation management system, the evaluation automation execution system, and the simulation system are separately deployed on different computing devices. For example, the simulation evaluation management system is deployed as a cloud-side service on a cloud-side computing device, the evaluation automation execution system can be deployed on an edge-side computing device, and the simulation system can be deployed on a cloud-side computing device where the cloud-side service is deployed or on an edge-side computing device.

[0159] Figure 7 This is a schematic structural diagram of a simulation evaluation management device provided in an embodiment of the present application. As Figure 7 shown, the device 700 includes a processor 701 and a communication circuit 702. The processor 701 and the communication circuit 702 are coupled to each other.

[0160] It can be understood that the communication circuit 702 can be a transceiver or an input / output interface.

[0161] Optionally, the device 700 may further include a memory 703, which is used to store instructions executed by the processor 701, or store input data required for the processor 701 to run instructions, or store data generated after the processor 701 runs instructions.

[0162] It can be understood that the memory 703 can be located outside the processor 701, or located inside the processor 701.

[0163] As an example, the processor 701 is used to implement the functions of the above-mentioned processing module 701, and the communication circuit 702 is used to implement the functions of the above-mentioned communication module 702.

[0164] As an example, the device 700 can be a computing device or a chip applied to a computing device.

[0165] Some embodiments of the present application also provide a computer-readable storage medium, which contains computer instructions. When the computer instructions run on a processor, they can implement the methods implemented by one or more modules in the simulation evaluation management system in any of the above embodiments.

[0166] In some embodiments of the present application, a simulation evaluation management system is further provided, which can implement the methods implemented by the simulation evaluation management system in any of the above embodiments.

[0167] It can be understood that the processor in the embodiments of the present application can be the following devices or all or part of the circuits for processing functions in the following devices: a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0168] The method steps in the embodiments of the present application can be implemented in a hardware manner or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, register, hard disk, removable hard disk, CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in the ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and the storage medium can also exist as discrete components in a network device or a terminal device.

[0169] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable devices. The computer program or instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it can also be an optical medium, such as a digital video disc; or it can be a semiconductor medium, such as a solid-state drive.

[0170] In various embodiments of the present application, if there is no special description and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other. The technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0171] It can be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. The magnitudes of the serial numbers of the above processes do not mean the order of execution, and the order of execution of each process should be determined by its function and internal logic.

Claims

1. A simulation evaluation management system, characterized in that, The simulation evaluation management system includes a simulation scenario library, an evaluation criterion library, an evaluation object management module, and an evaluation task management module. The evaluation object management module is used to maintain the attribute information of multiple autonomous driving softwares. The evaluation task management module is used for: Receiving a first operation instruction, where the first operation instruction indicates to perform a simulation evaluation on a first autonomous driving software among the multiple autonomous driving softwares; Generating an evaluation task package based on the first operation instruction. The evaluation task package contains files indicating a first simulation scenario in the simulation scenario library, a first evaluation criterion in the evaluation criterion library, and the first autonomous driving software. The attribute information of the first simulation scenario matches the attribute information of the first autonomous driving software, and the attribute information of the first evaluation criterion matches the attribute information of the first autonomous driving software.

2. The simulation evaluation management system according to claim 1, characterized in that, The attribute information of the first autonomous driving software includes operating design domain (ODD) attribute information.

3. The simulation evaluation management system according to claim 1 or 2, characterized in that, The simulation scenario library has at least one of the following functions: querying the simulation scenarios in the simulation scenario library, counting the information of the simulation scenarios in the simulation scenario library, creating a new simulation scenario in the simulation scenario library, deleting the simulation scenarios in the simulation scenario library, or modifying the attribute information of the simulation scenarios in the simulation scenario library.

4. The simulation evaluation management system according to any one of claims 1 to 3, characterized in that, The evaluation criterion library includes at least one of the following evaluation criteria: traffic rule evaluation criterion, safety evaluation criterion, or comfort evaluation criterion.

5. The simulation evaluation management system according to any one of claims 1 to 4, characterized in that, The evaluation criterion library has at least one of the following functions: querying the evaluation criteria in the evaluation criterion library, counting the information of the evaluation criteria in the evaluation criterion library, creating a new evaluation criterion in the evaluation criterion library, deleting the evaluation criteria in the evaluation criterion library, or modifying the attribute information of the evaluation criteria in the evaluation criterion library.

6. The simulation evaluation management system according to any one of claims 1 to 5, characterized in that, The first autonomous driving software corresponds to the first model of the first vehicle factory.

7. The simulation evaluation management system according to any one of claims 1 to 6, characterized in that, The evaluation object management module is used to maintain the attribute information of multiple autonomous driving softwares, including: The evaluation object management module is used to perform at least one of the following operations: querying an autonomous driving software, deleting an autonomous driving software, querying the attribute information of an autonomous driving software, counting the attribute information of an autonomous driving software, creating a new autonomous driving software, deleting the attribute information of an autonomous driving software, or modifying the attribute information of an autonomous driving software.

8. The simulation evaluation management system according to any one of claims 1 to 7, characterized in that, The simulation evaluation management system further includes an evaluation report management module. The evaluation report management module is used to perform at least one of the following operations: querying an evaluation report, or performing statistical analysis on an evaluation report.

9. A simulation evaluation management method, characterized in that, The method includes: Receiving a first operation instruction, where the first operation instruction indicates to perform a simulation evaluation on a first autonomous driving software; Generating an evaluation task package based on the first operation instruction. The evaluation task package contains files indicating the attribute information of a first simulation scenario, a first evaluation criterion, and a first autonomous driving software. The attribute information of the first simulation scenario matches the attribute information of the first autonomous driving software, and the attribute information of the first evaluation criterion matches the attribute information of the first autonomous driving software.

10. The method according to claim 9, characterized in that, The attribute information of the first autonomous driving software includes operating design domain (ODD) attribute information.

11. The method according to claim 9 or 10, characterized in that, The method further includes: Receive a second operation instruction, where the second operation instruction is used to indicate the following operations: query simulation scenarios in a simulation scenario library, count information of the simulation scenarios in the simulation scenario library, create a new simulation scenario in the simulation scenario library, delete a simulation scenario in the simulation scenario library, or modify attribute information of a simulation scenario in the simulation scenario library; Execute the operations indicated by the second operation instruction.

12. The method according to any one of claims 9 to 11, characterized in that, The method further includes: Receive a third operation instruction, where the third operation instruction is used to indicate the following operations: query evaluation criteria in an evaluation criteria library, count information of the evaluation criteria in the evaluation criteria library, create a new evaluation criteria in the evaluation criteria library, delete an evaluation criteria in the evaluation criteria library, or modify attribute information of an evaluation criteria in the evaluation criteria library; Execute the operations indicated by the third operation instruction.

13. The method according to claim 12, characterized in that, The evaluation criteria library includes at least one of the following evaluation criteria: traffic rule evaluation criteria, safety evaluation criteria, or comfort evaluation criteria.

14. The method according to any one of claims 9 to 13, characterized in that, The method further includes: Receive a fourth operation instruction, where the fourth operation instruction is used to indicate the following operations: query an autonomous driving software, delete the autonomous driving software, query attribute information of the autonomous driving software, count the attribute information of the autonomous driving software, create a new autonomous driving software, delete the attribute information of the autonomous driving software, or modify the attribute information of the autonomous driving software; Execute the operations indicated by the fourth operation instruction.

15. The method according to any one of claims 9 to 14, characterized in that, The first autonomous driving software corresponds to the first model of the first vehicle factory.

16. The method according to any one of claims 9 to 15, characterized in that, The method further includes: Receive a fifth operation instruction, where the fifth operation instruction is used to indicate the following operations: query an evaluation report, or perform statistical analysis on the evaluation report; Execute the operations indicated by the fifth operation instruction.

17. A simulation evaluation management device, characterized in that, Comprising: A processor and a memory; The memory stores computer instructions; The processor executes the computer instructions stored in the memory, so that the device executes the method according to any one of claims 9 to 16.

18. A computer-readable storage medium, characterized in that, A computer program or instruction is stored in the computer-readable storage medium, and when the computer program or instruction is run, the method according to any one of claims 9 to 16 is implemented.

Images