Systems, methods, and storage media for validating and optimizing intelligent testing algorithms

Abstract

The application provides a system, method and storage medium for verifying and optimizing an intelligent test algorithm, and relates to the technical field of automatic driving. The system for verifying and optimizing the intelligent test algorithm comprises an input module configured to generate a state space file and a score space file based on acquired test task information; a simulation module connected to the input module and configured to parse the received state space file to obtain sampling information, and determine score information based on the state space file and the received score space file; an intelligent test algorithm module connected to the simulation module and configured to determine attribute sequence values based on the sampling information, and determine a target test sample set based on the sampling information and the score information; and a tool module connected to the intelligent test algorithm module and configured to construct a sample-score distribution graph based on the received target test sample set. The application can reduce the hardware configuration requirements of a joint simulation platform and the difficulty of constructing an intelligent test algorithm.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a system, method, and storage medium for verifying and optimizing intelligent test algorithms. Background Technology

[0002] Autonomous driving systems have made significant strides in recent years thanks to advancements in machine learning-based sensing and decision-making algorithms. A key challenge for their large-scale real-world deployment is the evaluation of their safety, performance, and other metrics. Most existing driving systems are still trained and evaluated on natural scenarios collected from everyday life or heuristically generated adversarial scenarios. However, given the scarcity of critical scenarios in collected real-world data, methods for generating critical scenarios through intelligent testing algorithms are crucial for measuring risk and reducing costs.

[0003] Existing development, testing, and verification frameworks for autonomous driving technology often employ co-simulation platforms that integrate high-fidelity vehicle dynamics, sensor, and environmental modeling. These platforms test high-fidelity autonomous driving systems in realistic Natural Driving Environment (NDE) simulations. Examples include Autoware, the first "all-in-one" open-source software for autonomous vehicles; Apollo; and Driveworks. These systems typically have built-in and compatibility with high-fidelity simulators, such as CarCraft and SurfelGAN used by Waymo, Webviz and the Matrix used by Waymo, and DataViz used by Uber. While most of these are proprietary tools, many open-source simulators are also available, such as MATLAB / Simulink, CarSim, PreScan, Gazebo, CARLA, and LGSVL.

[0004] However, in existing co-simulation platforms, the high-fidelity simulator has a large number of elements available for sampling, resulting in an excessively high dimensionality of the entire state space. This can further exacerbate the expansion of the state space due to the excessive fineness of the scale in a single dimension, increasing the difficulty of constructing intelligent testing algorithms, requiring high-end hardware configurations, and resulting in long simulation times per run. Summary of the Invention

[0005] To reduce the hardware configuration requirements of the co-simulation platform, lower the difficulty of constructing intelligent testing algorithms, and improve simulation efficiency, this application provides a system, method, and storage medium for verifying and optimizing intelligent testing algorithms.

[0006] This application provides a system for verifying and optimizing intelligent testing algorithms, employing the following technical solution:

[0007] A method for verifying and optimizing intelligent testing algorithms includes: an input module for generating a state space file and a score space file based on acquired test task information; a simulation module connected to the input module for parsing the received state space file to obtain sampling information, and determining score information based on the state space file and the received score space file; an intelligent testing algorithm module connected to the simulation module for determining attribute sequence values ​​based on the sampling information, and determining a target test sample set based on the sampling information and score information; and a tool module connected to the intelligent testing algorithm module for constructing a sample-score distribution map based on the received target test sample set.

[0008] According to some embodiments, the above-mentioned simulation module includes: a task file parsing unit, connected to the input module, for parsing the received state space file to obtain sampling information and outputting a score space file; a scenario generation unit, connected to the intelligent testing algorithm module, for generating a scenario file based on the received attribute sequence values; a simulation management unit, connected to the scenario generation unit, for performing simulation based on the received scenario file to obtain simulation results; and a scorer, connected to the task file parsing unit and the simulation management unit, for receiving the score space file and the simulation results, and determining score information based on the score space file and the simulation results.

[0009] According to some embodiments, the above-mentioned simulation management unit includes at least one simulation engine and at least one unmanned system, wherein there is a correspondence between the at least one simulation engine and the at least one unmanned system; the at least one unmanned system includes an execution algorithm; the at least one simulation engine is used to construct a simulation environment based on a scenario file, and to execute a simulation process based on the execution algorithm and the simulation environment to obtain simulation results.

[0010] According to some embodiments, the above-mentioned intelligent testing algorithm module includes: an intelligent testing algorithm unit connected to the simulation module, used to receive sampling information, determine attribute sequence values ​​based on the sampling information, and determine a test sample set based on the sampling information and score information; and a test sample construction unit connected to the intelligent testing algorithm unit, used to filter out a target test sample set from the test sample set.

[0011] According to some embodiments, the above-mentioned tool module includes: a plotting tool unit connected to the intelligent testing algorithm module, used to receive a target test sample set and construct a sample-score distribution map based on the target test sample set; and a state space traversal tool unit used to provide a traversal algorithm for the plotting tool unit and to perform plotting sample sampling on the target test sample set based on the traversal algorithm.

[0012] According to some embodiments, the state space file mentioned above includes environment description information, carrier description information, task element description information, and collision rule description information; the score space file includes performance index information and the weight information corresponding to the performance index information.

[0013] According to some embodiments, the above-mentioned environmental description information includes base environment information and attribute information corresponding to the base environment; the carrier description information includes information on visualization resources, collision shape and mounted equipment, as well as the algorithms and parameters used to indicate the carrier's perception, planning and decision-making behavior.

[0014] This application provides a method for verifying and optimizing intelligent testing algorithms, employing the following technical solution:

[0015] A method for verifying and optimizing intelligent testing algorithms, executed by the aforementioned system for verifying and optimizing intelligent testing algorithms, includes: an input module generating a state space file and a score space file based on acquired test task information; a simulation module parsing the received state space file to obtain sampling information, and determining score information based on the state space file and the received score space file; an intelligent testing algorithm module determining attribute sequence values ​​based on the sampling information, and determining a target test sample set based on the sampling information and the score information; and a tool module constructing a sample-score distribution map based on the received target test sample set.

[0016] According to some embodiments, the above-mentioned tool module constructs a sample-score distribution map based on the received target test sample set, including: when the intelligent test algorithm included in the intelligent test algorithm module is a cyclic iterative test algorithm, the sequence value output by the intelligent test algorithm unit in the intelligent test algorithm module at the current cyclic iteration round is input to the simulation module, the simulation module performs the next round of simulation based on the sequence value, and the intelligent test module outputs the target test sample set corresponding to the current cyclic iteration round, and constructs a sample-score distribution map based on the target test sample set; when the intelligent test algorithm included in the intelligent test algorithm module is a non-cyclic test algorithm, the sample-score distribution map is constructed based on the target test sample set.

[0017] This application provides a computer-readable storage medium, which adopts the following technical solution:

[0018] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the aforementioned method for verifying and optimizing intelligent testing algorithms.

[0019] According to the embodiments provided in this application, a state space file and a score space file are generated by an output module. A simulation module executes a simulation process based on the received state space file and score space file, and determines the score information of the simulation results. An intelligent testing algorithm module determines a target test sample set based on the score information and sampling information, and uses a tool module to construct a sample-score distribution map corresponding to the target test sample set. Thus, a modular system for constructing, simulating, and evaluating test tasks is built, improving the efficiency of constructing test tasks for testing intelligent testing algorithms. Furthermore, by decoupling the testing process through modularization, the system can meet the testing needs of different types of simulation units, improving the adaptability of the intelligent testing algorithm system for verification and optimization, reducing the hardware configuration requirements of the co-simulation platform, and increasing simulation efficiency. Attached Figure Description

[0020] Figure 1 This is a block diagram illustrating the use of an embodiment of the present application to verify and optimize an intelligent testing algorithm system;

[0021] Figure 2 This is a block diagram of the simulation module and the intelligent testing algorithm module in an embodiment of this application;

[0022] Figure 3 This is a block diagram of the tool module in an embodiment of this application;

[0023] Figure 4 This is a block diagram illustrating a method for verifying and optimizing intelligent testing algorithms according to an embodiment of this application. Detailed Implementation

[0024] The following is in conjunction with the appendix Figures 1-4 This application will be described in further detail.

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] This application provides a system for verifying and optimizing intelligent testing algorithms, with reference to... Figure 1A system for verifying and optimizing intelligent testing algorithms includes: an input module, a simulation module, an intelligent testing algorithm module, and a tool module; the input module generates a state space file and a score space file; the simulation module, connected to the input module, parses the received state space file to obtain sampling information, and determines score information based on the state space file and the received score space file; the intelligent testing algorithm module, connected to the simulation module, determines attribute sequence values ​​based on the sampling information, and determines the target test sample set based on the sampling information and score information; the tool module, connected to the intelligent testing algorithm module, constructs a sample-score distribution map based on the received target test sample set, wherein the sample-score distribution map includes the distribution of test samples and their corresponding scores in the target test sample set, and highlights key test samples and the boundaries of sample composition in the target test sample set; the sampling information consists of element information that needs to be sampled in the simulation environment.

[0027] In some embodiments, the user subjectively designs test task information and sends the test task information to the input module. Based on the received test task information, the input module generates a state space file and a score space file, wherein both the state space file and the score space file exist in YAML format.

[0028] Subsequently, the input module inputs the state space file and score space into the simulation module. The simulation module analyzes, parses, and samples the state space file, and sends the sampled information obtained from the analysis, parsing, and sampling to the intelligent testing algorithm module. Based on the sampled information, the intelligent testing algorithm module determines the attribute sequence values ​​and sends the attribute sequence values ​​to the simulation module. The simulation module uses the attribute sequence values ​​to start the simulation process and obtains the score information after the simulation is completed. Based on the score information and sampling information, the intelligent testing algorithm module determines the target test sample set and sends the target test sample set to the tool module. Using the tool module, a sample-score distribution graph is constructed. Thus, by constructing a test task construction, simulation, and evaluation system in a modular form, the efficiency of constructing test tasks to test the intelligent testing algorithm is improved. Through modular decoupling of the test process, it can meet the testing needs of different types of simulation units, improve the adaptability of the verification and optimization of the intelligent testing algorithm system, reduce the hardware configuration requirements of the co-simulation platform, and improve simulation efficiency.

[0029] In some embodiments, the state space file includes environment description information, carrier description information, task element description information, and collision rule description information; the score space file includes performance index information and the weight information corresponding to the performance index information, wherein the task element description information includes test task information for a specific test task, such as waypoints and related attributes and routes in a path planning task.

[0030] In some embodiments, the environmental description information includes base environment information and attribute information corresponding to the base environment; the carrier description information includes information on visualization resources, collision shape and mounted devices, as well as algorithms and parameters indicating the carrier's perception, planning and decision-making behaviors.

[0031] In some embodiments, the base environment information includes available elements in the simulation environment, attribute information is the attribute information corresponding to the available elements, and environment description information is used to inform the sampler whether the sampler can use the identifier, the range of data to be sampled, or a fixed value; the carrier description information samples the same identifier and value type for the attributes that can be sampled; the collision rule description information indicates whether all objects need to record the current contact event when they come into contact during the simulation process and the corresponding operation that needs to be simulated after the contact is released; the score space file is used to specify which specific aspects of the unmanned system the simulation module will score, such as evaluating the destination and fuel quantity indicators of the unmanned system, i.e., determining whether the carrier has reached the destination and how much fuel has been consumed, and finally obtaining the final score by weighted summation.

[0032] In some embodiments, refer to Figure 2 The simulation module includes: a task file parsing unit, a scenario generation unit, a simulation management unit, and a scorer. The task file parsing unit, connected to the input module, parses the received state space file to obtain sampling information and outputs a score space file. The scenario generation unit, connected to the intelligent testing algorithm module, generates a scenario file based on the received attribute sequence values. The simulation management unit, connected to the scenario generation unit, performs simulation based on the received scenario file to obtain simulation results. The scorer, connected to the task file parsing unit and the simulation management unit, receives the score space file and simulation results, and determines the score information based on the score space file and simulation results.

[0033] In some embodiments, refer to Figure 2 The intelligent testing algorithm module includes: an intelligent testing algorithm unit and a test sample construction unit; the intelligent testing algorithm unit, connected to the simulation module, is used to receive sampling information, determine attribute sequence values ​​based on the sampling information, and determine the test sample set based on the sampling information and score information; the test sample construction unit, connected to the intelligent testing algorithm unit, is used to select the target test sample set from the test sample set.

[0034] In some embodiments, the task file parsing unit receives the state space file and score space file sent by the input module, and analyzes, parses and samples the state space file. That is, by retrieving all sampling identifiers in the state space file and using the sampling identifiers, the sampling information is determined. Subsequently, the task file parsing unit sends the sampling information to the intelligent testing algorithm unit included in the intelligent testing algorithm module. The intelligent testing algorithm unit uses its internally installed testing algorithm to select attribute sequence values ​​from the information range specified in the sampling information.

[0035] Subsequently, the intelligent testing algorithm unit sends the attribute sequence values ​​to the scenario generation unit in the simulation module. The scenario generation unit combines the attribute sequence values ​​to generate scenario files. For example, for a static obstacle's X-axis position attribute Position_X, if the sub-attribute Attribute_Changeable has a value of True, then a sequence of fixed values ​​needs to be sampled from the sub-attribute Value_Range values ​​[300, 500]. The testing algorithm installed in the intelligent testing algorithm unit is a uniform sampling algorithm, and when the number of samples is 5, then the sampled sequence values ​​are [300, 350, 400, 450, 500], and 5 scenario files are generated based on the sequence values. The scenario generation unit sends the scenario files to the simulation management unit, which is used to schedule available simulation resources and perform parallel simulations on batches of scenario files.

[0036] The simulation management unit executes the simulation process based on the scenario file and outputs the simulation results; the scorer receives the simulation results output by the simulation management unit and the score space file output by the task file parsing unit, analyzes the recorded data in the simulation results, calculates the scores corresponding to each performance index in the simulation results, and outputs the score information; the intelligent testing algorithm receives the score information output by the scorer, constructs a test sample set based on the score information, and sends the test sample set to the test sample construction unit, which constructs the target test sample set from the test sample set.

[0037] In some embodiments, the scorer is used to manage evaluation indicators and specific calculation methods. The score information output by the counter is obtained by weighting the indicator values. For example, when calculating the target indicator, if the simulation results are retrieved and a collision event between the carrier and the destination is recorded in the simulation results, then the score for that item is 1. When calculating the fuel quantity indicator and a collision event with the refueling point is retrieved, then the score for that item is 1. If the weight of both indicators is 1, then the weight is first normalized to 0.5, and finally weighted to obtain the final score of 1.

[0038] In some embodiments, when the visualization mode is executed in the scenario file, the simulation management unit executes single simulations sequentially, and the current simulation only begins after the previous simulation ends. When the non-visual parallel mode is executed in the scenario file, the simulation management unit first queries the current simulation computing resources. For example, if a single simulation requires the use of one computer process, it first queries the number of CPU cores m in the system, and then queries the number of currently running processes n to obtain the maximum number of processes (mn) available for parallel simulation. To ensure the redundancy of system operation, a process pool of size 0.8*(mn) is created, and then parallel simulation is performed.

[0039] In some embodiments, refer to Figure 2 The simulation management unit includes at least one simulation engine and at least one unmanned system, wherein there is a correspondence between the at least one simulation engine and at least one unmanned system; the at least one unmanned system includes an execution algorithm; the at least one simulation engine is used to construct a simulation environment based on a scenario file, and to execute the simulation process based on the execution algorithm and the simulation environment to obtain simulation results.

[0040] In some embodiments, the simulation engine employs a simplified two-dimensional simulation engine to reduce simulation events and improve simulation efficiency. The simulation engine is based on Pymunk, a Python library for 2D physics simulation, and utilizes Pygame for visualization, which is also a Python library.

[0041] In some embodiments, the simulation engine abstracts all its internal elements into elements such as the simulation world, carriers, static obstacles, dynamic obstacles, and waypoints. Each element contains images and collision shapes for visualization. During the simulation process, all events of interest are abstracted into collision events between any two elements in the simulation world, and corresponding subsequent events are implemented according to collision rules. The simulation world records all elements created in the environment and monitors and performs subsequent operations on all collision events. Carriers are distinguished by labels as either the unmanned system under test or a system in the background, and are configured with their own devices and perception, decision-making, and planning algorithms. Static obstacles are used to specify impassable areas in the test task. Dynamic obstacles specify non-carrier elements that can move in the simulation and are configured with specified simple motion algorithms. Waypoints are used to specify the starting point and resupply points along the target path.

[0042] For example, in the event of a collision between the carrier and a dynamic obstacle, the simulation environment will remove the carrier and the dynamic obstacle when the event is detected, based on the destruction rules included in the collision rules; in the event of a collision between the carrier and a refueling point, the simulation environment will increase the carrier's fuel level and remove the refueling point according to the corresponding rules; in the event of a collision between the carrier and the destination, and according to the corresponding specified rules, the simulation environment will end the simulation.

[0043] In some embodiments, by constructing a simple simulation module, uniformly abstracting events of interest, and standardizing the design language for constructing the simulation environment, the efficiency of a single simulation is improved. Furthermore, parallel simulation can be carried out without the support of excessive hardware resources, further improving simulation efficiency, while also enabling traversal of the state space.

[0044] In some embodiments, refer to Figure 3 The tool module includes: a plotting tool unit, connected to the intelligent testing algorithm module, used to receive the target test sample set and construct a sample-score distribution map based on the target test sample set; and a state space traversal tool unit, used to provide a traversal algorithm for the plotting tool unit, and to perform plotting sample sampling on the target test sample set based on the traversal algorithm. The traversal algorithm provides the intelligent testing algorithm in the intelligent testing algorithm unit with the target to be approached, thereby evaluating the performance and optimization direction of the intelligent testing algorithm.

[0045] In some embodiments, because the state space dimension m is large, it is difficult to visualize the final score distribution. Therefore, the plotting tool unit in the tool module is used to group all space dimensions pairwise to obtain the final score distribution. There are several combination methods. Then, under each combination method, the sampling elements in the sampling information and their corresponding score information are combined to display the score distribution related to the elements.

[0046] The state space traversal tool unit defaults to a traversal algorithm that uniformly collects data in the intelligent testing algorithm throughout the entire simulation testing process. The simulation mode adopts a parallel mode, which can quickly obtain the scores corresponding to all state space distributions and display them through the graphing tool unit. This makes it easy to intuitively show the effect that the intelligent testing algorithm should achieve, which is convenient for subsequent optimization.

[0047] This application provides a method for verifying and optimizing intelligent testing algorithms.

[0048] Reference Figure 4 A method for verifying and optimizing intelligent testing algorithms, executed by the aforementioned system for verifying and optimizing intelligent testing algorithms, includes steps S101, S102, S103, and S104, wherein...

[0049] S101, the input module obtains the state space file and the fraction space file.

[0050] In some embodiments, the user generates test task information through subjective human settings and sends the test task information to the input module. The input module analyzes the test task information and generates a state space file and a score space file. The state space file and the score space file exist in YAML format. The state space file contains environment description information, carrier description information, task element description information, and collision rule description information. The score space file includes performance index information and the weight information corresponding to the performance index information.

[0051] S102, the simulation module parses the received state space file to obtain sampling information, and determines the score information based on the state space file and the received score space file.

[0052] In some embodiments, the simulation module parses the state space file through its included task file parsing unit, retrieves the sampling identifiers of all sampling information in the state space file, and uses the sampling identifiers to determine the sampling information. The simulation module then uses the state space file and the score space file to perform the simulation process and determines the score information corresponding to the simulation results output by the simulation process.

[0053] S103, the intelligent testing algorithm module determines the attribute sequence values ​​based on the sampling information, and determines the target test sample set based on the sampling information and score information.

[0054] In some embodiments, the simulation module sends the sampling information to the intelligent testing algorithm module. The intelligent testing algorithm module uses its internally installed intelligent testing algorithm to determine the attribute sequence values ​​from the numerical range corresponding to the sampling information and sends the attribute sequence values ​​to the simulation module. The simulation module fills the scenario file of the value-driven simulation with the attribute sequence values ​​to obtain the scenario file and uses the scenario file to execute the simulation process. In addition, the intelligent testing algorithm module combines and filters the sampling information and score information to obtain the target test sample set.

[0055] S104, the tool module constructs a sample-score distribution map based on the received target test sample set.

[0056] In some embodiments, the intelligent testing algorithm module sends the target test sample set to the tool module. The tool module uses its internally installed plotting tools and state space traversal tools to construct a sample-score distribution map corresponding to the sampling information. Thus, by constructing a test task construction, simulation, and evaluation system in a modular form, the efficiency of constructing test tasks to conduct tests on the intelligent testing algorithm is improved. Furthermore, by decoupling the test process through modularization, the system meets the testing needs of different types of simulation units, thereby improving the adaptability of the intelligent testing algorithm system for verification and optimization.

[0057] Step S104: The tool module constructs a sample-score distribution map based on the received target test sample set, including: when the intelligent test algorithm included in the intelligent test algorithm module is a cyclic iterative test algorithm, the sequence value output by the intelligent test algorithm unit in the intelligent test algorithm module at the current cyclic iteration round is input to the simulation module, the simulation module performs the next round of simulation based on the sequence value, and the intelligent test module outputs the target test sample set corresponding to the current cyclic iteration round, and constructs a sample-score distribution map based on the target test sample set; when the intelligent test algorithm included in the intelligent test algorithm module is a non-cyclic test algorithm, the sample-score distribution map is constructed based on the target test sample set.

[0058] In some embodiments, when the intelligent testing algorithm included in the intelligent testing algorithm module is a cyclic iterative testing algorithm, it indicates that the sequence values ​​output by the intelligent testing algorithm unit in the intelligent testing algorithm module need to be input into the simulation module again for the next round of simulation based on the number of cyclic iterations of the testing algorithm. Subsequently, the intelligent testing algorithm module inputs the sequence values ​​output by the intelligent testing algorithm unit contained within it to the simulation module at the current cyclic iteration round. The simulation module performs the next round of simulation based on the sequence values. At the same time, the intelligent testing algorithm module outputs the target test sample set and inputs the target test sample set into the tool module to construct the sample-score distribution map.

[0059] When the intelligent testing algorithm included in the intelligent testing algorithm module is a non-cyclic testing algorithm, it means that there is no need to iterate over the target test sample set output by the intelligent testing algorithm module. That is, the target test sample set is the final target test sample set. Subsequently, the intelligent testing algorithm module only inputs the output target test sample set into the tool module. The tool module constructs the sample-score distribution graph corresponding to the target test sample set. Thus, by utilizing the graphical construction capability of the tool module, the convenience of verifying and optimizing the intelligent testing algorithm is improved.

[0060] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0061] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to perform a method for verifying and optimizing an intelligent testing algorithm.

[0062] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0063] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A system for verifying and optimizing intelligent testing algorithms, characterized in that, include: The input module is used to generate state space files and score space files based on the acquired test task information; The simulation module, connected to the input module, is used to parse the received state space file to obtain sampling information, and to determine score information based on the state space file and the received score space file, wherein the sampling information is the range of attribute information of available elements for the simulation environment in the state space file; An intelligent testing algorithm module, connected to the simulation module, is used to determine attribute sequence values ​​based on the sampling information, and to determine a target test sample set based on the sampling information and the score information. The module utilizes a testing algorithm installed within it to select attribute sequence values ​​from a range of information specified in the sampling information, combines and filters the sampling information and the score information to obtain the target test sample set, and sends the attribute sequence values ​​to the simulation module. The tool module, connected to the intelligent testing algorithm module, is used to construct a sample-score distribution map based on the received target test sample set.

2. The system according to claim 1, characterized in that, The simulation module includes: The task file parsing unit, connected to the input module, is used to parse the received state space file to obtain the sampling information and output the score space file; The scenario generation unit, connected to the intelligent testing algorithm module, is used to generate a scenario file based on the received attribute sequence values; The simulation management unit, connected to the scenario generation unit, is used to perform simulation based on the received scenario file and obtain simulation results. The scorer, connected to the task file parsing unit and the simulation management unit, is used to receive the score space file and the simulation results, and determine the score information based on the score space file and the simulation results.

3. The system according to claim 2, characterized in that, The simulation management unit includes at least one simulation engine and at least one unmanned system, wherein, There is a correspondence between the at least one simulation engine and the at least one unmanned system; The at least one unmanned system includes an execution algorithm; The at least one simulation engine is used to construct a simulation environment based on the scenario file, and to execute the simulation process based on the execution algorithm and the simulation environment to obtain the simulation results.

4. The system according to claim 1, characterized in that, The intelligent testing algorithm module includes: The intelligent testing algorithm unit, connected to the simulation module, is used to receive the sampling information, determine the attribute sequence value based on the sampling information, and determine the test sample set based on the sampling information and the score information. The test sample construction unit, connected to the intelligent test algorithm unit, is used to select the target test sample set from the test sample set.

5. The system according to claim 1, characterized in that, The tool module includes: The graphing tool unit, connected to the intelligent testing algorithm module, is used to receive the target test sample set and construct the sample-score distribution graph based on the target test sample set. The state space traversal tool unit is used to provide a traversal algorithm for the plotting tool unit, and to perform plotting sample sampling on the target test sample set based on the traversal algorithm.

6. The system according to claim 1, characterized in that, The state space file includes environmental description information, carrier description information, task element description information, and collision rule description information; The score space file includes performance index information and the weight information corresponding to the performance index information.

7. The system according to claim 6, characterized in that, The environmental description information includes the base environment information and the attribute information corresponding to the base environment; The carrier description information includes information on visualization resources, collision shape, and onboard equipment, as well as the algorithms and parameters used to indicate the carrier's perception, planning, and decision-making behaviors.

8. A method for verifying and optimizing intelligent testing algorithms, characterized in that, Performed by the system for verifying and optimizing intelligent testing algorithms as described in any one of claims 1-7. The method includes: The input module generates the state space file and the score space file based on the acquired test task information; The simulation module parses the received state space file to obtain sampling information, and determines score information based on the state space file and the received score space file; The intelligent testing algorithm module determines the attribute sequence value based on the sampling information, and determines the target test sample set based on the sampling information and the score information; The tool module constructs the sample-score distribution map based on the received target test sample set.

9. The method according to claim 8, characterized in that, The tool module constructs the sample-score distribution map based on the received target test sample set, including: When the intelligent testing algorithm included in the intelligent testing algorithm module is a cyclic iterative testing algorithm, the sequence value output by the intelligent testing algorithm unit in the intelligent testing algorithm module at the current cyclic iteration round is input to the simulation module. The simulation module performs the next round of simulation based on the sequence value, and the intelligent testing module outputs the target test sample set corresponding to the current cyclic iteration round, and constructs the sample-score distribution map based on the target test sample set. When the intelligent testing algorithm included in the intelligent testing algorithm module is a non-cyclic testing algorithm, the sample-score distribution map is constructed based on the target test sample set.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the processor performs the method according to any one of claims 8-9.

Images