Safety test method and device for automatic driving simulation

Abstract

The invention discloses a safety test method and device for automatic driving simulation. The method comprises the following steps: generating a full evaluation task parameter set; creating an evaluation task sequence according to the full evaluation task parameter set, traversing all servers in an automatic driving simulation safety test when any scene evaluation task in the evaluation task sequence is executed, and calculating server scores corresponding to all the servers and the number of currently available idle display cards; the scene evaluation task is scheduled to a target server with the highest server score and the number of currently available idle display cards conforming to the scene evaluation task; and controlling a target server to execute the scene evaluation task, and monitoring a task execution state. According to the method and the device, full-coverage testing of automatic driving simulation safety testing tasks in massive scenes is realized, and the utilization rate of server resources is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving simulation safety testing, and in particular to a method and device for autonomous driving simulation safety testing. Background Art

[0002] As autonomous driving technology becomes increasingly common in cars, and as driving is crucial to personal safety, many automakers and evaluation agencies are placing significant emphasis on safety-related testing of autonomous driving technology. Field testing is often limited by site and weather conditions, making it impossible to conduct testing in extreme scenarios. Using simulation software for testing can effectively address these issues and has become a mainstream choice. However, when using simulation software to simulate autonomous driving scenarios for safety testing, the following issues currently exist:

[0003] (1) The autonomous driving model evaluation task needs to cover a variety of key parameters such as weather, maps, vehicle models, tested autonomous driving models, attack algorithms, and evaluation algorithms. The full set of evaluation tasks formed by the permutations and combinations of the above parameters is extremely large. At the same time, in the process of continuous iteration of autonomous driving safety evaluation technology, new key parameters and the addition or adjustment of dictionary items of existing key parameters will cause the full set of evaluation tasks to be constantly updated and adjusted. Relying on manual creation of evaluation tasks covering all key parameters is not only a huge workload, but also prone to omissions and difficult to update in a timely manner.

[0004] (2) The operation of the autonomous driving model evaluation task requires the use of graphics card resources. Given the limited hardware resources and the large number of evaluation tasks, relying on manual coordination not only takes up manpower, but also cannot avoid the contradiction between graphics card resources being idle on holidays and graphics card resources being occupied on weekdays;

[0005] (3) The workload of collecting and feeding back the evaluation results is huge. How to efficiently and timely feed back the evaluation results so as to optimize the model in a timely manner is also an urgent problem that needs to be solved. Summary of the Invention

[0006] In order to solve the technical problems existing in the background technology, the present invention proposes a simulation safety testing method and device for autonomous driving.

[0007] The present invention proposes a simulation safety testing method for autonomous driving, comprising:

[0008] Generate a full set of evaluation task parameters;

[0009] Creating an evaluation task sequence based on the full evaluation task parameter set; when executing any scenario evaluation task in the evaluation task sequence, traversing all servers in the autonomous driving simulation safety test, and calculating the server scores corresponding to all servers and the number of currently available idle graphics cards;

[0010] Dispatching the scenario evaluation task to the target server with the highest server score and the number of currently available idle graphics cards that meets the requirement of the scenario evaluation task;

[0011] Control the target server to execute the scenario evaluation task and monitor the task execution status.

[0012] Preferably, the full evaluation task parameter set includes multiple scenario evaluation tasks, and the multiple scenario evaluation tasks correspond one-to-one to multiple graphics card requirements. The types of the scenario evaluation tasks include exclusive graphics card tasks and shared graphics card tasks, and the graphics card requirements include the number of exclusive graphics cards and the number of shared graphics cards.

[0013] Preferably, generating a full set of evaluation task parameters specifically includes:

[0014] Dynamically collect multi-dimensional parameters that affect the safety of the autonomous driving model through the simulation platform interface. The multi-dimensional parameters include but are not limited to weather type, map route, vehicle type, tested model, attack algorithm, and evaluation algorithm.

[0015] The Cartesian product is used to combine the dictionary items of all multidimensional parameters to generate a set of evaluation tasks covering all possible scenarios. The evaluation task set serves as the full evaluation task parameter set. The generated parameter set is stored in JSON format, where each row corresponds to a parameter combination of an evaluation task and supports real-time updates to adapt to the addition or adjustment of parameters to ensure that the test covers all dynamic scenarios. The total number of tasks is determined by the product of the number of dictionary items in each parameter category.

[0016] Preferably, the server scores corresponding to all servers are calculated as follows:

[0017]

[0018] Among them, score i is the server score of server i; N is the total number of graphics cards in server i; m is the number of exclusive graphics cards required for the current scenario evaluation task; Card j The usage score of the j-th graphics card in the server. If the task running on the j-th graphics card is an exclusive task, the usage score is 1. If the task running on the j-th graphics card is a shared task, the usage score is the current memory usage of the graphics card.

[0019] Preferably, the number of currently available idle graphics cards specifically includes:

[0020] Real-time query of the total number of graphics cards on the current server and the number of idle graphics cards, and calculation of the currently available idle graphics card number based on the graphics card resources dynamically reserved according to time rules;

[0021] The specific calculation process of the currently available number of idle graphics cards is as follows:

[0022] The number of currently available idle graphics cards = the total number of graphics cards - the current number of idle graphics cards - the dynamically reserved graphics card resources.

[0023] Preferably, the time rule is specifically:

[0024] The number of graphics cards reserved during holidays is 0;

[0025] During the preset working hours on weekdays, the number of graphics cards reserved is 1 / 3 of the total number of graphics cards;

[0026] The number of graphics cards reserved during the preset non-working time period on working days is 1 / 6 of the total number of graphics cards.

[0027] Preferably, it also includes:

[0028] When all scenario evaluation tasks in the evaluation task sequence have completed their execution, the total number of evaluation tasks, the number of successes, the number of failures, and the number of evaluation tasks whose results did not meet expectations are counted and output.

[0029] Preferably, it also includes:

[0030] For evaluation tasks that fail to execute or whose results do not meet expectations, a sequence of tasks to be verified is generated. The evaluation is re-executed according to the sequence of tasks to be verified until all scenario evaluation tasks pass verification, thus forming a closed-loop iterative test.

[0031] Preferably, the evaluation tasks whose effects do not meet expectations are:

[0032] In the attack algorithm test, the average precision AP after the attack a or average recall AR a Higher than the average precision AP before the attack b or average recall AR b ;

[0033] The calculation process for the number of evaluation tasks whose results did not meet expectations is as follows:

[0034] FF=Count(AP b <AP a ∪AR b <AR a );

[0035] Among them, AP b The average precision calculated for different categories of the autonomous driving model output for the pre-attack evaluation task; AP a The average precision of the autonomous driving model output for the post-attack evaluation task calculated under different categories; AR b The average recall rate calculated for different categories of the autonomous driving model output for the pre-attack evaluation task; AR aThe average recall rate calculated for the autonomous driving model output by the post-attack evaluation task under different categories; Count is the number of tasks whose average precision or recall rate before the attack is less than the average precision or recall rate after the attack; FF is the number of evaluation tasks whose results do not meet expectations.

[0036] The present invention proposes a simulation safety test device for autonomous driving, comprising:

[0037] Data generation module, used to generate the full set of evaluation task parameters;

[0038] A data processing module is configured to create an evaluation task sequence based on the full evaluation task parameter set. When executing any scenario evaluation task in the evaluation task sequence, the module traverses all servers in the autonomous driving simulation safety test and calculates the server scores corresponding to all servers and the number of currently available idle graphics cards.

[0039] An allocation module is used to dispatch the scenario evaluation task to a target server with the highest server score and a number of currently available idle graphics cards that meets the requirement of the scenario evaluation task;

[0040] The execution module is used to control the target server to execute the scenario evaluation task and monitor the task execution status.

[0041] The proposed method and device for autonomous driving simulation safety testing in the present invention achieves full-scenario coverage evaluation of autonomous driving models. By obtaining key parameters that have been maintained by the platform and affect the effectiveness of autonomous driving model algorithms, including weather, maps, vehicle models, tested autonomous driving models, attack algorithms, evaluation algorithms, etc., a full set of evaluation task parameters is generated. When key parameters are adjusted, the full set of evaluation task parameters can be flexibly and real-time updated. Then, based on the number of idle graphics cards on the server, a corresponding number of evaluation tasks are dynamically and automatically initiated until all evaluation tasks of the parameter sets are covered, and then the evaluation results are collected. For evaluation tasks that failed to run and evaluation tasks that succeeded but did not meet expectations, parameter sets to be verified are collected. After R&D and repair, the evaluation tasks with the corresponding parameters are rerun until all evaluation tasks run successfully. Full coverage testing of autonomous driving simulation safety test tasks in a large number of scenarios is achieved, improving the utilization rate of server resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a schematic diagram of the workflow of a simulation safety testing method for autonomous driving proposed in the present invention;

[0043] Figure 2 This is a schematic diagram of the implementation process of a simulation safety testing method for autonomous driving proposed in the present invention;

[0044] Figure 3This is a schematic diagram of a process flow for querying the idle number of graphics cards for an autonomous driving simulation safety test method proposed by the present invention;

[0045] Figure 4 This is a flowchart of analyzing and processing time series data for an autonomous driving simulation safety testing method proposed by the present invention;

[0046] Figure 5 This is a schematic diagram of the device architecture of an autonomous driving simulation safety testing device proposed in the present invention. DETAILED DESCRIPTION

[0047] Reference Figure 1-5 The present invention proposes a safety testing method for autonomous driving simulation, comprising the following steps:

[0048] S1. Generate a full set of evaluation task parameters.

[0049] In this embodiment, the full evaluation task parameter set includes multiple scenario evaluation tasks, and the multiple scenario evaluation tasks correspond one to one to multiple graphics card requirements. The types of scenario evaluation tasks include exclusive graphics card tasks and shared graphics card tasks, and the graphics card requirements include the number of exclusive graphics cards and the number of shared graphics cards.

[0050] In this embodiment, generating a full set of evaluation task parameters specifically includes:

[0051] Dynamically collect multi-dimensional parameters that affect the safety of the autonomous driving model through the simulation platform interface. The multi-dimensional parameters include but are not limited to weather type, map route, vehicle type, tested model, attack algorithm, and evaluation algorithm.

[0052] The Cartesian product is used to combine the dictionary items of all multidimensional parameters to generate a set of evaluation tasks covering all possible scenarios. The evaluation task set serves as the full evaluation task parameter set. The generated parameter set is stored in JSON format, where each row corresponds to a parameter combination of an evaluation task and supports real-time updates to adapt to the addition or adjustment of parameters to ensure that the test covers all dynamic scenarios. The total number of tasks is determined by the product of the number of dictionary items in each parameter category.

[0053] Specifically, Python implements RSA-encrypted identity information, enabling simulation of platform requests. Key parameter dictionary entries are returned through the interface to construct a dataset, and evaluation tasks are created and run. Python requests are used to simulate the platform interface's response to obtain information such as weather, map, vehicle type, model under test, test type, and attack type. Test types include white-box and black-box testing, and attack types include physical and digital attacks. Attack algorithms are matched to the attack type and model under test, and evaluation algorithms are matched to the model under test. All key factors influencing the safety evaluation of autonomous driving algorithms, including weather, map, vehicle type, model under test, test type, attack algorithm, and evaluation algorithm, are traversed to generate the input parameter set required for the full evaluation task, thereby improving test coverage. The full evaluation task parameter set is a JSON-formatted file, with each line corresponding to a parameter required for the evaluation task. As the models under test and evaluation algorithms are continuously updated and expanded, the full evaluation task parameter set can be dynamically updated based on the latest platform data.

[0054] In a white-box environment, the fast gradient descent method is a common method for generating attack samples. The implementation logic is as follows:

[0055] Use cross entropy to calculate the loss function: Loss = L(y,y s ); where y represents the true output of the sample in the model, y s Represents the true value label of the sample, and L represents the cross entropy function;

[0056] By finding the derivative of the model with respect to the input, and then using the sign function to get its specific gradient direction, and then multiplying it by the step size, the obtained perturbation is added to the original input to obtain the sample under attack. The formula is as follows:

[0057]

[0058] in, represents the gradient of the loss function with respect to the input x; sign represents the direction of the gradient, that is, only the positive and negative directions are retained and the magnitude is removed; ε is the step size, that is, the amplitude of the control update; x is the original sample; adv_x is the sample after the attack.

[0059] In the object detection task, Average Precision (AP) and Average Recall (AR) are important indicators for evaluating the model effect. The implementation logic is as follows:

[0060] Calculate the IoU (Intersection over Union) of the predicted box and the real box: IoU is a ratio, which is the ratio of the area of ​​the intersection of the predicted box and the real box to the area of ​​the union of the two;

[0061] For each category, the prediction results are counted separately, where:

[0062] TP: The number of correctly classified positive examples, that is, the IoU between the predicted box and the true box is ≥ 0.5 and the predicted box is correctly classified;

[0063] FP: The number of instances that are incorrectly classified as positive examples, that is, the IoU between the predicted box and the true box is less than 0.5; or the IoU between the predicted box and the true box is ≥ 0.5, but the predicted box is incorrectly classified; or repeated detection, that is, the same true box matches multiple predicted boxes, and only one predicted box meets the TP judgment, and the remaining predicted boxes are counted as FP;

[0064] FN: The number of instances that are incorrectly classified as negative examples, i.e., the predicted box does not appear on the real box;

[0065] Calculate the average precision of all categories Average recall

[0066] The total number of full evaluation tasks is:

[0067] Where i is the key field that affects the evaluation task results, such as weather, map, vehicle type, model to be tested, test type, evaluation algorithm, etc., with a total of n. i is the number of dictionary entries corresponding to the key field that affects the evaluation task result.

[0068] S2. Create an evaluation task sequence based on the full evaluation task parameter set. When executing any scenario evaluation task in the evaluation task sequence, traverse all servers in the autonomous driving simulation safety test and calculate the server scores corresponding to all servers and the number of currently available idle graphics cards.

[0069] In this embodiment, the detailed implementation process of querying the number of idle graphics cards is as follows: Figure 3 As shown, the specific process is as follows:

[0070] The automated task automatically calls the "Graphics Card Idle Quantity Query Service" according to the task trigger time rules; the "Graphics Card Idle Quantity Query Service" will execute the PromSQL statement: "DCGM_FI_DEV_FB_USED", which is a field indicator in NVIDIA's DCGM (Data Center GPU Manager) monitoring tool. Its full name is "Device Frame BufferUsed", which indicates the current usage of GPU video memory. After executing this statement, it will return a list of sampled time series data of the usage of each graphics card; after obtaining the sampled time series data list of each graphics card, the "Graphics Card Idle Quantity Query Service" will analyze and process the time series data, calculate the current number of idle graphics cards, and return it to the automated task.

[0071] In this embodiment, each server in the cluster deploys a dcgm-exporter service through a daemon thread. This service calls the graphics card usage interface in the corresponding graphics card driver once per second and reports the collected data to Prometheus for storage in the format of "time: graphics card name: usage rate". The detailed process of analyzing and processing time series data by the "graphics card idle number query service" is as follows: Figure 4 As shown, the details are as follows: execute the DCGM_FI_DEV_FB_USED statement to obtain all graphics card collection data in the last 15 minutes; group the collected data by graphics card name, sort each group of data by time, and then remove the grouped data with a graphics card usage rate of 0 in the last minute; obtain the current time, if the current time is a holiday or rest day, the number of reserved graphics cards is 0, if the current time is working time on a weekday, the number of reserved graphics cards is one-third of the total number of graphics cards, if the current time is non-working time on a weekday, the number of reserved graphics cards is one-sixth of the total number of graphics cards; calculate the number of idle graphics cards according to the following formula: the total number of graphics cards in the cluster - the number of groups of filtered sampling data - the number of reserved graphics cards.

[0072] S3. Schedule the scenario evaluation task to the target server with the highest server score and the number of currently available idle graphics cards that meets the requirements of the scenario evaluation task.

[0073] S4. Control the target server to execute the scenario evaluation task and monitor the task execution status.

[0074] In this embodiment, if no graphics card resources are available, the system waits five minutes and then checks the number of available graphics cards again. When graphics card resources are available, the system reads the entire evaluation task parameter set JSON file, row by row, from the front to the back, based on the number of available graphics cards. Evaluation tasks are then automatically created and executed sequentially until all N evaluation tasks in the parameter set have been completed. Given limited hardware resources, this system can maximize server resource utilization during nighttime and holidays.

[0075] Benchmark tasks can be categorized as dedicated graphics card tasks or shared graphics card tasks. Dedicated graphics card tasks require exclusive use of one or more graphics cards, while shared graphics card tasks share graphics card resources. When you create a benchmark task, each server in the cluster is automatically scored and the task is then dispatched to the server with the highest score.

[0076] In this embodiment, the server scores corresponding to all servers are calculated, specifically:

[0077]

[0078] Among them, score i is the server score of server i; N is the total number of graphics cards in server i; m is the number of exclusive graphics cards required for the current scenario evaluation task; Card j The usage score of the j-th graphics card in the server. If the task running on the j-th graphics card is an exclusive task, the usage score is 1. If the task running on the j-th graphics card is a shared task, the usage score is the current memory usage of the graphics card.

[0079] In this embodiment, the number of currently available idle graphics cards specifically includes: querying the total number of graphics cards in the current server and the current number of idle graphics cards in real time, and calculating the number of currently available idle graphics cards in combination with graphics card resources dynamically reserved according to time rules; the calculation process of the currently available idle graphics cards specifically includes:

[0080] The number of currently available idle graphics cards = the total number of graphics cards - the current number of idle graphics cards - the dynamically reserved graphics card resources.

[0081] In this embodiment, the time rules are as follows: the number of graphics cards reserved on holidays is 0; the number of graphics cards reserved on weekdays during the preset working time period is 1 / 3 of the total number of graphics cards; the number of graphics cards reserved on weekdays during the preset non-working time period is 1 / 6 of the total number of graphics cards.

[0082] In this embodiment, it also includes:

[0083] S5. After all scenario evaluation tasks in the evaluation task sequence have completed their execution, the total number of evaluation tasks, the number of successes, the number of failures, and the number of evaluation tasks whose results did not meet expectations are counted and output.

[0084] S6. Generate a sequence of tasks to be verified for evaluation tasks that fail to execute or whose results do not meet expectations. Re-execute the evaluation according to the sequence of tasks to be verified until all scenario evaluation tasks pass verification, thereby forming a closed-loop iterative test.

[0085] In this embodiment, the evaluation tasks whose results do not meet expectations are specifically:

[0086] In the attack algorithm test, the average precision AP after the attack a or average recall AR a Higher than the average precision AP before the attack b or average recall AR b ;

[0087] The calculation process for the number of evaluation tasks that did not meet expectations is as follows:

[0088] FF=Count(AP b <AP a ∪AR b <AR a);

[0089] Among them, AP b The average precision calculated for different categories of the autonomous driving model output for the pre-attack evaluation task; AP a The average precision of the autonomous driving model output for the post-attack evaluation task calculated under different categories; AR b The average recall rate calculated for different categories of the autonomous driving model output for the pre-attack evaluation task; AR a The average recall rate calculated for the autonomous driving model output by the post-attack evaluation task under different categories; Count is the number of tasks whose average precision or recall rate before the attack is less than the average precision or recall rate after the attack; FF is the number of evaluation tasks whose results do not meet expectations.

[0090] Specifically, the completion of all evaluation tasks in the evaluation task sequence is considered the end of a round of testing. At this time, the evaluation result report corresponding to the full set of evaluation task parameters is output, including the total number of evaluation tasks, the number of successes, the number of failures, the start time and the end time of the task execution. For the evaluation tasks that are successfully executed, the average precision and average recall calculated in different categories before and after the attack on the autonomous driving model are collected. For tasks that fail to execute and tasks that succeed but the results do not meet expectations, a unique ID and input parameters are provided to facilitate subsequent problem location. At the same time, for tasks that fail to execute and tasks that succeed but the results do not meet expectations, the corresponding input parameters are collected to form a new set of input parameters to be verified. Then, steps S2-S6 are re-executed for the new set of input parameters to be verified until all evaluation tasks can be successfully executed, ultimately ensuring the efficiency and coverage of the autonomous driving model safety test.

[0091] For example, the platform currently supports three weather conditions: rain, snow, and fog; five map routes, including merging ramps; two vehicle types: armored vehicles and jeeps; a tested autonomous driving model, Yolov3; an attack algorithm based on gradient descent; and two evaluation algorithms: Average Precision (AP) and Average Recall (AR). This results in a total of 3*5*2*1*1*2=60 parameter sets, or 60 evaluation tasks. Specifically, on a specific parameter, for example, on the Town01 map on a rainy day, the Yolov3 model's average precision and recall for detecting a jeep before and after an attack are tested. When the full evaluation task parameter set is generated, the parameters for that scenario are included in the 60 generated evaluation tasks. Based on the number of available graphics cards, the corresponding number of Create and Run evaluation task APIs are called, starting with the first parameter set as the input parameter for the evaluation task, and continuing until all parameters are exhausted, including the evaluation task for that scenario. After all evaluation tasks are created and running, the evaluation task running status interface is called every two hours to count the number of completed tasks until the number of completed tasks equals the total number of tasks, that is, all evaluation tasks are completed. Then call the evaluation task running result interface to count the total number of evaluation tasks, the number of successes, and the number of failures, and collect the average precision and average recall rate of successful tasks. If the average precision and average recall rate before the attack are lower than those after the attack, it is considered that the effect does not meet expectations. Summarize the unique IDs and input parameters of tasks that failed to execute and tasks that succeeded but did not meet expectations, so that R&D can locate the problem. After the R&D has repaired the problem, create and run the evaluation task again based on the summarized input parameters (that is, the input parameters of the task whose evaluation results did not meet expectations) until the running results of all evaluation tasks meet expectations.

[0092] Reference Figure 1-5 The present invention proposes a safety test device for autonomous driving simulation, comprising:

[0093] The data generation module is used to generate the full set of evaluation task parameters.

[0094] The data processing module is used to create an evaluation task sequence based on the full evaluation task parameter set. When executing any scenario evaluation task in the evaluation task sequence, it traverses all servers in the autonomous driving simulation safety test and calculates the server scores corresponding to all servers and the number of currently available idle graphics cards.

[0095] The allocation module is used to schedule the scenario evaluation task to the target server with the highest server score and the number of currently available idle graphics cards that meets the scenario evaluation task.

[0096] The execution module is used to control the target server to execute the scenario evaluation task and monitor the task execution status.

[0097] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A simulation safety testing method for autonomous driving, characterized in that: include: Generate a full set of evaluation task parameters; Creating an evaluation task sequence based on the full evaluation task parameter set; when executing any scenario evaluation task in the evaluation task sequence, traversing all servers in the autonomous driving simulation safety test, and calculating the server scores corresponding to all servers and the number of currently available idle graphics cards; Dispatching the scenario evaluation task to the target server with the highest server score and the number of currently available idle graphics cards that meets the requirement of the scenario evaluation task; Control the target server to execute the scenario evaluation task and monitor the task execution status.

2. The autonomous driving simulation safety testing method according to claim 1, characterized in that: The full evaluation task parameter set includes multiple scenario evaluation tasks, and the multiple scenario evaluation tasks correspond one to one to multiple graphics card requirements. The types of the scenario evaluation tasks include exclusive graphics card tasks and shared graphics card tasks, and the graphics card requirements include the number of exclusive graphics cards and the number of shared graphics cards.

3. The autonomous driving simulation safety testing method according to claim 1, characterized in that: Generating a full set of evaluation task parameters specifically includes: Dynamically collect multi-dimensional parameters that affect the safety of the autonomous driving model through the simulation platform interface. The multi-dimensional parameters include but are not limited to weather type, map route, vehicle type, tested model, attack algorithm, and evaluation algorithm. The Cartesian product is used to combine the dictionary items of all multidimensional parameters to generate a set of evaluation tasks covering all possible scenarios. The evaluation task set serves as the full evaluation task parameter set. The generated parameter set is stored in JSON format, where each row corresponds to a parameter combination of an evaluation task and supports real-time updates to adapt to the addition or adjustment of parameters to ensure that the test covers all dynamic scenarios. The total number of tasks is determined by the product of the number of dictionary items in each parameter category.

4. The autonomous driving simulation safety testing method according to claim 2, characterized in that: The server scores corresponding to all servers are calculated as follows: Among them, score i is the server score of server i; N is the total number of graphics cards in server i; m is the number of exclusive graphics cards required for the current scenario evaluation task; Card j The usage score of the j-th graphics card in the server. If the task running on the j-th graphics card is an exclusive task, the usage score is 1. If the task running on the j-th graphics card is a shared task, the usage score is the current memory usage of the graphics card.

5. The autonomous driving simulation safety testing method according to claim 1, characterized in that: The currently available idle graphics card number specifically includes: Real-time query of the total number of graphics cards on the current server and the number of idle graphics cards, and calculation of the currently available idle graphics card number based on the graphics card resources dynamically reserved according to time rules; The specific calculation process of the currently available number of idle graphics cards is as follows: The number of currently available idle graphics cards = the total number of graphics cards - the current number of idle graphics cards - the dynamically reserved graphics card resources.

6. The autonomous driving simulation safety testing method according to claim 5, characterized in that: The time rules are specifically as follows: The number of graphics cards reserved during holidays is 0; During the preset working hours on weekdays, the number of graphics cards reserved is 1 / 3 of the total number of graphics cards; The number of graphics cards reserved during the preset non-working time period on working days is 1 / 6 of the total number of graphics cards.

7. The autonomous driving simulation safety testing method according to claim 3, characterized in that: Also includes: When all scenario evaluation tasks in the evaluation task sequence have completed their execution, the total number of evaluation tasks, the number of successes, the number of failures, and the number of evaluation tasks whose results did not meet expectations are counted and output.

8. The autonomous driving simulation safety testing method according to claim 7, characterized in that: Also includes: For evaluation tasks that fail to execute or whose results do not meet expectations, a sequence of tasks to be verified is generated. The evaluation is re-executed according to the sequence of tasks to be verified until all scenario evaluation tasks pass verification, thus forming a closed-loop iterative test.

9. The autonomous driving simulation safety testing method according to claim 7, characterized in that: The evaluation tasks whose results did not meet expectations are as follows: In the attack algorithm test, the average precision AP after the attack a or average recall AR a Higher than the average precision AP before the attack b or average recall AR b ; The calculation process for the number of evaluation tasks whose results did not meet expectations is as follows: FF=Count(AP b <AP a ∪AR b <AR a ); Among them, AP b The average precision calculated for different categories of the autonomous driving model output for the pre-attack evaluation task; AP a The average precision of the autonomous driving model output for the post-attack evaluation task calculated under different categories; AR b The average recall rate calculated for different categories of the autonomous driving model output for the pre-attack evaluation task; AR a The average recall rate calculated for the autonomous driving model output by the post-attack evaluation task under different categories; Count is the number of tasks whose average precision or recall rate before the attack is less than the average precision or recall rate after the attack; FF is the number of evaluation tasks whose results do not meet expectations.

10. A simulation safety test device for autonomous driving, characterized in that: include: Data generation module, used to generate the full set of evaluation task parameters; A data processing module is configured to create an evaluation task sequence based on the full evaluation task parameter set. When executing any scenario evaluation task in the evaluation task sequence, the module traverses all servers in the autonomous driving simulation safety test and calculates the server scores corresponding to all servers and the number of currently available idle graphics cards. An allocation module is used to dispatch the scenario evaluation task to a target server with the highest server score and a number of currently available idle graphics cards that meets the requirement of the scenario evaluation task; The execution module is used to control the target server to execute the scenario evaluation task and monitor the task execution status.

Images