Autonomous driving system test method and system, computer device, storage medium and program product

Abstract

An autonomous driving system test method, comprising: sampling scenario data of a plurality of scenarios (S10), and sending the scenario data of the plurality of scenarios to a plurality of simulation emulators (200) for distributed testing, so as to obtain test result data of the plurality of scenarios, the plurality of simulation emulators (200) being each integrated with an autonomous driving domain controller; and determining a dangerous scenario boundary for an autonomous driving system on the basis of the test result data of the plurality of scenarios (S20). The method is an improvement on the a scenario-based simulation test technology for autonomous driving systems, and using a server and a plurality of simulation emulators integrated with autonomous driving system domain controllers to perform joint testing can greatly improve the simulation test efficiency and the test effect for autonomous driving systems.

Description

Automatic driving system testing method, system, computer equipment, storage medium and program product Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an autonomous driving system testing method, system, computer equipment, storage medium, and program product. Background Art

[0002] Current scenario-based simulation testing for autonomous driving systems typically uses methods such as traversal generation, empirical generation, and accelerated generation to generate scenarios. Accelerated generation is the most efficient, often employing inference algorithms to generate test scenarios. However, these accelerated autonomous driving scenario testing methods typically employ software-in-the-loop (SIL) testing, resulting in low simulation efficiency and poor test results. Summary of the Invention

[0003] The embodiments of the present application provide an autonomous driving system testing method, system, computer equipment, storage medium and program product for solving at least one of the above-mentioned technical problems.

[0004] In a first aspect, an embodiment of the present application provides a method for testing an autonomous driving system, comprising:

[0005] Scenario data of multiple scenarios are sampled, and the scenario data of the multiple scenarios are sent to multiple simulation simulators for distributed testing to obtain test result data of the multiple scenarios; the multiple simulation simulators are all integrated with an autonomous driving domain controller; and the dangerous scenario boundaries of the autonomous driving system are determined based on the test result data of the multiple scenarios.

[0006] In some embodiments, scene data of multiple scenarios are sampled, and the scene data of the multiple scenarios are sent to multiple simulation simulators for distributed testing to obtain test result data of the multiple scenarios, including: scene data sampling and testing in groups to obtain test result data for each group of scenarios that completes the test, and the number of each group of sampled scenes is determined according to the number of multiple simulation simulators; if the sampling stop condition is met after sampling the scene data of the nth group of scenes, the test result data corresponding to the first n groups of scenes are determined as the test result data of the multiple scenarios.

[0007] In some embodiments, the sampling stop condition is a preset sampling number.

[0008] In some embodiments, the autonomous driving system testing method of the present application also includes: archiving the test result data of each set of scenarios that have completed the test; when a test interruption occurs, determining the next set of scenes to be sampled based on the archived test result data of each set of scenarios; and performing scene sampling based on the next set of scenes to be sampled.

[0009] In some embodiments, the dangerous scene boundaries of the autonomous driving system are determined based on the test result data of multiple scenarios, including: using the test result data of each group of scenarios that have been tested to train a dangerous scene recognition model; and determining the dangerous scene boundaries of the autonomous driving system based on the dangerous scene recognition model.

[0010] In some embodiments, determining the dangerous scene boundary of the autonomous driving system based on a dangerous scene recognition model includes: determining a parameter range of a logical scene; inputting the parameter range of the logical scene into the dangerous scene recognition model to obtain a danger threshold value of multiple scenes; determining dangerous scenes and safe scenes in the multiple scenes based on the danger threshold values ​​of the multiple scenes; and determining the boundary between the dangerous scenes and the safe scenes in the multiple scenes as the dangerous scene boundary of the autonomous driving system.

[0011] In some embodiments, the sampling of the first group of scenes in the group scene data sampling is the initial scene sampling; the group scene data sampling and testing includes: determining whether the dangerous scene recognition model converges; if so, performing model update scene sampling; if not, performing normal scene sampling.

[0012] In some embodiments, the autonomous driving system testing method of the present application also includes: verifying whether the dangerous scene recognition model has converged based on the scene data and test results obtained by model update scene sampling; if so, performing a new round of initial scene sampling and testing; if not, performing normal scene sampling.

[0013] In some embodiments, scene data sampling and testing are performed in groups, including: randomly determining a first group of scenes and performing scene sampling and testing; and determining the next group of scenes to be sampled based on a trained dangerous scene recognition model.

[0014] In some embodiments, the autonomous driving system testing method of the present application further includes:

[0015] The simulation test state of the simulation simulator is detected in real time to obtain speed information and distance information of the vehicle and the external object; when it is determined that a collision between the vehicle and the external object is unlikely to occur based on the speed information and distance information, the current simulation simulator test is stopped.

[0016] In some embodiments, the autonomous driving system testing method of the present application further includes: detecting abnormal information of the simulation simulator, and when abnormal information of the simulation simulator is detected, restarting the simulation simulator where the abnormality occurs.

[0017] In some embodiments, scene data of multiple scenes are sampled and sent to multiple simulation simulators for distributed testing, including: sampling scene data of multiple scenes; determining idle simulation simulators among multiple simulation simulators; and allocating at least part of the scene data of the multiple scenes to the idle simulation simulators for testing.

[0018] In some embodiments, the scene data of at least part of multiple scenes are assigned to idle simulation simulators for testing, including: determining the number of idle simulation simulators and selecting a corresponding number of scene data; establishing a one-to-one correspondence between each simulation simulator and a corresponding number of scene data; and sending each scene data in the scene data of the corresponding data to the corresponding idle simulation simulator for testing.

[0019] In some embodiments, a one-to-one correspondence is established between each simulation simulator and a corresponding number of scene data, including: establishing a mapping relationship between each simulator label and each scene label based on the simulator label of each simulation simulator and the scene label of each scene data in the scene data of the corresponding data.

[0020] In a second aspect, an embodiment of the present application provides an autonomous driving system testing system, comprising: a server and multiple simulation simulators, each of which is integrated with an autonomous driving domain controller; the server is configured to sample scene data of multiple scenes, and send the scene data of the multiple scenes to multiple simulation simulators for distributed testing; the multiple simulation simulators are configured to perform distributed testing on the scene data of the multiple scenes to obtain test result data of the multiple scenes; the server is further configured to determine the dangerous scene boundaries of the autonomous driving system based on the test result data of the multiple scenes.

[0021] In a third aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in any embodiment of the present application.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program / instruction is stored, characterized in that when the computer program / instruction is executed by a processor, the steps of the method described in any embodiment of the present application are implemented.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the method described in any embodiment of the present application are implemented.

[0024] The embodiment of the present application is an improvement to the scenario-based simulation testing technology of the autonomous driving system. It adopts a method of joint testing of a server and multiple simulation simulators integrated with autonomous driving system domain controllers, which can greatly improve the efficiency of the simulation testing of the autonomous driving system and improve the testing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] FIG1 is a schematic diagram of an embodiment of an automatic driving system test system of the present application;

[0027] FIG2 is a flow chart of an embodiment of a method for testing an autonomous driving system of the present application;

[0028] FIG3 is a flow chart of another embodiment of the automatic driving system testing method of the present application;

[0029] FIG4 is a flow chart of another embodiment of the automatic driving system testing method of the present application;

[0030] FIG5 is a flowchart of another embodiment of the automatic driving system testing method of the present application;

[0031] FIG6 is a flowchart of another embodiment of the automatic driving system testing method of the present application;

[0032] FIG7 is a flowchart of another embodiment of the automatic driving system testing method of the present application;

[0033] FIG8 is a flowchart of another embodiment of the automatic driving system testing method of the present application;

[0034] FIG9 is a flowchart of another embodiment of the automatic driving system testing method of the present application;

[0035] FIG10 is a flowchart of another embodiment of the automatic driving system testing method of the present application;

[0036] FIG11a is a schematic diagram of the experimental results of the traversal test method using a fixed step length;

[0037] FIG11b is a schematic diagram of the experimental results of the autonomous driving system testing method of the present application;

[0038] FIG12 is a schematic structural diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0040] It should also be noted that, in this document, the terms "include" and "comprising" include not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, the elements defined by the phrase "include..." do not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the elements.

[0041] Figure 1 shows a schematic diagram of an autonomous driving system testing system according to one embodiment of the present application. This embodiment includes a server 100 and multiple simulators 200, each of which incorporates an autonomous driving domain controller (ADCU). The server 100 utilizes a distributed controllable approach for multiple simulators 200, enabling multiple simulators 200 to perform tests in parallel across multiple scenarios.

[0042] As shown in Figure 2, an embodiment of the present application provides a flowchart of an embodiment of an autonomous driving system testing method. This embodiment includes: S10, sampling scenario data for multiple scenarios and sending the scenario data for the multiple scenarios to multiple simulation simulators for distributed testing to obtain test result data for the multiple scenarios; the multiple simulation simulators are each integrated with an autonomous driving domain controller. S20, determining the dangerous scenario boundaries of the autonomous driving system based on the test result data for the multiple scenarios.

[0043] Exemplarily, the server 100 samples scenario data of multiple scenarios and sends the scenario data of the multiple scenarios to multiple simulation simulators 200 for distributed testing. The server 100 and the multiple simulation simulators 200 communicate via a bus connection. The server 100 sends the sampled scenario data of the multiple scenarios to each simulation simulator 200 via the bus. For example, the server 100 stores simulator tags of the simulation simulators. The server 100 sends the scenario data carrying the simulator tags of each simulator to the bus. Each simulator 200 identifies and obtains its own scenario data based on the simulator tags carried by the scenario data, and constructs a test scenario based on the scenario data to complete the test.

[0044] Exemplarily, multiple scenarios include but are not limited to: vehicle entry recognition and response (for example, a vehicle cutting into the lane where the vehicle is located), recognition and response of oncoming vehicles using the vehicle's lane, obstacle recognition and response, pedestrians crossing the road, stable following driving, overtaking, etc.

[0045] Among them, the scene data in the vehicle entry recognition and response scenario includes but is not limited to: the speed of the vehicle itself, the speed of the vehicle in front, and the relative distance between the two vehicles when the vehicle in front cuts into the lane of the vehicle itself; the scene data for the recognition and response of the oncoming vehicle using the lane of the vehicle itself includes but is not limited to: the speed of the vehicle itself, the speed of the oncoming vehicle, and the relative distance between the two vehicles when the oncoming vehicle uses the lane; the scene data for obstacle recognition and response includes but is not limited to: the speed of the vehicle itself and the distance between the vehicle and the obstacle; the scene data for pedestrians crossing the road includes but is not limited to: the speed of the vehicle itself, the speed of the pedestrian, and the relative distance between the vehicle and the pedestrian when the pedestrian crosses the road.

[0046] This application improves scenario-based simulation testing technology for autonomous driving systems. This distributed testing method, using a server and multiple simulators integrated with autonomous driving domain controllers, significantly improves the efficiency of autonomous driving system simulation testing. Furthermore, hardware-in-the-loop testing of autonomous driving systems is achieved through multiple simulators, improving both accuracy and effectiveness.

[0047] FIG3 is a flow chart of another embodiment of the automated driving system testing method of the present application. In this embodiment, scene data of multiple scenes are sampled and sent to multiple simulation simulators for distributed testing to obtain test result data for the multiple scenes, including:

[0048] S11 , sampling and testing scene data in groups to obtain test result data for each group of scenes that have completed the test, wherein the number of each group of sampled scenes is determined according to the number of the multiple simulation simulators.

[0049] S12: If a sampling stop condition is met after sampling the scene data of the nth group of scenes, the test result data corresponding to the first n groups of scenes are determined as the test result data of the multiple scenes.

[0050] Exemplarily, in step S11, scenario data for a corresponding number of scenarios is sampled based on the number of multiple simulation simulators, and the sampled scenario data for the corresponding number of scenarios is sent to the multiple simulation simulators, thereby implementing parallel testing of the corresponding number of scenarios. For example, if the number of simulation simulators is 5, scenario sampling is performed according to each group of 5 scenarios, and the sampled scenario data for the 5 scenarios is sent to the 5 simulators respectively, thereby implementing parallel testing of the 5 scenarios (each simulation simulator completes the test of the corresponding scenario).

[0051] After completing the sampling of the scene data for the first group of scenes, the scene data for the second group of scenes is sampled. For example, after completing the sampling of the scene data for the first group of five scenes, the scene data for the second group of five scenes is sampled and sent to five simulation simulators for simulation testing. Following the above grouped sampling and testing method, after completing the sampling and testing of the i-th group of scene data, the sampling and testing of the i+1-th group of scene data is performed.

[0052] Exemplarily, for step S12, after completing the sampling of each set of scene data, determine whether the sampling stop condition is met. If not, continue with the next set of scene data sampling and testing. If so, use the multiple sets of test results obtained from the completed multiple sets of scene data sampling and testing as the final test result data for the multiple scenes. For example, when the sampling stop condition is met after completing the scene data sampling of the nth group of scenes, the test results corresponding to the 1st to nth groups of scenes are used as the final test results for the multiple scenes. The sampling stop condition can be a pre-set number of scene sampling groups or a pre-set total number of scene sampling times. For example, the pre-set total number of scene sampling times is 10,000. It should be noted that the above is only an example of the sampling stop condition, and other conditions can also be set, which is not limited in this application.

[0053] For example, when scene sampling is performed according to 5 scenes per group, and the value of n is 2000 and the sampling stop condition is reached, the 10,000 test results corresponding to the scene data of the 10,000 scenes sampled from the 1st to the 2000th group are summarized to obtain the final required test result data for multiple scenes.

[0054] In this embodiment, grouping is used for scenario sampling and testing, and a corresponding number of simulation servers are used for simulation testing, so that the simulation simulator is utilized to the maximum extent and the efficiency of the simulation testing is improved.

[0055] FIG4 is a flow chart of another embodiment of the automatic driving system testing method of the present application. In this embodiment, the automatic driving system testing method of the present application further includes:

[0056] S30: Archiving the test result data for each completed scenario. For example, the first i groups of test result data for the first i groups of completed scenarios are archived accordingly. For example, the test result data for the first i groups of scenarios may be archived directly, or the first i groups of test result data may be post-processed and the post-processed results of the first i groups of test result data archived.

[0057] S40. When a test interruption occurs, determine the next set of scenarios to be sampled based on the archived test result data for each set of scenarios. Exemplarily, when a test interruption occurs, determine the next set of scenarios to be sampled based on all data in the archived test result data for each set of scenarios. For example, when the sampling and testing of the scenario data for the (i+1)th set of scenarios is interrupted, the next set of scenarios to be sampled may be determined based on the test result data or post-processing results of the first i sets of scenarios that have completed testing.

[0058] Exemplarily, when a test is interrupted, the next set of scenarios to be sampled is determined based on a portion of the archived test result data for each set of scenarios. For example, when the sampling and testing of the scenario data for the (i+1)th set of scenarios is interrupted, the next set of scenarios to be sampled can be determined based on the test result data or post-processing results of the already completed (i)th set of scenarios.

[0059] Determining the next set of scenarios to be sampled using archived test result data helps ensure the correlation between the scenarios whose sampling is resumed and the scenarios that have been sampled and tested previously, avoiding the uncertainty in testing caused by randomly determining the sampling scenarios.

[0060] S50. Sampling scenarios according to the next set of scenarios to be sampled. In order to prevent the test task from being interrupted due to various reasons, thereby causing the test task execution to fail, the present application designs a self-healing mechanism for accelerating the test of the above embodiment. That is, after completing the test of a set of scenarios, the corresponding test result data is archived, so that when a test interruption occurs, the next set of scenarios to be sampled can be determined based on the archived test result data, without having to start sampling and testing from the first set of scenarios again. This ensures test efficiency in the event of a test interruption.

[0061] FIG5 is a flowchart of another embodiment of the automated driving system testing method of the present application. In this embodiment, the method determines the dangerous scenario boundaries of the automated driving system based on test result data of multiple scenarios, including:

[0062] S21. Train a dangerous scene recognition model using the test result data for each set of completed scenarios. For example, the server 100 of this application is pre-configured with a dangerous scene recognition model. During the testing process of the autonomous driving system testing method of this application, convergence training is continuously performed based on the test result data output by multiple simulation simulators to improve the performance of the dangerous scene recognition model in identifying dangerous scenes.

[0063] S22. Determine the dangerous scenario boundary of the autonomous driving system based on the dangerous scenario recognition model. The dangerous scenario boundary refers to the boundary between dangerous scenarios and safe scenarios, i.e., the performance boundary of dangerous scenarios that the autonomous driving system can handle. Operations within this performance boundary will not result in a collision. Operations outside this performance boundary will result in a collision or a risk of collision.

[0064] In this embodiment, the dangerous scene boundary of the autonomous driving system under test is determined based on the dangerous scene recognition model after convergence training, and the performance safety boundary of the autonomous driving system is obtained.

[0065] FIG6 is a flowchart of another embodiment of the autonomous driving system testing method of the present application. In this embodiment, determining the dangerous scene boundaries of the autonomous driving system based on the dangerous scene recognition model includes:

[0066] S221. Determine the parameter range of the logical scenario. Exemplarily, the logical scenario is a parameterized representation of a set of scenarios. Exemplarily, the logical scenario includes multiple influencing factors (for example, a typical logical scenario includes but is not limited to the following influencing factors: basic road information, dynamic targets, and environment), and the influencing factors are described by parameter ranges and distributions, and parameter changes are allowed. The specific values ​​of all parameters are not specified in the logical scenario, but an optional range of values ​​is provided. Its purpose is to allow parameter values ​​to change during the simulation or testing process by selecting specific values ​​within the range, which can be achieved through predefined steps, distribution functions, or randomness. The parameter range and distribution of the influencing factors of the logical scenario constitute a scenario space.

[0067] S222. Input the parameter ranges of the logical scenarios into the dangerous scenario recognition model to obtain danger thresholds for multiple scenarios. For example, a typical logical scenario is described as follows: ① Basic road information: lane width 3-3.5m, curvature radius 0.6-0.9km, traffic sign location 0-200m; ② Dynamic target: traffic jam end 10-200m, traffic jam speed 0-30km / h, main vehicle distance 50-300m, main vehicle speed 80-130km / h; ③ Environment: temperature 10-40°C, rainwater particle size 20-100μm. The parameter ranges for the corresponding logical scenarios may include, but are not limited to, one of the following: basic road information range, curvature radius range, traffic sign location range, dynamic target range, traffic jam speed range, main vehicle distance range, main vehicle speed range, ambient temperature range, and rainwater particle size range. Input the parameter ranges of one or more of the above logical scenarios into the dangerous scenario recognition model to obtain danger thresholds for multiple scenarios. By testing within the input parameter range, the test efficiency is improved. The dangerous scene recognition mode can adopt a Bayesian optimization model based on the trust region or an Actor-Critic model in the reinforcement learning algorithm, or other models that can realize dangerous scene recognition, which is not limited in this application.

[0068] S223. Determine dangerous and safe scenarios among the multiple scenarios based on the danger thresholds of the multiple scenarios. Exemplarily, preset danger thresholds are set for the multiple scenarios. The danger thresholds of the multiple scenarios obtained by the dangerous scenario recognition model based on the parameter range of the logical scenario are compared with the corresponding preset danger thresholds. Scenarios corresponding to danger thresholds exceeding the preset danger thresholds are determined as dangerous scenarios; otherwise, they are determined as safe scenarios.

[0069] S224: Determine a boundary between a dangerous scene and a safe scene in the multiple scenes as a dangerous scene boundary of the autonomous driving system.

[0070] This application uses a hazard scene recognition model trained and fitted based on the test results data of each set of completed test scenarios. This model processes the parameter ranges of logical scenarios and determines hazard thresholds for multiple scenarios. Using statistical analysis methods (for example, by comparing with preset hazard thresholds, but this application is not limited to this), the boundaries of hazard scenes for the autonomous driving system at different speeds can be determined. This allows the autonomous driving system to assess its performance by determining its ability to respond to these hazard scenarios.

[0071] FIG7 is a flow chart of another embodiment of the automated driving system testing method of the present application. In this embodiment, the sampling of the first group of scenes in the grouped scene data sampling is the initial scene sampling. The grouped scene data sampling and testing includes:

[0072] S211: Determine whether the dangerous scene recognition model has converged. For example, during the process of sampling and testing scene data in groups, the dangerous scene recognition model is continuously trained using the test result data of each group of scenes that have been tested, so as to gradually achieve convergence of the dangerous scene recognition model.

[0073] S212: If it is determined that the dangerous scene recognition model has converged, then perform model update scene sampling.

[0074] S213: If it is determined that the dangerous scene recognition model does not converge, normal scene sampling is performed.

[0075] For example, initial scene sampling is random sampling, i.e., scene data from a preset number of scenes are randomly sampled in groups. Model update scene sampling and general scene sampling are performed for scenes determined by the hazard scene recognition model for prediction. This ensures the correlation between scenes, helps accelerate the convergence of the hazard scene recognition model, and improves testing efficiency. Model update scene sampling is used to verify the reliability of the hazard scene recognition model's convergence, while general scene sampling is used for convergence training of the hazard scene recognition model. Once the training results show that the hazard scene recognition model has converged, model update scene sampling is performed to verify the reliability of the hazard scene recognition model's convergence.

[0076] In this embodiment, after the dangerous scene recognition model is determined to have converged, model update scene sampling is further performed to further verify whether the dangerous scene recognition model has indeed converged, ensuring the reliability of the dangerous scene recognition model convergence. If the dangerous scene recognition model is determined not to have converged, normal scene sampling is continued, and the dangerous scene recognition model is trained for convergence.

[0077] In some embodiments, the autonomous driving system testing method of the present application further includes:

[0078] Based on the scene data and test results obtained from the model update scene sampling, verify whether the dangerous scene recognition model has converged; if so, perform a new round of initial scene sampling and testing; if not, perform normal scene sampling.

[0079] For example, after completing one round of convergence of the dangerous scene recognition model, a new round of initial scene sampling and testing is performed, followed by a random determination of the next scene to be tested and the convergence of the dangerous scene recognition model. This cycle repeats until the sampling stop condition is met. For example, this includes: randomly determining the first group of scenes, sampling and testing them; and determining the next group of scenes to be sampled based on the trained dangerous scene recognition model.

[0080] FIG8 is a flow chart of another embodiment of the autonomous driving system testing method of the present application. In this embodiment, the autonomous driving system testing method of the present application further includes:

[0081] S60: Detect the simulation test state of the simulation simulator in real time and obtain speed information and distance information between the vehicle and the external object; illustratively, obtain the vehicle speed and / or the external object speed and the relative distance between the vehicle and the external object.

[0082] S70: When it is determined based on the speed information and distance information that a collision between the ego vehicle and the external object is unlikely, the current simulation simulator test is terminated. For example, the external object may be an external vehicle. The ego vehicle speed and / or the external vehicle speed, as well as the relative distance between the ego vehicle and the external vehicle, are obtained.

[0083] In some embodiments, it includes judging whether a collision will occur between the vehicle and an external vehicle based on the speed information and distance information; when it is judged that a collision is unlikely to occur between the vehicle and the external vehicle based on the speed information and distance information, stopping the test of the current simulation simulator.

[0084] Exemplarily, the speed information includes the vehicle's speed, and the distance information includes the relative distance between the vehicle and the external vehicle. Determining whether a collision will occur between the vehicle and the external vehicle based on the speed and distance information includes: determining the headway time (THW) between the vehicle and the external vehicle based on the vehicle's speed and the relative distance between the vehicle and the external vehicle; if the headway time remains negative for a continuous period of time t1, then determining that a collision between the vehicle and the external vehicle is unlikely; if the headway time remains positive for a continuous period of time t1, then determining whether a collision will occur between the vehicle and the external vehicle based on the THW and TTC between the vehicle and the preceding vehicle. Exemplarily, the continuous period of time t1 can be 3 seconds.

[0085] In some embodiments, it includes: judging whether a collision will occur between the vehicle and an external vehicle based on the speed information and distance information; and stopping the test of the current simulation simulator when it is judged that a collision is unlikely to occur between the vehicle and the external vehicle based on the speed information and distance information.

[0086] Exemplarily, the speed information includes the vehicle's speed and the speed of the external vehicle, and the distance information includes the relative distance between the vehicle and the external vehicle. Determining whether a collision will occur between the vehicle and the external vehicle based on the speed and distance information includes determining a time to collision (TTC) between the vehicle and the external vehicle based on the vehicle's speed, the external vehicle's speed, and the relative distance between the vehicle and the external vehicle. If the time to collision exceeds a first threshold for a continuous time t, then determining that a collision between the vehicle and the external vehicle is unlikely. Exemplarily, the first threshold may be 6 seconds, and the continuous time t may be 3 seconds.

[0087] Exemplarily, the speed information includes the vehicle's speed, and the distance information includes the relative distance between the vehicle and the external vehicle. Determining whether a collision will occur between the vehicle and the external vehicle based on the speed and distance information includes determining a headway time (THW) between the vehicle and the external vehicle based on the vehicle's speed and the relative distance between the vehicle and the external vehicle. If the headway time exceeds a second threshold value for a continuous time t, then determining that a collision between the vehicle and the external vehicle is unlikely. Exemplarily, the second threshold value may be 7 seconds, and the continuous time t may be 3 seconds.

[0088] Exemplarily, the speed information includes the vehicle's speed and the speed of the external vehicle, and the distance information includes the relative distance between the vehicle and the external vehicle. Determining whether a collision will occur between the vehicle and the external vehicle based on the speed and distance information includes determining the time to collision (TTC) and time headway (THW) between the vehicle and the external vehicle based on the vehicle's speed, the external vehicle's speed, and the relative distance between the vehicle and the external vehicle. If the time to collision exceeds a first threshold and the time headway exceeds a second threshold for a continuous time t, then it is determined that a collision between the vehicle and the external vehicle is unlikely. Exemplarily, the first threshold value may be 6 seconds, the second threshold value may be 7 seconds, and the continuous time t may be 3 seconds.

[0089] In this embodiment, multiple simulation test states are monitored in real time. The likelihood of a collision between the ego vehicle and the external vehicle is determined based on their TTC (Time To Collision) and / or THW (Time Headway). The forward and backward relationship of the two vehicles is first determined by determining the positive or negative value of THW. For example, when THW is negative, the ego vehicle is determined to be in front and the external vehicle is behind; when THW is positive, the ego vehicle is determined to be behind and the external vehicle is in front. If THW is continuously negative for a continuous period of time t1, the two vehicles are determined to be unlikely to collide. Simulation of this scenario is terminated without waiting for the end of the current simulation and the next sampled scenario is simulated. If THW is continuously positive for a continuous period of time t1, a first threshold a for TTC is set and a second threshold b for THW is set. If the TTC and / or THW values ​​continuously exceed thresholds a and / or b for a continuous period of time t, the two vehicles are determined to be unlikely to collide. Simulation of this scenario is terminated without waiting for the end of the current simulation and the next sampled scenario is simulated. Among them, TTC = distance between two vehicles / relative speed of two vehicles; THW = distance between two vehicles / speed of own vehicle.

[0090] In this embodiment, the danger level of the scene is judged by setting and judging the two parameters TTC and THW. Once it is determined that there is no danger, the current round of testing is directly ended and the next round of testing is carried out, which greatly improves the test efficiency.

[0091] In some embodiments, the automatic driving system testing method of the present application also includes: detecting abnormal information of the simulation simulator, and restarting the abnormal simulation simulator when abnormal information of the simulation simulator is detected. In the process of testing the simulation simulator, in order to prevent task interruption due to various reasons, thereby causing task execution failure, the present embodiment designs a self-healing mechanism of the simulation simulator. When the simulation simulator detects that the environment is abnormal due to software or hardware (for example, the software does not cut into automatic driving, the simulation simulator does not build a sampling scene, etc.), it will send an abnormal flag, save the sampling scene, and automatically restart. After restarting, it will detect whether the software and hardware status of the simulation simulator is restored, test whether the hardware IP is connected, and whether the software environment is normal. If not, it will restart again. If normal, it will read the saved sampling scene, build the scene, and perform scene testing.

[0092] In some embodiments, scene data of multiple scenes are sampled, and the scene data of the multiple scenes are sent to multiple simulation simulators for distributed testing, including: a. sampling scene data of multiple scenes; b. determining idle simulation simulators among multiple simulation simulators; c. allocating at least part of the scene data of the multiple scenes to the idle simulation simulators for testing. Exemplarily, allocating at least part of the scene data of the multiple scenes to the idle simulation simulators for testing includes: determining the number of idle simulation simulators and selecting a corresponding number of scene data; establishing a one-to-one correspondence between each simulation simulator and a corresponding number of scene data; sending each scene data in the scene data of the corresponding data to the corresponding idle simulation simulator for testing (for example, establishing a mapping relationship between each simulator tag and each scene tag based on the simulator tag of each simulation simulator and the scene tag of each scene data in the scene data of the corresponding data).

[0093] This embodiment establishes a mapping relationship between simulators and scenario data, clearly defining the simulator corresponding to each scenario. This facilitates post-processing of test result data based on different simulators, avoiding interference between test result data from different simulators. It also constantly detects and allocates scenario data to idle simulators, improving the efficiency of simulator usage.

[0094] FIG9 is a flowchart of another embodiment of the automatic driving system testing method of the present application. In this embodiment, the automatic driving system testing method of the present application includes:

[0095] Step 1: Determine whether it is in the continuous optimization mode. If it is not in the continuous optimization mode, perform scene sampling; the optimization object refers to the dangerous scene recognition model configured in the server;

[0096] Step 2: Integrate the sampled scene data and put it into the simulator scene scheduler;

[0097] Step 3: The simulator scenario scheduler is responsible for a unified scenario allocation strategy, allocating scenarios to multiple simulation simulators integrated with autonomous driving domain controllers for testing;

[0098] Step 4: Multiple simulation simulators conduct scenario testing and return the test results to the server;

[0099] Step 5: Collect the test results and perform post-processing;

[0100] Step 6: Determine whether the sampling stop condition has been met. If so, the result is output. If not, the existing data is archived and step 1 is continued. The sampling stop condition can be the number of samples (e.g., 10,000 or 20,000). The specific setting of the sampling stop condition is related to the model algorithm and is not limited in this application. For example, it can also be the model output result. When the model output result meets the preset conditions, the sampling stop condition is met.

[0101] Step seven: When the sampling stop condition is reached and the test is completed, the test results are fitted, and the danger boundaries under different conditions are output. The danger boundaries are statistically analyzed to obtain the performance safety boundary of the autonomous driving system.

[0102] As shown in Figure 10, it is a flow chart of another embodiment of the automatic driving system testing method of the present application. In this embodiment, the initial scene sampling is first performed, and the scene data obtained by sampling (for example, environmental data (for example, how heavy the rain is, how heavy the snow is, visibility), lane lines, lane curvature, the speed of the vehicle itself, the speed of the adjacent vehicle, etc.) are distributed. According to the operating status of each simulation simulator, it is reasonably distributed to each simulation simulator to perform scene testing of the automatic driving system. After the scene test in the simulation simulator is completed, the result is saved and returned to the dangerous scene recognition model of the server for initial sampling post-processing (for example, sorting, converting, data integration and various data processing logics of the sampled scene data, etc., and converting them into a language that the simulation simulator can understand. For example, the scene data obtained by sampling is converted into a format that the simulation simulator can process. The data integration is, for example, to classify the scene data of multiple scenes sampled, and each scene corresponds to a set of scene data, so that the corresponding scene data can be sent to the corresponding simulation simulator for testing according to the scene). Determine whether the stopping condition is reached. If not, determine whether the dangerous scene recognition model has converged (used to evaluate the test results of the simulation simulator, determine whether the vehicle will collide, and obtain all dangerous scene data. For example, scene data of 5 scenes are sampled, of which 2 are dangerous scenes and 3 are safe scenes. The purpose of training the dangerous scene recognition model to converge is that the model can quickly and accurately identify 2 dangerous scenes in the end). If the dangerous scene recognition model has not converged, perform normal sampling, continue scene sampling, and according to the status of the simulation simulator, assign it to an empty simulation simulator for scene testing, return the results to the dangerous scene recognition model, and perform model prediction (the test results of multiple simulation simulators are input to the dangerous scene recognition model, and the dangerous scene recognition model has been training. Then, based on the prediction convergence results of the dangerous scene recognition model, a new round of sampling is performed to predict the next dangerous scene and perform scene sampling, so that scene sampling can be more targeted and avoid the blindness of scene sampling.The initial scene sampling can be a random sampling of 5 scenes, or Latin hypercube sampling, a probability sampling in a space, for example, a range of a scene (for example, a vehicle speed of 0-130km / h), sampling within this range), and judging whether the stopping condition is reached. If not, continue sampling, and judge whether the dangerous scene recognition model converges. If converged, perform model update sampling, and distribute the scene data to the simulation simulator for scene testing, and return the results to the dangerous scene recognition model for model prediction, and judge whether the stopping condition is reached. If not, judge whether the model converges. If the model converges, perform model update sampling, and then distribute the sampled scene to the simulator for scene testing. Repeat the above process until the stopping condition is reached, output the sampling scene and test results, and fit the test results to obtain the dangerous scene boundary of the autonomous driving system. Exemplarily, the test result data of each set of scenes that have been tested are used to train the dangerous scene recognition model; the dangerous scene boundary of the autonomous driving system is determined based on the dangerous scene recognition model. Among them, determining the dangerous scene boundaries of the autonomous driving system based on the dangerous scene recognition model includes: determining the parameter range of the logical scene; inputting the parameter range of the logical scene into the dangerous scene recognition model to obtain the dangerous threshold values ​​of multiple scenes; and determining the dangerous scenes and safe scenes in the multiple scenes according to the dangerous threshold values ​​of the multiple scenes.

[0103] The embodiment of the present application is described as follows with reference to FIG9 and FIG10 :

[0104] For step one, in order to prevent the test task from being interrupted due to various reasons, thereby causing the test task to fail, a self-healing mechanism for accelerated testing is designed. First, it is determined whether it is in the continue optimization mode: if the server's self-healing mechanism determines that the task is in an interrupted state, the flag of the dangerous scene recognition model in the server is changed to the continue optimization mode. After it is determined to be in the continue optimization mode, the previous archived data will be read and input into the sampling process. According to the archived data, the scene to be sampled next time is predicted, and multiple sampling scenes are obtained, and the task before the connection interruption is continued. If it is in the non-continue optimization mode, direct prediction sampling is performed to obtain multiple sampling scenes. In normal continuous sampling, for example, an interruption occurs between the third and fourth rounds of sampling (caused by network disconnection or power outage, etc.). At this time, if the test is retested (the test has been done for 1 hour before), the previous test will be wasted. The method of the present application has already stored the sampling and test results of the first three rounds. The test results of the first three rounds or the third round can be input into the dangerous scene recognition model to predict the sampling scene of the fourth round, thereby realizing the breakpoint continuation of the test and improving the test efficiency.

[0105] Scene sampling is divided into initial sampling, normal sampling and model update sampling. Initial sampling occurs in two situations: Situation 1. When the dangerous scene recognition model starts running for the first time; Situation 2. After the model is updated and sampled, and it is verified that the dangerous scene recognition model has indeed converged. When the dangerous scene recognition model starts running for the first time, initial sampling is performed, and the scene data of the initial sampling is placed in the simulation simulator for simulation testing, and the test results are post-processed, and the model is converged (that is, the post-processed test results are used to train the dangerous scene recognition model). Then, it is judged whether the dangerous scene recognition model has converged. If it has not converged, normal sampling is continued, and the sampled scene is placed in the simulation simulator for simulation testing, and the test results are post-processed; if it has converged, model update sampling is performed, and the sampled scene data is placed in the simulation simulator for simulation testing, and the test results are post-processed, and the new dangerous scene recognition model is converged, and it is judged whether the model has converged. This cycle is repeated until the test stop condition is reached and the test stops.

[0106] Model update sampling involves performing another round of sampling after model convergence to ensure more accurate model convergence and ensure that the model has indeed converged. If the dangerous scene recognition model has not converged after model update sampling, normal sampling will continue until the dangerous scene recognition model truly converges, avoiding misjudgment of dangerous scene recognition model convergence.

[0107] For step 2, the sampled scenario data are integrated, labeled and sorted (e.g. the first scenario is labeled "scenarioA"), converted into a data format suitable for the simulation simulator, and input into the simulator scenario scheduler.

[0108] For step 3, the simulator scenario scheduler can be integrated into the functional module of the server and specifically perform the following steps:

[0109] The simulator scenario scheduler receives scenario data from multiple scenarios and aggregates them into a set of sampled scenarios;

[0110] Determine whether there is any scenario data that has not been loaded into the simulator for testing;

[0111] If yes, then determine whether there is an idle simulator

[0112] If yes, then obtain scenario parameters equal to the number of idle simulation simulators and call the simulation simulator to perform simulation testing;

[0113] If not, determine whether there is a simulation simulator to complete the simulation;

[0114] If yes, the result is stored in the test result set, and the idle simulator list is updated to further determine whether there is any scene data that has not been loaded into the simulator for testing;

[0115] If not, determine whether all scenarios have completed testing;

[0116] If yes, get the test result set;

[0117] If not, jump to step: determine whether there is a simulation simulator to complete the simulation.

[0118] In some embodiments, it is determined which of multiple simulation simulators is in an idle state. If it is determined that simulation simulator 1 is in an idle state, the scene "scenario A" is marked with the simulator label "hil1", and the scenario data is placed in simulation simulator 1 for scenario testing. Otherwise, it is detected and determined whether any simulation simulator has completed simulation (for example, if all simulation simulators are in working state, wait until which simulation simulator is in an idle state first, then mark the scenario data with the corresponding simulator number, and place it in the corresponding simulation simulator for testing).

[0119] In some embodiments, when allocating multiple sampling scenarios to multiple simulation simulators, a scenario allocation process is designed. First, all sampling scenarios are marked. When the initial scenario is sampled, all simulation simulators are in a non-operating state. The scene data obtained by sampling are allocated to different simulation simulators for scenario testing. The running status of the simulation simulator is detected in real time. If the scenario test in simulation simulator 2 has been completed, it is determined whether there are sampling scenarios that have not been tested by the simulation simulator. If there are sampling scenarios that have not been tested by the simulation simulator, the sampling scenario is marked as "hil2" and the scene data of the sampling scenario is allocated to simulation simulator 2 for simulation testing. If all sampling scenarios have been tested, all test results are post-processed, and the test results are saved for the next round of scenario sampling.

[0120] For step four, during the testing of the simulation simulator, a self-healing mechanism of the simulation simulator is designed to prevent task interruption due to various reasons, thereby causing task execution failure. When the simulation simulator detects an environmental abnormality due to software or hardware (such as the software does not enter autonomous driving, the simulation simulator does not build a sampling scene, etc.), it will send an abnormal flag, save the sampling scene, and restart automatically. After restarting, it will check whether the software and hardware status of the simulation simulator has recovered, test whether the hardware IP is connected, and whether the software environment is normal. If it is not normal, restart again; if it is normal, read the saved sampling scene, build the scene, and perform scene testing. After the simulation simulator test is completed, the test results are marked with the corresponding scene label and simulator label, and output through the communication protocol.

[0121] In order to speed up the test efficiency, during the test process, the present application detects the simulation test status of the simulation simulator in real time. For example, the possibility of a collision between the two vehicles is judged based on the TTC and THW of the two vehicles. First, the front-to-back relationship of the two vehicles is judged by judging the positive and negative of THW. Then the first threshold a of TTC is set, and the second threshold b of THW is set. If the values ​​of TTC and THW continue to exceed the thresholds a and b within the continuous time t, it is judged that the two vehicles are unlikely to collide. There is no need to wait until the simulation of this scene is completed. The simulation of this scene can be stopped directly and the simulation of the next sampling scene can be carried out.

[0122] For step five, the test results are aggregated, extracted and sorted based on the scenario and simulator tags. These results are then integrated with all test results. Based on the test results indicating whether the two vehicles collided, the relative speed, and the relative distance between the two vehicles, a scenario hazard threshold is designed to determine the hazard level of the scenario. The model is then fitted based on this hazard level and converged. It should be noted that the above example uses only the two-vehicle collision scenario; scenarios involving collisions between the vehicle and an external object are also possible, and this application does not limit this.

[0123] For step six, determine whether the sampling stop condition is met. If so, output the result; if not, archive the existing data and continue with step one: determine whether the model has converged. If so, perform model update sampling in step one. If the model has not converged, perform normal sampling in step one.

[0124] For step 7, after the acceleration test is completed and the test results are obtained, the boundary model is predicted using the test results: 7.1. Process the test results (e.g., remove noise) and use the processed test results to optimize the trained dangerous scenario recognition model. 7.2. Determine the parameter range of the logical scenarios and input these scenario parameter ranges into the trained dangerous scenario recognition model. The dangerous scenario recognition model calculates the danger threshold for all scenarios and, based on the danger threshold, classifies the danger level of different scenarios into dangerous and safe scenarios. The boundary between dangerous and safe scenarios represents the performance boundary of dangerous scenarios that the autonomous driving system can handle. Scenarios within this performance boundary are collision-free, while scenarios outside this performance boundary may result in collisions or pose a risk of collision. 7.3. Assuming the danger threshold ranges from 0 to 1, with 1 representing safety and 0 representing collision, dangerous scenarios with a danger threshold between 0 and 0.1 are selected to form the dangerous boundary scenarios for the autonomous driving system and serve as the dangerous boundary of the autonomous driving system. By observing dangerous scenarios on the danger boundary, it is possible to intuitively identify the extreme scenarios that the autonomous driving system can handle and clearly understand the capabilities of the autonomous driving system.

[0125] This application uses a simulator with an integrated autonomous driving domain controller for simulation testing, which is more realistic than simulation testing using pure software. When finding the same number of dangerous scenarios, this application needs to simulate 3000 rounds (50 hours) using a single simulation simulator, while a traversal test with a fixed step length requires simulation of 9000+ rounds (150+ hours), which is more efficient than a traversal test with a fixed step length. At the same time, a large number of hazardous scenarios will be missed in a traversal test with a fixed step length compared to an accelerated test. If three simulators are used for testing, the efficiency of the simulation test is 3 times higher than that of a single simulation simulator, and 9 times higher than that of a traversal test with a fixed step length.

[0126] In some embodiments, experiments have shown that, under the same test rounds, the proportion of dangerous scenarios in the test results of the present application is twice that of the fixed-step traversal test. As shown in Figures 11a and 11b, there are comparison diagrams of the experimental results of the autonomous driving test method of the present application and the fixed-step traversal test method. The experiment tested the scenario of the vehicle in front cutting into the lane of the vehicle while the vehicle was driving. Specifically, the test results were tested under many scenarios with a vehicle speed of 20-80km / h, a vehicle speed of 20-80km / h, and a cutting distance between the vehicle in front and the vehicle in the range of -1m to 50m (i.e., the relative distance between the vehicle in front and the vehicle when the vehicle in front cuts into the lane where the vehicle is located). Among them, Figure 11a is a schematic diagram of the experimental results of the fixed-step traversal test method; Figure 11b is a schematic diagram of the experimental results of the autonomous driving system test method of the present application. In this embodiment, 2662 rounds of testing were performed using both the method of the present application and the convenient method of fixed step length. Among them, the proportion of dangerous scenes determined by the fixed-step traversal test is 40.1%, and the proportion of dangerous scenes determined by the test method of this application is 79.5%. Among them, the proportion of dangerous scenes is: the proportion of scenes below the scene danger threshold (dangerous scenes, i.e., the dark dots in the attached figure) to the total sampled scenes. Through the comparison of the above experimental results, it can be seen that the method of this application can more efficiently identify more dangerous scenes.

[0127] In some embodiments, the present application provides an autonomous driving system testing system, comprising: a server and multiple simulation simulators, each of which is integrated with an autonomous driving domain controller; the server is configured to sample scene data of multiple scenes, and send the scene data of the multiple scenes to multiple simulation simulators for distributed testing; the multiple simulation simulators are configured to perform distributed testing on the scene data of the multiple scenes to obtain test result data of the multiple scenes; the server is also configured to determine the dangerous scene boundaries of the autonomous driving system based on the test result data of the multiple scenes.

[0128] In some embodiments, scene data of multiple scenarios are sampled, and the scene data of the multiple scenarios are sent to multiple simulation simulators for distributed testing to obtain test result data of the multiple scenarios, including: performing scene data sampling and testing in groups to obtain test result data for each group of scenarios that completes the test, and the number of each group of sampled scenes is determined according to the number of the multiple simulation simulators; if the sampling stop condition is met after sampling the scene data of the nth group of scenes, the test result data corresponding to the first n groups of scenes are determined as the test result data of the multiple scenarios.

[0129] In some embodiments, the server is further configured to: archive the test result data of each group of scenarios that have completed the test; when a test interruption occurs, determine the next group of scenarios to be sampled based on the archived test result data of each group of scenarios; and perform scene sampling based on the next group of scenarios to be sampled.

[0130] In some embodiments, the dangerous scene boundaries of the autonomous driving system are determined based on the test result data of the multiple scenarios, including: using the test result data of each group of scenarios that have completed the test to train a dangerous scene recognition model; and determining the dangerous scene boundaries of the autonomous driving system based on the dangerous scene recognition model.

[0131] In some embodiments, determining the dangerous scene boundary of the autonomous driving system based on a dangerous scene recognition model includes: determining the parameter range of the logical scene; inputting the parameter range of the logical scene into the dangerous scene recognition model to obtain the dangerous threshold value of multiple scenes; determining the dangerous scenes and safe scenes in the multiple scenes according to the dangerous threshold value of the multiple scenes; and determining the boundary between the dangerous scenes and the safe scenes in the multiple scenes as the dangerous scene boundary of the autonomous driving system.

[0132] In some embodiments, the sampling of the first group of scenes in the group scene data sampling is the initial scene sampling; the group scene data sampling and testing includes: determining whether the dangerous scene recognition model converges; if so, performing model update scene sampling; if not, performing normal scene sampling.

[0133] In some embodiments, the server is further configured to: verify whether the dangerous scene recognition model has converged based on the scene data and test results obtained from the model update scene sampling; if so, perform a new round of initial scene sampling and testing; if not, perform normal scene sampling.

[0134] In some embodiments, scene data sampling and testing are performed in groups, including: randomly determining a first group of scenes and performing scene sampling and testing; and determining the next group of scenes to be sampled based on a trained dangerous scene recognition model.

[0135] In some embodiments, the server is further configured to: detect the simulation test status of the simulation simulator in real time, obtain speed information and distance information between the vehicle and the external object; when it is judged that a collision is unlikely to occur between the vehicle and the external object based on the speed information and distance information, stop the current simulation simulator test.

[0136] In some embodiments, the server is further configured to: detect abnormal information of the simulation simulator, and when abnormal information of the simulation simulator is detected, restart the simulation simulator where the abnormality occurs.

[0137] In some embodiments, scene data of multiple scenes are sampled and sent to multiple simulation simulators for distributed testing, including: sampling scene data of multiple scenes; determining idle simulation simulators among multiple simulation simulators; and allocating at least part of the scene data of the multiple scenes to the idle simulation simulators for testing.

[0138] The automated driving system testing system of the embodiment of the present application described above can be used to implement the automated driving system testing method of the embodiment of the present application. The details of the related embodiments can be mutually referenced and can accordingly achieve the technical effects achieved by the automated driving system testing method of the embodiment of the present application described above. These details will not be repeated here. In the embodiment of the present application, the relevant functional modules can be implemented using a hardware processor.

[0139] In some embodiments, the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in any embodiment of the present application.

[0140] In some embodiments, the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, characterized in that when the computer program / instruction is executed by a processor, the steps of the method described in any embodiment of the present application are implemented.

[0141] In some embodiments, the present application provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the method described in any embodiment of the present application are implemented.

[0142] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of combined actions, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0143] Figure 12 is a schematic diagram of the hardware structure of a computer device for executing an autonomous driving system testing method according to another embodiment of the present application. As shown in Figure 12, the device includes one or more processors 1210 and a memory 1220. Figure 12 uses one processor 1210 as an example. The device for executing the autonomous driving system testing method may also include an input device 1230 and an output device 1240. The processor 1210, memory 1220, input device 1230, and output device 1240 may be connected via a bus or other means. Figure 12 uses a bus connection as an example. Memory 1220, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the program instructions / modules corresponding to the autonomous driving system testing method in the embodiment of the present application. By executing the non-volatile software programs, instructions, and modules stored in memory 1220, processor 1210 executes various server functions and data processing, thereby implementing the autonomous driving system testing method in the aforementioned method embodiment.

[0144] Memory 1220 may include a program storage area and a data storage area. The program storage area may store an operating system and application programs required for at least one function; the data storage area may store data generated based on the use of the automated driving system test device. Furthermore, memory 1220 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, memory 1220 may optionally include memory remote from processor 1210. Such remote memory may be connected to the automated driving system test device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0145] The input device 1230 can receive input digital or character information and generate signals related to user settings and function control of the automated driving system testing device. The output device 1240 may include a display device such as a display screen. The one or more modules are stored in the memory 1220 and, when executed by the one or more processors 1210, perform the automated driving system testing method described in any of the above method embodiments.

[0146] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0147] The computer devices of the embodiments of the present application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by having mobile communication functions and are mainly aimed at providing voice and data communications. Such terminals include: smart phones (such as iPhones), multimedia phones, functional phones, and low-end phones. (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. Such terminals include: PDAs, MIDs, and UMPC devices, such as iPads. (3) Servers: Devices that provide computing services. The server's composition includes a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, manageability, etc. (4) Other electronic devices with data interaction functions.

[0148] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0149] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, or of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiment.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for testing an autonomous driving system, comprising: Sampling scene data of multiple scenes, and sending the scene data of the multiple scenes to multiple simulation simulators for distributed testing, to obtain test result data of the multiple scenes; The multiple simulation simulators are all integrated with an autonomous driving domain controller; Based on the test result data of the multiple scenarios, dangerous scenario boundaries of the autonomous driving system are determined.

2. The method according to claim 1, characterized in that Sampling scene data of multiple scenes, and sending the scene data of the multiple scenes to multiple simulation simulators for distributed testing, to obtain test result data of the multiple scenes, including: Sampling and testing scenario data in groups to obtain test result data for each group of scenarios that have completed the test, wherein the number of sampled scenarios in each group is determined according to the number of the multiple simulation simulators; If the sampling stop condition is met after sampling the scene data of the nth group of scenes, the test result data corresponding to the first n groups of scenes are determined as the test result data of the multiple scenes.

3. The method according to claim 2, characterized in that Also includes: Archive the test result data of each set of scenarios that have been tested; When a test interruption occurs, the next set of scenarios to be sampled is determined based on the archived test result data for each set of scenarios; Scene sampling is performed according to the next set of scenes to be sampled.

4. The method according to claim 3, characterized in that Determining a dangerous scenario boundary of the autonomous driving system based on the test result data of the multiple scenarios includes: The test result data of each set of scenarios that have been tested are used to train the dangerous scenario recognition model; Determine the dangerous scene boundaries of the autonomous driving system based on the dangerous scene recognition model.

5. The method according to claim 4, characterized in that Determine the dangerous scene boundaries of the autonomous driving system based on the dangerous scene recognition model, including: Determine the parameter range of the logical scenario; Inputting the parameter range of the logical scenario into the dangerous scenario recognition model to obtain dangerous thresholds of multiple scenarios; Determining dangerous scenes and safe scenes among the multiple scenes according to the danger thresholds of the multiple scenes; Determine a boundary between a dangerous scene and a safe scene in the multiple scenes as a dangerous scene boundary of the autonomous driving system.

6. The method according to claim 4, characterized in that The sampling of the first group of scenes in the grouped scene data sampling is the initial scene sampling; The grouping performs scene data sampling and testing, including: Determining whether the dangerous scene recognition model has converged; If yes, perform model update scene sampling; If not, normal scene sampling is performed.

7. The method according to claim 4, characterized in that Also includes: Verify whether the dangerous scene recognition model has converged based on the scene data obtained by sampling the model update scene and the test results; If yes, a new round of initial scene sampling and testing is carried out; If not, normal scene sampling is performed.

8. The method according to claim 4, characterized in that Sampling and testing scene data in groups, including: Randomly determine the first group of scenarios and conduct scenario sampling and testing; The next set of scenes to be sampled is determined based on the trained dangerous scene recognition model.

9. The method according to any one of claims 1 to 8, characterized in that Also includes: Real-time detection of the simulation test status of the simulator to obtain the speed and distance information between the vehicle and external objects; When it is determined based on the speed information and the distance information that a collision between the vehicle and the external object is unlikely to occur, the test of the current simulation simulator is stopped.

10. The method according to any one of claims 1 to 8, characterized in that Also includes: Detect abnormal information of the simulation simulator, and when abnormal information of the simulation simulator is detected, restart the simulation simulator where the abnormality occurs.

11. The method according to any one of claims 1 to 8, characterized in that Sampling scene data of multiple scenes and sending the scene data of the multiple scenes to multiple simulation simulators for distributed testing, including: Sampling scene data of multiple scenes; determining an idle simulator among a plurality of simulators; At least part of the scenario data of the plurality of scenarios is distributed to an idle simulation simulator for testing.

12. An autonomous driving system testing system, comprising: A server and multiple simulation simulators, each of which is integrated with an autonomous driving domain controller; The server is configured to sample scenario data of a plurality of scenarios and send the scenario data of the plurality of scenarios to a plurality of simulation simulators for distributed testing; The multiple simulation simulators are configured to perform distributed testing on the scenario data of the multiple scenarios to obtain test result data of the multiple scenarios; The server is further configured to determine the dangerous scene boundaries of the autonomous driving system based on the test result data of the multiple scenes.

13. The system according to claim 12, wherein: Sampling scene data of multiple scenes, and sending the scene data of the multiple scenes to multiple simulation simulators for distributed testing, to obtain test result data of the multiple scenes, including: Sampling and testing scenario data in groups to obtain test result data for each group of scenarios that have completed the test, wherein the number of sampled scenarios in each group is determined according to the number of the multiple simulation simulators; If the sampling stop condition is met after sampling the scene data of the nth group of scenes, the test result data corresponding to the first n groups of scenes are determined as the test result data of the multiple scenes.

14. The system according to claim 13, wherein: The server is further configured to: archive the test result data of each group of scenarios that have completed the test; when a test interruption occurs, determine the next group of scenarios to be sampled based on the archived test result data of each group of scenarios; and perform scene sampling based on the next group of scenarios to be sampled.

15. The system according to claim 14, wherein: Determining a dangerous scenario boundary of the autonomous driving system based on the test result data of the multiple scenarios includes: The test result data of each set of scenarios that have been tested are used to train the dangerous scenario recognition model; Determine the dangerous scene boundaries of the autonomous driving system based on the dangerous scene recognition model.

16. The system according to claim 15, wherein: Determine the dangerous scene boundaries of the autonomous driving system based on the dangerous scene recognition model, including: Determine a parameter range for a logical scenario; input the parameter range for the logical scenario into the dangerous scenario recognition model to obtain dangerous thresholds for multiple scenarios; determine dangerous scenarios and safe scenarios in the multiple scenarios based on the dangerous thresholds for the multiple scenarios; and determine a boundary between the dangerous scenarios and the safe scenarios in the multiple scenarios as a dangerous scenario boundary for the autonomous driving system.

17. The system according to claim 15, wherein: The sampling of the first group of scenes in the grouped scene data sampling is the initial scene sampling; The grouping performs scene data sampling and testing, including: Determine whether the dangerous scene recognition model has converged; if so, perform model update scene sampling; if not, perform normal scene sampling.

18. The system according to claim 15, wherein: The server is also configured to: Based on the scene data and test results obtained from the model update scene sampling, verify whether the dangerous scene recognition model has converged; if so, perform a new round of initial scene sampling and testing; if not, perform normal scene sampling.

19. The system according to claim 15, wherein: Sampling and testing scene data in groups, including: Randomly determine the first group of scenarios and conduct scenario sampling and testing; The next set of scenes to be sampled is determined based on the trained dangerous scene recognition model.

20. The system according to any one of claims 12 to 19, characterized in that The server is also configured to: Real-time detection of the simulation test status of the simulator to obtain the speed and distance information between the vehicle and external objects; When it is determined based on the speed information and the distance information that a collision between the vehicle and the external object is unlikely to occur, the test of the current simulation simulator is stopped.

21. The system according to any one of claims 12 to 19, characterized in that The server is further configured to detect abnormal information of the simulation simulator, and when abnormal information of the simulation simulator is detected, restart the abnormal simulation simulator.

22. The system according to any one of claims 12 to 19, characterized in that Sampling scene data of multiple scenes and sending the scene data of the multiple scenes to multiple simulation simulators for distributed testing, including: Sampling scenario data of a plurality of scenarios; determining an idle simulation simulator among a plurality of simulation simulators; and allocating at least a portion of the scenario data of the plurality of scenarios to the idle simulation simulator for testing.

23. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 11.

24. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

25. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images