An automatic sampling method, device and equipment for simulation testing and a storage medium

Abstract

The present disclosure relates to an automatic sampling method, device and equipment for simulation testing and a storage medium, comprising: obtaining a test scene and a parameter space of automatic driving simulation testing, determining a first parameter value according to the test scene, running the test scene to obtain a scene running result, using at least two candidate sampling methods to sample the test scene at least once to obtain a corresponding sampling data set, obtaining effective data of the candidate sampling method according to the sampling data set, comparing the effective data corresponding to the candidate sampling method to obtain a target sampling method, and completing the test of the test scene according to the target sampling method. The present disclosure can obtain an optimal sampling method from several sampling methods, and realize simulation testing of the automatic driving system through the optimal sampling method, so as to find more safety problems of the automatic driving system, ensure the effectiveness of sampling, and improve the efficiency of simulation testing.

Description

Technical Field

[0001] This disclosure relates to the field of simulation testing technology, and in particular to an automatic sampling method, apparatus, electronic device, and computer-readable storage medium for simulation testing. Background Technology

[0002] In related technologies, comprehensive scenario data is required for simulation testing and analysis to fully test autonomous driving systems. Testing all massive amounts of scenario data is extremely time-consuming and resource-intensive. Therefore, we use enhanced testing methods to test autonomous driving systems, avoiding unnecessary scenario testing and resource consumption, and helping users identify problems more quickly.

[0003] Parameter sampling (hereinafter referred to as sampling) is an enhancement testing method. A complete scenario is composed of multiple elements, and the combination of parameters contained in each element is explosive. Through parameter sampling, it is possible to quickly search the explosive parameter space, identify key scenarios, and thus quickly, efficiently, and massively generate key simulation scenarios, expose problems in autonomous driving systems, and reduce the number of actual test scenarios.

[0004] Currently, when conducting autonomous driving simulation tests, it is usually necessary to manually select one or more sampling methods to sample parameters and run scenarios based on the scenario and parameter space. This method relies on human experience and cannot determine the sampling method with better results, resulting in the need to repeat different sampling multiple times, which leads to low testing efficiency.

[0005] Therefore, there is an urgent need for an automatic sampling method that can achieve better sampling results, ensure the validity of sampling data, and thus improve sampling efficiency. Summary of the Invention

[0006] To overcome the problems existing in related technologies, this disclosure provides an automatic sampling method, apparatus, electronic device, and computer-readable storage medium for simulation testing.

[0007] According to a first aspect of the present disclosure, an automatic sampling method for simulation testing is provided, comprising: acquiring a test scenario and parameter space for autonomous driving simulation testing, wherein the parameter space includes a range of parameter values ​​corresponding to the test scenario; determining a first parameter value based on the test scenario, and running the test scenario based on the first parameter value to obtain a scenario running result; employing at least two candidate sampling methods to sample the test scenario at least once to obtain a corresponding sampling data set; obtaining valid data for the candidate sampling methods based on the sampling data set; comparing the valid data corresponding to the candidate sampling methods to obtain a target sampling method; and completing the testing of the test scenario according to the target sampling method.

[0008] In some embodiments, determining a first parameter value based on the test scenario, running the test scenario based on the first parameter value, and obtaining the scenario running result includes: obtaining historical test data of the test scenario; selecting parameter values ​​that correspond to historical valid data that meet preset conditions based on the historical test data; using the parameter value as the first parameter value; running the test scenario based on the first parameter value; and obtaining the corresponding scenario running result, wherein the scenario running result includes the first parameter value and initial valid data.

[0009] In some embodiments, employing at least two candidate sampling methods to sample the test scenario at least once to obtain a corresponding sampled data set includes: setting a maximum number of samplings for each candidate sampling method; selecting a set of parameter values ​​in the parameter space using the candidate sampling method; running the test scenario based on the parameter values ​​and the scenario execution results to obtain a set of sampled data; and repeatedly sampling using the candidate sampling method until the maximum number of samplings is reached to obtain a sampled data set for the candidate sampling method.

[0010] In some embodiments, obtaining valid data for the candidate sampling method based on the sampled data set includes: determining abnormal driving data during the operation of the test scenario based on the sampled data set, wherein the abnormal driving data includes collision data and violation data; counting the frequency of occurrence of abnormal driving data in the sampled data set; and determining valid data for the candidate sampling method based on the frequency and the abnormal driving data.

[0011] In some embodiments, comparing the valid data corresponding to the candidate sampling methods to obtain the target sampling method includes: comparing the valid data to determine the candidate sampling methods whose valid data reaches a preset threshold; and obtaining the target sampling method based on the candidate sampling methods whose valid data reaches the preset threshold.

[0012] In some embodiments, comparing the effective data of the candidate sampling methods to obtain the target sampling method includes: acquiring the frequency of occurrence of the collision data and the violation data to obtain the collision frequency and violation frequency; comparing the candidate sampling methods according to the collision frequency and violation frequency to obtain a comparison result; and acquiring the candidate sampling method whose collision frequency reaches a preset frequency according to the comparison result as the target sampling method.

[0013] In some embodiments, obtaining a target sampling method based on the candidate sampling methods where the effective data reaches a preset threshold includes: when there are at least two candidate sampling methods where the effective data reaches the preset threshold, recording the corresponding candidate sampling method as a pending sampling method; obtaining the sampling data set and the effective data of the pending sampling method, and using them as feature values ​​and label data respectively; training the feature values ​​and label data using an automatic ensemble learning method to obtain a sampling determination model; setting a maximum number of calls, and calling the sampling determination model through the pending sampling method to obtain target effective data; calculating the sum of the effective data of the pending sampling method and the target effective data; and comparing the pending sampling methods based on the sum to obtain the target sampling method.

[0014] According to a second aspect of the present disclosure, an automatic sampling device for simulation testing is provided, comprising: a test scenario acquisition module, configured to acquire a test scenario and parameter space for autonomous driving simulation testing, wherein the parameter space includes a range of parameter values ​​corresponding to the test scenario; a test scenario execution module, configured to determine a first parameter value based on the test scenario, and execute the test scenario based on the first parameter value to obtain a scenario execution result; a test scenario sampling module, configured to use at least two candidate sampling methods to sample the test scenario at least once to obtain a corresponding sampling data set; an effective data acquisition module, configured to acquire effective data of the candidate sampling methods based on the sampling data set; an effective data comparison module, configured to compare the effective data corresponding to the candidate sampling methods to obtain a target sampling method; and a testing module, configured to complete the testing of the test scenario according to the target sampling method.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement an automatic sampling method for simulation testing provided in the first aspect of the present disclosure.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of an automatic sampling method for simulation testing provided in the first aspect of the present disclosure.

[0017] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: by acquiring the test scenario and parameter space of autonomous driving simulation test, and determining the first parameter value according to the test scenario, the test scenario is run to obtain the scenario running results. At least two candidate sampling methods are used to sample the test scenario at least once to obtain the corresponding sampling data set. The effective data of the candidate sampling methods is obtained according to the sampling data set. The effective data corresponding to the candidate sampling methods are compared to obtain the target sampling method. The test scenario is tested according to the target sampling method. The preferred sampling method can be obtained from several sampling methods, and the simulation test of the autonomous driving system is realized through the preferred sampling method, so as to discover more safety problems of the autonomous driving system, ensure the effectiveness of sampling, and improve the efficiency of simulation test.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] Figure 1 This is a flowchart illustrating an automatic sampling method for simulation testing according to an exemplary embodiment.

[0021] Figure 2 This is a flowchart illustrating a method for obtaining the results of a scenario execution according to an exemplary embodiment.

[0022] Figure 3 This is a flowchart illustrating a method for acquiring a sampled data set according to an exemplary embodiment.

[0023] Figure 4 This is a flowchart illustrating an effective data acquisition method according to an exemplary embodiment.

[0024] Figure 5 This is a flowchart illustrating an effective data comparison method according to an exemplary embodiment.

[0025] Figure 6 This is a flowchart illustrating an effective data comparison method according to another exemplary embodiment.

[0026] Figure 7 This is a flowchart illustrating a target sampling method for obtaining a method according to an exemplary embodiment.

[0027] Figure 8 This is a flowchart illustrating a method for obtaining parameter values ​​according to an exemplary embodiment.

[0028] Figure 9 This is a block diagram illustrating an automatic sampling device for simulation testing according to an exemplary embodiment.

[0029] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0030] It should be noted that the relevant embodiments and accompanying drawings are only for describing and illustrating exemplary embodiments provided by this disclosure, and not all embodiments of this disclosure, nor should this disclosure be understood to be limited to the relevant exemplary embodiments.

[0031] It should be noted that the terms "first," "second," etc., used in this disclosure are only used to distinguish different steps, devices, or modules. These terms do not represent any specific technical meaning, nor do they indicate any order or interdependence between them.

[0032] It should be noted that the terms “a,” “a plurality of,” and “at least one” used in this disclosure are illustrative rather than restrictive. Unless otherwise expressly indicated in the context, they should be understood as “one or more.”

[0033] It should be noted that the term "and / or" used in this disclosure is used to describe the relationship between related objects, and generally indicates that there are at least three relationships. For example, A and / or B can at least indicate: the existence of A alone, the existence of both A and B, and the existence of B alone.

[0034] It should be noted that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Unless otherwise specified, the scope of this disclosure is not limited by the order in which the steps are described in the relevant embodiments.

[0035] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0036] Exemplary methods

[0037] Figure 1 This is a flowchart illustrating an automatic sampling method for simulation testing according to an exemplary embodiment, such as... Figure 1 As shown, the automatic sampling method for simulation testing is used in an autonomous driving safety verification platform, and includes the following steps:

[0038] In step S110, the test scenario and parameter space for autonomous driving simulation testing are obtained, and the parameter space includes the range of parameter values ​​corresponding to the test scenario.

[0039] In some embodiments, various test scenarios exist depending on the testing requirements, such as urban scenarios, highway scenarios, and mountain road scenarios. Therefore, before conducting simulation testing on an autonomous driving system, the corresponding test scenario must first be determined. A test scenario contains multiple parameters, and the test scenario varies depending on the composition of different parameters, such as different road widths and the number of traffic lights. The range of values ​​for all parameters in a given test scenario constitutes the parameter space. By obtaining parameter values ​​from the parameter space, the parameters for the determined test scenario can be configured, thereby enabling the simulation testing of the autonomous driving system within that test scenario.

[0040] Step S120: Determine the first parameter value according to the test scenario, and run the test scenario based on the first parameter value to obtain the scenario running result.

[0041] In some embodiments, after determining the test scenario, a first parameter value is obtained. The first parameter value includes one or more parameters used to configure the test scenario, such as the configuration of road width, obstacles, etc. When setting the first parameter value, it can be empty or a system preset value, or it can be a parameter value that is likely to generate effective data in historical simulation tests, so as to obtain more effective data in subsequent sampling processes. The test scenario is run using the first parameter value to test the autonomous driving system and obtain the scenario operation results.

[0042] In step S130, at least two candidate sampling methods are used to sample the test scenario at least once to obtain the corresponding sampled data set.

[0043] In some embodiments, since there are multiple sampling methods, such as cross-entropy sampling, random sampling, Hatton sampling, Bayesian sampling, and simulated annealing sampling, this method is needed to determine the preferred sampling method in order to obtain more effective data and better judge the test status of the autonomous driving system.

[0044] Therefore, by using at least two candidate sampling methods, the test scenario is sampled at least once to obtain the corresponding sampling parameter values, and the corresponding sampling data set is obtained based on all the sampling parameter values. Each sampling involves selecting a set of parameter values ​​from the parameter space using the corresponding candidate sampling method, sampling that set of parameter values, running the test scenario, and obtaining the corresponding scenario execution result.

[0045] In step S140, valid data for candidate sampling methods are obtained based on the sampled data set.

[0046] In some embodiments, after all parameter values ​​in the sampling dataset have been processed, valid data from the corresponding candidate sampling methods are obtained. If an anomaly exists in the scenario execution result corresponding to a sampling parameter value in a certain sampling dataset, such as running a red light or a collision, then that set of sampling parameter values ​​is considered valid data. Valid data allows for better assessment of the performance of the autonomous driving system, thereby enabling corresponding performance optimization.

[0047] In step S150, the valid data corresponding to the candidate sampling methods are compared to obtain the target sampling method.

[0048] In some embodiments, after obtaining valid data corresponding to all candidate sampling methods, the valid data in the candidate sampling methods are compared to obtain the target sampling method among the candidate sampling methods. For example, the method that obtains the most valid data is selected as the target sampling method. The target sampling method can perform better parameter sampling for the test scenario, thereby obtaining more valid test data, ensuring the effectiveness of sampling, and improving test efficiency.

[0049] In step S160, the test scenario is tested according to the target sampling method.

[0050] In some embodiments, since the obtained target sampling method is the preferred sampling method for the test scenario, testing the test scenario using the target sampling method can yield more effective data, thereby discovering more problems in the autonomous driving system, reducing unnecessary scenario testing and resource consumption, and improving the efficiency of simulation testing.

[0051] In this embodiment, by acquiring the test scenario and parameter space for autonomous driving simulation testing, and determining the first parameter value based on the test scenario, the test scenario is run to obtain the scenario running results. At least two candidate sampling methods are used to sample the test scenario at least once to obtain the corresponding sampling data set. Based on the sampling data set, the effective data of the candidate sampling methods is obtained. The effective data corresponding to the candidate sampling methods are compared to obtain the target sampling method. The test scenario is then tested based on the target sampling method. This allows for the selection of the preferred sampling method from several sampling methods, and the simulation test of the autonomous driving system is achieved through the preferred sampling method. This facilitates the discovery of more safety issues in the autonomous driving system, ensures the effectiveness of sampling, and improves the efficiency of simulation testing.

[0052] Figure 2 This is a flowchart illustrating a method for obtaining scene execution results according to an exemplary embodiment, such as... Figure 2 As shown, the method for obtaining the scene execution result is used in step S102, and includes the following steps:

[0053] In step S210, historical test data of the test scenario is obtained, and parameter values ​​that meet the preset conditions are selected based on the historical test data.

[0054] Specifically, after determining the test scenario, the historical test data for that test scenario is queried, and parameter values ​​that meet the preset conditions are selected from the corresponding historical valid data. For example, if the preset condition is that there are more than five valid data points, then a set of parameter values ​​with more than five valid data points is selected, so as to better discover the potential problems of the autonomous driving system in that test scenario.

[0055] In step S220, the parameter value is used as the first parameter value, and the test scenario is run according to the first parameter value to obtain the corresponding scenario running result. The scenario running result includes the first parameter value and the initial valid data.

[0056] Specifically, a set of parameter values ​​that meet preset conditions is used as the first parameter value, and the test scenario is run according to the first parameter value to obtain the test result, such as the test result being normal operation, collision during operation, or violation of operation. The test result information and the first parameter value are stored as the scenario operation result to realize data update under the test scenario, so as to facilitate data traceability and subsequent selection of parameter values.

[0057] In this embodiment, by acquiring historical test data of the test scenario, selecting parameter values ​​that meet the preset conditions of the corresponding historical valid data, using the parameter values ​​as the first parameter values, and running the test scenario according to the first parameter values, the corresponding scenario running results are obtained. The preferred first parameter values ​​can be obtained based on historical data, so as to better discover the problems of the autonomous driving system in the test scenario. The running results are stored to realize data updates in the test scenario, which facilitates data tracing and subsequent selection of parameter values.

[0058] Figure 3 This is a flowchart illustrating a method for acquiring a sampled data set according to an exemplary embodiment, such as... Figure 3 As shown, the method for obtaining the sampled data set is used in step S103, and includes the following steps:

[0059] In step S310, the maximum number of samplings for the candidate sampling method is set.

[0060] Specifically, when sampling using candidate sampling methods, the maximum number of samplings for all candidate sampling methods is first set to determine the upper limit of sampling.

[0061] In step S320, a set of parameter values ​​are selected in the parameter space using a candidate sampling method.

[0062] Specifically, a set of parameter values ​​is selected in the parameter space through a candidate sampling method. Different sampling methods yield different parameter values, and the difference in parameter values ​​will lead to differences in the test scenario, thereby causing differences in the test results of the autonomous driving system.

[0063] In step S330, the test scenario is run based on the parameter values ​​and the scenario running results to obtain a set of sampled data.

[0064] Specifically, based on the selected parameter values ​​and the results of the previous scenario, the test scenario is run to obtain a set of sampled data. The sampled data consists of the generated autonomous driving safety issues, such as the number of collisions and violations. Combined with the scenario running results, the test will identify more autonomous driving safety issues that are more likely to occur during the test.

[0065] In step S340, sampling is repeated according to the candidate sampling method until the maximum number of samplings is reached, and the sampling data set of the candidate sampling method is obtained.

[0066] Specifically, in setting a maximum number of sampling attempts, candidate sampling methods are used to repeatedly sample parameters in the parameter space until the maximum number of sampling attempts is reached, thus obtaining the sampled data set acquired by the candidate sampling methods. When multiple candidate sampling methods exist, multiple candidate sampling methods can be processed in parallel according to the resource conditions of the scenario, and the sampled data set corresponding to each candidate sampling method can be obtained to improve processing speed.

[0067] In this embodiment, a maximum number of sampling times for the candidate sampling method is set. A set of parameter values ​​is selected in the parameter space using the candidate sampling method. Based on the parameter values ​​and the scenario execution results, the test scenario is run to obtain a set of sampling data. Sampling is repeated according to the candidate sampling method until the maximum number of sampling times is reached, thereby obtaining a set of sampling data for the candidate sampling method. This enables the processing of multiple candidate sampling methods and the obtaining of corresponding sampling data sets, which facilitates the determination of the preferred sampling method for the test scenario based on the sampling data sets.

[0068] Figure 4 This is a flowchart illustrating an effective data acquisition method according to an exemplary embodiment, such as... Figure 4 As shown, the effective data acquisition method used in step S104 includes the following steps:

[0069] In step S410, abnormal driving data during the test scenario operation is determined based on the sampled data set. Abnormal driving data includes collision data and violation data.

[0070] Specifically, after obtaining the sampling data set of candidate sampling methods, abnormal driving data during the test scenario operation is determined based on the sampling data set, including collision data and violation data. Collision data can be collision data with vehicles or obstacles, and violation data can be violation data such as running red lights or changing lanes illegally.

[0071] In step S420, the frequency of abnormal driving data occurrences in the sampled data set is counted.

[0072] Specifically, the frequency of abnormal driving data in the statistical sampling dataset is used to determine the frequency of abnormal driving in the candidate sampling method, thereby obtaining the frequency of safety problems occurring in the autonomous driving system through the candidate sampling method.

[0073] In step S430, valid data for candidate sampling methods are determined based on frequency and abnormal driving data.

[0074] Specifically, based on the abnormal data and their frequency of occurrence, the effective data generated by the candidate sampling method during the sampling process is determined. The effective data includes frequency and abnormal driving data, which enables a more accurate assessment of the effectiveness of the candidate sampling method in the simulation test of the autonomous driving system, so as to obtain the preferred sampling method.

[0075] In this embodiment, abnormal driving data during the test scenario is determined based on the sampled data set, the frequency of abnormal driving data occurrence in the sampled data set is counted, and effective data for candidate sampling methods is determined based on the abnormal driving data and its frequency. This allows for comparison of candidate sampling methods using effective data, thereby obtaining the preferred sampling method and improving simulation testing efficiency.

[0076] Figure 5 This is a flowchart illustrating an effective data comparison method according to an exemplary embodiment, such as... Figure 5 As shown, the effective data comparison method is used in step S105, which includes the following steps:

[0077] In step S510, the valid data are compared to determine the candidate sampling method whose valid data reaches a preset threshold.

[0078] Specifically, after all candidate sampling methods have been sampled, the effective data corresponding to all candidate sampling methods is compared to determine the candidate sampling method whose effective data reaches a preset threshold. For example, the effective data can be set to include collisions and violations, and the candidate sampling method whose sum of collision and violation frequencies reaches five times is obtained. Of course, if none of the candidate sampling methods reach the preset threshold, the candidate sampling method with the highest effective data frequency or the most collision data can be obtained through comparison.

[0079] In step S520, the target sampling method is obtained based on the candidate sampling methods whose effective data reaches a preset threshold.

[0080] Specifically, the frequency of valid data, collision data, and violation data can directly reflect the effectiveness of the corresponding candidate sampling methods for autonomous driving system simulation testing. Therefore, candidate sampling methods with valid data reaching a preset threshold are used as target sampling methods. The target sampling method is the preferred sampling method for autonomous driving system simulation testing in this test scenario, which can generate more safety issues to facilitate optimization of the autonomous driving system.

[0081] In this embodiment, by comparing valid data, candidate sampling methods whose valid data reaches a preset threshold are determined and used as target sampling methods. This allows for obtaining the preferred sampling method in the test scenario, thereby preventing the autonomous driving system from generating more safety issues and improving test efficiency.

[0082] Figure 6 This is a flowchart illustrating an effective data comparison method according to another exemplary embodiment, such as... Figure 6 As shown, the effective data comparison method, after step S420, includes the following steps:

[0083] In step S610, the frequency of occurrence of collision data and violation data is obtained to obtain the collision frequency and violation frequency.

[0084] Specifically, since collisions and violations are different types of safety issues, it is necessary to statistically analyze the frequency of collision and violation data based on valid data to obtain the collision frequency and violation frequency, and then optimize the system accordingly based on the collisions and violations.

[0085] In step S620, candidate sampling methods are compared based on collision frequency and violation frequency to obtain comparison results.

[0086] Specifically, by comparing candidate sampling methods based on collision frequency and violation frequency, we can obtain candidate sampling methods with higher collision frequency or higher violation frequency. Thus, we can optimize the autonomous driving system for collision and violation by using the corresponding candidate sampling methods.

[0087] In step S630, candidate sampling methods whose collision frequencies reach a preset frequency are obtained based on the comparison results and used as target sampling methods.

[0088] Specifically, collisions are generally considered a more serious safety issue than violations. Therefore, candidate sampling methods that reach a preset frequency of collisions can be obtained based on the comparison results and used as the target sampling method to solve the collision problem in the autonomous driving system and improve the safety performance of the autonomous driving system.

[0089] In this embodiment, the collision frequency and violation frequency are obtained by acquiring the frequency of collision data and violation data, and are used to compare candidate sampling methods to obtain comparison results. The candidate sampling method whose fall collision frequency reaches a preset frequency according to the comparison results is used as the target sampling method, thereby obtaining the preferred sampling method for the collision problem, so as to generate more collision problems through the preferred sampling method and perform targeted processing.

[0090] Figure 7 This is a flowchart illustrating a target sampling method for obtaining a method according to an exemplary embodiment, such as... Figure 7 As shown, the target sampling method is used in step S520, and includes the following steps:

[0091] In step S710, when there are at least two candidate sampling methods with valid data reaching a preset threshold, the corresponding candidate sampling method is recorded as a pending sampling method.

[0092] Specifically, since there are multiple candidate sampling methods, when comparing the relationship between effective data and preset thresholds, there may be at least two candidate sampling methods whose effective data reach the preset thresholds. The corresponding candidate sampling methods are then designated as undetermined sampling methods.

[0093] In step S720, the sampling data set and valid data of the sampling method to be determined are obtained and used as feature values ​​and label data, respectively.

[0094] Specifically, the corresponding sampling data set and valid data are obtained according to the undetermined sampling method, and the sampling data set and label data are used as feature values ​​and label data, respectively.

[0095] In step S730, an automatic ensemble learning method is used to train the feature values ​​and label data to obtain a sampling determination model.

[0096] Specifically, an automatic ensemble learning method is adopted, which trains multiple learners using feature values ​​and label data, and merges them into a new learner to obtain a sampling determination model, which is used to determine the preferred sampling method.

[0097] In step S740, a maximum number of calls is set, and the sampling determination model is called respectively through the undetermined sampling method to obtain the target valid data.

[0098] Specifically, the maximum number of times the sampling determination model is called is set, and the sampling determination model is called respectively through the undetermined sampling method to obtain the corresponding target valid data. The input data of the sampling determination model is a set of parameter values ​​obtained after each sampling run, and the output data of the model after calling the model is the target valid data.

[0099] In step S750, the sum of the effective data of the sampling method to be determined and the effective data of the target is calculated.

[0100] Specifically, after the model is called by the undetermined sampling method, the sum of the effective data of the undetermined sampling method and the target effective data is calculated, so that the target sampling method can be obtained based on the data sum.

[0101] In step S760, the target sampling method is obtained by comparing the data and the undetermined sampling methods.

[0102] Specifically, after obtaining the data of all undetermined sampling methods, the data is compared, and the undetermined sampling method with the highest frequency of valid data or the highest collision frequency is selected as the target sampling method. This allows for better discovery of safety issues in autonomous driving systems based on the target sampling method, thereby improving the effectiveness of the sampling data.

[0103] In this embodiment, when the effective data of at least two candidate sampling methods reaches a preset threshold, the corresponding candidate sampling methods are recorded as pending sampling methods. The corresponding sampling data set and effective data are obtained and used as feature values ​​and label data, respectively. The feature values ​​and label data are trained using an automatic ensemble learning method to obtain a sampling determination model. A maximum number of calls is set, and the sampling determination model is called through the pending sampling methods to obtain target effective data. The sum of the target effective data and the effective data is calculated, and the sum of the data between the pending sampling methods is compared to obtain the target sampling method. Thus, the preferred sampling method is obtained and used to discover safety issues in the autonomous driving system, thereby improving the effectiveness of the sampling data and increasing the efficiency of simulation testing.

[0104] Figure 8 This is a flowchart illustrating a parameter value acquisition method according to an exemplary embodiment, such as... Figure 8 As shown, the parameter value acquisition method is used after step S160 and includes the following steps:

[0105] In step S810, after all candidate sampling methods have been run, all sampling data sets and valid data are obtained.

[0106] Specifically, in order to obtain an optimal set of parameter values, after all candidate sampling methods have been run, all sampled data sets and valid data are obtained.

[0107] In step S820, the probability density distribution calculation method of multidimensional continuous random variables is used to calculate the sampled data set and the effective data to obtain the sampled data that reaches the effective threshold.

[0108] Specifically, the probability density distribution calculation method of multidimensional continuous random variables is used to calculate the sampled data set and the effective data to obtain the sampled data that reaches the effective threshold, or to compare the sampled data with the highest frequency of effective data occurrence.

[0109] In step S830, the corresponding parameter values ​​are obtained based on the sampled data and stored.

[0110] Specifically, since the sampled data is a set of parameter values ​​sampled in the parameter space, the corresponding parameter values ​​can be obtained and stored based on the sampled data. When the test scenario needs to be used again, the stored parameter values ​​can be used as the first parameter values, enabling the autonomous driving system to generate more safety issues and improving the effectiveness of the sampled data.

[0111] In this embodiment, after all candidate sampling methods have been run, in order to obtain the preferred parameter values, all sampled data sets and valid data can be obtained. The probability density distribution calculation method of multidimensional continuous random variables can be used to calculate the sampled data sets and valid data to obtain the sampled data that reaches the valid threshold. The corresponding parameter values ​​can be obtained based on the sampled data and stored. This yields parameter values ​​that are prone to safety issues, making it easier to discover more autonomous driving safety problems and improve testing efficiency.

[0112] Exemplary device

[0113] Figure 9 This is a block diagram of an automatic sampling device for simulation testing according to an exemplary embodiment. (Refer to...) Figure 9 The device 900 includes a test scenario acquisition module 910, a test scenario operation module 920, a test scenario sampling module 930, an effective data acquisition module 940, an effective data comparison module 950, and a test module 960.

[0114] The test scenario acquisition module 910 is used to acquire the test scenario and parameter space for autonomous driving simulation testing. The parameter space includes the range of parameter values ​​corresponding to the test scenario.

[0115] The test scenario execution module 920 is used to determine the first parameter value according to the test scenario, run the test scenario based on the first parameter value, and obtain the scenario execution results;

[0116] The test scenario sampling module 930 is used to sample the test scenario at least once using at least two candidate sampling methods to obtain the corresponding sampling data set.

[0117] The effective data acquisition module 940 is used to acquire effective data of candidate sampling methods based on the sampled data set;

[0118] The effective data comparison module 950 is used to compare the effective data corresponding to the candidate sampling methods to obtain the target sampling method;

[0119] The test module 960 is used to complete the testing of the test scenario according to the target sampling method.

[0120] In this embodiment, the test scenario acquisition module 910 acquires the test scenario and corresponding parameter space for autonomous driving simulation testing. The test scenario execution module 920 determines the first parameter value based on the test scenario and runs the test scenario based on the first parameter value to obtain the scenario execution results. The test scenario sampling module 930 uses at least two candidate sampling methods to sample the test scenario at least once to obtain the corresponding sampling data set. The effective data acquisition module 940 obtains the effective data of the candidate sampling methods based on the adopted data set, and the effective data comparison module 950 compares the effective data corresponding to the candidate sampling methods to obtain the target sampling method. Finally, the test module 960 uses the target sampling method to complete the test of the test scenario, thereby obtaining the preferred sampling method and realizing the simulation test of the autonomous driving system through the preferred sampling method, so as to discover more safety issues of the autonomous driving system, improve the effectiveness of sampling, and thus improve the efficiency of simulation testing.

[0121] In some embodiments, the device further includes: a first parameter value selection module, configured to acquire historical test data of the test scenario, and select parameter values ​​that meet preset conditions for corresponding historical valid data based on the historical test data; and a running result acquisition module, configured to use the parameter values ​​as the first parameter values, run the test scenario based on the first parameter values, and acquire the corresponding scenario running results, wherein the scenario running results include the first parameter values ​​and initial valid data.

[0122] In some embodiments, the device further includes: a sampling count setting module for setting the maximum sampling count of the candidate sampling method; a second parameter value selection module for selecting a set of parameter values ​​in the parameter space using the candidate sampling method; a sampling data acquisition module for running the test scenario based on the parameter values ​​and the scenario running results to acquire a set of sampling data; and a sampling data set acquisition module for repeatedly sampling according to the candidate sampling method until the maximum sampling count is reached to acquire the sampling data set of the candidate sampling method.

[0123] In some embodiments, the apparatus further includes: an abnormal driving data determination module, configured to determine abnormal driving data during the operation of a test scenario based on a sampled data set, wherein the abnormal driving data includes collision data and violation data; a frequency statistics module, configured to count the frequency of occurrence of abnormal driving data in the sampled data set; and a valid data determination module, configured to determine valid data for candidate sampling methods based on the frequency and the abnormal driving data.

[0124] In some embodiments, the apparatus further includes: a candidate sampling method determination module, configured to compare valid data and determine candidate sampling methods in which the valid data reaches a preset threshold; and a target sampling method acquisition module, configured to acquire a target sampling method based on the candidate sampling methods in which the valid data reaches the preset threshold.

[0125] In some embodiments, the device further includes: a frequency acquisition module, configured to acquire the frequency of occurrence of collision data and violation data, and obtain collision frequency and violation frequency; a candidate sampling method comparison module, configured to compare candidate sampling methods according to collision frequency and violation frequency, and obtain comparison results; and a candidate sampling method acquisition module, configured to acquire candidate sampling methods whose collision frequency reaches a preset frequency according to the comparison results, and use them as target sampling methods.

[0126] In some embodiments, the apparatus further includes: a pending sampling method acquisition module, configured to record the corresponding candidate sampling method as a pending sampling method when there are at least two candidate sampling methods with valid data reaching a preset threshold; a feature value acquisition module, configured to acquire the sampling data set and valid data of the pending sampling method, and use them as feature values ​​and label data respectively; a model training module, configured to train the feature values ​​and label data using an automatic ensemble learning method to obtain a sampling determination model; a sampling determination model invocation module, configured to set a maximum number of invocations, and invoke the sampling determination model through the pending sampling method to obtain target valid data; a data and calculation module, configured to calculate the sum of the valid data of the pending sampling method and the target valid data; and a pending sampling method comparison module, configured to compare the pending sampling methods based on the sum of the data to obtain the target sampling method.

[0127] In some embodiments, the apparatus further includes: a data acquisition module, configured to acquire all sampled data sets and valid data after all candidate sampling methods have been run; a data calculation module, configured to calculate the sampled data sets and valid data using a probability density distribution calculation method for multidimensional continuous random variables, to obtain sampled data where the valid data reaches the valid threshold; and a parameter value storage module, configured to acquire and store the corresponding parameter values ​​based on the sampled data.

[0128] In the above embodiments, the operation of the device enables an automatic sampling method based on simulation testing, ensuring that the device can implement all aspects of an automatic sampling method for simulation testing.

[0129] Exemplary electronic devices

[0130] Figure 10 This is a block diagram illustrating an electronic device 1000 according to an exemplary embodiment. The electronic device 1000 may be a vehicle controller, an in-vehicle terminal, an in-vehicle computer, or other types of electronic devices.

[0131] Reference Figure 10 The electronic device 1000 may include at least one processor 1010 and a memory 1020. The processor 1010 can execute instructions stored in the memory 1020. The processor 1010 is communicatively connected to the memory 1020 via a data bus. In addition to the memory 1020, the processor 1010 may also be communicatively connected to an input device 1030, an output device 1040, and a communication device 1050 via the data bus.

[0132] The processor 1010 can be any conventional processor, such as a commercially available CPU. The processor may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0133] The memory 1020 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0134] In this embodiment of the present disclosure, the memory 1020 stores executable instructions, and the processor 1010 can read the executable instructions from the memory 1020 and execute the instructions to implement all or part of the steps of the automatic sampling method for simulation testing as described in any of the exemplary embodiments above.

[0135] Exemplary computer-readable storage media

[0136] In addition to the methods and apparatus described above, exemplary embodiments of this disclosure may also be a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in any of the methods in the exemplary embodiments described above.

[0137] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0138] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination thereof.

[0139] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0140] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An automatic sampling method for simulation testing, characterized in that, include: Obtain the test scenario and parameter space for autonomous driving simulation testing, wherein the parameter space includes the parameter value range corresponding to the test scenario; Based on the test scenario, determine the first parameter value, run the test scenario based on the first parameter value, and obtain the scenario running result; At least two candidate sampling methods are used to sample the test scenario at least once to obtain the corresponding sampled data set; Valid data for the candidate sampling method are obtained based on the sampling data set. The method of comparing the effective data corresponding to the candidate sampling methods to obtain the target sampling method includes: comparing the effective data to determine the candidate sampling methods whose effective data reaches a preset threshold; obtaining the target sampling method based on the candidate sampling methods whose effective data reaches the preset threshold, including: when there are at least two candidate sampling methods whose effective data reaches the preset threshold, recording the corresponding candidate sampling methods as pending sampling methods; obtaining the sampling data set and the effective data of the pending sampling methods, and using them as feature values ​​and label data respectively; training the feature values ​​and label data using an automatic ensemble learning method to obtain a sampling determination model; setting a maximum number of calls, and calling the sampling determination model through the pending sampling methods to obtain target effective data; calculating the sum of the effective data of the pending sampling methods and the target effective data; and comparing the pending sampling methods based on the sum to obtain the target sampling method. The test scenario is tested according to the target sampling method.

2. The automatic sampling method for simulation testing according to claim 1, characterized in that, The step of determining a first parameter value based on the test scenario, running the test scenario based on the first parameter value, and obtaining the scenario running result includes: Obtain historical test data for the test scenario, and select parameter values ​​that meet preset conditions based on the historical test data; The parameter value is used as the first parameter value, and the test scenario is run according to the first parameter value to obtain the corresponding scenario running result. The scenario running result includes the first parameter value and initial valid data.

3. The automatic sampling method for simulation testing according to claim 1, characterized in that, The process employs at least two candidate sampling methods to sample the test scenario at least once each, obtaining corresponding sampled data sets, including: Set the maximum number of samples for the candidate sampling method; The candidate sampling method is used to select a set of parameter values ​​in the parameter space; Based on the parameter values ​​and the scenario execution results, run the test scenario and obtain a set of sampled data; The sampling is repeated according to the candidate sampling method until the maximum number of samplings is reached, thereby obtaining the sampling data set of the candidate sampling method.

4. The automatic sampling method for simulation testing according to claim 1, characterized in that, The step of obtaining valid data for the candidate sampling method based on the sampled data set includes: Based on the sampled data set, abnormal driving data during the operation of the test scenario is determined, including collision data and violation data; Statistically count the frequency of abnormal driving data occurrences in the sampled data set; Based on the frequency and the abnormal driving data, the effective data for the candidate sampling method are determined.

5. The automatic sampling method for simulation testing according to claim 4, characterized in that, The step of comparing the effective data of the candidate sampling methods to obtain the target sampling method includes: Obtain the frequency of occurrence of the collision data and the violation data to obtain the collision frequency and violation frequency; Based on the collision frequency and violation frequency, the candidate sampling methods are compared to obtain the comparison results; Based on the comparison results, the candidate sampling method whose collision frequency reaches a preset frequency is obtained and used as the target sampling method.

6. The automatic sampling method for simulation testing according to claim 1, characterized in that, Also includes: After all the candidate sampling methods have been run, obtain all the sampled data sets and the valid data; The probability density distribution calculation method of multidimensional continuous random variables is used to calculate the sampled data set and the effective data to obtain the sampled data that reaches the effective threshold. The corresponding parameter values ​​are obtained from the sampled data and stored.

7. An automatic sampling device for simulation testing, characterized in that, include: The test scenario acquisition module is used to acquire the test scenario and parameter space for autonomous driving simulation testing. The parameter space includes the range of parameter values ​​corresponding to the test scenario. The test scenario execution module is used to determine a first parameter value based on the test scenario, and to run the test scenario based on the first parameter value to obtain the scenario execution result; The test scenario sampling module is used to sample the test scenario at least once using at least two candidate sampling methods to obtain the corresponding sampling data set. The effective data acquisition module is used to acquire effective data of the candidate sampling method based on the sampled data set; An effective data comparison module, used to compare the effective data corresponding to the candidate sampling methods to obtain the target sampling method, specifically includes: a candidate sampling method determination module, used to compare the effective data and determine the candidate sampling methods whose effective data reaches a preset threshold; a target sampling method acquisition module, used to acquire the target sampling method based on the candidate sampling methods whose effective data reaches the preset threshold; a pending sampling method acquisition module, used to record the corresponding candidate sampling method as a pending sampling method when there are at least two candidate sampling methods whose effective data reaches the preset threshold; a feature value acquisition module, used to acquire the sampling data set and the effective data of the pending sampling method, and use them as feature values ​​and label data respectively; a model training module, used to train the feature values ​​and label data using an automatic ensemble learning method to obtain a sampling determination model; a sampling determination model invocation module, used to set a maximum number of invocations, and invocation the sampling determination model through the pending sampling methods to obtain target effective data; a data and calculation module, used to calculate the data sum of the effective data of the pending sampling method and the target effective data; and a pending sampling method comparison module, used to compare the pending sampling methods based on the data sum to obtain the target sampling method. The testing module is used to perform tests on the test scenario according to the target sampling method.

8. The automatic sampling device for simulation testing according to claim 7, characterized in that, The device further includes: The first parameter value selection module is used to obtain historical test data of the test scenario and select parameter values ​​that meet the preset conditions based on the historical test data. The execution result acquisition module is used to take the parameter value as the first parameter value, run the test scenario according to the first parameter value, and obtain the corresponding scenario execution result. The scenario execution result includes the first parameter value and initial valid data.

9. The automatic sampling device for simulation testing according to claim 7, characterized in that, The device further includes: The sampling count setting module is used to set the maximum sampling count of the candidate sampling method; The second parameter value selection module is used to select a set of parameter values ​​in the parameter space through the candidate sampling method. The sampling data acquisition module is used to run the test scenario based on the parameter values ​​and the scenario running results, and acquire a set of sampling data; The sampling data set acquisition module is used to repeatedly sample according to the candidate sampling method until the maximum number of samplings is reached, and to acquire the sampling data set of the candidate sampling method.

10. The automatic sampling device for simulation testing according to claim 7, characterized in that, The device further includes: An abnormal driving data determination module is used to determine abnormal driving data during the operation of the test scenario based on the sampled data set. The abnormal driving data includes collision data and violation data. The frequency statistics module is used to count the frequency of abnormal driving data occurrences in the sampled data set; The effective data determination module is used to determine the effective data of the candidate sampling method based on the frequency and the abnormal driving data.

11. The automatic sampling device for simulation testing according to claim 10, characterized in that, The device further includes: The frequency acquisition module is used to acquire the frequency of occurrence of the collision data and the violation data, and obtain the collision frequency and violation frequency; The candidate sampling method comparison module is used to compare the candidate sampling methods according to the collision frequency and violation frequency, and obtain the comparison result. The candidate sampling method acquisition module is used to acquire, based on the comparison result, the candidate sampling methods whose collision frequency reaches a preset frequency, and use them as the target sampling methods.

12. The automatic sampling device for simulation testing according to claim 7, characterized in that, The device further includes: The data acquisition module is used to acquire all the sampled data sets and the valid data after all the candidate sampling methods have been run; The data calculation module is used to calculate the sampled data set and the effective data using the probability density distribution calculation method of multidimensional continuous random variables, so as to obtain the sampled data that reaches the effective threshold. The parameter value storage module is used to obtain and store the corresponding parameter values ​​based on the sampled data.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the automatic sampling method for simulation testing as described in any one of claims 1-6.

14. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the steps of the automatic sampling method for simulation testing as described in any one of claims 1-6.

Images