Test scene processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure disclose a test scene processing method and device, electronic equipment and storage medium, the method comprising: obtaining a target test scene, and generating a plurality of test cases of the target test scene; executing the plurality of test cases to obtain a plurality of target test indicators; processing based on the plurality of target test indicators; determining a target test case from the plurality of test cases; and storing the target test scene and the target test case in a scene database in association. By using the above scheme, a large number of test cases of the test scene are generated, and high-quality test cases and test scenes are stored in association, so as to ensure the subsequent automatic driving test effect.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, and in particular to a test scenario processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, the testing of autonomous driving algorithms installed in autonomous vehicles generally adopts driving simulation technology, which requires the construction of autonomous driving simulation test scenarios that meet the testing requirements.

[0003] In related technologies, only one or a few test cases corresponding to specific scenarios are executed in actual testing, resulting in insufficient test completeness, which in turn affects the test results of autonomous driving. Summary of the Invention

[0004] To address the aforementioned technical problems, or at least partially address them, this disclosure provides a test scenario processing method, apparatus, electronic device, and storage medium to acquire a large number of high-quality test cases and test scenario associated storage, thereby ensuring the effectiveness of subsequent autonomous driving tests.

[0005] In a first aspect, embodiments of this disclosure provide a test scenario processing method, the method comprising:

[0006] Obtain the target test scenario and generate multiple test cases for the target test scenario;

[0007] Execute the multiple test cases to obtain multiple target test metrics;

[0008] Based on the processing of the multiple target test metrics, target test cases are determined from the multiple test cases;

[0009] The target test scenario and the target test case are associated and stored in the scenario database.

[0010] Secondly, embodiments of this disclosure also provide a test scenario processing apparatus, the apparatus comprising:

[0011] The first acquisition module is used to acquire the target test scenario;

[0012] The generation module is used to generate multiple test cases for the target test scenario;

[0013] The execution module is used to execute the multiple test cases and obtain multiple target test metrics;

[0014] The processing and determination module is used to process the multiple target test metrics and determine the target test cases from the multiple test cases;

[0015] An associated storage module is used to associate and store the target test scenario and the target test case in a scenario database.

[0016] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the test scenario processing method as described above.

[0017] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the test scenario processing method described above.

[0018] The test scenario processing method provided in this disclosure involves acquiring a target test scenario, generating multiple test cases for the target test scenario, executing the multiple test cases to obtain multiple target test metrics, processing based on the multiple target test metrics, determining the target test case from the multiple test cases, and associating and storing the target test scenario and the target test case in a scenario database. By adopting the above scheme, a large number of test cases for the test scenario are generated, and high-quality test cases are selected and associated with the test scenario for storage, thereby ensuring the effectiveness of subsequent autonomous driving tests. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 This is a flowchart of a test scenario processing method according to an embodiment of this disclosure;

[0021] Figure 2 This is a flowchart of another test scenario processing method in this disclosure embodiment;

[0022] Figure 3 This is a schematic diagram of the structure of a test scenario processing device according to an embodiment of the present disclosure;

[0023] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0024] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0026] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0027] Specifically, test scenarios are classified according to the level of abstraction of the representation, including functional scenarios, logical scenarios, and specific scenarios. Functional scenarios describe relevant information through semantic description; logical scenarios use state space parameters to describe functional scenarios in detail; the key to describing logical scenarios is to define these state space parameters and their value ranges; describing specific scenarios involves determining the specific value of each parameter in the state space. Specific scenarios, plus the expected behavior of the object under test, are converted into test cases. Therefore, the smallest unit of test scenarios for simulation testing is usually a specific scenario.

[0028] This disclosure provides a test scenario processing method, which aims to generate a large number of test cases for test scenarios, and to screen high-quality test cases and store them in association with test scenarios to ensure the effectiveness of subsequent autonomous driving tests.

[0029] Figure 1 This is a flowchart illustrating a test scenario processing method according to an embodiment of this disclosure. The method can be executed by a test scenario processing device, which can be implemented in software and / or hardware and can be configured in an electronic device, such as a server.

[0030] like Figure 1 As shown, the method specifically includes the following steps:

[0031] Step 110: Obtain the target test scenario and generate multiple test cases for the target test scenario.

[0032] The target test scenario refers to a specific scenario description with atomicity (atomic scenario). An atomic scenario is an indivisible scenario. For example, scenario 0: a straight right lane of two lanes, with a static cube on the right at 120m and a slow-moving vehicle at 150m. It can be divided into: scenario 1: a straight right lane of two lanes, with a static cube on the right at 120m; scenario 2: a straight right lane of two lanes, with a slow-moving vehicle at 150m. Scenario 0 is not an atomic scenario, while scenarios 1 and 2 are atomic scenarios.

[0033] In this embodiment of the disclosure, a target test scenario is obtained, such as determining a functional scenario based on test requirements, describing the functional scenario in detail using state space parameters to obtain a logical scenario, determining the specific value of each parameter in the state space of the logical scenario, and obtaining a series of atomic specific scenarios as target test scenarios.

[0034] Among them, a test case refers to an atomic scenario plus the expected behavior of the object under test, which is converted into a test case. For example, the atomic scenario is a 200m straight road with only one lane. The width of the lane is twice the width of the vehicle boundary. There is a static cube in the lane at 120m. The expected behavior is that when the obstacle occupies no more than the left center line, the autonomous vehicle should pass on the right side without crossing the line. A test case is generated.

[0035] In this embodiment of the disclosure, there are many ways to generate multiple test cases for the target test scenario. In one embodiment, the state space parameters of the target test scenario are obtained, and different values ​​are assigned to the state space parameters under different conditions. Based on the different values ​​of each judgment indicator and the state space parameters, multiple test cases are generated.

[0036] For example, the target test scenario is a 200m straight road with only one lane, the width of which is twice the width of the vehicle's boundary. There is a static cube in the road at 120m. The semantic keywords of the state space are determined to be school and passenger car, the environmental state space is light and rain / snow, the participant state space is the lane-occupying space, the autonomous vehicle state space is the perception range (similar to lane-occupying space) and speed, and the indicators include whether there is a collision, whether the designated destination has been reached, the longitudinal safe distance from the nearest face of the cube when starting to move to the right, and the lateral safe distance from the nearest face of the cube when passing the cube. Thus, different diagnosis of state space parameters and various indicators generate multiple different test cases.

[0037] Step 120: Execute multiple test cases to obtain multiple target test metrics.

[0038] The target test metric refers to the test result of executing the test cases. For example, if the metric is the lateral safety distance, the target test metric is 0.2m.

[0039] In the embodiments of this disclosure, there are many ways to execute multiple test cases to obtain multiple target test metrics. In some implementations, each test case is executed N times to obtain N candidate test metrics corresponding to each test case; where N is a positive integer, the N candidate test metrics are processed based on preset filtering rules to obtain M candidate test metrics corresponding to each test case; where M is a positive integer less than or equal to N, the M candidate test metrics are calculated to obtain the target test metric corresponding to each test case.

[0040] In other implementations, each test case is executed directly to obtain the target test metric for each test case. The above two methods are merely examples of executing multiple test cases to obtain multiple target test metrics; this disclosure does not impose specific limitations on the method of executing multiple test cases to obtain multiple target test metrics.

[0041] Step 130: Process based on multiple target test metrics to determine the target test cases from multiple test cases.

[0042] Each test case corresponds to a target test metric. Multiple target test metrics are processed, and there are many ways to determine the target test case from multiple test cases. In some implementations, multiple target test metrics are clustered to obtain multiple cluster categories. From the multiple cluster categories, a first cluster category is obtained where the number of test cases corresponding to the target test metric is greater than one. From the test cases corresponding to the first cluster category, one test case is selected as the target test case.

[0043] In other implementations, multiple target test metrics are calculated to obtain target values, and target test cases are determined from multiple test cases based on the multiple target test metrics and target values. The above two methods are merely examples of processing multiple target test metrics to determine target test cases from multiple test cases. This disclosure does not impose specific limitations on the method of processing multiple target test metrics to determine target test cases from multiple test cases.

[0044] Step 140: Associate and store the target test scenario and target test cases in the scenario database.

[0045] In this embodiment of the disclosure, the scenario database is used to store multiple different target test scenarios and target test cases corresponding to each target test scenario, so that the number of target test cases and high-quality test cases can be directly selected based on the target test scenario to test autonomous driving, thereby further improving the test effect and efficiency.

[0046] The test scenario processing method provided in this embodiment obtains a target test scenario, generates multiple test cases for the target test scenario, executes multiple test cases to obtain multiple target test metrics, processes these multiple target test metrics, determines the target test case from the multiple test cases, and associates and stores the target test scenario and target test case in a scenario database. Using this scheme, a large number of test cases for the test scenario are generated, and high-quality test cases are selected and associated with the test scenario for storage, thus ensuring the effectiveness of subsequent autonomous driving tests.

[0047] In some embodiments, generating multiple test cases for a target test scenario includes: obtaining state space parameters of the target test scenario; obtaining the target value range of the state space parameters; and generating multiple test cases based on a preset set of judgment indicators and the target value range.

[0048] In this embodiment of the disclosure, in order to ensure the test effect, it is necessary to determine the target value range of the state space parameters, thereby assigning the corresponding values ​​to the state space parameters within the target value range, and generating multiple test cases by combining various judgment indicators.

[0049] In one specific implementation of this disclosure, obtaining the target value range of the state space parameters includes: obtaining multiple value ranges of the state space parameters under each semantic keyword, processing the multiple value ranges, and obtaining the target value range.

[0050] For example, for an atomic scene, iterate through all state space parameters. For each state space parameter, iterate through all semantic keywords and obtain the value range of the state space parameter under all semantic keywords. For example, the state space parameter of an autonomous vehicle is speed. In addition to the value range defined by the scene as [-120, 120], it has a value range of [-30, 30] according to the semantic keyword "school" and a value range of [-140, 140] according to the semantic keyword "passenger vehicle". By processing multiple value ranges, the target value range is obtained.

[0051] In the embodiments of this disclosure, there are many ways to process multiple value ranges to obtain a target value range. For example, one can directly obtain the intersection of multiple value ranges as the target value; another example is to select different processing methods for value ranges of different data formats to obtain the target value range.

[0052] Specifically, a first value range with a data format of array and continuous is obtained from multiple value ranges. The first value range is discretized according to a preset discretization dimension to obtain a first candidate value range. A second value range with a data format of array and non-contiguous is obtained from multiple value ranges. The second value range is transformed according to a preset step size to obtain a second candidate value range. A third value range with a data format of tuple is obtained from multiple value ranges as a third candidate value range. The target value range is obtained by processing based on the first candidate value range, the second candidate value range, and the third candidate value range.

[0053] For example, if the data format is an array, such as with the format symbol [], if the length is 2 values, it is continuous data; if the length is 3 values, it is non-continuous data. If the data format is a tuple, such as with the format symbol (), it is a candidate value range. Other data formats are incorrect and are returned. If the data is continuous, it is converted to non-continuous data according to a preset discrete dimension (e.g., 10%, which can be configured according to the application scenario). That is, the step size is 10% of the difference between the maximum and minimum values. For example, [-2, -1] will be converted to [-2, -1, 0.1] as a candidate value range. If the data is non-continuous, it is converted to a candidate value range according to a preset step size. For example, [-2, -1, 0.1] is converted to [-2, -1.9, -1.8, -1.7, -1.6, -1.5, -1.4, -1.3, -1.2, -1.1]. The intersection of all candidate value ranges is used as the target value range for this state space parameter.

[0054] Therefore, each value in the target range is written into the specific scenario description, and a set of judgment indicators is added according to the agreed format to generate multiple test cases.

[0055] By adopting the above scheme, a large number of test cases can be generated efficiently, which greatly improves the efficiency of test case generalization and the coverage that can be achieved, and further improves the effect of subsequent autonomous driving tests.

[0056] In some embodiments, multiple test cases are executed to obtain multiple target test metrics, including: executing each test case N times to obtain N candidate test metrics corresponding to each test case; where N is a positive integer; processing the N candidate test metrics based on preset filtering rules to obtain M candidate test metrics corresponding to each test case; where M is a positive integer less than or equal to N; and calculating the M candidate test metrics to obtain the target test metric corresponding to each test case.

[0057] In this embodiment of the disclosure, each test case is executed N times, that is, each test case can obtain N test results, that is, N candidate test indicators. In order to further improve the test accuracy, the N candidate test indicators can be filtered based on preset filtering rules to obtain M candidate test indicators less than or equal to N, which are then calculated to obtain the target test indicator corresponding to each test case.

[0058] The preset filtering rules can be selected and set according to the application scenario requirements. For example, considering the randomness that may exist in the simulation environment, such as network fluctuations that may cause singular values, or using the Grubbs criterion as the filtering rule.

[0059] In this embodiment of the disclosure, after screening, M candidate test indicators are obtained, and the average value is calculated as the target test indicator.

[0060] For example, for an atomic scenario S, all generated test cases are executed. For each test case, the test is executed N times (e.g., 1000 times, which can be configured as needed), resulting in N candidate test metrics. By using the Grubbs criterion to eliminate singular values, M candidate test metrics are obtained, and the average value is taken as the target test metric for that test case.

[0061] Therefore, by executing test cases multiple times and filtering out substandard ones, the target test metrics are determined, thereby improving the accuracy of obtaining the target test metrics and thus improving the accuracy of subsequent autonomous driving tests to meet the testing requirements.

[0062] In some embodiments, multiple target test metrics are clustered to obtain multiple cluster categories. A first cluster category with more than one test case corresponding to the target test metric is obtained from the multiple cluster categories. A test case is selected from the test cases corresponding to the first cluster category as the target test case.

[0063] In some embodiments, a second clustering category with one test case corresponding to the target test metric is obtained from multiple clustering categories. The state space value of the test case corresponding to the target test metric in the second clustering category is obtained. The state space value is processed based on a preset discrete dimension to obtain the target state space value. Based on a preset set of judgment indicators and the target state space value, a new test case is generated. The new test case is executed to obtain a new target test metric. The new target test metric is processed to obtain the target test case.

[0064] Specifically, all state space parameter values ​​of the target test scenario are clustered according to the target test index (the clustering algorithm for state space parameter values ​​can be selected as needed, such as K-Means or K-means clustering algorithm).

[0065] Specifically, if there are multiple state space parameter values ​​in the first cluster category, it means that the performance of autonomous vehicles in these specific scenarios tends to be consistent. Selecting one test case to intercept the problem is very close to the effect of intercepting all test cases. Therefore, one value can be randomly retained and other state space parameter values ​​can be removed, that is, only one test case needs to be retained.

[0066] It should be noted that all target test scenarios and test cases can also be retained and stored according to the aggregated categories. Each time a test case is executed, a target test scenario is randomly selected from each category for testing.

[0067] Specifically, if there is only one state space parameter value in the second cluster category, there may be missing valuable test cases (i.e., there are test cases that can intercept more problems but have not been generalized). Then, the state space parameter values ​​to its left and right can be used as boundaries. Based on the preset discrete dimension (e.g., 10%, which can be configured as needed) and the target state space value, new test cases can be generated, executed, and new target test indicators can be obtained. Based on the new target test indicators, the target test cases can be obtained.

[0068] Therefore, clustering can be performed based on multiple target test metrics, and test cases with the same effect can be selected. Furthermore, generalization and filtering can be repeated for individual test cases, thereby efficiently generating a large number of test cases, greatly improving the efficiency of generalization and the coverage that can be achieved. In addition, filtering can improve test coverage efficiency and also enhance the probability of discovering deep-seated or low-frequency problems brought about by randomness, further meeting the needs of autonomous driving testing.

[0069] In some embodiments, a test scenario is obtained, a matching is performed in a scenario database based on the test scenario to obtain test cases, the test cases are executed, and test results for the test scenario are generated.

[0070] In this embodiment of the disclosure, test results are generated by directly obtaining generalized and high-quality test cases from the scenario library for the scenario to be tested, thereby further ensuring the accuracy of the test results.

[0071] Figure 2 This is a flowchart of another test scenario processing method according to an embodiment of this disclosure. This embodiment further optimizes the above-described test scenario processing method based on the previous embodiment. Figure 2 As shown, the method includes:

[0072] Step 210: Obtain the target test scenario, obtain the state space parameters of the target test scenario, obtain the target value range of the state space parameters, and generate multiple test cases based on the preset judgment index set and the target value range.

[0073] In this embodiment of the disclosure, multiple value ranges of the state space parameters under each semantic keyword are obtained, and the multiple value ranges are processed to obtain the target value range.

[0074] In this embodiment of the disclosure, a first value range with a data format of array and continuous is obtained from multiple value ranges. The first value range is discretized according to a preset discrete dimension to obtain a first candidate value range. A second value range with a data format of array and non-contiguous is obtained from multiple value ranges. The second value range is transformed according to a preset step size to obtain a second candidate value range. A third value range with a data format of tuple is obtained from multiple value ranges as a third candidate value range. The target value range is obtained by processing based on the first candidate value range, the second candidate value range, and the third candidate value range.

[0075] Specifically, the test scenarios are categorized into functional scenarios, logical scenarios, and specific scenarios. The semantics of each functional scenario are obtained and semantic keywords are labeled. All parameters of the participants and environment are identified from the scenario content and detailed steps. The functional scenarios are transformed into logical scenario descriptions. Correlation analysis is performed on the parameters of all logical scenarios. The relevant parameters are transformed and normalized. A semantic-based state space dictionary is created (during the creation process, conflicting semantics are checked to ensure test accuracy). That is, the value range of each parameter under each relevant semantic keyword is defined.

[0076] Furthermore, for each atomic logical scenario (referred to as an "atomic scenario"), a specific scenario description is generated according to a predefined format (such as JSON (a lightweight data exchange format)). Parameter values ​​are left blank. The parameters in its state space are traversed. For each parameter: its semantic keywords are traversed to obtain the value range of the parameter under all semantic keywords. For each value range: all value ranges are traversed, the value range format is parsed, and if it is continuous data, it is converted into non-continuous data according to a preset discrete dimension (such as 10%). The intersection of the discrete value ranges is used as the target value range.

[0077] Finally, iterate through the target value range, write each value into the specific scenario description, add a preset set of judgment indicators in a pre-defined format (such as JSON), and generate multiple test cases.

[0078] In other words, preprocessing yields atomic specific scenario descriptions (referred to as "atomic scenarios"), which include the value range of each state space parameter under each relevant semantic keyword and an index set, with parameter values ​​left blank.

[0079] For example, the semantic description of atomic scene S0 is as follows: premise: there is only one straight road with a lane of 200m (long enough for the scene to occur), the width of the road is twice the width of the vehicle's boundary, and there is a static cube (participant, which is also an obstacle) in the road at 120m (long enough for the vehicle to complete acceleration). Scenario: there is a static cube in the road. Expectation: when the obstacle occupies no more than the left center line, the vehicle should pass by on the right side without crossing the line.

[0080] Therefore, the logical description of the atomic scene includes semantic keywords: school and passenger vehicle; environmental state space: light and rain / snow; participant state space: lane-occupying space (the corner of a cube, including multiple coordinate parameters); and vehicle state space: perception range (similar to lane-occupying space) and speed. Additionally, the judgment indicator set includes: whether a collision has occurred, whether the designated destination has been reached, the longitudinal safe distance from the nearest face of the cube when starting to move to the right, and the lateral safe distance from the nearest face of the cube when passing through it.

[0081] For example, for each atomic scenario S, iterate through the parameters in its state space. For each parameter P: iterate through its semantic keywords and obtain the value range of the parameter under all semantic keywords. Continuing with the above atomic scenario S0 as an example, one of the vehicle parameters, speed, has a value range of [-120, 120] in addition to the value range defined by the atomic scenario, a value range of [-30, 30] according to the semantic keyword "school", and a value range of [-140, 140] according to the semantic keyword "passenger vehicle". For each value range R: parse the value range format (the specific format definition is not limited, just follow the convention; the following is an example). If the format is an array, for example, the format symbol is []. If the length is 2 values, it is continuous data; if the length is 3 values, it is non-continuous data. If the format is a tuple, for example, the format symbol is (), it is a candidate value range; others are incorrect formats, return; if it is continuous... Data is transformed into non-continuous data according to a set discrete dimension (e.g., 10%, configurable), that is, the step size is 10% of the difference between the maximum and minimum values. For example, [-2, -1] will be transformed into [-2, -1, 0.1]. If it is non-continuous data, it is transformed into a candidate value range according to the step size. For example, [-2, -1, 0.1] will be transformed into [-2, -1.9, -1.8, -1.7, -1.6, -1.5, -1.4, -1.3, -1.2, -1.1]. The intersection of all candidate value ranges is used as the target value range Rf of this state space parameter. Traversing the above target value range Rf, for each value V: write this value V into the specific scenario description (a pre-agreed format), and add a preset set of judgment indicators according to the pre-agreed format (e.g., JSON, YAML (a highly readable format used to express data serialization) etc.) to generate test cases.

[0082] Step 220: Execute N times for each test case to obtain N candidate test metrics for each test case; where N is a positive integer. Process the N candidate test metrics based on preset filtering rules to obtain M candidate test metrics for each test case; where M is a positive integer less than or equal to N. Calculate the M candidate test metrics to obtain the target test metric for each test case.

[0083] Step 230: Cluster the multiple target test metrics to obtain multiple cluster categories.

[0084] Step 240: Obtain the first cluster category from multiple cluster categories where the number of test cases corresponding to the target test metric is greater than one, and select one test case from the test cases corresponding to the first cluster category as the target test case.

[0085] Step 250: Obtain a second cluster category with one test case corresponding to the target test metric from multiple cluster categories, obtain the state space value of the test case corresponding to the target test metric in the second cluster category, process the state space value based on the preset discrete dimension, and obtain the target state space value.

[0086] Step 260: Based on the preset set of judgment indicators and the target state space values, generate new test cases, execute the new test cases to obtain new target test indicators, and process the new target test indicators to obtain target test cases.

[0087] Specifically, for the atomic scenario S0 above, all test cases generated by the generalization above are executed. For each test case C, it is executed N times (e.g., 1000, configurable). For each resulting test metric K, outliers are removed (e.g., network fluctuations may cause outliers; this is done by selecting an outlier removal algorithm, such as the Grubbs criterion). The average of all test metrics is then taken as the target test metric Kv for that test case. For example, in the above example atomic scenario S0, one of the test metrics is the lateral safety distance. After removing outliers from 1000 test metrics using the Grubbs criterion, the average is taken as 0.2m.

[0088] Specifically, all state space parameter values ​​of the atomic scenario are clustered according to the target test indicators, that is, multiple target test indicators are clustered, each cluster category corresponds to a test case, and the test case corresponds to the state space parameter value.

[0089] If a cluster contains multiple test cases, i.e. multiple state space parameter values, it indicates that the vehicle's performance in these specific scenarios tends to be consistent. Using one test case to intercept the problem is very close to the effect of using all test cases to intercept it. Therefore, one test case, i.e. one state space parameter value, can be randomly retained, and the other test cases can be deleted, i.e., the other state space parameter values ​​can be eliminated. (Alternatively, all test cases can be retained and stored according to cluster categories. Each time test cases are executed, a test case can be randomly selected from each cluster category for testing.)

[0090] If a cluster category contains only one test case, i.e., only one state space parameter value, then there may be missing valuable test cases (i.e., test cases that could intercept more problems have not been generalized). In this case, the left and right state spaces of the state space parameter should be bounded by the parameter value, and the target state space value should be obtained by setting a discrete dimension (e.g., 10%, configurable). Based on the preset judgment index set and the target state space value, a new test case should be generated, the new test case should be executed, a new target test index should be obtained, and the new target test index should be processed to obtain the target test case. That is, the test case generalization process should be repeated, and subsequent clustering screening or further generalization processing should be repeated.

[0091] This allows for the efficient generation of a large number of test cases, greatly improving the efficiency of generalization and the coverage that can be achieved.

[0092] It should be noted that after step 230, steps 240 and / or steps 250-260 can be performed.

[0093] Step 270: Associate and store the target test scenario and target test cases in the scenario database.

[0094] Step 280: Obtain the scenario to be tested, match it with the scenario database to obtain the test cases, execute the test cases, and generate the test results of the scenario to be tested.

[0095] Specifically, the target test scenario and the target test cases are associated and stored in the scenario database, so that high-quality test cases can be directly selected based on the test scenario, generating effective test results and further improving test effectiveness.

[0096] For example, the atomic scenario S0 described above has 9 parameters. With a 10% discrete dimension, a single generalization generates 10^9 test cases, which takes approximately 3 minutes to execute. A high-performance server can execute 50 test cases simultaneously, requiring 10^7 hours to complete even the simplest scenario. However, the filtering method of this disclosure yields only 4 test cases for atomic scenario S0, saving 10^8 orders of magnitude in testing time or equipment, while maintaining consistent problem interception effectiveness—that is, a highly efficient test case set can be extracted.

[0097] This ensures that no effective scenarios are missed, achieving truly high-quality interception.

[0098] Figure 3 This is a schematic diagram of the structure of a test scenario processing device according to an embodiment of the present disclosure.

[0099] like Figure 3 As shown: The device includes:

[0100] The first acquisition module 310 is used to acquire the target test scenario;

[0101] The generation module 320 is used to generate multiple test cases for the target test scenario;

[0102] Execution module 330 is used to execute the multiple test cases to obtain multiple target test metrics;

[0103] The processing and determination module 340 is used to process the multiple target test indicators and determine the target test cases from the multiple test cases;

[0104] The associated storage module 350 is used to associate and store the target test scenario and the target test case in the scenario database.

[0105] Optionally, the generation module 320 includes:

[0106] The first acquisition unit is used to acquire the state space parameters of the target test scenario;

[0107] The second acquisition unit is used to acquire the target value range of the state space parameters;

[0108] The generation unit is used to generate the multiple test cases based on a preset set of judgment indicators and the target value range.

[0109] Optionally, the second acquisition unit includes:

[0110] The first acquisition subunit is used to acquire multiple value ranges of the state space parameters under each semantic keyword;

[0111] The processing subunit is used to process the multiple value ranges to obtain the target value range.

[0112] Optionally, the first acquisition subunit is specifically used for:

[0113] From the plurality of value ranges, obtain a first value range that is array-formatted and continuous, and discretize the first value range according to a preset discretization dimension to obtain a first candidate value range;

[0114] From the plurality of value ranges, obtain a second value range that is array-formatted and non-contiguous, and transform the second value range according to a preset step size to obtain a second candidate value range;

[0115] From the plurality of value ranges, a third value range in data format of tuples is selected as the third candidate value range;

[0116] The target value range is obtained by processing the first candidate value range, the second candidate value range, and the third candidate value range.

[0117] Optionally, execution module 330 is specifically used for:

[0118] For each test case, the test is executed N times to obtain N candidate test metrics for each test case; where N is a positive integer.

[0119] The N candidate test metrics are processed based on preset filtering rules to obtain M candidate test metrics corresponding to each test case; where M is a positive integer less than or equal to N.

[0120] The target test metric corresponding to each test case is obtained by calculating the M candidate test metrics.

[0121] Optionally, the processing and determination module 340 is specifically used for:

[0122] Clustering is performed on the multiple target test metrics to obtain multiple cluster categories;

[0123] From the multiple clustering categories, obtain a first clustering category where the number of test cases corresponding to the target test metric is greater than one;

[0124] Select one test case from the test cases corresponding to the first cluster category as the target test case.

[0125] Optionally, the device further includes:

[0126] The second acquisition module is used to acquire a second clustering category from the plurality of clustering categories, wherein the number of test cases corresponding to the target test metric is one;

[0127] The third acquisition module is used to acquire the state space values ​​of the test cases corresponding to the target test indicators in the second clustering category;

[0128] The processing module is used to process the state space values ​​based on a preset discrete dimension to obtain the target state space values.

[0129] The generation module is also used to generate new test cases based on a preset set of judgment indicators and the target state space values;

[0130] The execution module is also used to execute the new test cases to obtain new target test metrics;

[0131] The processing determination module is further configured to process the new target test metric to obtain the target test cases.

[0132] Optionally, the device further includes:

[0133] The third acquisition module is used to acquire the scenario to be tested;

[0134] The matching module is used to match the scenario to be tested in the scenario database to obtain the test cases;

[0135] The testing module is used to execute the test cases and generate test results for the test scenarios.

[0136] The test scenario processing apparatus provided in this disclosure embodiment can execute the steps in the test scenario processing method provided in this disclosure method embodiment, and has the execution steps and beneficial effects, which will not be repeated here.

[0137] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 4 It shows a schematic diagram of a structure suitable for implementing the electronic device 400 in the embodiments of this disclosure. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0138] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 401, which can perform various appropriate actions and processes to implement the methods of the embodiments described herein, based on a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the electronic device 400. The processing device 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0139] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0140] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the test scenario processing method as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of embodiments of this disclosure.

[0141] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0142] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a target test scenario and generate multiple test cases for the target test scenario; execute the multiple test cases to obtain multiple target test metrics; process based on the multiple target test metrics to determine a target test case from the multiple test cases; and associate and store the target test scenario and the target test case in a scenario database.

[0143] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0144] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0145] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A test scenario processing method, characterized in that, include: Obtain the target test scenario and generate multiple test cases for the target test scenario; wherein, the target test scenario refers to an atomic scenario description, and an atomic scenario refers to an indivisible scenario; Execute the multiple test cases to obtain multiple target test metrics; Based on the multiple target test metrics, a target test case is determined from the multiple test cases. The process of determining the target test case based on the multiple target test metrics includes: clustering the multiple target test metrics to obtain multiple cluster categories; obtaining a second cluster category from the multiple cluster categories where the number of test cases corresponding to the target test metric is one; obtaining the state space value of the test case corresponding to the target test metric in the second cluster category; processing the state space value based on a preset discrete dimension to obtain a target state space value; generating a new test case based on a preset set of judgment metrics and the target state space value; executing the new test case to obtain a new target test metric; and processing the new target test metric to obtain the target test case. The target test scenario and the target test case are associated and stored in the scenario database.

2. The test scenario processing method according to claim 1, characterized in that, The multiple test cases generated for the target test scenario include: Obtain the state space parameters of the target test scenario; Obtain the target range of values ​​for the state space parameters; Based on the preset set of judgment indicators and the target value range, the multiple test cases are generated.

3. The test scenario processing method according to claim 2, characterized in that, The target value range for obtaining the state space parameters includes: Obtain multiple value ranges for the state space parameters under each semantic keyword; The multiple value ranges are processed to obtain the target value range.

4. The test scenario processing method according to claim 3, characterized in that, The process of processing the multiple value ranges to obtain the target value range includes: From the plurality of value ranges, obtain a first value range that is array-formatted and continuous, and discretize the first value range according to a preset discretization dimension to obtain a first candidate value range; From the plurality of value ranges, obtain a second value range that is array-formatted and non-contiguous, and transform the second value range according to a preset step size to obtain a second candidate value range; From the plurality of value ranges, a third value range in data format of tuples is selected as the third candidate value range; The target value range is obtained by processing the first candidate value range, the second candidate value range, and the third candidate value range.

5. The test scenario processing method according to claim 1, characterized in that, The execution of the multiple test cases yields multiple target test metrics, including: For each test case, the test is executed N times to obtain N candidate test metrics for each test case; where N is a positive integer. The N candidate test metrics are processed based on preset filtering rules to obtain M candidate test metrics corresponding to each test case; where M is a positive integer less than or equal to N. The target test metric corresponding to each test case is obtained by calculating the M candidate test metrics.

6. The test scenario processing method according to claim 1, characterized in that, The method further includes: From the multiple clustering categories, obtain a first clustering category where the number of test cases corresponding to the target test metric is greater than one; Select one test case from the test cases corresponding to the first cluster category as the target test case.

7. The test scenario processing method according to claim 1, characterized in that, Also includes: Obtain the scenario to be tested; Based on the scenario to be tested, the test cases are obtained by matching them with the scenario database; Execute the test cases to be tested and generate test results for the test scenario.

8. A test scenario processing device, characterized in that, The device includes: The first acquisition module is used to acquire the target test scenario; wherein, the target test scenario refers to an atomic scenario description, and an atomic scenario refers to an indivisible scenario; The generation module is used to generate multiple test cases for the target test scenario; The execution module is used to execute the multiple test cases and obtain multiple target test metrics; A processing and determination module is used to process the multiple target test metrics and determine target test cases from the multiple test cases; wherein, the process of processing the multiple target test metrics and determining target test cases from the multiple test cases includes: clustering the multiple target test metrics to obtain multiple cluster categories; obtaining a second cluster category from the multiple cluster categories where the number of test cases corresponding to the target test metrics is one; obtaining the state space value of the test cases corresponding to the target test metrics in the second cluster category; processing the state space value based on a preset discrete dimension to obtain a target state space value; generating a new test case based on a preset set of judgment metrics and the target state space value; executing the new test case to obtain a new target test metric; and processing the new target test metric to obtain the target test case. An associated storage module is used to associate and store the target test scenario and the target test case in a scenario database.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images