An evaluation method, device, electronic device and storage medium for an autonomous driving test scenario set

Abstract

The present application provides an evaluation method, device, electronic device, and storage medium for an autonomous driving test scenario set, including: inputting design operating conditions and a test scenario set into the natural language processing network layer of a test scenario set evaluation model, performing semantic similarity calculation on the design operating conditions and the test scenario set, and determining the mapping result between each test scenario and the design operating conditions; inputting the mapping result between each test scenario and the design operating conditions into the data processing network layer of the test scenario set evaluation model, calculating the coverage of the test scenario set for the design operating conditions and the redundancy of the test scenario set based on the mapping result, and outputting the coverage and redundancy of the test scenario set. The coverage and redundancy provide a quantitative evaluation of the quality of the test scenario set, thereby optimizing resource allocation, avoiding unnecessary repeated testing, and significantly optimizing the safety verification process of the autonomous driving system.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an evaluation method, device, electronic device, and storage medium for an autonomous driving test scenario set. Background Art

[0002] With the rapid development of autonomous driving technology, safety has become a key factor in determining its widespread deployment. Before achieving fully autonomous driving, it is crucial to ensure that vehicles can operate safely and reliably under various conditions. To this end, rigorous and comprehensive safety testing of autonomous vehicles is particularly important. Operational Design Conditions (ODCs) refer to a set of factors that autonomous vehicle suppliers should clearly specify, including but not limited to road type, weather conditions, time of day, traffic density, and other factors.

[0003] In the current field of autonomous vehicle safety testing, methods for constructing test scenario sets primarily rely on expert experience or data-driven approaches. While these two approaches can meet testing needs to a certain extent, they also have some significant limitations. Expert experience-based approaches rely heavily on individual expertise and past experience, which can lead to highly subjective and biased test scenario selection. Differences in opinion may arise between experts, impacting the consistency and comprehensiveness of the scenario set. While data-driven approaches can leverage extensive historical data to guide test scenario design, historical data itself can be biased and not fully representative of all possible future scenarios. This means that even with a seemingly extensive scenario library, significant safety risks may not be effectively identified and verified. Therefore, evaluating autonomous driving test scenario sets has become a technical challenge that should not be underestimated. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an evaluation method, device, electronic device and storage medium for an autonomous driving test scenario set, which uses a natural language processing network layer to calculate the design operating conditions in the test scenario and the semantic similarity between the test scenario sets, so as to accurately map the scenario elements to the most similar ODC elements, and calculate the coverage and redundancy of the test scenario set for the ODC elements. The coverage and redundancy provide a quantitative evaluation of the quality of the test scenario set, helping users to intuitively understand the advantages and disadvantages of the test plan, thereby optimizing resource allocation, avoiding unnecessary repeated testing, and significantly optimizing the safety verification process of the autonomous driving system.

[0005] This embodiment of the present application provides an evaluation method for an autonomous driving test scenario set, the evaluation method comprising:

[0006] Obtain the design operating conditions and test scenario set of the autonomous driving system to be tested;

[0007] Inputting the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation on the design operating conditions and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model;

[0008] The mapping result between each test scenario and the designed operating conditions is input into the data processing network layer of the test scenario set evaluation model. Based on the mapping result of each test scenario, the coverage of the test scenario set for the designed operating conditions and the redundancy of the test scenario set are calculated, and the coverage and redundancy of the test scenario set are output.

[0009] In one possible implementation, inputting the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation on the design operating conditions and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operating conditions includes:

[0010] Performing identification generation processing on each ODC element in the design operation condition to generate an identification of each ODC element;

[0011] The natural language processing network layer performs vector conversion processing on the scenario elements of each test scenario and the multiple ODC elements in the designed operating conditions to determine each scenario element vector and each ODC element vector of the test scenario;

[0012] For each of the test scenes, calculating each scene element vector and each ODC element vector of the test scene based on cosine similarity, determining a similarity value between each scene element vector of the test scene and each ODC element vector, and using the ODC element corresponding to the highest similarity value as the mapping target of the scene element;

[0013] For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, the mapping result between the test scenario and the design operating conditions is determined; wherein the mapping result is a mapping list of the identifiers of the ODC elements corresponding to each of the scenario elements in the test scenario.

[0014] In one possible implementation, inputting the mapping result between each test scenario and the design operating condition into the data processing network layer of the test scenario set evaluation model, calculating the coverage of the test scenario set for the design operating condition and the redundancy of the test scenario set based on the mapping result of each test scenario, and outputting the coverage and redundancy of the test scenario set includes:

[0015] Processing the mapping results between each test scenario and the design operating conditions to generate a marked ODC element distribution map;

[0016] Determining the coverage of the test scenario set based on a ratio between the labeled results of the ODC elements in the ODC element distribution map and the total number of the ODC elements;

[0017] The test scenarios in the test scenario set with the same identifier of the ODC element are used as repeated test scenarios, and the redundancy of the test scenario set is determined based on the ratio between the number of repeated test scenarios and the total number of test scenarios.

[0018] In a possible implementation manner, processing the mapping result between each test scenario and the design operating condition to generate a marked ODC element distribution map includes:

[0019] Marking the ODC elements covered by each test scenario based on the mapping result between the test scenario and the design operating condition;

[0020] The identifications of the ODC elements of each test scenario are integrated to generate a marked ODC element distribution map.

[0021] In a possible implementation manner, after determining the redundancy of the test scenario set based on a ratio between the number of repeated test scenarios and the total number of test scenarios by taking the test scenarios with the same ODC element identifiers in the test scenario set as repeated test scenarios, the evaluation method further includes:

[0022] Based on the ODC element distribution map and the design operating conditions, determining uncovered ODC elements, and outputting the uncovered ODC elements in the form of a list;

[0023] The repeated test scenarios and their corresponding ODC element identification lists are correlated and output.

[0024] In a possible implementation manner, the design operating conditions and the test scenario set are determined in the following manner:

[0025] The initial design operating conditions and the initial test scenario set are subjected to scenario structuring and normalization processing to determine the design operating conditions and the test scenario set; wherein the scenario structuring includes a road layer, an environment layer, an interaction layer, and a vehicle layer.

[0026] The present application also provides an evaluation device for an autonomous driving test scenario set, the evaluation device comprising:

[0027] An acquisition module is used to obtain the design operating conditions and test scenario set of the autonomous driving system to be tested;

[0028] a semantic similarity calculation module, configured to input the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, perform semantic similarity calculation on the design operating conditions and the test scenario set, and determine a mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model;

[0029] A calculation and processing module is used to input the mapping results between each test scenario and the designed operating conditions into the data processing network layer of the test scenario set evaluation model, calculate the coverage of the test scenario set for the designed operating conditions and the redundancy of the test scenario set based on the mapping results of each test scenario, and output the coverage and redundancy of the test scenario set.

[0030] In one possible implementation, when the semantic similarity calculation module is used to input the design operating conditions and the test scenario set into the natural language processing network layer of a pre-trained test scenario set evaluation model, perform semantic similarity calculation processing on the design operating conditions and the test scenario set, and determine the mapping result between each test scenario in the test scenario set and the design operating conditions, the semantic similarity calculation module is specifically used to:

[0031] Performing identification generation processing on each ODC element in the design operation condition to generate an identification of each ODC element;

[0032] The natural language processing network layer performs vector conversion processing on the scenario elements of each test scenario and the multiple ODC elements in the designed operating conditions to determine each scenario element vector and each ODC element vector of the test scenario;

[0033] For each of the test scenes, calculating each scene element vector and each ODC element vector of the test scene based on cosine similarity, determining a similarity value between each scene element vector of the test scene and each ODC element vector, and using the ODC element corresponding to the highest similarity value as the mapping target of the scene element;

[0034] For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, the mapping result between the test scenario and the design operating conditions is determined; wherein the mapping result is a mapping list of the identifiers of the ODC elements corresponding to each of the scenario elements in the test scenario.

[0035] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the evaluation method for the autonomous driving test scenario set as described above are performed.

[0036] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the evaluation method for the autonomous driving test scenario set as described above are executed.

[0037] An embodiment of the present application provides an evaluation method, device, electronic device and storage medium for an autonomous driving test scenario set, the evaluation method comprising: obtaining design operating conditions and a test scenario set of an autonomous driving system to be tested; inputting the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation on the design operating conditions and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model; inputting the mapping result between each test scenario and the design operating conditions into a data processing network layer of the test scenario set evaluation model, calculating the coverage of the test scenario set on the design operating conditions and the redundancy of the test scenario set based on the mapping result of each test scenario, and outputting the coverage and redundancy of the test scenario set. The natural language processing network layer is used to calculate the semantic similarity between the design operating conditions in the test scenario and the test scenario set, so as to accurately map the scenario elements to the most similar ODC elements, and calculate the coverage and redundancy of the test scenario set for the ODC elements. The coverage and redundancy provide a quantitative evaluation of the quality of the test scenario set, helping users to intuitively understand the advantages and disadvantages of the test plan, thereby optimizing resource allocation, avoiding unnecessary repeated testing, and significantly optimizing the safety verification process of the autonomous driving system.

[0038] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A flowchart of an evaluation method for an autonomous driving test scenario set provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of an evaluation method for an autonomous driving test scenario set provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the structure of an evaluation device for an autonomous driving test scenario set provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0045] First, the application scenarios to which this application is applicable are introduced. This application can be applied in the field of autonomous driving technology.

[0046] Research has found that in the current field of autonomous vehicle safety testing, the methods for constructing test scenario sets primarily rely on expert experience or data-driven approaches. While these two approaches can meet testing needs to a certain extent, they also have some significant limitations. Expert experience-based approaches rely heavily on individual expertise and past experience, which can lead to highly subjective and biased test scenario selection. Differences in opinion may arise between experts, impacting the consistency and comprehensiveness of the scenario set. While data-driven approaches can leverage extensive historical data to guide test scenario design, historical data itself can be biased and not fully representative of all possible future scenarios. This means that even with a seemingly extensive scenario library, significant safety risks may not be effectively identified and verified. Therefore, evaluating autonomous driving test scenario sets has become a technical challenge that should not be underestimated.

[0047] Based on this, an embodiment of the present application provides an evaluation method for an autonomous driving test scenario set, which uses a natural language processing network layer to calculate the design operating conditions in the test scenario and the semantic similarity between the test scenario set, so as to accurately map the scenario elements to the most similar ODC elements, and calculate the coverage and redundancy of the test scenario set for the ODC elements. The coverage and redundancy provide a quantitative evaluation of the quality of the test scenario set, helping users to intuitively understand the advantages and disadvantages of the test plan, thereby optimizing resource allocation, avoiding unnecessary repeated testing, and significantly optimizing the safety verification process of the autonomous driving system.

[0048] See also Figure 1 , Figure 1 This is a flow chart of an evaluation method for an autonomous driving test scenario set provided in an embodiment of the present application. Figure 1 As shown in , the evaluation method provided in the embodiment of the present application includes:

[0049] S101: Obtain the design operating conditions and test scenario set of the autonomous driving system to be tested.

[0050] In this step, the design operating conditions and test scenario set of the autonomous driving system to be tested are obtained.

[0051] The ODC refers to the design operating conditions for a specific autonomous vehicle, determined by the manufacturer. The ODC details the specific factors under which the system is expected to operate safely, including but not limited to road type, weather conditions, and traffic density. The test scenario set refers to a collection of test scenarios for the autonomous vehicle. The scenario set serves as a vehicle for verifying the intended functional correctness of the system under test and the overall safety of the vehicle.

[0052] In a possible implementation manner, the design operating conditions and the test scenario set are determined in the following manner:

[0053] The initial design operating conditions and the initial test scenario set are subjected to scenario structuring and normalization processing to determine the design operating conditions and the test scenario set; wherein the scenario structuring includes a road layer, an environment layer, an interaction layer, and a vehicle layer.

[0054] Here, the design operating conditions and test scenario sets input into the test scenario set evaluation model should be structured and standardized. For example, the test scenario can refer to the four-layer scenario architecture of road layer, environment layer, interaction layer, and vehicle layer, and the design operating conditions should also have corresponding structural levels.

[0055] Among them, structured processing means that the input ODC (design operating conditions) and test scenario sets should be structured, which is convenient for model processing and analysis. For example, ODC can be classified according to factors such as road type, weather conditions, traffic density, etc., and test scenario sets can also be organized according to similar standards. Normalization processing means ensuring that the input formats of ODC and scenario sets are consistent to facilitate model matching and analysis. For example, each element in ODC should have a clear label or index, and each scene in the test scenario set should also have a corresponding label or index.

[0056] S102: Input the design operating conditions and the test scenario set into the natural language processing network layer of a pre-trained test scenario set evaluation model, perform semantic similarity calculation on the design operating conditions and the test scenario set, and determine the mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model.

[0057] In this step, the design operating conditions and the test scenario set are input into the natural language processing network layer of the test scenario set evaluation model, and the semantic similarity calculation is performed on the design operating conditions and the test scenario set to determine the mapping result between each test scenario in the test scenario set and the design operating conditions.

[0058] Among them, the test scenario set evaluation model is obtained by iteratively training the neural network model through sample design operating conditions and sample test scenario sets.

[0059] In one possible implementation, inputting the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation on the design operating conditions and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operating conditions includes:

[0060] A: Perform identification generation processing on each ODC element in the design operation condition to generate an identification for each ODC element.

[0061] Here, the identifier generation process is performed on each ODC element in the design operation condition to generate an identifier for each ODC element.

[0062] The identifier of an ODC element is unique and is the index or label of each ODC element. The identifier can be an ordered value or a string. For example, the road layer of the ODC includes ODC elements and identifiers: highway (1), intersection (2), urban expressway (3), etc.

[0063] B: The natural language processing network layer performs vector conversion processing on the scenario elements of each test scenario and the multiple ODC elements in the designed operating conditions to determine each scenario element vector and each ODC element vector of the test scenario.

[0064] Here, the natural language processing network layer performs vector conversion processing on the scenario elements of each test scenario and multiple ODC elements in the design operating conditions to determine each scenario element vector and each ODC element vector of the test scenario.

[0065] C: For each of the test scenes, each scene element vector and each ODC element vector of the test scene are calculated based on cosine similarity, and the similarity value between each scene element vector and each ODC element vector of the test scene is determined, and the ODC element corresponding to the highest similarity value is used as the mapping target of the scene element.

[0066] Here, for each test scene, each scene element vector and each ODC element vector of the test scene are calculated according to the cosine similarity, and the similarity value between each scene element vector of the test scene and each ODC element vector is determined, and the ODC element corresponding to the highest similarity value is used as the mapping target of the scene element.

[0067] Among them, the scene elements in the test scenario should ideally be completely equivalent to the ODC elements, but in fact, due to the differences between the manufacturer's description of the ODC and the test organization's description of the scene, the elements are also named differently. For example, the conditional element required by ODC: "light rain", the corresponding scene element may be "light rain 3mm", although the two are named differently, they refer to the same thing. Therefore, a natural language processing method is needed to map the ODC elements and scene elements that point to the same thing in an equivalent manner. The present invention uses a natural language processing network layer to map the input sentences or phrases into semantic vectors. The core is to represent sentences as high-dimensional vectors to achieve similarity calculation of input semantics.

[0068] Specifically, the model converts each scene element and each ODC element into a fixed-length vector representation. Let the vector generated by the model be v i Represents the scene element vector, u i Represents an ODC feature vector. Use cosine similarity to calculate the input scene feature v i With all ODC element vectors u i The similarity is calculated as follows:

[0069]

[0070] Among them, ∥v i ∥ and ∥u j ∥ are the norms of the vectors, sin(v i ,u j ) is in the range of [-1,1]. The closer it is to 1, the higher the semantic similarity. Based on the similarity results, the ODC element j with the highest similarity is selected as the mapping target of the scene element. That is:

[0071]

[0072] The final output is a mapping list of unique identifiers of each scene element and its corresponding ODC element, which is used to mark the ODC elements that can be covered by the test scene.

[0073] D: For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, determine the mapping result between the test scenario and the design operating conditions; wherein the mapping result is a mapping list of the identifiers of the ODC elements corresponding to each of the scenario elements in the test scenario.

[0074] Here, after completing the semantic similarity matching between the scene elements and the ODC elements, the entire test scene set is traversed. For each test scene, all ODC element labels covered by the scene are recorded according to the mapping relationship established in the previous step. For example, if a scene contains the two elements "highway" and "light rain", the scene will be marked as containing ODC element 1 (mapped to the scene's "highway") and ODC element 4 (mapped to the scene's "light rain"). Finally, a mapping list of ODC element identifiers is generated for each test scene, clearly indicating which design operating conditions the scenario can verify.

[0075] S103: Input the mapping result between each test scenario and the designed operating conditions into the data processing network layer of the test scenario set evaluation model, calculate the coverage of the test scenario set for the designed operating conditions and the redundancy of the test scenario set based on the mapping result of each test scenario, and output the coverage and redundancy of the test scenario set.

[0076] In this step, the mapping results between each test scenario and the designed operating conditions are input into the data processing network layer of the test scenario set evaluation model. According to the mapping results of each test scenario, the coverage of the test scenario set for the designed operating conditions and the redundancy of the test scenario set are calculated, and the coverage and redundancy of the test scenario set are output.

[0077] Among them, coverage refers to the extent to which a given set of test scenarios can cover the design and operating conditions of the target autonomous driving system. An ideal coverage should be close to 100%, indicating that the test scenario set has fully considered all key factors that may affect system performance. Lower coverage means that there are risk areas that have not been fully tested. Redundancy refers to the proportion of repeated or similar scenarios in the test scenario set. High redundancy indicates that there are many similar test cases, which will lead to waste of resources in the testing process and fail to effectively discover new problems. Ideally, redundancy should be as low as possible to ensure that each test case corresponds to different elements in the ODC to provide unique testing value.

[0078] In one possible implementation, inputting the mapping result between each test scenario and the design operating condition into the data processing network layer of the test scenario set evaluation model, calculating the coverage of the test scenario set for the design operating condition and the redundancy of the test scenario set based on the mapping result of each test scenario, and outputting the coverage and redundancy of the test scenario set includes:

[0079] a: Process the mapping results between each test scenario and the design operating conditions to generate a marked ODC element distribution map.

[0080] Here, the mapping results between each test scenario and the design operating conditions are processed to generate a marked ODC element distribution map.

[0081] In a possible implementation manner, processing the mapping result between each test scenario and the design operating condition to generate a marked ODC element distribution map includes:

[0082] Based on the mapping result between each test scenario and the designed operating condition, the test scenario is marked with the identifiers of the covered ODC elements; the identifiers of the ODC elements of each test scenario are integrated to generate a marked ODC element distribution map.

[0083] Here, the identification of the covered ODC elements of each test scenario is marked according to the mapping result between each test scenario and the design operation condition; the identification of the ODC elements of each test scenario is integrated to generate a distribution map of the marked ODC elements.

[0084] The ODC element distribution map shows the identifiers of the ODC elements covered by each test scenario.

[0085] b: Determine the coverage of the test scenario set based on the ratio between the labeled results of the ODC elements in the ODC element distribution map and the total number of the ODC elements.

[0086] Here, the number of ODC elements marked by the tested scene set in the generated ODC element distribution map is determined, and the proportion of the number of ODC elements marked by the tested scene set to the total number of ODC elements is calculated. Let N be the total number of ODC elements and n be the number of marked ODC elements, then the coverage C is

[0087] c: taking the test scenarios with the same ODC element identifier in the test scenario set as repeated test scenarios, and determining the redundancy of the test scenario set based on the ratio between the number of repeated test scenarios and the total number of test scenarios.

[0088] Here, the test scenarios with the same ODC element identifier in the test scenario set are regarded as repeated test scenarios, and the redundancy of the test scenario set is determined according to the ratio between the number of repeated test scenarios and the total number of test scenarios.

[0089] In a specific embodiment, all test scenarios are traversed to check whether their marked ODC element label lists are the same. If the ODC label lists of two or more scenarios are consistent, these scenarios are considered to be repeated. The number of these repeated scenarios is counted and marked so that these repeated scenarios can be considered for merging or deleting in subsequent optimization. Let M be the total number of test scenarios and m be the number of repeated scenarios. The redundancy R is

[0090] In a possible implementation manner, after determining the redundancy of the test scenario set based on a ratio between the number of repeated test scenarios and the total number of test scenarios by taking the test scenarios with the same ODC element identifiers in the test scenario set as repeated test scenarios, the evaluation method further includes:

[0091] (1): Based on the ODC element distribution map and the design operation conditions, uncovered ODC elements are determined, and the uncovered ODC elements are output in the form of a list.

[0092] Here, based on the ODC element distribution map and the design operating conditions, we count the ODC elements that are not marked in any test scenario. These unmarked ODC elements represent the parts that are not covered by the current test scenario set, that is, coverage gaps. These unmarked ODC elements are listed separately for subsequent output.

[0093] (2): The repeated test scenario and its corresponding ODC element identification list are associated and output.

[0094] Here, the repeated test scenario and its corresponding ODC element identification list are associated and outputted, so as to facilitate subsequent correction of the repeated test scenario.

[0095] For further information, see Figure 2 , Figure 2 This is a schematic diagram of an evaluation method for an autonomous driving test scenario set provided in an embodiment of the present application. Figure 2As shown, S201: the model receives the input of the ODC element list and the test scenario set containing the tested object, ensuring the structuring of the input data; S202: generates the identifier of the ODC element, calculates the semantic similarity between each scene element and the ODC element in the test scenario set, and maps the scene element to the most similar ODC element; S203: traverses all test scenarios, records the identifier list of the ODC element of each test scenario according to the mapping results, and integrates the marking status of the ODC elements contained in the test scenario set to generate a distribution map of the marked ODC elements; S204: the coverage is determined by the proportion of the ODC elements marked by the test scenario set, and the unmarked ODC elements are counted to identify coverage gaps; S205: the test scenarios with the same ODC element identifier in the test scenario set are identified and counted to obtain the number of repeated scenes in the scenario set, calculate the redundancy, and mark the identified repeated scenes; S206: outputs the coverage, redundancy, uncovered ODC elements and repeated test scenarios.

[0096] In this application, a one-to-one mapping between test scenarios and ODC elements is achieved by generating a unique identifier for each ODC element and using a natural language processing network layer to calculate the semantic similarity between each scenario element and the ODC element in the test scenario. This method can effectively overcome the traditional manual matching of scenarios and ODC based on expert experience and can automatically identify and label the ODC elements contained in the scenario. This automated mapping process can improve work efficiency, reduce human error, and enable the test scenario set to more accurately reflect actual operating conditions. By calculating coverage and redundancy, a quantitative evaluation of the quality of the test scenario set is provided. Coverage reflects the coverage rate of the test scenario set for ODC elements, ensuring that all key operating conditions are fully tested; redundancy reveals the proportion of repeated scenarios in the test scenario set, which helps optimize resource allocation and avoid unnecessary repeated testing. Through these quantitative indicators, users can intuitively see the strengths and weaknesses of the test scenario set and make targeted improvements. In addition, a list of uncovered ODC elements and repeated scenarios is output, providing users with specific optimization directions, making the test plan more accurate and efficient.

[0097] An embodiment of the present application provides an evaluation method for an autonomous driving test scenario set, the evaluation method comprising: obtaining design operating conditions and a test scenario set of an autonomous driving system to be tested; inputting the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation on the design operating conditions and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model; inputting the mapping result between each test scenario and the design operating conditions into a data processing network layer of the test scenario set evaluation model, calculating the coverage of the test scenario set on the design operating conditions and the redundancy of the test scenario set based on the mapping result of each test scenario, and outputting the coverage and redundancy of the test scenario set. The natural language processing network layer is used to calculate the semantic similarity between the design operating conditions in the test scenario and the test scenario set, so as to accurately map the scenario elements to the most similar ODC elements, and calculate the coverage and redundancy of the test scenario set for the ODC elements. The coverage and redundancy provide a quantitative evaluation of the quality of the test scenario set, helping users to intuitively understand the advantages and disadvantages of the test plan, thereby optimizing resource allocation, avoiding unnecessary repeated testing, and significantly optimizing the safety verification process of the autonomous driving system.

[0098] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of an evaluation device for an autonomous driving test scenario set provided in an embodiment of the present application. Figure 3 As shown in , the evaluation device 300 for the autonomous driving test scenario set includes:

[0099] An acquisition module 310 is configured to acquire design operating conditions and a test scenario set of the autonomous driving system to be tested;

[0100] a semantic similarity calculation module 320 for inputting the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation on the design operating conditions and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model;

[0101] The calculation processing module 330 is used to input the mapping results between each test scenario and the designed operating conditions into the data processing network layer of the test scenario set evaluation model, calculate the coverage of the test scenario set for the designed operating conditions and the redundancy of the test scenario set based on the mapping results of each test scenario, and output the coverage and redundancy of the test scenario set.

[0102] Furthermore, when the semantic similarity calculation module 320 is used to input the design operating conditions and the test scenario set into the natural language processing network layer of the pre-trained test scenario set evaluation model, perform semantic similarity calculation processing on the design operating conditions and the test scenario set, and determine the mapping result between each test scenario in the test scenario set and the design operating conditions, the semantic similarity calculation module 320 is specifically used to:

[0103] Performing identification generation processing on each ODC element in the design operation condition to generate an identification of each ODC element;

[0104] The natural language processing network layer performs vector conversion processing on the scenario elements of each test scenario and the multiple ODC elements in the designed operating conditions to determine each scenario element vector and each ODC element vector of the test scenario;

[0105] For each of the test scenes, calculating each scene element vector and each ODC element vector of the test scene based on cosine similarity, determining a similarity value between each scene element vector of the test scene and each ODC element vector, and using the ODC element corresponding to the highest similarity value as the mapping target of the scene element;

[0106] For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, the mapping result between the test scenario and the design operating conditions is determined; wherein the mapping result is a mapping list of the identifiers of the ODC elements corresponding to each of the scenario elements in the test scenario.

[0107] Furthermore, when the calculation processing module 330 is used to input the mapping result between each test scenario and the design operating condition into the data processing network layer of the test scenario set evaluation model, calculate the coverage of the test scenario set for the design operating condition and the redundancy of the test scenario set based on the mapping result of each test scenario, and output the coverage and redundancy of the test scenario set, the calculation processing module 330 is specifically used to:

[0108] Processing the mapping results between each test scenario and the design operating conditions to generate a marked ODC element distribution map;

[0109] Determining the coverage of the test scenario set based on a ratio between the labeled results of the ODC elements in the ODC element distribution map and the total number of the ODC elements;

[0110] The test scenarios with the same ODC element identifier in the test scenario set are used as repeated test scenarios, and the redundancy of the test scenario set is determined based on the ratio between the number of repeated test scenarios and the total number of test scenarios.

[0111] Furthermore, when the calculation processing module 330 is used to process the mapping result between each test scenario and the designed operating condition to generate the marked ODC element distribution map, the calculation processing module 330 is further used to:

[0112] Marking the ODC elements covered by each test scenario based on the mapping result between the test scenario and the design operating condition;

[0113] The identifications of the ODC elements of each test scenario are integrated to generate a marked ODC element distribution map.

[0114] Furthermore, the calculation processing module 330 is further configured to:

[0115] Based on the ODC element distribution map and the design operating conditions, determining uncovered ODC elements, and outputting the uncovered ODC elements in the form of a list;

[0116] The repeated test scenarios and their corresponding ODC element identification lists are correlated and output.

[0117] Furthermore, the acquisition module 310 is further configured to:

[0118] The initial design operating conditions and the initial test scenario set are subjected to scenario structuring and normalization processing to determine the design operating conditions and the test scenario set; wherein the scenario structuring includes a road layer, an environment layer, an interaction layer, and a vehicle layer.

[0119] An embodiment of the present application provides an evaluation device for an autonomous driving test scenario set, the evaluation device comprising: an acquisition module for acquiring design operating conditions and a test scenario set of an autonomous driving system to be tested; a semantic similarity calculation module for inputting the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation processing on the design operating conditions and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model; a calculation processing module for inputting the mapping result between each test scenario and the design operating conditions into a data processing network layer of the test scenario set evaluation model, calculating the coverage of the test scenario set on the design operating conditions and the redundancy of the test scenario set based on the mapping result of each test scenario, and outputting the coverage and redundancy of the test scenario set. The natural language processing network layer is used to calculate the semantic similarity between the designed operating conditions in the test scenario and the test scenario set, so as to accurately map the scenario elements to the most similar ODC elements, and calculate the coverage and redundancy of the test scenario set for the ODC elements. The coverage and redundancy provide a quantitative evaluation of the quality of the test scenario set, helping users to intuitively understand the advantages and disadvantages of the test plan, thereby optimizing resource allocation and avoiding unnecessary repeated testing, so as to significantly optimize the safety verification process of the autonomous driving system.

[0120] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in FIG, the electronic device 400 includes a processor 410, a memory 420 and a bus 430.

[0121] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The steps of the evaluation method for the autonomous driving test scenario set in the method embodiment shown are specifically implemented in accordance with the method embodiment and will not be described in detail here.

[0122] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the evaluation method for the autonomous driving test scenario set in the method embodiment shown are specifically implemented in accordance with the method embodiment and will not be described in detail here.

[0123] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0125] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0126] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0127] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0128] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for evaluating an autonomous driving test scenario set, characterized in that: The evaluation method includes: Obtain the design and operating conditions and test scenario set of the autonomous driving system to be tested; Inputting the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation on the design operating conditions and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model; Inputting the mapping result between each test scenario and the designed operating condition into the data processing network layer of the test scenario set evaluation model, calculating the coverage of the test scenario set for the designed operating condition and the redundancy of the test scenario set based on the mapping result of each test scenario, and outputting the coverage and redundancy of the test scenario set; Determine the mapping result between each test scenario in the test scenario set and the design operating condition by the following steps: Performing identification generation processing on each ODC element in the design operation condition to generate an identification of each ODC element; The natural language processing network layer performs vector conversion processing on the scenario elements of each test scenario and the multiple ODC elements in the designed operating conditions to determine each scenario element vector and each ODC element vector of the test scenario; For each of the test scenes, calculating each scene element vector and each ODC element vector of the test scene based on cosine similarity, determining a similarity value between each scene element vector of the test scene and each ODC element vector, and using the ODC element corresponding to the highest similarity value as the mapping target of the scene element; For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, the mapping result between the test scenario and the design operating conditions is determined; wherein the mapping result is a mapping list of the identifiers of the ODC elements corresponding to each of the scenario elements in the test scenario.

2. The evaluation method according to claim 1, wherein Inputting the mapping result between each test scenario and the design operating condition into the data processing network layer of the test scenario set evaluation model, calculating the coverage of the test scenario set for the design operating condition and the redundancy of the test scenario set based on the mapping result of each test scenario, and outputting the coverage and redundancy of the test scenario set, includes: Processing the mapping results between each test scenario and the design operating conditions to generate a marked ODC element distribution map; Determining the coverage of the test scenario set based on a ratio between the labeled results of the ODC elements in the ODC element distribution map and the total number of the ODC elements; The test scenarios with the same ODC element identifier in the test scenario set are used as repeated test scenarios, and the redundancy of the test scenario set is determined based on the ratio between the number of repeated test scenarios and the total number of test scenarios.

3. The evaluation method according to claim 2, wherein: The processing of the mapping result between each test scenario and the design operating condition to generate a marked ODC element distribution map includes: Marking the ODC elements covered by each test scenario based on the mapping result between the test scenario and the design operating condition; The identifications of the ODC elements of each test scenario are integrated to generate a marked ODC element distribution map.

4. The evaluation method according to claim 2, wherein: After determining the redundancy of the test scenario set based on a ratio between the number of repeated test scenarios and the total number of test scenarios, the evaluation method further includes: Based on the ODC element distribution map and the design operating conditions, determining uncovered ODC elements, and outputting the uncovered ODC elements in the form of a list; The repeated test scenarios and their corresponding ODC element identification lists are correlated and output.

5. The evaluation method according to claim 1, wherein: The design operating conditions and the test scenario set are determined by: The initial design operating conditions and the initial test scenario set are subjected to scenario structuring and normalization processing to determine the design operating conditions and the test scenario set; wherein the scenario structuring includes a road layer, an environment layer, an interaction layer, and a vehicle layer.

6. An evaluation device for an autonomous driving test scenario set, characterized in that: The evaluation device comprises: An acquisition module is used to obtain the design operating conditions and test scenario set of the autonomous driving system to be tested; a semantic similarity calculation module, configured to input the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, perform semantic similarity calculation on the design operating conditions and the test scenario set, and determine a mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model; a calculation processing module, configured to input the mapping result between each test scenario and the design operating condition into the data processing network layer of the test scenario set evaluation model, calculate the coverage of the test scenario set for the design operating condition and the redundancy of the test scenario set based on the mapping result of each test scenario, and output the coverage and redundancy of the test scenario set; The semantic similarity calculation module determines the mapping result between each test scenario in the test scenario set and the design operating condition through the following steps: Performing identification generation processing on each ODC element in the design operation condition to generate an identification of each ODC element; The natural language processing network layer performs vector conversion processing on the scenario elements of each test scenario and the multiple ODC elements in the designed operating conditions to determine each scenario element vector and each ODC element vector of the test scenario; For each of the test scenes, calculating each scene element vector and each ODC element vector of the test scene based on cosine similarity, determining a similarity value between each scene element vector of the test scene and each ODC element vector, and using the ODC element corresponding to the highest similarity value as the mapping target of the scene element; For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, the mapping result between the test scenario and the design operating conditions is determined; wherein the mapping result is a mapping list of the identifiers of the ODC elements corresponding to each of the scenario elements in the test scenario.

7. An electronic device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. When the processor is running, the machine-readable instructions execute the steps of the evaluation method for the autonomous driving test scenario set as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the evaluation method for the autonomous driving test scenario set according to any one of claims 1 to 5.

Images