Evaluation method and device of automatic driving test scene set, electronic equipment and storage medium

Abstract

The invention provides an automatic driving test scene set evaluation method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting a design operation condition and a test scene set into a natural language processing network layer of a test scene set evaluation model, and carrying out the semantic similarity calculation processing of the design operation condition and the test scene set, determining a mapping result between each test scene and a design operation condition; inputting a mapping result between each test scene and the design operation condition into a data processing network layer of the test scene set evaluation model, and calculating the coverage of the test scene set to the design operation condition and the redundancy of the test scene set based on the mapping result; and outputting the coverage and redundancy of the test scene set. According to the method, the coverage and the redundancy are realized, and quantitative evaluation on the quality of the test scene set is provided, so that resource allocation is optimized, unnecessary repeated tests are avoided, and the safety verification process of the automatic driving system is remarkably optimized.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an evaluation method, device, electronic device and storage medium for an autonomous driving test scenario set. Background Art

[0002] With the rapid development of autonomous driving technology, safety has become one of the key factors that determine whether it can be widely deployed. Before achieving fully autonomous driving, it is crucial to ensure that the vehicle can operate safely and reliably under various conditions. To this end, it is particularly important to conduct rigorous and comprehensive safety testing on autonomous vehicles. Operational Design Conditions (ODC) refers to a set of factors that autonomous vehicle suppliers should clearly specify, including but not limited to road type, weather conditions, time, traffic density and other factors.

[0003] In the current field of safety testing for autonomous vehicles, the methods for constructing test scenario sets mainly rely on expert experience or data-driven methods. Although these two methods can meet the testing needs to a certain extent, they also have some obvious limitations. The method based on expert experience is highly dependent on personal expertise and past experience, which may lead to the selection of test scenarios being highly subjective and one-sided. There may be differences of opinion between different experts, which will affect the consistency and comprehensiveness of the scenario set. Although the data-driven method can use a large amount of historical data to guide the design of test scenarios, the historical data itself may be biased and cannot fully represent all possible situations that may occur in the future. This means that even if a seemingly large scenario library has been established, there may be important safety risks that have not been effectively identified and verified. Therefore, how to evaluate the autonomous driving test scenario set has become a technical issue that cannot be underestimated. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide an evaluation method, device, electronic device and storage medium for an autonomous driving test scenario set, which uses a natural language processing network layer to calculate the design operating conditions in the test scenario and the semantic similarity between the test scenario set, so as to accurately map the scenario elements to the most similar ODC elements, and calculate the coverage and redundancy of the test scenario set for the ODC elements. The coverage and redundancy provide a quantitative evaluation of the quality of the test scenario set, helping users to intuitively understand the advantages and disadvantages of the test solution, thereby optimizing resource allocation and avoiding unnecessary repeated testing, so as to significantly optimize the safety verification process of the autonomous driving system.

[0005] The present application provides an evaluation method for an autonomous driving test scenario set, the evaluation method comprising:

[0006] Obtain the design operating conditions and test scenario set of the autonomous driving system to be tested;

[0007] Input the design operation conditions and the test scenario set into the natural language processing network layer of a pre-trained test scenario set evaluation model, perform semantic similarity calculation on the design operation conditions and the test scenario set, and determine the mapping result between each test scenario in the test scenario set and the design operation conditions; wherein the test scenario set evaluation model is a neural network model;

[0008] The mapping result between each test scenario and the designed operating conditions is input into the data processing network layer of the test scenario set evaluation model, and the coverage of the test scenario set on the designed operating conditions and the redundancy of the test scenario set are calculated based on the mapping result of each test scenario, and the coverage and redundancy of the test scenario set are output.

[0009] In a possible implementation manner, the inputting the design operation condition and the test scenario set into the natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation processing on the design operation condition and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operation condition, includes:

[0010] Performing identification generation processing on each ODC element in the design operation condition to generate an identification of each ODC element;

[0011] The natural language processing network layer performs vector conversion processing on the scenario elements of each of the test scenarios and the multiple ODC elements in the designed operating conditions to determine each scenario element vector and each ODC element vector of the test scenario;

[0012] For each of the test scenes, each scene element vector and each ODC element vector of the test scene are calculated based on cosine similarity, and the similarity value between each scene element vector of the test scene and each ODC element vector is determined, and the ODC element corresponding to the highest similarity value is used as the mapping target of the scene element;

[0013] For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, determine the mapping result between the test scenario and the design operating conditions; wherein the mapping result is a mapping list of identifiers of the ODC elements corresponding to each scenario element in the test scenario.

[0014] In a possible implementation manner, the mapping result between each test scenario and the design operating condition is input into the data processing network layer of the test scenario set evaluation model, the coverage of the test scenario set for the design operating condition and the redundancy of the test scenario set are calculated based on the mapping result of each test scenario, and the coverage and redundancy of the test scenario set are output, including:

[0015] Processing the mapping results between each test scenario and the design operating conditions to generate a marked ODC element distribution map;

[0016] Determining the coverage of the test scenario set based on a ratio between the labeled results of the ODC elements in the ODC element distribution map and the total number of the ODC elements;

[0017] The test scenarios in the test scenario set with the same identifier of the ODC element are used as repeated test scenarios, and the redundancy of the test scenario set is determined based on the ratio between the number of repeated test scenarios and the total number of test scenarios.

[0018] In a possible implementation manner, the processing of the mapping result between each test scenario and the design operating condition to generate a marked ODC element distribution map includes:

[0019] Based on the mapping result between each test scenario and the designed operating condition, marking the identification of the covered ODC elements for the test scenario;

[0020] The identifications of the ODC elements of each of the test scenarios are integrated to generate a marked ODC element distribution map.

[0021] In a possible implementation manner, after taking the test scenarios with the same identifier of the ODC element in the test scenario set as repeated test scenarios and determining the redundancy of the test scenario set based on the ratio between the number of repeated test scenarios and the total number of test scenarios, the evaluation method further includes:

[0022] Based on the ODC element distribution map and the design operation conditions, determining uncovered ODC elements, and outputting the uncovered ODC elements in the form of a list;

[0023] The repeated test scenarios and their corresponding ODC element identification lists are associated and output.

[0024] In a possible implementation manner, the design operating condition and the test scenario set are determined in the following manner:

[0025] The initial design operating conditions and the initial test scenario set are subjected to scenario structuring and normalization processing to determine the design operating conditions and the test scenario set; wherein the scenario structuring includes a road layer, an environment layer, an interaction layer, and a self-vehicle layer.

[0026] The embodiment of the present application also provides an evaluation device for an autonomous driving test scenario set, the evaluation device comprising:

[0027] An acquisition module, used to obtain the design operating conditions and test scenario set of the autonomous driving system to be tested;

[0028] A semantic similarity calculation module, used for inputting the design operation conditions and the test scenario set into the natural language processing network layer of the pre-trained test scenario set evaluation model, performing semantic similarity calculation processing on the design operation conditions and the test scenario set, and determining the mapping result between each test scenario in the test scenario set and the design operation conditions; wherein the test scenario set evaluation model is a neural network model;

[0029] A calculation processing module is used to input the mapping result between each test scenario and the designed operating conditions into the data processing network layer of the test scenario set evaluation model, calculate the coverage of the test scenario set for the designed operating conditions and the redundancy of the test scenario set based on the mapping result of each test scenario, and output the coverage and redundancy of the test scenario set.

[0030] In a possible implementation manner, when the semantic similarity calculation module is used to input the design operation conditions and the test scenario set into the natural language processing network layer of the pre-trained test scenario set evaluation model, perform semantic similarity calculation processing on the design operation conditions and the test scenario set, and determine the mapping result between each test scenario in the test scenario set and the design operation conditions, the semantic similarity calculation module is specifically used to:

[0031] Performing identification generation processing on each ODC element in the design operation condition to generate an identification of each ODC element;

[0032] The natural language processing network layer performs vector conversion processing on the scenario elements of each of the test scenarios and the multiple ODC elements in the designed operating conditions to determine each scenario element vector and each ODC element vector of the test scenario;

[0033] For each of the test scenes, each scene element vector and each ODC element vector of the test scene are calculated based on cosine similarity, and the similarity value between each scene element vector of the test scene and each ODC element vector is determined, and the ODC element corresponding to the highest similarity value is used as the mapping target of the scene element;

[0034] For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, determine the mapping result between the test scenario and the design operating conditions; wherein the mapping result is a mapping list of identifiers of the ODC elements corresponding to each scenario element in the test scenario.

[0035] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the evaluation method of the autonomous driving test scenario set as described above are performed.

[0036] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the evaluation method for the autonomous driving test scenario set as described above are executed.

[0037] An embodiment of the present application provides an evaluation method, device, electronic device and storage medium for an autonomous driving test scenario set, the evaluation method comprising: obtaining design operating conditions and a test scenario set of an autonomous driving system to be tested; inputting the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation processing on the design operating conditions and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model; inputting the mapping result between each test scenario and the design operating conditions into a data processing network layer of the test scenario set evaluation model, calculating the coverage of the test scenario set on the design operating conditions and the redundancy of the test scenario set based on the mapping result of each test scenario, and outputting the coverage and redundancy of the test scenario set. It realizes the use of natural language processing network layer to calculate the semantic similarity between the designed operating conditions in the test scenario and the test scenario set, so as to accurately map the scenario elements to the most similar ODC elements, and calculate the coverage and redundancy of the test scenario set for the ODC elements. The coverage and redundancy provide a quantitative evaluation of the quality of the test scenario set, helping users to intuitively understand the advantages and disadvantages of the test solution, thereby optimizing resource allocation and avoiding unnecessary repeated testing, so as to significantly optimize the safety verification process of the autonomous driving system.

[0038] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A flowchart of an evaluation method for an autonomous driving test scenario set provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of an evaluation method for an autonomous driving test scenario set provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the structure of an evaluation device for an autonomous driving test scenario set provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0045] First, the application scenarios to which this application is applicable are introduced. This application can be applied in the field of autonomous driving technology.

[0046] Research has found that in the current field of safety testing for autonomous vehicles, the methods for constructing test scenario sets mainly rely on expert experience or data-driven methods. Although these two methods can meet the testing needs to a certain extent, they also have some obvious limitations. The method based on expert experience is highly dependent on personal expertise and past experience, which may lead to the selection of test scenarios being highly subjective and one-sided. There may be differences of opinion between different experts, which will affect the consistency and comprehensiveness of the scenario set. Although the data-driven method can use a large amount of historical data to guide the design of test scenarios, the historical data itself may be biased and cannot fully represent all possible situations that may occur in the future. This means that even if a seemingly large scenario library has been established, there may be important safety risks that have not been effectively identified and verified. Therefore, how to evaluate the autonomous driving test scenario set has become a technical issue that cannot be underestimated.

[0047] Based on this, an embodiment of the present application provides an evaluation method for an autonomous driving test scenario set, which uses a natural language processing network layer to calculate the semantic similarity between the design operating conditions in the test scenario and the test scenario set, so as to accurately map the scenario elements to the most similar ODC elements, and calculate the coverage and redundancy of the test scenario set for the ODC elements. The coverage and redundancy provide a quantitative evaluation of the quality of the test scenario set, helping users to intuitively understand the advantages and disadvantages of the test solution, thereby optimizing resource allocation and avoiding unnecessary repeated testing, so as to significantly optimize the safety verification process of the autonomous driving system.

[0048] See also Figure 1 , Figure 1 This is a flow chart of an evaluation method for an autonomous driving test scenario set provided in an embodiment of the present application. Figure 1 As shown in , the evaluation method provided in the embodiment of the present application includes:

[0049] S101: Obtain design operating conditions and a test scenario set for the autonomous driving system to be tested.

[0050] In this step, the design operating conditions and test scenario set of the autonomous driving system to be tested are obtained.

[0051] Among them, ODC refers to the design operating conditions of a certain autonomous vehicle, which is determined by the manufacturer. ODC describes in detail the specific factors in which the system is expected to operate safely, including but not limited to road types, weather conditions, traffic density, etc. The test scenario set refers to the collection of test scenarios for the autonomous vehicle. The scenario set is a carrier for verifying the correctness of the expected functions of the tested system and the safety of the entire vehicle.

[0052] In a possible implementation manner, the design operating condition and the test scenario set are determined in the following manner:

[0053] The initial design operating conditions and the initial test scenario set are subjected to scenario structuring and normalization processing to determine the design operating conditions and the test scenario set; wherein the scenario structuring includes a road layer, an environment layer, an interaction layer, and a self-vehicle layer.

[0054] Here, the design operating conditions and test scenario sets input into the test scenario set evaluation model should be structured and standardized. For example, the test scenario can refer to the four-layer scenario architecture of road layer, environment layer, interaction layer, and vehicle layer, and the design operating conditions should also have a corresponding structural hierarchy.

[0055] Among them, the structured processing is: the input ODC (design operating conditions) and test scenario set should be structured, so that the model can be processed and analyzed easily. For example, ODC can be classified according to factors such as road type, weather conditions, traffic density, etc., and the test scenario set can also be organized according to similar standards. The normalized processing is: ensure that the input format of ODC and scenario set is consistent, so that the model can be matched and analyzed easily. For example, each element in ODC should have a clear label or index, and each scene in the test scenario set should also have a corresponding label or index.

[0056] S102: Input the design operating conditions and the test scenario set into the natural language processing network layer of a pre-trained test scenario set evaluation model, perform semantic similarity calculation on the design operating conditions and the test scenario set, and determine the mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model.

[0057] In this step, the design operating conditions and the test scenario set are input into the natural language processing network layer of the test scenario set evaluation model, and the semantic similarity calculation is performed on the design operating conditions and the test scenario set to determine the mapping result between each test scenario in the test scenario set and the design operating conditions.

[0058] Among them, the test scenario set evaluation model is obtained by iteratively training the neural network model through sample design operating conditions and sample test scenario sets.

[0059] In a possible implementation manner, the inputting the design operation condition and the test scenario set into the natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation processing on the design operation condition and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operation condition, includes:

[0060] A: Perform identification generation processing on each ODC element in the design operation condition to generate an identification of each ODC element.

[0061] Here, the identification generation process is performed on each ODC element in the design operation condition to generate an identification of each ODC element.

[0062] The identifier of an ODC element is unique. The identifier is the index or label of each ODC element. The identifier can be an ordered value or a string. For example, the road layer of ODC includes ODC elements and identifiers: expressway (1), intersection (2), urban expressway (3), etc.

[0063] B: The natural language processing network layer performs vector conversion processing on the scenario elements of each of the test scenarios and the multiple ODC elements in the design operating conditions to determine each scenario element vector and each ODC element vector of the test scenario.

[0064] Here, the natural language processing network layer performs vector conversion processing on the scenario elements of each test scenario and multiple ODC elements in the design operating conditions to determine each scenario element vector and each ODC element vector of the test scenario.

[0065] C: For each of the test scenes, each scene element vector and each ODC element vector of the test scene are calculated based on cosine similarity, the similarity value between each scene element vector and each ODC element vector of the test scene is determined, and the ODC element corresponding to the highest similarity value is used as the mapping target of the scene element.

[0066] Here, for each test scene, each scene element vector and each ODC element vector of the test scene are calculated according to the cosine similarity, the similarity value between each scene element vector and each ODC element vector of the test scene is determined, and the ODC element corresponding to the highest similarity value is used as the mapping target of the scene element.

[0067] Among them, the scene elements in the test scenario should ideally be completely equivalent to the ODC elements. However, in reality, due to the differences between the manufacturer's description of ODC and the test organization's description of the scene, the elements are also named differently. For example, the conditional element required by ODC: "light rain" may correspond to the scene element "light rain 3mm". Although the two are named differently, they refer to the same thing. Therefore, a natural language processing method is needed to map the ODC elements and scene elements that point to the same thing equivalently. The present invention uses a natural language processing network layer to map the input sentences or phrases into semantic vectors. The core is to represent sentences as high-dimensional vectors to achieve similarity calculation of input semantics.

[0068] Specifically, the model converts each scene element and each ODC element into a fixed-length vector representation. Let the vector generated by the model be v i Represents the scene element vector, u i Represents an ODC feature vector. Use cosine similarity to calculate the input scene feature v i With all ODC element vectors u i The similarity is calculated as:

[0069]

[0070] Among them, ∥v i ∥ and ∥u j ∥ are the norms of the vectors, sin(v i ,u j ) is in the range of [-1,1]. The closer it is to 1, the higher the semantic similarity. According to the similarity results, the ODC element j with the highest similarity is selected as the mapping target of the scene element. That is:

[0071]

[0072] The final output is a mapping list of unique identifiers of each scene element and its corresponding ODC element, which is used to mark the ODC elements that can be covered by the test scenario.

[0073] D: For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, determine the mapping result between the test scenario and the design operating conditions; wherein the mapping result is a mapping list of the identifiers of the ODC elements corresponding to each scenario element in the test scenario.

[0074] Here, after completing the semantic similarity matching between the scene elements and the ODC elements, the entire test scene set is traversed. For each test scene, all ODC element labels covered by the scene are recorded according to the mapping relationship established in the previous step. For example, if a scene contains the two elements "highway" and "light rain", the scene will be marked as containing ODC element 1 (mapped to the scene's "highway") and ODC element 4 (mapped to the scene's "light rain"). Finally, a mapping list of ODC element identifiers is generated for each test scene, clearly indicating which design operating conditions the scenario can verify.

[0075] S103: Input the mapping result between each test scenario and the designed operating conditions into the data processing network layer of the test scenario set evaluation model, calculate the coverage of the test scenario set for the designed operating conditions and the redundancy of the test scenario set based on the mapping result of each test scenario, and output the coverage and redundancy of the test scenario set.

[0076] In this step, the mapping results between each test scenario and the designed operating conditions are input into the data processing network layer of the test scenario set evaluation model, and the coverage of the test scenario set for the designed operating conditions and the redundancy of the test scenario set are calculated according to the mapping results of each test scenario, and the coverage and redundancy of the test scenario set are output.

[0077] Among them, coverage refers to the extent to which a given set of test scenarios can cover the design and operating conditions of the target autonomous driving system. An ideal coverage should be close to 100%, indicating that the test scenario set has fully considered all key factors that may affect system performance. Lower coverage means that there are risk areas that have not been fully tested. Redundancy refers to the proportion of repeated or similar scenarios in the test scenario set. High redundancy indicates that there are many similar test cases, which will lead to waste of resources during the testing process and fail to effectively discover new problems. Ideally, redundancy should be as low as possible to ensure that each test case corresponds to different elements in the ODC to provide unique testing value.

[0078] In a possible implementation manner, the mapping result between each test scenario and the design operating condition is input into the data processing network layer of the test scenario set evaluation model, the coverage of the test scenario set for the design operating condition and the redundancy of the test scenario set are calculated based on the mapping result of each test scenario, and the coverage and redundancy of the test scenario set are output, including:

[0079] a: Process the mapping results between each test scenario and the design operating conditions to generate a marked ODC element distribution map.

[0080] Here, the mapping result between each test scenario and the design operation condition is processed to generate a marked ODC element distribution map.

[0081] In a possible implementation manner, the processing of the mapping result between each test scenario and the design operating condition to generate a marked ODC element distribution map includes:

[0082] Based on the mapping result between each test scenario and the designed operating condition, the identification of the ODC elements covered by the test scenario is marked; the identification of the ODC elements of each test scenario is integrated to generate a distribution map of the marked ODC elements.

[0083] Here, the identification of the ODC elements covered by each test scenario is marked according to the mapping result between each test scenario and the design operation condition; the identification of the ODC elements of each test scenario is integrated to generate a distribution map of the marked ODC elements.

[0084] The ODC element distribution map shows the identifiers of the ODC elements covered by each test scenario.

[0085] b: Based on the ratio between the labeled results of the ODC elements in the ODC element distribution map and the total number of the ODC elements, the coverage of the test scenario set is determined.

[0086] Here, the number of ODC elements marked by the tested scene set in the generated ODC element distribution map is determined, and the proportion of the number of ODC elements marked by the tested scene set to the total number of ODC elements is calculated. Let N be the total number of ODC elements, n be the number of marked ODC elements, then the coverage C is

[0087] c: taking the test scenarios with the same ODC element identifier in the test scenario set as repeated test scenarios, and determining the redundancy of the test scenario set based on the ratio between the number of repeated test scenarios and the total number of test scenarios.

[0088] Here, the test scenarios with the same ODC element in the test scenario set are regarded as repeated test scenarios, and the redundancy of the test scenario set is determined according to the ratio between the number of repeated test scenarios and the total number of test scenarios.

[0089] In a specific embodiment, all test scenes are traversed to check whether their marked ODC element label lists are the same. If the ODC label lists of two or more scenes are consistent, these scenes are considered to be repeated, and the number of these repeated scenes is counted and marked so that these repeated scenes can be considered to be merged or deleted in subsequent optimization. Let M be the total number of test scenes, m is the number of repeated scenes, and the redundancy R is

[0090] In a possible implementation manner, after taking the test scenarios with the same identifier of the ODC element in the test scenario set as repeated test scenarios and determining the redundancy of the test scenario set based on the ratio between the number of repeated test scenarios and the total number of test scenarios, the evaluation method further includes:

[0091] (1): Based on the ODC element distribution map and the design operation conditions, uncovered ODC elements are determined, and the uncovered ODC elements are output in the form of a list.

[0092] Here, according to the ODC element distribution map and the design operation conditions, the ODC elements that are not marked in any test scenario are counted. These unmarked ODC elements represent the parts that are not covered in the current test scenario set, that is, coverage loss. The unmarked ODC elements are listed separately for subsequent output.

[0093] (2): The repeated test scenarios and their corresponding ODC element identification lists are associated and output.

[0094] Here, the repeated test scenario and its corresponding ODC element identification list are associated and outputted, so as to facilitate subsequent correction of the repeated test scenario.

[0095] For further information, see Figure 2 , Figure 2 A schematic diagram of an evaluation method for an autonomous driving test scenario set provided in an embodiment of the present application. Figure 2As shown, S201: the model receives the input of the element list containing the ODC of the object under test and the test scenario set to ensure the structuring of the input data; S202: generate the identification of the ODC element, calculate the semantic similarity between each scene element in the test scenario set and the ODC element, and map the scene element to the most similar ODC element; S203: traverse all test scenarios, record the identification list of the ODC elements of each test scenario according to the mapping results, and integrate the marking of the ODC elements contained in the test scenario set to generate a distribution map of the marked ODC elements; S204: determine the coverage by the proportion of the ODC elements marked in the test scenario set, count the unmarked ODC elements, and identify the coverage gap; S205: identify and count the test scenarios with the same ODC element identification in the test scenario set, obtain the number of repeated scenes in the scenario set, calculate the redundancy, and mark the identified repeated scenes; S206: output the coverage, redundancy, uncovered ODC elements and repeated test scenarios.

[0096] In this application, by generating a unique identifier for each ODC element, and using the natural language processing network layer to calculate the semantic similarity between each scenario element and the ODC element in the test scenario, a one-to-one mapping between the test scenario and the ODC element is achieved. This method can effectively overcome the traditional manual matching problem of scenarios and ODCs based on expert experience, and can automatically identify and mark the ODC elements contained in the scenario. This automated mapping process can improve work efficiency, reduce human errors, and enable the test scenario set to more accurately reflect the actual operating conditions. By calculating coverage and redundancy, a quantitative evaluation of the quality of the test scenario set is provided. Coverage reflects the coverage rate of the test scenario set for ODC elements, ensuring that all key operating conditions are fully tested; while redundancy reveals the proportion of repeated scenarios in the test scenario set, which helps to optimize resource allocation and avoid unnecessary repeated testing. Through these quantitative indicators, users can intuitively see the advantages and disadvantages of the test scenario set, so as to make targeted improvements. In addition, the output of uncovered ODC elements and repeated scenario lists provides users with specific optimization directions, making the test plan more accurate and efficient.

[0097] An embodiment of the present application provides an evaluation method for an autonomous driving test scenario set, the evaluation method comprising: obtaining design operating conditions and a test scenario set of an autonomous driving system to be tested; inputting the design operating conditions and the test scenario set into a natural language processing network layer of a pre-trained test scenario set evaluation model, performing semantic similarity calculation processing on the design operating conditions and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model; inputting the mapping result between each test scenario and the design operating conditions into a data processing network layer of the test scenario set evaluation model, calculating the coverage of the test scenario set on the design operating conditions and the redundancy of the test scenario set based on the mapping result of each test scenario, and outputting the coverage and redundancy of the test scenario set. It realizes the use of natural language processing network layer to calculate the semantic similarity between the designed operating conditions in the test scenario and the test scenario set, so as to accurately map the scenario elements to the most similar ODC elements, and calculate the coverage and redundancy of the test scenario set for the ODC elements. The coverage and redundancy provide a quantitative evaluation of the quality of the test scenario set, helping users to intuitively understand the advantages and disadvantages of the test solution, thereby optimizing resource allocation and avoiding unnecessary repeated testing, so as to significantly optimize the safety verification process of the autonomous driving system.

[0098] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of an evaluation device for an autonomous driving test scenario set provided in an embodiment of the present application. Figure 3 As shown in , the evaluation device 300 of the autonomous driving test scenario set includes:

[0099] An acquisition module 310 is used to acquire design operating conditions and a test scenario set of the autonomous driving system to be tested;

[0100] The semantic similarity calculation module 320 is used to input the design operation conditions and the test scenario set into the natural language processing network layer of the pre-trained test scenario set evaluation model, perform semantic similarity calculation processing on the design operation conditions and the test scenario set, and determine the mapping result between each test scenario in the test scenario set and the design operation conditions; wherein the test scenario set evaluation model is a neural network model;

[0101] The calculation processing module 330 is used to input the mapping result between each test scenario and the design operating condition into the data processing network layer of the test scenario set evaluation model, calculate the coverage of the test scenario set for the design operating condition and the redundancy of the test scenario set based on the mapping result of each test scenario, and output the coverage and redundancy of the test scenario set.

[0102] Further, when the semantic similarity calculation module 320 is used to input the design operation conditions and the test scenario set into the natural language processing network layer of the pre-trained test scenario set evaluation model, perform semantic similarity calculation processing on the design operation conditions and the test scenario set, and determine the mapping result between each test scenario in the test scenario set and the design operation conditions, the semantic similarity calculation module 320 is specifically used to:

[0103] Performing identification generation processing on each ODC element in the design operation condition to generate an identification of each ODC element;

[0104] The natural language processing network layer performs vector conversion processing on the scenario elements of each of the test scenarios and the multiple ODC elements in the designed operating conditions to determine each scenario element vector and each ODC element vector of the test scenario;

[0105] For each of the test scenes, each scene element vector and each ODC element vector of the test scene are calculated based on cosine similarity, and the similarity value between each scene element vector of the test scene and each ODC element vector is determined, and the ODC element corresponding to the highest similarity value is used as the mapping target of the scene element;

[0106] For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, determine the mapping result between the test scenario and the design operating conditions; wherein the mapping result is a mapping list of identifiers of the ODC elements corresponding to each scenario element in the test scenario.

[0107] Further, when the calculation processing module 330 is used to input the mapping result between each test scenario and the design operating condition into the data processing network layer of the test scenario set evaluation model, calculate the coverage of the test scenario set for the design operating condition and the redundancy of the test scenario set based on the mapping result of each test scenario, and output the coverage and redundancy of the test scenario set, the calculation processing module 330 is specifically used to:

[0108] Processing the mapping results between each test scenario and the design operating conditions to generate a marked ODC element distribution map;

[0109] Determining the coverage of the test scenario set based on a ratio between the labeled results of the ODC elements in the ODC element distribution map and the total number of the ODC elements;

[0110] The test scenarios with the same identification of the ODC element in the test scenario set are used as repeated test scenarios, and the redundancy of the test scenario set is determined based on the ratio between the number of repeated test scenarios and the total number of test scenarios.

[0111] Furthermore, when the calculation processing module 330 is used to process the mapping result between each test scenario and the design operating condition to generate the marked ODC element distribution map, the calculation processing module 330 is also used to:

[0112] Based on the mapping result between each test scenario and the designed operating condition, marking the identification of the covered ODC elements for the test scenario;

[0113] The identifications of the ODC elements of each of the test scenarios are integrated to generate a marked ODC element distribution map.

[0114] Furthermore, the calculation processing module 330 is also used for:

[0115] Based on the ODC element distribution map and the design operation conditions, determining uncovered ODC elements, and outputting the uncovered ODC elements in the form of a list;

[0116] The repeated test scenarios and their corresponding ODC element identification lists are associated and output.

[0117] Furthermore, the acquisition module 310 is also used for:

[0118] The initial design operating conditions and the initial test scenario set are subjected to scenario structuring and normalization processing to determine the design operating conditions and the test scenario set; wherein the scenario structuring includes a road layer, an environment layer, an interaction layer, and a self-vehicle layer.

[0119] An evaluation device for an autonomous driving test scenario set provided in an embodiment of the present application comprises: an acquisition module, used to acquire the design operating conditions and the test scenario set of the autonomous driving system to be tested; a semantic similarity calculation module, used to input the design operating conditions and the test scenario set into the natural language processing network layer of a pre-trained test scenario set evaluation model, perform semantic similarity calculation processing on the design operating conditions and the test scenario set, and determine the mapping result between each test scenario in the test scenario set and the design operating conditions; wherein the test scenario set evaluation model is a neural network model; a calculation processing module, used to input the mapping result between each test scenario and the design operating conditions into the data processing network layer of the test scenario set evaluation model, calculate the coverage of the test scenario set on the design operating conditions and the redundancy of the test scenario set based on the mapping result of each test scenario, and output the coverage and redundancy of the test scenario set. The natural language processing network layer is used to calculate the semantic similarity between the designed operating conditions in the test scenario and the test scenario set, so as to accurately map the scenario elements to the most similar ODC elements, and calculate the coverage and redundancy of the test scenario set for the ODC elements. The coverage and redundancy provide a quantitative evaluation of the quality of the test scenario set, helping users to intuitively understand the advantages and disadvantages of the test solution, thereby optimizing resource allocation and avoiding unnecessary repeated testing, so as to significantly optimize the safety verification process of the autonomous driving system.

[0120] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0121] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The steps of the evaluation method for the autonomous driving test scenario set in the method embodiment shown, and the specific implementation method thereof can be referred to the method embodiment, which will not be described in detail here.

[0122] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the evaluation method for the autonomous driving test scenario set in the method embodiment shown in the embodiment and the specific implementation method thereof can be found in the method embodiment and will not be described in detail here.

[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0127] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0128] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for evaluating an autonomous driving test scenario set, characterized in that: The evaluation method includes: Obtain the design and operating conditions of the autonomous driving system to be tested and the test scenario set; Input the design operation conditions and the test scenario set into the natural language processing network layer of a pre-trained test scenario set evaluation model, perform semantic similarity calculation on the design operation conditions and the test scenario set, and determine the mapping result between each test scenario in the test scenario set and the design operation conditions; wherein the test scenario set evaluation model is a neural network model; The mapping result between each test scenario and the designed operating conditions is input into the data processing network layer of the test scenario set evaluation model, and the coverage of the test scenario set on the designed operating conditions and the redundancy of the test scenario set are calculated based on the mapping result of each test scenario, and the coverage and redundancy of the test scenario set are output.

2. The evaluation method according to claim 1, characterized in that: The step of inputting the design operation condition and the test scenario set into the natural language processing network layer of the pre-trained test scenario set evaluation model, performing semantic similarity calculation processing on the design operation condition and the test scenario set, and determining a mapping result between each test scenario in the test scenario set and the design operation condition, includes: Performing identification generation processing on each ODC element in the design operation condition to generate an identification of each ODC element; The natural language processing network layer performs vector conversion processing on the scenario elements of each of the test scenarios and the multiple ODC elements in the designed operating conditions to determine each scenario element vector and each ODC element vector of the test scenario; For each of the test scenes, each scene element vector and each ODC element vector of the test scene are calculated based on cosine similarity, and the similarity value between each scene element vector of the test scene and each ODC element vector is determined, and the ODC element corresponding to the highest similarity value is used as the mapping target of the scene element; For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, determine the mapping result between the test scenario and the design operating conditions; wherein the mapping result is a mapping list of identifiers of the ODC elements corresponding to each scenario element in the test scenario.

3. The evaluation method according to claim 1, characterized in that: The mapping result between each test scenario and the designed operating condition is input into the data processing network layer of the test scenario set evaluation model, the coverage of the test scenario set for the designed operating condition and the redundancy of the test scenario set are calculated based on the mapping result of each test scenario, and the coverage and redundancy of the test scenario set are output, including: Processing the mapping results between each test scenario and the design operating conditions to generate a marked ODC element distribution map; Determining the coverage of the test scenario set based on a ratio between the labeled results of the ODC elements in the ODC element distribution map and the total number of the ODC elements; The test scenarios with the same identification of the ODC element in the test scenario set are used as repeated test scenarios, and the redundancy of the test scenario set is determined based on the ratio between the number of repeated test scenarios and the total number of test scenarios.

4. The evaluation method according to claim 3, characterized in that: The processing of the mapping result between each test scenario and the design operation condition to generate a marked ODC element distribution map includes: Based on the mapping result between each test scenario and the designed operating condition, marking the identification of the covered ODC elements for the test scenario; The identifications of the ODC elements of each of the test scenarios are integrated to generate a marked ODC element distribution map.

5. The evaluation method according to claim 3, characterized in that: After taking the test scenarios with the same identifier of the ODC element in the test scenario set as repeated test scenarios and determining the redundancy of the test scenario set based on the ratio between the number of repeated test scenarios and the total number of test scenarios, the evaluation method further includes: Based on the ODC element distribution map and the design operation conditions, determining uncovered ODC elements, and outputting the uncovered ODC elements in the form of a list; The repeated test scenarios and their corresponding ODC element identification lists are associated and output.

6. The evaluation method according to claim 1, characterized in that: The design operating conditions and the test scenario set are determined in the following manner: The initial design operating conditions and the initial test scenario set are subjected to scenario structuring and normalization processing to determine the design operating conditions and the test scenario set; wherein the scenario structuring includes a road layer, an environment layer, an interaction layer, and a self-vehicle layer.

7. An evaluation device for an autonomous driving test scenario set, characterized in that: The evaluation device comprises: An acquisition module, used to obtain the design operating conditions and test scenario set of the autonomous driving system to be tested; A semantic similarity calculation module, used for inputting the design operation conditions and the test scenario set into the natural language processing network layer of the pre-trained test scenario set evaluation model, performing semantic similarity calculation processing on the design operation conditions and the test scenario set, and determining the mapping result between each test scenario in the test scenario set and the design operation conditions; wherein the test scenario set evaluation model is a neural network model; A calculation processing module is used to input the mapping result between each test scenario and the designed operating conditions into the data processing network layer of the test scenario set evaluation model, calculate the coverage of the test scenario set for the designed operating conditions and the redundancy of the test scenario set based on the mapping result of each test scenario, and output the coverage and redundancy of the test scenario set.

8. The evaluation device according to claim 7, characterized in that The semantic similarity calculation module is used to input the design operation conditions and the test scenario set into the natural language processing network layer of the pre-trained test scenario set evaluation model, perform semantic similarity calculation processing on the design operation conditions and the test scenario set, and determine the mapping result between each test scenario in the test scenario set and the design operation conditions. The semantic similarity calculation module is specifically used to: Performing identification generation processing on each ODC element in the design operation condition to generate an identification of each ODC element; The natural language processing network layer performs vector conversion processing on the scenario elements of each of the test scenarios and the multiple ODC elements in the designed operating conditions to determine each scenario element vector and each ODC element vector of the test scenario; For each of the test scenes, each scene element vector and each ODC element vector of the test scene are calculated based on cosine similarity, and the similarity value between each scene element vector of the test scene and each ODC element vector is determined, and the ODC element corresponding to the highest similarity value is used as the mapping target of the scene element; For each of the test scenarios, based on the mapping target corresponding to each scenario element of the test scenario, determine the mapping result between the test scenario and the design operating conditions; wherein the mapping result is a mapping list of identifiers of the ODC elements corresponding to each scenario element in the test scenario.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the evaluation method of the autonomous driving test scenario set as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the evaluation method of the autonomous driving test scenario set according to any one of claims 1 to 6 are executed.

Images