Performance evaluation methods, devices, equipment, and storage media for autonomous driving algorithms

Abstract

This invention relates to the field of autonomous driving testing technology, and discloses a performance evaluation method, apparatus, device, and storage medium for autonomous driving algorithms, which improves the accuracy of performance evaluation, shortens the algorithm iteration cycle, and enhances development efficiency. The method acquires simulation test data of various autonomous driving algorithms in an autonomous driving system, analyzes the simulation test data to obtain abnormal indicators and corresponding scene data, determines the category of the abnormal indicators, matches the corresponding decision tree from a pre-constructed indicator diagnostic logic table based on the category, evaluates the corresponding scene data according to the evaluation content corresponding to each node in the decision tree, and generates a simulation test report of the autonomous driving algorithm based on the evaluation results.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving testing technology, and in particular to a method, apparatus, device, and storage medium for evaluating the performance of an autonomous driving algorithm. Background Technology

[0002] With the development of artificial intelligence technology, especially in the research of autonomous driving in automobiles, automatic control programs are developed to automatically control various components in vehicles. However, the scenarios to be handled in the application of autonomous driving technology are very complex and diverse. Therefore, it is necessary to conduct simulation tests in a large number of test scenarios and compare the behavior of the new algorithm with that of the original algorithm in the test scenarios to evaluate the changes in the performance of the new algorithm.

[0003] Currently, various evaluation metrics are used to assess the performance of the main vehicle in test scenarios, such as safety, comfort, and lane change completion. When comparing these metrics, the focus is on scenarios where significant changes occur. Diagnostics are used to identify scenarios where the vehicle's behavior has indeed improved or worsened. However, the identification and labeling of these scenarios are primarily done manually, which is inefficient. Summary of the Invention

[0004] This invention provides a performance evaluation method, apparatus, device, and storage medium for autonomous driving algorithms. Its main purpose is to solve the problem that the existing algorithm simulation result diagnosis process requires a large amount of manual intervention, resulting in excessively long diagnosis and evaluation time and low accuracy, which is not conducive to algorithm iterative development.

[0005] The first aspect of this invention provides a performance evaluation method for an autonomous driving algorithm, comprising:

[0006] The simulation test data of each autonomous driving algorithm in the autonomous driving system is obtained, and the simulation test data is analyzed to obtain abnormal indicators and corresponding scene data.

[0007] The category of the abnormal indicator is determined, and the corresponding decision tree is matched from the pre-built indicator diagnosis logic table based on the category, wherein the decision tree contains at least one node, and each node corresponds to a different evaluation content of the indicator.

[0008] The corresponding scenario data is evaluated according to the evaluation content of each node in the decision tree, and a simulation test report of the autonomous driving algorithm is generated based on the evaluation results.

[0009] Optionally, the step of acquiring simulation test data of each autonomous driving algorithm in the autonomous driving system and analyzing the simulation test data to obtain abnormal indicators and corresponding scene data includes:

[0010] Identify the autonomous driving algorithms to be evaluated;

[0011] Based on the aforementioned autonomous driving algorithm, corresponding real driving data and simulation test data are collected from the autonomous driving system and the algorithm simulation platform, respectively.

[0012] The real driving data is compared with the simulation test data, and the corresponding scene data is extracted from the simulation test data based on the comparison results.

[0013] The scenario data is analyzed to obtain abnormal indicators.

[0014] Optionally, the step of parsing the scene data to obtain abnormal indicators includes:

[0015] Identify the vehicle's driving trajectory and control data during the driving process from the scene data;

[0016] The vehicle's driving trajectory and the control data are matched with preset scenario definition information, and the scenario type is determined based on the matching result.

[0017] Based on the scenario type, scenario-related performance parameters are extracted from the control data, and abnormal indicators are determined based on the performance parameters.

[0018] Optionally, determining the category of the abnormal indicator and matching the corresponding decision tree from a pre-built indicator diagnosis logic table based on the category includes:

[0019] Based on the correspondence between scene types and various indicators, the category of the abnormal indicator in the scene under the given scene type is determined;

[0020] The scene-related questions are determined according to the scene type, and the corresponding annotation information is determined based on the scene-related questions. The scene-related questions are evaluation items used to evaluate the control performance of the autonomous driving algorithm in the corresponding scene.

[0021] Based on the annotation information, the corresponding decision tree is matched from the pre-constructed indicator diagnosis logic table, wherein the indicator diagnosis logic table contains decision trees for evaluating different indicators.

[0022] Optionally, the step of evaluating the corresponding scenario data according to the evaluation content corresponding to each node in the decision tree, and generating a simulation test report of the autonomous driving algorithm based on the evaluation results, includes:

[0023] The evaluation items of each trunk in the decision tree and the evaluation content of each node under the evaluation item are analyzed.

[0024] Based on the evaluation items, vehicle control parameters are extracted from the control data;

[0025] The evaluation content of each node is used as the evaluation labeling rule. The corresponding vehicle control parameters are evaluated and labeled in sequence to obtain the labeling results.

[0026] The annotation results are uploaded to the algorithm simulation platform for diagnostic evaluation, and a corresponding simulation test report is generated.

[0027] Optionally, uploading the annotation results to the algorithm simulation platform for diagnostic evaluation and generating a corresponding simulation test report includes:

[0028] Based on the annotation results, the corresponding algorithm failure problem type is determined;

[0029] Using the algorithm simulation platform, simulation diagnosis is performed using scenario models corresponding to the algorithm's fault problem types to obtain diagnostic results.

[0030] The diagnostic results are used to generate a corresponding simulation test report according to the report format corresponding to the fault problem type of the algorithm.

[0031] Optionally, before acquiring simulation test data of each autonomous driving algorithm in the autonomous driving system and analyzing the simulation test data to obtain abnormal indicators and corresponding scene data, the method further includes:

[0032] Extract the autonomous driving algorithms from the autonomous driving system and determine the scenarios corresponding to the algorithms;

[0033] The test content is determined based on the scenario, and the incident problem is constructed based on the test content;

[0034] Analyze the scenario-related issues corresponding to the aforementioned accident problem and configure the evaluation content;

[0035] The evaluation content corresponding to each accident problem is used to construct nodes and leaf nodes using a tree structure to generate a corresponding decision tree, wherein the leaf nodes are connected to the nodes, and all nodes are connected to form a decision tree.

[0036] A second aspect of the present invention provides a performance evaluation device for an autonomous driving algorithm, comprising:

[0037] The data acquisition module is used to acquire simulation test data of various autonomous driving algorithms in the autonomous driving system, and to analyze the simulation test data to obtain abnormal indicators and corresponding scene data.

[0038] A matching module is used to determine the category of the abnormal indicator and match the corresponding decision tree from a pre-built indicator diagnosis logic table based on the category. The decision tree contains at least one node, and each node corresponds to a different evaluation content of the indicator.

[0039] The evaluation module is used to evaluate the corresponding scenario data according to the evaluation content corresponding to each node in the decision tree, and generate a simulation test report of the autonomous driving algorithm based on the evaluation results.

[0040] Optionally, the acquisition module includes:

[0041] The determination unit is used to determine the autonomous driving algorithm to be evaluated;

[0042] The data acquisition unit is used to acquire corresponding real driving data and simulation test data from the autonomous driving system and the algorithm simulation platform, respectively, based on the autonomous driving algorithm.

[0043] The extraction unit is used to compare the real driving data with the simulation test data, and extract the corresponding scene data from the simulation test data based on the comparison results.

[0044] The first parsing unit is used to parse the scene data into indicators to obtain abnormal indicators.

[0045] Optionally, the first parsing unit is specifically used for:

[0046] Identify the vehicle's driving trajectory and control data during the driving process from the scene data;

[0047] The vehicle's driving trajectory and the control data are matched with preset scenario definition information, and the scenario type is determined based on the matching result.

[0048] Based on the scenario type, scenario-related performance parameters are extracted from the control data, and abnormal indicators are determined based on the performance parameters.

[0049] Optionally, the matching module includes:

[0050] The type query unit is used to determine the category of the abnormal indicator in the scenario under the scenario type based on the correspondence between the scenario type and each indicator;

[0051] The annotation unit is used to determine the corresponding scene-related questions according to the scene type, and to determine the corresponding annotation information based on the scene-related questions, wherein the scene-related questions are evaluation items used to evaluate the control performance of the autonomous driving algorithm in the corresponding scene;

[0052] The matching unit is used to match the corresponding decision tree from a pre-built indicator diagnosis logic table based on the annotation information, wherein the indicator diagnosis logic table contains decision trees for evaluating different indicators.

[0053] Optionally, the evaluation module includes:

[0054] The second parsing unit is used to parse the evaluation items of each trunk in the decision tree, as well as the evaluation content of each node under the evaluation item.

[0055] The parameter extraction unit is used to extract vehicle control parameters from the control data based on the evaluation item.

[0056] The evaluation unit is used to evaluate and label the corresponding vehicle control parameters sequentially, taking the evaluation content of each node as the evaluation labeling rules, and obtain the labeling results.

[0057] The report generation unit is used to upload the annotation results to the algorithm simulation platform for diagnostic evaluation and generate a corresponding simulation test report.

[0058] Optionally, the report generation unit is specifically used for:

[0059] Based on the annotation results, the corresponding algorithm failure problem type is determined;

[0060] Using the algorithm simulation platform, simulation diagnosis is performed using scenario models corresponding to the algorithm's fault problem types to obtain diagnostic results.

[0061] The diagnostic results are used to generate a corresponding simulation test report according to the report format corresponding to the fault problem type of the algorithm.

[0062] Optionally, the performance evaluation device for the autonomous driving algorithm further includes a decision tree construction module, which is specifically used for:

[0063] Extract the autonomous driving algorithms from the autonomous driving system and determine the scenarios corresponding to the algorithms;

[0064] The test content is determined based on the scenario, and the incident problem is constructed based on the test content;

[0065] Analyze the scenario-related issues corresponding to the aforementioned accident problem and configure the evaluation content;

[0066] The evaluation content corresponding to each accident problem is used to construct nodes and leaf nodes using a tree structure to generate a corresponding decision tree, wherein the leaf nodes are connected to the nodes, and all nodes are connected to form a decision tree.

[0067] A third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores a computer program; the at least one processor invokes the computer program in the memory to cause the computer device to execute the various steps of the performance evaluation method of the above-described autonomous driving algorithm.

[0068] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the various steps of the performance evaluation method for the above-described autonomous driving algorithm.

[0069] The technical solution provided by this invention acquires simulation test data of various autonomous driving algorithms in an autonomous driving system, analyzes the simulation test data to obtain abnormal indicators and corresponding scene data, determines the category of abnormal indicators, matches the corresponding decision tree from a pre-built indicator diagnosis logic table based on the category, evaluates the corresponding scene data according to the evaluation content corresponding to each node in the decision tree, and generates a simulation test report of the autonomous driving algorithm based on the evaluation results. This solution requires a deep understanding of autonomous vehicle algorithms and evaluation indicators, necessitates a significant investment of time by test engineers in this task, and is therefore costly. With a limited number of test engineers, the inability to promptly complete numerous version comparison and diagnosis requirements can delay algorithm iteration cycles and impact development efficiency. Furthermore, the decision tree approach avoids the problem of misdiagnosis that can easily occur due to subjective human judgment. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of an embodiment of the performance evaluation method for autonomous driving algorithms in this invention;

[0071] Figure 2 This is a schematic diagram of another embodiment of the performance evaluation method for autonomous driving algorithms in this invention;

[0072] Figure 3 This is a flowchart of the comparative evaluation in an embodiment of the present invention;

[0073] Figure 4 This is a flowchart illustrating the feedback process of the simulation test report in this embodiment of the invention.

[0074] Figure 5 This is a schematic diagram of one embodiment of the performance evaluation device for autonomous driving algorithms in this invention.

[0075] Figure 6 This is a schematic diagram of another embodiment of the performance evaluation device for the autonomous driving algorithm in this invention;

[0076] Figure 7 This is a schematic diagram of one embodiment of the computer device in this invention. Detailed Implementation

[0077] This invention provides a method, apparatus, device, and storage medium for performance evaluation of autonomous driving algorithms. It utilizes decision trees to evaluate abnormal indicators in simulation test data, obtaining evaluation results and outputting a test report. This method transforms the labeled and evaluated indicators into a decision tree, which automatically evaluates the indicators. This approach requires a deep understanding of autonomous vehicle algorithms and evaluation indicators, demanding significant time from test engineers and resulting in high costs. With a limited number of test engineers, the inability to promptly complete numerous version comparison and diagnostic requests can delay algorithm iteration cycles and impact development efficiency. Furthermore, the decision tree approach avoids the problem of misdiagnosis that can easily occur due to subjective human judgment.

[0078] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0079] It is understood that the executing entity of this invention can be a performance evaluation device for autonomous driving algorithms, or it can be a terminal or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as an example.

[0080] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the performance evaluation method for autonomous driving algorithms in this invention includes:

[0081] 101. Obtain simulation test data of each autonomous driving algorithm in the autonomous driving system, and analyze the simulation test data to obtain abnormal indicators and corresponding scenario data;

[0082] In this step, the various indicators in the autonomous driving scenario are important parameters for evaluating the merits of autonomous driving technologies such as autonomous driving algorithms or systems. During autonomous driving, abnormal control of indicators such as erratic steering wheel movements, continuous intermittent braking, and excessive acceleration in the automatic control algorithm can lead to driving accidents. By using predefined indicators of accidents or algorithm interruptions, stops, or terminations in different driving scenarios, the algorithm is simulated to extract indicators in different scenarios, and abnormal indicators are analyzed.

[0083] In this embodiment, the extraction of abnormal indicators and scene data is achieved through comparative analysis of real driving data and simulated scene data. Real driving data is primarily read from the vehicle's autonomous driving system. Based on a scene model, the read real driving data is segmented according to scene, and after segmentation, it is arranged chronologically. Code segments in the autonomous driving algorithm are parsed; these code segments should be understood as different control algorithms in autonomous driving. The segmented real scene data is then mapped to the control algorithms, and the corresponding scene type is labeled. A driving simulation program is then invoked based on the control algorithm to perform simulation and obtain simulation test data.

[0084] Then, the corresponding simulation test data and real scene data are analyzed separately to extract key information. The key information of the two are compared one by one. Based on the comparison results, the inconsistencies are selected as abnormal indicators. For the abnormal indicators, the corresponding data are extracted from the simulation test data and real scene data. The extracted data from both sides are combined to obtain the scene data corresponding to the abnormal indicators.

[0085] 102. Determine the category of abnormal indicators, and match the corresponding decision tree from the pre-built indicator diagnosis logic table based on the category;

[0086] In this step, each decision tree corresponds to one indicator. The indicator diagnosis logic table contains decision trees for multiple indicators across multiple scenarios. This decision tree can be understood as a set of rules reflecting the indicator diagnosis logic and content. Each decision tree contains at least one node, and each node corresponds to different evaluation contents of the indicator. The indicator category can be understood as the fault category that occurs during vehicle driving. Specifically, the corresponding scenario category is determined by analyzing scenario data. Based on the scenario category, multiple indicators and their corresponding fault categories are parsed out. Fault categories include unrealistic scenario problems, evaluation indicator algorithm problems, incomparable scenarios, planning algorithm problems, etc. Based on the fault category, corresponding abnormal indicators are matched to obtain the category of the abnormal indicator. Then, the indicator diagnosis logic table is queried based on the category to retrieve the matching decision tree.

[0087] In practical applications, firstly, based on the scenario information corresponding to the scenario data, all decision trees belonging to the scenario information are found from the indicator diagnosis logic table. Then, based on the matching information corresponding to the abnormal indicators, the corresponding decision tree is selected from all the found decision trees. The root node and child nodes of the decision tree are extracted, and an evaluation rule sequence is constructed. This evaluation rule sequence is obtained by sorting all child nodes sequentially with the root node as the first element.

[0088] 103. Evaluate the corresponding scenario data according to the evaluation content of each node in the decision tree, and generate a simulation test report of the autonomous driving algorithm based on the evaluation results.

[0089] In this step, the evaluation content of each node is obtained by parsing the knowledge graph of the decision tree. First, the root nodes in the decision tree are identified. Starting from the root node, the branch points on the tree trunk where the root node is located are extracted in turn by parsing upwards. Then, the child nodes on the branches corresponding to the branch points are extracted in turn according to the upward order of each branch point, and the content of each child node is extracted to obtain the evaluation content. The evaluation content is the scenario-related questions and corresponding standards or conditions proposed based on the evaluation index.

[0090] Then, based on the extracted questions, standards, or conditions, the corresponding parameters in the scene data are evaluated, and a result decision tree is output based on the evaluation results. A simulation test report of the autonomous driving algorithm is generated based on the result decision tree.

[0091] In this embodiment, the result decision tree is obtained by evaluating the corresponding parameters using the evaluation content on the child nodes, and then replacing the evaluation content on the child nodes with the evaluation result.

[0092] Furthermore, before evaluating the scene data based on the decision tree, the process includes parsing the scene data based on scene-related questions at the child nodes of the decision tree, constructing a data structure tree corresponding to the decision tree. Then, based on the data structure tree, each child node of the decision tree is matched with each child node of the data structure tree. Finally, based on the matching results, the criteria or conditions at each child node of the decision tree are used to evaluate the corresponding child node data in the data structure tree, the evaluation results are calculated, and a simulation test report is generated based on the evaluation results.

[0093] In this embodiment of the invention, to improve the accuracy of algorithm performance evaluation, a decision tree is constructed to evaluate the evaluation metrics in the algorithm. Based on the decision tree, the metric data in the obtained simulation and scenario data of the algorithm execution are evaluated, thus achieving automatic metric evaluation. This evaluation method has a unified evaluation standard, ensuring the accuracy of the evaluation. Furthermore, the implementation of the decision tree improves the efficiency of the evaluation, shortens the cycle of algorithm iteration based on the evaluation results, and enhances development efficiency. At the same time, the decision tree approach avoids the problem of misdiagnosis that can easily occur due to subjective judgment.

[0094] Please see Figure 2 Another embodiment of the performance evaluation method for autonomous driving algorithms in this invention includes:

[0095] 201. Identify the autonomous driving algorithm to be evaluated, collect the corresponding simulation test data for analysis, and obtain abnormal indicators and corresponding scenario data;

[0096] Furthermore, before step 201, the process includes: collecting control algorithms in the current vehicle to be evaluated, classifying the control algorithms according to the scenario to obtain a set of algorithms to be evaluated, collecting data based on the set of algorithms, and executing step 201.

[0097] In this embodiment, after determining the algorithm to be evaluated, the data is prepared through the following steps:

[0098] Identify the autonomous driving algorithms to be evaluated;

[0099] Based on the aforementioned autonomous driving algorithm, corresponding real driving data and simulation test data are collected from the autonomous driving system and the algorithm simulation platform, respectively.

[0100] The real driving data is compared with the simulation test data, and the corresponding scene data is extracted from the simulation test data based on the comparison results.

[0101] The scenario data is analyzed to obtain abnormal indicators.

[0102] In this embodiment, the method is mainly used to compare and evaluate new and old versions of autonomous driving algorithms to determine the stability of the new algorithm. Specifically, when analyzing the algorithm data to obtain abnormal indicators and scenario data, this step involves acquiring simulation test data and real driving data for both the new and old versions of the autonomous driving algorithm. Then, the indicators of the new and old versions are analyzed separately. The indicator data from the simulation and real scenarios are then compared. If anomalies are identified, the indicator data from both the new and old versions is extracted and compared with the abnormal indicator. If both comparisons result in anomalies, the indicator is identified as an abnormal indicator, and the corresponding scenario data from both the new and old versions is extracted.

[0103] In this embodiment, the process of parsing scene data to obtain abnormal indicators can be achieved in the following way:

[0104] Identify the vehicle's driving trajectory and control data during the driving process from the scene data;

[0105] The vehicle's driving trajectory and the control data are matched with preset scenario definition information, and the scenario type is determined based on the matching result.

[0106] Based on the scenario type, scenario-related performance parameters are extracted from the control data, and abnormal indicators are determined based on the performance parameters.

[0107] In practical applications, different metrics are set for different scenarios. The corresponding metrics are selected based on the scenario type, and the metric data in the control data are extracted in sequence. That is, the scenario-related performance parameters are extracted. Then, the performance parameters are compared between the old and new versions to identify abnormal metrics.

[0108] 202. Based on the correspondence between scenario types and various indicators, determine the category of abnormal indicators within the scenario under the scenario type;

[0109] It should be noted that each scenario contains multiple types of indicators, and each type of indicator corresponds to a fault problem, such as unrealistic scenario problems, evaluation indicator algorithm problems, incomparable scenarios, planning algorithm problems, etc. Each category contains multiple indicators, such as safety, comfort, lane changing capability, etc. Each indicator corresponds to a decision tree, which is actually the scoring algorithm model for that indicator.

[0110] Specifically, the correspondence between scene types and various indicators is actually the correspondence between scene types and indicator categories. The indicator category corresponding to the scene data is retrieved, and then each indicator within that category is matched against the abnormal indicators to determine the category of the abnormal indicator. In practical applications, matching can also be based on a matching code. For example, a random number generation algorithm can be used to first calculate a first random number associated with the abnormal indicator and the scene type. Then, a second random number can be calculated based on the same algorithm to associate the scene type with the indicator category. The first and second random numbers are compared; if they match, the category of the abnormal indicator is determined. To ensure comparability, the random number generation rules must first be configured as equality rules between scene type and indicator, and between scene type and category. Then, the random number generation algorithm learns these two equality rules. Finally, in application, the learned random number generation algorithm is used to calculate the category of the abnormal indicator.

[0111] 203. Determine the corresponding scene-related questions based on the scene type, and determine the corresponding annotation information based on the scene-related questions;

[0112] In this step, the determination of the annotation information is specifically achieved through the annotation model. The annotation model extracts the problem features in the relevant problems of the scenario, and the category number is calculated based on the problem features and the classification algorithm to obtain the annotation information.

[0113] 204. Based on the annotation information, match the corresponding decision tree from the pre-built indicator diagnosis logic table;

[0114] In this step, the scenario-related questions are evaluation items used to assess the control performance of the autonomous driving algorithm in the corresponding scenario; the indicator diagnosis logic table contains decision trees for evaluating different indicators.

[0115] In this embodiment, the decision tree can be understood as a logic tree constructed based on the evaluation algorithm of the autonomous driving algorithm, and its specific implementation is as follows:

[0116] Extract the autonomous driving algorithms from the autonomous driving system and determine the scenarios corresponding to the algorithms;

[0117] The test content is determined based on the scenario, and the incident problem is constructed based on the test content;

[0118] Analyze the scenario-related issues corresponding to the aforementioned accident problem and configure the evaluation content;

[0119] The evaluation content corresponding to each accident problem is used to construct nodes and leaf nodes using a tree structure to generate a corresponding decision tree, wherein the leaf nodes are connected to the nodes, and all nodes are connected to form a decision tree.

[0120] In practical applications, a labeling information field is added to the standard diagnostic logic table. After the labeling information is determined, a decision tree whose field information is equal to the labeling information is matched from the indicator diagnostic logic table based on the labeling information. Then, the decision trees selected by the labeling information are filtered based on the category to obtain the decision tree corresponding to the abnormal indicator.

[0121] 205. Analyze the evaluation items of each trunk in the decision tree, and the evaluation content of each node under the evaluation item;

[0122] In practical applications, decision trees are structured by logically organizing various evaluation metrics into a tree-like structure. Each internal node of a decision tree represents a scenario-related question based on that evaluation metric, while the leaf nodes represent different diagnostic results. Specifically, the decision tree is constructed by obtaining the corresponding decision tree based on the type of evaluation metrics for the scenario. The final evaluation result for the scenario is obtained by answering the questions, and the decision tree that meets the given conditions is output based on the evaluation results.

[0123] 206. Based on the evaluation items, extract the vehicle control parameters from the control data;

[0124] In this step, after extracting the evaluation items from the decision tree, the control data is extracted and classified based on the evaluation items to generate a vehicle control parameter structure diagram that matches the decision tree. This structure diagram is in the form of a tree structure.

[0125] 207. The evaluation content of each node is used as the evaluation labeling rule. The corresponding vehicle control parameters are evaluated and labeled in sequence to obtain the labeling results.

[0126] 208. Upload the annotation results to the algorithm simulation platform for diagnostic evaluation and generate the corresponding simulation test report.

[0127] Understandably, simulation test reports are output separately based on different problems or scenarios. For example, if the scenario is not realistic, the report is given to the smart agent team to develop more realistic interactions between obstacles and the main vehicle; if the evaluation index algorithm is not realistic, the report is given to the evaluation algorithm engineer to optimize the accuracy of the algorithm; if the scenario is not comparable, the report is given to the simulation group to optimize the simulation method for version comparison; and if the planning algorithm is not realistic, the report is given to the planning algorithm engineer for planning algorithm iteration.

[0128] Specifically, this step involves determining the corresponding algorithm failure problem type based on the annotation results;

[0129] Using the algorithm simulation platform, simulation diagnosis is performed using scenario models corresponding to the algorithm's fault problem types to obtain diagnostic results.

[0130] The diagnostic results are used to generate a corresponding simulation test report according to the report format corresponding to the fault problem type of the algorithm.

[0131] In practical applications, the above steps are mainly based on a system consisting of a simulation testing platform, a labeling platform, and a database to evaluate the performance of autonomous driving algorithms. Figure 3 As shown, firstly, the algorithm engineer submits a test task on the cloud simulation computing platform. After the task is completed, the scenario in which the evaluation index changes is sent to the annotation platform for diagnosis. The annotation platform displays the corresponding decision tree according to different evaluation index categories. The annotator annotates according to the decision tree logic. Finally, the diagnostic results of the annotation platform are written into the simulation result database, a test report is generated and sent to the algorithm engineer.

[0132] The decision tree for evaluating the algorithm uses a tree structure to diagnose the simulation comparison results based on the algorithm type, classifying internal nodes according to the type of accident / problem. The results of each diagnosis are fed back to the corresponding algorithm team to drive algorithm development. See the flowchart for the feedback process of the diagnostic results. Figure 4 Issues with unrealistic scenarios are reported to the smart agent team to develop more realistic interactions between obstacles and the main vehicle; issues with evaluation metric algorithms are reported to evaluation algorithm engineers to optimize the accuracy of the algorithms; issues with incomparable scenarios are reported to the simulation team for version comparison and optimization of simulation methods; and issues with planning algorithms are reported to planning algorithm engineers for planning algorithm iteration.

[0133] In practical applications, the development of decision trees for evaluation indicators is based on accident problem classification. The realism of both scenarios is determined through problem diagnosis, and the quality of the vehicle's behavior is judged based on this realism. Taking a safety degradation decision tree as an example, the internal nodes in the decision tree are judged using the following questions:

[0134] 1. Based on the camera footage and the safety officer's audio recordings, determine the type of error in the driving behavior of the vehicle involved in the original accident;

[0135] 2. Was the collision caused by a delayed detection of the object, preventing the vehicle from braking in time?

[0136] 3. If the object involved in the collision rear-ends the main vehicle, did the main vehicle brake at a speed of less than -2 m / s before the collision?

[0137] 4. In road testing, is the behavior of the main vehicle towards obstacles that pose a safety risk, as in simulation, reckless?

[0138] 5. Whether the interaction between obstacles and the main vehicle is realistic in both software versions;

[0139] 6. In version b, is there a risk of collision when the main vehicle interacts with objects that the evaluation algorithm identifies as dangerous?

[0140] In this embodiment of the invention, a decision tree approach is used to evaluate abnormal indicators in simulation test data, obtain evaluation results, and output a test report. This method transforms the indicator annotation and evaluation content into a decision tree, which automatically evaluates the indicators by executing the decision tree. This eliminates the need for diagnostic personnel to have knowledge of autonomous vehicle algorithms and evaluation algorithms, which typically involves a long training period. By transforming the diagnostic logic into a decision tree, data annotators can perform version comparison diagnostics, significantly reducing personnel training time and costs.

[0141] Meanwhile, the judgment of version comparison diagnosis is relatively subjective. By standardizing the diagnostic logic into a tree-like logical structure, the number of misdiagnoses can be effectively reduced.

[0142] Furthermore, version comparison diagnostics have high requirements for timeliness. Previously, due to the limited number of test engineers, it was impossible to quickly complete all diagnostic needs. With annotation by annotators, diagnostic tasks can be completed more quickly, effectively accelerating the iteration speed of planning algorithms.

[0143] The performance evaluation method for the autonomous driving algorithm in the embodiments of the present invention has been described above. The performance evaluation device for the autonomous driving algorithm in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 5 One embodiment of the performance evaluation device for autonomous driving algorithms in this invention includes:

[0144] The acquisition module 501 is used to acquire simulation test data of each autonomous driving algorithm in the autonomous driving system, and analyze the simulation test data to obtain abnormal indicators and corresponding scene data.

[0145] The matching module 502 is used to determine the category of the abnormal indicator and match the corresponding decision tree from a pre-built indicator diagnosis logic table based on the category. The decision tree contains at least one node, and each node corresponds to a different evaluation content of the indicator.

[0146] The evaluation module 503 is used to evaluate the corresponding scenario data according to the evaluation content corresponding to each node in the decision tree, and generate a simulation test report of the autonomous driving algorithm based on the evaluation results.

[0147] In this embodiment of the invention, an evaluation decision tree is constructed from the evaluation metrics in the algorithm. Based on this decision tree, the metric data in the obtained simulation and scenario data of the algorithm execution are evaluated, thus achieving automatic metric evaluation. This evaluation method ensures consistent evaluation standards and accuracy. Furthermore, the implementation of the decision tree improves evaluation efficiency, shortens the algorithm iteration cycle based on evaluation results, and enhances development efficiency. Simultaneously, the decision tree approach avoids the problem of misdiagnosis that can easily occur due to subjective human judgment.

[0148] Please see Figure 6 Another embodiment of the performance evaluation device for autonomous driving algorithms in this invention includes:

[0149] The acquisition module 501 is used to acquire simulation test data of each autonomous driving algorithm in the autonomous driving system, and analyze the simulation test data to obtain abnormal indicators and corresponding scene data.

[0150] The matching module 502 is used to determine the category of the abnormal indicator and match the corresponding decision tree from a pre-built indicator diagnosis logic table based on the category. The decision tree contains at least one node, and each node corresponds to a different evaluation content of the indicator.

[0151] The evaluation module 503 is used to evaluate the corresponding scenario data according to the evaluation content corresponding to each node in the decision tree, and generate a simulation test report of the autonomous driving algorithm based on the evaluation results.

[0152] Optionally, the acquisition module 501 includes:

[0153] The determining unit 5011 is used to determine the autonomous driving algorithm to be evaluated;

[0154] The acquisition unit 5012 is used to acquire corresponding real driving data and simulation test data from the autonomous driving system and the algorithm simulation platform, respectively, based on the autonomous driving algorithm.

[0155] Extraction unit 5013 is used to compare the real driving data with the simulation test data, and extract the corresponding scene data from the simulation test data based on the comparison result;

[0156] The first parsing unit 5014 is used to parse the scene data into indicators to obtain abnormal indicators.

[0157] Optionally, the first parsing unit 5014 is specifically used for:

[0158] Identify the vehicle's driving trajectory and control data during the driving process from the scene data;

[0159] The vehicle's driving trajectory and the control data are matched with preset scenario definition information, and the scenario type is determined based on the matching result.

[0160] Based on the scenario type, scenario-related performance parameters are extracted from the control data, and abnormal indicators are determined based on the performance parameters.

[0161] Optionally, the matching module 502 includes:

[0162] The type query unit 5021 is used to determine the category of the abnormal indicator in the scene under the scene type based on the correspondence between the scene type and each indicator;

[0163] The annotation unit 5022 is used to determine the corresponding scene-related questions according to the scene type, and to determine the corresponding annotation information based on the scene-related questions, wherein the scene-related questions are evaluation items used to evaluate the control performance of the autonomous driving algorithm in the corresponding scene;

[0164] The matching unit 5023 is used to match the corresponding decision tree from the pre-built indicator diagnosis logic table based on the annotation information, wherein the indicator diagnosis logic table contains decision trees for evaluating different indicators.

[0165] Optionally, the evaluation module 503 includes:

[0166] The second parsing unit 5031 is used to parse the evaluation items of each trunk in the decision tree, as well as the evaluation content of each node under the evaluation item.

[0167] The parameter extraction unit 5032 is used to extract vehicle control parameters from the control data based on the evaluation item;

[0168] Evaluation unit 5033 is used to evaluate and label the corresponding vehicle control parameters in sequence by taking the evaluation content of each node as the evaluation labeling rules, and obtain the labeling results.

[0169] The report generation unit 5034 is used to upload the annotation results to the algorithm simulation platform for diagnostic evaluation and generate a corresponding simulation test report.

[0170] Optionally, the report generation unit 5034 is specifically used for:

[0171] Based on the annotation results, the corresponding algorithm failure problem type is determined;

[0172] Using the algorithm simulation platform, simulation diagnosis is performed using scenario models corresponding to the algorithm's fault problem types to obtain diagnostic results.

[0173] The diagnostic results are used to generate a corresponding simulation test report according to the report format corresponding to the fault problem type of the algorithm.

[0174] Optionally, the performance evaluation device for the autonomous driving algorithm further includes a decision tree construction module 504, which is specifically used for:

[0175] Extract the autonomous driving algorithms from the autonomous driving system and determine the scenarios corresponding to the algorithms;

[0176] The test content is determined based on the scenario, and the incident problem is constructed based on the test content;

[0177] Analyze the scenario-related issues corresponding to the aforementioned accident problem and configure the evaluation content;

[0178] The evaluation content corresponding to each accident problem is used to construct nodes and leaf nodes using a tree structure to generate a corresponding decision tree, wherein the leaf nodes are connected to the nodes, and all nodes are connected to form a decision tree.

[0179] In this embodiment of the invention, a decision tree approach is used to evaluate abnormal indicators in simulation test data, obtain evaluation results, and output a test report. This method transforms the labeling and evaluation content of the indicators into a decision tree, which automatically evaluates the indicators. This approach requires a deep understanding of autonomous vehicle algorithms and evaluation indicators, necessitating significant time investment from test engineers and resulting in high costs. With a limited number of test engineers, the inability to promptly complete numerous version comparison and diagnostic requests can delay algorithm iteration cycles and impact development efficiency. Furthermore, the decision tree approach avoids the problem of misdiagnosis that can easily occur due to subjective human judgment.

[0180] above Figure 5 and Figure 6 The performance evaluation device for the autonomous driving algorithm in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The computer device in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0181] Figure 7 This is a schematic diagram of the structure of a computer device 700 provided in an embodiment of the present invention. The computer device 700 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 710 (e.g., one or more processors) and a memory 720, and one or more storage media 730 (e.g., one or more mass storage devices) for storing application programs 733 or data 732. The memory 720 and storage media 730 can be temporary or persistent storage. The program stored in the storage media 730 may include one or more modules (not shown in the diagram), each module including a series of computer program operations on the computer device 700. Furthermore, the processor 710 may be configured to communicate with the storage media 730 and execute the series of computer program operations in the storage media 730 on the computer device 700.

[0182] Computer device 700 may also include one or more power supplies 740, one or more wired or wireless network interfaces 550, one or more input / output interfaces 760, and / or one or more operating systems 731, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 7 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0183] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer-readable computer program, which, when executed by the processor, causes the processor to perform the various steps of the performance evaluation method of the autonomous driving algorithm in the above embodiments.

[0184] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein a computer program is stored in the computer program, and when the computer program is run on a computer, the computer executes the various steps of the performance evaluation method of the autonomous driving algorithm.

[0185] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0186] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several computer programs to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A performance evaluation method for an autonomous driving algorithm, characterized in that, The performance evaluation method for the autonomous driving algorithm includes: Extract the autonomous driving algorithm and the evaluation algorithm of the autonomous driving algorithm from the autonomous driving system, and construct a decision tree based on the evaluation algorithm; The simulation test data of each autonomous driving algorithm in the autonomous driving system is obtained, and the simulation test data is analyzed to obtain abnormal indicators and corresponding scene data. The category of the abnormal indicator is determined, and the corresponding decision tree is matched from the pre-built indicator diagnosis logic table based on the category, wherein the decision tree contains at least one node, and each node corresponds to a different evaluation content of the indicator. The corresponding scenario data is evaluated according to the evaluation content of each node in the decision tree, and a simulation test report of the autonomous driving algorithm is generated based on the evaluation results.

2. The performance evaluation method for autonomous driving algorithms according to claim 1, characterized in that, The process of acquiring simulation test data for each autonomous driving algorithm in the autonomous driving system and analyzing the simulation test data to obtain abnormal indicators and corresponding scene data includes: Identify the autonomous driving algorithms to be evaluated; Based on the aforementioned autonomous driving algorithm, corresponding real driving data and simulation test data are collected from the autonomous driving system and the algorithm simulation platform, respectively. The real driving data is compared with the simulation test data, and the corresponding scene data is extracted from the simulation test data based on the comparison results. The scenario data is analyzed to obtain abnormal indicators.

3. The performance evaluation method for autonomous driving algorithms according to claim 2, characterized in that, The process of parsing the scenario data to obtain abnormal indicators includes: Identify the vehicle's driving trajectory and control data during the driving process from the scene data; The vehicle's driving trajectory and the control data are matched with preset scenario definition information, and the scenario type is determined based on the matching result. Based on the scenario type, scenario-related performance parameters are extracted from the control data, and abnormal indicators are determined based on the performance parameters.

4. The performance evaluation method for autonomous driving algorithms according to claim 3, characterized in that, The step of determining the category of the abnormal indicator and matching the corresponding decision tree from a pre-built indicator diagnosis logic table based on the category includes: Based on the correspondence between scene types and various indicators, the category of the abnormal indicator in the scene under the given scene type is determined; The scene-related questions are determined according to the scene type, and the corresponding annotation information is determined based on the scene-related questions. The scene-related questions are evaluation items used to evaluate the control performance of the autonomous driving algorithm in the corresponding scene. Based on the annotation information, the corresponding decision tree is matched from the pre-built indicator diagnosis logic table, wherein the indicator diagnosis logic table contains decision trees for evaluating different indicators.

5. The performance evaluation method for autonomous driving algorithms according to claim 3, characterized in that, The step of evaluating the corresponding scenario data according to the evaluation content corresponding to each node in the decision tree, and generating a simulation test report of the autonomous driving algorithm based on the evaluation results, includes: The evaluation items of each trunk in the decision tree and the evaluation content of each node under the evaluation item are analyzed. Based on the evaluation items, vehicle control parameters are extracted from the control data; The evaluation content of each node is used as the evaluation labeling rule. The corresponding vehicle control parameters are evaluated and labeled in sequence to obtain the labeling results. The annotation results are uploaded to the algorithm simulation platform for diagnostic evaluation, and a corresponding simulation test report is generated.

6. The performance evaluation method for autonomous driving algorithms according to claim 5, characterized in that, The step of uploading the annotation results to the algorithm simulation platform for diagnostic evaluation and generating a corresponding simulation test report includes: Based on the annotation results, the corresponding algorithm failure problem type is determined; Using the algorithm simulation platform, simulation diagnosis is performed using scenario models corresponding to the algorithm's fault problem types to obtain diagnostic results. The diagnostic results are used to generate a corresponding simulation test report according to the report format corresponding to the fault problem type of the algorithm.

7. The performance evaluation method for an autonomous driving algorithm according to any one of claims 1-6, characterized in that, Before acquiring simulation test data of each autonomous driving algorithm in the autonomous driving system and analyzing the simulation test data to obtain abnormal indicators and corresponding scene data, the process further includes: Extract the autonomous driving algorithms from the autonomous driving system and determine the scenarios corresponding to the algorithms; The test content is determined based on the scenario, and the incident problem is constructed based on the test content; Analyze the scenario-related issues corresponding to the aforementioned accident problem and configure the evaluation content; The evaluation content corresponding to each accident problem is used to construct nodes and leaf nodes using a tree structure to generate a corresponding decision tree, wherein the leaf nodes are connected to the nodes, and all nodes are connected to form a decision tree.

8. A performance evaluation device for an autonomous driving algorithm, characterized in that, The performance evaluation device for the autonomous driving algorithm includes: The decision tree construction module is used to extract the autonomous driving algorithm and the evaluation algorithm of the autonomous driving algorithm in the autonomous driving system, and construct a decision tree based on the evaluation algorithm; The data acquisition module is used to acquire simulation test data of various autonomous driving algorithms in the autonomous driving system, and to analyze the simulation test data to obtain abnormal indicators and corresponding scene data. A matching module is used to determine the category of the abnormal indicator and match the corresponding decision tree from a pre-built indicator diagnosis logic table based on the category. The decision tree contains at least one node, and each node corresponds to a different evaluation content of the indicator. The evaluation module is used to evaluate the corresponding scenario data according to the evaluation content corresponding to each node in the decision tree, and generate a simulation test report of the autonomous driving algorithm based on the evaluation results.

9. A computer device, characterized in that, The computer device includes: a memory and at least one processor, wherein the memory stores a computer program; The at least one processor invokes the computer program in the memory to cause the computer device to perform the various steps of the performance evaluation method for the autonomous driving algorithm as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the various steps of the performance evaluation method for the autonomous driving algorithm as described in any one of claims 1-7.

Images