Data-driven automatic driving system performance test method, medium and equipment

Abstract

The invention discloses a data-driven automatic driving system performance test method, medium and equipment, and relates to the technical field of automatic driving tests.The method comprises the steps that driving scene data mining is conducted on an automatic driving system, and an initial driving scene set is obtained; performing driving record retrieval to obtain a real vehicle test case library of each scene; performing risk-associated reward and punishment on the initial driving scene set; performing simulation test analysis on the automatic driving system to obtain a first driving performance evaluation sequence; performing simulation test analysis on the automatic driving system to obtain a second driving performance evaluation sequence; and carrying out abnormal traceability to generate an automatic driving performance test report. According to the method and the device, the technical problem of poor evaluation accuracy caused by incomplete scene coverage and difficulty in accurately tracing abnormities in performance evaluation of the automatic driving system in the prior art is solved, and the technical effects of performing comprehensive simulation test and abnormity tracing on the automatic driving system based on different scene clusters and improving the performance evaluation accuracy are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving testing, and particularly to a data-driven method, medium, and device for testing the performance of an autonomous driving system. Background Art

[0002] With the rapid development of technology, autonomous driving technology has become a research hotspot and development trend in the global transportation field. In the process of the autonomous driving system from research and development to actual application, its performance testing is crucial. However, there are many problems in the current existing technologies in this regard. On the one hand, traditional performance testing often relies on limited test scenarios and cannot comprehensively cover complex and changeable actual traffic conditions, resulting in limitations in the performance evaluation of the autonomous driving system under different traffic environments, road conditions, and weather conditions. On the other hand, once an abnormality occurs in the autonomous driving system during testing or actual operation, the existing testing methods are difficult to quickly and accurately trace the root cause of the problem.

[0003] The existing technology has technical problems such as incomplete scene coverage and difficulty in accurately tracing the source of anomalies in the performance evaluation of the autonomous driving system, resulting in poor evaluation accuracy. Summary of the Invention

[0004] This application provides a data-driven method, medium, and device for testing the performance of an autonomous driving system, which is used to solve the technical problems in the prior art that the scene coverage in the performance evaluation of the autonomous driving system is incomplete and it is difficult to accurately trace the source of anomalies, resulting in poor evaluation accuracy.

[0005] In view of the above problems, this application provides a data-driven method, medium, and device for testing the performance of an autonomous driving system.

[0006] In the first aspect of the embodiments of this application, a data-driven method for testing the performance of an autonomous driving system is provided. The method includes: Mining driving scenario data of the autonomous driving system according to driving scenario factors to obtain an initial driving scenario set; retrieving driving records of the autonomous driving system according to the initial driving scenario set to obtain a real vehicle test case library for each scenario; performing risk-associated rewards and punishments on the initial driving scenario set according to the real vehicle test case library for each scenario to obtain a first driving scenario cluster and a second driving scenario cluster; based on multiple channels of driving performance evaluation, performing simulation test analysis on the autonomous driving system according to the first driving scenario cluster to obtain a first sequence of driving performance evaluation; based on the multiple channels of driving performance evaluation, performing simulation test analysis on the autonomous driving system according to the second driving scenario cluster to obtain a second sequence of driving performance evaluation; and performing anomaly tracing according to the first sequence of driving performance evaluation and the second sequence of driving performance evaluation to generate an autonomous driving performance test report.

[0007] In the second aspect of the embodiments of the present application, the present application provides a computer-readable storage medium storing a computer program for executing the data-driven performance testing method of the autonomous driving system provided by the present application.

[0008] In the third aspect of the embodiments of the present application, the present application provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; wherein, the processor is used to execute the data-driven performance testing method of the autonomous driving system provided by the present application.

[0009] One or more technical solutions provided in the present application have at least the following technical effects or advantages: Perform driving scenario data mining on the autonomous driving system according to driving scenario factors to obtain an initial driving scenario set; retrieve driving records of the autonomous driving system to obtain a real vehicle test case library for each scenario; perform risk-associated rewards and punishments on the initial driving scenario set to obtain a first driving scenario cluster and a second driving scenario cluster; perform simulation test analysis on the autonomous driving system to obtain a first sequence of driving performance evaluations; perform simulation test analysis on the autonomous driving system according to the second driving scenario cluster to obtain a second sequence of driving performance evaluations; perform anomaly tracing based on the first sequence of driving performance evaluations and the second sequence of driving performance evaluations to generate an autonomous driving performance test report. It achieves the technical effect of realizing comprehensive simulation testing and anomaly tracing of the autonomous driving system based on different scenario clusters, and improving the accuracy of performance evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0011] Figure 1 It is a flowchart of the data-driven performance testing method of the autonomous driving system provided by the embodiments of the present application; Figure 2 It is a schematic structural diagram of an electronic device provided by the present application.

[0012] Description of reference numerals: Processor 21, Memory 22, Input device 23, Output device 24. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0013] The present application provides a method, medium, and device for testing the performance of a data-driven autonomous driving system, which are used to solve the technical problems in the prior art that the scenario coverage in the performance evaluation of the autonomous driving system is not comprehensive and it is difficult to accurately trace the source of anomalies, resulting in poor evaluation accuracy.

[0014] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0015] Embodiment 1, as Figure 1 shown, the present application provides a method for testing the performance of a data-driven autonomous driving system, and the method includes: Step S100: Perform driving scenario data mining on the autonomous driving system according to driving scenario factors to obtain an initial driving scenario set.

[0016] Specifically, the driving scenario factors include traffic environment, road environment, and weather conditions. First, from the huge driving scenario record set of the autonomous driving system, feature recognition is carried out according to these factors, and the traffic environment data set, road environment data set, and weather condition data set are accurately separated. Subsequently, frequency sorting work is carried out for each data set. For example, the frequency coefficient of each data in the traffic environment data set is calculated, and the frequently occurring data is screened out, and it is vectorized to construct a traffic environment vector space. The road environment and weather condition data sets are also operated in this way. Finally, the constructed traffic environment, road environment, and weather condition vector spaces are randomly combined to obtain an initial driving scenario set covering various complex scenarios, laying a foundation for the subsequent performance test of the autonomous driving system.

[0017] Step S200: Retrieve driving records of the autonomous driving system according to the initial driving scenario set to obtain a real vehicle test case library for each scenario.

[0018] Specifically, using each scenario in the initial driving scenario set as a retrieval condition, a comprehensive retrieval is performed on a large number of driving records stored in the autonomous driving system. By comparing and matching the feature information of the initial driving scenarios with data such as scenario descriptions and vehicle operating states in the driving records, real vehicle test records corresponding to each initial driving scenario are screened out. These records contain the operating performance of the autonomous driving system in different scenarios, vehicle control data, and surrounding environment information during actual road driving. The screened records are classified and sorted according to different scenarios, and finally, a real vehicle test case library for each scenario is formed. These case libraries truly reflect the operating conditions of the autonomous driving system in actual scenarios and provide key actual data support for subsequent performance evaluation and analysis.

[0019] Step S300: Perform risk-associated rewards and punishments on the initial driving scenario set according to the real vehicle test case libraries for each scenario, and obtain a first driving scenario cluster and a second driving scenario cluster.

[0020] Specifically, first, with the help of the actual operation data contained in the real vehicle test case libraries for each scenario, a risk assessment is performed on each scenario in the initial driving scenario set, and the corresponding scenario risk coefficient is calculated. Based on the calculated multiple scenario risk coefficients, the initial driving scenario set is divided into two categories: a first driving scenario set with a scenario risk coefficient less than or equal to a preset scenario risk threshold, and a second driving scenario set with a coefficient greater than this threshold. For the first driving scenario set, the scenario coupling coefficient is obtained by evaluating the pairwise similarity between scenarios. According to the comparison result with the scenario coupling threshold, the scenarios are adaptively decoupled and optimized to obtain a first driving scenario cluster; for the second driving scenario set, the mutation quantity analysis function for driving scenarios is used to calculate the mutation quantity of each scenario, and then the scenarios are mutated and expanded based on these quantities to finally obtain a second driving scenario cluster, thereby realizing the differential processing and optimization of scenarios with different risk levels and providing a more targeted scenario set for subsequent accurate simulation test analysis.

[0021] Step S400: Based on the multi-channel driving performance evaluation, perform simulation test analysis on the autonomous driving system according to the first driving scenario cluster, and obtain a first sequence of driving performance evaluations.

[0022] Specifically, starting from the first driving scenario cluster, each driving scenario therein is traversed one by one. After each scenario is extracted, multiple simulation tests are conducted on the autonomous driving system for this scenario to obtain multiple first-scenario test data sets. To ensure the reliability of the data, the central value of these data sets is calculated to obtain the first-scenario credible test data set. The multi-channel driving performance evaluation consists of a basic driving performance evaluation channel, a dynamic interaction driving performance evaluation channel, and a comprehensive driving performance calculation channel. The first-scenario credible test data set is sequentially input into the basic driving performance evaluation channel and the dynamic interaction driving performance evaluation channel to obtain the first basic driving performance coefficient and the first dynamic interaction driving performance coefficient respectively. Then, these two coefficients are input into the comprehensive driving performance calculation channel to further obtain the first comprehensive driving performance coefficient. These coefficients together constitute the driving performance evaluation result of the first scenario and are added to the first sequence of driving performance evaluation in order for a comprehensive and systematic analysis of the performance of the autonomous driving system in different scenarios in the future.

[0023] Step S500: Based on the multi-channel driving performance evaluation, perform simulation test analysis on the autonomous driving system according to the second driving scenario cluster to obtain the second sequence of driving performance evaluation.

[0024] Specifically, based on the second driving scenario cluster, each driving scenario therein is traversed in sequence. For each extracted driving scenario, multiple simulation tests are carried out on the autonomous driving system to collect multiple second-scenario test data sets. To obtain more reliable data, these data sets are processed by calculating the central value to obtain the second-scenario credible test data set. The multi-channel driving performance evaluation includes a basic driving performance evaluation channel, a dynamic interaction driving performance evaluation channel, and a comprehensive driving performance calculation channel. The second-scenario credible test data set is input into the basic driving performance evaluation channel to obtain the second basic driving performance coefficient; input into the dynamic interaction driving performance evaluation channel to obtain the second dynamic interaction driving performance coefficient. Then, these two coefficients are input into the comprehensive driving performance calculation channel to obtain the second comprehensive driving performance coefficient. These coefficients together constitute the driving performance evaluation result of the second scenario and are added one by one to the second sequence of driving performance evaluation in order to provide key data support for the subsequent comprehensive evaluation of the performance of the autonomous driving system.

[0025] Step S600: According to the first sequence of driving performance evaluation and the second sequence of driving performance evaluation, perform abnormal traceability to generate an autonomous driving performance test report.

[0026] Specifically, perform element-by-element comparison on the data of the first sequence and the second sequence of driving performance evaluation, and highlight the differences in aspects such as driving performance coefficients and test results. Using the fault tree analysis method, starting from the system architecture level, take driving scenarios, sensors, algorithm modules, etc. as nodes, and construct a fault tree according to the logical relationship. Combine the abnormal data in the two sequences, trace back step by step from the underlying events to find out the reasons for the anomalies. For example, whether the sensor data transmission is abnormal in a specific scenario, which in turn affects the algorithm decision-making and ultimately leads to abnormal driving performance. At the same time, use the causal analysis method to analyze the causal relationship between the abnormal driving performance and each link of the system, and sort out the path of the anomaly generation. After completing the anomaly tracing, according to the industry-standard test report template, organize information such as the reasons for the anomalies, the involved scenarios, and the impact on the performance of the autonomous driving system, and generate a detailed autonomous driving performance test report. The report will also include corresponding improvement suggestions and optimization directions to provide strong support for improving the performance of the autonomous driving system.

[0027] In a possible implementation manner, step S300 further includes: Step S310: Perform risk assessment on the initial driving scenario set according to the real vehicle test case library for each scenario to obtain multiple scenario risk coefficients.

[0028] Step S320: Classify the initial driving scenario set according to the multiple scenario risk coefficients to obtain a first driving scenario set with a scenario risk coefficient less than or equal to the scenario risk threshold, and a second driving scenario set with a scenario risk coefficient greater than the scenario risk threshold.

[0029] Step S330: Decouple and optimize the first driving scenario set to obtain the first driving scenario cluster.

[0030] Step S340: Mutate and expand the second driving scenario set to obtain the second driving scenario cluster.

[0031] Specifically, the Analytic Hierarchy Process (AHP) is used for risk assessment to obtain the risk coefficients of multiple scenarios. First, the goal of risk assessment is determined to be evaluating the risk levels of each scenario in the initial driving scenario set, and a hierarchical structure model is established that includes a criterion layer such as vehicle behavior stability, traffic conflict possibility, environmental complexity, etc., and the corresponding index layer. For the data in the in-vehicle test case library of each scenario, data is extracted as indicators from aspects such as the vehicle driving trajectory, the distance change with other road participants, the road type and the number of traffic signs in the scenario. By means of expert scoring or data statistical analysis, etc., a judgment matrix is constructed to determine the relative weights of each criterion and index. For example, for vehicle behavior stability and traffic conflict possibility, if it is considered that the former has a greater impact on the scenario risk, corresponding weight values can be assigned in the judgment matrix. Finally, by synthesizing the weights and index values of each index, the comprehensive risk score of each scenario is calculated, and these scores are the risk coefficients of multiple scenarios, so as to quantify the risk levels of each scenario.

[0032] The obtained risk coefficients of multiple scenarios are used as the classification basis. A scenario risk threshold is preset, which is comprehensively determined according to the safety standards of the autonomous driving system, past experience, and industry norms, etc. For each scenario in the initial driving scenario set, its corresponding scenario risk coefficient is compared with this threshold. Those scenarios with scenario risk coefficients less than or equal to the threshold are selected and grouped together to form the first driving scenario set. These scenarios are considered to have relatively low risks, and the possibility of dangerous situations occurring in these scenarios for the autonomous driving system is relatively small; while the scenarios with scenario risk coefficients greater than the threshold are classified into the second driving scenario set. These scenarios have higher risks and need to be focused on and processed subsequently, so as to realize the risk stratification and classification of the initial driving scenario set and lay a foundation for more targeted test analysis work in the future.

[0033] The Euclidean distance between each pair of scenarios in the first driving scenario set is calculated to measure the pairwise similarity, so as to obtain multiple scenario coupling coefficients. A reasonable scenario coupling threshold is set. If the scenario coupling coefficient is greater than or equal to this threshold, it indicates that the coupling degree of the two scenarios is relatively high. For scenarios with a high coupling degree, the Principal Component Analysis (PCA) method is used for decoupling. PCA transforms the original scenario data into a new orthogonal coordinate system through linear transformation, so that the data variance in the new coordinate system is as large as possible and mutually independent. In this way, the main features can be extracted, redundant information can be removed, and the adaptive decoupling of the scenario can be realized. Then, according to the decoupled scenario data, clustering and integration are carried out according to factors such as scenario type and environmental conditions to form a scenario set with clear discrimination and independence, that is, the first driving scenario cluster, so as to more accurately evaluate the performance of the autonomous driving system in low-risk scenarios in the future.

[0034] According to the driving scenario mutation quantity analysis function, combined with the scenario risk coefficients corresponding to each driving scenario in the second driving scenario set, the mutation quantities of each driving scenario are calculated. This function comprehensively considers factors such as the difference between the scenario risk coefficient and the scenario risk threshold, and the upper and lower limits of the driving scenario mutation quantity. After obtaining the mutation quantities of each driving scenario, based on these quantities, a mutation expansion operation is performed on the second driving scenario set. For example, by changing certain parameters in the scenario (such as traffic flow, road slope, weather condition details, etc.), multiple new scenarios that are similar to the original scenario but have differences are generated, thereby expanding the second driving scenario set and finally forming the second driving scenario cluster, enriching the diversity of high-risk scenarios, so as to more comprehensively test the performance of the autonomous driving system in complex high-risk scenarios.

[0035] In a possible implementation manner, step S330 further includes: Step S331: Perform pairwise similarity evaluation on the first driving scenario set to obtain multiple scenario coupling coefficients.

[0036] Step S332: Determine whether the multiple scenario coupling coefficients are greater than or equal to the scenario coupling threshold to obtain multiple scenario coupling judgment results.

[0037] Step S333: Perform adaptive decoupling on the first driving scenario set according to the multiple scenario coupling judgment results to obtain the first driving scenario cluster.

[0038] Specifically, a series of operations are carried out for the first driving scenario set. First, each scenario in the first driving scenario set is subjected to feature extraction in multiple dimensions such as traffic environment, road environment, and weather condition, and each scenario is transformed into a multi-dimensional feature vector. Using appropriate calculation methods such as Euclidean distance, these feature vectors are calculated pairwise, thereby obtaining multiple scenario coupling coefficients, which intuitively reflect the similarity degree between each scenario.

[0039] Compare each scenario coupling coefficient with a preset scenario coupling threshold, which is determined based on a deep understanding of the performance testing of the autonomous driving system, the analysis of a large amount of past test data, and industry standard specifications. During the comparison process, a judgment operation is performed for each scenario coupling coefficient: if the coefficient is greater than or equal to the scenario coupling threshold, it means that the corresponding two scenarios have a high degree of coupling and can be regarded as having a strong correlation in subsequent test analysis, and it is recorded as a positive scenario coupling judgment result; conversely, if the coefficient is less than the scenario coupling threshold, it indicates that the coupling degree of these two scenarios is low and they need to be considered separately in the test, and it is recorded as a negative scenario coupling judgment result. By judging and recording all the scenario coupling coefficients one by one, multiple scenario coupling judgment results are finally generated, which provide an important basis for the subsequent adaptive decoupling operation of the first driving scenario set and determine whether the scenarios need to be merged or tested separately.

[0040] Based on the obtained multiple scenario coupling judgment results, an adaptive decoupling operation is performed on the first driving scenario set. When the scenario coupling coefficient is greater than or equal to the scenario coupling threshold, it means that the similarity between these scenarios is high and there are a large number of repeated features and situations. To improve the test efficiency and avoid wasting resources caused by repeated testing, these scenarios need to be merged. During the merging process, the common features of these similar scenarios are extracted, and the redundant parts are removed to form a comprehensive scenario. For example, if multiple scenarios are mostly similar in terms of traffic flow, road type, weather conditions, etc., but only differ in some details, these similar parts are integrated, and the representative differences are retained as the new merged scenario. For those scenarios with a coupling degree less than the threshold, they each have unique features and irreplaceable value in testing the performance of the autonomous driving system. To comprehensively evaluate the performance of the autonomous driving system in different scenarios and ensure the integrity and accuracy of the test, these scenarios need to be tested separately. Through this differentiated processing method, resources can be utilized efficiently, and the comprehensiveness of the test scenarios can be ensured. Finally, through such an adaptive decoupling operation, the first driving scenario cluster is obtained, which can provide strong support for more accurate and efficient testing of the performance of the autonomous driving system in the future.

[0041] In a possible implementation manner, step S340 further includes: According to the driving scenario mutation quantity analysis function, calculate the respective driving scenario mutation quantities corresponding to the second driving scenario set, where the driving scenario mutation quantity analysis function is: ; where R i represents the scenario risk coefficient corresponding to the i-th driving scenario in the second driving scenario set, i is a positive integer, S iCharacterize the number of driving scenario variations corresponding to the i-th driving scenario. Floor means rounding down, R 0 Characterize the scenario risk threshold, S max Characterize the upper limit of the number of driving scenario variations, S min Characterize the lower limit of the number of driving scenario variations. Expand the second driving scenario set according to the numbers of driving scenario variations of each to obtain the second driving scenario cluster.

[0042] Specifically, first, according to a specific parsing function for the number of driving scenario variations, that is , calculate the number of variations of each driving scenario in the second driving scenario set. R i represents the scenario risk coefficient corresponding to the i-th driving scenario in the second driving scenario set, and its numerical value reflects the potential risk degree of this scenario. i is a positive integer starting from 1 to ensure correspondence to each scenario. R 0 is a preset scenario risk threshold, which is the benchmark for judging the level of scenario risk and is determined comprehensively by the safety standards of the autonomous driving system, past test experience, etc. S max and S min are respectively the upper and lower limits of the number of driving scenario variations, which limit the variation range of each scenario. For example, it is stipulated that the minimum number of variations is 2 and the maximum is 10. This is to avoid excessive variations while ensuring the comprehensiveness of testing. The Floor function is used to round down to ensure that the calculated number of variations is an integer.

[0043] Calculate the number of variations of each scenario through this function After that, the second driving scenario set is expanded by variation. For example, if the calculated number of variations of a certain scenario is 3, factors such as traffic flow (such as increasing or decreasing the number of vehicles, changing the vehicle speed distribution), road environment (such as adjusting the road slope, curvature), or weather conditions (such as changing the light intensity, precipitation probability) of this scenario are changed, and a new scenario is generated each time. The newly generated scenarios are merged with the scenarios in the original second driving scenario set to finally form the second driving scenario cluster. The purpose of this is to make the test scenarios more diverse, so as to more comprehensively and deeply test the performance of the autonomous driving system in high-risk scenarios and provide more sufficient data support for evaluating the reliability of the autonomous driving system in complex and dangerous environments.

[0044] In a possible implementation manner, step S400 further includes: Step S410: Traverse the first driving scenario cluster and extract the first driving scenario.

[0045] Step S420: Conduct multiple simulation tests on the autonomous driving system according to the first driving scenario to obtain multiple first scenario test data sets.

[0046] Step S430: Calculate the central value based on the multiple first-scenario test data sets to obtain a first-scenario credible test data set.

[0047] Step S440: Input the first-scenario credible test data set into the multi-channel driving performance evaluation to obtain a first-scenario driving performance evaluation result.

[0048] Step S450: Add the first-scenario driving performance evaluation result to the first sequence of driving performance evaluations.

[0049] Specifically, first, locate the generated first driving scenario cluster, which is a set of driving scenarios obtained after previous risk-associated rewards and punishments and decoupling and optimization of the low-risk scenario set. Through an iterative algorithm, in a specific order (such as sequential traversal, random traversal), visit each scenario element in the first driving scenario cluster in turn. During the traversal, each time a scenario is visited, it is extracted as the current first driving scenario to be processed. The extraction operation can be completed through a data reading interface or a relevant data processing module, separating the data of this scenario from the storage structure of the first driving scenario cluster to prepare for subsequent simulation tests of the autonomous driving system for this scenario.

[0050] After the extraction of the first driving scenario is completed, it enters the multiple simulation test session of the autonomous driving system. Using the autonomous driving simulation test platform, based on the detailed features such as the traffic environment, road environment, and weather conditions included in the first driving scenario, construct a virtual test environment that highly restores the real scenario. In this environment, simulate a variety of different driving conditions, such as changing the driving speed of the vehicle, adjusting the density of traffic flow, setting different road gradients and curve curvatures, etc. For each simulated condition, the autonomous driving system will conduct an operation test, and during each test process, a large amount of data will be recorded, covering aspects such as the driving trajectory of the vehicle, the feedback information of the sensors, and the instruction output of the decision-making system. After multiple such tests, the data recorded in each test is sorted and saved separately, and finally, multiple first-scenario test data sets are obtained. These data sets provide a rich data basis for subsequent analysis of the performance of the autonomous driving system in this scenario.

[0051] Perform central value calculation on the obtained multiple first-scenario test data sets to obtain a first-scenario credible test data set. The multiple first-scenario test data sets contain data generated from multiple simulation tests of the autonomous driving system in the same driving scenario. Due to various random factors during the test process, there are fluctuations and errors in these data. In order to obtain data that can more accurately reflect the true performance of the autonomous driving system in this scenario, central value calculation is required. During the calculation, the mean value is used. The same type of data (such as vehicle speed, steering angle, etc.) in the multiple first-scenario test data sets are added separately and then divided by the number of tests to obtain the average value of these data. Through such central value calculation, the influence of abnormal data and random errors can be eliminated to a certain extent, making the obtained data more representative and reliable. Finally, a first-scenario credible test data set is obtained, providing a reliable basis for accurately evaluating the performance of the autonomous driving system in this scenario in the future.

[0052] After obtaining the first-scenario credible test data set, it is input into the multi-channel driving performance evaluation to obtain the first-scenario driving performance evaluation result. The multi-channel driving performance evaluation consists of a basic driving performance evaluation channel, a dynamic interaction driving performance evaluation channel, and a comprehensive driving performance calculation channel. First, the first-scenario credible test data set is input into the basic driving performance evaluation channel. This channel analyzes and calculates the key indicators related to basic driving in the data set, such as the acceleration and deceleration performance of the vehicle, driving stability, etc., and then obtains the first basic driving performance coefficient. Next, the same data set is input into the dynamic interaction driving performance evaluation channel. This channel focuses on the performance of the autonomous driving system in dynamic interaction scenarios, such as the interaction with other vehicles and pedestrians, and thus obtains the first dynamic interaction driving performance coefficient. Subsequently, the first basic driving performance coefficient and the first dynamic interaction driving performance coefficient are input into the comprehensive driving performance calculation channel together. This channel comprehensively considers the above two coefficients and other relevant factors, and uses a weighted algorithm for calculation, and finally obtains the first comprehensive driving performance coefficient. These three coefficients together constitute the first-scenario driving performance evaluation result, comprehensively reflecting the performance of the autonomous driving system in this scenario.

[0053] Add the obtained first-scenario driving performance evaluation result to the first sequence of driving performance evaluation. This sequence is used to record the performance evaluation of the autonomous driving system in each scenario in the first driving scenario cluster, providing key data support for subsequent comparative analysis, anomaly tracing, and generating a complete autonomous driving performance test report.

[0054] In a possible implementation manner, step S440 further includes: Step S441: The multi-channel driving performance evaluation includes a basic driving performance evaluation channel, a dynamic interaction driving performance evaluation channel, and a comprehensive driving performance calculation channel.

[0055] Step S442: Input the first-scenario trusted test data set into the basic driving performance evaluation channel to obtain the first basic driving performance coefficient.

[0056] Step S443: Input the first-scenario trusted test data set into the dynamic interaction driving performance evaluation channel to obtain the first dynamic interaction driving performance coefficient.

[0057] Step S444: Input the first basic driving performance coefficient and the first dynamic interaction driving performance coefficient into the comprehensive driving performance calculation channel to obtain the first comprehensive driving performance coefficient.

[0058] Step S445: Output the first basic driving performance coefficient, the first dynamic interaction driving performance coefficient, and the first comprehensive driving performance coefficient as the first-scenario driving performance evaluation result.

[0059] Specifically, the multi-channel driving performance evaluation includes a basic driving performance evaluation channel, a dynamic interaction driving performance evaluation channel, and a comprehensive driving performance calculation channel. These channels cooperate with each other to evaluate the performance of the autonomous driving system in a specific scenario from different dimensions.

[0060] Input the first-scenario trusted test data set into the basic driving performance evaluation channel and use the decision tree algorithm model to obtain the first basic driving performance coefficient. The decision tree algorithm makes step-by-step partitions based on the characteristics of the data set and constructs a tree structure to achieve classification or regression tasks. For the basic driving performance evaluation, first extract the characteristics related to the basic driving performance in the data set, such as the vehicle speed change rate, steering angle deviation, braking response time, etc. The algorithm starts from the root node and partitions the data according to different values of a certain characteristic. For example, taking the speed change rate as the partitioning basis, if the speed change rate exceeds a certain threshold, the data is partitioned into one branch; if it does not exceed, it is partitioned into another branch. Continuously repeat this process until the data in each branch has high homogeneity, forming a decision tree. After constructing the decision tree, input the first-scenario trusted test data set. The decision tree performs classification or regression calculations along the branches of the tree according to the characteristics of the data, and finally outputs a value representing the basic driving performance. This value is the first basic driving performance coefficient, which comprehensively reflects the performance of the autonomous driving system in basic driving.

[0061] Input the first-scenario trusted test data set into the dynamic interactive driving performance evaluation channel, and use the convolutional neural network (CNN) algorithm model to obtain the first dynamic interactive driving performance coefficient. The CNN algorithm model performs excellently in processing images and data with spatial structures, and is also applicable to the dynamic interactive data between vehicles and the surrounding environment in the autonomous driving scenario. First, extract the feature data related to dynamic interaction from the first-scenario trusted test data set, such as the positions, speeds, driving directions of other vehicles around the vehicle, and the positions and moving directions of pedestrians, etc., and preprocess these data into a format suitable for the input of the CNN, for example, convert the position information into a two-dimensional matrix form. The CNN model contains multiple convolutional layers, pooling layers, and fully connected layers. The convolutional layer slides the convolutional kernel over the data to extract features from local regions, capturing the spatial features and patterns in the data, such as the change patterns of the relative positions between vehicles. The pooling layer is used to reduce the dimension of the data, reducing the computational amount while retaining the key features. After being processed by multiple convolutional layers and pooling layers, the features of the data are gradually abstracted and extracted. Finally, the features extracted previously are integrated through the fully connected layer and mapped to an output node, and the value of this output node is the first dynamic interactive driving performance coefficient. This coefficient comprehensively reflects the performance of the autonomous driving system in the dynamic interaction scenario. The higher the coefficient, the better the system performs in dynamic interaction with other vehicles, pedestrians, etc., such as being able to more accurately predict the behaviors of other road users and make reasonable decisions more timely.

[0062] Input the first basic driving performance coefficient and the first dynamic interactive driving performance coefficient into the comprehensive driving performance calculation channel to obtain the first comprehensive driving performance coefficient that more comprehensively reflects the performance of the autonomous driving system. First, clarify the weights of the two coefficients. Assume that the weight of the basic driving performance coefficient is W 1 , and the weight of the dynamic interactive driving performance coefficient is W 2 , and W 1 +W 2 =1. The setting of the weights needs to comprehensively consider the emphasis of the autonomous driving system on basic driving performance and dynamic interactive performance in different scenarios. For example, in scenarios with heavy traffic flow and complex road conditions, dynamic interactive performance is more important, and the value of W 2 can be increased accordingly; while in scenarios with relatively simple road conditions, the weight W 1 of the basic driving performance can be appropriately increased. Then perform weighted calculation using the formula: the first comprehensive driving performance coefficient = W 1 ×the first basic driving performance coefficient + W 2 ×the first dynamic interactive driving performance coefficient. When calculating, multiply the previously obtained first basic driving performance coefficient by W 1 , and multiply the first dynamic interactive driving performance coefficient by W 2Multiply them, and then add the two products together. The final result is the first comprehensive driving performance coefficient. This coefficient integrates information from both basic driving performance and dynamic interaction driving performance, enabling a more comprehensive and accurate assessment of the overall performance of the autonomous driving system in the first driving scenario, and providing key data support for subsequent performance evaluation and analysis.

[0063] After a series of complex operations mentioned above, the first basic driving performance coefficient, the first dynamic interaction driving performance coefficient, and the first comprehensive driving performance coefficient are obtained. These three coefficients respectively reflect the performance of the autonomous driving system in the first driving scenario from different dimensions. The first basic driving performance coefficient reflects the basic performance of the system in conventional driving operations such as acceleration, deceleration, and steering. The first dynamic interaction driving performance coefficient focuses on the system's performance when interacting with other road users (such as vehicles and pedestrians). The first comprehensive driving performance coefficient combines the former two and comprehensively reflects the overall performance of the system in this scenario. Integrate and output these three coefficients to form the driving performance evaluation result for the first scenario. This result is a quantitative representation of the performance of the autonomous driving system in the first driving scenario, providing important data basis for subsequent evaluation of the system's comprehensive performance in various scenarios, comparison of performance differences between different scenarios, and anomaly tracing, etc., helping to comprehensively understand the advantages and disadvantages of the autonomous driving system, and thus promoting the optimization and improvement of its performance.

[0064] In a possible implementation manner, step S100 further includes: Step S110: The driving scenario factors include traffic environment, road environment, and weather conditions.

[0065] Step S120: According to the driving scenario factors, perform feature recognition on the driving scenario record set of the autonomous driving system to obtain a traffic environment data set, a road environment data set, and a weather condition data set.

[0066] Step S130: Sort out the frequencies of the traffic environment data set, the road environment data set, and the weather condition data set, and construct a traffic environment vector space, a road environment vector space, and a weather condition vector space.

[0067] Step S140: Randomly combine according to the traffic environment vector space, the road environment vector space, and the weather condition vector space to generate the initial driving scenario set.

[0068] Specifically, the driving scenario factors cover three key dimensions: traffic environment, road environment, and weather conditions. As an important part of the driving scenario, the traffic environment encompasses many dynamic and static elements. The dynamic elements include vehicle flow, vehicle speed distribution, the proportion of different types of vehicles (such as cars, trucks, buses) and their driving trajectories and following distances, etc. The static elements include traffic signal states, the setting and types of traffic signs, etc. These factors together shape the traffic operation conditions faced by vehicles when driving on the road. The road environment mainly involves the physical properties of the road itself, such as the road category, which can be divided into highways, urban arterial roads, secondary arterial roads, rural roads, etc. The number of lanes and lane widths of different category roads vary; the road gradient, including uphill, downhill, and flat roads, and its gradient size affects vehicle power requirements and driving stability; there is also the curve curvature, which determines the handling difficulty and speed limit when the vehicle turns. Weather conditions also have a significant impact on driving, covering weather with good lighting and no precipitation such as sunny days, cloudy days, and overcast days, as well as bad weather such as rainy days, snowy days, and foggy days. The differences in visibility, road surface wetness, and lighting intensity under different weather conditions will change the perception and response strategies of drivers or autonomous driving systems to the road and the surrounding environment.

[0069] Adopt methods based on machine learning and data mining. First, preprocess the raw data in the record set to remove noise and missing values to improve data quality. For traffic environment feature recognition, use object detection algorithms (such as the YOLO series) to process the image or video data in the record set, identify objects such as vehicles, pedestrians, traffic signs, and traffic lights, count information such as the number, type, and movement direction of vehicles, and at the same time combine sensor data (such as radar, lidar) to calculate the speed and spacing of vehicles, and then construct a traffic environment data set. In terms of road environment feature recognition, use semantic segmentation algorithms (such as U-Net) to analyze the image data, distinguish different types of roads (such as highways, urban streets), lane lines, and road edges, and extract information such as the gradient, curvature, and elevation of the road from map data and sensor data to form a road environment data set. For weather condition recognition, use image classification algorithms (such as ResNet) to classify the images in the record, judge the weather type (such as sunny days, rainy days, foggy days), and combine meteorological sensor data (such as humidity, lighting intensity) for verification and supplementation to finally obtain a weather condition data set.

[0070] To construct a vector space, it is necessary to conduct a frequency analysis on the datasets of traffic environment, road environment, and weather conditions. For the traffic environment dataset, relevant features are first determined, such as vehicle density, traffic flow speed, traffic rule compliance, etc. With the help of data mining techniques, the frequency of each feature value appearing in the dataset is statistically calculated. For example, the number of occurrences of vehicle density at different time periods and different road sections. Based on this frequency information, each feature value is mapped to a dimension in the vector space, and the frequency of the feature value is used as the coordinate value on this dimension, thereby constructing the traffic environment vector space. For the road environment dataset, features such as road type, slope, and bend curvature are identified, and the occurrence frequency of each feature value is statistically calculated. For example, the number of occurrences of different road types (highways, urban streets, etc.). The feature values are corresponding to the dimensions in the vector space, and with the frequency as the coordinates, the road environment vector space is constructed. For the weather condition dataset, features such as weather type (sunny, rainy, foggy, etc.), visibility, and precipitation intensity are determined, and the occurrence frequency of each feature value is statistically calculated. Using these feature values as dimensions and frequency as coordinates, the weather condition vector space is constructed.

[0071] Using the random number generation algorithm, a vector is randomly selected from the traffic environment vector space, road environment vector space, and weather condition vector space respectively. The traffic environment vector represents a specific traffic condition, such as a specific vehicle density, traffic flow speed, and traffic rule compliance pattern; the road environment vector corresponds to specific road attributes, such as road type, slope, and bend curvature; the weather condition vector represents specific weather conditions, such as a specific weather type, visibility, and precipitation intensity. Combining these three randomly selected vectors together forms a complete initial driving scenario, which comprehensively combines the characteristics of traffic, road, and weather. By repeating such a random selection and combination process multiple times, a large number of different initial driving scenarios can be generated. These scenarios form the initial driving scenario set, providing a rich variety of simulation scenarios for the subsequent comprehensive testing and evaluation of the autonomous driving system.

[0072] In a possible implementation manner, step S130 further includes: Step S131: Calculate the frequency of each traffic environment data in the traffic environment dataset to obtain the traffic environment frequency coefficients.

[0073] Step S132: Screen the traffic environment dataset based on the traffic environment frequency coefficients to obtain multiple frequent traffic environment data that meet the predetermined traffic environment frequency coefficients.

[0074] Step S133: Perform vectorization processing based on the multiple frequent traffic environment data to generate the traffic environment vector space.

[0075] Specifically, the traffic environment dataset contains rich and diverse traffic environment data, such as vehicle flow at different time periods, vehicle density on different road sections, status changes of traffic lights, occurrence of traffic violation events, etc. To obtain the frequency coefficients of each traffic environment, it is first necessary to clarify the statistical scope and time span to ensure the consistency and integrity of the data. Then, for each specific traffic environment data in the dataset, count the number of times it appears in the entire dataset. Next, divide the number of occurrences of each data by the total number of the dataset, and the obtained ratio is the frequency coefficient of the traffic environment data. For example, if in 1000 traffic environment records, the high vehicle density at a specific time period on a certain road section appears 200 times, then the frequency coefficient of this traffic environment data is 200÷1000 = 0.2. In this way, calculate the frequency of all traffic environment data in the dataset, and finally obtain the traffic environment frequency coefficients corresponding to each data. These coefficients will provide key basis for subsequent screening and construction of the vector space.

[0076] Screen the traffic environment dataset according to the calculated frequency coefficients of each traffic environment. Preset a predetermined traffic environment frequency coefficient as the threshold, and exclude the traffic environment data with a frequency coefficient lower than this threshold, only retain the traffic environment data with a frequency coefficient reaching or exceeding this threshold. These retained data are multiple frequent traffic environment data that meet the predetermined conditions. They represent the traffic environment characteristics that appear relatively frequently in the actual traffic scenario and are of great significance for constructing a representative traffic environment vector space.

[0077] Perform vectorization processing on the multiple selected frequent traffic environment data. Assign a vector dimension to each frequent traffic environment data, and use its frequency coefficient in the dataset as the coordinate value on this dimension. In this way, each frequent traffic environment data can be represented by a vector, and all these vectors together constitute the traffic environment vector space. This vector space can intuitively reflect various frequently occurring traffic environment characteristics and their relative importance, providing an effective data basis for subsequent simulation and performance evaluation of the autonomous driving scenario based on the traffic environment.

[0078] Embodiment 2. Based on the same inventive concept as the data-driven autonomous driving system performance testing method in the foregoing embodiment, this embodiment provides a computer-readable storage medium, which can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the data-driven autonomous driving system performance testing method in this application embodiment. The processor executes various functional applications and data processing of the computer device by running the software programs, instructions, and modules stored in the memory, that is, implements the above-mentioned data-driven autonomous driving system performance testing method.

[0079] Embodiment 3. Based on the same inventive concept as the data-driven performance testing method for autonomous driving systems in the foregoing embodiments, this embodiment provides an electronic device. Figure 2 FIG. Figure 2 is a schematic structural diagram of the electronic device provided in Embodiment 3 of the present invention, showing a block diagram of an exemplary electronic device suitable for implementing the embodiments of the present invention. Figure 2 The displayed electronic device is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention. As Figure 2 shown, the electronic device includes a processor 21, a memory 22, an input device 23, and an output device 24; the number of processors 21 in the electronic device can be one or more. Figure 2 Taking one processor 21 as an example, the processor 21, the memory 22, the input device 23, and the output device 24 in the electronic device can be connected by a bus or other means. Figure 2 Taking connection by a bus as an example.

[0080] It should be noted that the above sequence of the embodiments of the present application is only for description and does not represent the superiority or inferiority of the embodiments. And the above specific embodiments of this specification have been described. Additionally, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0081] The above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

[0082] This specification and the drawings are only exemplary descriptions of the present application and are considered to have covered any and all modifications, variations, combinations, or equivalents within the scope of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalent technologies, the present application is intended to include these changes and modifications.

Claims

1. A data-driven automatic driving system performance testing method, characterized in that: include: Perform driving scenario data mining on the autonomous driving system according to the driving scenario factors to obtain an initial driving scenario set; Retrieving driving records of the automatic driving system according to the initial driving scenario set to obtain a real vehicle test case library for each scenario; Perform risk-related rewards and penalties on the initial driving scenario set according to the real vehicle test case library for each scenario to obtain a first driving scenario cluster and a second driving scenario cluster; Based on the driving performance evaluation multi-channel, performing simulation test analysis on the automatic driving system according to the first driving scenario cluster to obtain a first sequence of driving performance evaluation; Based on the driving performance evaluation multi-channel, performing simulation test analysis on the automatic driving system according to the second driving scenario cluster to obtain a second driving performance evaluation sequence; According to the first driving performance evaluation sequence and the second driving performance evaluation sequence, abnormality tracing is performed to generate an automatic driving performance test report.

2. The data-driven automatic driving system performance testing method according to claim 1, characterized in that: According to the real vehicle test case library of each scenario, the initial driving scenario set is rewarded or punished based on risk association, so as to obtain a first driving scenario cluster and a second driving scenario cluster, including: Performing risk assessment on the initial driving scenario set according to the real vehicle test case library of each scenario to obtain multiple scenario risk coefficients; Classifying the initial driving scenario set according to the multiple scenario risk coefficients to obtain a first driving scenario set that is less than or equal to a scenario risk threshold, and a second driving scenario set that is greater than the scenario risk threshold; Performing decoupling optimization on the first driving scenario set to obtain the first driving scenario cluster; The second driving scene set is mutated and expanded to obtain the second driving scene cluster.

3. The data-driven automatic driving system performance testing method according to claim 2, characterized in that: Performing decoupling optimization on the first driving scenario set to obtain the first driving scenario cluster includes: Performing pairwise similarity evaluation on the first driving scene sets to obtain multiple scene coupling coefficients; Determine whether the multiple scene coupling coefficients are greater than or equal to a scene coupling threshold, and obtain multiple scene coupling determination results; The first driving scene set is adaptively decoupled according to the multiple scene coupling judgment results to obtain the first driving scene cluster.

4. The data-driven automatic driving system performance testing method according to claim 2, characterized in that: Mutating and expanding the second driving scene set to obtain the second driving scene cluster includes: The driving scenario variation quantity corresponding to the second driving scenario set is calculated according to the driving scenario variation quantity analytical function, wherein the driving scenario variation quantity analytical function is: ； Among them, R i represents the scenario risk coefficient corresponding to the i-th driving scenario in the second driving scenario set, i is a positive integer, S i represents the number of driving scenario variations corresponding to the i-th driving scenario, Floor refers to rounding down, R0 represents the scenario risk threshold, S max Characterizes the upper limit of the number of driving scene variations, S min Characterize the lower limit of the number of driving scene variations; The second driving scene set is mutated and expanded according to the number of variations of each driving scene to obtain the second driving scene cluster.

5. The data-driven automatic driving system performance testing method according to claim 1, characterized in that: Based on the driving performance evaluation multi-channel, the automatic driving system is simulated tested and analyzed according to the first driving scenario cluster to obtain a first sequence of driving performance evaluation, including: Traversing the first driving scene cluster to extract a first driving scene; Performing multiple simulation tests on the autonomous driving system according to the first driving scenario to obtain multiple first scenario test data sets; Performing concentrated value calculation according to the multiple first-scenario test data sets to obtain a first-scenario credible test data set; Inputting the first scenario credible test data set into the driving performance evaluation multi-channel to obtain a first scenario driving performance evaluation result; The first scenario driving performance evaluation result is added to the driving performance evaluation first sequence.

6. The data-driven automatic driving system performance testing method according to claim 5, characterized in that: Inputting the first scenario credible test data set into the driving performance evaluation multi-channel to obtain the first scenario driving performance evaluation result, including: The driving performance evaluation multi-channels include a basic driving performance evaluation channel, a dynamic interactive driving performance evaluation channel and a comprehensive driving performance calculation channel; Inputting the first scenario credible test data set into the basic driving performance evaluation channel to obtain a first basic driving performance coefficient; Inputting the first scenario credible test data set into the dynamic interactive driving performance evaluation channel to obtain a first dynamic interactive driving performance coefficient; Inputting the first basic driving performance coefficient and the first dynamic interactive driving performance coefficient into the comprehensive driving performance calculation channel to obtain a first comprehensive driving performance coefficient; The first basic driving performance coefficient, the first dynamic interactive driving performance coefficient and the first comprehensive driving performance coefficient are output as the first scenario driving performance evaluation result.

7. The data-driven automatic driving system performance testing method according to claim 1, characterized in that: According to the driving scenario factors, the driving scenario data of the autonomous driving system is mined to obtain the initial driving scenario set, including: The driving scenario factors include traffic environment, road environment and weather conditions; According to the driving scene factors, feature recognition is performed on the driving scene record set of the automatic driving system to obtain a traffic environment data set, a road environment data set, and a weather condition data set; Performing frequency sorting on the traffic environment data set, the road environment data set, and the weather condition data set to construct a traffic environment vector space, a road environment vector space, and a weather condition vector space; The initial driving scenario set is generated by randomly combining the traffic environment vector space, the road environment vector space and the weather condition vector space.

8. The data-driven automatic driving system performance testing method according to claim 7, characterized in that: The traffic environment data set, the road environment data set and the weather condition data set are frequently combed to construct a traffic environment vector space, a road environment vector space and a weather condition vector space, including: Calculating the frequency of each traffic environment data in the traffic environment data set to obtain a frequency coefficient of each traffic environment; Filter the traffic environment data set based on the traffic environment frequency coefficients to obtain a plurality of frequent traffic environment data satisfying a predetermined traffic environment frequency coefficient; Vectorization processing is performed according to the plurality of frequent traffic environment data to generate the traffic environment vector space.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data-driven autonomous driving system performance testing method as described in any one of claims 1 to 8 is implemented.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, used to implement the data-driven autonomous driving system performance testing method as described in any one of claims 1 to 8 when executing the executable instructions stored in the memory.

Images