Data-driven autonomous driving system performance testing methods, media, and equipment

Abstract

The present invention discloses a data-driven autonomous driving system performance testing method, medium, and equipment, relating to the field of autonomous driving testing technology, including: mining driving scenario data for the autonomous driving system to obtain an initial set of driving scenarios; searching driving records to obtain a library of real-vehicle test cases for each scenario; performing risk-related rewards and penalties on the initial set of driving scenarios; conducting simulation testing and analysis of the autonomous driving system to obtain a first sequence of driving performance evaluations; conducting simulation testing and analysis of the autonomous driving system to obtain a second sequence of driving performance evaluations; and tracing anomalies to generate an autonomous driving performance test report. The present invention solves the technical problems of prior art in autonomous driving system performance evaluations, such as incomplete scenario coverage and difficulty in accurately tracing anomalies, resulting in poor evaluation accuracy. It achieves the technical effect of enabling comprehensive simulation testing and anomaly tracing of autonomous driving systems based on different scenario clusters, thereby improving the accuracy of performance evaluations.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving testing technology, and in particular to data-driven autonomous driving system performance testing methods, media, and equipment. Background Art

[0002] With the rapid advancement of science and technology, autonomous driving technology has become a research hotspot and development trend in the global transportation sector. Performance testing of autonomous driving systems is crucial throughout their development and deployment. However, current technologies face numerous challenges in this regard. Traditional performance testing often relies on a limited number of test scenarios, failing to fully encompass the complex and ever-changing nature of real-world traffic conditions. This limits the performance evaluation of autonomous driving systems in diverse traffic environments, road conditions, and weather conditions. Furthermore, if an autonomous driving system experiences an anomaly during testing or actual operation, existing testing methods struggle to quickly and accurately trace the root cause.

[0003] Existing technologies have technical problems such as incomplete scene coverage and difficulty in accurately tracing anomalies in the performance evaluation of autonomous driving systems, resulting in poor evaluation accuracy. Summary of the Invention

[0004] This application provides a data-driven autonomous driving system performance testing method, medium and equipment, which are used to solve the technical problems in the existing technology of autonomous driving system performance evaluation, such as incomplete scene coverage, difficulty in accurately tracing anomalies, and poor evaluation accuracy.

[0005] In view of the above problems, this application provides a data-driven autonomous driving system performance testing method, medium and equipment.

[0006] A first aspect of an embodiment of the present application provides a data-driven autonomous driving system performance testing method, the method comprising:

[0007] Driving scenario data mining is performed on the autonomous driving system based on driving scenario factors to obtain an initial driving scenario set; driving record retrieval is performed on the autonomous driving system based on the initial driving scenario set to obtain a library of real-vehicle test cases for each scenario; risk-associated rewards and penalties are performed on the initial driving scenario set based on the library of real-vehicle test cases for each scenario to obtain a first driving scenario cluster and a second driving scenario cluster; based on a multi-channel driving performance evaluation, a simulation test analysis is performed on the autonomous driving system based on the first driving scenario cluster to obtain a first driving performance evaluation sequence; based on the multi-channel driving performance evaluation, a simulation test analysis is performed on the autonomous driving system based on the second driving scenario cluster to obtain a second driving performance evaluation sequence; based on the first driving performance evaluation sequence and the second driving performance evaluation sequence, anomaly tracing is performed to generate an autonomous driving performance test report.

[0008] In a second aspect of the embodiments of the present application, the present application provides a computer-readable storage medium storing a computer program, which is used to execute the data-driven autonomous driving system performance testing method provided by the present application.

[0009] In a third aspect of the embodiments of the present application, the present application provides an electronic device comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is used to execute the data-driven autonomous driving system performance testing method provided in the present application.

[0010] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0011] The system conducts driving scenario data mining on the autonomous driving system based on driving scenario factors to obtain an initial set of driving scenarios. The system then searches driving records to obtain a library of real-vehicle test cases for each scenario. Risk-related rewards and penalties are applied to the initial set of driving scenarios to obtain a first driving scenario cluster and a second driving scenario cluster. The system then performs simulation testing and analysis on the autonomous driving system to obtain a first sequence of driving performance evaluations. The system then performs simulation testing and analysis on the autonomous driving system based on the second driving scenario cluster to obtain a second sequence of driving performance evaluations. Based on the first and second driving performance evaluation sequences, anomaly tracing is performed to generate an autonomous driving performance test report. This achieves the technical effect of conducting comprehensive simulation testing and anomaly tracing for the autonomous driving system based on different scenario clusters, improving the accuracy of performance evaluations. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A flowchart of a data-driven autonomous driving system performance testing method provided in an embodiment of the present application;

[0014] Figure 2 This is a schematic diagram of the structure of an electronic device provided in this application.

[0015] Description of reference numerals: processor 21 , memory 22 , input device 23 , output device 24 . DETAILED DESCRIPTION

[0016] This application provides a data-driven autonomous driving system performance testing method, medium and equipment to solve the technical problems in the existing technology of autonomous driving system performance evaluation, such as incomplete scene coverage, difficulty in accurately tracing anomalies, and poor evaluation accuracy.

[0017] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work are within the scope of protection of this application.

[0018] Example 1, as Figure 1 As shown, the present application provides a data-driven autonomous driving system performance testing method, the method comprising:

[0019] Step S100: performing driving scenario data mining on the automatic driving system according to the driving scenario factors to obtain an initial driving scenario set.

[0020] Specifically, driving scenario factors include traffic environment, road environment, and weather conditions. First, based on these factors, feature recognition is performed on the autonomous driving system's vast set of driving scenario records, accurately separating the traffic environment dataset, road environment dataset, and weather condition dataset. Next, a frequency analysis is performed on each dataset. For example, the frequency coefficient is calculated for each data point in the traffic environment dataset, frequently occurring data is filtered out, and vectorized to construct a traffic environment vector space. The same process is performed on the road environment and weather condition datasets. Finally, the constructed traffic environment, road environment, and weather condition vector spaces are randomly combined to obtain an initial set of driving scenarios covering a variety of complex scenarios, laying the foundation for subsequent autonomous driving system performance testing.

[0021] Step S200: searching driving records of the automatic driving system according to the initial driving scenario set to obtain a library of real vehicle test cases for each scenario.

[0022] Specifically, a comprehensive search is conducted across the vast amount of driving records stored by the autonomous driving system, using each scenario in the initial driving scenario set as search criteria. By comparing and matching the characteristic information of the initial driving scenario with the scenario description, vehicle operating status, and other data in the driving record, real vehicle test records corresponding to each initial driving scenario are selected. These records contain the autonomous driving system's operational performance in different scenarios, vehicle control data, and surrounding environment information during actual road driving. These selected records are categorized and organized according to different scenarios, ultimately forming a library of real-world vehicle test cases for each scenario. These use case libraries truly reflect the operation of the autonomous driving system in real-world scenarios, providing critical real-world data support for subsequent performance evaluation and analysis.

[0023] Step S300: performing risk-related rewards and penalties on the initial driving scenario set according to the real vehicle test case library for each scenario to obtain a first driving scenario cluster and a second driving scenario cluster.

[0024] Specifically, the initial set of driving scenarios is first risk-assessed using actual operational data from a library of real-vehicle test cases for each scenario, and the corresponding scenario risk coefficient is calculated. Based on the calculated multiple scenario risk coefficients, the initial set of driving scenarios is divided into two categories: a first set of driving scenarios with a scenario risk threshold less than or equal to a pre-set scenario risk threshold, and a second set of driving scenarios with a scenario risk threshold greater than that threshold. For the first set of driving scenarios, the scenario coupling coefficient is determined by evaluating pairwise similarities between scenarios. Based on the comparison of this coefficient with the scenario coupling threshold, the scenarios are adaptively decoupled and optimized to obtain the first driving scenario cluster. For the second set of driving scenarios, the number of variations in each scenario is calculated using a driving scenario variation number analytical function. Based on this number, the scenarios are then mutated and expanded to obtain the second driving scenario cluster. This achieves differentiated processing and optimization for scenarios of different risk levels, providing a more targeted scenario set for subsequent precise simulation testing and analysis.

[0025] Step S400: Based on the driving performance evaluation multi-channel, the automatic driving system is simulated tested and analyzed according to the first driving scenario cluster to obtain a first sequence of driving performance evaluation.

[0026] Specifically, starting from the first driving scenario cluster, each driving scenario therein is traversed one by one. After each scenario is extracted, the autonomous driving system is simulated and tested multiple times for that scenario to obtain multiple first scenario test data sets. To ensure data reliability, these data sets are subjected to centralized value calculation to obtain a first scenario credible test data set. The driving performance evaluation multi-channel consists of a basic driving performance evaluation channel, a dynamic interactive driving performance evaluation channel, and a comprehensive driving performance calculation channel. The first scenario credible test data set is sequentially input into the basic driving performance evaluation channel and the dynamic interactive driving performance evaluation channel to obtain the first basic driving performance coefficient and the first dynamic interactive driving performance coefficient, respectively. These two coefficients are then input into the comprehensive driving performance calculation channel to obtain the first comprehensive driving performance coefficient. These coefficients together constitute the first scenario driving performance evaluation results and are sequentially added to the first driving performance evaluation sequence to facilitate a comprehensive and systematic analysis of the performance of the autonomous driving system in different scenarios.

[0027] Step S500: Based on the driving performance evaluation multi-channel, the automatic driving system is simulated tested and analyzed according to the second driving scenario cluster to obtain a second sequence of driving performance evaluation.

[0028] Specifically, based on the second driving scenario cluster, each driving scenario therein is traversed in sequence. For each extracted driving scenario, multiple simulation tests are conducted on the autonomous driving system to collect multiple second scenario test data sets. To obtain more reliable data, these data sets are processed through centralized value calculation to obtain a second scenario credible test data set. The driving performance evaluation multi-channel includes a basic driving performance evaluation channel, a dynamic interactive driving performance evaluation channel, and a comprehensive driving performance calculation channel. The second scenario credible test data set is input into the basic driving performance evaluation channel to obtain a second basic driving performance coefficient; it is input into the dynamic interactive driving performance evaluation channel to obtain a second dynamic interactive driving performance coefficient. These two coefficients are then input into the comprehensive driving performance calculation channel to obtain a second comprehensive driving performance coefficient. These coefficients together constitute the second scenario driving performance evaluation results and are added one by one in sequence to the second driving performance evaluation sequence, providing key data support for the subsequent comprehensive evaluation of the autonomous driving system performance.

[0029] Step S600: Tracing the source of abnormalities based on the first driving performance evaluation sequence and the second driving performance evaluation sequence to generate an autonomous driving performance test report.

[0030] Specifically, the data from the first and second sequences of driving performance evaluations were compared element by element, highlighting any differences in driving performance coefficients and test results. Utilizing fault tree analysis, starting from the system architecture level, the system constructs a fault tree based on logical relationships, using driving scenarios, sensors, and algorithm modules as nodes. Combining the abnormal data from both sequences, the system traces back from the underlying events to the underlying causes, investigating whether the cause of the anomaly is due to abnormal sensor data transmission in a specific scenario, thereby affecting algorithmic decision-making and ultimately leading to abnormal driving performance. Causal analysis is also employed to analyze the causal relationship between driving performance anomalies and various system components, identifying the path through which the anomaly originated. After tracing the anomaly, the system generates a detailed autonomous driving performance test report, compiling information such as the cause of the anomaly, the scenarios involved, and the degree of impact on autonomous driving system performance according to a standard industry test report template. The report also includes corresponding improvement suggestions and optimization directions, providing strong support for improving autonomous driving system performance.

[0031] In one possible implementation, step S300 further includes:

[0032] Step S310: performing risk assessment on the initial driving scenario set according to the real vehicle test case library for each scenario to obtain multiple scenario risk coefficients.

[0033] Step S320: Classify the initial driving scenario set according to the multiple scenario risk coefficients to obtain a first driving scenario set that is less than or equal to a scenario risk threshold, and a second driving scenario set that is greater than the scenario risk threshold.

[0034] Step S330: performing decoupling optimization on the first driving scenario set to obtain the first driving scenario cluster.

[0035] Step S340: performing mutation and expansion on the second driving scene set to obtain the second driving scene cluster.

[0036] Specifically, the analytic hierarchy process (AHP) is used to conduct risk assessment to obtain risk coefficients for multiple scenarios. First, the goal of the risk assessment is to assess the risk level of each scenario in the initial set of driving scenarios. A hierarchical model is established, consisting of a criterion layer, such as vehicle behavior stability, traffic conflict potential, and environmental complexity, and a corresponding indicator layer. Based on data from a library of real-vehicle test cases for each scenario, indicators are extracted, including vehicle trajectory, changes in distance from other road users, road type, and the number of traffic signs in the scenario. Through expert scoring or statistical analysis, a judgment matrix is constructed to determine the relative weights of each criterion and indicator. For example, if vehicle behavior stability and traffic conflict potential are considered to have a greater impact on scenario risk, a corresponding weight can be assigned in the judgment matrix. Finally, the weights and values of each indicator are combined to calculate a comprehensive risk score for each scenario. These scores form multiple scenario risk coefficients, which are used to quantify the risk level of each scenario.

[0037] The multiple scenario risk coefficients obtained are used as the basis for classification. A scenario risk threshold is set in advance, which is determined based on the safety standards of the autonomous driving system, past experience, and industry specifications. For each scenario in the initial driving scenario set, its corresponding scenario risk coefficient is compared with the threshold. Scenarios with scenario risk coefficients less than or equal to the threshold are selected and grouped together to form the first driving scenario set. These scenarios are considered to have relatively low risks, and the autonomous driving system is less likely to encounter dangerous situations in these scenarios; scenes with scenario risk coefficients greater than the threshold are classified into the second driving scenario set. These scenarios have higher risks and require subsequent focus and processing, thereby achieving risk stratification and classification of the initial driving scenario set, laying the foundation for more targeted test and analysis work in the future.

[0038] The Euclidean distance between each scene in the first driving scenario set is calculated to measure pairwise similarity, thereby obtaining multiple scenario coupling coefficients. A reasonable scenario coupling threshold is set; if the scenario coupling coefficient is greater than or equal to the threshold, the two scenarios are highly coupled. For highly coupled scenarios, principal component analysis (PCA) is used for decoupling. PCA transforms the original scene data into a new orthogonal coordinate system through linear transformation, ensuring that the data variance in the new coordinate system is as large as possible and independent of each other. This extracts key features, removes redundant information, and achieves adaptive decoupling of the scenarios. The decoupled scenario data is then clustered and integrated according to factors such as scenario type and environmental conditions to form a set of scenarios with clear distinction and independence, namely the first driving scenario cluster. This allows for more accurate evaluation of the autonomous driving system's performance in low-risk scenarios.

[0039] The number of variations for each driving scenario in the second driving scenario set is calculated using a driving scenario variation number analytical function, combined with the scenario risk coefficient corresponding to each driving scenario in the second driving scenario set. This function comprehensively considers factors such as the difference between the scenario risk coefficient and the scenario risk threshold, as well as upper and lower limits on the number of driving scenario variations. After obtaining the number of variations for each driving scenario, mutation expansion operations are performed on the second driving scenario set based on these numbers. For example, by varying certain parameters within the scenario (such as traffic flow, road slope, and weather conditions), multiple new scenarios similar to but different from the original scenario are generated, thereby expanding the second driving scenario set. This ultimately forms a second driving scenario cluster, enriching the diversity of high-risk scenarios and enabling more comprehensive testing of the autonomous driving system's performance in complex, high-risk scenarios.

[0040] In one possible implementation, step S330 further includes:

[0041] Step S331: performing pairwise similarity evaluation on the first driving scene set to obtain multiple scene coupling coefficients.

[0042] Step S332: determining whether the plurality of scene coupling coefficients are greater than or equal to a scene coupling threshold, and obtaining a plurality of scene coupling determination results.

[0043] Step S333: Adaptively decoupling the first driving scene set according to the multiple scene coupling judgment results to obtain the first driving scene cluster.

[0044] Specifically, a series of operations are performed on the first set of driving scenarios. First, feature extraction is performed on each scenario in the first set based on multiple dimensions, such as traffic environment, road environment, and weather conditions, converting each scenario into a multidimensional feature vector. Using appropriate calculation methods such as Euclidean distance, these feature vectors are then pairwise calculated to obtain multiple scenario coupling coefficients, which intuitively reflect the degree of similarity between the scenarios.

[0045] Each scenario coupling coefficient is compared with a pre-set scenario coupling threshold, which is determined based on a deep understanding of autonomous driving system performance testing, analysis of a large amount of past test data, and industry standards and specifications. During the comparison process, a judgment operation is performed on each scenario coupling coefficient: if the coefficient is greater than or equal to the scenario coupling threshold, it means that the corresponding two scenarios are highly coupled, and can be considered to have a strong correlation in subsequent test analysis, and is recorded as a positive scenario coupling judgment result; conversely, if the coefficient is less than the scenario coupling threshold, it indicates that the two scenarios are less coupled and need to be considered separately in the test, and is recorded as a negative scenario coupling judgment result. By judging and recording all scenario coupling coefficients one by one, multiple scenario coupling judgment results are eventually generated. These results provide an important basis for the subsequent adaptive decoupling operation of the first driving scenario set, and determine whether the scenarios need to be merged or tested separately.

[0046] Based on the results of the multiple scenario coupling determinations, an adaptive decoupling operation is performed on the first set of driving scenarios. When the scenario coupling coefficient is greater than or equal to the scenario coupling threshold, it indicates that the scenarios are highly similar and contain a large number of overlapping features and situations. To improve testing efficiency and avoid the waste of resources caused by repeated testing, these scenarios are merged. During the merging process, common features of these similar scenarios are extracted, and redundant features are removed to form a comprehensive scenario. For example, if multiple scenarios are largely similar in terms of traffic flow, road type, weather conditions, etc., but differ only in minor details, these similar features are merged, retaining the representative differences, and forming the new merged scenario. Scenarios with coupling below the threshold each possess unique characteristics and are irreplaceable in testing the performance of the autonomous driving system. To comprehensively evaluate the performance of the autonomous driving system in different scenarios and ensure the completeness and accuracy of the test, these scenarios need to be tested separately. This differentiated processing method achieves efficient resource utilization while ensuring comprehensive test scenario coverage. Ultimately, after this adaptive decoupling operation, the first driving scenario cluster is obtained, which provides strong support for subsequent more accurate and efficient testing of the autonomous driving system performance.

[0047] In one possible implementation, step S340 further includes:

[0048] The number of driving scenario variations corresponding to the second driving scenario set is calculated according to a driving scenario variation number analytical function, wherein the driving scenario variation number analytical function is:

[0049] ;

[0050] Among them, R iRepresents the scenario risk coefficient corresponding to the i-th driving scenario in the second driving scenario set, i is a positive integer, S i represents the number of driving scenario variations corresponding to the i-th driving scenario, Floor refers to rounding down, R0 represents the scenario risk threshold, S max Characterizes the upper limit of the number of driving scene variations, S min The lower limit of the driving scene variation number is represented. The second driving scene set is mutated and expanded according to the variation number of each driving scene to obtain the second driving scene cluster.

[0051] Specifically, first, the function is analyzed according to the number of variations in a specific driving scenario, i.e. , calculate the number of variations of each driving scenario in the second driving scenario set. R i Represents the scenario risk coefficient corresponding to the i-th driving scenario in the second driving scenario set. Its value reflects the potential risk level of the scenario. i is a positive integer starting from 1 to ensure that it corresponds to each scenario. R0 is a pre-set scenario risk threshold, which is the benchmark for judging the level of scenario risk. It is determined by the safety standards of the autonomous driving system and past test experience. max and S min They are the upper and lower limits of the number of driving scenario mutations, which limit the range of each scenario mutation. For example, the minimum mutation is 2 times and the maximum mutation is 10 times. This is to avoid excessive mutation while ensuring the comprehensiveness of the test. The Floor function is used to round down to ensure that the calculated number of mutations is an integer.

[0052] The number of mutations in each scene is calculated through this function After that, the second set of driving scenarios is mutated and expanded. For example, if the calculated number of mutations for a scenario is three, a new scenario is generated each time by varying factors such as traffic flow (e.g., increasing or decreasing the number of vehicles, changing the vehicle speed distribution), road environment (e.g., adjusting road slope and curvature), or weather conditions (e.g., changing light intensity and precipitation probability). These newly generated scenarios are then merged with those in the original second set of driving scenarios to form the second driving scenario cluster. This approach aims to enrich the test scenarios, thereby enabling more comprehensive and in-depth testing of the autonomous driving system's performance in high-risk scenarios and providing more robust data support for evaluating the reliability of autonomous driving systems in complex and dangerous environments.

[0053] In one possible implementation, step S400 further includes:

[0054] Step S410: traverse the first driving scene cluster and extract the first driving scene.

[0055] Step S420: Perform multiple simulation tests on the autonomous driving system according to the first driving scenario to obtain multiple first scenario test data sets.

[0056] Step S430: performing centralized value calculation based on the multiple first-scenario test data sets to obtain a first-scenario credible test data set.

[0057] Step S440: Inputting the first scenario credible test data set into the driving performance evaluation multi-channel to obtain a first scenario driving performance evaluation result.

[0058] Step S450: Add the first scenario driving performance evaluation result to the first driving performance evaluation sequence.

[0059] Specifically, the system first locates the generated first driving scenario cluster. This cluster is a set of driving scenarios derived from prior risk-related rewards and penalties, as well as decoupling optimization of the low-risk scenario set. An iterative algorithm is then used to sequentially access each scenario element in the first driving scenario cluster in a specific order (e.g., sequential or random traversal). During the traversal process, each scenario is extracted as the first driving scenario to be processed. This extraction operation can be performed through a data reading interface or related data processing modules, separating the data for that scenario from the storage structure of the first driving scenario cluster in preparation for subsequent automated driving system simulation testing of that scenario.

[0060] After the first driving scenario is extracted, the autonomous driving system undergoes multiple simulation tests. Using the autonomous driving simulation test platform, a virtual test environment closely resembling the real-world scenario is constructed based on the detailed features of the first driving scenario, including the traffic environment, road conditions, and weather conditions. Within this environment, a variety of driving conditions are simulated, such as varying vehicle speeds, adjusting traffic density, and setting varying road slopes and curve curvatures. For each simulated condition, the autonomous driving system undergoes a test run, recording a large amount of data covering the vehicle's trajectory, sensor feedback, and command output from the decision-making system. After multiple such tests, the data recorded from each test is organized and stored, ultimately generating multiple first-scenario test datasets. These datasets provide a rich foundation for subsequent analysis of the autonomous driving system's performance in these scenarios.

[0061] A central value calculation is performed on the multiple first-scenario test data sets obtained to obtain a first-scenario credible test data set. The multiple first-scenario test data sets contain data generated by multiple simulation tests of the autonomous driving system in the same driving scenario. Due to various random factors in the testing process, this data is subject to fluctuations and errors. In order to obtain data that more accurately reflects the actual performance of the autonomous driving system in this scenario, a central value calculation is required. During the calculation, the mean is used to add the similar data (such as vehicle speed, steering angle, etc.) in the multiple first-scenario test data sets, and then divide it by the number of tests to obtain the average value of these data. Through this central value calculation, the influence of abnormal data and random errors can be eliminated to a certain extent, making the obtained data more representative and reliable. Ultimately, a first-scenario credible test data set is obtained, providing a reliable basis for the subsequent accurate evaluation of the performance of the autonomous driving system in this scenario.

[0062] Once the first-scenario credible test dataset is obtained, it is input into the driving performance evaluation multi-channel to obtain the first-scenario driving performance evaluation results. The driving performance evaluation multi-channel consists of a basic driving performance evaluation channel, a dynamic interactive driving performance evaluation channel, and a comprehensive driving performance calculation channel. First, the first-scenario credible test dataset is input into the basic driving performance evaluation channel. This channel analyzes and calculates key indicators related to basic driving in the dataset, such as vehicle acceleration and deceleration performance, and driving stability, to derive the first basic driving performance coefficient. Next, the same dataset is input into the dynamic interactive driving performance evaluation channel, which focuses on the autonomous driving system's performance in dynamic interactive scenarios, such as interactions with other vehicles and pedestrians, to obtain the first dynamic interactive driving performance coefficient. The first basic driving performance coefficient and the first dynamic interactive driving performance coefficient are then input into the comprehensive driving performance calculation channel. This channel comprehensively considers the first two coefficients and other relevant factors, applying a weighted algorithm to calculate the first comprehensive driving performance coefficient. These three coefficients together constitute the first-scenario driving performance evaluation results, comprehensively reflecting the autonomous driving system's performance in that scenario.

[0063] The obtained driving performance evaluation results of the first scenario are added to the first driving performance evaluation sequence. This sequence is used to record the performance evaluation of the autonomous driving system in each scenario in the first driving scenario cluster, providing key data support for subsequent comparative analysis, anomaly tracing, and generation of a complete autonomous driving performance test report.

[0064] In one possible implementation, step S440 further includes:

[0065] Step S441: The driving performance evaluation multi-channel includes a basic driving performance evaluation channel, a dynamic interactive driving performance evaluation channel and a comprehensive driving performance calculation channel.

[0066] Step S442: Inputting the first scenario credible test data set into the basic driving performance evaluation channel to obtain a first basic driving performance coefficient.

[0067] Step S443: Input the first scenario credible test data set into the dynamic interactive driving performance evaluation channel to obtain a first dynamic interactive driving performance coefficient.

[0068] Step S444: inputting the first basic driving performance coefficient and the first dynamic interactive driving performance coefficient into the comprehensive driving performance calculation channel to obtain a first comprehensive driving performance coefficient.

[0069] Step S445: Output the first basic driving performance coefficient, the first dynamic interactive driving performance coefficient, and the first comprehensive driving performance coefficient as the first scenario driving performance evaluation result.

[0070] Specifically, the multi-channel driving performance evaluation includes a basic driving performance evaluation channel, a dynamic interactive driving performance evaluation channel, and a comprehensive driving performance calculation channel. These channels work together to evaluate the performance of the autonomous driving system in specific scenarios from different dimensions.

[0071] The first scenario's credible test dataset is input into the basic driving performance evaluation channel, and a decision tree algorithm model is used to obtain the first basic driving performance coefficient. The decision tree algorithm gradually partitions the dataset based on its characteristics, constructing a tree structure to perform classification or regression tasks. For basic driving performance evaluation, the algorithm first extracts features from the dataset related to basic driving performance, such as vehicle speed change rate, steering angle deviation, and braking response time. Starting from the root node, the algorithm partitions the data based on the values of a specific feature. For example, if the speed change rate exceeds a certain threshold, the data is partitioned into one branch; if it does not, it is partitioned into another branch. This process is repeated until the data in each branch is highly homogeneous, forming a decision tree. After the decision tree is constructed, the first scenario's credible test dataset is input. The decision tree performs classification or regression calculations along the tree branches based on the data characteristics, ultimately outputting a numerical value representing basic driving performance. This value is the first basic driving performance coefficient, which comprehensively reflects the performance of the autonomous driving system in basic driving.

[0072] The first scenario's credible test dataset is fed into the dynamic interactive driving performance evaluation pipeline, where a convolutional neural network (CNN) algorithm model is used to obtain the first dynamic interactive driving performance coefficient. CNN algorithms excel at processing images and spatially structured data, making them equally applicable to dynamic interaction data between vehicles and their surroundings in autonomous driving scenarios. First, feature data related to dynamic interactions is extracted from the first scenario's credible test dataset, including information such as the position, speed, and direction of surrounding vehicles, as well as the position and direction of pedestrians. This data is preprocessed and converted into a format suitable for CNN input, such as converting position information into a two-dimensional matrix. The CNN model consists of multiple convolutional, pooling, and fully connected layers. Convolutional layers slide convolution kernels over the data, extracting features from local regions and capturing spatial features and patterns in the data, such as the relative position changes between vehicles. Pooling layers reduce the data's dimensionality, preserving key features while reducing computational complexity. Through multiple convolutional and pooling layers, data features are gradually abstracted and extracted. Finally, a fully connected layer integrates these extracted features and maps them to an output node, the value of which is the first dynamic interactive driving performance coefficient. This coefficient comprehensively reflects the performance of the autonomous driving system in dynamic interaction scenarios. The higher the coefficient, the better the system performs in dynamic interactions with other vehicles, pedestrians, etc., such as being able to more accurately predict the behavior of other road users and make reasonable decisions more timely.

[0073] The first basic driving performance coefficient and the first dynamic interactive driving performance coefficient are input into the comprehensive driving performance calculation channel to obtain a first comprehensive driving performance coefficient that more comprehensively reflects the performance of the autonomous driving system. First, the weights of the two coefficients are determined. Assume that the weight of the basic driving performance coefficient is W1, and the weight of the dynamic interactive driving performance coefficient is W2, with W1 + W2 = 1. The weightings should be determined based on the degree of emphasis the autonomous driving system places on basic driving performance and dynamic interactive performance in different scenarios. For example, in scenarios with heavy traffic and complex road conditions, dynamic interactive performance is more important, so the value of W2 can be increased accordingly. In contrast, in scenarios with simpler road conditions, the weight of basic driving performance, W1, can be appropriately increased. Next, a weighted calculation is performed using the formula: First Comprehensive Driving Performance Coefficient = W1 × First Basic Driving Performance Coefficient + W2 × First Dynamic Interactive Driving Performance Coefficient. To calculate this, the first basic driving performance coefficient obtained previously is multiplied by W1, and the first dynamic interactive driving performance coefficient is multiplied by W2. The two products are then added together to obtain the first comprehensive driving performance coefficient. This coefficient integrates information on basic driving performance and dynamic interactive driving performance, and can more comprehensively and accurately evaluate the comprehensive performance of the autonomous driving system in the first driving scenario, providing key data support for subsequent performance evaluation and analysis.

[0074] After a series of complex operations, the first basic driving performance coefficient, the first dynamic interactive driving performance coefficient, and the first comprehensive driving performance coefficient are obtained. These three coefficients reflect the performance of the autonomous driving system in the first driving scenario from different dimensions. The first basic driving performance coefficient reflects the system's basic performance in routine driving operations such as acceleration, deceleration, and steering; the first dynamic interactive driving performance coefficient focuses on the system's performance when interacting with other road users (such as vehicles and pedestrians); and the first comprehensive driving performance coefficient combines the first two, comprehensively reflecting the system's overall performance in this scenario. These three coefficients are integrated and output to form the first-scenario driving performance evaluation result. This result quantifies the performance of the autonomous driving system in the first driving scenario and provides important data for subsequent evaluation of the system's comprehensive performance in various scenarios, comparison of performance differences between different scenarios, and anomaly tracing. It helps to fully understand the strengths and weaknesses of the autonomous driving system, thereby promoting its optimization and improvement.

[0075] In one possible implementation, step S100 further includes:

[0076] Step S110: The driving scenario factors include traffic environment, road environment and weather conditions.

[0077] Step S120: Based on the driving scene factors, feature recognition is performed on the driving scene record set of the automatic driving system to obtain a traffic environment data set, a road environment data set, and a weather condition data set.

[0078] Step S130: performing frequency sorting on the traffic environment dataset, the road environment dataset, and the weather condition dataset to construct a traffic environment vector space, a road environment vector space, and a weather condition vector space.

[0079] Step S140: generating the initial driving scenario set by randomly combining the traffic environment vector space, the road environment vector space, and the weather condition vector space.

[0080] Specifically, driving scenario factors cover three key dimensions: traffic environment, road environment, and weather conditions. As an important component of the driving scenario, the traffic environment encompasses many dynamic and static elements. Dynamic elements include vehicle flow, speed distribution, the proportion of different types of vehicles (such as cars, trucks, and buses), their driving trajectories, and following distances. Static elements include the status of traffic lights, the setting and type of traffic signs, etc. These factors together shape the traffic conditions faced by vehicles when driving on the road. The road environment mainly involves the physical properties of the road itself, such as the type of road, which can be divided into highways, urban main roads, secondary roads, rural roads, etc. The number of lanes and lane widths of different types of roads vary; the slope of the road, which includes uphill, downhill, and flat roads. The size of the slope affects the vehicle's power demand and driving stability; and the curvature of the curve determines the difficulty of control and speed limit when the vehicle turns. Weather conditions also have a significant impact on driving, ranging from sunny, cloudy, and overcast days with good lighting and no precipitation to inclement weather such as rainy, snowy, and foggy days. The differences in visibility, road surface wetness, and light intensity in different weather conditions will change the driver or autonomous driving system's perception and response strategies of the road and surrounding environment.

[0081] A method based on machine learning and data mining is employed. First, the raw data in the record set is preprocessed to remove noise and missing values to improve data quality. For traffic environment feature recognition, object detection algorithms (such as the YOLO series) are used to process the image or video data in the record set to identify objects such as vehicles, pedestrians, traffic signs, and signals. Vehicle counts, types, and movement directions are counted, and vehicle speeds and distances are calculated using sensor data (such as radar and lidar). This constructs a traffic environment dataset. For road environment feature recognition, semantic segmentation algorithms (such as U-Net) are used to analyze image data to distinguish between different road types (such as highways and city streets), lane lines, and road edges. Information such as road slope, curvature, and elevation is extracted from map and sensor data to form a road environment dataset. For weather condition recognition, image classification algorithms (such as ResNet) are used to classify the images in the record set and determine the weather type (e.g., sunny, rainy, foggy). This is verified and supplemented with meteorological sensor data (such as humidity and light intensity) to ultimately generate a weather condition dataset.

[0082] To construct the vector space, frequency analysis is performed on traffic environment, road environment, and weather condition datasets. For the traffic environment dataset, relevant features such as vehicle density, traffic flow speed, and compliance with traffic rules are first identified. Using data mining techniques, the frequency of each eigenvalue in the dataset is counted, such as the number of vehicle density occurrences in different time periods and road sections. Based on this frequency information, each eigenvalue is mapped to a dimension in the vector space, with the eigenvalue's frequency serving as the coordinate value on that dimension, thereby constructing the traffic environment vector space. For the road environment dataset, features such as road type, slope, and curve curvature are identified, and the frequency of each eigenvalue is counted, such as the number of occurrences of different road types (freeways, urban streets, etc.). Eigenvalues are mapped to the dimensions of the vector space, and the road environment vector space is constructed using frequency as the coordinate. For the weather condition dataset, features such as weather type (sunny, rainy, foggy, etc.), visibility, and precipitation intensity are identified. The frequency of each eigenvalue is counted, and the weather condition vector space is constructed using these eigenvalues as the dimensions and frequency as the coordinate.

[0083] Using a random number generation algorithm, a vector is randomly selected from the traffic environment vector space, the road environment vector space, and the weather condition vector space. The traffic environment vector represents a specific traffic condition, such as a specific vehicle density, traffic flow speed, and traffic rule compliance pattern; the road environment vector corresponds to specific road attributes, such as road type, slope, and curve curvature; and the weather condition vector represents specific weather conditions, such as weather type, visibility, and precipitation intensity. Combining these three randomly selected vectors forms a complete initial driving scenario that integrates traffic, road, and weather characteristics. By repeating this random selection and combination process multiple times, a large number of different initial driving scenarios can be generated. These scenarios constitute the initial driving scenario set, providing a rich and diverse simulation scenario for subsequent comprehensive testing and evaluation of the autonomous driving system.

[0084] In one possible implementation, step S130 further includes:

[0085] Step S131: performing frequency calculation on each traffic environment data in the traffic environment data set to obtain a frequency coefficient of each traffic environment.

[0086] Step S132: filtering the traffic environment data set based on the traffic environment frequency coefficients to obtain a plurality of frequent traffic environment data that meet a predetermined traffic environment frequency coefficient.

[0087] Step S133: performing vectorization processing on the plurality of frequent traffic environment data to generate the traffic environment vector space.

[0088] Specifically, the traffic environment dataset contains a rich variety of traffic environment data, such as vehicle flow during different time periods, vehicle density on different road sections, changes in traffic light status, and the occurrence of traffic violations. To obtain the frequency coefficient for each traffic environment data point, the statistical scope and time span must be clearly defined to ensure data consistency and integrity. Next, for each specific traffic environment data point in the dataset, the number of times it appears in the entire dataset is counted. The resulting ratio is the frequency coefficient for that traffic environment data point. For example, if a high vehicle density occurs 200 times during a specific time period on a particular road section among 1000 traffic environment records, the frequency coefficient for that traffic environment data point is 200 ÷ 1000 = 0.2. By calculating the frequency coefficient for all traffic environment data points in this manner, the corresponding traffic environment frequency coefficient for each data point is obtained. These coefficients provide a key basis for subsequent screening and construction of the vector space.

[0089] Traffic environment datasets are filtered based on the calculated frequency coefficients of each traffic environment. A predetermined frequency coefficient threshold is set, and traffic environment data with a frequency coefficient below the threshold is excluded. Only traffic environment data with a frequency coefficient at or above the threshold is retained. These retained data are multiple frequent traffic environment data that meet the predetermined conditions. They represent traffic environment features that appear frequently in real traffic scenarios and are of great significance for constructing a representative traffic environment vector space.

[0090] The selected frequent traffic environment data are vectorized, each frequently occurring data point being assigned a vector dimension, with its frequency coefficient in the dataset serving as the coordinate value along that dimension. This allows each frequently occurring data point to be represented by a vector, and all these vectors together form a traffic environment vector space. This vector space intuitively reflects various frequently occurring traffic environment features and their relative importance, providing an effective data foundation for subsequent autonomous driving scenario simulation and performance evaluation based on traffic environments.

[0091] Example 2, based on the same inventive concept as the data-driven autonomous driving system performance testing method described in the previous example, provides a computer-readable storage medium that can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the data-driven autonomous driving system performance testing method described in the present example. A processor executes the software programs, instructions, and modules stored in the memory to perform various functional applications and data processing on the computer device, thereby implementing the aforementioned data-driven autonomous driving system performance testing method.

[0092] Embodiment 3: Based on the same inventive concept as the data-driven automatic driving system performance testing method in the previous embodiment, this embodiment provides an electronic device, Figure 2 This is a structural diagram of an electronic device provided in accordance with a third embodiment of the present invention, showing a block diagram of an exemplary electronic device suitable for implementing the embodiments of the present invention. Figure 2 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. Figure 2 As shown, the electronic device includes a processor 21, a memory 22, an input device 23 and an output device 24; the number of processors 21 in the electronic device can be one or more. Figure 2 Taking a processor 21 as an example, the processor 21, memory 22, input device 23 and output device 24 in the electronic device can be connected through a bus or other means. Figure 2 The bus connection is taken as an example.

[0093] It should be noted that the order in which the embodiments of the present application are presented is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. Furthermore, the foregoing descriptions of specific embodiments of this specification are provided. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

[0095] This specification and drawings are merely illustrative of the present application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Obviously, those skilled in the art may make various modifications and variations to this application without departing from the scope of this application. Thus, this application is intended to include such modifications and variations as fall within the scope of this application and its equivalents.

Claims

1. A data-driven autonomous driving system performance testing method, characterized in that: include: Perform driving scenario data mining on the autonomous driving system based on driving scenario factors to obtain an initial driving scenario set; Searching driving records of the automatic driving system according to the initial driving scenario set to obtain a real vehicle test case library for each scenario; Perform risk-related rewards and penalties on the initial driving scenario set according to the real vehicle test case library for each scenario to obtain a first driving scenario cluster and a second driving scenario cluster; Based on the driving performance evaluation multi-channel, performing simulation test analysis on the autonomous driving system according to the first driving scenario cluster to obtain a first driving performance evaluation sequence; Based on the driving performance evaluation multi-channel, performing simulation test analysis on the automatic driving system according to the second driving scenario cluster to obtain a second driving performance evaluation sequence; performing abnormality tracing based on the first driving performance evaluation sequence and the second driving performance evaluation sequence, and generating an autonomous driving performance test report; Performing risk-related rewards and penalties on the initial driving scenario set according to the real vehicle test case library for each scenario to obtain a first driving scenario cluster and a second driving scenario cluster, including: Performing a risk assessment on the initial driving scenario set based on the real vehicle test case library for each scenario to obtain multiple scenario risk coefficients; Classifying the initial driving scenario set according to the multiple scenario risk coefficients to obtain a first driving scenario set less than or equal to a scenario risk threshold, and a second driving scenario set greater than the scenario risk threshold; performing decoupling optimization on the first driving scenario set to obtain the first driving scenario cluster; performing mutation and expansion on the second driving scene set to obtain the second driving scene cluster; Performing decoupling optimization on the first driving scenario set to obtain the first driving scenario cluster includes: Performing pairwise similarity evaluation on the first driving scene set to obtain multiple scene coupling coefficients; Determining whether the multiple scene coupling coefficients are greater than or equal to a scene coupling threshold, and obtaining multiple scene coupling determination results; Adaptively decoupling the first driving scene set according to the multiple scene coupling judgment results to obtain the first driving scene cluster; Mutating and expanding the second driving scene set to obtain the second driving scene cluster includes: The number of driving scenario variations corresponding to the second driving scenario set is calculated according to a driving scenario variation number analytical function, wherein the driving scenario variation number analytical function is: ； Among them, R i Represents the scenario risk coefficient corresponding to the i-th driving scenario in the second driving scenario set, i is a positive integer, S i represents the number of driving scenario variations corresponding to the i-th driving scenario, Floor refers to rounding down, R0 represents the scenario risk threshold, S max Characterizes the upper limit of the number of driving scene variations, S min Characterize the lower limit of the number of driving scene variations; The second driving scene set is mutated and expanded according to the number of variations of each driving scene to obtain the second driving scene cluster.

2. The data-driven automatic driving system performance testing method according to claim 1, wherein: Based on the driving performance evaluation multi-channel, a simulation test analysis is performed on the automatic driving system according to the first driving scenario cluster to obtain a first driving performance evaluation sequence, including: Traversing the first driving scene cluster to extract a first driving scene; performing multiple simulation tests on the autonomous driving system according to the first driving scenario to obtain multiple first scenario test data sets; Performing concentrated value calculation based on the multiple first-scenario test data sets to obtain a first-scenario credible test data set; Inputting the first scenario credible test data set into the driving performance evaluation multi-channel to obtain a first scenario driving performance evaluation result; The first scenario driving performance evaluation result is added to the first driving performance evaluation sequence.

3. The data-driven automatic driving system performance testing method according to claim 2, wherein: Inputting the first scenario credible test data set into the driving performance evaluation multi-channel to obtain the first scenario driving performance evaluation result, including: The driving performance evaluation multi-channel includes a basic driving performance evaluation channel, a dynamic interactive driving performance evaluation channel and a comprehensive driving performance calculation channel; Inputting the first scenario credible test data set into the basic driving performance evaluation channel to obtain a first basic driving performance coefficient; Inputting the first scenario credible test data set into the dynamic interactive driving performance evaluation channel to obtain a first dynamic interactive driving performance coefficient; Inputting the first basic driving performance coefficient and the first dynamic interactive driving performance coefficient into the comprehensive driving performance calculation channel to obtain a first comprehensive driving performance coefficient; The first basic driving performance coefficient, the first dynamic interactive driving performance coefficient, and the first comprehensive driving performance coefficient are output as the first scenario driving performance evaluation result.

4. The data-driven automatic driving system performance testing method according to claim 1, wherein: The driving scenario data of the autonomous driving system is mined based on the driving scenario factors to obtain the initial driving scenario set, including: The driving scenario factors include traffic environment, road environment and weather conditions; performing feature recognition on a driving scene record set of the automatic driving system according to the driving scene factor to obtain a traffic environment dataset, a road environment dataset, and a weather condition dataset; Performing frequency sorting on the traffic environment dataset, the road environment dataset, and the weather condition dataset to construct a traffic environment vector space, a road environment vector space, and a weather condition vector space; The initial driving scenario set is generated by randomly combining the traffic environment vector space, the road environment vector space, and the weather condition vector space.

5. The data-driven automatic driving system performance testing method according to claim 4, wherein: Frequency sorting is performed on the traffic environment dataset, the road environment dataset, and the weather condition dataset to construct a traffic environment vector space, a road environment vector space, and a weather condition vector space, including: Calculating the frequency of each traffic environment data in the traffic environment data set to obtain a frequency coefficient of each traffic environment; Filtering the traffic environment data set based on the traffic environment frequency coefficients to obtain a plurality of frequent traffic environment data that meet a predetermined traffic environment frequency coefficient; Vectorization processing is performed on the plurality of frequent traffic environment data to generate the traffic environment vector space.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data-driven autonomous driving system performance testing method as described in any one of claims 1 to 5 is implemented.

7. An electronic device, characterized in that: The electronic device comprises: a memory for storing executable instructions; A processor, configured to implement the data-driven autonomous driving system performance testing method according to any one of claims 1 to 5 when executing the executable instructions stored in the memory.

Images