A Performance Evaluation Method for Autonomous Driving Systems Based on Scene Complexity
By calculating the scenario complexity of each test sample and performing hierarchical evaluation, and combining weighted Bayesian fusion of statistical confidence and source credibility, the problems of unclear scenario complexity and difficulty in multi-source data fusion in the performance evaluation of autonomous driving systems are solved, and efficient and reliable evaluation results are achieved.
Patent Information
- Application Number
- CN202611276496.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-11-25
- Filing Date
- 2026-08-21
- Publication Date
- 2026-11-17
AI Technical Summary
Existing performance evaluation methods for autonomous driving systems lack scientific quantification of driving scenario complexity, cannot effectively distinguish scenarios of different levels of complexity, resulting in large errors in evaluation results, difficulties in fusion of multi-source test data, and a huge workload for testing.
By calculating the scenario complexity of each test sample, a test sample library with complexity labels is formed. Basic simulation, high-precision simulation and real vehicle road test are organized according to the complexity level. Weighted Bayesian fusion is performed by combining statistical confidence and source credibility to obtain the system performance evaluation results.
It achieves explicit complexity labeling for each test sample, solves the sample stratification problem, improves the interpretability and reliability of evaluation results, reduces evaluation error, and unifies multi-source test data.
Smart Images

Figure CN123113807A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of autonomous driving system performance evaluation, specifically relating to an autonomous driving system performance evaluation method based on scenario complexity. Background Technology
[0002] Performance evaluation of autonomous driving systems is a crucial step in ensuring the safe and reliable operation of autonomous vehicles. With the rapid development of autonomous driving technology, evaluation methods not only need to provide overall system performance, but also need to differentiate performance differences under driving scenarios of varying complexity, and explain the statistical reliability of the evaluation results.
[0003] Traditional methods for evaluating the performance of autonomous driving systems have the following limitations: First, they lack a scientific quantification of the complexity of driving scenarios, making it impossible to effectively distinguish and stratify scenarios with different levels of complexity. Second, the evaluation process involves a large sample space and a huge workload, and the evaluation results obtained under limited testing conditions are prone to large fluctuations. Third, the data sources, reliability, and statistical risks of simulation tests and real-vehicle road tests are different, making it difficult for traditional methods to unify and integrate multi-source test data.
[0004] Specifically, traditional evaluation methods have the following problems:
[0005] 1. Excessively large evaluation error range: Traditional evaluation methods typically do not stratify the complexity of the scenario sample space, but instead attempt to directly obtain performance indicators across the entire sample space, resulting in a huge workload for sample evaluation. In actual evaluation, only limited driving conditions can be selected for testing, and the evaluation results obtained under different test conditions may fluctuate within a wide range.
[0006] 2. Lack of quantitative standards for scenario complexity: Existing technologies lack quantifiable methods for calculating the complexity of driving scenarios, making it difficult to unify factors such as traffic density, road geometry, weather conditions, and dynamic obstacles into the same evaluation framework.
[0007] 3. Difficulty in fusing multi-source test data: The costs, authenticity, sample size and statistical risks of basic simulation tests, high-precision simulation tests and real vehicle road tests are different. If the results are simply averaged or only from a single source, the reliability of the evaluation results will be easily reduced.
[0008] 4. The test sample space is too large: The performance of autonomous driving systems is affected by a variety of driving scenario factors, and the sample space of driving scenarios that need to be evaluated is extremely large, lacking effective sample selection, stratification and data fusion strategies.
[0009] Therefore, a performance evaluation method for autonomous driving systems is needed that can calculate the scenario complexity for each test sample, stratify the samples based on the complexity, and fuse multi-source test results according to statistical confidence and source credibility. Summary of the Invention
[0010] To address the issues of unclear definition of scene complexity, unknown sources of hierarchical thresholds, and insufficient basis for multi-source test data fusion in existing autonomous driving system performance evaluation methods, this invention provides a scene complexity-based autonomous driving system performance evaluation method. Using a single driving test sample as the basic evaluation object, the method first calculates the scene complexity of each test sample, forming a test sample library with complexity labels. Then, it organizes basic simulation tests, high-precision simulation tests, and real-vehicle road tests according to complexity levels. Finally, it performs weighted Bayesian fusion based on statistical confidence and source credibility to obtain system performance evaluation results under different complexity levels.
[0011] The performance evaluation method for autonomous driving systems based on scenario complexity includes the following steps:
[0012] Step 1, obtain A driving test sample containing a sequence of images and their corresponding reference images;
[0013] No. One driving test sample The Middle Frame image is The corresponding reference image is ,in =1,2,..., , =1,2,..., ; For this test sample The number of frames.
[0014] Reference images and sequence images are matched according to timestamps, vehicle poses, road mileage, or lane topology to characterize the comparison status of the same road location under standard road, standard weather, or labeled conditions.
[0015] Step 2: Calculate the scenario complexity of each driving test sample to obtain a test sample library with scenario complexity.
[0016] First, for the test samples The Frame Image Compared with reference image Calculate frame-level scene complexity based on basic complexity and inter-frame variability. :
[0017]
[0018] Inter-frame variation correction factor ≥0; when When =1, it is stipulated that =0;
[0019] For test samples No. The basic complexity of a frame:
[0020]
[0021] For traffic density complexity, For the geometric complexity of the road, Due to the complexity of the weather environment, For dynamic obstacle complexity; , , and These are the weight coefficients for the four complexity dimensions, and ;
[0022] For the first The basic complexity of the frame image and the (t-1)th frame image, and the calculated inter-frame variability:
[0023]
[0024] Then, the test sample The scene complexity of the test sample is obtained by fusing the scene complexity of all frames. :
[0025]
[0026] in, For the first Time weighting of frames.
[0027] Finally, a test sample library is compiled from the scenario complexities corresponding to all test samples. .
[0028] Step 3: Based on the complexity of the scenario, the test samples are divided into three levels: low complexity, medium complexity, and high complexity.
[0029] The test sample library contains a total of One test sample, , ,..., Sorted from smallest to largest , ,..., The complexity threshold is determined using quantiles:
[0030]
[0031] in, This serves as the threshold for distinguishing between low and medium complexity. This is the threshold for distinguishing between medium and high complexity.
[0032] Step four: At each complexity level, basic simulation tests, high-precision simulation tests, and real-vehicle road tests are conducted to obtain system performance test results from the three test sources at the same complexity level.
[0033] Test results include the number of tests. Number of times the standard is met Performance index estimates and its corresponding evidence ,in These represent basic simulation testing, high-precision simulation testing, and real-vehicle road testing, respectively. These represent low, medium, and high complexity levels, respectively.
[0034] Step 5: Merge the test results of multiple types under the same complexity level to obtain the system performance evaluation results under different scenario complexity levels.
[0035] The fusion process includes statistical confidence calculation, source credibility calculation, fusion weight calculation, and weighted Bayesian data fusion.
[0036] Regarding the first Type of test, statistical confidence level The calculation is as follows:
[0037]
[0038] in, For the first The test source pre-sets the developer risk or significance level. The smaller the value, the lower the risk of misjudgment allowed by the test source, and the higher the corresponding statistical confidence level.
[0039] Source credibility The calculation is as follows:
[0040]
[0041]
[0042] in, This is the error attenuation coefficient. ; For the first The source of the test is in The average error across a calibration scenario; To calibrate the number of scenes, ; For the source of this test in the Performance estimates in each calibration scenario; Reference performance values for the calibration scenario;
[0043] Fusion weights The calculation is as follows:
[0044]
[0045] In weighted Bayesian data fusion, the first The posterior probability of a test source is obtained based on its own test evidence. as follows:
[0046]
[0047]
[0048] in, The posterior probability is the performance of the merged system. For "autonomous driving systems at complexity level" The system performance assumption is that it "meets the preset performance target"; at the complexity level The test evidence obtained below is ; Defined as the opposing assumption of "not meeting the preset performance target"; This indicates that the system meets the preset performance targets. Under the conditions, the first Obtaining test evidence from various test sources The conditional probability; This indicates that the system does not meet the preset performance targets. Under the conditions, the first The same test evidence was obtained from different test sources. The conditional probability.
[0049] Compared with the prior art, the present invention has the following advantages:
[0050] 1. A sample-by-sample scenario complexity calculation system was established; a scenario complexity value was calculated for each test sample, avoiding the problem of only obtaining the overall complexity of the sample library without being able to stratify individual samples.
[0051] 2. The four complexity dimensions and their image matching criteria have been clarified; the four dimensions of traffic density, road geometry, weather environment and dynamic obstacles are all calculated based on the current frame image and the reference image. The comprehensive geometric difference map is only used as an internal variable of road geometric complexity and is no longer mistakenly regarded as an independent dimension.
[0052] 3. The source of complexity thresholds has been clarified; the thresholds for low, medium and high complexity levels can be determined by statistical quantiles, regulatory thresholds or historical benchmarks, avoiding unclear sources of thresholds.
[0053] 4. The basis for multi-source data fusion was clarified; the statistical confidence level was calculated by the developer's risk, the source credibility was calculated by the calibration set error, and the final fusion weight was obtained by normalizing the two and then used for Bayesian posterior probability fusion, thereby improving the interpretability and reliability of the evaluation results. Attached Figure Description
[0054] Figure 1 This is a flowchart of a performance evaluation method for an autonomous driving system based on scene complexity according to the present invention.
[0055] Figure 2 This is a flowchart of the scenario complexity calculation method for the performance evaluation method of the autonomous driving system of the present invention.
[0056] Figure 3 This is a flowchart illustrating the test sample complexity classification of the performance evaluation method for autonomous driving systems according to the present invention.
[0057] Figure 4 This is a flowchart of the weighted Bayesian fusion of multi-source test results for the autonomous driving system performance evaluation method of the present invention. Detailed Implementation
[0058] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.
[0059] This invention provides a performance evaluation method for autonomous driving systems based on scene complexity, which mainly consists of test sample acquisition, scene complexity calculation, complexity stratification, stratified testing, and multi-source data fusion. First, the scene complexity of each driving test sample is calculated, then tests are carried out according to the complexity level, and finally the results from different test sources are fused to obtain the performance evaluation results of autonomous driving systems at low, medium, and high complexity levels.
[0060] like Figure 1 As shown, it includes the following steps:
[0061] Step 1, obtain A driving test sample containing a sequence of images and their corresponding reference images;
[0062] No. One driving test sample It includes a sequence of images and their corresponding reference images; where the first... Frame image is The corresponding reference image is ,in =1,2,..., , =1,2,..., ; This represents the number of time frames in the sequence.
[0063] Reference images and sequence images are matched according to timestamps, vehicle poses, road mileage, or lane topology to characterize the comparison status of the same road location under standard road, standard weather, or labeled conditions.
[0064] Step 2: Calculate the scenario complexity of each driving test sample to obtain a test sample library with scenario complexity.
[0065] like Figure 2 As shown, specifically:
[0066] Step 201: For each frame of the sequence of test sample images, calculate the traffic density complexity, road geometry complexity, weather environment complexity, and dynamic obstacle complexity of the frame image based on the frame image and the corresponding reference image.
[0067] The calculation of the traffic density complexity includes:
[0068] If the number of detected vehicles corresponding to the frame exceeds the first threshold, the global traffic density complexity is calculated based on the frame as the traffic density complexity corresponding to the frame; otherwise, the local traffic density complexity is calculated based on the frame as the traffic density complexity corresponding to the frame.
[0069] The global traffic density complexity is calculated as follows: based on the vehicle detection results in the image frame, traffic impact regions are searched in the image, features of each vehicle target and each traffic impact region are extracted, the similarity between each vehicle target and each traffic impact region is calculated based on the extracted features, and the weighted summation of the similarity between each vehicle target and each traffic impact region is used to obtain the global traffic density complexity of the image frame.
[0070] The global traffic density complexity is expressed by the following formula:
[0071]
[0072] in, M represents the target number of vehicles, and M represents the number of traffic impact areas. Let be the weight of the i-th vehicle target. Let j be the weight of the traffic impact area. For vehicle target Vi and traffic impact area Similarity;
[0073] The calculation of the road's geometric complexity includes:
[0074] Calculate the geometric features of the road region in the frame image and the road template in the corresponding reference image, including curvature features, slope features, and lane line features; calculate the geometric feature difference map between the road region and the road template to obtain a comprehensive geometric difference map; calculate the weight of each pixel in the road region based on the Gaussian kernel model to obtain the Gaussian weight map of the road region; calculate the road geometric complexity corresponding to the frame image based on the comprehensive geometric difference map and the Gaussian weight map.
[0075] The formula for calculating the geometric complexity of a road is:
[0076]
[0077] Where Ω represents the road region, Gw(x, y) is the pixel weight calculated based on the Gaussian kernel model, and Dg(x, y) is the value of the comprehensive geometric difference map at pixel (x, y).
[0078] The comprehensive geometric difference diagram is calculated in the following way:
[0079] Dg (x, y) = λ1Dc (x, y) + λ2Ds (x, y) + λ3Dl (x, y)
[0080] Where Dc(x, y), Ds(x, y), and Dl(x, y) are the values of the difference maps of curvature features, slope features, and lane line features at pixel (x, y), respectively, and λ1, λ2, and λ3 are the corresponding weight coefficients.
[0081] The calculation of the complexity of the weather environment includes:
[0082] The weather environment region is extracted from the image frame, and multiple feature maps of the weather environment region are calculated. The saliency map of each feature map is calculated based on the image visual saliency algorithm, and the comprehensive saliency map of the weather environment region is obtained based on the saliency map of each feature map. The comprehensive saliency map of the key weather region is extracted from the comprehensive saliency map of the weather environment region, and the weather environment complexity corresponding to the image frame is obtained based on the average saliency value of the comprehensive saliency map of the key weather region.
[0083] The formula for calculating weather and environmental complexity is:
[0084]
[0085] in, Let C be the weather environment complexity corresponding to this frame image, denoted as C in the t-th frame image of the m-th test sample. w m,t ; For key weather areas, | | represents the number of pixels in the key weather area; S(x,y) represents the saliency value of the fused weather environment comprehensive saliency map at pixel (x,y).
[0086] This formula calculates the regional mean of the comprehensive saliency map; therefore, S(x,y) is retained in the final weather and environmental complexity formula and is no longer written as S. c (x,y). If it is necessary to describe the construction process of the comprehensive saliency map, then S c (x,y) represents the saliency value of the c-th type of weather environmental feature saliency map at pixel (x,y), ρ c N represents the fusion weight of the c-th type of weather environmental features. w This represents the number of weather and environmental feature types; each type of saliency map is synthesized into S(x,y) according to its corresponding fusion weight, and the sum of all fusion weights is 1.
[0087] The calculation of the dynamic obstacle complexity includes:
[0088] The motion features, including velocity, trajectory, and behavior features, of the dynamic obstacle region in the frame image and the dynamic obstacle template in the corresponding reference image are calculated respectively. The motion feature difference map between the dynamic obstacle region and the dynamic obstacle template is calculated to obtain a comprehensive motion difference map. The weight of each pixel in the dynamic obstacle region is calculated based on the Gaussian kernel model to obtain the Gaussian weight map of the dynamic obstacle region. The dynamic obstacle complexity corresponding to the frame image is calculated based on the comprehensive motion difference map and the Gaussian weight map.
[0089] The complexity of dynamic obstacles is expressed by the formula: Co = Σ Go (x, y).Dm (x, y) (x, y)∈ Ωo; where Ωo is the dynamic obstacle region, Go (x, y) is the Gaussian weight of the dynamic obstacle region, and Dm (x, y) is the comprehensive motion difference map.
[0090] Step 202: Obtain the basic complexity of the image frame based on the above-mentioned complexity dimensions;
[0091] For the The first test sample Frame image, based on the current frame image Compared with reference image Calculate the complexity of traffic density Road geometric complexity Complexity of weather and environment and dynamic obstacle complexity The basic complexity of the composition :
[0092]
[0093] in, , , and These are the weight coefficients for the four complexity dimensions, and .
[0094] Step 203: Calculate the inter-frame variation of the image based on the basic complexity of the current frame and the previous frame;
[0095] Based on the The basic complexity of the frame image and the (t-1)th frame image, and the calculated inter-frame variability:
[0096]
[0097] Step 204: Based on the basic complexity and inter-frame variability of each frame in the sequence of test samples, obtain the complexity of each frame.
[0098] Frame-level scene complexity :
[0099]
[0100] Inter-frame variation correction factor ≥0; when When =1, it is stipulated that =0;
[0101] Step 205: Calculate the scene complexity of the test sample by weighted summation of the complexity of each frame image.
[0102] The test sample The scene complexity of the test sample is obtained by fusing the scene complexity of all frames. :
[0103]
[0104] in, For the first The scenario complexity of each test sample, The number of frames in this test sample. For the first Time weighting of frames. Can be taken The weights can also be increased based on key event frames such as intersection merging, obstruction, close-range interaction, sudden braking, or pedestrian crossing; exponential decay weights are usually used to highlight the importance of key frames.
[0105] Therefore, each test sample corresponds to a scenario complexity value, and the scenario complexities corresponding to all test samples form the test sample library, represented as follows. .
[0106] Step 3: Based on the complexity of the scenario, the test samples are divided into three levels: low complexity, medium complexity, and high complexity.
[0107] like Figure 3 As shown, the test sample classification process includes: sorting the samples in the test sample library according to the scenario complexity from small to large, determining the first threshold and the second threshold, and dividing them into low complexity, medium complexity and high complexity sample sets respectively.
[0108] The test sample library contains a total of One test sample, , ,..., Sorted from smallest to largest , ,..., In the absence of external regulations or manually specified thresholds, quantiles are used to determine the complexity threshold:
[0109]
[0110] in, This serves as the threshold for distinguishing between low and medium complexity. This serves as the threshold for distinguishing between medium and high complexity. If existing regulations, enterprise testing standards, or historical benchmark libraries provide a threshold, this threshold can be used as a fixed threshold; otherwise, the statistical quantiles described above are used for calculation.
[0111] Three complexity levels: Low complexity: CL ∈ [0, θ1); Medium complexity: CM ∈ [θ1, θ2); High complexity: CH ∈ [θ2, ∞); where, and The complexity threshold is determined through statistical analysis;
[0112] Step four: For each complexity level, perform basic simulation tests, high-precision simulation tests, and real-vehicle road tests in sequence to obtain system performance test results from the three test sources at the same complexity level.
[0113] The basic simulation test uses a basic physical model and simplified environmental parameters, which is highly efficient and can cover a large number of test scenarios. It mainly verifies the basic functions and logical correctness of the system. The high-precision simulation test uses a high-precision physical model, sensor model and detailed environmental parameters, which is closer to the real driving environment and has higher test accuracy. It mainly verifies the performance of the system in complex scenarios. The real vehicle road test is conducted in a real road environment or closed test track. The test results are the most reliable and have the highest authenticity, but the cost and risk are higher. It mainly verifies the safety and reliability of the system in practical applications.
[0114] For each type of system performance test, based on a test sample library with scenario complexity, system performance tests are conducted at different scenario complexity levels to obtain test results for system performance at different scenario complexity levels.
[0115] Test results include the number of tests. Number of times the standard is met Performance index estimates and its corresponding evidence ,in These represent basic simulation testing, high-precision simulation testing, and real-vehicle road testing, respectively. These represent low, medium, and high complexity levels, respectively.
[0116] Step 5: Merge the test results of multiple types under the same complexity level to obtain the system performance evaluation results under different scenario complexity levels.
[0117] like Figure 4 As shown, the multi-source test result fusion process includes: acquiring the test results of basic simulation test, high-precision simulation test and real vehicle road test respectively, calculating the statistical confidence and source credibility of each test source, and performing weighted Bayesian fusion according to complexity level to obtain the system performance evaluation result.
[0118] The fusion process includes statistical confidence calculation, source credibility calculation, fusion weight calculation, and weighted Bayesian data fusion.
[0119] Regarding the first Type of test, statistical confidence level The calculation is as follows:
[0120]
[0121] in, For the first The test source pre-sets the developer risk or significance level. This represents the statistical confidence level obtained by normalizing the developer's risk. It can be determined by test specifications, sample size requirements, or historical validation data; The smaller the value, the lower the risk of misjudgment allowed by the test source, and the higher the corresponding statistical confidence level.
[0122] Source credibility The calculation is as follows:
[0123]
[0124]
[0125] in, To calibrate the number of scenes, ; For the first The source of the test is in The average error across a calibration scenario; For the source of this test in the Performance estimates in each calibration scenario; Reference performance values for the calibration scenario; This is the error attenuation coefficient. ; This represents the credibility of the test source. Credibility is not an arbitrarily assigned value, but rather calculated from calibration set errors or historical consistency statistics.
[0126] Fusion weights The calculation is as follows:
[0127]
[0128] in, For the first The final fusion weights of the various test sources, and This article adopts This indicates the fusion weight, and should not be confused with traffic density weight, time weight, or other intermediate weight symbols.
[0129] In weighted Bayesian data fusion, Defined as "autonomous driving system at the complexity level" The system performance assumption that "meets the preset performance target" will be implemented. Defined as the opposing assumption of "not meeting the preset performance target". This assumption is not a fixed value, but rather refers to whether the system performance reaches the target threshold at a certain complexity level, such as an obstacle avoidance success rate of not less than . No. The test source is in the complexity level The test evidence obtained below is .
[0130] No. The posterior probability of a test source is obtained based on its own test evidence. as follows:
[0131]
[0132]
[0133] in, and The prior probability can be given by historical test results, expert priors, or uniform priors; This represents the posterior probability of the integrated system meeting performance standards. For proportional indicators such as pass rate and obstacle avoidance success rate, For continuous performance indicators, it can be... Replace with the normalized score of the corresponding continuous index, and simultaneously calculate the fusion performance estimate. . This indicates that the system meets the preset performance targets. Under the conditions, the first Obtaining test evidence from various test sources The conditional probability; This indicates that the system does not meet the preset performance targets. Under the conditions, the first The same test evidence was obtained from different test sources. The conditional probability.
[0134] Example:
[0135] I. Test Sample Acquisition
[0136] Get One driving test sample Each test sample Corresponding to a continuous driving segment, containing a sequence of images and reference image The sequence images are the current frame images obtained by autonomous vehicles during real-vehicle testing, simulation testing, or data acquisition and playback; the reference images can be images collected on the same road segment under standard weather and traffic conditions, or reference images generated from high-precision maps, simulated standard scenes, or manual annotations.
[0137] The current frame image is matched with a reference image based on timestamp, vehicle pose, road mileage, or lane topology. The matched image pair is used to calculate the complexity difference of that frame relative to the reference state in terms of vehicle targets, road geometry, weather conditions, and dynamic obstacles.
[0138] II. Calculation of Scene Complexity
[0139] Calculate the scenario complexity for each test sample: for the... The first test sample For each frame image, first calculate the frame-level complexity across four dimensions, then fuse them into a frame-level scene complexity. Finally, the sample-level scene complexity is calculated along the time dimension. .
[0140] 1. Traffic density complexity:
[0141] For the current frame image Vehicle detection is performed to obtain the vehicle target set { , ,..., },in This indicates the number of vehicle targets in the current frame image that have been filtered by confidence, rather than the number of vehicles in the entire sequence or reference image.
[0142] The reference image or road semantic map is divided into regions to obtain a set of traffic impact areas. , ,..., The traffic impact areas include intersections, merging points, ramps, lane narrowing areas, bus stop areas, and areas adjacent to pedestrian crossings. Indicates the number of traffic-affected areas, used to avoid comparisons with medium complexity levels. or total number of samples Confusing.
[0143]
[0144] in, For the first The target weight of each vehicle target can be determined by vehicle type, size, speed, and distance from the vehicle itself; For the first The regional weight of each traffic impact zone can be determined by the zone type, the number of conflict points, and the risk level. For vehicle target Traffic impact area The spatial correlation can be calculated from the normalized distance from the target center to the region boundary, the overlapping area between the target and the region, or the lane topology. This means normalizing the result to the interval between 0 and 1.
[0145] 2. Road geometric complexity:
[0146] Extract the road region based on the current frame image and the reference image. It also extracts geometric features such as curvature, slope, lane lines, and lane width changes.
[0147] Comprehensive geometric difference diagram It is only used for internal calculations of road geometry complexity and is not used as a separate complexity dimension.
[0148]
[0149]
[0150] in, , , These are the difference maps of curvature features, slope features, and lane line features at pixel (x,y). , , For the corresponding weight coefficients and ; The pixel weights are obtained based on the road centerline, drivable area, or Gaussian kernel model.
[0151] 3. Complexity of weather and environment:
[0152] The differences in weather conditions such as illumination, visibility, precipitation, water accumulation, haze, glare, and shadows are calculated based on the current frame image and the reference image.
[0153]
[0154] in, Differences in light intensity or brightness distribution For differences in visibility or image sharpness, To account for differences in weather conditions such as rainfall, snowfall, flooding, and fog, Additional environmental disturbances such as glare, shadows, stains, or dust may cause differences. , , , For the corresponding weight coefficients and All the above differences are normalized to the range of 0 to 1.
[0155] 4. Dynamic obstacle complexity:
[0156] For the current frame image Target detection and tracking are performed to obtain a set of obstacles including pedestrians, non-motorized vehicles, construction vehicles, temporarily parked vehicles, or other obstacles that pose a risk of movement or obstruction of traffic. , ,..., The reference image is used to determine whether the obstacle belongs to a regular target in the reference state, and to calculate its positional and motion differences relative to the reference state.
[0157]
[0158] in, This represents the number of obstacles in the current frame image. For the first The category weights of each obstacle; To normalize the speed, For normalized acceleration, To establish a normalized safe distance from other vehicles or planned trajectories, The interactive risk items are derived based on collision time TTC, traversal trend, or differences in movement direction. , , , For the corresponding weight coefficients and .when hour, =0.
[0159] 5. Frame-level complexity and sample-level complexity:
[0160] The basic complexity is obtained by integrating the four dimensions:
[0161]
[0162] in, , , , The weights are the fusion weights for the four dimensions.
[0163] The aforementioned weights can be determined through the analytic hierarchy process, entropy weighting, historical accident statistical regression, sensitivity analysis, or validation set training, and should be kept consistent in the same batch of evaluations.
[0164] The frame-level scene complexity is obtained by calculating the inter-frame variability by combining the basic complexity between adjacent frames. The scenario complexity of the test sample is obtained by weighted summation:
[0165]
[0166] in, This is a time-weighted approach. If each frame has the same importance, then... If the test samples contain critical events such as intersection merging, pedestrian crossing, sudden braking, occlusion, or close-range interaction, the corresponding frames can be upgraded according to the event risk. and for all Normalization is performed. After calculation, each test sample... Each yields a scene complexity. .
[0167] III. Complexity Level Classification
[0168] The scenarios are stratified based on their complexity values across all test samples. one sample Sort by size from smallest to largest, and determine the threshold using the following formula:
[0169]
[0170] when At that time, the sample was classified as low complexity. ;when At that time, the sample was classified as medium complexity. ;when At that time, the sample was classified as high complexity. If the complexity is not normalized, then the high-complexity interval can be represented as... If a historical benchmark database or regulatory threshold is used, then and It is provided directly by the benchmark library or regulations.
[0171] IV. Layered Performance Testing
[0172] Performance tests were conducted on three types of autonomous driving systems at low, medium, and high complexity levels:
[0173] For each test source and each complexity level Obtain test evidence =( , , ).in For the number of tests, To meet the preset performance target number of times, Alternatively, it can be obtained by normalizing continuous performance indicators. Preset performance targets can be obstacle avoidance success, no collisions, takeover rate below a threshold, minimum safe distance meeting a threshold, or comprehensive safety score meeting a threshold.
[0174] V. Multi-source data fusion
[0175] At the same level of complexity, test results from different test sources are not directly averaged. Instead, the statistical confidence and credibility of each source are calculated first, and then the final fusion weight is obtained.
[0176] 1. Regarding the first Type of test source setting developer risk Calculate the statistical confidence level ;in, =0 indicates basic simulation testing. =1 indicates high-precision simulation testing. =2 indicates real-vehicle road testing. Developer risk. It can be determined by the test specifications, sample size, and historical false positive rate.
[0177] 2. Credibility of the source The error between each test source and the reference performance is calculated based on the calibration scenario;
[0178] 3. Calculate the final fusion weight based on statistical confidence level and source credibility. :
[0179] 4. Weighted Bayesian fusion and Evidence likelihood. For pass / fail test results, evidence likelihood can be calculated based on a binomial distribution or a beta-binomial distribution; for continuous performance metrics, evidence likelihood can be calculated based on a normal distribution, kernel density estimation, or empirical distribution.
[0180]
[0181]
[0182] in, This represents the posterior probability of achieving the target after fusion. These are the estimated performance metrics after fusion. From this, we can obtain system performance evaluation results for low, medium, and high complexity scenarios.
[0183] The fusion output includes the complexity level, the number of samples from each test source, the performance estimate of each test source, and the final fusion weights. Fusion performance estimates and the posterior probability of achieving the standard This results in a traceable performance evaluation of the autonomous driving system.
[0184] The performance evaluation of an autonomous driving system was conducted on a complex traffic section in a city, which included scenarios such as multiple intersections, pedestrian crossings, vehicle merging, and temporary parking. First, a driving test sample from this section was obtained. The test sample contains 1,000 frames of sequential images and corresponding reference images. The sequential images and reference images are matched according to vehicle pose and road mileage.
[0185] The complexity of the test sample was calculated frame by frame across four dimensions: Regarding traffic density, an average of 15 vehicles were detected in the current frame, with vehicles mostly distributed at intersections and merging areas. The calculated complexity was... In terms of road geometry, this section includes curves, slopes, and lane marking changes, which are calculated to... Regarding the weather conditions, the test was conducted in mostly cloudy weather with localized shadows and variations in lighting. The calculations yielded... Regarding dynamic obstacles, there are targets such as pedestrians, non-motorized vehicles, and temporarily parked vehicles, which are calculated... .
[0186] Let the weights of the four dimensions be... =0.35、 =0.25、 =0.15、 =0.25, then the average frame-level complexity of this sample is approximately:
[0187]
[0188] If 1000 frames use uniform time weighting =1 / 1000, and the average complexity of each frame is 0.705, then the scene complexity of this test sample is... Based on the quantiles of the historical sample database, , Therefore, the test sample was classified as medium complexity.
[0189] For samples of medium complexity, three types of tests were performed sequentially: basic simulation tests were used to obtain the obstacle avoidance success rate. High-precision simulation tests yielded the obstacle avoidance success rate. Real-world road tests yielded an obstacle avoidance success rate. .
[0190] Set developer risks , , The statistical confidence level is:
[0191]
[0192]
[0193]
[0194] Source reliability is calculated based on historical calibration set errors. , , The final fusion weight is:
[0195]
[0196]
[0197]
[0198] The success rate of obstacle avoidance in medium-complexity scenarios after fusion is:
[0199]
[0200] If Defined as "the obstacle avoidance success rate of an autonomous driving system in medium-complex scenarios is not less than 85%", and the posterior probability is obtained based on three types of test evidence. , , Then the posterior probability of achieving the standard after fusion is:
[0201]
[0202] In this embodiment, the autonomous driving system achieves an obstacle avoidance success rate of 88.2% in a moderately complex urban traffic scenario, meeting the preset performance target of an obstacle avoidance success rate of no less than 85%, with a posterior probability of 0.854 after fusion. This result provides both a performance estimate and a success probability, which can be used to determine whether the system meets the test admission requirements in a moderately complex scenario.
[0203] This invention calculates the scenario complexity on a sample-by-sample basis, enabling each test sample to obtain a clear complexity label, thus solving the problem of only obtaining the overall results of the sample library without being able to stratify the data.
[0204] This invention unifies four dimensions—traffic density, road geometry, weather environment, and dynamic obstacles—into frame-level complexity calculation, and clarifies the matching relationship between each dimension and the current frame image and reference image, thereby improving the interpretability of complexity calculation.
[0205] This invention determines the complexity level threshold by using statistical quantiles, historical benchmark databases, or regulatory thresholds, so that the classification of low, medium, and high complexity levels has a clear source and can be dynamically updated according to changes in the size of the sample database.
[0206] This invention calculates statistical confidence levels based on developer risk, calculates source confidence levels based on calibration set errors, and employs... Using Bayesian fusion as the final fusion weight avoids problems such as mixed use of weight symbols and unclear sources of credibility, thus improving the reliability of the fusion results of multi-source test data.
[0207] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. All equivalent modifications, substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A performance evaluation method for an autonomous driving system based on scene complexity, characterized in that, The specific steps are as follows: Step 1, obtain A driving test sample containing a sequence of images and their corresponding reference images; No. One driving test sample The Middle Frame image is The corresponding reference image is , in =1,2,..., , =1,2,..., ; For test samples The number of frames; Step 2: Calculate the scenario complexity of each driving test sample to obtain a test sample library with scenario complexity. First, regarding the test samples Calculate the frame-level scene complexity of each frame; Then, the test sample The scene complexity of the test sample is obtained by fusing the scene complexity of all frames. ; Finally, a test sample library is compiled from the scenario complexities corresponding to all test samples. ; Step 3: Based on the complexity of the scenario, the test samples are divided into three levels: low complexity, medium complexity, and high complexity. Step 4: At each complexity level, three types of autonomous driving system performance tests are conducted sequentially, including basic simulation tests, high-precision simulation tests, and real-vehicle road tests, to obtain system performance test results from the three test sources at the same complexity level; Step 5: Merge the test results of multiple types under the same complexity level to obtain the system performance evaluation results under different scenario complexity levels; The fusion process includes statistical confidence calculation, source credibility calculation, fusion weight calculation, and weighted Bayesian data fusion.
2. The method as described in claim 1, characterized in that, In step one, the reference image and the sequence image are matched according to timestamp, vehicle pose, road mileage or lane topology to characterize the comparison status of the same road location under standard road, standard weather or labeled conditions.
3. The method as described in claim 1, characterized in that, In step two, the test sample is first calculated. No. The basic complexity of a frame for: For traffic density complexity, For the geometric complexity of the road, Due to the complexity of the weather environment, For dynamic obstacle complexity; , , and These are the weight coefficients for the four complexity dimensions, and ; Then, based on the first The basic complexity of the frame image and the (t-1)th frame image, calculating the degree of change between frames. : Based on the basic complexity and inter-frame variability, the test samples are calculated. No. Frame-level scene complexity of a frame image: Inter-frame variation correction factor ≥0; when When =1, it is stipulated that =0; Test samples Scene complexity for: in, For the first Time weighting of frames.
4. The method as described in claim 3, characterized in that, In step three, the test sample library will be... test samples , ,..., Sort by their respective scene complexity from smallest to largest. , ,..., The complexity threshold is determined using quantiles: in, This serves as the threshold for distinguishing between low and medium complexity. This is the threshold for distinguishing between medium and high complexity.
5. The method as described in claim 4, characterized in that, In step four, the test results include the number of tests. Number of times the standard is met Performance index estimates and its corresponding evidence ,in These represent basic simulation testing, high-precision simulation testing, and real-vehicle road testing, respectively. These represent low, medium, and high complexity levels, respectively.
6. The method as described in claim 5, characterized in that, In step five, regarding the first... Type of test, statistical confidence level The calculation is as follows: in, For the first The test source pre-sets the developer risk or significance level. The smaller the value, the lower the allowable risk of misjudgment from the test source, and the higher the corresponding statistical confidence level. Source credibility The calculation is as follows: in, This is the error attenuation coefficient. ; For the first The source of the test is The average error across a calibration scenario; To calibrate the number of scenes, ; For the source of this test in the Performance estimates in each calibration scenario; Reference performance values for the calibration scenario; Fusion weights The calculation is as follows: In weighted Bayesian data fusion, firstly, the first step is to calculate the... The posterior probability of a test source is obtained based on its own test evidence. as follows: Then, the posterior probabilities obtained from all test sources based on their own test evidence are fused to obtain the posterior probability of system performance meeting the standard. : For "autonomous driving systems at complexity level" The system performance assumption is that it "meets the preset performance target"; at the complexity level The test evidence obtained below is ; Defined as the opposing assumption of "not meeting the preset performance target"; This indicates that the system meets the preset performance targets. Under the conditions, the first Various test sources obtain test evidence The conditional probability; This indicates that the system does not meet the preset performance targets. Under the conditions, the first The same test evidence was obtained from different test sources. The conditional probability.
