Online multi-dimensional comprehensive test evaluation method for automatic driving test

Abstract

The invention relates to an online multi-dimensional comprehensive test evaluation method for an automatic driving test. The method comprises the following steps: constructing a test scene library; constructing a test scene complexity model; building an intelligent driving joint simulation test platform, and embedding the intelligent driving algorithm into the intelligent driving joint simulation test platform; constructing an automatic driving test multi-dimensional evaluation index system for performing multi-dimensional performance evaluation on the automatic driving vehicle; an intelligent driving joint simulation test platform is utilized to obtain vehicle test data, an evaluation index system and a test scene complexity model are combined to calculate objective evaluation index weights, then comprehensive performance quantitative scores are calculated, and finally an automatic evaluation report is output and obtained. Compared with the prior art, the subjectivity of manual evaluation can be reduced, and the evaluation efficiency of the automatic driving vehicle is improved; comprehensive and objective quantitative scoring can be carried out on the multi-dimensional performance of the automatic driving vehicle, and the influence of different test scenes on the scoring difference of the automatic driving vehicle is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving testing, and in particular to an online multi-dimensional comprehensive testing and evaluation method for autonomous driving testing. Background Art

[0002] For the actual application and deployment of autonomous vehicles, the evaluation of driving intelligence is a key challenge. The comprehensive performance of autonomous vehicles is a key factor in determining whether they can be widely deployed and win the public's trust. The comprehensive performance of autonomous vehicles can only be demonstrated when the performance of autonomous vehicles exceeds that of humans in various complex and critical test scenarios. Currently, there are relatively few standards and test methods for evaluating the performance of autonomous vehicles. In order to ensure the safety, comfort and standard compliance of autonomous vehicles in various road environments and conditions, it is necessary to conduct comprehensive tests to address the long-tail problems that arise in the development of intelligent driving systems.

[0003] A key bottleneck in testing and improving the performance of autonomous vehicles is the lack of objective multi-dimensional quantitative evaluation methods for the comprehensive performance of autonomous vehicles. Current methods usually use collision scenarios to test the safety performance of autonomous vehicles. Many researchers have conducted extensive research on scenario construction and key parameter selection. Existing research includes accelerated evaluation methods with collision and injury probability as the main indicators. In addition, there is a series of adversarial testing method studies that attack autonomous vehicles to test their safety. The above methods only focus on testing the safety of autonomous vehicles, but it is difficult to evaluate the comprehensive performance of autonomous vehicles. For example, when other vehicles suddenly cut into the lane of the autonomous vehicle, automatic emergency braking is considered to be effective and will not cause a collision. However, the maximum braking force and braking time greatly affect the perception and comfort of passengers, and also reflect the multi-dimensional performance of autonomous vehicles. The above methods face major challenges in testing and evaluating the comprehensive performance of autonomous vehicles. For the development of autonomous vehicle test and evaluation systems, evaluating driving intelligence also brings new challenges. We need to study how to design a more comprehensive evaluation metric system to quantify the multi-dimensional performance of autonomous vehicles.

[0004] In addition, existing research has paid little attention to the impact of different test scenarios on intelligent driving tests, resulting in significant differences in the evaluation results of the same autonomous driving vehicle in scenarios of different complexity. This difference makes it possible for an autonomous driving vehicle that performs well in simple scenarios to perform poorly in complex scenarios, resulting in two extreme evaluation scores. Therefore, in the testing and evaluation of autonomous driving vehicles, it is necessary to find a way to reduce the impact of different test scenarios on the test results. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an online multi-dimensional comprehensive test and evaluation method for autonomous driving testing, which can comprehensively and objectively quantify the multi-dimensional performance of autonomous driving vehicles and reduce the impact of different test scenarios on the differences in autonomous driving vehicle scores.

[0006] The purpose of the present invention can be achieved by the following technical solution: An online multi-dimensional comprehensive test evaluation method for autonomous driving test, comprising the following steps:

[0007] S1. Build a test scenario library;

[0008] S2. Construct a test scene complexity model, including static scene complexity, dynamic scene complexity and weather environment complexity models;

[0009] S3. Build an intelligent driving joint simulation test platform and embed the intelligent driving algorithm into the intelligent driving joint simulation test platform;

[0010] S4. Build a multi-dimensional evaluation index system for autonomous driving tests to evaluate the multi-dimensional performance of autonomous driving vehicles;

[0011] S5. Use the intelligent driving joint simulation test platform to obtain vehicle test data. Combined with the evaluation index system and the test scenario complexity model, the automated evaluation report is output through evaluation index weight calculation and comprehensive performance quantitative scoring.

[0012] Furthermore, the specific process of step S1 is as follows:

[0013] S11, generalization generation of natural driving test scenarios;

[0014] S12, optimizing scene parameters for static traffic scenes;

[0015] S13, adaptively adjust dynamic traffic scenarios to optimize the behavior of other major traffic participants;

[0016] S14. Output key boundary test scenarios with different complexity.

[0017] Furthermore, the scene complexity model in step S2 is specifically:

[0018] C=α S C S +α D C D +α W C W

[0019] Among them, α S , α D and αW They are the static scene complexity C S , Dynamic scene complexity C D and weather environment complexity C W The weight coefficient of

[0020] The scene complexity model in step S2 only considers the impact of static targets and dynamic traffic participants within the impact area around the autonomous driving vehicle under test on the autonomous driving vehicle under test. The effective range of the impact area around the autonomous driving vehicle is described by a semicircle centered on the center point of the rear axle of the vehicle, and the radius of the semicircle is:

[0021] r range =max{r safe ,2r l}

[0022]

[0023] Among them, r range is the radius of the autonomous vehicle’s influence area, r safe is the radius of the safety zone of the autonomous driving vehicle under test, r l is the length of the intersection area, v i is the initial speed of the autonomous vehicle, τ is the response lag time of the autonomous vehicle, and are the maximum acceleration and minimum deceleration of the autonomous vehicle, respectively.

[0024] Furthermore, the static scene complexity is specifically:

[0025]

[0026] in, represents the scene complexity between the static traffic target i and the autonomous driving vehicle under test, N is the number of static traffic targets in the scene, is the virtual charge of the ith static target, ε0 is the dielectric constant, is the relative distance between the static traffic target i and the autonomous driving vehicle under test, (x i ,y i ) is the center of gravity coordinate of the static traffic target i, (x0, y0) is the center of gravity coordinate of the tested autonomous driving vehicle, and r0 is the equivalent radius of the static traffic target.

[0027] Furthermore, the dynamic scene complexity is specifically:

[0028]

[0029]

[0030] in, is the complexity between the dynamic traffic participant j and the autonomous driving vehicle under test, is the scaling enhancement factor, is the electron energy level coefficient, is the virtual charge of the jth dynamic traffic participant, ∈0 is the dielectric constant, is the distance between the dynamic traffic participant j and the autonomous driving vehicle under test, l j is the difference between the lane number of the tested autonomous vehicle and the lane number of the jth traffic participant. When the tested autonomous vehicle and the jth traffic participant are in the same lane, l j =1,θ j,0 and v j,0 The distribution is the encounter angle and relative speed between the dynamic traffic participant j and the autonomous driving vehicle under test, v j ′ ,0 is the normalized relative velocity, v max is the relative speed in the most complex scenario, v min is the relative speed in the least complex scenario.

[0031] Furthermore, the weather environment complexity is specifically:

[0032]

[0033] Among them, γ v and γ f are the weight coefficients of weather visibility complexity and road friction complexity, P max is the visibility distance when lighting and clarity are optimal, P v is the visibility distance around the tested autonomous driving vehicle, P min F is the visibility distance when the lighting and clarity are the worst. max is the road friction coefficient under the driest conditions, F r is the road friction coefficient of the test scene, F min is the friction coefficient of the road under the most slippery conditions.

[0034] Furthermore, the multi-dimensional evaluation index system of the autonomous driving test in step S4 includes two dimensions, five primary indicators and fourteen secondary indicators, wherein the two dimensions include the vehicle performance dimension and the altruism performance dimension;

[0035] The performance dimension of this vehicle includes four first-level indicators, namely safety, comfort, driving performance and standard compliance;

[0036] The altruism performance dimension includes a first-level indicator, which is the traffic coordination of autonomous vehicles.

[0037] Furthermore, the step S5 specifically includes the following steps:

[0038] S51. Determine whether the currently tested autonomous driving vehicle has passed the safety test based on the vehicle test data. After n simulation tests, if the accident rate of the autonomous driving vehicle is greater than a preset threshold, it indicates that the autonomous driving vehicle is unqualified and the current process ends; otherwise, execute step S52.

[0039] S52, using the evaluation index system, calculating the evaluation index weights to obtain an evaluation index weight matrix;

[0040] S53. Combine the scene complexity model and the evaluation index weight matrix to perform multi-dimensional comprehensive performance quantitative scoring and output an automated evaluation report.

[0041] Furthermore, the specific process of step S52 is as follows:

[0042] S521. Construct an evaluation matrix based on vehicle test data:

[0043]

[0044] Where n is the number of completed intelligent driving tests, m is the number of evaluation indicators, and the test data of the zth evaluation indicator in the kth intelligent driving test is x. kz express;

[0045] S522, for the evaluation matrix X n×m Perform standardization, scale all test data to the minimum type, and use the data of the standardized evaluation matrix To express, the standardized formula is:

[0046]

[0047] Among them, B is the positive evaluation index set, and C is the negative evaluation index set;

[0048] S523, according to the standardized evaluation matrix, the smaller This will result in a larger performance score f k :

[0049]

[0050] S524, calculating the performance score of each intelligent driving test after removing each evaluation index, so that m groups of performance functions are associated with m evaluation indexes, and after removing the zth evaluation index, the performance score f′ of the kth intelligent driving test kz for:

[0051]

[0052] S525. After determining the elimination effect of the zth evaluation index, calculate the sum of absolute deviations, where the elimination effect of the zth evaluation index is expressed as η z :

[0053]

[0054] S526. Calculate the objective weight ω of the zth evaluation index z for:

[0055]

[0056] Among them, ∑ r η r It is the sum of the impacts of all evaluation indicators.

[0057] Furthermore, the specific process of the multi-dimensional comprehensive performance quantitative scoring in step S53 is as follows:

[0058] S531: Based on the test evaluation matrix, construct an ideal autonomous driving vehicle (IAV) and a worst autonomous driving vehicle (AIAV), and calculate the optimal test value IT and the worst test value AIT to obtain an extended evaluation matrix X E :

[0059]

[0060] S532. Standardize the extended evaluation matrix of positive and negative evaluation indicators:

[0061]

[0062] Among them, x kz ∈X E , x tkz ∈X E ;

[0063] S533, Standardize the evaluation matrix and the evaluation index weight ω z Multiply and calculate the weighted evaluation matrix x w :

[0064]

[0065] S534, calculating the utility U between the kth intelligent driving test and the worst test value and the best test value k :

[0066]

[0067] Among them, S k is the weighted evaluation matrix X wThe sum of the values ​​of ;

[0068] S535: Determine the utility function of the kth intelligent driving test, where the utility function is determined based on the relative relationship between the worst test value and the best test value, and the calculation formula is:

[0069]

[0070]

[0071] S536, according to the evaluation index weight ω z and utility function f(U k ), calculate the specific performance score of each evaluation indicator and the comprehensive performance evaluation score s of the tested autonomous driving vehicle:

[0072]

[0073] in, is the weighted coefficient of the standardized scene complexity, C k is the scene complexity of the kth intelligent driving test, C max It is the maximum value of the scene complexity of n intelligent driving tests.

[0074] Compared with the prior art, the present invention has the following advantages:

[0075] The present invention first constructs a test scenario library, then respectively establishes a scenario complexity model, builds a joint simulation test platform, and constructs a multi-dimensional evaluation index system for autonomous driving tests, then uses the intelligent driving joint simulation test platform to obtain vehicle test data, and then combines the evaluation index system and the test scenario complexity model, and outputs an automated evaluation report through evaluation index weight calculation and comprehensive performance quantitative scoring. Among them, the scenario complexity model is a bridge connecting the test scenario and the evaluation index system, which can adaptively scale the evaluation scale of test scenarios of different difficulty levels, thereby reducing the score differences caused by test scenarios of different difficulty levels on the tested autonomous driving vehicles, reducing the subjectivity of manual evaluation, accelerating the evaluation efficiency of autonomous driving vehicles, and also improving the versatility of the evaluation index system and the unbiasedness of the evaluation results.

[0076] When establishing the scene complexity model, the present invention only considers the influence of static targets and dynamic traffic participants in the impact area around the tested autonomous driving vehicle on the tested autonomous driving vehicle. The scene complexity model is designed to consist of the static scene complexity C S , Dynamic scene complexity C D and weather environment complexity C W By using the scene complexity model as a scaling indicator, the difference in scores of the tested autonomous driving vehicles caused by scenes of different complexities can be effectively reduced.

[0077] The present invention constructs a multi-dimensional evaluation index system for autonomous driving tests, specifically defining the evaluation index based on the interaction effect between the tested autonomous driving vehicle and the surrounding environment. On the basis of the existing four vehicle performance evaluation indicators (safety, comfort, driving performance and standard compliance), the concept of altruistic performance is further proposed, which focuses on describing the impact of the behavior of the autonomous driving vehicle on other surrounding traffic participants. In this way, the performance of the intelligent driving vehicle can be comprehensively characterized from two dimensions and five aspects through a hierarchical structure.

[0078] The present invention can minimize human intervention and reduce subjectivity by constructing an evaluation matrix, using a method based on standard removal effect to determine the evaluation index weights, and calculating a comprehensive evaluation score. By defining the best and worst autonomous driving vehicles, the present invention can achieve real-time quantitative scoring of different autonomous driving vehicles and combine it with a scenario complexity model. Based on the defined scenario complexity model, the entire process of intelligent driving testing is closed-loop, thereby achieving adaptive scenario selection and accelerated testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0080] Figure 2 It is the framework of the multi-dimensional integrated testing and evaluation method of the autonomous driving vehicle in the embodiment;

[0081] Figure 3 is a schematic diagram of the scene complexity model;

[0082] Figure 4 It is a schematic diagram of the multi-dimensional evaluation indicator system;

[0083] Figure 5 It is a flowchart of the objective multi-dimensional comprehensive evaluation method;

[0084] Figure 6 Schematic diagram of comparison of simulation test evaluation results of different autonomous driving algorithms in the embodiments. DETAILED DESCRIPTION

[0085] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0086] Example

[0087] like Figure 1 As shown, an online multi-dimensional comprehensive test evaluation method for autonomous driving testing includes the following steps:

[0088] S1. Build a test scenario library;

[0089] S2. Construct a test scene complexity model, including static scene complexity, dynamic scene complexity and weather environment complexity models;

[0090] S3. Build an intelligent driving joint simulation test platform and embed the intelligent driving algorithm into the intelligent driving joint simulation test platform;

[0091] S4. Build a multi-dimensional evaluation index system for autonomous driving tests to evaluate the multi-dimensional performance of autonomous driving vehicles;

[0092] S5. Use the intelligent driving joint simulation test platform to obtain vehicle test data. Combined with the evaluation index system and the test scenario complexity model, the automated evaluation report is output through evaluation index weight calculation and comprehensive performance quantitative scoring.

[0093] This embodiment applies the above technical solution, mainly including:

[0094] In order to comprehensively and objectively evaluate the multi-dimensional performance of autonomous driving vehicles, this embodiment builds a multi-dimensional integrated test system for intelligent driving, which integrates test scenario generation, scenario complexity quantification, intelligent driving algorithm embedding and automatic evaluation method. Figure 2 The figure shows the architecture of the multi-dimensional integrated test and evaluation method for autonomous driving vehicles in this embodiment. The entire automated evaluation system consists of four modules: test scenario library, scenario complexity model, simulation test platform and automated evaluation system.

[0095] 1. Build a test scenario library

[0096] Test scenarios are the basis of intelligent driving tests, providing different simulation test scenarios for intelligent driving tests. These test scenarios are constructed by data analysis and feature extraction of standard regulations, traffic accident data, natural driving data, and expert experience. The specific process includes:

[0097] S11, generalization generation of natural driving test scenarios;

[0098] S12. Optimize scene parameters for static traffic scenes to improve the diversity and complexity of static scenes;

[0099] S13, adaptively adjust dynamic traffic scenarios to optimize the behavior of other major traffic participants;

[0100] S14. Output key boundary test scenarios with different complexity.

[0101] This embodiment optimizes the scene parameters and the behavior of dynamic traffic participants from static and dynamic scenes to generate test scenes of different complexity. The generated static scene elements include static road structure, static obstacles, weather visibility and road friction coefficient. In addition, this embodiment also fully considers the impact of mixed traffic flows consisting of autonomous vehicles, human-driven vehicles, bicycles and pedestrians on the autonomous vehicle under test.

[0102] 2. Building a test scenario complexity model

[0103] The scene complexity model in this embodiment is based on potential field theory. Objects in the scene related to the autonomous driving vehicle under test are abstracted as uniformly charged wires or positive point charges. According to field theory, objects in the scene will generate an electric potential field in the regional space that is inversely proportional to the distance, and the potential field is determined by the category of the object. The overall environmental potential field can capture the complexity of the scene by combining and superimposing different potential fields. The overall architecture of the scene complexity model is as follows: Figure 3 shown.

[0104] The proposed scene complexity model consists of the static scene complexity C S , Dynamic scene complexity C D and weather environment complexity C W The specific calculation method of the scene complexity model is:

[0105] C=α S C S +α D C D +α W C W

[0106] Among them, α S , α D and α W They are the static scene complexity C S , Dynamic scene complexity C D and weather environment complexity C W The weight coefficient of .

[0107] The scene complexity model only considers the impact of static targets and dynamic traffic participants within the impact area around the tested autonomous vehicle on the tested autonomous vehicle. The effective range of the impact area around the autonomous vehicle is described by a semicircle centered on the center point of the rear axle of the vehicle. The radius of the semicircle is:

[0108] r range =max{r safe ,2r l}

[0109]

[0110] Among them, r range is the radius of the autonomous vehicle’s influence area, r safe is the radius of the safety zone of the autonomous driving vehicle under test, r l is the length of the intersection area, v i is the initial speed of the autonomous vehicle, τ is the response lag time of the autonomous vehicle, and are the maximum acceleration and minimum deceleration of the autonomous vehicle, respectively;

[0111] The complexity of static scenes is as follows:

[0112]

[0113] in, represents the scene complexity between the static traffic target i and the autonomous driving vehicle under test, N is the number of static traffic targets in the scene, is the virtual charge of the ith static target, ε0 is the dielectric constant, is the relative distance between the static traffic target i and the autonomous driving vehicle under test, (x i ,y i ) is the center of gravity coordinate of the static traffic target i, (x0, y0) is the center of gravity coordinate of the tested autonomous driving vehicle, and r0 is the equivalent radius of the static traffic target.

[0114] The dynamic scene complexity is as follows:

[0115]

[0116] in, is the complexity between the dynamic traffic participant j and the autonomous driving vehicle under test, is the scaling enhancement factor, is the electron energy level coefficient, is the virtual charge of the jth dynamic traffic participant, ∈0 is the dielectric constant, is the distance between the dynamic traffic participant j and the autonomous driving vehicle under test, l j is the difference between the lane number of the tested autonomous vehicle and the lane number of the jth traffic participant. When the tested autonomous vehicle and the jth traffic participant are in the same lane, l j =1. θ j,0 and v j,0 The distribution is the encounter angle and relative speed between the dynamic traffic participant j and the autonomous driving vehicle under test, v j ′ ,0 is the normalized relative velocity, v max is the relative speed in the most complex scenario, vmin is the relative speed in the least complex scenario.

[0117] The complexity of the weather environment is as follows:

[0118]

[0119] Among them, γ v and γ f are the weight coefficients of weather visibility complexity and road friction complexity, P max is the visibility distance when lighting and clarity are optimal, P v is the visibility distance around the tested autonomous driving vehicle, P min F is the visibility distance when the lighting and clarity are the worst. max is the road friction coefficient under the driest conditions, F r is the road friction coefficient of the test scene, F min is the friction coefficient of the road under the most slippery conditions.

[0120] 3. Build an intelligent driving joint simulation test platform

[0121] In order to meet the intelligent driving test requirements in various scenarios, this embodiment develops a joint simulation platform tool chain consisting of virtual simulation test software (VTD), Matlab / Simulink and CarSim. VTD provides intelligent driving simulation for complex traffic scenarios, including road network modeling, traffic scene modeling, weather and environment simulation, and high-fidelity image rendering. CarSim is responsible for realizing high-precision vehicle dynamics simulation. Simulink supports embedding different intelligent driving algorithms and simulating them. Therefore, it is necessary to realize real-time interaction of vehicle dynamics in CarSim and VTD. VTD sends the environment and position information perceived by the vehicle to CarSim for vehicle dynamics calculation, and CarSim feeds back the obtained dynamic state information to VTD. At the same time, through the intelligent driving algorithm written in Simulink, the vehicle in VTD can realize the corresponding automatic driving task. On this joint simulation platform, the driving performance of the intelligent driving algorithm is tested. The simulation test is carried out at the algorithm level, which is conducive to optimizing and improving the development and testing efficiency of the intelligent driving algorithm.

[0122] 4. Build a multi-dimensional evaluation index system for autonomous driving tests

[0123] The multi-dimensional evaluation index system aims to comprehensively characterize the intelligence level of autonomous driving vehicles and realize automated evaluation. Therefore, this scheme proposes a joint evaluation index system composed of multi-dimensional evaluation indicators. The proposed evaluation system defines the evaluation indicators according to the interaction effect between the tested autonomous driving vehicle and the surrounding environment.

[0124] Based on the existing four vehicle performance evaluation indicators, the concept of altruism performance is further proposed. Altruism performance focuses on describing the impact of the behavior of the autonomous driving vehicle on other traffic participants around it. This embodiment uses the traffic coordination index to characterize altruism performance. This evaluation index system comprehensively describes the performance of intelligent driving vehicles from two dimensions and five aspects.

[0125] In the evaluation index system, safety is evaluated by proxy safety indicators. Three commonly used driving safety indicators are selected: time to collision (TTC), collision exposure time (TET) and post-intrusion time (PET). The comfort index focuses on the feelings of passengers during driving, including irregular driving, sudden braking and sharp turns caused by defects in the intelligent driving system. The maximum acceleration, maximum acceleration change rate, maximum heading angle change rate and speed standard deviation are used to represent driving comfort. Driving performance describes the task completion quality and task completion efficiency of the autonomous driving vehicle under the corresponding operation design domain. The driving performance is described by the task completion time, the maximum offset to the center of the lane, the average speed and the distance when the autonomous vehicle first detects an obstacle. Standard compliance focuses on the degree of compliance of the tested autonomous vehicle with traffic regulations. It is mainly used to describe whether the autonomous vehicle violates traffic regulations, such as speeding. In the altruism performance dimension, the traffic coordination index is used to measure the impact of the behavior of the autonomous vehicle on other vehicles and the overall traffic flow. The maximum deceleration of surrounding vehicles and the average speed of surrounding vehicles are used to represent traffic coordination. Such as Figure 4 As shown in the figure (including two dimensions, five first-level indicators and fourteen second-level indicators, two of which include the vehicle performance dimension and the altruism performance dimension; the vehicle performance dimension includes four first-level indicators, namely safety, comfort, driving performance and standard specification; the altruism performance dimension includes one first-level indicator, which is the traffic coordination of the autonomous driving vehicle), the above two dimensions and five evaluation indicators constitute the evaluation index system of the autonomous driving vehicle through a hierarchical structure.

[0126] 5. Automated Evaluation System

[0127] The overall process of the automated evaluation system is as follows: Figure 5As shown. First, the test scenarios are automatically extracted from the test scenario library, and then the autonomous driving vehicle is tested on the simulation test platform. Then, it is determined whether the autonomous driving vehicle has passed the test scenario without a collision. Assume that the autonomous driving vehicle completes all n test scenarios without a collision. In this case, a comprehensive performance evaluation will be conducted, including constructing an evaluation matrix, determining the evaluation indicator weights through a standard removal effect-based method, and calculating a comprehensive evaluation score. If a collision occurs during the test of the autonomous driving vehicle, the number of collisions j is calculated. If the accident rate of the autonomous driving vehicle is greater than 10%, the autonomous driving vehicle is considered unqualified and no further scoring is required. If the accident rate of the autonomous driving vehicle is less than 10%, the comprehensive performance evaluation continues and the evaluation score is calculated.

[0128] The weights of the evaluation indicators are calculated using a method based on standard removal effect. The method based on standard removal effect is an objective weight calculation method. When calculating the weights of the evaluation indicators, it is necessary to comprehensively study the impact of each evaluation indicator on the overall performance of the intelligent driving test. The method for determining the objective weight is as follows:

[0129] First, an evaluation matrix is ​​constructed based on the test data obtained from the intelligent driving test:

[0130]

[0131] Where n is the number of completed intelligent driving tests, m is the number of evaluation indicators, and the test data of the zth evaluation indicator in the kth intelligent driving test is x. kz express;

[0132] For the evaluation matrix X n×m Perform normalization and scale all test data to the minimum type. The data of the normalized evaluation matrix is ​​used The standardized formula can be expressed as follows:

[0133]

[0134] Among them, B is the positive evaluation indicator set, and C is the negative evaluation indicator set.

[0135] The overall performance of each intelligent driving test is calculated using the logarithmic measurement method with equal weights of the evaluation indicators. According to the standardized evaluation matrix, the smaller This will result in a larger performance score f k , and its calculation formula is:

[0136]

[0137] Then, the performance score of each intelligent driving test after removing each evaluation index is calculated. In this way, there are m groups of performance functions associated with m evaluation indexes. After removing the zth evaluation index, the performance score f′ of the kth intelligent driving test is kz It can be calculated as:

[0138]

[0139] After determining the elimination effect of the zth evaluation indicator, the sum of absolute deviations is calculated. The elimination effect of the zth evaluation indicator can be expressed as η z :

[0140]

[0141] The objective weight of the zth evaluation indicator ∈ z It can be calculated by the following formula:

[0142]

[0143] The specific calculation process of the multi-dimensional comprehensive performance quantitative score is as follows:

[0144] Based on the test evaluation matrix, the ideal autonomous driving vehicle (IAV) and the worst autonomous driving vehicle (AIAV) are constructed, and the optimal test value (IT) and the worst test value (AIT) are further calculated to obtain the extended evaluation matrix X E :

[0145]

[0146] Normalize the extended evaluation matrix for both positive and negative evaluation metrics:

[0147]

[0148] Among them, x kz ∈X E , x tkz ∈X E ;

[0149] By using a standardized assessment matrix and the evaluation index weight ω z Multiply to calculate the weighted evaluation matrix X w :

[0150]

[0151] The utility U between the kth intelligent driving test and the worst test value and the best test value k The calculation formula is:

[0152]

[0153] Among them, S k is the weighted evaluation matrix X w The sum of the values ​​of is calculated as:

[0154]

[0155] Then, the utility function of the kth intelligent driving test is determined. The utility function is determined based on the relative relationship with the worst test value and the best test value, and its calculation formula is:

[0156]

[0157]

[0158] According to the evaluation index weight ω z and utility function f(U k ), the specific performance score of each evaluation indicator and the comprehensive performance evaluation score s of the tested autonomous driving vehicle can be calculated:

[0159]

[0160] in, is the weighted coefficient of the standardized scene complexity, C k is the scene complexity of the kth intelligent driving test, C max It is the maximum value of the scene complexity of n intelligent driving tests.

[0161] The most important part of this technical solution is to conduct a comprehensive and objective quantitative evaluation of the performance of the autonomous driving vehicle. In order to prove the effectiveness of this technical solution, this embodiment uses the VTD autonomous driving model developed by VIRES and the rule-based DPA autonomous driving model for simulation verification. The VTD autonomous driving model is equivalent to a black box, which is used to represent a type of black box intelligent driving algorithm. DPA is an intelligent driving algorithm that integrates mixed flow intersection decision-making, path planning and real-time control.

[0162] Figure 6 The evaluation scores and comprehensive scores of the VTD autonomous driving model and the DPA intelligent driving algorithm in different test scenarios are shown. As can be seen from the figure, the evaluation score of the DPA autonomous driving model is significantly higher than that of the VTD autonomous driving model. This is mainly because the rule-based DPA intelligent driving algorithm is relatively conservative and aims to prevent collisions. In addition, combining the comprehensive score with the complexity of the test scenario can effectively reduce the impact of different test scenarios on the inconsistency of test results, thereby effectively improving the objectivity and effectiveness of the evaluation results, and ensuring the unbiased performance evaluation of different autonomous driving vehicles.

[0163] In summary, the goal of this technical solution is to study a four-layer automated evaluation architecture for autonomous vehicles to improve the objectivity and unbiasedness of the test. First, the automated evaluation system can generate test scenarios of different complexity for comprehensive autonomous vehicle testing. Second, the potential field method is used to quantify the complexity of the scenario to help evaluate various driving challenges. Third, the joint simulation platform combines VTD, Matlab / Simulink and CarSim to achieve accurate scenario simulation and high-precision vehicle dynamics modeling. Fourth, the automated evaluation system uses multi-dimensional indicators and evaluation algorithms to quantitatively evaluate the comprehensive multi-dimensional performance of autonomous vehicles, reduce manual participation and accelerate evaluation efficiency. When this solution is applied to practice, users can automatically select test scenarios, automatically perform simulation tests and data collection, automatically process and analyze test data, and calculate test results online. The subjectivity of manual evaluation is greatly reduced and the evaluation process is accelerated. In addition, the evaluation indicator system is also flexible and scalable, and can integrate more evaluation dimensions and indicators in the future.

Claims

1. An online multi-dimensional comprehensive test evaluation method for autonomous driving testing, characterized in that: The following steps are involved: S1. Build a test scenario library; S2. Construct a test scene complexity model, including static scene complexity, dynamic scene complexity and weather environment complexity models; S3. Build an intelligent driving joint simulation test platform and embed the intelligent driving algorithm into the intelligent driving joint simulation test platform; S4. Build a multi-dimensional evaluation index system for autonomous driving tests to evaluate the multi-dimensional performance of autonomous driving vehicles; S5. Use the intelligent driving joint simulation test platform to obtain vehicle test data. Combined with the evaluation index system and the test scenario complexity model, the automated evaluation report is output through evaluation index weight calculation and comprehensive performance quantitative scoring.

2. The online multi-dimensional comprehensive test evaluation method for autonomous driving test according to claim 1, characterized in that: The specific process of step S1 is as follows: S11, generalization generation of natural driving test scenarios; S12, optimizing scene parameters for static traffic scenes; S13, adaptively adjust dynamic traffic scenarios to optimize the behavior of other major traffic participants; S14. Output key boundary test scenarios with different complexity.

3. The online multi-dimensional comprehensive test evaluation method for autonomous driving test according to claim 1, characterized in that: The scene complexity model in step S2 is specifically: C=a S C S +a D C D +a W C W Among them, α S , α D and α W They are the static scene complexity C S , Dynamic scene complexity C D and weather environment complexity C W The weight coefficient of The scene complexity model in step S2 only considers the impact of static targets and dynamic traffic participants within the impact area around the autonomous driving vehicle under test on the autonomous driving vehicle under test. The effective range of the impact area around the autonomous driving vehicle is described by a semicircle centered on the center point of the rear axle of the vehicle, and the radius of the semicircle is: r range =max{r safe ,2r l } Among them, r range is the radius of the autonomous vehicle’s influence area, r safe is the radius of the safety zone of the autonomous driving vehicle under test, r l is the length of the intersection area, v i is the initial speed of the autonomous vehicle, τ is the response lag time of the autonomous vehicle, and are the maximum acceleration and minimum deceleration of the autonomous vehicle, respectively.

4. The online multi-dimensional comprehensive test evaluation method for autonomous driving test according to claim 3, characterized in that: The static scene complexity is specifically: in, represents the scene complexity between the static traffic target i and the autonomous driving vehicle under test, N is the number of static traffic targets in the scene, is the virtual charge of the ith static target, ε0 is the dielectric constant, is the relative distance between the static traffic target i and the autonomous driving vehicle under test, (x i ,y i ) is the center of gravity coordinate of the static traffic target i, (x0, y0) is the center of gravity coordinate of the tested autonomous driving vehicle, and r0 is the equivalent radius of the static traffic target.

5. The online multi-dimensional comprehensive test evaluation method for autonomous driving test according to claim 4, characterized in that: The dynamic scene complexity is specifically: in, is the complexity between the dynamic traffic participant j and the autonomous driving vehicle under test, is the scaling enhancement factor, is the electron energy level coefficient, is the virtual charge of the jth dynamic traffic participant, ∈0 is the dielectric constant, is the distance between the dynamic traffic participant j and the autonomous driving vehicle under test, l j is the difference between the lane number of the tested autonomous vehicle and the lane number of the jth traffic participant. When the tested autonomous vehicle and the jth traffic participant are in the same lane, l j =1,θ j,0 and v j,0 The distribution is the encounter angle and relative speed between the dynamic traffic participant j and the autonomous driving vehicle under test, v j ′ ,0 is the normalized relative velocity, v max is the relative speed in the most complex scenario, v min is the relative speed in the least complex scenario.

6. The online multi-dimensional comprehensive test evaluation method for autonomous driving test according to claim 5, characterized in that: The complexity of the weather environment is specifically: Among them, γ v and γ f are the weight coefficients of weather visibility complexity and road friction complexity, P max is the visibility distance when lighting and clarity are optimal, P v is the visibility distance around the tested autonomous driving vehicle, P min F is the visibility distance when the lighting and clarity are the worst. max is the road friction coefficient under the driest conditions, F r is the road friction coefficient of the test scene, F min is the friction coefficient of the road under the most slippery conditions.

7. The online multi-dimensional comprehensive test evaluation method for autonomous driving test according to claim 1, characterized in that: The multi-dimensional evaluation index system for the autonomous driving test in step S4 includes two dimensions, five primary indicators and fourteen secondary indicators, wherein the two dimensions include the vehicle performance dimension and the altruism performance dimension; The performance dimension of this vehicle includes four first-level indicators, namely safety, comfort, driving performance and standard compliance; The altruism performance dimension includes a first-level indicator, which is the traffic coordination of autonomous vehicles.

8. The online multi-dimensional comprehensive test evaluation method for autonomous driving test according to claim 1, characterized in that: The step S5 specifically comprises the following steps: S51. Determine whether the currently tested autonomous driving vehicle has passed the safety test based on the vehicle test data. After n simulation tests, if the accident rate of the autonomous driving vehicle is greater than a preset threshold, it indicates that the autonomous driving vehicle is unqualified and the current process ends; otherwise, execute step S52. S52, using the evaluation index system, calculating the evaluation index weights to obtain an evaluation index weight matrix; S53. Combine the scene complexity model and the evaluation index weight matrix to perform multi-dimensional comprehensive performance quantitative scoring and output an automated evaluation report.

9. The online multi-dimensional comprehensive test evaluation method for autonomous driving test according to claim 8, characterized in that: The specific process of step S52 is as follows: S521. Construct an evaluation matrix based on vehicle test data: Where n is the number of completed intelligent driving tests, m is the number of evaluation indicators, and the test data of the zth evaluation indicator in the kth intelligent driving test is x. kz express; S522, for the evaluation matrix X n×m Perform standardization, scale all test data to the minimum type, and use the data of the standardized evaluation matrix To express, the standardized formula is: Among them, B is the positive evaluation index set, and C is the negative evaluation index set; S523, according to the standardized evaluation matrix, the smaller This will result in a larger performance score f k : S524, calculating the performance score of each intelligent driving test after removing each evaluation index, so that m groups of performance functions are associated with m evaluation indexes, and after removing the zth evaluation index, the performance score f′ of the kth intelligent driving test kz for: S525. After determining the elimination effect of the zth evaluation index, calculate the sum of absolute deviations, where the elimination effect of the zth evaluation index is expressed as η z : S526. Calculate the objective weight ω of the zth evaluation index z for: Among them, ∑ r η r It is the sum of the impacts of all evaluation indicators.

10. The online multi-dimensional comprehensive test evaluation method for autonomous driving test according to claim 9, characterized in that: The specific process of multi-dimensional comprehensive performance quantitative scoring in step S53 is as follows: S531: Based on the test evaluation matrix, construct an ideal autonomous driving vehicle (IAV) and a worst autonomous driving vehicle (AIAV), and calculate the optimal test value IT and the worst test value AIT to obtain an extended evaluation matrix X E : S532. Standardize the extended evaluation matrix of positive and negative evaluation indicators: Among them, x kz ∈X E , x tkz ∈X E ; S533, Standardize the evaluation matrix and the evaluation index weight ω z Multiply and calculate the weighted evaluation matrix x w : S534, calculating the utility U between the kth intelligent driving test and the worst test value and the best test value k : Among them, S k is the weighted evaluation matrix X w The sum of the values ​​of ; S535: Determine the utility function of the kth intelligent driving test, where the utility function is determined based on the relative relationship between the worst test value and the best test value, and the calculation formula is: S536, according to the evaluation index weight ω z and utility function f(U k ), calculate the specific performance score of each evaluation indicator and the comprehensive performance evaluation score s of the tested autonomous driving vehicle: in, is the weighted coefficient of the standardized scene complexity, C k is the scene complexity of the kth intelligent driving test, C max It is the maximum value of the scene complexity of n intelligent driving tests.

Images