An online multi-dimension comprehensive test evaluation method for automatic driving test

Abstract

The application relates to an online multidimensional comprehensive test evaluation method for automatic driving test, which comprises the following steps: constructing a test scene library; constructing a test scene complexity model; building an intelligent driving joint simulation test platform and embedding an intelligent driving algorithm into the intelligent driving joint simulation test platform; constructing an automatic driving test multidimensional evaluation index system for multidimensional performance evaluation of an automatic driving vehicle; obtaining vehicle test data by using the intelligent driving joint simulation test platform, combining the evaluation index system and the test scene complexity model, calculating objective evaluation index weights, then calculating comprehensive performance quantitative scores, and finally outputting an automatic evaluation report. Compared with the prior art, the application can reduce the subjectivity of manual evaluation, accelerate the evaluation efficiency of the automatic driving vehicle, comprehensively and objectively quantitatively score the multidimensional performance of the automatic driving vehicle, and reduce the influence of different test scenes on the score difference of the automatic driving vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving test, and in particular to an online multi-dimensional comprehensive test evaluation method for automatic driving test. BACKGROUND

[0002] For the practical application deployment of autonomous vehicles, the evaluation of driving intelligence is a key challenge. The comprehensive performance of autonomous vehicles is a key factor in determining whether they can be widely deployed and win public trust. Only when the performance of autonomous vehicles exceeds that of humans in various complex key test scenarios can their comprehensive performance be proven. There are relatively few standards and test methods for evaluating the performance of autonomous vehicles at present. In order to ensure the safety, comfort and standard compliance of autonomous vehicles in various road environments and conditions, it is necessary to conduct comprehensive tests to address the long-tail problems that arise during the development of intelligent driving systems.

[0003] A key bottleneck in testing and improving the performance of autonomous vehicles is the lack of objective multi-dimensional quantitative evaluation methods for the comprehensive performance of autonomous vehicles. Current methods typically use collision scenarios to test the safety performance of autonomous vehicles. Many researchers have conducted extensive research on scenario construction and key parameter selection. Existing research includes acceleration evaluation methods that focus on collision and injury probability. In addition, there is a series of adversarial testing methods that attack autonomous vehicles to test their safety. The above methods only focus on testing the safety of autonomous vehicles, but it is difficult to evaluate the comprehensive performance of autonomous vehicles. For example, when other vehicles suddenly cut into the lane of an autonomous vehicle, the automatic emergency braking is considered effective and does not result in a collision. However, the maximum braking force and braking time greatly affect the perception and comfort of passengers, also reflecting the multi-dimensional performance of autonomous vehicles. The above methods face major challenges in testing and evaluating the comprehensive performance of autonomous vehicles. The evaluation of driving intelligence also presents new challenges for the development of autonomous vehicle testing and evaluation systems. We need to study how to design a more comprehensive evaluation metric system to quantify the multi-dimensional performance of autonomous vehicles.

[0004] In addition, existing research pays less attention to the impact of different test scenarios on intelligent driving tests, resulting in significant differences in evaluation results for the same autonomous vehicle in different complexity scenarios. This difference makes autonomous vehicles that perform well in simple scenarios may not perform well in complex scenarios, resulting in two extreme evaluation scores. Therefore, in the testing and evaluation of autonomous vehicles, it is necessary to find a way to reduce the impact of different test scenarios on test results. SUMMARY

[0005] The present application aims to overcome the defects of the prior art and provides an online multi-dimensional comprehensive test evaluation method for automatic driving test, which can comprehensively and objectively quantify the multi-dimensional performance of the automatic driving vehicle and reduce the influence of different test scenes on the scoring difference of the automatic driving vehicle.

[0006] The purpose of the present application can be achieved by the following technical solutions: an online multi-dimensional comprehensive test evaluation method for automatic driving test, comprising the following steps:

[0007] S1, constructing a test scene library;

[0008] S2, constructing a test scene complexity model, including static scene complexity, dynamic scene complexity and weather environment complexity model;

[0009] S3, building an intelligent driving joint simulation test platform, and embedding the intelligent driving algorithm into the intelligent driving joint simulation test platform;

[0010] S4, constructing an automatic driving test multi-dimensional evaluation index system for multi-dimensional performance evaluation of the automatic driving vehicle;

[0011] S5, using the intelligent driving joint simulation test platform to obtain vehicle test data, combining the evaluation index system and the test scene complexity model, and through evaluation index weight calculation and comprehensive performance quantitative scoring, outputting an automatic evaluation report.

[0012] Further, the specific process of step S1 is:

[0013] S11, generating natural driving test scenes in generalization;

[0014] S12, optimizing the scene parameters of the static traffic scene;

[0015] S13, self-adaptive adjustment of the dynamic traffic scene, optimizing the behavior of other traffic participants;

[0016] S14, outputting key boundary test scenes with different complexity.

[0017] Further, the scene complexity model in step S2 is specifically:

[0018] C=α S C S +α D C D +α W C W

[0019] Wherein, α S , α D and αW are weight coefficients of static scene complexity C S , dynamic scene complexity C D and weather environment complexity C W , respectively.

[0020] The scene complexity model in the step S2 only considers the influence of static targets and dynamic traffic participants in the surrounding influence area of the measured autonomous vehicle on the measured autonomous vehicle, and the effective range of the surrounding influence area of the autonomous vehicle is described as a semicircle with the center point of the rear axle of the vehicle as the center, and the radius of the semicircle is:

[0021] r range =max{r safe ,2r l}

[0022]

[0023] wherein r range is the influence area radius of the autonomous vehicle, r safe is the safety zone radius of the measured autonomous vehicle, r l is the length of the intersection area range, v i is the initial speed of the autonomous vehicle, τ is the response lag time of the autonomous vehicle, and are the maximum acceleration and the minimum deceleration of the autonomous vehicle, respectively.

[0024] Further, the static scene complexity is specifically:

[0025]

[0026] wherein C represents the scene complexity between the static traffic target i and the measured autonomous vehicle, N is the number of static traffic targets in the scene, is the virtual electric quantity of the i-th static target, ε0 is the dielectric constant, is the relative distance between the static traffic target i and the measured autonomous vehicle, (x i ,y i ) is the barycentric coordinate of the static traffic target i, (x0,y0) is the barycentric coordinate of the measured autonomous vehicle, and r0 is the equivalent radius of the static traffic target.

[0027] Further, the dynamic scene complexity is specifically:

[0028]

[0029]

[0030] wherein, is the complexity between the dynamic traffic participant j and the tested autonomous vehicle, is the scaling enhancement factor, is the electronic energy level coefficient, is the virtual electric quantity of the jth dynamic traffic participant, ∈0 is the dielectric constant, is the distance between the dynamic traffic participant j and the tested autonomous vehicle, l j is the difference between the lane number where the tested autonomous vehicle is located and the lane number where the jth traffic participant is located, when the tested autonomous vehicle and the jth traffic participant are located in the same lane, l j = 1, θ j,0 and v j,0 is the encounter angle and relative speed between the dynamic traffic participant j and the tested autonomous vehicle, v j ′ ,0 is the normalized relative speed, v max is the relative speed under the most complex scenario, v min is the relative speed under the least complex scenario.

[0031] Further, the weather environment complexity is specifically:

[0032]

[0033] wherein, γ v and γ f are the weight coefficients of the weather visibility complexity and the road friction complexity, P max is the visibility distance when the lighting and clarity are best, P v is the visibility distance around the tested autonomous vehicle, P min is the visibility distance when the lighting and clarity are worst, F max is the road friction coefficient under the driest condition, F r is the road friction coefficient of the test scenario, F min is the road friction coefficient under the slipperiest condition.

[0034] Further, the automatic driving test multi-dimensional evaluation index system in the step S4 includes two dimensions, five first-level indexes and fourteen second-level indexes, wherein the two dimensions include the ego performance dimension and the altruistic performance dimension;

[0035] The ego performance dimension includes four first-level indexes, which are safety, comfort, driving performance and standard normativity.

[0036] The altruistic performance dimension includes one first-level index, which is the traffic coordination of the autonomous vehicle.

[0037] Further, the step S5 specifically comprises the following steps:

[0038] S51, according to the vehicle test data, determine whether the current tested automatic driving vehicle passes the safety test, after n times of simulation test, if the accident rate of the automatic driving vehicle is greater than the preset threshold, it indicates that the automatic driving vehicle is unqualified, end the current process; otherwise, execute step S52;

[0039] S52, using the evaluation index system, perform evaluation index weight calculation to obtain an evaluation index weight matrix;

[0040] S53, combine the scene complexity model and the evaluation index weight matrix to perform multi-dimensional comprehensive performance quantification scoring, and output an automatic evaluation report.

[0041] Further, the specific process of the step S52 is as follows:

[0042] S521, based on the vehicle test data, construct an evaluation matrix:

[0043]

[0044] Wherein, n is the number of completed intelligent driving tests, m is the number of evaluation indexes, and the test data of the zth evaluation index in the kth intelligent driving test is represented by x kz ;

[0045] S522, standardize the evaluation matrix X n×m , scale all test data to the minimum type, and represent the data of the standardized evaluation matrix by , and the standardization formula is:

[0046]

[0047] Wherein, B is a set of positive evaluation indexes, and C is a set of negative evaluation indexes;

[0048] S523, according to the standardized evaluation matrix, smaller will lead to larger performance score f k :

[0049]

[0050] S524, calculate the performance score of each intelligent driving test after removing each evaluation index, so that m groups of performance functions are associated with m evaluation indexes, and the performance score f′ kz of the kth intelligent driving test after removing the zth evaluation index is:

[0051]

[0052] S525、In the case of determining the elimination effect of the zth evaluation index, calculate the sum of absolute deviations, wherein the elimination effect of the zth evaluation index is denoted as η z :

[0053]

[0054] S526、Calculate the objective weight ω z of the zth evaluation index:

[0055]

[0056] wherein ∑ r η r is the sum of the effects of all evaluation indexes.

[0057] Further, the specific process of the multi-dimensional comprehensive performance quantification score in the step S53 is:

[0058] S531、Based on the test evaluation matrix, construct the ideal automatic driving vehicle (IAV) and the worst automatic driving vehicle (AIAV), and calculate the optimal test value IT and the worst test value AIT, to obtain the extended evaluation matrix X E :

[0059]

[0060] S532、Standardize the extended evaluation matrix of the positive and negative evaluation indexes:

[0061]

[0062] wherein x kz ∈X E , x tkz ∈X E ;

[0063] S533、Multiply the standardized evaluation matrix by the evaluation index weight ω z to calculate the weighted evaluation matrix x w :

[0064]

[0065] S534、Calculate the utility degree U k between the kth intelligent driving test and the worst test value and the optimal test value:

[0066]

[0067] wherein S k is the weighted evaluation matrix X wThe sum of the values of the first and second evaluation indicators.

[0068] S535, determine the utility function of the kth intelligent driving test, which is determined according to the relative relationship with the worst test value and the optimal test value, and the calculation formula is:

[0069]

[0070]

[0071] S536, according to the evaluation index weight ω z and the utility function f(U k ), the specific performance score of each evaluation index and the comprehensive performance evaluation score s of the measured autonomous vehicle are calculated:

[0072]

[0073] Wherein, is the standardized scene complexity weighting coefficient, C k is the scene complexity of the kth intelligent driving test, C max is the maximum scene complexity of n intelligent driving tests.

[0074] Compared with the prior art, the present application has the following advantages:

[0075] The present application firstly constructs a test scene library, then respectively establishes a scene complexity model, builds a joint simulation test platform and constructs an automatic driving test multi-dimensional evaluation index system, then obtains vehicle test data by using the intelligent driving joint simulation test platform, and then combines the evaluation index system and the test scene complexity model, calculates the evaluation index weight and the comprehensive performance quantitative score, and outputs the automatic evaluation report. The scene complexity model is a bridge connecting the test scene and the evaluation index system, which can adaptively scale the evaluation scale of different difficulty test scenes, thereby reducing the score difference caused by different difficulty test scenes to the measured autonomous vehicle, reducing the subjectivity of manual evaluation, speeding up the evaluation efficiency of autonomous vehicles, and improving the universality of the evaluation index system and the unbiasedness of the evaluation result.

[0076] When the scene complexity model is established, only the influence of static targets and dynamic traffic participants in the influence area around the measured autonomous vehicle on the measured autonomous vehicle is considered, and the scene complexity model is designed to consist of static scene complexity C S , dynamic scene complexity C D and weather environment complexity C W , so that the scene complexity model is used as a scaling index, which can effectively reduce the score difference caused by different complexity scenes to the measured autonomous vehicle.

[0077] The application constructs an automatic driving test multi-dimensional evaluation index system, specifically defines evaluation indexes according to the interaction effect of the tested automatic driving vehicle and the surrounding environment. On the basis of the existing four vehicle performance evaluation indexes (safety, comfort, driving performance and standard normativity), the concept of altruistic performance is further proposed, which focuses on describing the influence of the behavior of the automatic driving vehicle on other traffic participants. Thus, the performance of the intelligent driving vehicle can be fully described from two dimensions and five aspects through a hierarchical structure.

[0078] The application can maximize the reduction of artificial intervention and subjectivity by constructing an evaluation matrix, determining the evaluation index weight by using a method based on standard removal effect and calculating the comprehensive evaluation score. The real-time quantitative score of different automatic driving vehicles is realized by defining the best automatic driving vehicle and the worst automatic driving vehicle, and the score is combined with the scene complexity model. Based on the defined scene complexity model, the closed loop of the whole intelligent driving test process is realized, so as to realize adaptive scene selection and accelerated testing. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 It is a method flowchart of the application;

[0080] Figure 2 It is the architecture of the multi-dimensional integrated test and evaluation method of the automatic driving vehicle in the embodiment;

[0081] Figure 3 It is a schematic diagram of the scene complexity model;

[0082] Figure 4 It is a schematic diagram of the multi-dimensional evaluation index system;

[0083] Figure 5 It is a flowchart of the objective multi-dimensional comprehensive evaluation method;

[0084] Figure 6 It is a comparison schematic diagram of the simulation test evaluation results of different automatic driving algorithms in the embodiment. DETAILED DESCRIPTION

[0085] The application will be described in detail below in combination with the drawings and specific embodiments.

[0086] EMBODIMENT

[0087] As shown in the figure, an online multi-dimensional comprehensive test and evaluation method for automatic driving test includes the following steps: Figure 1

[0088] S1, construct a test scene library;

[0089] ​S2, a test scene complexity model is constructed, including a static scene complexity, a dynamic scene complexity and a weather environment complexity model;

[0090] S3, an intelligent driving joint simulation test platform is built, and an intelligent driving algorithm is embedded into the intelligent driving joint simulation test platform;

[0091] S4, an automatic driving test multi-dimensional evaluation index system is constructed, which is used for multi-dimensional performance evaluation of the automatic driving vehicle;

[0092] S5, the intelligent driving joint simulation test platform is used to obtain vehicle test data, and the evaluation index system and the test scene complexity model are combined to output an automatic evaluation report through evaluation index weight calculation and comprehensive performance quantification scoring.

[0093] The embodiment applies the above technical solution, mainly including:

[0094] In order to comprehensively and objectively evaluate the multi-dimensional performance of the automatic driving vehicle, the embodiment builds an intelligent driving multi-dimensional integrated test system, which integrates test scene generation, scene complexity quantification, intelligent driving algorithm embedding and automatic evaluation method. Figure 2 The architecture of the automatic driving vehicle multi-dimensional integrated test and evaluation method in the embodiment is shown. The entire automatic evaluation system is composed of four modules of a test scene library, a scene complexity model, a simulation test platform and an automatic evaluation system.

[0095] I. Constructing a test scene library

[0096] The test scene is the basis of intelligent driving test, which provides different simulation test scenes for intelligent driving test. These test scenes are constructed by data analysis and feature extraction on standard regulations, traffic accident data, natural driving data and expert experience. The specific process includes:

[0097] S11, natural driving test scenes are generated in a general way;

[0098] S12, scene parameter optimization is performed on the static traffic scene to improve the diversity and complexity of the static scene;

[0099] S13, adaptive adjustment is performed on the dynamic traffic scene to optimize the behavior of other traffic participants;

[0100] S14, key boundary test scenes with different complexities are output.

[0101] The embodiment optimizes scene parameters and behaviors of dynamic traffic participants from static and dynamic scenes to generate test scenes with different complexities. The generated static scene elements include static road structure, static obstacles, weather visibility, and road friction coefficient. In addition, the embodiment also comprehensively considers the influence of mixed traffic flow composed of autonomous vehicles, human-driven vehicles, bicycles, and pedestrians on the tested autonomous vehicle.

[0102] II. Constructing a test scene complexity model

[0103] The scene complexity model in the embodiment is established based on potential field theory. The objects in the scene related to the tested autonomous vehicle are abstracted as uniformly charged wires or positive point charges. According to the field theory, the objects in the scene will generate an electric potential field in the region space, which is inversely proportional to the distance, and the electric potential field is determined by the category of the object. The overall environmental potential field can capture the complexity of the scene by combining and superimposing different potential fields. The overall architecture of the scene complexity model is shown in Figure 3

[0104] The proposed scene complexity model is composed of static scene complexity C S , dynamic scene complexity C D , and weather environment complexity C W . The specific calculation method of the scene complexity model is as follows:

[0105] C=α S C S +α D C D +α W C W

[0106] wherein, α S , α D and α W are the weight coefficients of static scene complexity C S , dynamic scene complexity C D and weather environment complexity C W .

[0107] The scene complexity model only considers the influence of static targets and dynamic traffic participants within the influence area range around the tested autonomous vehicle on the tested autonomous vehicle, wherein the effective range of the influence area around the autonomous vehicle is described as a semicircle with the center point of the rear axle of the vehicle as the center, and the radius of the semicircle is:

[0108] r range =max{r safe ,2r l}

[0109] ​

[0110] where r range is the influence area radius of the ego vehicle, r safe is the safety zone radius of the ego vehicle, r l is the length of the intersection area range, v i is the initial speed of the ego vehicle, and τ is the response lag time of the ego vehicle, and are the maximum acceleration and the minimum deceleration of the ego vehicle, respectively.

[0111] The static scenario complexity is specifically:

[0112]

[0113] where represents the scenario complexity between the static traffic target i and the ego vehicle, and N is the number of static traffic targets in the scenario, is the virtual electric quantity of the i-th static target, and ε0is the permittivity, is the relative distance between the static traffic target i and the ego vehicle, (x i ,y i ) are the barycentric coordinates of the static traffic target i, (x0,y0) are the barycentric coordinates of the ego vehicle, and r0is the equivalent radius of the static traffic target.

[0114] The dynamic scenario complexity is specifically:

[0115]

[0116] where is the complexity between the dynamic traffic participant j and the ego vehicle, is the scaling enhancement factor, is the electronic energy level coefficient, is the virtual electric quantity of the j-th dynamic traffic participant, and ε0is the permittivity, is the distance between the dynamic traffic participant j and the ego vehicle, l j is the difference between the lane number where the ego vehicle is located and the lane number where the j-th traffic participant is located. When the ego vehicle and the j-th traffic participant are located in the same lane, l j = 1. θ j,0 and v j,0 are the encounter angle and the relative speed between the dynamic traffic participant j and the ego vehicle, v j ′ ,0 is the normalized relative speed, v max is the relative speed under the most complex scenario, vmin For the relative speed in the most simple scenario.

[0117] The weather environment complexity is specifically:

[0118]

[0119] where γ v and γ f are the weight coefficients of weather visibility complexity and road friction complexity, P max is the visibility distance when the lighting and clarity are best, P v is the visibility distance around the measured autonomous vehicle, P min is the visibility distance when the lighting and clarity are worst, F max is the road friction coefficient in the driest case, F r is the road friction coefficient of the test scenario, and F min is the road friction coefficient in the slipperiest case.

[0120] III. Building an intelligent driving joint simulation test platform

[0121] To meet the intelligent driving test requirements in various scenarios, the embodiment develops a joint simulation platform tool chain composed of a virtual simulation test software (VTD), Matlab / Simulink, and CarSim. The VTD provides intelligent driving simulation of complex traffic scenarios, including road network modeling, traffic scenario modeling, weather and environment simulation, and high-fidelity image rendering. CarSim is responsible for implementing high-precision vehicle dynamics simulation. Simulink supports embedding different intelligent driving algorithms and simulating them. Therefore, it is necessary to realize real-time interaction of vehicle dynamics in CarSim and VTD. The VTD sends the environment and position information perceived by the vehicle to CarSim for vehicle dynamics calculation, and CarSim feeds back the obtained dynamics state information to the VTD. At the same time, through the intelligent driving algorithm written by Simulink, the vehicle in the VTD can implement the corresponding automatic driving task. On this joint simulation platform, the driving performance of the intelligent driving algorithm is tested. The simulation test is carried out at the algorithm level, which is beneficial to optimize and improve the development and test efficiency of the intelligent driving algorithm.

[0122] IV. Building an automatic driving test multi-dimensional evaluation index system

[0123] The multi-dimensional evaluation index system aims to comprehensively depict the intelligence level of the autonomous vehicle and realize automatic evaluation. Therefore, the scheme proposes a joint evaluation index system composed of multi-dimensional evaluation indexes. The proposed evaluation system defines the evaluation indexes according to the interaction effect between the measured autonomous vehicle and the surrounding environment.

[0124] On the basis of the four existing performance evaluation indexes of the ego vehicle, the concept of altruistic performance is further proposed. The altruistic performance focuses on describing the influence of the ego vehicle's behavior on other surrounding traffic participants. In this embodiment, the traffic coordination index is used to represent the altruistic performance. The evaluation index system comprehensively characterizes the performance of the intelligent driving vehicle from two dimensions and five aspects.

[0125] In the evaluation index system, the safety is evaluated by the proxy safety index. Three commonly used driving safety indexes are selected: time to collision (TTC), time to exposure to collision (TET) and post encroachment time (PET). The comfort index focuses on the feelings of passengers during driving, including irregular driving, sudden braking and sharp turning caused by defects of the intelligent driving system. The maximum acceleration, maximum acceleration rate, maximum heading angle rate and speed standard deviation are used to represent the driving comfort. The driving performance describes the task completion quality and task completion efficiency of the ego vehicle in the corresponding operation design domain. The task completion time, maximum offset to the lane center, average speed and distance when the ego vehicle first detects the obstacle are used to describe the driving performance. The standard compliance mainly focuses on the degree of compliance of the ego vehicle to the traffic regulations. It is mainly used to describe whether the ego vehicle violates the traffic rules, such as speeding. In the altruistic performance dimension, the traffic coordination index is used to measure the influence of the ego vehicle's behavior on other vehicles and the overall traffic flow. The maximum deceleration of the surrounding vehicles and the average speed of the surrounding vehicles are used to represent the traffic coordination. As shown in Figure 4 The above two dimensions and five evaluation indexes constitute the evaluation index system of the ego vehicle through the hierarchical structure.

[0126] V. Automated evaluation system

[0127] The overall process of the automated evaluation system is shown in Figure 5The test scene is automatically extracted from the test scene library, and the autonomous vehicle is tested on the simulation test platform. Then, it is determined whether the autonomous vehicle passes the test scene without collision. Assuming that the autonomous vehicle completes all n test scenes without collision. In this case, the comprehensive performance evaluation is carried out, including constructing the evaluation matrix, determining the evaluation index weight based on the standard removal effect method, and calculating the comprehensive evaluation score. If a collision occurs during the test of the autonomous vehicle, the number of collisions j is calculated. If the accident rate of the autonomous vehicle is greater than 10%, the autonomous vehicle is considered unqualified and does not need to be scored further. If the accident rate of the autonomous vehicle is less than 10%, the comprehensive performance evaluation is continued and the evaluation score is calculated.

[0128] The weight of the evaluation index is calculated by the standard removal effect based method. The standard removal effect based method is an objective weight calculation method. When calculating the weight of the evaluation index, the influence of each evaluation index on the overall performance of intelligent driving test is comprehensively studied. The determination method of the objective weight is as follows:

[0129] Firstly, the evaluation matrix is constructed based on the test data obtained by intelligent driving test:

[0130]

[0131] Wherein, n is the number of completed intelligent driving tests, m is the number of evaluation indexes, and the test data of the zth evaluation index in the kth intelligent driving test is represented by x kz

[0132] The evaluation matrix X n×m is standardized, and all test data is scaled to the minimum type. The data of the standardized evaluation matrix is represented by . The standardization formula can be expressed as follows:

[0133]

[0134] Wherein, B is the positive evaluation index set, and C is the negative evaluation index set.

[0135] The overall performance of each intelligent driving test is calculated by the logarithmic measurement method under the condition that the evaluation index weight is equal. According to the standardized evaluation matrix, smaller will result in larger performance score f k , and its calculation formula is as follows:

[0136]

[0137] ​Then, the performance score of each intelligent driving test after removing each evaluation index is calculated. In this way, m sets of performance functions are associated with m evaluation indexes. The performance score of the kth intelligent driving test after removing the zth evaluation index f′ kz may be calculated as:

[0138]

[0139] After determining the elimination effect of the zth evaluation index, the sum of absolute deviations is calculated. The elimination effect of the zth evaluation index can be expressed as η z :

[0140]

[0141] The objective weight of the zth evaluation index ∈ z may be calculated according to the following formula:

[0142]

[0143] The specific calculation process of the multi-dimensional comprehensive performance quantification score is as follows:

[0144] Based on the test evaluation matrix, an ideal autonomous driving vehicle (IAV) and a worst autonomous driving vehicle (AIAV) are constructed, and further, an optimal test value (IT) and a worst test value (AIT) are calculated, thereby obtaining an extended evaluation matrix X E :

[0145]

[0146] The extended evaluation matrix of the positive and negative evaluation indexes is normalized:

[0147]

[0148] wherein x kz ∈ X E , x tkz ∈ X E ;

[0149] By multiplying the normalized evaluation matrix and the evaluation index weight ω z , a weighted evaluation matrix X w is calculated:

[0150]

[0151] The utility degree U k of the kth intelligent driving test with respect to the worst test value and the optimal test value is calculated according to the following formula:

[0152]

[0153] wherein S k is the sum of the values of the weighted evaluation matrix X w , and the calculation formula is:

[0154]

[0155] Then, the utility function of the kth intelligent driving test is determined. The utility function is determined according to the relative relationship with the worst test value and the optimal test value, and the calculation formula is:

[0156]

[0157]

[0158] According to the evaluation index weight ω z and the utility function f(U k ), the specific performance score of each evaluation index and the comprehensive performance evaluation score s of the tested autonomous vehicle can be calculated:

[0159]

[0160] wherein, is the standardized scene complexity weighting coefficient, C k is the scene complexity of the kth intelligent driving test, C max is the maximum scene complexity of the n intelligent driving tests.

[0161] The most important part of the technical solution is to comprehensively and objectively quantify the performance of the autonomous vehicle. In order to prove the effectiveness of the technical solution, the VTD autonomous driving model developed by VIRES and the DPA autonomous driving model based on rules are used for simulation verification in this embodiment. The VTD autonomous driving model is equivalent to a black box, which represents a class of black box intelligent driving algorithms. DPA is an intelligent driving algorithm that integrates hybrid flow intersection decision, path planning and real-time control.

[0162] Figure 6 The evaluation score and the comprehensive score of the VTD autonomous driving model and the DPA intelligent driving algorithm under different test scenes are shown. As can be seen from the figure, the evaluation score of the DPA autonomous driving model is significantly higher than that of the VTD autonomous driving model. This is mainly because the DPA intelligent driving algorithm based on rules is relatively conservative and aims to prevent collisions. In addition, the combination of the comprehensive score and the test scene complexity can effectively reduce the influence of different test scenes on the inconsistency of the test results, thereby effectively improving the objectivity and effectiveness of the evaluation results and ensuring the unbiasedness of the performance evaluation of different autonomous vehicles.

[0163] In summary, the target of the technical scheme is to study a four-layer automatic driving vehicle automatic evaluation architecture to improve the objectivity and impartiality of the test. First, the automatic evaluation system can generate test scenarios of different complexity for comprehensive automatic driving vehicle testing. Second, the potential field method is used to quantify the complexity of the scene to help evaluate various driving challenges. Third, the joint simulation platform combines VTD, Matlab / Simulink and CarSim to achieve accurate scene simulation and high-precision vehicle dynamics modeling. Fourth, the automatic evaluation system uses multi-dimensional indicators and evaluation algorithms to quantitatively evaluate the comprehensive multi-dimensional performance of the automatic driving vehicle, reducing human involvement and accelerating the evaluation efficiency. When applied to practice, users can automatically select test scenarios, automatically perform simulation testing and data collection, automatically process and analyze test data, and online calculate test results. This greatly reduces the subjectivity of manual evaluation and speeds up the evaluation process. In addition, the evaluation index system is flexible and extensible, and can integrate more evaluation dimensions and indicators in the future.

Claims

1. An online multi-dimensional comprehensive testing and evaluation method for autonomous driving testing, characterized in that, Includes the following steps: S1. Build a test scenario library; S2. Construct test scenario complexity models, including static scenario complexity, dynamic scenario complexity, and weather environment complexity models; S3. Build an intelligent driving joint simulation test platform and embed intelligent driving algorithms into the intelligent driving joint simulation test platform; S4. Construct a multi-dimensional evaluation index system for autonomous driving testing to evaluate the performance of autonomous vehicles in multiple dimensions. S5. Using the intelligent driving joint simulation test platform, vehicle test data is obtained. Combined with the evaluation index system and the test scenario complexity model, an automated evaluation report is output through the calculation of evaluation index weights and the comprehensive performance quantification score. Step S5 specifically includes the following steps: S51. Based on the vehicle test data, determine whether the currently tested autonomous driving vehicle has passed the safety test, and proceed accordingly. After the simulation test, if the accident rate of the autonomous vehicle is greater than the preset threshold, it indicates that the autonomous vehicle is unqualified and the current process ends; otherwise, proceed to step S52. S52. Using the evaluation index system, calculate the weights of the evaluation indicators to obtain the evaluation index weight matrix. S53. Combining the scenario complexity model and the evaluation index weight matrix, perform multi-dimensional comprehensive performance quantification scoring and output an automated evaluation report; The specific process of step S52 is as follows: S521. Based on vehicle test data, construct an evaluation matrix: ， in, The number of intelligent driving tests completed. To determine the number of evaluation indicators, the first... In the first intelligent driving test Test data for each evaluation indicator express; S522, Evaluation Matrix Standardization was performed, scaling all test data to a minimum size, and the standardized evaluation matrix data was then processed using... To represent, the standardized formula is: ， in, For positive evaluation indicators, This is a set of negative evaluation indicators; S523. According to the standardized evaluation matrix, the smaller one... This will result in a larger performance score. : ， ， S524. Calculate the performance score for each intelligent driving test after removing each evaluation indicator, so that... Group performance functions and The evaluation indicators are associated, and the first one is removed. After the first evaluation indicator, the second... Performance score of intelligent driving test for: ， S525, in determining the first After eliminating the impact of each evaluation indicator, the sum of absolute deviations is calculated, where the sum of the absolute deviations is the result of eliminating the impact of the first indicator. The impact of each evaluation indicator is expressed as: : ， ， S526, Calculate the first Objective weights of each evaluation indicator for: ， in, This represents the sum of the effects of all evaluation indicators.

2. The online multi-dimensional comprehensive testing and evaluation method for autonomous driving testing according to claim 1, characterized in that, The specific process of step S1 is as follows: S11, Generalize and generate natural driving test scenarios; S12. Optimize scene parameters for static traffic scenarios; S13. Adaptively adjust to dynamic traffic scenarios and optimize the behavior of other major traffic participants; S14. Output the critical boundary test scenarios with different complexities.

3. The online multi-dimensional comprehensive testing and evaluation method for autonomous driving testing according to claim 1, characterized in that, The scene complexity model in step S2 is specifically as follows: ， in, , and These are the static scene complexity Dynamic scene complexity and the complexity of the weather environment Weighting coefficients; In step S2, the scene complexity model only considers the influence of static targets and dynamic traffic participants within the influence area surrounding the tested autonomous vehicle. The effective range of the influence area surrounding the autonomous vehicle is described by a semicircle centered on the rear axle center point of the vehicle, and the radius of this semicircle is: ， ， in, The radius of the area affected by autonomous vehicles. The radius of the safe zone for the tested autonomous vehicle. The length of the intersection area. The initial speed of the autonomous vehicle. The response lag time for autonomous vehicles. and These represent the maximum acceleration and minimum deceleration of an autonomous vehicle, respectively.

4. The online multi-dimensional comprehensive testing and evaluation method for autonomous driving testing according to claim 3, characterized in that, The specific complexity of the static scene is as follows: ， ， ， in, Representing static traffic targets The complexity of the scenario between the autonomous vehicle under test and the test vehicle This represents the number of static traffic targets in the scene. For the first Virtual power of a static target Where is the dielectric constant. For static traffic targets The relative distance between the autonomous vehicle being tested and the test vehicle. For static traffic targets The coordinates of the center of gravity, The coordinates of the center of gravity of the autonomous vehicle under test. The equivalent radius of the static traffic target.

5. The online multi-dimensional comprehensive testing and evaluation method for autonomous driving testing according to claim 4, characterized in that, The complexity of the dynamic scene is specifically as follows: ， ， ， ， ， ， ， ， in, For dynamic traffic participants The complexity between the autonomous vehicle under test and the test vehicle For scaling enhancement factor, For electronic energy level coefficients, For the first Virtual battery level of each dynamic traffic participant Where is the dielectric constant. For dynamic traffic participants Distance between the autonomous vehicle being tested and the test vehicle The lane number of the autonomous vehicle under test and the lane number of the test vehicle The difference in lane numbers between each traffic participant and the tested autonomous vehicle. When multiple traffic participants are in the same lane , and Distribution of dynamic traffic participants Encounter angle and relative speed with the tested autonomous vehicle The standardized relative speed, The relative velocity in the most complex scenario. The relative speed is the speed in the least complex scenario.

6. The online multi-dimensional comprehensive testing and evaluation method for autonomous driving testing according to claim 5, characterized in that, The specific complexity of the weather environment is as follows: ， in, and These are the weighting coefficients for weather visibility complexity and road friction complexity, respectively. The visibility distance at which lighting and clarity are optimal. The visibility distance around the tested autonomous vehicle. The visibility distance at which lighting and clarity are at their worst. The coefficient of friction for the road under the driest conditions. To test the road friction coefficient in the scenario, This represents the coefficient of friction of the road under the most slippery conditions.

7. The online multi-dimensional comprehensive testing and evaluation method for autonomous driving testing according to claim 1, characterized in that, The multi-dimensional evaluation index system for autonomous driving testing in step S4 includes two dimensions, five primary indicators, and fourteen secondary indicators. The two dimensions include the vehicle performance dimension and the altruistic performance dimension. The performance dimensions of this vehicle include four primary indicators: safety, comfort, driving performance, and standardization. The altruistic performance dimension includes a primary metric: traffic coordination for autonomous vehicles.

8. The online multi-dimensional comprehensive testing and evaluation method for autonomous driving testing according to claim 1, characterized in that, The specific process of multi-dimensional comprehensive performance quantification and scoring in step S53 is as follows: S531. Based on the test evaluation matrix, construct the ideal autonomous vehicle (IAV) and the worst autonomous vehicle (AIAV), and calculate the optimal test value IT and the worst test value AIT to obtain the extended evaluation matrix. : ， ， ， S532. Standardize the extended evaluation matrix of positive and negative evaluation indicators: ， in, , ; S533, Standardize the evaluation matrix Weights of evaluation indicators Multiply by each other to obtain the weighted evaluation matrix. : ， S534, Calculate the first Utility between sub-intelligent driving tests and worst and best test values : ， ， ， in, Weighted evaluation matrix The sum of the values; S535, Determine the first The utility function for this intelligent driving test is determined based on its relative relationship with the worst and best test values, and its calculation formula is as follows: ， ， ， S536, Based on the weights of the evaluation indicators and utility function The specific performance score for each evaluation indicator and the overall performance evaluation score of the tested autonomous vehicle are calculated. : ， ， in, To standardize the weighting coefficients for scenario complexity, For the first The complexity of the scenarios in this intelligent driving test yes The maximum scenario complexity for this intelligent driving test.

Images