Quantitative evaluation method for reliability of whole vehicle in-the-loop simulation test

Abstract

The embodiment of the invention discloses a quantitative evaluation method for the reliability of a whole vehicle in-the-loop simulation test, and the method comprises the steps: constructing a plurality of test scenes, and testing a simulation vehicle and a real vehicle in the plurality of test scenes; original simulation vehicle information of a simulation vehicle and original test vehicle information of a real vehicle are collected, data optimization processing and screening are carried out, and a simulation data set and a test data set are obtained; based on a preset subjective score data set, a simulation data set and a test data set, a plurality of error scores and a multi-curve trend consistency score are obtained, and the error scores comprise an objective phase score, an objective amplitude score and an objective slope score; based on the multiple error scores and the trend consistency score, the test score of simulation vehicle-in-the-loop simulation is obtained, error calculation can be carried out in multiple dimensions to obtain the corresponding error score, the accuracy of the evaluation result can be improved, and the scientificity and consistency of simulation evaluation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving simulation testing, and particularly to a quantitative evaluation method for the credibility of vehicle-in-the-loop simulation testing. Background Art

[0002] Autonomous driving simulation testing can accelerate the development of intelligent driving and the process of algorithm deployment. However, the differences between the reproduced and generalized virtual simulation environment and the real environment will affect the test results. Currently, the research on simulation credibility evaluation mainly focuses on SIL (Safety Integrity Level) and HIL (hardware-in-the-loop) and is subjective. There is a lack of quantifiable evaluation indicators and methods at the vehicle performance level. Abroad, in the field of passive safety, by introducing objective evaluation criteria such as CORA (Correlation and Analysis) and ISO (International Organization for Standardization), the scientific nature and consistency of simulation evaluation have been significantly improved. Currently, the domestic research on the credibility of autonomous driving simulation testing is still in its infancy, and there is no set of mature and complete evaluation systems.

[0003] As the last link in the "V" - shaped development process of the autonomous driving system, vehicle-in-the-loop simulation testing of autonomous driving vehicles can discover and solve 70% of systematic problems, and determines whether the autonomous driving vehicle can finally correctly execute perception and regulation algorithms and drive well on the road. The traditional evaluation of credibility is mainly subjective, and there is no quantifiable evaluation method. Moreover, the mass production of autonomous driving vehicles requires a large number of simulation tests for algorithm verification and performance improvement. Therefore, a scientific quantitative evaluation of the credibility of vehicle-in-the-loop simulation testing of autonomous driving vehicles is an urgent problem to be solved currently. Summary of the Invention

[0004] The purpose of the present invention is to provide a quantitative evaluation method for the credibility of vehicle-in-the-loop simulation testing, and the method includes:

[0005] Constructing a plurality of test scenarios, and respectively conducting multiple tests on the simulation vehicle and the real vehicle under the plurality of test scenarios;

[0006] In each test, collecting test data of the simulation vehicle and the real vehicle from multiple dimensions to obtain a simulation data set corresponding to the simulation vehicle and a test data set corresponding to the real vehicle;

[0007] It is characterized in that:

[0008] Based on a preset subjective scoring dataset, the simulation dataset, and the test dataset, error scores for multiple data dimensions and trend consistency scores for the corresponding data curves are obtained. The error scores include an objective phase score, an objective amplitude score, and an objective slope score.

[0009] Based on the error scores and the trend consistency scores, a test score for quantifying the credibility evaluation of the in-loop simulation of the simulation vehicle is obtained.

[0010] The embodiments of the present application have the following beneficial effects:

[0011] The embodiments of the present application disclose a method for quantitatively evaluating the credibility of a vehicle-in-the-loop simulation test. The method for quantitatively evaluating the credibility of the in-loop simulation proposed by the present invention can perform simulation tests using a vehicle-in-the-loop simulation system according to different test scenarios, create different subjective scoring datasets for vehicle-in-the-loop simulation, and can calculate errors for the longitudinal and lateral output parameters of the vehicle-in-the-loop simulation from a single dimension to multiple dimensions to obtain corresponding error scores. Moreover, the objective weights of the comprehensive scores are determined according to the importance of each score in multiple tests, and the confidence of the vehicle-in-the-loop simulation test is quantified, thereby knowing the simulation accuracy of the vehicle-in-the-loop simulation test system, and significantly improving the scientificity and consistency of the simulation evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the present invention, the drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the protection scope of the present invention. In each drawing, similar components are numbered similarly.

[0013] Figure 1 The figure shows a schematic diagram of a method for quantitatively evaluating the credibility of a vehicle-in-the-loop simulation test proposed by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0014] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.

[0015] Embodiment 1

[0016] In order to more clearly show the implementation steps and advantages of the invention, the following describes the specific implementation manners with reference to the legends.

[0017] The embodiments of the present application propose a method for quantitatively evaluating the credibility of a vehicle-in-the-loop simulation test. As Figure 1 shown, exemplarily, the specific implementation steps of the method are as follows:

[0018] Construct multiple test scenarios, and test the simulated vehicle and the real vehicle separately under multiple test scenarios; collect the original simulated vehicle information of the simulated vehicle and the original test vehicle information of the real vehicle, perform data optimization processing and screening to obtain a simulation data set and a test data set; based on the preset subjective scoring data set, simulation data set and test data set, obtain multiple error scores and trend consistency scores of multiple curves; based on the multiple error scores and trend consistency scores, obtain the test score of the simulated vehicle in the in-loop simulation.

[0019] In this example, the actual test vehicle model can be Lotus Eletre. The industrial test machine can be tested by an axle-coupled vehicle-in-the-loop simulation system. The vehicle working conditions of this vehicle can use a VEHICO robot control vehicle with the model SR15+CBAR600. Among them, the Racelogic VBOX positioning data acquisition system and the i-TESTER video acquisition system can be used during the test. The Racelogic VBOX adopts a fixed base station mode, and the i-TESTER video acquisition system is used to monitor the test process and collect the vehicle body CAN (Controller Area Network) signal at the same time. During the test, the information to be collected includes but is not limited to the vehicle motion attitude information and steering wheel angle information of the vehicle, that is, the transverse and longitudinal output parameters of the vehicle are collected. The vehicle motion attitude information includes wheel speed, torque, vehicle speed and acceleration, etc. Among them, the steering wheel angle is the transverse output parameter, and the wheel speed, torque, vehicle speed and acceleration are the longitudinal output parameters. In other words, during the test, the transverse and longitudinal output information of the simulated vehicle and the real vehicle will be collected separately.

[0020] It can be understood that multiple different test scenarios can be created during the test, and the information corresponding to each time point of the simulated vehicle and the test vehicle in each test scenario can be collected multiple times, that is, the original simulated vehicle information corresponding to the simulated vehicle and the original test vehicle information corresponding to the test vehicle are collected. The original simulated vehicle information is the transverse and longitudinal output parameters of the simulated vehicle during the in-loop simulation test, including simulated wheel speed, simulated torque, simulated vehicle speed, simulated acceleration and simulated steering wheel angle, etc.; the original test vehicle information is the transverse and longitudinal output parameters of the real vehicle during each test, including test wheel speed, test torque, test vehicle speed, test acceleration and test steering wheel angle, etc.

[0021] Among them, the test scenarios can be set according to the actual situation, and multiple tests are carried out under each test scenario to improve data accuracy. As shown in Table 1, whether there are interfering vehicles in the test scenario, whether the vehicle is driving on a straight road or a curve, whether the vehicle is driving at a constant speed, accelerating, decelerating or turning, and the percentage of the vehicle's accelerator pedal can all be set by itself. Demonstratively, the test scenario can be that there are no interfering vehicles, the vehicle is driving at a constant speed on a straight road, and the vehicle is kept driving at a constant speed for the same time. Different speeds are different test scenarios. Under each test scenario, the simulation vehicle and the test vehicle can be tested three times. For example, when the vehicle is driving at a constant speed on a non-interfering straight road for 60s, the vehicle speed can be kept at 30km / h, 60km / h, 80km / h or 120km / h. Each vehicle speed corresponds to a test scenario. The simulation vehicle and the test vehicle will both be tested three times under each test scenario, and the original simulation vehicle information and the original test vehicle information at each test will be collected.

[0022]

[0023] Table 1

[0024] Collect the original simulation vehicle information of the simulation vehicle and the original test vehicle information of the actual vehicle, clean the original simulation vehicle information and the original test vehicle information to remove the information corresponding to the meaningless timestamps, and respectively obtain the cleaned original simulation vehicle information and the cleaned original test vehicle information; use the spline interpolation method to fill in the missing values of the cleaned original simulation vehicle information and the cleaned original test vehicle information to obtain the corresponding original simulation data and original test data; process and screen the original simulation data and the original test data through the convolution smoothing algorithm to form a simulation data set and a test data set.

[0025] It can be understood that the original simulation vehicle information and the original test vehicle information are respectively cleaned to remove the information corresponding to the meaningless timestamps caused by the inconsistent time points when the simulation vehicle and the test vehicle trigger responses during the test. Among them, the cleaning process includes, but is not limited to, processing invalid values, deleting duplicates, and correcting error data. Such cleaning methods are well-known to those skilled in the art and will not be elaborated here. After the cleaning process, the spline interpolation method is used to process the missing values of the cleaned original simulation vehicle information and the cleaned original test vehicle information respectively to generate the corresponding original simulation data and original test data. The spline interpolation method is well-known to those skilled in the art and will not be elaborated here. In this application, the data optimization process and screening can make the data in the data set more accurate, complete, consistent and reliable.

[0026] After generating the original simulation data and original test data in this application, the Savitzky-Golay convolution smoothing algorithm is used to process the above original simulation data and original test data for the same time series curve. Different windows are set according to different noise characteristics. Let the width of the filtering window be 2m + 1, where m is the value that maximizes the signal-to-noise ratio. Within each window, a polynomial of order n is used to fit the signal. n represents the order of the polynomial, and n ≤ 2m + 1. Here, the signal-to-noise ratio refers to the ratio of the signal to the noise in an electronic device or electronic system. The polynomial of order n is specifically as follows:

[0027] P(t) = a 0 + a 1 t + a 2 t 2 + … a n t n

[0028] In the formula, P(t) represents the signal value after polynomial fitting, a 0 , a 1 , …, a n represent the coefficients of the polynomial, and t represents the time variable. Through the above polynomial of order n, the signal values after polynomial fitting corresponding to the original simulation data and the signal values after polynomial fitting corresponding to the original test data can be determined respectively.

[0029] In this application, the coefficients of the polynomial are determined by minimizing the fitting error using Equation (1), and the smoothed signal is obtained using Equation (2). The specific expressions are as follows:

[0030]

[0031]

[0032] In Equation (1), t represents the time variable, t i represents the i-th time point, represents the squared error between the original signal and the signal value after fitting at time t i . y(t i ) represents the value of the original signal at time t i , and P(t i ) represents the signal value after polynomial fitting at time t i . In Equation (2), represents the smoothed signal, c i represents the convolution coefficient, y(t + i·Δt) represents the value of the original signal within the sliding window, and Δt represents the sampling frequency. The difference between multiple curves

[0033] It is understandable that the smoothed original simulation data corresponding to the original simulation data and the smoothed original test data corresponding to the original test data can be obtained through the nth-order polynomial, Equation (1), and Equation (2), that is, the smoothed data corresponding to the original data is obtained.

[0034] In this application, screening will be performed on the smoothed data according to a pre-determined time starting point and sampling frequency to obtain corresponding simulation data and test data, and form corresponding simulation data sets and test data sets. Among them, the time starting point and sampling frequency can be determined according to the actual situation. For example, the time starting point can be preset to t = 0.00, and the sampling frequency is Δt = 0.02 s. Screening is performed on the smoothed data according to the time starting point t = 0.00 and the sampling frequency Δt = 0.02 s. Data such as the position, speed, acceleration, wheel torque, and steering wheel angle corresponding to each time point are screened from the above-mentioned smoothed data according to the pre-determined time starting point and sampling frequency to form a simulation data set k ∈ {s, v, a, n, θ} and a test data set , k ∈ {s, v, a, n, θ}; where k represents different dimensions in the data set, θ represents the steering wheel angle, s represents the position, v represents the speed, a represents the acceleration, n represents the wheel torque, θ represents the steering wheel angle, N represents the total number of data points in a single dimension, and the dimensions in this application can be the position, speed, acceleration, wheel torque, or steering wheel angle.

[0035] In this example, the error score and the trend consistency score are used to quantify the credibility of the vehicle-in-the-loop simulation;

[0036] Specifically, the error score includes the phase error score, amplitude error score, and slope error score for different data dimensions;

[0037] The trend consistency score includes the overall trend consistency score and the change inflection point consistency score for different data curves.

[0038] Therefore, on the one hand, after collecting the original simulation vehicle information and the original test vehicle information, the original simulation vehicle information and the original test vehicle information collected multiple times in different test scenarios will be provided to multiple domain experts. Multiple domain experts will input the subjective scores corresponding to the vehicle-in-the-loop simulation of the simulation vehicle in each dimension under different test conditions according to the original simulation vehicle information and the original test vehicle information. The subjective scores include, but are not limited to, the scores for the vehicle-in-the-loop simulation time response synchronization ability, the output numerical accuracy, the curve change rate matching degree, the overall trend consistency, and the significant change inflection point consistency. They are respectively represented as the subjective phase error score F P 、subjective amplitude error score Fm Subjective slope score F s Subjective overall trend consistency score F r and subjective inflection point consistency score F c .

[0039] Receive multiple subjective scores input according to the original simulation vehicle information and the original test vehicle information; perform fuzzy processing on the multiple subjective scores to form a preset subjective score data set. In addition, in this application, the evaluation scores are divided into "very poor (D 1 )", "poor (D 2 )", "average (D 3 )", "good (D 4 )", "excellent (D 5 )" 5 fuzzy levels, and the basic score fuzzy intervals corresponding to each fuzzy level are [0, 40], [40, 60], [60, 80], [80, 90], [90, 100] respectively. Calculate the membership degrees of different fuzzy levels corresponding to each score in each dimension k through the membership function of the fuzzy interval Calculate the membership degrees of different fuzzy levels corresponding to each score in each dimension k

[0040] Among them, the membership function of the fuzzy interval is specifically as follows:

[0041]

[0042] In formula (3), Δτ k is the fuzzy factor of dimension k, |Δτ k | ≤ 5, the default value is 0, D e represents the e-th fuzzy level, e = 1, 2... 5, a e represents the lower boundary value of the basic score interval of the fuzzy level, c e represents the upper boundary value of the basic score interval of the fuzzy level, b e represents the midpoint of the basic score interval of the fuzzy level F can be the subjective phase error score F p Subjective amplitude error score F m Subjective slope error score F s Subjective trend consistency score F r or subjective inflection point consistency score F c .

[0043] Taking the phase error score as an example, represents the j-th subjective phase error score. When receiving multiple subjective scores input by five domain experts, construct the subjective phase score membership matrix of dimension k j = 1, 2... 5, as shown in formula (4):

[0044]

[0045] Among them, represents the membership degree matrix of the subjective phase error score. The rows of the membership degree matrix represent the membership degrees of each subjective phase error score at different fuzzy levels.

[0046] The membership degree matrix of the subjective phase score provides important basic information for defuzzification. The final defuzzified phase score of the k dimension is calculated using Equation (5). Equation (5) is specifically as follows:

[0047]

[0048] Among them, represents the defuzzified phase score of the k dimension.

[0049] In addition, using the same steps above, that is, using the above Equation (3), Equation (4) and Equation (5) and their corresponding steps, the membership degree matrices corresponding to the subjective amplitude error score and the subjective slope error score of each dimension k in each test scenario and each experiment are constructed in the same way, and the corresponding defuzzified amplitude scores and defuzzified slope scores are obtained respectively. According to each experiment of each test scenario and the corresponding defuzzified phase score defuzzified amplitude score defuzzified slope score a preset subjective score data set is formed. In other words, the preset subjective score data set also includes specific phase error score items, amplitude error score items, slope error score items and their corresponding defuzzified scores calculated, as well as overall trend consistency score items and change inflection point consistency score items.

[0050] On the other hand, in this example, the error values will be calculated based on the simulation data set and the test data set; multiple objective scores will be calculated respectively through the objective score formula based on the error values, the maximum acceptable error in the subjective score data set and the preset error sensitivity coefficient; data interaction will be carried out through the preset subjective score data set and multiple objective scores, and the maximum acceptable error, the fuzzy factor in the subjective score data set and the preset error sensitivity coefficient will be iterated until it is determined that the fitness between the defuzzified subjective score in the subjective score data set and each objective score is the maximum value among all calculated fitnesses; multiple objective scores will be calculated based on the optimized maximum acceptable error, the optimized fuzzy factor and the optimized error sensitivity coefficient.

[0051] In this example, the data set includes multiple dimensions and is derived from multiple experiments under multiple test environments. It is necessary to calculate the corresponding objective scores for each dimension separately in each experiment under each test environment. The general objective score formula is as follows:

[0052]

[0053] In the formula, k represents different dimensions in the subjective score data set, and E k is the objective phase score of the k-th dimension during each experiment, is the maximum acceptable error of the k-th dimension in the subjective score data set, and β k is the error sensitivity coefficient of the k-th dimension, and β k can both be set according to the actual situation. ε k is the error value of the k-th dimension. Among them, E k can be the objective phase score of the k-th dimension the objective amplitude score of the k-th dimension or the objective slope score of the k-th dimension β k can be the amplitude error sensitivity coefficient of the k-th dimension the phase error sensitivity coefficient of the k-th dimension or the amplitude error sensitivity coefficient of the k-th dimension can be the maximum acceptable slope error of the k-th dimension the maximum acceptable amplitude error of the k-th dimension the maximum acceptable phase error of the k-th dimension ε k can be the slope error of the k-th dimension the amplitude error of the k-th dimension the phase error of the k-th dimension

[0054] It can be understood that after obtaining the simulation data set and the test data set, the phase error corresponding to each dimension between the simulation data set and the test data set will be calculated. The time series x k of the k-th dimension in the simulation data is shifted left or right along the time axis, one data point at a time. The Pearson correlation coefficient between the time series after each shift and the time series y k of the k-th dimension in the test data set is calculated. When the Pearson correlation coefficient between the two time series is the largest, the ratio of the step size by which the time series x k of the k-th dimension in the simulation data is shifted and the time axis is the phase error of the k-th dimension

[0055] Calculate the objective phase score of the k dimension for each experiment under all test scenarios using Equation (7). As follows:

[0056]

[0057] Among them, is the maximum acceptable phase error of the k dimension in the subjective score dataset, which can be set according to the actual situation. is the phase error sensitivity coefficient of the k dimension.

[0058] Calculate the simulation dataset of the k dimension after time alignment using Equation (8) and Equation (9) respectively and the test dataset The average amplitude error and the maximum amplitude error The specific expressions are as follows:

[0059]

[0060]

[0061] Finally, use Equation (10) to calculate the amplitude error of the k dimension Calculate the objective amplitude score of the k dimension using Equation (11)

[0062]

[0063]

[0064] Among them, is the maximum acceptable amplitude error of the k dimension in the subjective score dataset, which can be set according to the actual situation. is the amplitude error sensitivity coefficient of the k dimension, which can be set according to the actual situation.

[0065] Derive each time point in the simulation dataset of the k dimension after time alignment and the test dataset to obtain the simulation slope dataset and the test slope dataset Calculate the slope error between the simulation and the test at each time point of the k dimension For g ∈ [1, M] and h ∈ [1, N], use Equation (12) and Equation (13) to obtain the average slope error and the maximum slope error The specific expressions are as follows:

[0066]

[0067]

[0068] Finally, the slope error is calculated using Equation (14). Calculate the objective slope score for dimension k using Equation (15). The specific expression is as follows:

[0069]

[0070]

[0071] Wherein, represents the objective slope score for dimension k, is the maximum acceptable amplitude error for dimension k in the subjective scoring dataset, which can be set according to the actual situation, is the amplitude error sensitivity coefficient for dimension k.

[0072] In different test scenarios, when conducting A trials in a single test scenario, in order to determine the values of β k and Δτ k the mean squared error (abbreviated as MSE) between the minimized defuzzified subjective score and the objective score E k is used as the optimization objective, and the particle swarm algorithm is used to optimize and Δτ k .

[0073] It can be understood that β k represents the error sensitivity coefficient for dimension k, and β k can be the amplitude error sensitivity coefficient for dimension k the phase error sensitivity coefficient for dimension k or the amplitude error sensitivity coefficient for dimension k represents the defuzzified score for dimension k, which can be the defuzzified phase score the defuzzified amplitude score or the defuzzified slope score represents the average value of the subjective scores for dimension k in the subjective scoring dataset, which can be the average value of the subjective phase scores for dimension k the average value of the subjective amplitude scores for dimension k or the average value of the subjective slope scores for dimension k E k represents the objective score, that is, the error score, and E kCan score objective phases in k dimensions Objective magnitude scores in k dimensions Or objective slope score in k dimensions Δτ k represents the fuzzy factor in the subjective data set, Δτ k The phase ambiguity factor can be calculated for the subjective data set. Amplitude fuzziness factor of subjective data or the slope fuzziness factor of the subjective data set

[0074] In this example, the average of the subjective ratings of k dimensions in the subjective ratings dataset is calculated. calculate and A k-dimensional defuzzified subjective ratings The sum of squared differences between tot , calculate the k-dimensional defuzzified subjective score and the objective score E of k dimensions k The sum of squared differences between res , calculate the fit R between the subjective and objective scores of k dimensions 2 , use grid search to determine the fit R 2 The largest beta k and the fuzzy factor Δτ in the subjective data set k The final objective score E is obtained k , that is, the error score E k , in other words, the optimized ε max_s , and Based on the optimized ε max_s , and Determine the corresponding and The value of β is determined above. k , Δτ k , The process can be expressed by the optimization calculation in the following formula (16):

[0075]

[0076] Where A represents the number of experiments in each test scenario, a∈[1,A], is the defuzzified subjective phase score of the ath experiment in the subjective rating dataset, is the objective score of the k-th dimension of the a-th experiment.

[0077] In terms of consistency scoring, in this example, the Pearson correlation coefficient between each dimension of the simulation dataset and the experimental dataset is calculated, and the overall trend consistency score is obtained based on multiple Pearson correlation coefficients; the inflection point consistency score is obtained by calculating the simulation dataset and the experimental dataset through the inflection point detection algorithm and the preset inflection point matching formula.

[0078] The k dimensions of the dataset influence each other. Therefore, the trend consistency of the curve can measure the differences in the motion trends and inflection points of state changes between the simulation dataset and the experimental dataset. First, use Equation (17) to calculate the simulation dataset after time alignment and the experimental dataset of the Pearson correlation coefficient r between each dimension k , and then use Equation (18) to calculate the motion consistency score E r . The specific expressions are as follows:

[0079]

[0080]

[0081] In the formula represents the b-th simulation data in the simulation dataset after time alignment, represents the time-averaged simulation data, represents the average experimental data of the experimental dataset after time alignment, and K represents the total number of dimensions.

[0082] Use the inflection point detection algorithm to perform inflection point detection on each dimension of the simulation dataset and the experimental dataset respectively, and obtain the simulation inflection point data index set and the experimental inflection point data index set where D k represents the number of inflection points in the k dimension, and represent the sequential index of the D k -th inflection point in the k dimension in the simulation dataset;

[0083] Merge the inflection point indexes of all dimensions of the simulation data and the inflection point indexes of all dimensions of the experimental data respectively, and sort the two sets of inflection point index sets in ascending order according to the position to form the simulation inflection point data index set G = {g 1 , g 2 , …, g D} and the experimental inflection point data index set Q = {q 1 , q 2 , …, q D}, where D represents the total number of inflection points. The DBSCAN clustering algorithm is used for the two groups of inflection point datasets respectively. The clustering radius r is set to 500 and min_samples is set to 1. The clustering algorithm groups adjacent core points into the same cluster. For each cluster, the average value of all inflection point indices within the cluster is calculated as the comprehensive inflection point position, and finally the comprehensive simulation inflection point data position set and the comprehensive test inflection point data position set Then, using Equation (19) for and to perform matching to obtain the set M of all successfully matched inflection point pairs. The specific expression is as follows:

[0084]

[0085] Using Equation (22) and Equation (23) respectively to calculate the number N of successfully matched simulation inflection points sim and the number N of successfully matched test inflection points real , and the specific expression is as follows:

[0086]

[0087]

[0088] Using Equation (22) and Equation (23) to calculate the matching precision P and the matching coverage rate R. The specific expression is as follows:

[0089]

[0090]

[0091] Among them, L represents the number of inflection points in the comprehensive simulation inflection point dataset, and J represents the number of inflection points in the comprehensive test inflection point dataset. Finally, use Equation (24) to calculate the inflection point consistency score E c , and the specific expression is as follows:

[0092]

[0093] In the formula, E c represents the inflection point matching score.

[0094] Based on the number of experiments in different test scenarios, determine the mean and entropy values corresponding to multiple error scores and trend consistency scores; based on the mean and entropy values, calculate the objective weights corresponding to multiple error scores and trend consistency scores respectively; weight and add multiple error scores and trend consistency scores with their corresponding objective weights respectively to obtain the test score of the simulation vehicle in the in-loop simulation.

[0095] For different test scenarios, the test data of n error scores are obtained after m simulation test experiments. Then, the data of the error score z in the w-th simulation test experiment can be expressed as x wz , and the evaluation matrix can be constructed as X m×n , and the specific expression (25) is as follows:

[0096]

[0097] where m represents the number of experiments for a single test scenario, n represents the number of error scores, and the error score z can be the motion consistency score E r , the objective amplitude score the objective phase score the objective slope score and the inflection point matching score E c .

[0098] Calculate the standardized value p of the error score z in the w-th simulation test experiment using Equation (27) wz , and the data of the error score z in the w-th simulation test experiment is expressed as x wz , and the specific expression is as follows:

[0099]

[0100] Then, calculate the mean value and the entropy value H of each error score using Equation (28) and Equation (29) respectively z

[0101]

[0102]

[0103] where Calculate the difference degree D of each scoring index in m trials using Equation (30) z , and the specific expression is as follows:

[0104]

[0105] Finally, calculate the weight ω of each error score using Equation (31) z , and the specific expression is as follows:

[0106]

[0107] Use Equation (32) to weightedly combine the above five error scores to obtain the final in-loop simulation test score of the entire autonomous driving vehicle In other words, calculate the scores of each error rating and the corresponding objective weights to obtain the corresponding comprehensive ratings, and add up the comprehensive ratings to obtain the test rating of the simulation vehicle-in-the-loop simulation. This evaluation rating represents the evaluation rating of a single dimension, and the specific expression is as follows:

[0108]

[0109] where ω p represents the objective weight of the phase rating, ω m represents the objective weight of the amplitude rating, ω s represents the objective weight of the slope rating, ω r represents the objective weight of the overall trend rating, ω c represents the objective weight of the inflection point matching rating, represents the objective phase rating of the k-th dimension, represents the objective amplitude rating of the k-th dimension, represents the objective slope rating of the k-th dimension, e r represents the motion consistency rating, E c represents the inflection point matching rating.

[0110] The quantitative evaluation method for the credibility of the vehicle-in-the-loop simulation test of the present application can use the vehicle-in-the-loop simulation system to conduct simulation tests according to different test scenarios, create different vehicle-in-the-loop simulation subjective rating data sets, be able to calculate the errors of the horizontal and vertical output parameters of the vehicle-in-the-loop simulation from a single dimension to multiple dimensions to obtain the corresponding error ratings, and interact through the subjective rating data sets and each error rating to improve the accuracy of each error rating, thereby optimizing the evaluation process and improving the accuracy of the evaluation results. It can also determine the objective weights of the comprehensive ratings according to the importance of each rating in multiple tests, quantify the confidence of the vehicle-in-the-loop simulation test, and thus know the simulation accuracy of the vehicle-in-the-loop simulation test system, significantly improving the scientificity and consistency of the simulation evaluation.

[0111] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention.

Claims

1. A quantitative evaluation method for the credibility of vehicle-in-the-loop simulation test, comprising: Constructing multiple test scenarios, and performing multiple tests on the simulated vehicle and the real vehicle respectively under the multiple test scenarios; In each test, test data of the simulated vehicle and the real vehicle are collected from multiple dimensions to obtain a simulation data set corresponding to the simulated vehicle and a test data set corresponding to the real vehicle; Features: Based on the preset subjective scoring data set, the simulation data set and the test data set, error scores of multiple data dimensions and trend consistency scores of corresponding data curves are obtained, wherein the error scores include objective phase scores, objective amplitude scores and objective slope scores; Based on the error score and the trend consistency score, a test score for quantifying the credibility assessment of the simulated vehicle-in-the-loop simulation is obtained.

2. The method for quantitatively evaluating the credibility of vehicle-in-the-loop simulation test according to claim 1 is characterized in that: The row data is optimized and screened, including: Collecting original simulated vehicle information of the simulated vehicle and original test vehicle information of the real vehicle, cleaning the original simulated vehicle information and the original test vehicle information to remove information corresponding to meaningless timestamps, and obtaining cleaned original simulated vehicle information and cleaned original test vehicle information respectively; Using a spline interpolation method to complete the missing values ​​of the cleaned original simulation vehicle information and the cleaned original test vehicle information, to obtain corresponding original simulation data and original test data; The original simulation data and the original test data are processed and screened by a convolution smoothing algorithm to form a simulation data set and a test data set.

3. The method for quantitatively evaluating the credibility of vehicle-in-the-loop simulation test according to claim 1 is characterized in that: The trend consistency score includes a motion consistency score and an inflection point matching score. The trend consistency score is calculated based on the simulation data set and the test data set, including: Calculating the Pearson correlation coefficients between the simulation data set and the test data set in each dimension, and obtaining the motion consistency score based on multiple Pearson correlation coefficients; The simulation data set and the test data set are calculated using an inflection point detection algorithm and a preset inflection point matching formula to obtain the inflection point matching score.

4. The method for quantitatively evaluating the credibility of vehicle-in-the-loop simulation test according to claim 1 is characterized in that: The “obtaining multiple error scores and multiple curve trend consistency scores based on the preset subjective score data set, the simulation data set and the test data set”, the defuzzified subjective score includes multiple defuzzified subjective scores, including: Calculating an error value based on the simulation data set and the test data set; Calculating a plurality of objective scores respectively through an objective scoring formula based on the error value, the maximum acceptable error in the subjective scoring data set, and a preset error sensitivity coefficient; By performing data interaction between a preset subjective scoring data set and the plurality of objective scores, iterative optimization is performed on the preset acceptable maximum error, the fuzzy factor in the subjective scoring data set, and the preset error sensitivity coefficient, with the goal of maximizing the fit between the defuzzified subjective score in the subjective scoring data set and each of the objective scores; The multiple objective scores are calculated based on the optimized acceptable maximum error, the optimized fuzzy factor, and the optimized error sensitivity coefficient.

5. The method for quantitatively evaluating the credibility of vehicle-in-the-loop simulation testing according to claim 4 is characterized in that: The objective scoring formula is: In the formula, k represents the different dimensions in the subjective rating dataset, E k is the objective score of k dimensions in each experiment, is the maximum acceptable error in the k-dimensional space, β k is the error sensitivity coefficient, ε k is the error value in dimension k.

6. The method for quantitatively evaluating the credibility of vehicle-in-the-loop simulation testing according to claim 4 is characterized in that: The formula for maximizing the fit between the defuzzified subjective score and the objective score is expressed as follows: Where k represents the different dimensions in the subjective rating dataset, A represents the number of experiments in each test scenario, a∈[1,A], is the defuzzified subjective phase score of the ath experiment in the subjective score dataset, is the average of the subjective ratings of k dimensions in the subjective rating dataset, is the objective score of the k-th dimension of the a-th experiment, SS tot It represents the average value of the subjective rating of k dimensions in the subjective rating dataset and the sum of the squared differences of the defuzzified subjective ratings of A k dimensions, SS res represents the k-dimensional defuzzified subjective score and k-dimensional objective score E k The sum of squared differences between 2 Represents the fitness.

7. The method for quantitatively evaluating the credibility of vehicle-in-the-loop simulation test according to claim 6 is characterized in that: Particle swarm algorithm is used to solve the optimization problem.

8. The method for quantitatively evaluating the credibility of vehicle-in-the-loop simulation test according to claim 1 is characterized in that: “Obtaining a test score of the simulated vehicle-in-the-loop simulation based on the multiple error scores and the trend consistency score” includes: The mean and entropy values ​​corresponding to each of the error scores and the trend consistency scores are determined based on the number of experiments in different test scenarios. The formula for calculating the mean is as follows: Where m represents the number of experiments for a single test scenario. represents the mean value corresponding to each error score, x wz The data representing the error score z in the w-th simulation test experiment, the formula for calculating the entropy value is as follows: In the formula, H z represents the entropy value corresponding to each error score, p wz represents the normalized value of the error score z in the wth simulation test experiment; Based on the mean and the entropy value, calculating objective weights corresponding to the plurality of error scores and the trend consistency score respectively; The multiple error scores and the trend consistency scores are weighted and added to the corresponding objective weights to obtain a test score of the simulated vehicle-in-the-loop simulation.

9. The method for quantitatively evaluating the credibility of vehicle-in-the-loop simulation testing according to claim 6 is characterized in that: The weighted addition formula is: In the formula, ω p represents the objective weight of the phase score, ω m represents the objective weight of the magnitude score, ω s represents the objective weight of the slope score, ω r represents the objective weight of the overall trend score, ω c represents the objective weight of the inflection point matching score, represents the objective phase score of k dimensions, represents the objective magnitude score of k dimensions, represents the objective slope score of the k dimension, E r represents the motion consistency score, E c Represents the inflection point matching score.

Images