Credibility measurement method and system for unmanned driving multi-mode perception algorithm

Abstract

The invention provides a credibility measurement method and system for an unmanned multi-mode perception algorithm, a data-driven virtual simulation test platform is constructed, and feature distribution is counted on the basis of actually-collected actual measurement data. Through dynamic model modeling and state transition, an end-to-end closed loop sensing process is described. Afterwards, for the problem that a traditional combined test method is low in efficiency due to the high dimension of scene parameters, a black box substitution model between the scene parameters and a perception test result is established, historical test data is evaluated by using a Bayesian optimization algorithm to generate the most antagonistic scene parameters in a next iteration test, and the efficiency of the traditional combined test method is improved. In this way, the confrontation of the test environment is continuously improved, and the test efficiency is effectively improved. And finally, a perception algorithm credibility measurement module is constructed, a perception failure scene is extracted through playback test data, a credibility evaluation index system is constructed, and comprehensive credibility evaluation of the tested perception system is realized.

Description

Technical Field

[0001] The present invention relates to the field of driverless vehicle testing and evaluation, and in particular to a credibility measurement method and system for a driverless multimodal perception algorithm. Background Art

[0002] Human-vehicle-road cooperative autonomous driving is a safety-critical system that is highly dependent on multiple sensors and intelligent algorithms. As the level of autonomous driving increases, the requirements for the comprehensiveness, accuracy, and timeliness of environmental information obtained by the perception system are becoming increasingly higher. The overall performance of autonomous vehicles depends largely on the performance of the perception system. To ensure the safe and reliable operation of autonomous vehicles, the operation process and output results of the perception system must meet the expected requirements, that is, be credible.

[0003] At present, the testing and evaluation methods of unmanned driving perception systems are mainly divided into three categories. The first category is mainly based on formal methods. Based on the model structure and logical mechanism of the perception system, a mathematical analytical model is constructed to traverse the state space of the system to identify and eliminate dangerous scenarios. It aims to detect design defects of the perception system in the early stage of system development, which helps to solve problems in advance and reduce the cost of later repairs. However, most of the unmanned driving perception systems at this stage are based on deep learning algorithms, and the internal structure and working principle of such algorithms are difficult to model. At the same time, the output results are greatly affected by the training data and show obvious cognitive uncertainty. Therefore, traditional formal methods are difficult to apply in the credibility evaluation of intelligent perception algorithms.

[0004] The second category is to conduct closed-field testing on unmanned vehicles equipped with perception systems. This method manually sets one or several sets of scene parameters in a real environment and conducts a pass-through test. This method is carried out in a controlled environment, and the experimental process is reproducible and repeatable, which can guarantee the validity of the test results to a certain extent. However, due to the limitations of closed venues, this method often cannot provide enough test mileage to cover various situations and scenarios, and cannot fully evaluate the performance of the perception system in a complex actual environment; the high cost of building and maintaining closed test sites further limits the scale of testing; the evaluation criteria usually include statistical indicators such as false detection rate, false negative rate, and error, which are difficult to fully reflect the credibility of the perception system in a dynamic, complex and uncertain real environment.

[0005] The third method is virtual simulation testing, which uses computers to abstract the actual traffic environment and build a simulation test system to complete the test. Domestic and foreign researchers have conducted a lot of research on traffic scenes and their data construction generation technology, using simulation engine generation and game model rendering methods to build virtual test scenes to obtain high-quality test data, and use large-scale generated test data for unmanned driving intelligent perception algorithm training and test verification. At the same time, this method allows the generation of diverse traffic scenes in a virtual simulation environment, which is crucial for evaluating the credibility of perception algorithms in multiple scenarios; the test environment is highly controllable, and testers can modify the parameters of the scene so that the algorithm can be repeatedly tested under controlled conditions to better evaluate the performance of the perception algorithm; compared with closed testing, virtual simulation testing is less expensive and effectively reduces testing costs. However, the authenticity of the virtual environment is crucial to the accuracy of the test results. The virtual environment needs to accurately and unbiasedly simulate various scene elements in the real world; at the same time, specific test execution technologies need to be considered to ensure that the dangerous state space of the perception algorithm is covered with a limited number of simulation tests.

[0006] Therefore, the present invention conducts credibility evaluation of multimodal intelligent perception algorithms for unmanned driving based on virtual simulation testing.

[0007] In response to the above-mentioned scene restoration problem, a general test scene generation technology for multimodal perception was developed. By summarizing the common characteristics of multimodal algorithms and analyzing the attribute characteristics and interaction relationships of various elements in the scene;

[0008] At the same time, the value range of simulation scene elements is defined based on the distribution of massive measured data. The constructed simulation scene is both realistic and can comprehensively test the performance of the perception algorithm.

[0009] In response to the problem of dangerous scene coverage mentioned above, an accelerated test method for intelligent perception algorithm evaluation was studied. The mapping relationship between scene parameters and perception results in a set of historical tests was modeled by Gaussian process, and the cumulative error of a complete test process was set as the cost function. Through the Bayesian optimization algorithm, the potential scene parameter δ was found to maximize the expected cost function, that is, the most adversarial scene was automatically selected as the next perception test scene, and multiple iterations were performed to determine whether the perception algorithm under test was poorly designed or poorly trained.

[0010] In addition, the present invention constructs a credibility measurement model for intelligent perception algorithms, considers the impact of perception failure on the safety of unmanned driving operations, and designs evaluation indicators to comprehensively evaluate the credibility of intelligent perception algorithms. Summary of the invention

[0011] The flowchart of the method for measuring the credibility of multimodal perception of human-vehicle-road cooperative unmanned driving in the present invention is as follows:Figure 1 As shown, it includes three steps: general test scenario generation, adversarial test method research, and credibility measurement model construction:

[0012] 1. Generic test scenario generation for multimodal perception

[0013] Based on massive measured data, the present invention constructs a general simulation test environment suitable for multimodal intelligent perception algorithms. First, data preprocessing is performed to extract scene element types and corresponding attribute feature values ​​based on measured data; then, the joint distribution of attribute values ​​of each type of elements in the test scene is determined, and random sampling based on the joint distribution can obtain a simulation test scene that restores the real world, thus completing the construction of a general test scene.

[0014] (1) Measured data preprocessing

[0015] In order to ensure that the general test scene is unbiased, the present invention performs data preprocessing based on the collected measured data (images, point cloud data, etc.), sorts out the scene elements in the real world, and extracts the attribute feature values ​​of the scene elements according to the element type.

[0016] 1) Sorting out scene elements

[0017] According to the definition of scenes in ISO 21488, the present invention divides scene elements into entity class, environment class and interference class, as shown in Table 1. The entity class is an element that interacts with the driverless car during dynamic driving and has specific physical properties. It must completely cover all expected perception contents of the perception system under test and cover various traffic elements in the real traffic environment as comprehensively as possible. The environment class is a scene element that significantly affects the perception system in achieving its expected functions. The interference class is generally not used as the expected perception content of the perception system, but this class may affect the perception system in achieving its expected functions.

[0018] Table 1 Summarization of scene elements

[0019]

[0020] 2) Extraction of scene element attribute feature values

[0021] For the specific scene elements in Table 1, it is necessary to extract the characteristic values ​​of the attribute features that affect the performance of the multimodal intelligent perception algorithm. For entity and interference classes, the attribute features include reflectivity, reflection area, data generation, and variability features; for environment classes, the attribute features include reflectivity and transmittance. For example, Tables 2 and 3 respectively give the attribute features of specific elements such as motor vehicles and rainfall. Subsequently, the corresponding feature extraction algorithms such as convolutional neural networks are used from the measured data to extract the characteristic values ​​of the corresponding attributes.

[0022] Table 2 Attribute characteristics of motor vehicles

[0023]

[0024]

[0025] Table 3 Rainfall attribute characteristics

[0026]

[0027] (2) Calculation of joint distribution of simulation test scenario parameters

[0028] To generate a simulation test scene, a set of determined scene parameters δ is required, that is, a set of determined scene element attribute feature values, assuming that there are m in total (such as vehicle surface material, vehicle color, vehicle structure, rainfall composition, rainfall density, etc.). To ensure the unbiasedness of the simulation test, a joint probability distribution of m attribute features should be constructed, and a set of scene parameters should be randomly sampled from the distribution. The specific steps are as follows:

[0029] a. Feature type classification and preprocessing

[0030] When constructing the joint probability distribution of attribute features, the attributes include two types of features: discrete and continuous.

[0031] Discrete features: features with a finite number of possible values, such as the color of a vehicle (black, white, gray), the surface material of a vehicle (metal, plastic, etc.). During preprocessing, label encoding is used to convert category labels into numerical form.

[0032] Continuous features: features with infinite possible values, such as vehicle speed, vehicle heading angle, etc. During preprocessing, each type of feature value is normalized to ensure that different features have the same scale.

[0033] b. Probability distribution modeling of attribute features

[0034] Discrete features: Probability Mass Function (PMF) is used for modeling, and the frequency of occurrence of each possible value in the measured data is calculated as its probability:

[0035]

[0036] Among them, c k is the kth category, and n is the total number of attribute values ​​of this feature in the measured data.

[0037] Continuous features: Kernel Density Estimation (KDE) is used to model each type of attribute feature, and Gaussian kernel function is selected Select the bandwidth parameter h and calculate its probability density:

[0038]

[0039] in, Indicates that the feature is in X i Density estimate of location, X j The eigenvalue of the jth sample point in the measured data, n is the total number of attribute values ​​of the feature in the measured data, indicating how many X j is used to construct a density estimate for this feature.

[0040] c. Construction of Joint Probability Distribution

[0041] Combine m discrete and continuous features to construct a mixed joint probability distribution. The independent parameter group I contains k independent discrete and continuous parameters; the dependent parameter group D contains l discrete and continuous parameters that depend on independent parameters or other features, where the sum of l and k is m. The joint probability distribution of the independent parameter group I is:

[0042] P(I)=P(X i1 )·P(X i2 )·...·P(X ik )

[0043] in, is the marginal distribution of k independent parameters, using P(X i )(discrete feature) or (continuous features) are calculated. The joint probability distribution of the dependent parameter group D is:

[0044] P(D|I)=P(X d1 ,X d2 ,…,X dl |X i1 ,X i2 ,…,X ik )

[0045] Among them, X d1 ,X d2 ,…,X dl The discrete or continuous features belong to the dependent parameter group D, and l is the number of dependent parameters. Finally, the joint probability distribution is expressed as follows. The schematic diagram of the joint probability distribution is as follows Figure 2 As shown:

[0046] P(X1,X2,…,X m )=P(I)·P(D|I)

[0047] In the subsequent intelligent perception algorithm simulation test process, a set of existing perception test results is required when constructing Bayesian optimization, that is, several sets of scene parameters δ (row vectors, a set of parameter values ​​containing all scene element items) are obtained by random sampling from the joint probability distribution constructed in this section several times, and specific test cases are generated respectively, and perception tests are performed to obtain corresponding perception test results. Each set of perception test results maps different scene parameters δ to the value of the cost function J.

[0048] 2. Design of an integrated framework for intelligent perception algorithm evaluation

[0049] The above-mentioned general test scenario generation process realizes the unbiased construction of test cases in a simulation environment based on measured data. However, massive test cases are inexhaustible. How to evaluate the performance of intelligent perception algorithms through limited testing will become the key to this invention. This section will study the adversarial scenario generation strategy based on the Bayesian optimization algorithm by constructing a dynamic model of simulation test scenarios to achieve the purpose of accelerating adversarial testing.

[0050] (1) Construction of dynamic model of simulation test scenario

[0051] In this step, the state of the simulated test scene, the state of the unmanned vehicle, and the output results of the perception algorithm under test are modeled as a dynamic model. The first two state variables are determined by a set of test scene parameters δ, and the output results are determined by the perception algorithm under test. The cumulative error of the output results is modeled as the cost function J of the dynamic model. The Bayesian optimization method in 2.2 is used to iterate and generate the value of δ to maximize J. The schematic diagram of dynamic model modeling is shown in the figure. Figure 6 shown.

[0052] The modeling process is as follows:

[0053] Order t is the state vector of the simulation test scene at time t, including three types of scene elements in the scene, which is defined as s t ={s interactive_entity ,s disturbing_entity ,s environmental_entity}, where s interactive_entity is an entity class, belonging to all the expected perception contents of the perception system under test, s disturbing_entity is the interference type, s environmental_entity is the environment class. Let x t is the state vector of the unmanned vehicle at t where the intelligent perception algorithm under test is located, which involves the position and direction of the unmanned vehicle. Both of the above initial states are determined by the initial scene parameter δ.

[0054] The process of the algorithm being tested perceiving the surrounding environment is described by o t ~h(o|xt ,s t ) indicates that t represents the output of the algorithm under test. The dynamics model of the unmanned vehicle in which the algorithm under test is located is represented by x t+1 ~p(x|x t ,a t ), where the vehicle behavior is represented by a t ~π θ (a|o t ) is determined by the strategy at which the strategy depends on the output of the perception module o t The transfer model of the state vector of the simulation test scenario is s t+1 ~q(x|s t ,u t ), the state transfer process is composed of Since the present invention evaluates the intelligent perception algorithm, the policy control parameters and θ are set as constants. Therefore, the simulation joint state time evolution of the multimodal intelligent perception algorithm test system in the present invention is given by the following generative model, and the meaning of each state vector is as follows: Figure 6 As shown:

[0055] δ~(x0,s0)

[0056] o t ~h(o|x t ,s t )

[0057] a t ~π θ (a|o t )

[0058]

[0059] x t+1 ~p(x|x t ,a t )

[0060] s t+1 ~q(x|s t ,u t )

[0061] x t : represents the state vector of the unmanned vehicle where the algorithm under test is located at time t, including position, speed, etc.;

[0062] s t : represents the state vector of the simulation test scenario at time t, including all entity, interference and environment states;

[0063] o t: represents the output result of the algorithm under test at time t, such as the detected target, obstacle, etc.;

[0064] h(o|x t ,s t ): indicates that given x t and t When the measured perception algorithm outputs o t The probability distribution function of

[0065] a t : represents the behavior decision of the unmanned vehicle where the algorithm under test is located at time t, including steering, acceleration, etc.;

[0066] π θ (a|o t ): indicates that it is based on o t To generate driverless vehicle behavior a t Strategy model, θ represents the control parameter;

[0067] u t : represents the strategy output of the control test scenario state transition at time t, including the change of the motion state of the surrounding motor vehicles and the change of the environment state;

[0068] Indicates based on s t and x t To determine the scene change u t The strategy model, Indicates control parameters;

[0069] x t+1 : represents the state of the unmanned vehicle at time t+1, which is represented by the current state x t and behavior a t Determine, through the transfer function p(x|x t ,a t )produce;

[0070] s t+1 : represents the state of the simulation test scenario at time t+1, which is represented by the current state s t and scene changes t Determine, through the transfer function q(x|s t ,u t )describe.

[0071] The cost function of the dynamic model is modeled as the cumulative error of the output of the perception module. The formula is as follows:

[0072]

[0073] Where c(o t ,st ) is the cost function at time t, which indicates the degree of deviation between the output perception result of the tested algorithm and the true value at that time. and θ are constants, so the instantaneous cost function is determined by the initial scene parameters δ.

[0074] The perception error is defined as existence error, i.e., missed detection (False Negative, FN, obstacles that actually exist are not detected) and false detection (False Positive, FP, obstacles that actually do not exist are mistakenly detected); classification error, i.e., the output result classification error;

[0075] Measurement error refers to the error in the position, speed, size and other parameters of the output results.

[0076] c(o t ,s t )The calculation process is as follows:

[0077]

[0078]

[0079]

[0080]

[0081] Among them, considering that missed detection has a great impact on the safety of driverless cars, the instantaneous cost function is set to the highest when a missed detection occurs, that is, Error m . Use w u 、w c 、w s represents the weights of existence error, classification error and state error, and the instantaneous error value is obtained by weighted average method. Indicates the distance between the misdetected target and the unmanned vehicle. The closer the distance, the higher the corresponding error score. Indicates the number of misclassified targets, sum total Represents the total number of targets. The more the number of classification errors, the higher the corresponding error score. and They respectively represent the measured value and the true value when the perception result is classified correctly. The greater the proportion of the average measurement error to the true value, the higher the corresponding error score.

[0082] In the subsequent generation of adversarial scenarios based on the Bayesian optimization algorithm, the present invention continuously adjusts δ to generate adversarial scenarios that are unfavorable to the intelligent perception algorithm, continuously reduces the accuracy of the output results of the perception module, and finds the performance boundary of the perception system under test and quantifies its credibility level. Therefore, when setting the cost function, it is necessary to maximize the cumulative error of the perception result, which is expressed as follows:

[0083] δ * = argmax δ J

[0084] (2) Adversarial scenario generation based on Bayesian optimization algorithm

[0085] Based on the Bayesian optimization algorithm, this section determines the initial scenario configuration strategy δ for the next round of simulation tests by modeling the prior distribution and acquiring the function. k+1 , the specific algorithm framework is shown in the following table:

[0086] Table 4 Bayesian optimization algorithm framework

[0087]

[0088] The meaning of each symbol is as follows:

[0089] 1. δ is a set of scenario parameters, and each set of δ generates a simulation test scenario;

[0090] 2.J is the cost function of the dynamic model of the simulation test scenario, Represents parameter δ j The corresponding cost function value;

[0091] 3.D is the historical observation set of the intelligent perception algorithm simulation test platform;

[0092] 4. EI is the acquisition function, that is, the expected improvement algorithm ExpectedImprovement (EI).

[0093] Based on the above steps, the content of this section is divided into establishing probability distribution for historical observation states and using acquisition function to generate the scenario parameter δ for the next iteration.

[0094] 1) Bayesian optimization prior distribution modeling

[0095] In the first row of Table 4, a set of common test scenarios (assuming there are n scenarios) is obtained by randomly sampling the content of Part 1 and performing the test to obtain the data set Use Gaussian process to establish an accurate probability distribution model for the data set D as the prior distribution for Bayesian optimization. Use ARD Matern 5 / 2 as the kernel function to characterize the nonlinear relationship between the output results of the tested perception algorithm and the scene parameter δ. The modeling process is as follows:

[0096]

[0097]

[0098]

[0099] Among them, K(δ,δ ′ ) is the value of the kernel function, which represents two points δ and δ in the input feature space ′ δ represents the scene parameters in this simulation test. 2 is the variance parameter of the kernel function, which controls the scale of the kernel function; l is a function of the feature space distance, allowing different dimensions to have different length scales; β is the noise variance in the observations; β, l, σ are hyperparameters of the Gaussian process and are determined by random variational inference techniques.

[0100] After completing Gaussian process modeling, the model outputs the predicted mean μ(δ) and predicted variance σ at each δ 2 (δ). The mean represents the predicted value of the cost function J at δ given the current dataset D; the variance represents the uncertainty at δ, reflecting the confidence of the model in the prediction at that point. A larger variance means that the model has a higher prediction uncertainty at that point, which encourages the Bayesian optimization algorithm to explore more in this area. The mean and variance are used by the Bayesian optimization algorithm to generate the scenario parameter δ in the fourth row of Table 4. j .

[0101] 2) Bayesian optimization acquisition function modeling

[0102] In the fourth row of Table 4, based on the predicted mean and variance output by the Gaussian process model, the acquisition function α(δ) is constructed to determine the selection of the scene parameter δ in the next test. The acquisition function needs to make a trade-off between "exploration" (exploring areas with high uncertainty) and "utilization" (utilizing areas near the current optimal point). In the present invention, the acquisition function adopts the expected improvement algorithm, and the modeling process is as follows:

[0103]

[0104]

[0105] Among them, I(δ) is the improvement function of the cost function at δ, J *is the best value in the test results of the current data set D; is the Gaussian process model's prediction of the cost function, and its distribution is:

[0106]

[0107] because The distribution of is Gaussian distribution, and the expected improvement function α EI (δ) can be calculated analytically by the integral property of Gaussian distribution, and the analytical expression of EI is as follows:

[0108]

[0109] in:

[0110] μ(δ) is the predicted mean of the cost function J by the Gaussian process at the scene parameter δ;

[0111] σ(δ) is the standard deviation of the Gaussian process prediction of J at δ;

[0112] Φ is the cumulative distribution function of the standard normal distribution;

[0113] φ is the probability density function of the standard normal distribution;

[0114] in, Used to measure the predicted value exceeding J at parameter δ * The probability of , reflects the “utilization” component, that is, selecting points with a higher probability of bringing an expected improvement in the cost function value; Used to measure the accuracy of the predicted value at parameter δ * The probability density of , reflects the “exploration” component, that is, exploring in high uncertainty areas to reduce uncertainty and possibly discover better improvements. Then, we look for α EI (δ) maximizes The present invention selects the gradient ascent method to utilize α EI The gradient information of (δ) is gradually updated to approach the maximum value. This completes the content of the fourth row of Table 4, and determines the scenario parameter δ that maximizes the improvement of the expected cost function value. * .

[0115] In the contents of rows 5-7 of Table 4, the scenario parameters δ output by the Bayesian optimization algorithm are * Conduct intelligent perception algorithm simulation test and obtain the new cost function value y=J(δ * ) and add it to the dataset D. The Gaussian process model is updated based on the newly acquired dataset D, and then a new round of iterations is performed back to row 4 of Table 4 until the K iteration requirement is met. This method can continuously generate scenario parameters δ that are adversarial to the perception algorithm under test, effectively improving the test efficiency.

[0116] 3. Construction of credibility measurement model of intelligent perception algorithm

[0117] Based on the contents of Parts 1 and 2, K groups of adversarial environments are obtained, and as the Bayesian optimization is iterated, the cost function value of each group increases (that is, the overall level of accuracy of the perception results in the clips decreases), as shown in Figure 7 The output result of each group is a continuous perception result segment with a fixed length. If it exceeds the preset perception failure threshold, it is considered a perception failure scenario. For example, Figure 8 This section will build a credibility evaluation index system to comprehensively evaluate the credibility of the intelligent perception algorithm based on the K groups of test results obtained in the previous steps.

[0118] The present invention constructs a credibility measurement model for intelligent perception algorithms based on exposure rate, penetration rate, and hazard rate, taking into account the error of the perception results themselves and their impact on the safety and stability of the operation of driverless cars, and comprehensively evaluates their credibility. The following are the general steps:

[0119] a. Exposure rate assessment: First, the perception algorithm failure threshold τ is set using expert experience, and the perception failure segment is determined by replaying the Log data recorded in the simulation test. The exposure rate represents the proportion of the duration of the perception failure segment to the total time, and the exposure rate level E is determined based on this.

[0120] b. Penetration rate evaluation: Penetration rate indicates the degree of decrease in the accuracy of the perception results. The three indicators of existence error, classification error and measurement error are selected to evaluate the penetration rate, and the weighted average method is used for calculation. The formula is as follows:

[0121] P=w u ×Score u +w c ×Score c +w s ×Score s

[0122] Among them, Score u Indicates the existence error score, Score c Represents the classification error score, Score s Denotes the measurement error score, using w u 、w c 、w s They represent the relative importance of existence error, classification error, and measurement error, respectively.

[0123] c. Hazard rate assessment: The hazard rate indicates the degree to which the output of the driverless planning control system is disturbed by the disturbance of the perception algorithm output. H is used to represent the hazard rate, and the calculation formula is as follows:

[0124]

[0125] Among them, S d It represents the total number of times in the simulation test that the TTC between the unmanned vehicle under the tested algorithm and the vehicle in front is less than 0.5s, and K represents the total number of tests performed.

[0126] d. Comprehensive evaluation: The credibility of the tested intelligent perception algorithm is calculated by weighting the scores of each dimension. The calculation formula is as follows:

[0127] Trustworthy=E×w E +P×w P +H×w H

[0128] Among them, w E 、w P 、w H They represent the weights corresponding to the exposure rate, penetration rate, and hazard rate, respectively. E, P, and H represent the scores corresponding to the exposure rate, penetration rate, and hazard rate calculated based on the test results of group K, respectively, and are normalized. Trustworthy represents the credibility of the tested intelligent perception algorithm.

[0129] The present invention also provides a device and system for evaluating the credibility of unmanned driving intelligent perception algorithms, such as Figure 5 This system adopts a cloud platform distributed simulation architecture, builds a simulation test scenario generation system based on the cloud platform, and builds an intelligent perception algorithm simulation test platform, a scenario configuration parameter optimization device, and an intelligent perception algorithm credibility evaluation system based on a local high-performance computing (HPC) cluster:

[0130] 1. Simulation test scenario generation system: This device analyzes and processes massive measured data, receives information from the scenario configuration parameter optimization device, edits a set of simulation scenario configuration files according to the specified scenario parameters, and inputs them into the intelligent perception algorithm simulation test platform. The module includes the following units:

[0131] (1) Data storage unit: used to store a large amount of real-world measured data, including data collected on open roads, data collected on closed sites, standards and regulations, traffic accident data, infrastructure data, meteorological data, and sensor data.

[0132] (2) Data processing unit: A module used to analyze and process the measured data in the data storage unit. It is responsible for extracting information from the original unstructured data and providing basic data for scene generation.

[0133] (3) Scenario editing unit: edit the simulation configuration file according to the scenario parameter requirements and output it to the simulation test platform.

[0134] 2. Intelligent perception algorithm simulation test platform: receives a set of configuration files from the simulation test scenario generation system, generates a simulation test environment with high fidelity, accesses the intelligent perception algorithm under test and performs simulation tests. The platform includes the following units:

[0135] (1) Simulation framework: The core of the simulation platform, responsible for coordinating the work of various modules and controlling the simulation process.

[0136] (2) Physics engine: simulates the movement, interaction, and V2X communication of objects in the simulation environment to ensure that the physical behavior in the simulation environment conforms to objective laws.

[0137] (3) Graphics Engine: Renders and generates virtual scenes with high fidelity, including sensor simulation, static scene restoration, and dynamic scene restoration.

[0138] (4) Perception algorithm interface unit: accesses the intelligent perception algorithm under test and integrates it into the simulation test platform.

[0139] 3. Scene configuration parameter optimization device: Receive the intelligent perception algorithm simulation test process and results, and build them into a dynamic model. Combine the historical simulation scene parameters and the corresponding test results, use the Bayesian optimization algorithm to construct the Gaussian process prior and acquisition function respectively, and maximize the acquisition function to obtain the current optimal scene parameters, and input the parameters into the scene editing unit.

[0140] (1) Dynamic model modeling module: The interaction between each element in the scene and the unmanned driving perception system over time during the simulation test is modeled as a dynamic model, and the corresponding cost function is designed.

[0141] (2) Prior distribution modeling module: A Gaussian process prior model is constructed based on the cost function. This model is used to estimate the nonlinear relationship between the output results of the intelligent perception algorithm and the scene parameters.

[0142] (3) Acquisition function modeling module: Receive the mean and variance information of the Gaussian process prior model, construct an acquisition function and find the maximum value of the acquisition function. This value corresponds to the most adversarial scenario configuration parameter in the next simulation test, and outputs the parameter to the simulation test scenario generation system.

[0143] 4. Intelligent perception algorithm credibility evaluation system: After the Bayesian optimization iteration process is completed, the virtual simulation test results are analyzed, and an evaluation index system is constructed to output the credibility comprehensive evaluation results. The system includes the following units:

[0144] (1) Log data storage unit: mainly responsible for collecting, storing and managing all logs and test data generated during the virtual simulation test, including the operation process and results of the tested perception algorithm in different scenarios. It supports the function of fast retrieval and backtracking of historical test data.

[0145] (2) Credibility comprehensive evaluation unit: Based on the simulation test process and results combined with the evaluation index system, it outputs the comprehensive evaluation results of the credibility of the intelligent perception algorithm.

[0146] The credibility evaluation method, device and system provided by the present invention have the following technical key points and advantages:

[0147] (1) The present invention constructs a test scene in a virtual simulation environment based on massive measured data, which is highly controllable while ensuring the fidelity of scene restoration.

[0148] (2) The present invention adopts the Bayesian optimization algorithm to iteratively optimize the cost function, so that the generated simulation scene has increasingly enhanced adversarial properties, and the ability boundary of the perception algorithm can be quickly obtained by using a limited number of tests. Figure 7 By using a finite number of test iterations, the upper bound of the cost function (the level of overall perceived performance degradation) is obtained, which can effectively improve the test evaluation efficiency.

[0149] (3) The test and evaluation of existing autonomous driving perception algorithms are generally based on fixed collected images or point clouds, which are open-loop evaluations. That is, the perception results will not affect the behavioral decisions of all traffic participants in the scene, and therefore cannot meet the requirements of end-to-end autonomous driving. This study, by building a virtual simulation test environment and dynamic modeling, depicts a dynamic process of "perception results affecting behavioral choices - behavioral choices affecting the perception input of the next frame", which can meet the closed-loop evaluation requirements of end-to-end autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0150] Figure 1 This is the overall flow chart of the credibility evaluation of the intelligent perception algorithm of the present invention;

[0151] Figure 2 Generate a general test scenario process for multimodal perception of the present invention;

[0152] Figure 3 This is a schematic diagram of the integrated framework design of the intelligent perception algorithm evaluation of the present invention;

[0153] Figure 4 A process for constructing a credibility measurement model for the intelligent perception algorithm of the present invention;

[0154] Figure 5 It is a schematic diagram of the credibility evaluation device and system of the intelligent perception algorithm of the present invention;

[0155] Figure 6 Schematic diagram of dynamic model building;

[0156] Figure 7 Schematic diagram of the Bayesian optimization process;

[0157] Figure 8 Schematic diagram of perceptual failure segment extraction. DETAILED DESCRIPTION

[0158] The present invention relates to a credibility measurement method, device and system for multimodal perception of human-vehicle-road collaborative unmanned driving. The specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0159] This example conducts a credibility evaluation on the 3D point cloud target detection algorithm PointPillars.

[0160] Among them, the simulation test scenario generation system is deployed in the cloud platform, which includes a massive data storage unit, a data processing unit, and a scenario editing unit;

[0161] The intelligent perception algorithm simulation test platform is developed based on the Unreal Engine and deployed in the local server. It receives the scene configuration file from the cloud platform and performs the intelligent perception algorithm simulation test.

[0162] The scene configuration parameter optimization device is deployed in the local server, and the high-performance computing unit supports the dynamic model modeling module, the prior distribution modeling module and the acquisition function modeling module;

[0163] The intelligent perception algorithm credibility evaluation system is deployed in the local server, including a log data storage unit that supports high bandwidth, high throughput, and low latency, as well as a simulation test result analysis unit and a credibility comprehensive evaluation unit. The modules deployed in the local server transmit and communicate with the cloud platform, together forming the multimodal intelligent perception credibility evaluation system proposed in the present invention.

[0164] The massive measured data in this example comes from:

[0165] Vehicle-side data, test data of open test roads for intelligent connected vehicles in Jiading District, Shanghai, and closed test data of closed test areas dedicated for intelligent connected vehicles in Jiading District, Shanghai;

[0166] Roadside data is collected through multi-source and multi-modal roadside sensing equipment from the Intelligent Connected Vehicle Demonstration Zone in Jiading District, Shanghai. The demonstration zone includes demonstration operations of more than 700 types of automatic vehicles, including driverless taxis, intelligent connected buses, smart trains, smart cleaning, and smart delivery.

[0167] Based on the above device and system, Figures 1 to 8The implementation method of the credibility measurement method for multimodal perception of human-vehicle-road collaborative unmanned driving includes the following steps:

[0168] Step 1: Build a general test scenario for intelligent perception algorithms based on massive measured data. The specific steps are as follows:

[0169] Step 1.1: Preprocess the measured data to extract specific element items and corresponding attribute feature values ​​in the real scene.

[0170] The data sources of this embodiment come from vehicle-side data and roadside data. The data types include structured and unstructured data, and the data modalities include video, 3D point cloud, etc. The above data are all stored in the storage server of the cloud platform. Through the data processing unit in the cloud platform, the corresponding attributes are extracted according to the predefined three types of data sources, and the three stages of data cleaning, feature extraction and data storage are successively carried out to convert the unstructured measured data into structured scene element feature values.

[0171] Step 1.2: Calculate the distribution of each scene element attribute feature value separately, and calculate their joint distribution.

[0172] Based on the scene element attribute characteristic values ​​obtained from the real scene in step 1.1, the data processing unit uses the probability mass function and kernel density estimation method to fit the characteristic value distribution of each element in the scene, and combines discrete and continuous features to construct a mixed joint probability distribution.

[0173] Step 1.3: Based on the intelligent perception algorithm simulation test platform, construct an intelligent perception algorithm simulation test scenario according to the scenario parameter δ, which specifically includes the following steps:

[0174] The scene element parameter δ is obtained from the joint distribution function of the attribute eigenvalues ​​by random sampling, and the virtual simulation test scene is rendered using the physics engine and the graphics engine.

[0175] Step 1.4: After the simulation test scenario is determined, the intelligent perception algorithm to be tested is connected to the simulation platform to start testing.

[0176] The intelligent perception algorithm selected for test in this embodiment is PointPillars, and the perception algorithm for test uses the pre-training model weights disclosed in existing research.

[0177] Step 2: Build an integrated framework for intelligent perception algorithm evaluation. The specific steps are as follows:

[0178] The scene configuration parameter optimization device receives the test results and scene parameter settings from the simulation platform, generates adversarial scene parameters through dynamic modeling, prior distribution modeling and acquisition function modeling, and guides the generation of the next simulation test scene. The specific steps are as follows:

[0179] Step 2.1: Model the interaction between each scene element and the unmanned vehicle in which the perception algorithm under test is located during the simulation test as a dynamic model:

[0180] δ~(x0,s0)

[0181] o t ~h(o|x t ,s t )

[0182] a t ~π θ (a|o t )

[0183]

[0184] x t+1 ~p(x|x t ,a t )

[0185] s t+1 ~q(x|s t ,u t )

[0186] Among them, δ determines the initial state of the state vector s of the simulation test scene element and the state vector x of the unmanned vehicle where the algorithm under test is located; h(o|x t ,s t ) means that given x t and t When the perception algorithm under test outputs the result o t The probability distribution function of θ (a|o t ) indicates that it is based on o t To generate driverless vehicle behavior a t Strategy model, θ represents the control parameter; Indicates based on s t and x t To determine the scene change u t The strategy model, Indicates control parameters.

[0187] The cost function of the dynamic model is modeled as the cumulative error of the output of the perception module. The formula is as follows:

[0188]

[0189] Where c(o t ,s t ) is the cost function at time t, which indicates the degree of deviation between the output perception result of the tested algorithm and the true value at that time. and θ are constants, so the instantaneous cost function is determined by the initial scene parameter δ. The perception error is defined as existence error, i.e., missed detection (False Negative, FN, obstacles that actually exist are not detected) and false detection (False Positive, FP, obstacles that actually do not exist are mistakenly detected); classification error, i.e., the classification error of the output result; and measurement error, i.e., the error of the position, speed, and size of the output result. c(o t ,s t )The calculation process is as follows:

[0190]

[0191]

[0192]

[0193]

[0194] Among them, considering that missed detection has a great impact on the safety of driverless cars, the instantaneous cost function is set to the highest when a missed detection occurs, that is, Error m . Use w u 、w c 、w s represents the weights of existence error, classification error and state error, and the instantaneous error value is obtained by weighted average method. Indicates the distance between the misdetected target and the unmanned vehicle. The closer the distance, the higher the corresponding error score. Indicates the number of misclassified targets, sum total Represents the total number of targets. The more the number of classification errors, the higher the corresponding error score. and They respectively represent the measured value and the true value when the perception result is classified correctly. The greater the proportion of the average measurement error to the true value, the higher the corresponding error score.

[0195] Step 2.2, receive the scene parameter δ and the dynamic model cost function J, randomly sample the test cases generated based on steps 1.3 and 1.4 and execute the test to obtain the data set D consisting of the corresponding perception results, use ARD Matern 5 / 2 as the kernel function to build a Gaussian process prior model to characterize the nonlinear relationship between the output results of the intelligent perception algorithm and the scene parameter δ. The modeling process is as follows:

[0196]

[0197]

[0198]

[0199] Among them, K(δ,δ ′ ) is the value of the kernel function, which represents two points δ and δ in the input feature space ′ The similarity between them, δ represents the initial scenario parameter configuration in this simulation test, σ 2 is the variance parameter of the kernel function, which controls the scale of the kernel function, l is a function of the feature space distance, allowing different dimensions to have different length scales, and β is the noise variance in the observations.

[0200] Based on the simulation test data and noise data generated by the new scenario parameter configuration δ, the Gaussian process regression algorithm is used to infer the posterior distribution of the cost function.

[0201] The hyperparameters β, l, σ of the above Gaussian process regression are obtained by maximizing the logarithmic marginal likelihood function logp(y i |δ,β,l,σ) and introduces the variational distribution solution using variational inference techniques.

[0202] Step 2.3: Based on the mean and variance of the Gaussian process obtained in step 2.2, use the expected improvement algorithm to create the acquisition function α(δ). The modeling process is as follows:

[0203]

[0204] I(δ)=max(yJ * ,0)

[0205] Where I(δ) is the improvement function of the cost function at δ, J * is the best value in the current test result. Given the historical test result data set D = {δ j ,y j} j＝1…k For the expectation of the distribution on y, due to the Gaussian assumption of the Gaussian process posterior, we can analyze α EI The maximum value of the expected improvement function will correspond to the most adversarial scenario configuration strategy δ in the next round of simulation tests. k+1 , and output the strategy to the scenario generation system.

[0206] Step 3: Conduct credibility evaluation based on the intelligent perception algorithm simulation test results. The specific steps are as follows:

[0207] The credibility comprehensive evaluation unit is based on the test process and results recorded in the Log data storage unit, and selects three indicators such as exposure rate, penetration rate and hazard rate to construct a credibility evaluation system. On the one hand, it evaluates the accuracy of the perception results, and on the other hand, it considers the impact of perception failure on the safety and stability of the unmanned vehicle where the algorithm is tested, and comprehensively evaluates the credibility of the intelligent perception algorithm. The specific steps are as follows:

[0208] a. Exposure rate assessment: First, the perception algorithm failure threshold τ is set using expert experience, and the perception failure segment is determined by replaying the Log data recorded in the simulation test. The exposure rate represents the proportion of the duration of the perception failure segment to the total time, and the exposure rate level E is determined based on this.

[0209] b. Penetration rate evaluation: Penetration rate indicates the degree of decrease in the accuracy of the perception results. The three indicators of existence error, classification error and measurement error are selected to evaluate the penetration rate, and the weighted average method is used for calculation. The formula is as follows:

[0210] P=w u ×Score u +w c ×Score c +w s ×Score s

[0211] Among them, Score u Indicates the existence error score, Score c Represents the classification error score, Score s Denotes the measurement error score, using w u 、w c 、w s They represent the relative importance of existence error, classification error, and measurement error, respectively.

[0212] c. Hazard rate assessment: The hazard rate indicates the degree to which the output of the driverless planning control system is disturbed by the disturbance of the perception algorithm output. H is used to represent the hazard rate, and the calculation formula is as follows:

[0213]

[0214] Among them, S d It represents the total number of times in the simulation test that the TTC between the unmanned vehicle under the tested algorithm and the vehicle in front is less than 0.5s, and K represents the total number of tests performed.

[0215] d. Comprehensive evaluation: The credibility of the tested intelligent perception algorithm is calculated by weighting the scores of each dimension. The calculation formula is as follows:

[0216] Trustworthy=E×w E +P×wP +H×w H

[0217] Among them, w E 、w P 、w H They represent the weights corresponding to the exposure rate, penetration rate, and hazard rate, respectively. E, P, and H represent the scores corresponding to the exposure rate, penetration rate, and hazard rate calculated based on the test results of group K, respectively, and are normalized. Trustworthy represents the credibility of the tested intelligent perception algorithm.

Claims

1. A credibility measurement method for an unmanned driving multimodal perception algorithm comprises the following steps: Step 1: Build a general test scenario for the intelligent perception algorithm based on measured data: Step 1.1: Preprocess the measured data to extract specific elements and corresponding attribute feature values ​​in the real scene; Step 1.2, respectively calculate the distribution of the attribute feature value of each scene element in the real scene, and calculate their joint distribution; Step 1.3, set the simulation test scene elements according to the specific elements in the real scene, and the attribute characteristic value is controlled by the simulation scene parameter δ; randomly sample the joint distribution obtained from the real scene several times, and obtain a scene parameter δ each time by random sampling, so as to construct a set of simulation test cases, connect to the tested intelligent perception algorithm to perform the test, and obtain the corresponding perception test results; Step 2: Design of an integrated framework for intelligent perception algorithm evaluation: Step 2.1: The state of the simulated test scene, the state of the unmanned vehicle, and the output result of the perception algorithm under test are modeled as a dynamic model. The first two state variables are determined by the test scene parameter δ, and the output result is determined by the perception algorithm under test. The cumulative error of the output result is modeled as the cost function J of the dynamic model. The value of δ is continuously iterated through the Bayesian optimization method composed of steps 2.2 and 2.3 to maximize J. Step 2.2: Based on a set of perception test results obtained in 1.3, select ARD Matern 5 / 2 as the kernel function and initialize the Gaussian process model; Step 2.3: Based on the cost function J, the predicted mean μ(δ) and variance σ can be provided for any scenario parameter δ. 2 (δ) Gaussian process model, select the expected improvement function α EI (δ) is used as the acquisition function of Bayesian optimization, and a numerical optimization method is used to find the maximum value of α EI (δ), and output δ to the scenario generation system to guide the next round of test scenario generation; Step 3: Construction of intelligent perception algorithm credibility measurement model: Step 3.1, extracting the perceived failure fragment based on the simulation test results; Step 3.2: Construct a credibility evaluation index system to comprehensively evaluate the credibility of the intelligent perception algorithm.

2. The credibility measurement method for unmanned driving multimodal perception algorithm according to claim 1 is characterized in that: The step 1.1 specifically comprises the following steps: Step 1.1.1, combing scene elements: according to the multimodal data actually collected, the elements appearing in the scene are divided into three categories: entity, environment, and interference, and the specific elements are refined by sub-category; Step 1.1.2, scene element attribute feature extraction: select the attributes that affect the performance of the multimodal intelligent perception algorithm in each type of scene specific element items, and use the feature extraction algorithm to extract the characteristic values ​​of each type of scene specific element attributes from the measured data.

3. The credibility measurement method for unmanned driving multimodal perception algorithm according to claim 1 is characterized in that: The step 1.2 specifically comprises the following steps: Step 1.2.1, distinguish the discrete features and continuous features in the scene element features, and use the probability mass function PMF to model the discrete features and the kernel density estimation KDE to model the continuous features; Step 1.2.2: Combine discrete and continuous features to construct a mixed joint probability distribution.

4. The credibility measurement method for unmanned driving multimodal perception algorithm according to claim 1 is characterized in that: The specific process of step 2.3 is as follows: Bayesian optimization acquisition function modeling: Use the expected improvement function as the acquisition function, which calculates a set of potential scenario parameters δ compared to the currently known cost function J max The expected improvement α EI (δ), which makes the cost function value at the current maximum value J max continue to increase under the premise of.

5. The credibility measurement method for unmanned driving multimodal perception algorithm according to claim 1 is characterized in that: The specific process of step 3.1 is as follows: The expert experience method is used to set the perception failure threshold, and the perception failure fragment is extracted based on the simulation process recorded during the test and the corresponding perception results.

6. The credibility measurement method for unmanned driving multimodal perception algorithm according to claim 1 is characterized in that: The specific process of step 3.2 is as follows: A credibility evaluation index system is constructed based on exposure rate, penetration rate and hazard rate. On the one hand, it evaluates the accuracy of the perception results. On the other hand, it considers the impact of perception failure on the safety and stability of the unmanned vehicle in which the algorithm is tested, and comprehensively evaluates the credibility of the intelligent perception algorithm.

7. A credibility measurement system for unmanned driving multimodal perception algorithm, used to implement the credibility measurement method for unmanned driving multimodal perception algorithm as claimed in any one of claims 1 to 6, characterized in that: include: Simulation test scenario generation system: This device analyzes and processes massive measured data, receives scenario parameters δ generated by the scenario configuration parameter optimization device, edits a set of simulation test scenario configuration files based on the scenario parameters, and inputs them into the intelligent perception algorithm simulation test platform; Intelligent perception algorithm simulation test platform: receives the configuration file from the simulation test scenario generation system, generates a simulation test environment with high fidelity, accesses the intelligent perception algorithm under test and performs simulation tests; Scenario configuration parameter optimization device: receives the simulation test process and results, builds a dynamic model, and uses the Bayesian optimization algorithm to obtain the current expected improvement α EI (δ) maximizes the scenario parameter δ, and inputs it into the scenario editing unit of the simulation test scenario generation system; Intelligent perception algorithm credibility evaluation system: Analyze the simulation test results of the intelligent perception algorithm under test, and output the comprehensive credibility evaluation results based on the evaluation index system.

Images