Credibility measurement method and system for multimodal perception algorithms for autonomous driving

Abstract

The present invention proposes a credibility measurement method and system for unmanned driving multimodal perception algorithms, constructs a data-driven virtual simulation test platform, and statistically analyzes its characteristic distribution based on real measured data. The end-to-end closed-loop perception process is characterized by dynamic model modeling and state transition. Subsequently, in response to the problem that the high dimensionality of scene parameters leads to the inefficiency of traditional combination testing methods, the present invention establishes a black box substitution model between scene parameters and perception test results, and uses a Bayesian optimization algorithm to evaluate historical test data to generate the most adversarial scene parameters in the next iterative test. In this way, the adversarial nature of the test environment is continuously improved, and the test efficiency is effectively improved. Finally, the present invention constructs a perception algorithm credibility measurement module, extracts perception failure scenarios by replaying test data, and constructs a credibility evaluation index system to achieve a comprehensive credibility evaluation of the perception system under test.

Description

Technical Field

[0001] The present invention relates to the field of driverless car testing and evaluation, and in particular to a credibility measurement method and system for a driverless multimodal perception algorithm. Background Art

[0002] Autonomous driving with human-vehicle-road collaboration is a safety-critical system that relies heavily on multiple sensors and intelligent algorithms. As autonomous driving levels increase, the requirements for the comprehensiveness, accuracy, and timeliness of environmental information acquired by the perception system become increasingly stringent. The overall performance of autonomous vehicles depends largely on the performance of the perception system. To ensure the safe and reliable operation of autonomous vehicles, the perception system's operation and output must meet expected requirements—that is, possess trustworthiness.

[0003] Current testing and evaluation methods for autonomous driving perception systems fall into three main categories. The first category primarily relies on formal methods. Based on the model structure and logical mechanisms of the perception system, mathematical analytical models are constructed, traversing the system's state space to identify and eliminate dangerous scenarios. These methods aim to detect design flaws in the perception system early in system development, helping to proactively address them and reduce the cost of later repairs. However, most current autonomous driving perception systems are based on deep learning algorithms, whose internal structure and operating principles are difficult to model. Furthermore, their output is significantly influenced by the training data, exhibiting significant cognitive uncertainty. Therefore, traditional formal methods are difficult to apply to the credibility evaluation of intelligent perception algorithms.

[0004] The second category involves closed-field testing of driverless vehicles equipped with perception systems. This method involves manually setting one or several sets of scenario parameters in a real-world environment and conducting a pass-through test. This method is conducted in a controlled environment, and the experimental process is reproducible and repeatable, which can guarantee the validity of the test results to a certain extent. However, due to the limitations of closed-field testing, this method often cannot provide sufficient test mileage to cover various situations and scenarios, and cannot fully evaluate the performance of the perception system in complex real-world environments. The high cost of building and maintaining closed test sites further limits the scale of testing. Evaluation criteria usually include statistical indicators such as false detection rate, false negative rate, and error, which cannot fully reflect the credibility of the perception system in dynamic, complex, and uncertain real-world environments.

[0005] The third approach is virtual simulation testing, which uses computers to abstract the actual traffic environment and construct a simulation test system to complete the test. Researchers at home and abroad have conducted extensive research on traffic scenarios and their data construction and generation techniques. They have used methods such as simulation engine generation and game model rendering to construct virtual test scenarios, thereby obtaining high-quality test data. This large-scale generated test data is then used for training and testing autonomous driving intelligent perception algorithms. Furthermore, this approach allows for the generation of diverse traffic scenarios within a virtual simulation environment, which is crucial for evaluating the credibility of perception algorithms in multiple scenarios. The test environment is highly controllable, allowing testers to modify scenario parameters to repeatedly test the algorithm under controlled conditions, thereby better evaluating the performance of the perception algorithm. Compared to closed testing, virtual simulation testing is less expensive, effectively reducing testing costs. However, the realism of the virtual environment is crucial to the accuracy of test results. The virtual environment must accurately and unbiasedly simulate various real-world scenario elements. Furthermore, specific test execution techniques must be considered to ensure that the critical state space of the perception algorithm is covered within a limited number of simulation tests.

[0006] Therefore, the present invention conducts a credibility evaluation of a multimodal intelligent perception algorithm for unmanned driving based on virtual simulation testing.

[0007] To address the aforementioned scene restoration issues, we developed a general test scene generation technology for multimodal perception. This technology summarizes the common characteristics of multimodal algorithms and analyzes the attributes and interactions of various elements in the scene.

[0008] At the same time, the value range of simulation scene elements is defined based on the distribution of massive measured data. The constructed simulation scene is both realistic and can comprehensively test the performance of the perception algorithm.

[0009] To address the aforementioned issue of dangerous scenario coverage, we developed an accelerated testing method for integrated intelligent perception algorithm evaluation. Using Gaussian processes to model the mapping between scenario parameters and perception results from a set of historical tests, we used the cumulative error of a complete test as the cost function. Using a Bayesian optimization algorithm, we searched for potential scenario parameters δ that maximized the expected cost function. This automatically selected the most adversarial scenario as the next perception test scenario. Through multiple iterations, we determined whether the perception algorithm under test was poorly designed or poorly trained.

[0010] In addition, the present invention constructs a credibility measurement model for intelligent perception algorithms, considers the impact of perception failure on the safety of unmanned driving operations, and designs evaluation indicators to comprehensively evaluate the credibility of intelligent perception algorithms. Summary of the Invention

[0011] The flowchart of the multimodal perception credibility measurement method for human-vehicle-road collaborative unmanned driving in this invention is as follows:Figure 1 As shown in the figure, it includes three steps: general test scenario generation, adversarial test method research, and credibility measurement model construction:

[0012] 1. Generate general test scenarios for multimodal perception

[0013] Based on massive amounts of measured data, this paper constructs a universal simulation test environment suitable for multimodal intelligent perception algorithms. First, data preprocessing is performed to extract scene element types and corresponding attribute feature values ​​based on the measured data. Subsequently, the joint distribution of attribute values ​​for each type of element in the test scene is determined. Random sampling based on this joint distribution creates a simulated test scene that recreates the real world, thus completing the construction of a universal test scenario.

[0014] (1) Measured data preprocessing

[0015] In order to ensure that the general test scenario is unbiased, the present invention performs data preprocessing based on the collected measured data (images, point cloud data, etc.), sorts out the scene elements in the real world, and extracts the attribute feature values ​​of the scene elements according to the element type.

[0016] 1) Sorting out scene elements

[0017] Based on the ISO 21488 definition of scenes, this paper divides scene elements into entity, environment, and interference categories, as detailed in Table 1. Entity elements are elements with specific physical properties that interact with the autonomous vehicle during dynamic driving. They must fully cover all expected perception content of the perception system under test and, as comprehensively as possible, cover various traffic elements in the real traffic environment. Environment elements are scene elements that significantly affect the perception system's ability to achieve its intended functions. Interference elements are generally not considered as expected perception content of the perception system, but they may affect the perception system's ability to achieve its intended functions.

[0018] Table 1 Sorting out scene elements

[0019]

[0020] 2) Scene element attribute feature value extraction

[0021] For the specific scene elements listed in Table 1, it is necessary to extract the characteristic values ​​of the attribute features that influence the performance of the multimodal intelligent perception algorithm. For entity and interference classes, these attribute features include reflectivity, reflection area, data generation, and variability; for environmental classes, these attribute features include reflectivity and transmittance. For example, Tables 2 and 3 provide the attribute features of specific elements such as motor vehicles and rainfall, respectively. Subsequently, feature extraction algorithms such as convolutional neural networks are used from the measured data to extract the characteristic values ​​of the corresponding attributes.

[0022] Table 2 Attribute characteristics of motor vehicles

[0023]

[0024]

[0025] Table 3 Rainfall attribute characteristics

[0026]

[0027] (2) Calculation of joint distribution of simulation test scenario parameters

[0028] Generating a simulation test scenario requires a set of defined scenario parameters δ, i.e., a set of defined scenario element attribute feature values. Assume there are m of these parameters (e.g., vehicle surface material, vehicle color, vehicle structure, rainfall composition, rainfall density, etc.). To ensure the unbiased nature of the simulation test, a joint probability distribution of the m attribute features should be constructed, and a set of scenario parameters should be randomly sampled from this distribution. The specific steps are as follows:

[0029] a. Feature type classification and preprocessing

[0030] When constructing the joint probability distribution of attribute features, attributes include two types of features: discrete and continuous.

[0031] Discrete features: Features with a finite number of possible values, such as the color of a vehicle (black, white, gray) or the surface material of a vehicle (metal, plastic, etc.). During preprocessing, label encoding is used to convert category labels into numerical form.

[0032] Continuous features: Features with an infinite number of possible values, such as vehicle speed and vehicle heading angle. During preprocessing, each feature value is normalized to ensure that different features have the same scale.

[0033] b. Probability distribution modeling of attribute features

[0034] Discrete features: Probability Mass Function (PMF) is used for modeling, and the frequency of occurrence of each possible value in the measured data is calculated as its probability:

[0035]

[0036] Among them, c k is the kth category, and n is the total number of attribute values ​​of this feature in the measured data.

[0037] Continuous features: Kernel Density Estimation (KDE) is used to model each attribute feature, and Gaussian kernel function is selected. Select the bandwidth parameter h and calculate its probability density:

[0038]

[0039] in, Indicates that the feature is in X i Density estimate of location, X j The eigenvalue of the jth sample point in the measured data, n is the total number of attribute values ​​of the feature in the measured data, indicating how many X j is used to construct a density estimate for this feature.

[0040] c. Construction of Joint Probability Distribution

[0041] Combine m discrete and continuous features to construct a mixed joint probability distribution. The independent parameter group I contains k independent discrete and continuous parameters; the dependent parameter group D contains l discrete and continuous parameters that depend on the independent parameters or other features, where the sum of l and k is m. The joint probability distribution of the independent parameter group I is:

[0042] P(I)=P(X i1 )·P(X i2 )·...·P(X ik )

[0043] in, is the marginal distribution of k independent parameters, using P(X i )(discrete feature) or (continuous features) are calculated. The joint probability distribution of the dependent parameter group D is:

[0044] P(D|I)=P(X d1 ,X d2 ,…,X dl |X i1 ,X i2 ,…,X ik )

[0045] Among them, X d1 ,X d2 ,…,X dl The discrete or continuous features belong to the dependent parameter group D, and l is the number of dependent parameters. Finally, the joint probability distribution is expressed as follows, and the joint probability distribution diagram is as follows Figure 2 As shown:

[0046] P(X1,X2,…,X m )=P(I)·P(D|I)

[0047] During subsequent intelligent perception algorithm simulation testing, Bayesian optimization requires a set of existing perception test results. Specifically, we randomly sample several sets of scenario parameters δ (row vectors containing parameter values ​​for all scenario elements) from the joint probability distribution constructed in this section. We then generate specific test cases for each set and execute perception tests to obtain the corresponding perception test results. Each set of perception test results maps a different scenario parameter δ to the value of the cost function J.

[0048] 2. Design of an integrated framework for intelligent perception algorithm evaluation

[0049] The general test scenario generation process described above enables unbiased construction of test cases in a simulation environment based on measured data. However, the vast number of test cases is inexhaustible. The key to this invention is how to evaluate the performance of intelligent perception algorithms through limited testing. This section will explore adversarial scenario generation strategies based on Bayesian optimization algorithms by constructing a dynamic model of simulated test scenarios to accelerate adversarial testing.

[0050] (1) Construction of dynamic model of simulation test scenario

[0051] In this step, the state of the simulated test scene, the state of the unmanned vehicle, and the output of the perception algorithm under test are modeled as a dynamic model. The first two state variables are determined by a set of test scene parameters δ, and the output is determined by the perception algorithm under test. The cumulative error of the output is modeled as the cost function J of the dynamic model. The Bayesian optimization method in 2.2 is used to iteratively generate the value of δ to maximize J. The schematic diagram of the dynamic model model is shown in the figure. Figure 6 shown.

[0052] The modeling process is as follows:

[0053] Order s t is the state vector of the simulation test scene at time t, including three types of scene elements in the scene, which is defined as s t ={s interactive_entity ,s disturbing_entity ,s environmental_entity}, where s interactive_entity is an entity class, belonging to all expected perception contents of the perception system under test, s disturbing_entity is the interference type, s environmental_entity is the environment class. Let x t is the state vector of the unmanned vehicle at t where the intelligent perception algorithm under test is located. This vector involves the position and direction of the unmanned vehicle. Both of the above initial states are determined by the initial scene parameter δ.

[0054] The process of the algorithm being tested perceiving the surrounding environment is described by o t ~h(o|xt ,s t ) means, t Represents the output of the algorithm under test. The dynamic model of the unmanned vehicle where the algorithm under test is located is represented by x t+1 ~p(x|x t ,a t ) is represented by a t ~π θ (a|o t ) is determined by the strategy at which the output of the perception module is determined. t The transfer model of the state vector of the simulation test scenario is s t+1 ~q(x|s t ,u t ), the state transfer process is composed of Since the present invention evaluates the intelligent perception algorithm, the policy control parameters and θ are set as constants. Therefore, the simulation joint state time evolution of the multimodal intelligent perception algorithm test system in the present invention is given by the following generative model, and the meaning of each state vector is as follows: Figure 6 As shown:

[0055] δ~(x0,s0)

[0056] o t ~h(o|x t ,s t )

[0057] a t ~π θ (a|o t )

[0058]

[0059] x t+1 ~p(x|x t ,a t )

[0060] s t+1 ~q(x|s t ,u t )

[0061] x t : represents the state vector of the unmanned vehicle where the algorithm under test is located at time t, including position, speed, etc.;

[0062] s t : represents the state vector of the simulation test scenario at time t, including all entity, interference and environment states;

[0063] o t: represents the output of the algorithm under test at time t, such as the detected target, obstacle, etc.

[0064] h(o|x t ,s t ): indicates a given x t and s t When the measured perception algorithm outputs o t The probability distribution function of

[0065] a t : represents the behavioral decision of the unmanned vehicle where the algorithm under test is located at time t, including steering, acceleration, etc.;

[0066] π θ (a|o t ): indicates that it is based on o t To generate unmanned vehicle behavior a t Strategy model, θ represents the control parameter;

[0067] u t : represents the policy output of the control test scenario state transition at time t, including the change of the surrounding motor vehicle motion state and the change of the environment state;

[0068] Indicates based on s t and x t To determine the scene change u t The strategy model, Indicates control parameters;

[0069] x t+1 : Indicates the state of the unmanned vehicle at time t+1, which is represented by the current state x t and behavior a t Determine, through the transfer function p(x|x t ,a t )produce;

[0070] s t+1 : Indicates the state of the simulation test scenario at time t+1, which is represented by the current state s t and scene changes t Determine, by the transfer function q(x|s t ,u t )describe.

[0071] The cost function of the dynamic model is modeled as the cumulative error of the perception module output results. The formula is as follows:

[0072]

[0073] Where c(o t ,st ) is the cost function at time t, which indicates the degree of deviation between the output perception result of the measured algorithm and the true value at that time. and θ are constants, so the instantaneous cost function is determined by the initial scene parameters δ.

[0074] Perception error is defined as existence error, i.e., missed detection (False Negative, FN) of an obstacle that actually exists and false detection (False Positive, FP) of an obstacle that actually does not exist. Classification error, i.e., incorrect classification of the output result.

[0075] Measurement error refers to the error in the position, speed, size and other parameters of the output results.

[0076] c(o t ,s t )The calculation process is as follows:

[0077]

[0078]

[0079]

[0080]

[0081] Among them, considering that missed detection has a greater impact on the safety of driverless cars, the instantaneous cost function is set to the highest when a missed detection occurs, that is, Error m Use w u 、w c 、w s Represents the weights of existence error, classification error and state error, and the instantaneous error value is obtained by weighted average method. Indicates the distance between the misdetected target and the unmanned vehicle. The closer the distance, the higher the corresponding error score. Indicates the number of misclassified targets, sum total Represents the total number of targets. The greater the number of classification errors, the higher the corresponding error score. and They represent the measured value and the true value when the perception result is correctly classified. The greater the proportion of the average measurement error to the true value, the higher the corresponding error score.

[0082] In the subsequent adversarial scenario generation based on the Bayesian optimization algorithm, the present invention continuously adjusts δ to generate adversarial scenarios that are unfavorable to the intelligent perception algorithm, continuously reducing the accuracy of the perception module's output results to find the performance boundary of the tested perception system and quantify its credibility level. Therefore, when setting the cost function, it is necessary to maximize the cumulative error of the perception results, which is expressed as follows:

[0083] δ * =argmax δ J

[0084] (2) Adversarial scenario generation based on Bayesian optimization algorithm

[0085] This section is based on the Bayesian optimization algorithm. By modeling the prior distribution and acquiring the function, the initial scenario configuration strategy δ for the next round of simulation test is determined. k+1 , the specific algorithm framework is shown in the following table:

[0086] Table 4 Bayesian optimization algorithm framework

[0087]

[0088] The meaning of each symbol is as follows:

[0089] 1. δ is a set of scenario parameters, each set of δ generates a simulation test scenario;

[0090] 2. J is the cost function of the dynamic model of the simulation test scenario, Represents parameter δ j The corresponding cost function value;

[0091] 3.D is the historical observation set of the intelligent perception algorithm simulation test platform;

[0092] 4. EI is the acquisition function, that is, the expected improvement algorithm ExpectedImprovement (EI).

[0093] Based on the above steps, the content of this section is divided into establishing a probability distribution for the historical observation state and using the acquisition function to generate the scenario parameter δ for the next iteration.

[0094] 1) Bayesian optimization prior distribution modeling

[0095] In the first row of Table 4, a set of common test scenarios (assuming there are n scenarios) is obtained by randomly sampling the content of Part 1 and performing the test to obtain the data set A Gaussian process is used to establish an accurate probability distribution model for the dataset D, which serves as the prior distribution for Bayesian optimization. ARD Matern 5 / 2 is used as the kernel function to characterize the nonlinear relationship between the output of the perception algorithm under test and the scene parameter δ. The modeling process is as follows:

[0096]

[0097]

[0098]

[0099] Among them, K(δ,δ ′ ) is the value of the kernel function, which represents two points δ and δ in the input feature space ′ The similarity between them, δ represents the scene parameters in this simulation test. σ 2 is the variance parameter of the kernel function, which controls the scale of the kernel function; l is a function of the distance in the feature space, allowing different dimensions to have different length scales; β is the noise variance in the observations; β, l, σ are hyperparameters of the Gaussian process and are determined by random variational inference techniques.

[0100] After completing Gaussian process modeling, the model outputs the predicted mean μ(δ) and predicted variance σ at each δ 2 (δ). The mean represents the predicted value of the cost function J at δ given the current dataset D; the variance represents the uncertainty at δ, reflecting the confidence of the model in the prediction at that point. A larger variance means that the model has a higher uncertainty in the prediction at that point, which encourages the Bayesian optimization algorithm to explore more in this area. The mean and variance are used by the Bayesian optimization algorithm to generate the scenario parameter δ in the fourth row of Table 4. j .

[0101] 2) Bayesian optimization acquisition function modeling

[0102] In row 4 of Table 4, based on the predicted mean and variance output by the Gaussian process model, an acquisition function α(δ) is constructed to determine the choice of scenario parameter δ for the next test. The acquisition function needs to strike a balance between "exploration" (exploring areas with high uncertainty) and "utilization" (utilizing areas near the current optimal point). In this invention, the acquisition function uses the expected improvement algorithm, and the modeling process is as follows:

[0103]

[0104]

[0105] Among them, I(δ) is the improvement function of the cost function at δ, J *is the best value among the test results of the current dataset D; is the Gaussian process model's prediction of the cost function, which is distributed as follows:

[0106]

[0107] because The distribution is Gaussian distribution, and the expected improvement function α EI (δ) can be calculated analytically using the integral property of the Gaussian distribution, and the analytical expression of EI is as follows:

[0108]

[0109] in:

[0110] μ(δ) is the predicted mean of the cost function J by the Gaussian process at the scene parameter δ;

[0111] σ(δ) is the standard deviation of the Gaussian process prediction of J at δ;

[0112] Φ is the cumulative distribution function of the standard normal distribution;

[0113] φ is the probability density function of the standard normal distribution;

[0114] in, Used to measure the predicted value exceeding J at parameter δ * The probability of , reflects the “utilization” component, that is, selecting points with a higher probability of bringing the expected improvement of the cost function value; Used to measure the accuracy of the predicted value close to J at parameter δ * The probability density of , reflects the “exploration” component, that is, exploring in high uncertainty areas to reduce uncertainty and possibly find better improvements. Then, we look for the solution that makes α EI (δ) maximized The present invention selects the gradient ascent method to use α EI The gradient information of (δ) is gradually updated to approach the maximum value. This completes the content of the fourth row of Table 4 and determines the scenario parameter δ that maximizes the improvement of the expected cost function value. * .

[0115] In the 5th to 7th rows of Table 4, the scenario parameters δ output by the Bayesian optimization algorithm are * Conduct intelligent perception algorithm simulation test and obtain the new cost function value y=J(δ * ) and adds it to dataset D. The Gaussian process model is updated based on the newly acquired dataset D, and then a new round of iteration is performed, returning to row 4 of Table 4, until the K iteration requirement is met. This method continuously generates scenario parameters δ that are adversarial to the perception algorithm being tested, effectively improving testing efficiency.

[0116] 3. Construction of credibility measurement model for intelligent perception algorithm

[0117] Based on the contents of Parts 1 and 2, K groups of adversarial environments are obtained, and as the Bayesian optimization iterates, the cost function value of each group continues to increase (that is, the overall accuracy of the perception results in the clip continues to decrease), as shown in Figure 7 The output of each group is a continuous perception result segment with a fixed length. If it exceeds the pre-set perception failure threshold, it is considered a perception failure scenario. For example, Figure 8 This section will build a credibility evaluation index system to comprehensively evaluate the credibility of the intelligent perception algorithm based on the K groups of test results obtained in the previous steps.

[0118] This paper constructs a credibility measurement model for intelligent perception algorithms based on exposure rate, penetration rate, and hazard rate. This model considers the errors in the perception results themselves and their impact on the safety and stability of autonomous vehicles, and comprehensively evaluates their credibility. The general steps are as follows:

[0119] a. Exposure Rate Assessment: First, we use expert experience to set the perception algorithm failure threshold τ. We then replay the log data recorded during the simulation test to identify the perception failure segments. The exposure rate represents the proportion of the duration of the perception failure segments to the total time, and this is used to determine the exposure rate level E.

[0120] b. Penetration rate evaluation: Penetration rate indicates the degree of decrease in the accuracy of the perception results. The three indicators of existence error, classification error, and measurement error are selected to evaluate the penetration rate. The weighted average method is used to calculate the penetration rate. The formula is as follows:

[0121] P=w u ×Score u +w c ×Score c +w s ×Score s

[0122] Among them, Score u Indicates the existence error score, Score c Represents the classification error score, Score s Denotes the measurement error score, using w u 、w c 、w s They represent the relative importance of existence error, classification error and measurement error respectively.

[0123] c. Hazard rate assessment: The hazard rate indicates the degree to which the output of the autonomous driving planning and control system is disturbed by the disturbance in the output of the perception algorithm. H is used to represent the hazard rate, and the calculation formula is as follows:

[0124]

[0125] Among them, S d It represents the total number of times in the simulation test that the TTC between the unmanned vehicle under test and the vehicle in front is less than 0.5s, and K represents the total number of tests performed.

[0126] d. Comprehensive evaluation: The credibility of the tested intelligent perception algorithm is calculated by weighting the scores of each dimension. The calculation formula is as follows:

[0127] Trustworthy=E×w E +P×w P +H×w H

[0128] Among them, w E 、w P 、w H They represent the weights corresponding to the exposure rate, penetration rate, and hazard rate, respectively. E, P, and H represent the scores corresponding to the exposure rate, penetration rate, and hazard rate calculated based on the K groups of test results, respectively. They are normalized separately. Trustworthy represents the credibility of the tested intelligent perception algorithm.

[0129] The present invention also provides a device and system for evaluating the credibility of unmanned driving intelligent perception algorithms, such as Figure 5 This system adopts a cloud platform distributed simulation architecture, builds a simulation test scenario generation system based on the cloud platform, and builds an intelligent perception algorithm simulation test platform, a scenario configuration parameter optimization device, and an intelligent perception algorithm credibility evaluation system based on a local high-performance computing (HPC) cluster:

[0130] 1. Simulation test scenario generation system: This device analyzes and processes massive amounts of measured data, receives information from the scenario configuration parameter optimization device, edits a set of simulation scenario configuration files according to the specified scenario parameters, and inputs them into the intelligent perception algorithm simulation test platform. The module includes the following units:

[0131] (1) Data storage unit: used to store a large amount of real-world measured data, including data collected on open roads, data collected on closed sites, standards and regulations, traffic accident data, infrastructure data, meteorological data, and sensor data.

[0132] (2) Data processing unit: A module used to analyze and process the measured data in the data storage unit. It is responsible for extracting information from the original unstructured data and providing basic data for scene generation.

[0133] (3) Scenario editing unit: According to the scenario parameter requirements, edit the simulation configuration file and output it to the simulation test platform.

[0134] 2. Intelligent Perception Algorithm Simulation Test Platform: This platform receives a set of configuration files from the simulation test scenario generation system, generates a highly realistic simulation test environment, connects to the intelligent perception algorithm under test, and performs simulation tests. The platform includes the following units:

[0135] (1) Simulation framework: The core of the simulation platform, responsible for coordinating the work of various modules and controlling the simulation process.

[0136] (2) Physics engine: simulates the movement, interaction and V2X communication of objects in the simulation environment to ensure that the physical behavior in the simulation environment conforms to objective laws.

[0137] (3) Image engine: Renders and generates virtual scenes with high fidelity, including sensor simulation, static scene restoration, and dynamic scene restoration.

[0138] (4) Perception algorithm interface unit: accesses the intelligent perception algorithm under test and integrates it into the simulation test platform.

[0139] 3. Scenario configuration parameter optimization device: Receives the intelligent perception algorithm simulation test process and results, and constructs them into a dynamic model. Combining the historical simulation scenario parameters with the corresponding test results, using the Bayesian optimization algorithm, constructs the Gaussian process prior and acquisition function respectively, and maximizes the acquisition function to obtain the current optimal scenario parameters, and inputs the parameters into the scenario editing unit.

[0140] (1) Dynamic model modeling module: The interaction between each element in the scene and the unmanned driving perception system over time during the simulation test is modeled as a dynamic model, and the corresponding cost function is designed.

[0141] (2) Prior distribution modeling module: A Gaussian process prior model is constructed based on the cost function. This model is used to estimate the nonlinear relationship between the output results of the intelligent perception algorithm and the scene parameters.

[0142] (3) Acquisition function modeling module: Receive the mean and variance information of the Gaussian process prior model, construct an acquisition function and find the maximum value of the acquisition function. This value corresponds to the most adversarial scenario configuration parameter in the next simulation test, and outputs the parameter to the simulation test scenario generation system.

[0143] 4. Intelligent perception algorithm credibility evaluation system: After the Bayesian optimization iteration process is completed, the virtual simulation test results are analyzed, and an evaluation index system is constructed to output a comprehensive credibility evaluation result. The system includes the following units:

[0144] (1) Log Data Storage Unit: This unit is responsible for collecting, storing, and managing all logs and test data generated during the virtual simulation test process. This unit includes the operation process and results of the perception algorithm under test in different scenarios. It supports the rapid retrieval and backtracking of historical test data.

[0145] (2) Credibility comprehensive evaluation unit: Based on the process and results of simulation testing combined with the evaluation index system, it outputs the comprehensive evaluation results of the credibility of the intelligent perception algorithm.

[0146] The credibility evaluation method, device, and system provided by the present invention have the following key technical points and advantages:

[0147] (1) The present invention constructs a test scene in a virtual simulation environment based on massive measured data, which is highly controllable while ensuring the fidelity of scene restoration.

[0148] (2) The present invention adopts the Bayesian optimization algorithm to iteratively optimize the cost function, so that the generated simulation scene has an increasingly stronger adversarial nature. The ability boundary of the perception algorithm can be quickly obtained by using a limited number of tests. Figure 7 By using a finite number of test iterations, we obtain the upper bound of the cost function (the level of overall perceived performance degradation), which can effectively improve the test evaluation efficiency.

[0149] (3) Existing testing and evaluation of autonomous driving perception algorithms are generally based on fixed-collection images or point clouds, which are open-loop evaluations. That is, the perception results do not affect the behavioral decisions of all traffic participants in the scene, and therefore cannot meet the requirements of end-to-end autonomous driving. This study, by constructing a virtual simulation test environment and dynamic modeling, depicts the dynamic process of "perception results affecting behavioral choices - behavioral choices affecting the perception input of the next frame", which can meet the closed-loop evaluation requirements of end-to-end autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0150] Figure 1 This is the overall flow chart of the credibility evaluation of the intelligent perception algorithm of the present invention;

[0151] Figure 2 Generate a general test scenario process for multimodal perception in the present invention;

[0152] Figure 3 This is a schematic diagram of the integrated framework design for intelligent perception algorithm evaluation in the present invention;

[0153] Figure 4 The process of constructing the credibility measurement model of the intelligent perception algorithm of the present invention;

[0154] Figure 5 Schematic diagram of the intelligent perception algorithm credibility evaluation device and system of the present invention;

[0155] Figure 6 Schematic diagram of dynamic model building;

[0156] Figure 7 Schematic diagram of the Bayesian optimization process;

[0157] Figure 8 Schematic diagram of perception failure segment extraction. DETAILED DESCRIPTION

[0158] The present invention relates to a credibility measurement method, device, and system for multimodal perception of human-vehicle-road collaborative unmanned driving. The specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and specific embodiments.

[0159] This example conducts a credibility evaluation on the 3D point cloud target detection algorithm PointPillars.

[0160] Among them, the simulation test scenario generation system is deployed in the cloud platform, which includes a massive data storage unit, a data processing unit, and a scenario editing unit;

[0161] The intelligent perception algorithm simulation test platform is developed based on the Unreal Engine and deployed on a local server. It receives scenario configuration files from the cloud platform and performs intelligent perception algorithm simulation tests.

[0162] The scenario configuration parameter optimization device is deployed in the local server, and the high-performance computing unit supports the dynamic model modeling module, the prior distribution modeling module and the acquisition function modeling module;

[0163] The intelligent perception algorithm credibility evaluation system, deployed on a local server, includes a log data storage unit that supports high bandwidth, high throughput, and low latency, as well as a simulation test result analysis unit and a comprehensive credibility evaluation unit. Data transmission and communication between these modules and the cloud platform form the multimodal intelligent perception credibility evaluation system proposed in this invention.

[0164] The massive measured data in this example comes from:

[0165] Vehicle-side data, test data from open test roads for intelligent connected vehicles in Jiading District, Shanghai, and closed test data from a dedicated closed test area for intelligent connected vehicles in Jiading District, Shanghai;

[0166] Roadside data is collected through multi-source and multi-modal roadside sensing equipment from the Intelligent Connected Vehicle Demonstration Zone in Jiading District, Shanghai. The demonstration zone includes demonstration operations of more than 700 types of automatic vehicles, including driverless taxis, intelligent connected buses, smart trains, smart cleaning, and smart delivery.

[0167] Based on the above device and system, Figures 1 to 8The implementation method of the credibility measurement method for multimodal perception of human-vehicle-road collaborative unmanned driving includes the following steps:

[0168] Step 1: Build a general test scenario for the intelligent perception algorithm based on massive measured data. The specific steps are as follows:

[0169] Step 1.1: Preprocess the measured data to extract specific elements and corresponding attribute feature values ​​in the real scene.

[0170] The data sources in this embodiment come from vehicle-side and roadside data, including both structured and unstructured data types and modalities such as video and 3D point clouds. All of this data is stored on the cloud platform's storage server. The cloud platform's data processing unit extracts attributes corresponding to the three predefined data sources. Data cleaning, feature extraction, and data storage are performed sequentially, transforming the unstructured measured data into structured scene element feature values.

[0171] Step 1.2: Calculate the distribution of each scene element attribute feature value separately, and calculate their joint distribution.

[0172] Based on the scene element attribute characteristic values ​​obtained from the real scene in step 1.1, the data processing unit uses the probability mass function and kernel density estimation method to fit the characteristic value distribution of each element in the scene respectively, and combines the discrete and continuous features to construct a mixed joint probability distribution.

[0173] Step 1.3: Based on the intelligent perception algorithm simulation test platform, construct an intelligent perception algorithm simulation test scenario according to the scenario parameter δ, which specifically includes the following steps:

[0174] The scene element parameter δ is obtained from the joint distribution function of attribute eigenvalues ​​by random sampling, and the virtual simulation test scene is rendered using the physics engine and graphics engine.

[0175] Step 1.4: After the simulation test scenario is determined, the intelligent perception algorithm to be tested is connected to the simulation platform to start testing.

[0176] The intelligent perception algorithm selected for testing in this embodiment is PointPillars, and the perception algorithm for testing uses the pre-training model weights disclosed in existing research.

[0177] Step 2: Build an integrated framework for intelligent perception algorithm evaluation. The specific steps are as follows:

[0178] The scenario configuration parameter optimization device receives the test results and scenario parameter settings from the simulation platform, and generates adversarial scenario parameters through dynamic model building, prior distribution modeling, and acquisition function modeling to guide the generation of the next simulation test scenario. The specific steps are as follows:

[0179] Step 2.1: Model the interaction between each scene element and the unmanned vehicle in which the perception algorithm is tested during the simulation test as a dynamic model:

[0180] δ~(x0,s0)

[0181] o t ~h(o|x t ,s t )

[0182] a t ~π θ (a|o t )

[0183]

[0184] x t+1 ~p(x|x t ,a t )

[0185] s t+1 ~q(x|s t ,u t )

[0186] Among them, δ determines the initial state of the state vector s of the simulation test scene element and the state vector x of the unmanned vehicle where the algorithm under test is located; h(o|x t ,s t ) means given x t and s t When the measured perception algorithm outputs the result o t The probability distribution function of π θ (a|o t ) indicates that it is based on o t To generate autonomous vehicle behavior a t Strategy model, θ represents the control parameter; Indicates based on s t and x t To determine the scene change u t The strategy model, Indicates control parameters.

[0187] The cost function of the dynamic model is modeled as the cumulative error of the perception module output results. The formula is as follows:

[0188]

[0189] Where c(o t ,s t ) is the cost function at time t, which indicates the degree of deviation between the output perception result of the measured algorithm and the true value at that time. and θ are constants, so the instantaneous cost function is determined by the initial scene parameter δ. The perception error is defined as existence error (False Negative, FN) is missing an obstacle that actually exists and false positive (FP is mistakenly detected when an obstacle does not exist); classification error (Classification error) is the error in the output classification; measurement error is the error in the output parameters such as position, speed, and size. c(o t ,s t )The calculation process is as follows:

[0190]

[0191]

[0192]

[0193]

[0194] Among them, considering that missed detection has a greater impact on the safety of driverless cars, the instantaneous cost function is set to the highest when a missed detection occurs, that is, Error m Use w u 、w c 、w s Represents the weights of existence error, classification error and state error, and the instantaneous error value is obtained by weighted average method. Indicates the distance between the misdetected target and the unmanned vehicle. The closer the distance, the higher the corresponding error score. Indicates the number of misclassified targets, sum total Represents the total number of targets. The greater the number of classification errors, the higher the corresponding error score. and They represent the measured value and the true value when the perception result is correctly classified. The greater the proportion of the average measurement error to the true value, the higher the corresponding error score.

[0195] Step 2.2: Receive the scene parameter δ and the dynamic model cost function J. Based on the test cases randomly sampled and generated in steps 1.3 and 1.4, execute the test to obtain a dataset D consisting of the corresponding perception results. Use ARD Matern 5 / 2 as the kernel function to construct a Gaussian process prior model to characterize the nonlinear relationship between the output of the intelligent perception algorithm and the scene parameter δ. The modeling process is as follows:

[0196]

[0197]

[0198]

[0199] Among them, K(δ,δ ′ ) is the value of the kernel function, which represents two points δ and δ in the input feature space ′ The similarity between them, δ represents the initial scenario parameter configuration in this simulation test, σ 2 is the variance parameter of the kernel function, which controls the scale of the kernel function, l is a function of the distance in the feature space, allowing different dimensions to have different length scales, and β is the noise variance in the observations.

[0200] Based on the simulation test data and noise data generated by the new scenario parameter configuration δ, the posterior distribution of the cost function is inferred through the Gaussian process regression algorithm.

[0201] The hyperparameters β, l, σ of the Gaussian process regression are obtained by maximizing the log-marginal likelihood function logp(y i |δ,β,l,σ) and introduces the variational distribution solution using variational inference techniques.

[0202] Step 2.3: Based on the mean and variance of the Gaussian process obtained in step 2.2, use the expected improvement algorithm to create the acquisition function α(δ). The modeling process is as follows:

[0203]

[0204] I(δ)=max(yJ * ,0)

[0205] Where I(δ) is the improvement function of the cost function at δ, J * is the best value in the current test result. Given the historical test result dataset D = {δ j ,y j} j＝1…k For the expectation of the distribution on y, due to the Gaussian assumption of the Gaussian process posterior, we can analyze α EI The maximum value of the expected improvement function will correspond to the most adversarial scenario configuration strategy δ in the next round of simulation tests. k+1 , and output the strategy to the scenario generation system.

[0206] Step 3: Conduct credibility evaluation based on the intelligent perception algorithm simulation test results. The specific steps are as follows:

[0207] The credibility comprehensive evaluation unit, based on the test process and results recorded in the log data storage unit, selects three indicators, namely exposure rate, penetration rate, and hazard rate, to construct a credibility evaluation system. On the one hand, it evaluates the accuracy of the perception results, and on the other hand, it considers the impact of perception failure on the safety and stability of the unmanned vehicle in which the algorithm is tested, and comprehensively evaluates the credibility of the intelligent perception algorithm. The specific steps are as follows:

[0208] a. Exposure Rate Assessment: First, we use expert experience to set the perception algorithm failure threshold τ. We then replay the log data recorded during the simulation test to identify the perception failure segments. The exposure rate represents the proportion of the duration of the perception failure segments to the total time, and this is used to determine the exposure rate level E.

[0209] b. Penetration rate evaluation: Penetration rate indicates the degree of decrease in the accuracy of the perception results. The three indicators of existence error, classification error, and measurement error are selected to evaluate the penetration rate. The weighted average method is used to calculate the penetration rate. The formula is as follows:

[0210] P=w u ×Score u +w c ×Score c +w s ×Score s

[0211] Among them, Score u Indicates the existence error score, Score c Represents the classification error score, Score s Denotes the measurement error score, using w u 、w c 、w s They represent the relative importance of existence error, classification error and measurement error respectively.

[0212] c. Hazard rate assessment: The hazard rate indicates the degree to which the output of the autonomous driving planning and control system is disturbed by the disturbance in the output of the perception algorithm. H is used to represent the hazard rate, and the calculation formula is as follows:

[0213]

[0214] Among them, S d It represents the total number of times in the simulation test that the TTC between the unmanned vehicle under test and the vehicle ahead is less than 0.5s, and K represents the total number of tests performed.

[0215] d. Comprehensive evaluation: The credibility of the tested intelligent perception algorithm is calculated by weighting the scores of each dimension. The calculation formula is as follows:

[0216] Trustworthy=E×w E +P×wP +H×w H

[0217] Among them, w E 、w P 、w H They represent the weights corresponding to the exposure rate, penetration rate, and hazard rate, respectively. E, P, and H represent the scores corresponding to the exposure rate, penetration rate, and hazard rate calculated based on the K groups of test results, respectively. They are normalized separately. Trustworthy represents the credibility of the tested intelligent perception algorithm.

Claims

1. A credibility measurement method for an unmanned driving multimodal perception algorithm, comprising the following steps: Step 1: Build a general test scenario for the intelligent perception algorithm based on measured data: Step 1.1: Preprocess the measured data to extract specific elements and corresponding attribute feature values ​​in the real scene; Step 1.2: Calculate the distribution of the attribute feature value of each scene element in the real scene separately, and calculate their joint distribution; Step 1.3: Set the simulation test scenario elements according to the specific elements in the real scene. The attribute characteristic values ​​are controlled by the simulation scenario parameter δ. Randomly sample the joint distribution obtained from the real scene several times, and obtain a scenario parameter δ each time. This is used to construct a set of simulation test cases. Connect the tested intelligent perception algorithm to perform the test and obtain the corresponding perception test results. Step 2: Design of an integrated framework for intelligent perception algorithm evaluation: Step 2.1: Model the state of the simulated test scene, the state of the unmanned vehicle, and the output of the perception algorithm under test as a dynamic model. The first two state variables are determined by the test scene parameter δ, and the output is determined by the perception algorithm under test. Model the cumulative error of the output as the cost function J of the dynamic model. Use the Bayesian optimization method composed of steps 2.2 and 2.3 to continuously iterate the value of δ to maximize J. Step 2.2: Based on the set of perception test results obtained in 1.3, select ARD Matern 5 / 2 as the kernel function and initialize the Gaussian process model; Step 2.3: Based on the cost function J, which can provide the predicted mean μ(δ) and variance σ for any scenario parameter δ 2 (δ) Gaussian process model, select the expected improvement function α EI (δ) is used as the acquisition function of Bayesian optimization, and numerical optimization methods are used to find the maximum value of α EI (δ), and output δ to the scenario generation system to guide the next round of test scenario generation; Step 3: Construction of intelligent perception algorithm credibility measurement model: Step 3.1, extracting the perceived failure fragment based on the simulation test results; Step 3.2: Construct a credibility evaluation index system to comprehensively evaluate the credibility of the intelligent perception algorithm.

2. The credibility measurement method for the multimodal perception algorithm for unmanned driving according to claim 1 is characterized in that: The step 1.1 specifically includes the following steps: Step 1.1.

1. Sorting out scene elements: Based on the actual multimodal data collected, classify the elements appearing in the scene into three categories: entity, environment, and interference. Specific elements are further refined by subcategory. Step 1.1.2, scene element attribute feature extraction: Select the attributes that affect the performance of the multimodal intelligent perception algorithm in the specific element items of each type of scene, and use the feature extraction algorithm to extract the characteristic values ​​of the specific element attributes of each type of scene from the measured data.

3. The credibility measurement method for the multimodal perception algorithm for unmanned driving according to claim 1 is characterized in that: The step 1.2 specifically includes the following steps: Step 1.2.1: Distinguish between discrete and continuous features in scene element features, and use probability mass function (PMF) to model discrete features and kernel density estimation (KDE) to model continuous features. Step 1.2.2: Combine discrete and continuous features to construct a mixed joint probability distribution.

4. The credibility measurement method for the multimodal perception algorithm for unmanned driving according to claim 1 is characterized in that: The specific process of step 2.3 is as follows: Bayesian optimization acquisition function modeling: Use the expected improvement function as the acquisition function, which calculates a set of potential scenario parameters δ compared to the currently known cost function J max The expected improvement α EI (δ), which makes the cost function value at the current maximum value J max continue to increase under the premise of .

5. The credibility measurement method for unmanned driving multimodal perception algorithm according to claim 1 is characterized in that: The specific process of step 3.1 is as follows: The expert experience method is used to set the perception failure threshold, and the perception failure fragments are extracted based on the simulation process recorded during the test and the corresponding perception results.

6. The credibility measurement method for unmanned driving multimodal perception algorithm according to claim 1 is characterized in that: The specific process of step 3.2 is as follows: A credibility evaluation index system is constructed based on exposure rate, penetration rate and hazard rate. On the one hand, it evaluates the accuracy of the perception results. On the other hand, it considers the impact of perception failure on the safety and stability of the unmanned vehicle in which the algorithm is tested, and comprehensively evaluates the credibility of the intelligent perception algorithm.

7. A credibility measurement system for an unmanned driving multimodal perception algorithm, used to implement the credibility measurement method for an unmanned driving multimodal perception algorithm according to any one of claims 1 to 6, characterized in that: include: Simulation test scenario generation system: This device analyzes and processes massive amounts of measured data, receives scenario parameters δ generated by the scenario configuration parameter optimization device, compiles a set of simulation test scenario configuration files based on the scenario parameters, and inputs them into the intelligent perception algorithm simulation test platform; Intelligent perception algorithm simulation test platform: Receives configuration files from the simulation test scenario generation system, generates a highly realistic simulation test environment, connects to the intelligent perception algorithm under test, and performs simulation tests; Scenario configuration parameter optimization device: receives the simulation test process and results, builds a dynamic model, and uses the Bayesian optimization algorithm to obtain the current expected improvement amount α EI (δ) maximizes the scenario parameter δ, and inputs it into the scenario editing unit of the simulation test scenario generation system; Intelligent perception algorithm credibility evaluation system: Analyzes the simulation test results of the intelligent perception algorithm under test and outputs comprehensive credibility evaluation results based on the evaluation index system.

Images