Autonomous driving simulation test method and device, vehicle, medium and program product

Abstract

This invention relates to the field of autonomous driving technology, and discloses autonomous driving simulation testing methods, devices, vehicles, media, and program products. The invention constructs a confusion matrix by forming perception fault pairs from the output category and the real category of the perception model, evaluating safety risk values, and then correcting the safety risk values ​​using real-vehicle perception sample data. Based on the corrected safety risk values, a set of perception fault test scenarios is constructed for simulation testing. By quantifying the semantic-level fault risk of perception fault pairs and correcting them using real-vehicle samples, the risk values ​​are made to match the real fault distribution of the perception model. The test scenarios generated based on the corrected confusion matrix can cover various types of perception semantic errors, systematically verifying the operational status of the autonomous driving target algorithm when receiving erroneous semantic information. This achieves full coverage and standardized simulation testing of sensor fusion semantic output faults, improving the completeness of fault verification from autonomous driving perception to autonomous driving algorithms.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to autonomous driving simulation testing methods, devices, vehicles, media, and software products. Background Technology

[0002] With the development of autonomous driving technology, autonomous driving systems have become one of the core functions of intelligent connected vehicles. Before real-world application, how to fully verify the safety and robustness of autonomous driving systems through simulation testing in a laboratory environment has become a key challenge for the industry. One of the core tasks of simulation testing is to simulate various fault conditions to examine the system's performance under abnormal circumstances.

[0003] Since autonomous driving decisions heavily rely on semantic information (such as "What is this object?") derived from vehicle perception signals interpreted by vehicle fusion perception algorithms, the accuracy of this semantic information directly impacts the safety of autonomous driving decisions. In real-world driving environments, inaccurate perception algorithm outputs are inevitable, such as misidentifying a small car as a large vehicle. Current fault injection tests often occur before or after perception, skipping the crucial "perception understanding" stage. For example, automatic parking scenarios (especially low-speed, near-field, and multi-obstacle scenarios) are high-risk areas for sensor fusion semantic errors. Therefore, effectively simulating and verifying autonomous driving control algorithms in situations where perception algorithm outputs abnormal information is a pressing issue. Summary of the Invention

[0004] This invention provides an autonomous driving simulation testing method, device, vehicle, medium, and program product to solve the problem that existing autonomous driving simulation testing solutions lack effective simulation verification when sensor fusion semantic output information is abnormal.

[0005] In a first aspect, the present invention provides an autonomous driving simulation testing method, the method comprising: Obtain the set of all output categories corresponding to at least one perception model of the vehicle; For each perception model, each output category is paired with each real category to form a perception fault pair, and the security risk value of each perception fault pair is evaluated. A fault confusion matrix is ​​constructed based on each of the perceived fault pairs, and the security risk value is determined as the matrix element of the corresponding perceived fault pair in the fault confusion matrix; Obtain the real vehicle perception sample data corresponding to the perception model, and correct each of the safety risk values ​​in the fault confusion matrix based on the real vehicle perception sample data; Based on the corrected fault confusion matrix, a set of perception fault test scenarios is constructed, and the target algorithm for autonomous driving is simulated and tested using the set of perception fault test scenarios.

[0006] This invention forms perception fault pairs by matching the output category and the true category of the perception model, constructs a confusion matrix by evaluating the safety risk value of each perception fault pair, and then corrects the safety risk values ​​using real-vehicle perception sample data from the perception model. Finally, based on the corrected safety risk values, a set of perception fault test scenarios is constructed to simulate and test the autonomous driving target algorithm. This quantifies the semantic-level fault risk of perception fault pairs and standardizes the storage of all semantic misjudgment types using the confusion matrix. Real-vehicle sample correction ensures that the risk values ​​closely match the actual fault distribution of the perception model. The test scenarios generated based on the corrected confusion matrix can cover various types of perception semantic errors, systematically verifying the operational status of the autonomous driving target algorithm when receiving erroneous semantic information. This achieves full coverage and standardized simulation testing of sensor fusion semantic output faults, improving the completeness of fault verification from autonomous driving perception to autonomous driving algorithm.

[0007] In one alternative implementation, the assessment of the security risk value for each perceived fault pair includes: Based on the preset driving safety risk level classification conditions, the driving safety risk level of each perceived fault pair is assessed to determine the driving safety risk level of each perceived fault pair. Based on the pre-defined correspondence between different driving safety risk levels and score values, the score value corresponding to the driving safety risk level of each perceived fault pair is determined. The score corresponding to the driving safety risk level of each perceived fault pair is determined as the safety risk value of that perceived fault pair.

[0008] This invention classifies perceived fault pairs by pre-defined risk level classification conditions and matches the pre-defined score values ​​of each level as safety risk values, achieving a standardized initial assessment of semantic fault risk. Through unified classification rules, the risk quantification of all perceived fault pairs is completed, ensuring that all perceived fault pairs have directly comparable standardized risk values, serving as the basic elements of the confusion matrix. Risk assignment for a large number of perceived fault pairs can be completed quickly without complex calculations, significantly reducing the computational and human resource costs of initial fault modeling. This provides a unified and quantifiable basic risk benchmark for subsequent real-vehicle data correction and test resource allocation, ensuring a unified risk assessment standard throughout the fault testing process and facilitating subsequent automated batch processing of massive amounts of perceived fault pairs.

[0009] In one optional implementation, the real-vehicle perception sample data includes: the real-vehicle output category and the corresponding real-vehicle true category; the step of correcting each of the safety risk values ​​in the fault confusion matrix based on the real-vehicle perception sample data includes: For each real vehicle output category, count the number of first samples in the real vehicle perception sample data for the current real vehicle output category and the number of second samples in the first sample data that are the same as the current real vehicle output category; Based on the ratio of the second sample size to the first sample size, the conditional probability of the current real vehicle output category is calculated; Based on the conditional probabilities corresponding to different real vehicle output categories, calculate the erroneous conditional entropy and correct conditional entropy for each perceived fault pair. Based on the difference between the erroneous conditional entropy and the correct conditional entropy of each perceived fault pair, the security risk value corresponding to that perceived fault pair in the fault confusion matrix is ​​corrected.

[0010] This invention derives conditional probabilities from the proportion of true categories under different output categories based on statistical analysis of real-vehicle perception sample data. It then uses these conditional probabilities to calculate and correct the safety risk value for perceived faults by adjusting the difference between the erroneous and correct conditional entropies. Conditional probabilities objectively reflect the frequency of various semantic misjudgments in real-vehicle scenarios, while conditional entropy quantifies the information uncertainty of the real environment. The entropy difference accurately characterizes the degree of environmental cognitive bias caused by different semantic misjudgments. This allows the corrected risk value to integrate the probability of real-vehicle fault occurrence with information uncertainty, ensuring that the safety risk value of the confusion matrix aligns with the statistical patterns of real-vehicle faults and guaranteeing the rationality of constructing subsequent perception fault test scenarios.

[0011] In one optional implementation, the step of correcting the security risk value corresponding to the perceived fault pair in the fault confusion matrix based on the difference between the error conditional entropy and the correct conditional entropy of each perceived fault pair includes: The difference between the error condition entropy and the correct condition entropy of each fault-sensing pair is determined as the priority weight of that fault-sensing pair. The priority weight of the perceived fault pair is multiplied by the security risk value corresponding to the perceived fault pair in the fault confusion matrix to obtain the corrected security risk value.

[0012] This invention prioritizes the perceived fault pair by setting the difference between the error conditional entropy and the correct conditional entropy as the priority weight. This weight is then multiplied by the original safety risk value to obtain a corrected risk value, thus completing the iterative update of the confusion matrix risk parameters. This priority weight represents the incremental uncertainty in environmental perception caused by single-class semantic misjudgment, while the original risk value represents the inherent hazard level of the fault. The combination of these two factors simultaneously considers both the inherent hazard of the fault and the probability of its occurrence in actual vehicles, optimizing the risk quantification result from a dual-dimensional perspective. This allows the corrected risk value to directly distinguish the comprehensive hazard level of different perceived fault pairs, achieving refined stratification of fault risk. This provides a precise quantitative basis for constructing subsequent test scenarios, avoiding misallocation of test resources due to relying solely on a single dimension to determine risk, and ensuring that high-probability, high-hazard semantic faults receive higher test priority.

[0013] In one optional implementation, constructing a set of fault-aware test scenarios based on the corrected fault confusion matrix includes: Based on the security risk values ​​in the corrected fault confusion matrix, determine the number of test scenarios corresponding to each security risk value and the perceived fault pair. Based on the number of test scenarios for different sensing fault pairs, construct sensing fault test scenarios for each sensing fault pair.

[0014] This invention determines the number of test scenarios corresponding to each perceived fault pair based on the security risk value of the modified confusion matrix, and then constructs corresponding fault test scenarios in batches according to the corresponding number. By using the quantified comprehensive risk value as the basis for allocating the number of test scenarios, it achieves differentiated allocation of test resources, ensuring that each type of semantic fault is matched with a corresponding number of independent test scenarios. This allows for precise and targeted injection of corresponding fused semantic errors, guaranteeing that every type of perceived misjudgment fault has simulation test condition coverage. It avoids the problems of insufficient high-risk fault scenarios and redundant low-risk fault scenarios, maximizing the coverage of high-risk semantic faults with limited simulation resources, and improving the targeting of fault injection testing and the efficiency of test scenario utilization.

[0015] In one optional implementation, determining the number of test scenarios corresponding to each perceived fault pair based on the security risk values ​​in the modified fault confusion matrix includes: Determine the total number of test scenarios for the simulation test; The objective is to minimize the overall fluctuation of the estimated failure rate of all perceived fault pairs, and the objective function is constructed with the constraint that the sum of the number of test scenarios for all perceived fault pairs equals the total number of test scenarios. The estimated failure rate of each perceived fault pair is determined by the safety risk value of that perceived fault pair. The estimated failure rate is used to characterize the estimated probability that the corresponding perceived fault pair will cause the simulation test of the autonomous driving target algorithm to fail. The objective function is solved to obtain the number of test scenarios for each perceived fault pair corresponding to each security risk value.

[0016] This invention utilizes the total number of test scenarios, constructs an objective function with the goal of minimizing the overall fluctuation of the estimated failure rate and the total number of scenarios as constraints, and solves for the optimal number of test scenarios for each fault pair. With minimizing the variance of the failure rate estimate as the core optimization direction, it achieves optimal resource allocation through mathematical optimization, avoiding the subjectivity and irrationality of manual allocation. Under the constraints of fixed simulation computing power and an upper limit on the number of simulations, it maximizes the data reliability of all semantic fault statistics, obtaining the highest accuracy fault risk statistics for the autonomous driving target algorithm with minimal testing resource consumption.

[0017] In an optional implementation, the method further includes: The safety risk value of each perceived fault pair is transformed using the Sigmoid function to obtain the estimated failure rate of each perceived fault pair.

[0018] This invention uses the Sigmoid function to transform safety risk values ​​to obtain the estimated failure rate for each perceived fault pair, establishing a quantitative mapping relationship between risk values ​​and failure probabilities. The Sigmoid function output is in the 0-1 range, consistent with the failure rate range, thus transforming the risk weights representing the overall risk level into statistically significant prior failure probabilities, providing standardized input parameters for the variance minimization optimization objective function. The unified transformation rule ensures consistent input parameters during the optimization process, avoiding computational distortion caused by inconsistent risk values ​​for different perceived fault pairs, and guaranteeing the stability and automated batch processing capability of the test resource allocation model.

[0019] In one optional implementation, determining the number of test scenarios corresponding to each perceived fault pair based on the security risk values ​​in the modified fault confusion matrix includes: Determine the total number of test scenarios for the simulation test; Sort the security risk values ​​in the corrected fault confusion matrix; Based on the ranking results and the total number of test scenarios, the number of test scenarios corresponding to each security risk value for the perceived fault pair is determined. The higher the security risk value of the perceived fault pair, the greater the number of test scenarios it corresponds to.

[0020] This invention utilizes the total number of test scenarios to sort the corrected safety risk values ​​and allocates the number of scenarios based on the sorting results, with higher risks corresponding to more scenarios. By relying solely on risk value sorting, test resource allocation can be quickly completed, reducing the computational complexity of the scenario allocation process and meeting the rapid modeling requirements of lightweight simulation testing platforms. It enables multiple rounds of testing for high-risk semantic faults and fewer testing rounds for low-risk faults, thereby concentrating limited simulation resources to cover catastrophic and high-level semantic misjudgment faults. This facilitates the rapid identification of failure issues in autonomous driving target algorithms under high-risk perceptual semantic errors, balancing the rationality of test scenario allocation with testing effectiveness.

[0021] In an optional implementation, after simulating the autonomous driving target algorithm using the perception fault test scenario set, the method further includes: The robustness of the proposed autonomous driving target algorithm is verified based on the simulation test results.

[0022] After completing the simulation test of the perception fault scenario, this invention verifies the robustness of the autonomous driving target algorithm based on the test results. Thus, after the fault simulation is completed, the algorithm safety quantitative assessment is completed directly based on the measured data. This achieves fully automated processing of fault injection, simulation operation, and robustness determination. It can output the safety performance conclusions of the autonomous driving target algorithm under various levels of semantic faults in batches, shortening the complete verification cycle of semantic faults in autonomous driving perception fusion.

[0023] In one optional implementation, the robustness verification of the autonomous driving target algorithm based on the simulation test results includes: Count the number of fault test scenarios that cause the simulation test results to fail for each perceived fault pair in the test results; Based on the actual failure rate corresponding to each perceived fault pair in the test results, a Bayesian probability model is established. The prior parameters of the Bayesian probability model are determined by the security risk value corresponding to each perceived fault pair. Based on the number of test scenarios and the number of fault test scenarios corresponding to each perceived fault pair, the Bayesian probability model is solved to determine the probability value of the pre-set upper bound of confidence corresponding to different driving safety risk levels. Based on the relationship between the probability value of the preset upper bound of confidence corresponding to each driving safety risk level and the probability threshold corresponding to that driving safety risk level, the robustness verification result of the autonomous driving target algorithm at that safety risk level is determined.

[0024] This invention constructs a Bayesian model based on the safety risk value as a prior parameter by statistically analyzing the number of failure scenarios for each perceived fault. It then calculates the probability of a pre-set upper bound on the confidence level for each risk level and determines robustness by comparing it to a threshold. This allows for the prior parameter setting of the risk value to be adjusted in the early stages, integrating prior fault patterns from real vehicles with the actual simulation data. By quantifying the statistical uncertainty of the risk probability through a pre-set upper bound on the confidence level, rather than relying solely on the failure rate of a single test, the evaluation results have statistical confidence. It can quantify the upper limit risk probability of the autonomous driving target algorithm under semantic faults of different danger levels, forming a standardized safety judgment conclusion with reliable statistical basis.

[0025] In one optional implementation, determining the robustness verification result of the autonomous driving target algorithm at a given safety risk level based on the relationship between the probability value of a preset confidence upper bound corresponding to each driving safety risk level and the probability threshold corresponding to that driving safety risk level includes: When the probability value of the preset upper bound of the confidence level corresponding to the current driving safety risk level is less than the probability threshold corresponding to the current driving safety risk level, the robustness verification result of the autonomous driving target algorithm at the current safety risk level is determined to be passed. If the probability value of the preset upper bound of the confidence level corresponding to the current driving safety risk level is not less than the probability threshold corresponding to the current driving safety risk level, the robustness verification result of the autonomous driving target algorithm at the current safety risk level is determined to be unsuccessful.

[0026] This invention determines robustness by comparing the probability of a pre-set upper bound of confidence level corresponding to a risk level with a preset safety threshold. If the probability of the pre-set upper bound of confidence level is less than the threshold, the verification passes; otherwise, it fails. Using a probability upper bound with confidence as the judgment criterion, compared to relying solely on the number of measured failures, fully considers the statistical errors caused by insufficient simulation sample size, avoiding misjudgments of algorithm robustness under small sample testing. A unified threshold judgment standard enables automated, uninterrupted robustness output, allowing different test batches and different vehicle models of autonomous driving systems to use the same set of judgment rules to compare safety performance, facilitating the establishment of standardized autonomous driving perception semantic fault access verification specifications.

[0027] In an optional implementation, when it is determined that the robustness verification result of the autonomous driving target algorithm at the current safety risk level fails, the method further includes: Filter target perceived fault pairs that belong to the current security risk level; Based on the target perception fault pair, the autonomous driving target algorithm is optimized and updated; Based on the target perceived fault pair, construct a set of secondary perceived fault test scenarios; The optimized and updated autonomous driving target algorithm is simulated and tested using the secondary perception fault test scenario set, and the test results based on the simulation test are re-executed to verify the robustness of the autonomous driving target algorithm.

[0028] This invention selects perception fault pairs of the same risk level when verification fails, optimizes the autonomous driving target algorithm based on these fault pairs, and builds a secondary perception fault test scenario set for retesting. This avoids the waste of R&D resources caused by indiscriminate full optimization of the autonomous driving target algorithm; and it constructs separate retest scenarios for problematic perception fault pairs to accurately verify whether the algorithm optimization measures effectively correct decision-making defects under semantic faults. The iterative testing mechanism continuously reduces the probability of vehicle failure caused by high-risk semantic faults and continuously improves the fault tolerance capability of the autonomous driving target algorithm for perception semantic errors.

[0029] Secondly, the present invention provides an autonomous driving simulation testing device, the device comprising: The acquisition module is used to acquire the set of all output categories corresponding to at least one perception model of the vehicle. The first processing module is used to form a perception fault pair by pairing each output category with each real category for each perception model, and to evaluate the security risk value of each perception fault pair. The second processing module is used to construct a fault confusion matrix based on each of the perceived fault pairs, and to determine the security risk value as the matrix element of the corresponding perceived fault pair in the fault confusion matrix; The third processing module is used to acquire the real vehicle perception sample data corresponding to the perception model, and to correct each of the safety risk values ​​in the fault confusion matrix based on the real vehicle perception sample data. The fourth processing module is used to construct a set of perception fault test scenarios based on the corrected fault confusion matrix, and to test the autonomous driving target algorithm using each of the perception fault test scenarios.

[0030] Thirdly, the present invention provides a vehicle comprising: a controller, the controller comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method provided in the first aspect or any corresponding embodiment thereof.

[0031] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method provided in the first aspect or any corresponding embodiment thereof.

[0032] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to perform the method provided in the first aspect or any corresponding embodiment thereof.

[0033] The technical solution provided by this invention has the following beneficial effects: This invention forms perception fault pairs by matching the output category and the true category of the perception model, constructs a confusion matrix by evaluating the safety risk value of each perception fault pair, and then corrects the safety risk value using real-vehicle perception sample data from the perception model. Finally, based on the corrected safety risk value, a fault test scenario is constructed to simulate and test the autonomous driving target algorithm. This quantifies the semantic-level fault risk of perception fault pairs and standardizes the storage of all semantic misjudgment types using the confusion matrix. Real-vehicle sample correction ensures that the risk value closely matches the actual fault distribution of the perception model. The test scenario generated based on the corrected confusion matrix can cover various types of perception semantic errors, systematically verifying the operational state of the autonomous driving target algorithm when receiving erroneous semantic information. This achieves full coverage and standardized simulation testing of sensor fusion semantic output faults, improving the completeness of fault verification from autonomous driving perception to autonomous driving algorithm. Attached Figure Description

[0034] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is a flowchart of a fault injection test for an autonomous driving system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the first type of autonomous driving simulation test method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a second process for an autonomous driving simulation test method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of test resource allocation and scene generation according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the working principle of the fault parameter interface according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the third process of the autonomous driving simulation test method according to an embodiment of the present invention; Figure 7 This is a structural block diagram of an autonomous driving simulation test device according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the hardware structure of the vehicle controller according to an embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0038] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0039] In related technologies, the main approach is to test the autonomous driving function of the vehicle under test in a target simulation test scenario using simulation software, fault injection boards, and control equipment. However, most research focuses only on the testing of fault injection into the raw data, meaning that the actual object being tested is the fusion perception algorithm itself, rather than on the impact of the fusion perception output on the autonomous driving function algorithm (hereinafter also known as the control algorithm).

[0040] Therefore, there is a lack of a testing method that can specifically and systematically interfere with the high-level semantic results output by the sensor fusion module. Existing fault injection methods mostly occur before or after perception, but they skip the core step of "perception understanding." There is an urgent need for a simulation test scheme that can deeply verify whether the vehicle control algorithm can still make safe and reasonable decisions when it receives "clear but incorrect" sensor fusion semantic output information.

[0041] To address the aforementioned issues, this invention provides a fault injection testing scheme for autonomous driving systems. This scheme proactively and systematically tests the robustness of control algorithms when receiving erroneous semantic information, thereby enabling deeper and more comprehensive verification of the autonomous driving system. This testing scheme employs a hierarchical testing framework, such as... Figure 1As shown, the main components include semantic fault modeling, constructing triggering scenarios, performing closed-loop testing, and conducting quantitative evaluation. The main testing process includes: Semantic fault modeling: By defining a set of standardized sensor fusion semantic fault models, namely fault confusion matrices, and correcting the standardized semantic fault models based on a large amount of real vehicle perception fusion output fault data, a corrected perception fusion semantic model is obtained.

[0042] Simulation environment construction: In the simulation platform, based on the selected modified perception fusion semantic model, the corresponding sensor fusion output module and other vehicle body signal modules are developed and loaded. The fusion perception semantic fault parameter setting panel and fault parameter YAML configuration file are also developed. The actual vehicle test control algorithm module, human-machine interaction module, state machine module and control module are connected.

[0043] Test scenario optimization and optimal allocation: Based on the above modified model, further processing is carried out to optimally allocate test resources, define the test benefit function, construct the optimization problem, solve and generate a set of test scenarios that can most effectively verify the robustness of the system, and complete the automated simulation.

[0044] Robustness Quantitative Assessment: Establish a Bayesian model, update the posterior distribution, calculate the upper bound of risk confidence and determine robustness.

[0045] Compared with existing technologies, the technical solution provided by this invention is specifically targeted at the vertical field of autonomous driving, and its focus is directly on sensor fusion semantic faults, solving the unique challenge of perception fusion fault testing in dense obstacle environments. By introducing a semantic fault risk quantification model based on confusion matrix and information theory (conditional entropy), a theoretical risk weight is calculated for each possible semantic error. The "degree of danger" is transformed from subjective experience-based judgment into a calculable, objective increment of information uncertainty. Furthermore, the initial theoretical model is empirically calibrated using a large amount of real-vehicle data, enabling the fault risk weights to more realistically reflect the statistical laws of the real world, continuously approaching the true risk distribution as data accumulates. An optimal test resource allocation model based on risk weights is implemented. By solving an optimization problem aimed at "minimizing the overall risk estimation variance," limited test resources (such as the number of simulations) are concentrated on allocating to the faults with the highest risk and highest uncertainty, achieving a shift from "uniform testing" to "optimal allocation." Finally, through Bayesian risk updates and statistical inference, the test results are transformed into a safety statement with clear confidence.

[0046] According to an embodiment of the present invention, an embodiment of an autonomous driving simulation test method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] This embodiment provides an autonomous driving simulation testing method, which can be used in electronic devices such as computers and servers, and can also be applied to vehicle controllers such as microcontrollers and CPUs. Figure 2 This is a flowchart of an autonomous driving simulation test method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the set of all output categories corresponding to at least one perception model of the vehicle.

[0048] The perception model in this context refers to a sensor fusion perception model within an autonomous driving system. This differs from single-sensor raw data acquisition modules such as cameras, LiDAR, and ultrasonic sensors. It is an algorithmic model used to process raw signals from one or more sensors and output high-level semantic recognition results, providing a basis for judgment in downstream regulatory algorithms. For example, this perception model can be a single model integrating image target recognition algorithms, an algorithmic model fusing image data and radar data for joint target recognition, or an algorithmic model integrating semantic recognition, such as a FusObjects model or a lane line static perception model. This invention is not limited to these limitations.

[0049] Furthermore, the sensor input data corresponding to perception models integrating different algorithms are different, and the corresponding output categories are also different. The outputs of different perception models include, but are not limited to, fused target object list (FusObjects), fused point cloud parameters of traversable area (Freespace), fused output of ultrasonic sector detection distance (Sector), fused output of target object boundary (Cluster), fused output of static environment perception (Env), and fused output of self-pose information (Odometry).

[0050] The set of all output categories corresponding to each perception model is all the semantic categories that the model can output. For example, for the FusObjects model, the output category set FUSOBJ_TYPE is extracted as {cars, vehicles, two-wheeled vehicles, pedestrians, animals, walls, low curbs, traffic cones, water-filled barriers, unknown}; for the lane line static perception model, the output set LANEMARKING_TYPE is extracted as {solid lines, dashed lines, double solid lines, double dashed lines, solid-dashed lines, dashed-solid lines, unknown}.

[0051] Step S202: For each perception model, each output category is paired with each real category to form a perception fault pair, and the security risk value of each perception fault pair is evaluated.

[0052] Here, the true category represents the category that matches the real environment object among all output categories. For example, if the real environment object is a pedestrian, then the true category corresponds to pedestrian; or if the real environment object is a car, then the true category corresponds to car. Each output category in the output category set is combined with a true category to form a perception fault pair. For example, if the output category car and the true category pedestrian form a perception fault pair, it indicates that the perception model misidentifies a certain true category as a certain output category, resulting in a perception fault. In particular, when the true category and the output category are the same, the perception fault pair indicates that the perception model has no perception fault.

[0053] For example, for a single perception model, the set of true environmental ground values ​​(GT) corresponding to the true category and the set of fused output values ​​(FO) corresponding to the perception model's output category are permuted and paired. Each pair (true category GT, fused output FO) constitutes an independent perception fault pair. If GT and FO are consistent, it is a fault-free sample; otherwise, it is a semantic fault sample. An initial risk assessment can be performed on each perception fault pair based on human experience or the degree of harm that the perception fault pair may cause to driving safety, resulting in a safety risk value for each perception fault pair. The larger the safety risk value, the greater the degree of harm to driving safety caused by the corresponding perception fault pair; conversely, the smaller the safety risk value, the smaller the degree of harm to driving safety caused by the corresponding perception fault pair.

[0054] Step S203: Construct a fault confusion matrix based on each perceived fault pair, and determine the safety risk value as the matrix element of the corresponding perceived fault pair in the fault confusion matrix.

[0055] Specifically, a two-dimensional fault confusion matrix is ​​constructed using the true category GT as the matrix rows and the fused output FO as the matrix columns. Each cell in the matrix stores the initial safety risk value of the corresponding perceived fault pair, and the diagonal cells where GT and FO are the same are uniformly filled with the risk-free value 0. It should be noted that different perception models correspond to different fault confusion matrices.

[0056] For example, the fault confusion matrix constructed for the FusObjects model is shown in Table 1, and the fault confusion matrix constructed for the lane line static perception model is shown in Table 2.

[0057] Table 1

[0058] Table 2

[0059] Step S204: Obtain the actual vehicle perception sample data corresponding to the perception model, and correct each safety risk value in the fault confusion matrix based on the actual vehicle perception sample data.

[0060] The real-vehicle perception sample data consists of the actual output category and the actual real category of the perception model under different real driving conditions. It can be extracted from real-vehicle road tests or historical operating data of the vehicle. Since the safety risk value in the above-mentioned fault confusion matrix is ​​determined by theory or human experience, there may be some deviation. Therefore, the deviation can be corrected by the real-vehicle perception sample data during the actual operation of the vehicle, so as to further ensure the consistency between the safety risk value and the actual vehicle operation.

[0061] Step S205: Based on the corrected fault confusion matrix, construct a set of perception fault test scenarios, and use the set of perception fault test scenarios to conduct simulation tests on the autonomous driving target algorithm.

[0062] Specifically, by obtaining all corrected safety risk values ​​within the confusion matrix, allocating the number of test scenarios corresponding to each perceived fault pair based on the risk values, generating several exclusive simulation-induced scenarios for each fault pair, and continuously injecting fused perception outputs with specified semantic errors into the autonomous driving target algorithm, i.e., the controlled algorithm under test, to form a simulation closed loop, driving the vehicle model to run and collecting full-time simulation data, thus completing the simulation verification of the controlled algorithm.

[0063] For example, for the perception fault pair with the highest risk after correction (pedestrian, car), more simulation test scenarios are assigned. For instance, in parking simulation, a pedestrian is set to appear on the side of the parking space. The recognition result of the fusion module is forced to be a car through the fault configuration file. The erroneous semantics are input into the parking control algorithm, and the decision-making behaviors of the vehicle such as deceleration, obstacle avoidance, and parking path are observed to complete the algorithm test under this high-risk semantic fault.

[0064] This invention constructs a confusion matrix by forming perception fault pairs from the output category and the true category of the perception model, evaluating the safety risk value of each perception fault pair, and then correcting the safety risk value using real-vehicle perception sample data from the perception model. Finally, based on the corrected safety risk value, a set of perception fault test scenarios is constructed to simulate and test the autonomous driving target algorithm. This quantifies the semantic-level fault risk of perception fault pairs and standardizes the storage of all semantic misjudgment types using the confusion matrix. The real-vehicle sample correction ensures that the risk value matches the actual fault distribution of the perception model. The test scenarios generated based on the corrected confusion matrix can cover various types of perception semantic errors, systematically verifying the operating state of the autonomous driving target algorithm when receiving erroneous semantic information. This achieves full coverage and standardized simulation testing of sensor fusion semantic output faults, improving the completeness of fault verification from autonomous driving perception to autonomous driving algorithm.

[0065] This embodiment provides an autonomous driving simulation testing method, which can be used in electronic devices such as computers and servers, and can also be applied to vehicle controllers such as microcontrollers and CPUs. Figure 3 This is a flowchart of an autonomous driving simulation test method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps: Step S301: Obtain the set of all output categories corresponding to at least one perception model of the vehicle. See details below. Figure 2 The relevant descriptions of step S201 shown will not be repeated here.

[0066] Step S302: For each perception model, each output category is paired with each real category to form a perception fault pair, and the security risk value of each perception fault pair is evaluated.

[0067] Specifically, the evaluation of the security risk value for each perceived fault pair in step S302 above includes: Step a1: Based on the preset driving safety risk level classification conditions, assess the driving safety risk level of each perceived fault pair and determine the driving safety risk level of each perceived fault pair.

[0068] The preset driving safety risk level classification conditions can be set based on human experience or the degree of harm that a perceived fault may cause to driving safety, such as classifying it into three or five levels, etc., and this invention is not limited thereto. For example, a five-level driving safety risk level classification standard is set as a unified basis for classifying perceived faults, and the specific definition of each level is as follows: Level I catastrophic failure: The failure is extremely likely to directly cause collisions between the vehicle and pedestrians, other road users, or result in fatal serious traffic violations; Level II high-risk fault: The fault is highly likely to trigger unexpected and violent operations such as sudden braking and sharp turning of the vehicle without reason, which may lead to secondary accidents such as rear-end collisions and loss of control. Level III Medium Risk Fault: The fault only causes the vehicle to make suboptimal decisions, reducing traffic and parking efficiency, and will not trigger a safety accident in the short term; Level IV low-risk fault: The fault only causes the autonomous driving function to be degraded. The system automatically activates a conservative backup strategy and there is no immediate risk of collision. Level V: Risk-free: The category output by the perception fusion is completely consistent with the category of the real environment, and there is no semantic misjudgment.

[0069] In the fault confusion matrix in Table 1 above, F-O28 mistakenly identifies a pedestrian (soft target, highly dynamic, irregular movement) as a vehicle (usually with a more regular motion model). This semantic fault is catastrophic, causing the control algorithm to severely underestimate the severity of the collision and adopt inappropriate collision avoidance strategies, which can easily lead to fatal accidents. F-O1 mistakenly identifies a small car as a large vehicle, which is of medium risk and may lead to excessive following distance and overtaking decisions. In the fault confusion matrix in Table 2 above, F-L13 mistakenly outputs double solid lines as dashed lines, which is catastrophic. In the lane line type semantic fault confusion matrix, F-L25 mistakenly outputs solid-dashed lines as solid lines, which is of medium risk. Only the lane on the solid line side is prohibited from changing lanes. Misjudging it as a complete lane prohibition may excessively restrict the behavior of the vehicle or other vehicles.

[0070] For example, for the perception fault pair (GT=pedestrian, FO=car), according to the classification conditions, this misjudgment will cause the parking control algorithm to underestimate the pedestrian collision risk and adopt a dangerous obstacle avoidance strategy, which is very likely to cause a fatal collision, and is matched as a Level I catastrophic fault; for the lane line perception model fault pair (GT=solid-dashed line, FO=solid line), it will only over-restrict the vehicle's lane changing behavior and will not directly cause a collision, and is matched as a Level III medium-risk fault.

[0071] Step a2: Based on the preset correspondence between different driving safety risk levels and score values, determine the score value corresponding to the driving safety risk level of each perceived fault pair.

[0072] Specifically, by pre-establishing a one-to-one mapping rule between risk levels and fixed numerical scores, this mapping relationship is used to reflect the relationship between risk levels and score values: for example, the score value corresponding to catastrophic level I is 1, the score value corresponding to high risk level II is 0.75, the score value corresponding to medium risk level III is 0.5, the score value corresponding to low risk level IV is 0.25, and the score value corresponding to no risk level V is 0.

[0073] Step a3: Determine the score corresponding to the driving safety risk level of each perceived fault pair as the safety risk value of that perceived fault pair.

[0074] Specifically, after determining the risk level of a single group of perceived fault pairs, the fixed score value bound to that level is directly retrieved as the safety risk value of that perceived fault pair.

[0075] Taking the fault confusion matrix in Table 1 above as an example, F-O28 corresponds to GT=pedestrian → FO=car. Catastrophic fault: Misclassifying pedestrians (soft targets, highly dynamic, irregular motion) as vehicles (usually with more regular motion models). This causes the regulation control algorithm to severely underestimate the severity of collisions and adopt inappropriate collision avoidance strategies (such as expecting "vehicles" to follow the rules), which can easily lead to fatal accidents. F-O46: GT=wall → FO=car. High-risk fault: Misclassifying static obstacles as dynamic vehicles. This causes the regulation control algorithm to make incorrect following or overtaking decisions for non-existent "vehicles in front," which may cause vehicles to deviate from the safe path and increase the risk of collisions with real obstacles or other traffic participants. F-O1: GT=car → FO=large vehicle. Medium risk: Incorrectly estimating the size of the vehicle in front. This may lead to following distances that are too far (affecting traffic flow) or too close (underestimating braking distance), and may affect overtaking decisions (being more cautious with large vehicles).

[0076] Taking the fault confusion matrix in Table 2 above as an example, F-L1: GT=solid line → FO=dashed line. High risk: Lane changing is allowed in areas where lane changing is prohibited, which is likely to cause collisions or violations. F-L25: GT=solid-dashed line → FO=solid line. Moderate risk: Lane changing is prohibited only in the lane with the solid line. Misjudging it as a complete lane prohibition will excessively restrict the behavior of this vehicle or other vehicles. F-L13: GT=double solid line → FO=dashed line. Catastrophic: Double solid lines are strictly prohibited from being crossed. Any misjudgment as a line that can be crossed will lead to serious violations and an extremely high risk of collision.

[0077] This invention classifies perceived fault pairs according to preset risk level classification conditions and matches preset score values ​​of each level as safety risk values, achieving a standardized initial judgment of semantic fault risk. By using unified classification rules to quantify the risk of all perceived fault pairs, all pairs have directly comparable standardized risk values, serving as the basic elements of the confusion matrix. This allows for rapid risk assignment of large-scale perceived fault pairs without complex calculations, significantly reducing the computational and human resource costs of initial fault modeling. It provides a unified and quantifiable basic risk benchmark for subsequent real-vehicle data correction and test resource allocation, ensuring a unified risk assessment standard throughout the fault testing process and facilitating subsequent automated batch processing of massive amounts of perceived fault pairs.

[0078] Step S303: Construct a fault confusion matrix based on each perceived fault pair, and determine the safety risk value as the matrix element of the corresponding perceived fault pair in the fault confusion matrix. See details below. Figure 2 The relevant description of step S203 shown will not be repeated here.

[0079] Step S304: Obtain the actual vehicle perception sample data corresponding to the perception model, and correct each safety risk value in the fault confusion matrix based on the actual vehicle perception sample data.

[0080] Specifically, the real-vehicle perception sample data includes: the real-vehicle output category and the corresponding real-vehicle true category, and step S304 above includes: Step S3041: Obtain the real vehicle perception sample data corresponding to the perception model. For each real vehicle output category, count the number of first samples of the current real vehicle output category in the real vehicle perception sample data and the number of second samples in the first sample number that are the same as the current real vehicle output category.

[0081] Specifically, by importing perception sample data from the perception model collected from real vehicles, each sample record contains the actual true category GT and the output category FO output by the perception model. For each output category FO, two levels of statistics are performed: the first sample number is the total number of all samples in all output categories that are equal to that FO; the second sample number is the number of samples in the first sample set that completely match the true GT and FO.

[0082] For example, taking the output category "car" of the fused target object perception model as an example, we traversed all real vehicle data and counted a total of 8,620 records of all output categories FO=car. This number is the first sample count. Among them, there are 7,956 samples of the real environment itself being a car, i.e., GT=car. This number is the second sample count. Similarly, we counted the number of samples of all output categories such as FO=pedestrian, FO=dashed line, etc.

[0083] Step S3042: Calculate the conditional probability of the current vehicle output category based on the ratio of the second sample size to the first sample size.

[0084] Specifically, the two sets of sample counts obtained in step S3041 above are divided by the number of the second sample and the number of the first sample to obtain the conditional probability P(X=FO|Y=FO) that the real environment is also FO when the known fusion output is the current FO; the remaining probability 1-P(X=FO|Y=FO) represents the overall probability of semantic misjudgment under this output category.

[0085] For example, continuing with the sample data of FO=car from the previous step, the second sample number 7956 divided by the first sample number 8620 yields a conditional probability of approximately 0.923, which means that when the fused output is identified as a car, the probability that the real scene is indeed a car is 92.3%; the remaining 7.7% is the misjudgment probability of various objects being misidentified as cars.

[0086] Step S3043: Based on the conditional probabilities corresponding to different real vehicle output categories, calculate the error conditional entropy and the correct conditional entropy for each perceived fault pair.

[0087] Specifically, by using the information-theoretic conditional entropy formula as the calculation benchmark, Represents a collection of objects in the real environment. This represents the fusion output set of the perception model. For any set of perception fault pairs... ,in For the real GT category, The fusion output FO category of the perception model: Correct conditional entropy This represents the uncertainty entropy of the real environment and the correct conditional entropy when the fused output matches the true category. The calculation formula is: Error conditional entropy Indicates that the merged output is an error category. At that time, the uncertainty entropy of the real environment, the entropy of error conditions The calculation formula is: Step S3044: Based on the difference between the error conditional entropy and the correct conditional entropy of each perceived fault pair, correct the security risk value corresponding to the perceived fault pair in the fault confusion matrix.

[0088] Specifically, for each perceived fault pair, the larger the difference between the error conditional entropy and the correct conditional entropy, the higher the uncertainty in environmental cognition caused by the semantic misjudgment; by using this difference to correct the security risk value in the confusion matrix, the subjective bias caused by human experience setting is replaced.

[0089] This invention, in its embodiments, derives conditional probabilities from the proportion of true categories under different output categories based on statistical analysis of real-vehicle perception sample data. It then uses these conditional probabilities to calculate and correct the safety risk value for perceived faults by adjusting the difference between the erroneous and correct conditional entropies. Conditional probabilities objectively reflect the frequency of various semantic misjudgments in real-vehicle scenarios, while conditional entropy quantifies the information uncertainty of the real environment. The entropy difference accurately characterizes the degree of environmental cognitive bias caused by different semantic misjudgments. This allows the corrected risk value to integrate the probability of actual fault occurrence with information uncertainty, ensuring that the safety risk value of the confusion matrix aligns with the statistical patterns of real-vehicle faults and guaranteeing the rationality of constructing subsequent perception fault test scenarios.

[0090] Further, step S3044 above includes: Step b1: The difference between the error condition entropy and the correct condition entropy of each fault-sensing pair is determined as the priority weight of that fault-sensing pair.

[0091] Specifically, since the aforementioned definition of the danger of safety risk value is derived solely from human experience and theoretical deduction, in order to increase the authenticity and robustness of the safety risk value, and to systematically manage fault models, high-risk faults are prioritized for testing. Based on a large amount of real vehicle perception data, perception fault information is automatically extracted and processed. Furthermore, an information theory-based weight allocation method is introduced, so that the safety risk value after weight allocation correction can more realistically and objectively reflect the safety risk situation of the real vehicle, providing an accurate basis for subsequent testing of autonomous driving target algorithms.

[0092] Step b2: Multiply the priority weight of the perceived fault pair by the corresponding security risk value of the perceived fault pair in the fault confusion matrix to obtain the corrected security risk value.

[0093] Specifically, for the priority weights of the fault-sensing pairs It can be calculated using the following formula: in, It is the conditional entropy, representing the condition where the fusion output is known to be... In real-world environments Uncertainty regarding priority weights. The higher, the more likely it is to be Misclassified as The greater the risk, the higher the priority weight. The calculation formula is as follows: Based on the perceived fault data collected from actual vehicles, the corresponding priority weights in each fault confusion matrix are calculated. Then, the calculated priority weights will be... Safety risk values ​​corresponding to each fault risk level Combined with the corrected security risk value The specific calculation formula is as follows: For example, using a fault perception pair (pedestrian, vehicle), the error conditional entropy calculated from real vehicle perception data is much greater than the correct conditional entropy, resulting in a higher entropy difference and corresponding higher priority weight. Conversely, for the fault perception pair (vehicle, vehicle), the entropy difference between the two types is smaller, leading to a lower weight. Thus, by utilizing the entropy difference to quantify the true uncertainty of different faults, objective calibration of the safety risk value in the fault confusion matrix based on real vehicle statistical patterns is achieved, solving the problem of relying solely on manual experience for value assignment and the disconnect from the actual road test fault distribution.

[0094] This invention employs a method where the difference between the error conditional entropy and the correct conditional entropy of a perceived fault pair is set as the priority weight for that pair. This weight is then multiplied by the original safety risk value to obtain a corrected risk value, thus completing the iterative update of the confusion matrix risk parameters. This priority weight represents the incremental uncertainty in environmental perception caused by a single type of semantic misjudgment, while the original risk value represents the inherent hazard level of the fault. The combination of these two factors simultaneously considers both the inherent hazard of the fault and the probability of its occurrence in a real vehicle, optimizing the risk quantification result from a dual-dimensional perspective. This allows the corrected risk value to directly distinguish the comprehensive hazard level of different perceived fault pairs, achieving refined stratification of fault risk. This provides a precise quantitative basis for constructing subsequent test scenarios, avoiding misallocation of test resources due to relying solely on a single dimension to determine risk, and ensuring that high-probability, high-hazard semantic faults receive higher test priority.

[0095] Step S305: Based on the corrected fault confusion matrix, construct a set of perception fault test scenarios, and use the set of perception fault test scenarios to conduct simulation tests on the autonomous driving target algorithm.

[0096] Due to the limited amount of real-vehicle fusion-perception fault data available, in order to improve the provable and high-confidence fault scenarios, the corrected safety risk value corresponding to each group of perceived faults in the corrected fault confusion matrix is ​​read. This value is used as the basis for allocating the number of scenarios, and each type of perceived fault is assigned a corresponding number of simulation tests. The higher the corrected safety risk value, the more test scenarios are allocated.

[0097] Specifically, step S305 includes: Step S3051: Based on the security risk values ​​in the corrected fault confusion matrix, determine the number of test scenarios for each security risk value corresponding to the perceived fault pair.

[0098] Furthermore, in some optional embodiments, step S3051 above includes: Step c1: Determine the total number of test scenarios for the simulation test.

[0099] In this simulation, the total number of test scenarios that each perception model can be deployed in is pre-defined. This value is determined by a combination of factors, including the computing power of the simulation platform and the constraints of the testing cycle.

[0100] Step c2: The optimization objective is to minimize the overall fluctuation of the estimated failure rate of all perceived fault pairs, and the objective function is constructed with the constraint that the sum of the number of test scenarios for all perceived fault pairs equals the total number of test scenarios. The estimated failure rate of each perceived fault pair is determined by the safety risk value of that perceived fault pair. The estimated failure rate is used to characterize the estimated probability that the corresponding perceived fault pair will cause a failure in the simulation test of the autonomous driving target algorithm.

[0101] Specifically, this is achieved by constructing an optimization objective function and constraints: the optimization objective is to minimize the weighted variance of the predicted failure rate for all fault pairs, and the constraint is that the sum of the number of assigned scenarios for all fault pairs equals the total field. The number of scenarios and the number of single-type fault scenarios are greater than or equal to 0. The specific objective function is as follows: in, , It is a fault detection pair The estimated failure rate, which represents the rate used to characterize the perceived failure pair. The estimated probability of failure in simulation testing of autonomous driving target algorithms. Indicates the perceived fault pair The corresponding number of test scenarios Indicates the perceived fault pair Security risk values ​​in the revised fault confusion matrix.

[0102] The above method adds a squared weighting term to the perceived fault pairs with high security risk values ​​in the objective function, forcing the optimization algorithm to prioritize reducing the statistical variance of these pairs. This estimates the failure rate. The variance is approximately expressed as follows:

[0103] In the above formula, multiply This indicates that for high-risk ( For high-risk (high-risk) perceived fault pairs, a smaller estimated variance of their corresponding fault rate is required (i.e., fewer test runs). (We need more), because the risks and costs associated with its uncertainty are higher.

[0104] In some optional implementations, the estimated failure rate of each perceived fault pair is obtained by transforming the security risk value of each perceived fault pair using a Sigmoid function. The specific transformation formula is as follows: in, This represents the Sigmoid function. Represents the coefficient.

[0105] This invention utilizes the Sigmoid function to transform safety risk values ​​to obtain the estimated failure rate for each perceived fault pair, establishing a quantitative mapping relationship between risk values ​​and failure probabilities. The Sigmoid function output is in the 0-1 range, consistent with the failure rate range, thus transforming the risk weights representing the overall risk level into statistically significant prior failure probabilities. This provides standardized input parameters for the variance minimization optimization objective function. The unified transformation rule ensures consistent input parameters during the optimization process, avoiding computational distortion caused by inconsistent risk values ​​for different perceived fault pairs, and guaranteeing the stability and automated batch processing capability of the test resource allocation model.

[0106] Step c3: Solve the objective function to obtain the number of test scenarios for each perceived fault pair corresponding to each security risk value.

[0107] Specifically, the solution method and specific solution process for the above objective function are existing technologies and can be implemented using existing technologies, so they will not be elaborated here.

[0108] For example, let's set the overall scenario for this simulation. With a threshold of 100, the corrected safety risk values ​​corresponding to all perceived fault pairs are substituted into the objective function for solution. Taking a total of three perceived fault pairs as an example, the allocation results are as follows: catastrophic faults (pedestrian misidentification of a car) are assigned to 61 test scenarios, high-risk wall misidentification of a car is assigned to 38 scenarios, and low-risk curb misidentification of a wall is assigned to only 1 scenario. This optimal allocation scheme minimizes the fault rate estimation error of high-risk semantic faults within a limited 100 simulations, resulting in higher confidence in the test statistics and avoiding the subjective imbalance problem caused by manual scenario allocation.

[0109] This invention utilizes the total number of test scenarios, constructs an objective function with the goal of minimizing the overall fluctuation of the estimated failure rate and the total number of scenarios as constraints, and solves for the optimal number of test scenarios for each fault pair. With minimizing the variance of the failure rate estimate as the core optimization direction, optimal resource allocation is achieved through mathematical optimization, avoiding the subjectivity and irrationality of manual allocation. Under the constraints of fixed simulation computing power and an upper limit on the number of simulations, the data reliability of all semantic fault statistics is maximized, obtaining the highest accuracy fault risk statistics for the autonomous driving target algorithm with minimal testing resource consumption.

[0110] Furthermore, in some alternative embodiments, step S3051 above includes: Step d1: Determine the total number of test scenarios for the simulation test. The specific details are described in step c1 above and will not be repeated here.

[0111] Step d2: Sort the security risk values ​​in the corrected fault confusion matrix.

[0112] Step d3: Based on the sorting results and the total number of test scenarios, determine the number of test scenarios for each security risk value corresponding to the perceived fault pair. The higher the security risk value of the perceived fault pair, the larger the number of test scenarios it corresponds to.

[0113] Specifically, the number of scenarios is allocated level by level according to the ranking of safety risk values, following the allocation logic that the higher the modified safety risk value, the more test scenarios are allocated. For example, a tiered allocation rule can be adopted: catastrophic failures are uniformly allocated the highest level of scenarios, followed by high-risk, medium-risk, and low-risk scenarios, which decrease in turn. The total number of all failure scenarios is equal to the preset total number of scenarios.

[0114] For example, with a total of 30 test scenarios, all perceived fault pairs are sorted in descending order of their corresponding safety risk values, and a tiered allocation rule is set: 18 scenarios are allocated to each group for catastrophic faults, 8 for high-risk faults, 3 for medium-risk faults, and 1 for low-risk faults. This eliminates the need to solve complex optimization equations, resulting in extremely low computational load. The lightweight simulation platform can quickly complete resource allocation, tilting limited simulation resources towards high-risk semantic faults to rapidly locate failure defects in the control algorithm under severe perceptual semantic misjudgment.

[0115] This invention utilizes the total number of test scenarios to sort the corrected safety risk values ​​and allocate the number of scenarios based on the sorting results, with higher risks corresponding to more scenarios. By relying solely on risk value sorting, test resource allocation can be quickly completed, reducing the computational complexity of the scenario allocation process and meeting the rapid modeling requirements of a lightweight simulation testing platform. It enables multiple rounds of testing for high-risk semantic faults and fewer testing rounds for low-risk faults, thereby concentrating limited simulation resources to cover catastrophic and high-level semantic misjudgment faults. This facilitates the rapid identification of failure issues in autonomous driving target algorithms under high-risk perceptual semantic errors, balancing the rationality of test scenario allocation with testing effectiveness.

[0116] Step S3052: Construct the perception fault test scenario for each perception fault pair according to the number of test scenarios for different perception fault pairs.

[0117] Specifically, for each perceived fault pair ,generate Each corresponding simulation test scenario, based on The system uses automated scripts to generalize simulation test scenario files (XML format). Generalization parameters include, but are not limited to, the vehicle's attitude, speed, and relative position to targets from the fused sensors. It also performs similar generalizations on other vehicles. Fault scenarios are matched with YAML configuration files containing the same fault parameters. For example, the test resource allocation and scenario generation process is as follows: Figure 4 As shown.

[0118] Step S3053: Simulate and test the autonomous driving target algorithm using various perception fault test scenarios.

[0119] Specifically, a simulation environment is built on the simulation platform, integrating the actual vehicle's test control algorithm module, human-machine interaction module, state machine module, and control module, and loading the simulation test scenario. In practical applications, the simulation platform can be enhanced with interfaces for fusion-based fault injection signal simulation development and fault parameter configuration. The fault parameter configuration interface consists of two parts: a fusion-based perception semantic fault parameter setting panel and a fault parameter YAML configuration file.

[0120] Specifically, for simulating fault injection signals in perception fusion, a simulated sensor fusion module is developed, and perception results with expected semantic errors are output. For example, the perception results include, but are not limited to: The FusObjects list contains targets that are misclassified or have biased attributes. Examples include targets that are misclassified, have extremely low classification confidence, have incorrect dynamic / static states, or exhibit changes in the IDs of multiple targets.

[0121] When merging point clouds into traversable freespace regions, the boundary points may suffer from shape distortion or changes in point attributes due to semantic errors. These errors could include variations in the location of specific probe points, sudden changes in the location of target objects, or errors in the point attributes (type, motion state, etc.) of target objects.

[0122] The ultrasonic sector detection range fusion output may contain errors due to detection errors. These errors can include probabilistic range jumps caused by detecting specific obstacles, and jumps at sector boundaries. The target object boundary (Cluster) fusion output contains obstacle boundary lines that may contain misclassifications or positional errors. These errors may include boundary line position deviations, incorrect boundary line types, or confusion between the boundary lines of high-risk obstacles (such as pedestrians) and the boundary lines of other obstacles (such as vehicles).

[0123] The static environment perception (Env) fusion output contains incorrect classifications or ID errors in static road information such as parking space information, lane lines, lane boundary lines, and intersection information. Examples include incorrect parking space type (perpendicular, horizontal, angled, perpendicular-angled, etc.), and offset parking space corner positions. Specifically, in valet parking AVP tests, there may be discrepancies between the perceived parking space and the high-precision map's parking space position; errors in the upstream and downstream connections of lane lines / lane boundary lines; and incorrect intersection IDs.

[0124] The fusion output of the vehicle's own pose information (Odometry) includes probabilistic jump anomalies in the vehicle's position (global coordinates, latitude and longitude) and attitude information (pitch, roll, yaw).

[0125] The system integrates a sensory semantic fault parameter setting panel and a fault parameter YAML configuration file, ensuring that one fault parameter YAML configuration file corresponds to one test scenario. Once simulation begins, the fault parameter persists throughout the entire simulation process; that is, the corresponding fault is always present in the sensor fusion output during simulation. For example, its working principle is as follows: Figure 5 As shown, the corresponding fusion perception output parameters are written into the fault parameter YAML configuration file, and a fault parameter setting panel is developed. If the fault parameter setting panel is activated, the parameter values ​​set in the panel are read first; otherwise, the default fault parameter YAML configuration file corresponding to the scenario is used. This fault parameter interface allows for both manual debugging and automatic operation. Manual debugging allows developers to modify specific parameters for single-point testing, and the fault parameter setting panel provides fast and convenient modification. Automatic operation requires the use of scripts to batch modify the fault parameter YAML configuration file for testing.

[0126] Then, automated simulation begins. Based on the perception results of semantic errors, the simulation is input to the control algorithm through a closed loop. The control commands generated by the control algorithm are executed to drive the simulated vehicle model to move, conduct simulation tests, and simultaneously collect time-series data throughout the entire simulation process.

[0127] This invention determines the number of test scenarios corresponding to each perceived fault pair based on the security risk value of the modified confusion matrix, and then constructs corresponding fault test scenarios in batches according to the corresponding number. By using the quantified comprehensive risk value as the basis for allocating the number of test scenarios, differentiated allocation of test resources is achieved. This ensures that each type of semantic fault is matched with a corresponding number of independent test scenarios, allowing for precise and targeted injection of corresponding fusion semantic errors. This guarantees that every type of perceived misjudgment fault has simulation test condition coverage, avoiding the problems of insufficient high-risk fault scenarios and redundant low-risk fault scenarios. It maximizes the coverage of high-risk semantic faults with limited simulation resources, improving the targeting of fault injection testing and the efficiency of test scenario utilization.

[0128] This embodiment provides an autonomous driving simulation testing method, which can be used in electronic devices such as computers and servers, and can also be applied to vehicle controllers such as microcontrollers and CPUs. Figure 6 This is a flowchart of an autonomous driving simulation test method according to an embodiment of the present invention, such as... Figure 6 As shown, the process includes the following steps: Step S601: Obtain the set of all output categories corresponding to at least one perception model of the vehicle. See details below. Figure 3 The relevant descriptions of step S301 shown will not be repeated here.

[0129] Step S602: For each perception model, each output category is paired with each real category to form a perception fault pair, and the security risk value of each perception fault pair is evaluated. See details below. Figure 3 The relevant description of step S302 shown will not be repeated here.

[0130] Step S603: Construct a fault confusion matrix based on each perceived fault pair, and determine the safety risk value as the matrix element of the corresponding perceived fault pair in the fault confusion matrix. See details below. Figure 3 The relevant description of step S303 shown will not be repeated here.

[0131] Step S604: Obtain the real-vehicle perception sample data corresponding to the perception model, and correct each safety risk value in the fault confusion matrix based on the real-vehicle perception sample data. See details below. Figure 3 The relevant description of step S304 shown will not be repeated here.

[0132] Step S605: Based on the corrected fault confusion matrix, a perception fault test scenario set is constructed, and the autonomous driving target algorithm is simulated and tested using the perception fault test scenario set. See details below. Figure 3 The relevant description of step S305 shown will not be repeated here.

[0133] Step S606: Based on the test results of the simulation test, verify the robustness of the autonomous driving target algorithm.

[0134] After completing the simulation test of the perception fault scenario, the embodiments of the present invention verify the robustness of the autonomous driving target algorithm based on the test results. Thus, after the fault simulation is completed, the algorithm safety quantitative evaluation is directly completed based on the measured data. The entire process of fault injection, simulation operation and robustness judgment is automated. The safety performance conclusions of the autonomous driving target algorithm under various levels of semantic faults can be output in batches, shortening the complete verification cycle of autonomous driving perception fusion semantic faults.

[0135] Specifically, step S606 includes: Step e1: Count the number of fault test scenarios that cause the simulation test results to fail for each perceived fault pair in the test results.

[0136] Specifically, through statistical analysis of each perceived fault pair The number of failure test scenarios that occurred during simulation testing The algorithm iterates through all simulation scenarios corresponding to the same fault pair and records the number of scenarios in which the control algorithm fails to perform a safety operation. For example, assuming that the perceived fault pair (pedestrian → car) is assigned 22 simulation test scenarios, and in 6 of these scenarios the vehicle fails to slow down in time and nearly collides with the pedestrian, then this perceived fault pair... Number of fault test scenarios .

[0137] Step e2: Based on the actual failure rate corresponding to each perceived fault pair in the test results, establish a Bayesian probability model. The prior parameters of the Bayesian probability model are determined by the security risk value corresponding to each perceived fault pair.

[0138] Specifically, the Beta conjugate prior distribution is chosen to describe the fault-to-true fault rate. For each perceived fault pair Conjugate prior distribution, Beta distribution:

[0139] The prior mean satisfies the following formula: Specifically, prior distribution parameters Corrected safety risk value based on perceived fault Sure, Indicates the perceived fault pair Number of test scenarios .

[0140] Step e3: Based on the number of test scenarios and the number of fault test scenarios corresponding to each perceived fault pair, solve the Bayesian probability model to determine the probability value of the pre-set upper bound of confidence corresponding to different driving safety risk levels.

[0141] Specifically, after simulation testing, based on the number of fault test scenarios mentioned above... and number of test scenarios According to Bayes' theorem, the posterior distribution is still a Beta distribution: Furthermore, based on the simulation test results of the perception model, according to the aforementioned preset driving safety risk level classification conditions, the test results are divided into driving safety risk levels such as catastrophic, high risk, medium risk, and low risk. Then, for each driving safety risk level, the failure probability of 95% is used as the preset upper bound of the confidence level for solution. The specific solution process is existing technology and will not be elaborated here.

[0142] It should be noted that the above-mentioned preset driving safety risk level classification conditions can be flexibly set according to actual needs. Similarly, the preset confidence upper limit can also be flexibly set according to actual needs, such as using 98% or 90% of the failure probability as the preset confidence upper limit, etc. This invention is not limited to this.

[0143] Step e4: Based on the relationship between the probability value of the preset upper bound of confidence corresponding to each driving safety risk level and the probability threshold corresponding to that driving safety risk level, determine the robustness verification result of the autonomous driving target algorithm at that safety risk level.

[0144] Specifically, by pre-setting independent safety probability thresholds for different driving safety risk levels, the higher the risk represented by the driving safety risk level, the smaller the corresponding safety probability threshold should be. This ensures that the robustness verification for higher-risk test scenarios is more rigorous, thereby guaranteeing the safety of autonomous driving. For example, the threshold for catastrophic failure is set to 0.01, the threshold for high-risk failure is set to 0.05, the threshold for medium-risk failure is set to 0.2, and the threshold for low-risk failure is set to 0.5. The upper confidence probability calculated for the current driving safety risk level is compared with the corresponding threshold to determine the robustness verification result for the current driving safety risk level.

[0145] Catastrophic failure set For example, define the overall catastrophic failure rate. for: because It is extremely small and can be approximated as: Although it is impossible to know The truth value can be determined, but its Bayesian posterior confidence upper bound can be calculated. For example, its 95th quantile can be calculated. , making .

[0146] Final security statement: If ,in For a predefined acceptable safety target, namely the aforementioned probability threshold, the regulatory control algorithm passes robustness verification if the risk of sensor fusion semantic faults is lower than the safety target in terms of statistical confidence; otherwise, it is deemed unsuccessful. For example, the sensor fusion semantic fault injection test process for the entire autonomous driving system is as follows: Figure 6 As shown.

[0147] This invention constructs a Bayesian model based on safety risk values ​​as prior parameters by statistically analyzing the number of failure scenarios for each perceived fault. It then calculates the probability of a pre-set upper bound on the confidence level for each risk level and determines robustness by comparing it to a threshold. This process refines the prior parameters for risk values ​​in the early stages, integrating real-vehicle fault patterns with actual simulation data. By quantifying the statistical uncertainty of risk probability through a pre-set upper bound on the confidence level, rather than relying solely on the failure rate of a single test, the evaluation results possess statistical confidence. This allows for the quantification of the upper risk probability of the autonomous driving target algorithm under semantic faults of different danger levels, forming a standardized safety judgment conclusion with reliable statistical basis.

[0148] Furthermore, step e4 above includes: Step e41: When the probability value of the preset upper bound of the confidence level corresponding to the current driving safety risk level is less than the probability threshold corresponding to the current driving safety risk level, the robustness verification result of the autonomous driving target algorithm at the current safety risk level is determined to be passed.

[0149] Specifically, if the upper confidence bound probability of the current driving safety risk level is less than the corresponding threshold, the robustness verification of the autonomous driving target algorithm under this risk level is deemed successful.

[0150] For example, the 95% confidence upper bound for catastrophic failure is 0.006, which is less than the preset threshold of 0.01. This means that, based on a statistical 95% confidence level, the probability of failure of the control algorithm when faced with catastrophic semantic misjudgment is lower than the safety red line, and the robustness verification of this level is passed.

[0151] Step e42: When the probability value of the preset upper bound of the confidence level corresponding to the current driving safety risk level is not less than the probability threshold corresponding to the current driving safety risk level, the robustness verification result of the autonomous driving target algorithm at the current safety risk level is determined to be unsuccessful.

[0152] Specifically, if the upper confidence bound probability of the current driving safety risk level is not less than the corresponding threshold, the robustness verification of the autonomous driving target algorithm under this risk level is deemed to have failed.

[0153] For example, if the upper bound of the confidence for high-risk faults is 0.07, which is higher than the preset threshold of 0.05, it means that at a 95% confidence level, the probability of the algorithm failing when encountering high-level semantic faults exceeds the acceptable safety range, and the robustness is deemed unsatisfactory.

[0154] This invention assesses robustness by comparing the probability of a pre-set upper bound of confidence level corresponding to a risk level with a preset safety threshold. If the probability of the pre-set upper bound of confidence level is less than the threshold, the verification passes; otherwise, it fails. Using a probability upper bound with confidence as the judgment criterion, compared to relying solely on the number of measured failures, fully considers the statistical errors caused by insufficient simulation sample size, avoiding misjudgments of algorithm robustness under small sample testing. A unified threshold judgment standard enables automated, uninterrupted robustness output, allowing different test batches and different vehicle models of autonomous driving systems to use the same set of judgment rules to compare safety performance, facilitating the establishment of standardized autonomous driving perception semantic fault access verification specifications.

[0155] Step S607: When it is determined that the robustness verification result of the autonomous driving target algorithm at the current safety risk level is not passed, target perception fault pairs belonging to the current safety risk level are selected.

[0156] Specifically, when the robustness verification of a certain driving safety risk level fails, the corrected fault confusion matrix is ​​automatically traversed to select all perceived fault pairs belonging to that risk level as the target perceived fault pair set that causes the algorithm to fail.

[0157] For example, assuming the robustness of the catastrophic failure level fails, all perceived fault pairs marked as catastrophic risks in the confusion matrix are automatically extracted, including GT=pedestrian → FO=car, etc., to form a target fault list, accurately locate the semantic misjudgment type that causes the safety risk to exceed the standard, and avoid the waste of manpower caused by indiscriminately checking all fault pairs.

[0158] Step S608: Based on the target perception fault pair, optimize and update the autonomous driving target algorithm.

[0159] Specifically, the R&D personnel used the target perception fault pairs obtained in the previous step as the optimization guide, and made targeted adjustments to the obstacle avoidance logic, path planning constraints, and lane line constraint strategies of the downstream control algorithm. They also added a fault tolerance mechanism specifically for this type of semantic misjudgment scenario. The specific optimization and update methods can be implemented with reference to existing technologies, and will not be elaborated here.

[0160] For example, for the core high-risk perception fault pair "pedestrian misjudged as a car", the original parking control algorithm has a short braking distance for vehicle-type obstacles and does not meet the needs of pedestrian protection. Based on the optimization algorithm for this perception fault pair, a fault-tolerant branch for obstacle type credibility is added. When the obstacle size and motion characteristics match the pedestrian but the fused output is a vehicle, the safe braking distance is forcibly lengthened and the parking speed is reduced, which specifically repairs the decision-making defects under this catastrophic semantic fault.

[0161] Step S609: Construct a set of secondary perception fault test scenarios based on the target perception fault pair.

[0162] Specifically, referring to the fault test scenario generation method in the above steps, test resources are reallocated only for the selected target perceived fault pairs, and secondary simulation scenarios are generated in batches with corresponding semantic fault YAML configuration files to focus on the problem faults and carry out centralized retesting without repeating the full set of fault scenarios.

[0163] For example, 30 sets of secondary parking scenarios are generated for catastrophic perception fault pairs (pedestrian → car). The pedestrians are set to appear in different positions such as the left front, right rear, and side of the parking space, and different parking speeds are used. The YAML file uniformly forces the perception module to output the target object type as a car, forming a lightweight set of secondary scenarios that only covers high-risk failure faults, reducing the computational power consumption of retest simulation.

[0164] Step S610: Use the secondary perception fault test scenario set to perform simulation test on the optimized and updated autonomous driving target algorithm, and then re-execute step S606.

[0165] Specifically, the optimized control algorithm is connected to the simulation platform, all secondary test scenarios are loaded to complete the automated closed-loop simulation, and the vehicle time-series operation data and failure records in the scenario are collected. After the simulation is completed, the robustness evaluation process is carried out again, the confidence upper bound probability of the corresponding risk level is recalculated, and the verification results are re-determined by comparing with the safety threshold to achieve iterative closed loop.

[0166] For example, the optimized parking algorithm is simulated in a secondary scenario with 30 sets of pedestrians misjudging cars, and the number of vehicle collisions, sudden braking, and other failures in the scenario is counted. Then, the Bayesian model is rebuilt to update the posterior distribution, and the 95% confidence upper bound of catastrophic failure is recalculated. If the updated confidence upper bound is lower than the preset threshold of 0.01, the robustness verification is considered to have passed. If it is still higher than the threshold, the iterative process of filtering failures, optimizing the algorithm, and building a retest scenario is repeated to continuously reduce the risk of semantic failure of the system.

[0167] This invention employs a screening and verification process to select perception fault pairs of the same risk level when verification fails. Based on these fault pairs, the autonomous driving target algorithm is optimized, and a secondary perception fault test scenario set is built for retesting. This avoids the waste of R&D resources caused by indiscriminate full optimization of the autonomous driving target algorithm. Furthermore, separate retest scenarios are constructed for problematic perception fault pairs to accurately verify whether the algorithm optimization measures effectively correct decision-making defects under semantic faults. The iterative testing mechanism continuously reduces the probability of vehicle failure caused by high-risk semantic faults, continuously improving the fault tolerance capability of the autonomous driving target algorithm for perception semantic errors.

[0168] This embodiment also provides an autonomous driving simulation testing device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0169] This embodiment provides an autonomous driving simulation testing device, such as... Figure 7 As shown, the autonomous driving simulation test device includes: The acquisition module 701 is used to acquire the set of all output categories corresponding to at least one perception model of the vehicle. The first processing module 702 is used to form a perception fault pair with each output category and each real category for each perception model, and to evaluate the safety risk value of each perception fault pair. The second processing module 703 is used to construct a fault confusion matrix based on each perceived fault pair, and to determine the safety risk value as the matrix element of the corresponding perceived fault pair in the fault confusion matrix; The third processing module 704 is used to acquire the real vehicle perception sample data corresponding to the perception model, and to correct each safety risk value in the fault confusion matrix based on the real vehicle perception sample data. The fourth processing module 705 is used to construct a set of perception fault test scenarios based on the corrected fault confusion matrix, and to test the autonomous driving target algorithm using each perception fault test scenario.

[0170] In some optional implementations, the first processing module 702 includes: The first processing unit is used to assess the driving safety risk level of each perceived fault pair based on preset driving safety risk level classification conditions, and determine the driving safety risk level of each perceived fault pair. The second processing unit is used to determine the score value corresponding to the driving safety risk level of each perceived fault pair based on the preset correspondence between different driving safety risk levels and score values. The third processing unit is used to determine the score value corresponding to the driving safety risk level of each perceived fault pair as the safety risk value of that perceived fault pair.

[0171] In some optional implementations, the real-vehicle perception sample data includes: the real-vehicle output category and the corresponding real-vehicle actual category. The third processing module 704 mentioned above includes: The fourth processing unit is used to count, for each real vehicle output category, the number of first samples in the real vehicle perception sample data for the current real vehicle output category and the number of second samples in the first sample data that are the same as the current real vehicle output category; The fifth processing unit is used to calculate the conditional probability of the current real vehicle output category based on the ratio of the second sample size to the first sample size. The sixth processing unit is used to calculate the error conditional entropy and the correct conditional entropy for each perceived fault pair based on the conditional probabilities corresponding to different real vehicle output categories. The seventh processing unit is used to correct the security risk value corresponding to the perceived fault pair in the fault confusion matrix based on the difference between the error condition entropy and the correct condition entropy of each perceived fault pair.

[0172] In some optional embodiments, the seventh processing unit described above includes: The first processing subunit is used to determine the difference between the error condition entropy and the correct condition entropy of each fault-sensing pair as the priority weight of that fault-sensing pair. The second processing subunit is used to multiply the priority weight of the perceived fault pair with the corresponding security risk value of the perceived fault pair in the fault confusion matrix to obtain the corrected security risk value.

[0173] In some optional implementations, the fourth processing module 705 described above includes: The eighth processing unit is used to determine the number of test scenarios corresponding to each security risk value for each perceived fault pair based on the security risk value in the corrected fault confusion matrix. The ninth processing unit is used to construct the perception fault test scenario for each perception fault pair according to the number of test scenarios for different perception fault pairs.

[0174] In some optional embodiments, the eighth processing unit described above includes: The third processing subunit is used to determine the total number of test scenarios for simulation testing; The fourth processing subunit is used to construct an objective function with the optimization objective of minimizing the overall fluctuation of the estimated failure rate of all perceived fault pairs and the constraint that the sum of the number of test scenarios for all perceived fault pairs equals the total number of test scenarios. The estimated failure rate of each perceived fault pair is determined by the safety risk value of that perceived fault pair. The estimated failure rate is used to characterize the estimated probability that the corresponding perceived fault pair will cause a failure in the simulation test of the autonomous driving target algorithm. The fifth processing subunit is used to solve the objective function and obtain the number of test scenarios for each perceived fault pair corresponding to each security risk value.

[0175] In some optional implementations, the fourth processing subunit is further configured to convert the security risk value of each perceived fault pair using a Sigmoid function to obtain the estimated failure rate of each perceived fault pair.

[0176] In some optional embodiments, the eighth processing unit described above includes: The sixth processing subunit is used to determine the total number of test scenarios for simulation testing; The seventh processing subunit is used to sort the security risk values ​​in the corrected fault confusion matrix; The eighth processing subunit is used to determine the number of test scenarios for each security risk value corresponding to the perceived fault pair based on the sorting results and the total number of test scenarios. The larger the security risk value of the perceived fault pair, the larger the number of test scenarios it corresponds to.

[0177] In some optional implementations, the autonomous driving simulation test apparatus further includes: The fifth processing module is used to verify the robustness of the autonomous driving target algorithm based on the test results of simulation tests.

[0178] In some optional implementations, the fifth processing module described above includes: The tenth processing unit is used to count the number of fault test scenarios that cause the simulation test results to fail for each perceived fault pair in the test results; The eleventh processing unit is used to establish a Bayesian probability model based on the actual failure rate corresponding to each perceived fault pair in the test results. The prior parameters of the Bayesian probability model are determined by the security risk value corresponding to each perceived fault pair. The twelfth processing unit is used to solve the Bayesian probability model based on the number of test scenarios and the number of fault test scenarios corresponding to each perceived fault pair, and to determine the probability value of the pre-set upper bound of the confidence level corresponding to different driving safety risk levels. The thirteenth processing unit is used to determine the robustness verification result of the autonomous driving target algorithm at the safety risk level based on the relationship between the probability value of the preset upper bound of the confidence level corresponding to each driving safety risk level and the probability threshold corresponding to that driving safety risk level.

[0179] In some optional embodiments, the thirteenth processing unit described above includes: The ninth processing subunit is used to determine that the robustness verification result of the autonomous driving target algorithm at the current safety risk level is passed when the probability value of the preset upper bound of the confidence level corresponding to the current driving safety risk level is less than the probability threshold corresponding to the current driving safety risk level. The tenth processing subunit is used to determine that the robustness verification result of the autonomous driving target algorithm at the current safety risk level is not passed when the probability value of the preset upper bound of the confidence level corresponding to the current driving safety risk level is not less than the probability threshold corresponding to the current driving safety risk level.

[0180] In some optional implementations, the autonomous driving simulation test apparatus further includes: The sixth processing module is used to filter target-perceived fault pairs that belong to the current security risk level; The seventh processing module is used to optimize and update the autonomous driving target algorithm based on the target perception fault pairs; The eighth processing module is used to construct a set of secondary perception fault test scenarios based on the target perception fault pair; The ninth processing module is used to perform simulation tests on the optimized and updated autonomous driving target algorithm using a set of secondary perception fault test scenarios, and to re-execute the test results based on the simulation test to verify the robustness of the autonomous driving target algorithm.

[0181] The autonomous driving simulation testing device provided in this embodiment of the invention can execute the autonomous driving simulation testing method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0182] Figure 8 This is a schematic diagram of the structure of a vehicle controller provided in an embodiment of the present invention.

[0183] The following is a detailed reference. Figure 8 The diagram illustrates a structural schematic suitable for implementing a controller in an embodiment of the present invention. The controller may include a processor (e.g., a central processing unit, graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 802 or a program loaded from memory 808 into random access memory (RAM) 803. RAM 803 also stores various programs and data required for the operation of the electronic device. The processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0184] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; memory 808 including, for example, magnetic tape, hard disk, etc.; and communication devices 809. Communication device 809 allows the controller to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 A controller with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and may alternatively implement or have more or fewer devices.

[0185] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a memory 808, or installed from a ROM 802. When the computer program is executed by the processor 801, it performs the functions defined in the autonomous driving simulation test method of the embodiments of the present invention.

[0186] Figure 8The controller shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0187] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the autonomous driving simulation testing method shown in the above embodiments is implemented.

[0188] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0189] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. An autonomous driving simulation testing method, characterized in that, The method includes: Obtain the set of all output categories corresponding to at least one perception model of the vehicle; For each perception model, each output category is paired with each real category to form a perception fault pair, and the security risk value of each perception fault pair is evaluated. A fault confusion matrix is ​​constructed based on each of the perceived fault pairs, and the security risk value is determined as the matrix element of the corresponding perceived fault pair in the fault confusion matrix; Obtain the real vehicle perception sample data corresponding to the perception model, and correct each of the safety risk values ​​in the fault confusion matrix based on the real vehicle perception sample data; Based on the corrected fault confusion matrix, a set of perception fault test scenarios is constructed, and the target algorithm for autonomous driving is simulated and tested using the set of perception fault test scenarios.

2. The method according to claim 1, characterized in that, The assessment of the security risk value for each perceived fault pair includes: Based on the preset driving safety risk level classification conditions, the driving safety risk level of each perceived fault pair is assessed to determine the driving safety risk level of each perceived fault pair. Based on the pre-defined correspondence between different driving safety risk levels and score values, the score value corresponding to the driving safety risk level of each perceived fault pair is determined. The score corresponding to the driving safety risk level of each perceived fault pair is determined as the safety risk value of that perceived fault pair.

3. The method according to claim 1, characterized in that, The real-vehicle perception sample data includes: the real-vehicle output category and the corresponding real-vehicle true category. The correction of each safety risk value in the fault confusion matrix based on the real-vehicle perception sample data includes: For each real vehicle output category, count the number of first samples in the real vehicle perception sample data for the current real vehicle output category and the number of second samples in the first sample data that are the same as the current real vehicle output category; Based on the ratio of the second sample size to the first sample size, the conditional probability of the current real vehicle output category is calculated; Based on the conditional probabilities corresponding to different real vehicle output categories, calculate the erroneous conditional entropy and correct conditional entropy for each perceived fault pair. Based on the difference between the erroneous conditional entropy and the correct conditional entropy of each perceived fault pair, the security risk value corresponding to that perceived fault pair in the fault confusion matrix is ​​corrected.

4. The method according to claim 3, characterized in that, The step of correcting the security risk value corresponding to the perceived fault pair in the fault confusion matrix based on the difference between the error conditional entropy and the correct conditional entropy of each perceived fault pair includes: The difference between the error condition entropy and the correct condition entropy of each fault-sensing pair is determined as the priority weight of that fault-sensing pair. The priority weight of the perceived fault pair is multiplied by the security risk value corresponding to the perceived fault pair in the fault confusion matrix to obtain the corrected security risk value.

5. The method according to claim 1, characterized in that, The set of fault-aware test scenarios is constructed based on the corrected fault confusion matrix, including: Based on the security risk values ​​in the corrected fault confusion matrix, determine the number of test scenarios corresponding to each security risk value and the perceived fault pair. Based on the number of test scenarios for different sensing fault pairs, construct sensing fault test scenarios for each sensing fault pair.

6. The method according to claim 5, characterized in that, The determination of the number of test scenarios corresponding to each perceived fault pair based on the security risk value in the corrected fault confusion matrix includes: Determine the total number of test scenarios for the simulation test; The objective is to minimize the overall fluctuation of the estimated failure rate of all perceived fault pairs, and the objective function is constructed with the constraint that the sum of the number of test scenarios for all perceived fault pairs equals the total number of test scenarios. The estimated failure rate of each perceived fault pair is determined by the safety risk value of that perceived fault pair. The estimated failure rate is used to characterize the estimated probability that the corresponding perceived fault pair will cause the simulation test of the autonomous driving target algorithm to fail. The objective function is solved to obtain the number of test scenarios for each perceived fault pair corresponding to each security risk value.

7. The method according to claim 6, characterized in that, The method further includes: The safety risk value of each perceived fault pair is transformed using the Sigmoid function to obtain the estimated failure rate of each perceived fault pair.

8. The method according to claim 5, characterized in that, The determination of the number of test scenarios corresponding to each perceived fault pair based on the security risk value in the corrected fault confusion matrix includes: Determine the total number of test scenarios for the simulation test; Sort the security risk values ​​in the corrected fault confusion matrix; Based on the ranking results and the total number of test scenarios, the number of test scenarios corresponding to each security risk value for the perceived fault pair is determined. The higher the security risk value of the perceived fault pair, the greater the number of test scenarios it corresponds to.

9. The method according to any one of claims 1-8, characterized in that, After simulating and testing the autonomous driving target algorithm using the aforementioned perception fault test scenario set, the method further includes: The robustness of the proposed autonomous driving target algorithm is verified based on the simulation test results.

10. The method according to claim 9, characterized in that, The test results based on simulation testing are used to verify the robustness of the autonomous driving target algorithm, including: Count the number of fault test scenarios that cause the simulation test results to fail for each perceived fault pair in the test results; Based on the actual failure rate corresponding to each perceived fault pair in the test results, a Bayesian probability model is established. The prior parameters of the Bayesian probability model are determined by the security risk value corresponding to each perceived fault pair. Based on the number of test scenarios and the number of fault test scenarios corresponding to each perceived fault pair, the Bayesian probability model is solved to determine the probability value of the pre-set upper bound of confidence corresponding to different driving safety risk levels. Based on the relationship between the probability value of the preset upper bound of confidence corresponding to each driving safety risk level and the probability threshold corresponding to that driving safety risk level, the robustness verification result of the autonomous driving target algorithm at that safety risk level is determined.

11. The method according to claim 10, characterized in that, The determination of the robustness verification result of the autonomous driving target algorithm at a given safety risk level based on the relationship between the probability value of the preset upper bound of confidence corresponding to each driving safety risk level and the probability threshold corresponding to that driving safety risk level includes: When the probability value of the preset upper bound of the confidence level corresponding to the current driving safety risk level is less than the probability threshold corresponding to the current driving safety risk level, the robustness verification result of the autonomous driving target algorithm at the current safety risk level is determined to be passed. If the probability value of the preset upper bound of the confidence level corresponding to the current driving safety risk level is not less than the probability threshold corresponding to the current driving safety risk level, the robustness verification result of the autonomous driving target algorithm at the current safety risk level is determined to be unsuccessful.

12. The method according to claim 11, characterized in that, When the robustness verification result of the autonomous driving target algorithm at the current safety risk level is determined to be unsuccessful, the method further includes: Filter target perceived fault pairs that belong to the current security risk level; Based on the target perception fault pairs, the autonomous driving target algorithm is optimized and updated. Based on the target perceived fault pair, construct a set of secondary perceived fault test scenarios; The optimized and updated autonomous driving target algorithm is simulated and tested using the secondary perception fault test scenario set, and the test results based on the simulation test are re-executed to verify the robustness of the autonomous driving target algorithm.

13. An autonomous driving simulation testing device, characterized in that, The device includes: The acquisition module is used to acquire the set of all output categories corresponding to at least one perception model of the vehicle. The first processing module is used to form a perception fault pair by pairing each output category with each real category for each perception model, and to evaluate the security risk value of each perception fault pair. The second processing module is used to construct a fault confusion matrix based on each of the perceived fault pairs, and to determine the security risk value as the matrix element of the corresponding perceived fault pair in the fault confusion matrix; The third processing module is used to acquire the real vehicle perception sample data corresponding to the perception model, and to correct each of the safety risk values ​​in the fault confusion matrix based on the real vehicle perception sample data. The fourth processing module is used to construct a set of perception fault test scenarios based on the corrected fault confusion matrix, and to test the autonomous driving target algorithm using each of the perception fault test scenarios.

14. A vehicle, characterized in that, The vehicle includes: a controller, the controller comprising: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 12.

16. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the method of any one of claims 1 to 12.

Images