A method and related equipment for determining the equivalence of autonomous driving simulation tests

By constructing a confidence transfer chain and a nonlinear coupling transfer equation, the problem of difficulty in quantifying the error transmission and coupling effects in the simulation process of autonomous driving simulation testing is solved, and the equivalence determination and substitutability assessment of simulation testing and real testing are realized.

CN123116587APending Publication Date: 2026-11-17CHANGAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611295109.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-25
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing autonomous driving simulation tests, there is a lack of quantitative reflection of the closed-loop transmission and coupling effects between the confidence level of each simulation stage and the final simulation test results and the real test results, making it difficult to determine the degree of equivalence between simulation tests and real tests.

Method used

A confidence propagation chain is constructed, the propagation coefficients between adjacent nodes are calibrated, and the convergence of confidence propagation in the feedback loop is determined through perturbation experiments. The overall system-level confidence is calculated by combining the nonlinear coupling transfer equation. The result-level equivalence is determined based on simulation test data and real test data, and the substitutable ratio of simulation test is determined.

Benefits of technology

It achieves a quantitative reflection of the equivalence between simulation tests and real tests, provides a credibility assessment of autonomous driving simulation tests, and improves the accuracy of substitutability determination for simulation tests.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116587A_ABST
    Figure CN123116587A_ABST
Patent Text Reader

Abstract

This application belongs to the field of autonomous driving simulation testing technology, and discloses a method and related equipment for determining the equivalence of autonomous driving simulation testing. The method obtains the confidence values ​​of a scene reconstruction model, a traffic flow simulation model, a sensor simulation model, and a vehicle dynamics model; constructs a confidence transfer chain containing feedback loops; calibrates the transfer coefficients between adjacent nodes through perturbation experiments and determines whether the confidence transfer in the feedback loop converges; when the confidence transfer converges, it calculates the overall system-level confidence based on the nonlinear coupling transfer equation; obtains result-level equivalence based on simulation test data and real test data; and determines the substitutable ratio of the simulation test based on the overall system-level confidence and result-level equivalence, generating an equivalence determination result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of autonomous driving simulation testing technology, specifically relating to an autonomous driving simulation testing equivalence determination method and related equipment. Background Technology

[0002] Autonomous driving simulation testing constructs road scenarios, traffic participants, sensors, and vehicle motion states in a virtual environment to test the perception, decision-making, and control processes of autonomous driving systems. It can be used to assist in verifying the performance of autonomous driving systems in different scenarios. An autonomous driving simulation testing system typically includes multiple simulation stages such as scene reconstruction, traffic flow simulation, sensor simulation, and vehicle dynamics. The vehicle motion states generated by vehicle dynamics simulation also feed back into and affect traffic flow states, creating a closed-loop operational relationship among the various simulation stages.

[0003] Current simulation reliability assessments typically evaluate the fidelity or confidence of different simulation stages separately, or assess their consistency by comparing simulation test results with actual test results. However, errors generated in each simulation stage propagate step-by-step along the simulation test chain and may repeatedly propagate in feedback loops; when errors exist simultaneously in different simulation stages, coupling effects may also occur. Evaluating each simulation stage independently or simply aggregating them is insufficient to reflect the impact of errors from each stage, after propagation and coupling, on the overall system-level reliability.

[0004] Therefore, in existing autonomous driving simulation tests, there is a lack of quantitative correlation between the confidence level of each simulation stage and the final simulation test results and real test results, which can reflect the closed-loop transmission and coupling effects. It is difficult to further quantify and determine the equivalence and substitutability of simulation tests and real tests based on the overall credibility of the autonomous driving simulation test system. Summary of the Invention

[0005] The purpose of this application is to provide a method and related equipment for determining the equivalence of autonomous driving simulation testing, so as to overcome the technical problem that existing autonomous driving simulation testing methods do not quantify the transmission and coupling relationship of the confidence of each simulation link in the closed-loop test chain, resulting in the overall confidence of the system level being unable to reflect the cumulative impact of errors in each link, and thus making it difficult to quantify the degree of equivalence between simulation testing and real testing.

[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, this application provides a method for determining the equivalence of autonomous driving simulation tests, including: Obtain the confidence values ​​of the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model in the autonomous driving simulation test system; A confidence transfer chain is constructed, which sequentially includes a scene reconstruction node, a traffic flow simulation node, a sensor simulation node, a test autonomous driving system response node, a vehicle dynamics simulation node, and a test result node. The vehicle dynamics simulation node feeds back to the traffic flow simulation node, forming a feedback loop. The transmission coefficients between adjacent nodes in the confidence transmission chain are calibrated through perturbation experiments. The loop gain is determined based on the transmission coefficients between adjacent nodes in the feedback loop. The convergence of the confidence transmission in the feedback loop is then determined based on the loop gain. The coefficients of the nonlinear coupling transfer equation are calibrated, and when the confidence transfer in the feedback loop converges, the overall system-level confidence is calculated based on the confidence values ​​of the scene reconstruction model, the traffic flow simulation model, the sensor simulation model, and the vehicle dynamics model, as well as the nonlinear coupling transfer equation. The result-level equivalence is determined based on simulation test data and real test data to obtain the result-level equivalence. The substitutable ratio of the simulation test is determined according to the overall system-level confidence and the result-level equivalence, and the equivalence determination result is generated.

[0007] Furthermore, the transfer coefficients include confidence transfer coefficients and equivalent signal transfer coefficients; For adjacent nodes that are not involved in the response node of the tested autonomous driving system, while keeping the input of other nodes unchanged, a perturbation of known magnitude is applied to the output of the upstream node, and the corresponding confidence transfer coefficient is calibrated according to the confidence change of the adjacent downstream nodes. The transmission coefficients between the sensor simulation node and the response node of the autonomous driving system under test, and between the response node of the autonomous driving system under test and the vehicle dynamics simulation node, are equivalent signal transmission coefficients, which are calibrated based on the changes in the response of the autonomous driving system under test caused by sensor output signal disturbances and the changes in the vehicle dynamics simulation output caused by control command disturbances.

[0008] Furthermore, when the third-order and higher-order cross terms are truncated, the nonlinear coupling transfer equation includes first-order linear terms and second-order cross terms, and the overall system-level confidence is calculated according to the following formula:

[0009] in, The system-level overall confidence level is... and Number the model. and The first The model and the first The confidence score of each model. For the first The coefficients of the first-order linear terms corresponding to each model For the first The model and the first The second-order cross term coefficients corresponding to each model; The coefficients of the first-order linear term were calibrated through a single-factor perturbation experiment, and the coefficients of the second-order cross term were calibrated through a two-factor perturbation experiment.

[0010] Furthermore, the step of determining the loop gain based on the transmission coefficients between adjacent nodes in the feedback loop, and determining whether the confidence propagation in the feedback loop converges based on the loop gain, includes: The absolute value of the product of the transmission coefficients between adjacent nodes in the feedback loop is determined as the loop gain. When the loop gain is less than 1, the confidence propagation in the feedback loop is determined to be converged; when the loop gain is greater than or equal to 1, the confidence propagation in the feedback loop is determined to be divergent.

[0011] Furthermore, before performing the result-level equivalence determination, scenario-level equivalence determination and behavior-level equivalence determination are also included; The road geometry parameters, traffic participant types and initial states, and environmental conditions in the simulation scenario and the real test scenario are matched, and the scenario-level equivalence is determined based on the average matching rate of scenario elements. The test scenarios that pass the scenario-level equivalence determination are used as the objects of behavior-level equivalence determination. The trajectory consistency score is determined based on the dynamic time warping distance between the simulation trajectory and the real test trajectory, and the decision consistency score is determined based on the edit distance of the key decision event sequences in the simulation test and the real test. The trajectory consistency score and the decision consistency score are weighted to obtain the behavior-level equivalence.

[0012] Furthermore, the result-level equivalence determination includes: A two-sided equivalence test is performed on the simulation test indicators and the real test indicators for each test scenario. The result-level equivalence is determined based on the proportion of the number of test scenarios that are determined to be equivalent in the two-sided equivalence test to the total number of test scenarios.

[0013] Furthermore, the substitutable ratio is calculated according to the following formula:

[0014] in, The aforementioned alternative ratio, For the aforementioned result-level equivalence, The system-level overall confidence level is given.

[0015] Secondly, this application provides an equivalence determination system for autonomous driving simulation testing, comprising: The confidence level acquisition module is used to acquire the confidence level values ​​of the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model in the autonomous driving simulation test system. The confidence transfer chain construction module is used to construct the confidence transfer chain, which sequentially includes a scene reconstruction node, a traffic flow simulation node, a sensor simulation node, a test autonomous driving system response node, a vehicle dynamics simulation node, and a test result node. The vehicle dynamics simulation node feeds back to the traffic flow simulation node to form a feedback loop. The closed-loop transfer analysis module is used to calibrate the transfer coefficient between adjacent nodes in the confidence transfer chain through perturbation experiments, determine the loop gain based on the transfer coefficient between each adjacent node in the feedback loop, and determine whether the confidence transfer in the feedback loop has converged based on the loop gain. The system-level confidence determination module is used to calibrate the coefficients of the nonlinear coupling transfer equation, and calculate the overall system-level confidence based on the confidence values ​​of the scene reconstruction model, the traffic flow simulation model, the sensor simulation model, and the vehicle dynamics model, as well as the nonlinear coupling transfer equation, when the confidence transfer in the feedback loop converges. The equivalence determination module is used to perform result-level equivalence determination based on simulation test data and real test data, obtain result-level equivalence, determine the substitutable ratio of simulation test according to the overall system-level confidence and the result-level equivalence, and generate equivalence determination results.

[0016] Thirdly, this application provides a computer device, including a processor and a computer-readable storage medium; The processor is adapted to execute computer programs; The computer-readable storage medium stores a computer program, which, when executed by the processor, implements the autonomous driving simulation test equivalence determination method as described above.

[0017] Fourthly, this application provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed as described above for determining the equivalence of autonomous driving simulation tests.

[0018] Compared with the prior art, this application has the following beneficial effects: Firstly, this application provides a method for determining the equivalence of autonomous driving simulation tests. This application constructs a confidence transfer chain containing feedback loops, calibrates the transfer coefficients between adjacent nodes based on perturbation experiments, and calculates the overall system-level confidence by combining the confidence transfer convergence of the feedback loops and the nonlinear coupling transfer equation. This ensures that the influence of confidence in different simulation stages during the transfer and coupling process is reflected in the overall system-level confidence. Furthermore, it combines simulation test data and real test data to obtain result-level equivalence, and determines the substitutable proportion of simulation tests based on the overall system-level confidence and result-level equivalence, thereby providing a quantitative basis for determining the equivalence between autonomous driving simulation tests and real tests.

[0019] Secondly, this application provides an equivalence determination system for autonomous driving simulation testing. This system, through a confidence level acquisition module, a confidence level transmission chain construction module, a closed-loop transmission analysis module, a system-level confidence level determination module, and an equivalence determination module, respectively achieves confidence level acquisition, confidence level transmission chain construction, confidence level transmission analysis in the feedback loop, system-level overall confidence level calculation, and result-level equivalence determination and substitutable ratio determination, enabling the data processing chain of the autonomous driving simulation testing equivalence determination method to be implemented through the corresponding functional modules.

[0020] Thirdly, this application provides a computer device. This computer device executes a computer program via a processor to implement the aforementioned method for determining the equivalence of autonomous driving simulation tests, enabling confidence transfer analysis, system-level overall confidence calculation, result-level equivalence determination, and determination of the substitutable ratio to be completed within the computer device.

[0021] Fourthly, this application provides a computer-readable storage medium. This computer-readable storage medium stores a computer program for implementing the above-described method for determining the equivalence of autonomous driving simulation tests, enabling the method to be implemented through the loading and execution of the computer program. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the method for determining the equivalence of autonomous driving simulation testing in this application. Figure 2 This is a schematic diagram of the confidence transmission chain and feedback loop in the embodiments of this application; Figure 3This is a schematic diagram of the structure of the autonomous driving simulation test equivalence determination system in the embodiments of this application; Figure 4 This is a diagram showing the internal structure of a computer device in an embodiment of this application. Detailed Implementation

[0024] Autonomous driving simulation testing systems typically consist of multiple cascaded simulation models. Simulation errors generated by each model not only propagate along the test chain, but vehicle dynamics simulation results also feed back into traffic flow simulations, causing errors to continue propagating in the feedback loop. Evaluating the confidence level of each model individually is insufficient to reflect the impact of the propagation and coupling of errors from each model on the overall reliability of the autonomous driving simulation test results.

[0025] Based on the above background, this application constructs a confidence transmission chain containing feedback loops, calibrates the transmission coefficients between adjacent nodes, and determines whether the confidence transmission in the feedback loops converges. When the confidence transmission converges, it calculates the overall system-level confidence by combining the confidence values ​​of each model and the nonlinear coupling transmission equations. Then, it combines simulation test data and real test data to obtain result-level equivalence, thereby determining the substitutable ratio of simulation tests and generating equivalence judgment results.

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] Example 1 This embodiment provides a method for determining the equivalence of autonomous driving simulation tests, such as... Figure 1 As shown, the method includes the following steps.

[0028] The confidence scores of the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model in the autonomous driving simulation test system are obtained. The scene reconstruction model characterizes the static environment in the autonomous driving simulation test; the traffic flow simulation model characterizes the behavior and state changes of dynamic traffic participants; the sensor simulation model generates the sensor simulation signals received by the autonomous driving system under test; and the vehicle dynamics model determines the vehicle's motion state based on the control commands output by the autonomous driving system under test. The confidence scores of the four models reflect the simulation reliability of their respective models and serve as input for subsequent system-level overall confidence score calculations. During the test, the models in the autonomous driving simulation test system are interconnected according to the processes of scene generation, traffic flow evolution, sensor perception, autonomous driving system response, and changes in vehicle motion state.

[0029] Construct a confidence propagation chain. For example... Figure 2 As shown, the confidence propagation chain sequentially includes a scene reconstruction node, a traffic flow simulation node, a sensor simulation node, a tested autonomous driving system response node, a vehicle dynamics simulation node, and a test result node. The scene reconstruction node forms a forward propagation relationship with the test result node, and the vehicle dynamics simulation node also feeds back to the traffic flow simulation node, thus creating a feedback loop between the traffic flow simulation node, sensor simulation node, tested autonomous driving system response node, and vehicle dynamics simulation node. This confidence propagation chain allows us to represent the sequential influence of each simulation stage during the testing process, according to the data transmission order. Figure 2 As shown, the transmission relationship between nodes is represented by the corresponding transmission coefficients.

[0030] Perturbation experiments are conducted on adjacent nodes in the confidence propagation chain, and the propagation coefficient between adjacent nodes is calibrated based on the response changes of adjacent downstream nodes after the upstream node is perturbed. The propagation coefficient characterizes the degree to which the perturbation effect is propagated from one node to its adjacent nodes. For the feedback loop, the loop gain is determined based on the propagation coefficient between adjacent nodes in the feedback loop, and the convergence of confidence propagation in the feedback loop is judged based on the loop gain. Thus, it is possible to determine whether the confidence effect cyclically propagated through the feedback loop can form a stable state before calculating the overall system-level confidence.

[0031] The coefficients of the nonlinear coupling transfer equation are calibrated. Upon convergence of the confidence transfer in the feedback loop, the confidence values ​​of the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model are input into the nonlinear coupling transfer equation to calculate the overall system-level confidence. This nonlinear coupling transfer equation is used to uniformly reflect the confidence of each model and the coupling effects between models to the overall system-level confidence, ensuring that the resulting overall system-level confidence not only characterizes the credibility of individual models but also reflects the combined effect of multiple simulation components on the overall credibility of the autonomous driving simulation test. The specific equation form and coefficient calibration method will be further explained in subsequent embodiments.

[0032] Simulation test data and real test data corresponding to the same test object are obtained, and result-level equivalence is determined based on the simulation test data and real test data to obtain result-level equivalence. Result-level equivalence reflects the degree of equivalence between simulation test results and real test results, while system-level overall confidence reflects the overall credibility of the autonomous driving simulation test system that generated the simulation test results. Based on the system-level overall confidence and the result-level equivalence, the substitutable ratio of the simulation test is determined, and an equivalence determination result is generated, thereby combining the overall credibility of the autonomous driving simulation test system itself with the degree of equivalence between simulation test results and real test results for quantitative determination.

[0033] Example 2 Based on Example 1, this embodiment further explains the calibration of the transmission coefficients between adjacent nodes in the confidence transmission chain and the convergence determination of confidence transmission in the feedback loop.

[0034] like Figure 2 As shown, the confidence propagation chain, in the forward direction, includes a scene reconstruction node, a traffic flow simulation node, a sensor simulation node, a test autonomous driving system response node, a vehicle dynamics simulation node, and a test result node. The vehicle dynamics simulation node feeds back to the traffic flow simulation node, forming a feedback loop. For adjacent nodes that are not involved in the test autonomous driving system response node, the confidence propagation coefficient can be expressed as the local propagation relationship of the confidence change of the downstream node relative to the confidence change of the upstream node:

[0035] in, and Number the nodes. For nodes To node The confidence transfer coefficient between them and These represent the confidence levels of the corresponding upstream and downstream nodes, respectively. This represents partial derivative operations. During calibration, while keeping the inputs of other nodes constant, a perturbation of known magnitude is applied to the output of the upstream node. The change in confidence level of adjacent downstream nodes is measured, and the corresponding confidence level transitivity is fitted through multiple perturbation experiments. The fitted transitivity is normalized to obtain the corresponding confidence level transitivity coefficient, so that the transitivity coefficients between different adjacent nodes adopt a unified relative perturbation transitivity. In some embodiments, when normalizing the original linear slope obtained from the perturbation experiment, the original linear slope is divided by the product of the confidence level benchmark value of the adjacent downstream node and the perturbation ratio to obtain the corresponding confidence level transitivity coefficient.

[0036] The following uses the confidence transfer coefficient between the scene reconstruction node and the traffic flow simulation node. Taking this as an example, the calibration process of the confidence transfer coefficient will be explained.

[0037] During the perturbation experiment, other inputs to the traffic flow simulation model, sensor simulation model, the tested autonomous driving system, and the vehicle dynamics model were kept constant, and perturbations were applied only to the road geometry parameters output by the scene reconstruction model. The perturbation objects included lane width, turning radius, and slope, with perturbation levels set to ±5%, ±10%, and ±15%, respectively.

[0038] Taking the experimental data corresponding to lane width disturbances as an example, the following table shows how the confidence level of the traffic flow simulation model changes with the output disturbance of the scene reconstruction model:

[0039] in, The magnitude of the perturbation applied to the output of the scene reconstruction model. The confidence level of the traffic flow simulation model. This represents the change in confidence level of the traffic flow simulation model relative to the baseline state.

[0040] With perturbation magnitude Using the relative rate of change of the confidence level of the traffic flow simulation model as the dependent variable, a linear regression was performed, yielding an original linear slope of 0.355. This means that for every 1% disturbance in the output of the scene reconstruction model, the confidence level of the traffic flow simulation model changes by 0.355%.

[0041] Since the confidence transfer coefficient is used to represent the transfer relationship of confidence changes between adjacent nodes, and the confidence values ​​of each model are between 0 and 1, the original linear slope is normalized. The confidence baseline value of the traffic flow simulation model is 0.818. According to the conversion relationship in this embodiment, the confidence transfer coefficient between the scene reconstruction node and the traffic flow simulation node is... for:

[0042] in, Here, represents the confidence transfer coefficient between the scene reconstruction node and the traffic flow simulation node; 0.355 is the original linear slope obtained from the perturbation experiment; 0.818 is the confidence baseline value of the traffic flow simulation model; and the coefficient is... Convert percentage perturbations to a conversion factor for the absolute change in confidence level.

[0043] The same perturbation experiment was conducted on other adjacent nodes in the confidence propagation chain using the corresponding perturbation object, and the normalization process was completed to obtain the propagation coefficient between each adjacent node.

[0044] The tested autonomous driving system response node corresponds to the real autonomous driving algorithm software stack to be verified. It is itself the object under test, not a simulation model. Therefore, the confidence level of the tested autonomous driving system response node is fixed at 1, and the same confidence level quantification as the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model is not applied to this node. Fixing the confidence level of the tested autonomous driving system response node to 1 does not affect the disturbance propagation calibration before and after this node. The transfer coefficient between the sensor simulation node and the tested autonomous driving system response node... And the transfer coefficient between the response node of the tested autonomous driving system and the vehicle dynamics simulation node. It is represented by the equivalent signal transmission coefficient. Among them, Calibration is performed based on the changes in the response of the tested autonomous driving system caused by disturbances in the sensor output signal. The calibration is performed based on the changes in vehicle dynamics simulation output caused by control command disturbances. The equivalent signal transfer coefficients are normalized after calibration, wherein the confidence baseline value of the response node of the tested autonomous driving system is set to 1. The normalized equivalent signal transfer coefficients are used together with other confidence transfer coefficients to calculate the loop gain of the feedback loop.

[0045] In this embodiment, the calibration results of the transmission coefficients between adjacent nodes are shown in the following table:

[0046] The calibration results described above characterize the extent to which a disturbance is transmitted from one node to its neighboring nodes along the confidence transmission chain. In some embodiments, a confidence transmission matrix can be established based on the transmission coefficients between neighboring nodes, recording the transmission relationships and corresponding transmission coefficients between nodes in matrix form. In this embodiment, the non-zero transmission relationships corresponding to the confidence transmission matrix include feedback relationships from the scene reconstruction node to the traffic flow simulation node, from the traffic flow simulation node to the sensor simulation node, from the sensor simulation node to the response node of the autonomous driving system under test, from the response node of the autonomous driving system under test to the vehicle dynamics simulation node, from the vehicle dynamics simulation node to the test result node, and from the vehicle dynamics simulation node to the traffic flow simulation node, with corresponding transmission coefficients of... , , , , and .

[0047] In some embodiments, the marginal contribution can also be calculated based on the confidence values ​​of each model and the propagation coefficients of that model along the forward propagation path to the test result node. Defined as the first Confidence of each model The product of the transfer coefficients of each adjacent node along the forward propagation path of the model to the test result node is used to quantify the degree to which the improvement of the corresponding model confidence improves the overall confidence of the system.

[0048] For the scene reconstruction model, its forward propagation path passes through nodes 1, 2, 3, 4, 5, and 6 in sequence. The product of the forward propagation coefficients is:

[0049] Therefore, the marginal contribution of the scene reconstruction model is:

[0050] For the traffic flow simulation model, its forward propagation path passes through nodes 2, 3, 4, 5, and 6 in sequence, and the product of the forward propagation coefficients is:

[0051] Therefore, the marginal contribution of the traffic flow simulation model is:

[0052] For the sensor simulation model, its forward propagation path passes through nodes 3, 4, 5, and 6 sequentially. The product of the forward propagation coefficients is:

[0053] Therefore, the marginal contribution of the sensor simulation model is:

[0054] For the vehicle dynamics model, the forward propagation path is from node 5 to node 6, and the forward propagation coefficient is 0.90. Therefore, the marginal contribution of the vehicle dynamics model is:

[0055]

[0056] In this embodiment, the vehicle dynamics model is closest to the test result node, and its forward propagation path only passes through one adjacent propagation link, so its marginal contribution is the highest; the scene reconstruction model has the longest forward propagation path to the test result node, and the confidence influence is transmitted step by step through multiple adjacent propagation links, so its marginal contribution is the lowest.

[0057] For example, when the confidence level of the vehicle dynamics model increases from 0.818 to 0.900, the increase in confidence level is 0.082. Based on the forward pass coefficient of the vehicle dynamics model being 0.90, the expected increase in system-level confidence level is approximately:

[0058] When the confidence level of the scene reconstruction model also increases by 0.082, based on the product of the forward pass coefficients of the scene reconstruction model (0.268), the expected increase in system-level confidence is approximately:

[0059] Therefore, under the test system and transfer coefficient conditions of this embodiment, the priority order for improving model confidence can be determined based on the marginal contribution of each model. In this embodiment, the marginal contribution of the vehicle dynamics model, sensor simulation model, traffic flow simulation model, and scene reconstruction model decreases in that order.

[0060] The feedback loop sequentially passes through the traffic flow simulation node, the sensor simulation node, the tested autonomous driving system response node, and the vehicle dynamics simulation node, and is fed back from the vehicle dynamics simulation node to the traffic flow simulation node. The loop gain of this feedback loop is... Determined based on the transmission coefficients between adjacent nodes in the loop:

[0061] Substitute the above calibration values:

[0062] Since the loop gain is less than 1, the confidence propagation in the decision feedback loop converges. After each complete feedback loop, the confidence bias decreases to approximately 22.7% of the previous value; after three loops, the corresponding proportion is approximately:

[0063] When the loop gain is greater than or equal to 1, it is determined that the confidence propagation in the feedback loop is diverging.

[0064] To further verify the convergence determination of confidence propagation in the feedback loop, this embodiment also analyzes the case where some propagation coefficients in the feedback loop increase.

[0065] Under normal operating conditions, the transfer coefficients in the feedback loop are respectively , , and The corresponding loop gain is 0.227.

[0066] When the vehicle dynamics model deviates significantly from the modeling of emergency braking conditions, the transmission coefficient from the vehicle dynamics simulation node to the traffic flow simulation node will be affected. The value increased from 0.55 to 0.95, while fluctuations in traffic flow simulation behavior affected the transfer coefficient from traffic flow simulation nodes to sensor simulation nodes. When the loop gain is increased from 0.68 to 0.95, the loop gain is:

[0067] At this point, the loop gain is still less than 1, and the confidence propagation in the feedback loop still converges. However, compared to normal operating conditions, the degree of attenuation of the confidence deviation after one feedback loop is reduced.

[0068] When the response of the autonomous driving system under test further exhibits high sensitivity to the sensor simulation output, the equivalent signal transfer coefficient between the sensor simulation node and the response node of the autonomous driving system under test is increased.

[0069] When increased to 0.95, the loop gain is:

[0070] When the equivalent signal transmission coefficient between the response node of the tested autonomous driving system and the vehicle dynamics simulation node is... When further increased to 0.95, the loop gain is:

[0071] At this point, the loop gain approaches 1 further, indicating that the decay rate of the confidence deviation in the feedback loop is further reduced. When the propagation coefficient in the feedback loop continues to increase, making the loop gain greater than or equal to 1, it is determined that the confidence propagation in the feedback loop is diverging, and the overall system-level confidence does not have a steady-state value. In some embodiments, when determining that the confidence propagation in the feedback loop is diverging, a subsystem consistency alarm can be generated, and the links with larger propagation coefficients in the feedback loop can be checked first.

[0072] Through the above processing, the influence of the confidence level in the feedback loop can be judged by the calibrated adjacent node transmission coefficients, and the convergence judgment result can be connected to the subsequent system-level overall confidence level calculation.

[0073] Example 3 This embodiment, based on Embodiment 1, further explains the coefficient calibration of the nonlinear coupling transfer equation and the calculation of the overall system-level confidence level.

[0074] The nonlinear coupling transfer equation expresses the overall system-level confidence as a polynomial function of the confidence of each model, including first-order linear terms, second-order cross terms, and third-order and higher-order cross terms. Its general form is:

[0075] in, The system-level overall confidence level is... and Number the model. and The first The model and the first The confidence score of each model. For the first The coefficients of the first-order linear terms corresponding to each model For the first The model and the first The second-order cross term coefficients corresponding to each model. In this embodiment, the coefficients of the first-order linear term are calibrated through a single-factor perturbation experiment, the coefficients of the second-order cross-term are calibrated through a two-factor perturbation experiment, and the contribution of the third-order cross-term is evaluated through a three-factor perturbation experiment. When the third-order and higher-order cross-terms meet the truncation condition, the third-order and higher-order cross-terms can be truncated. At this time, the overall system-level confidence score is calculated according to the following formula:

[0076] The coefficients of the first-order linear terms were calibrated through a single-factor perturbation experiment. Specifically, multiple perturbations of known magnitude were applied to the output of a model. While keeping other model inputs constant, the corresponding values ​​of the overall system confidence under each level of perturbation were recorded, and a linear regression was performed on the relationship between the model confidence and the overall system confidence.

[0077] The following explanation uses the calibration process of the first-order linear term coefficients in the scene reconstruction model as an example. Keeping the inputs of the traffic flow simulation model, sensor simulation model, and vehicle dynamics model unchanged, only the output of the scene reconstruction model is calibrated. 20% 15% 10% Multi-level perturbations of 5%, 0%, and +5%. The confidence scores of the scene reconstruction model and the overall system-level confidence scores at each perturbation level are shown in the table below:

[0078] in, The magnitude of the perturbation applied to the output of the scene reconstruction model. The confidence level of the scene reconstruction model. This represents the overall confidence level at the system level.

[0079] Reconstructing model confidence based on scene For independent variables, system-level overall confidence A linear regression was performed on the dependent variable, yielding a regression slope of 0.184. This regression slope represents the local response relationship between the overall system-level confidence and the change in the confidence of the scene reconstruction model.

[0080] The regression slope includes not only the contribution of the first-order linear term corresponding to the scene reconstruction model, but also the contribution of the second-order cross term related to the scene reconstruction model. Therefore, after completing the two-factor perturbation calibration of each second-order cross term coefficient, the nonlinear coupling transfer equation is fitted globally using the least squares method, and the second-order cross term contribution is removed to obtain the coefficients of the first-order linear term corresponding to the scene reconstruction model.

[0081] The same method was used to conduct single-factor perturbation experiments on the traffic flow simulation model, sensor simulation model and vehicle dynamics model, and after completing the second-order cross term calibration, the overall fitting was performed to obtain the first-order linear term coefficients of each model.

[0082]

[0083] The sum of the coefficients of the first-order linear terms is 0.83, which is less than 1. This indicates that the overall confidence level of the system is not simply obtained by summing the first-order linear terms of the confidence levels of each model. It is also necessary to combine the contribution of the cross terms between the models for calculation.

[0084] The coefficients of the second-order cross term were calibrated through a two-factor perturbation experiment. For the... The model and the first Each model was individually perturbed to obtain the overall system-level confidence change. and Then, by simultaneously applying perturbations of the same magnitude to both models, the overall system-level confidence change is obtained. Interaction effect term Calculate according to the following formula:

[0085] when When, it indicates a positive coupling contribution between the two corresponding models; when When this occurs, it indicates a negative coupling contribution. The interaction effect term is normalized by dividing by the square of the corresponding perturbation magnitude and the product of the confidence values ​​of the two models, yielding the coefficients of the second-order interaction term. .

[0086] The following uses the second-order cross-term coefficients between the traffic flow simulation model and the sensor simulation model. Taking a two-factor perturbation experiment as an example, the specific calibration process is explained. In this embodiment, the confidence level of the traffic flow simulation model is... Confidence level of sensor simulation model The baseline value for the overall system-level confidence level is 0.748, and the disturbance magnitude is... .

[0087] The first step is to apply only the output of the traffic flow simulation model. With a 30% perturbation, keeping other models unchanged, the overall system-level confidence score after the perturbation is 0.686. Therefore, the change in overall system-level confidence score caused by perturbing only the traffic flow simulation model is:

[0088] in, This refers to the overall system-level confidence change when only a disturbance is applied to the output of the traffic flow simulation model. This represents the overall confidence level at the system level.

[0089] The second step is to apply the following to only the output of the sensor simulation model. With a 30% perturbation, keeping other models unchanged, the overall system-level confidence level after the perturbation is 0.675. Therefore, the change in the overall system-level confidence level caused by perturbing only the sensor simulation model is:

[0090] in, This represents the change in overall system-level confidence when only the output of the sensor simulation model is perturbed.

[0091] The third step is to simultaneously apply pressure to the outputs of the traffic flow simulation model and the sensor simulation model. A 30% perturbation was applied, keeping other models unchanged. The overall system-level confidence score after the perturbation was 0.612. Therefore, the change in overall system-level confidence score caused by simultaneously perturbing the traffic flow simulation model and the sensor simulation model is:

[0092] in, This refers to the overall system-level confidence change when perturbations are applied to the outputs of both the traffic flow simulation model and the sensor simulation model simultaneously.

[0093] The fourth step is to calculate the interaction effect term between the traffic flow simulation model and the sensor simulation model based on the system-level overall confidence change obtained from individual and simultaneous disturbances. :

[0094] in, This represents the interaction effect term between the traffic flow simulation model and the sensor simulation model. Because... This indicates that in this perturbation experiment, when both models simultaneously exhibit confidence decay, the overall system-level confidence decrease is less than the sum of the decreases when each model exhibits confidence decay separately, indicating a negative coupling contribution between the two models. In this embodiment, when traffic flow behavior is slightly distorted, the sensor simulation has a certain tolerance for detection errors of surrounding vehicles. Small deviations in traffic flow behavior do not directly cause corresponding changes in the target detection result. Therefore, there is a certain mutual compensation relationship between traffic flow behavior deviation and sensor detection error.

[0095] The interaction effect term is normalized. The normalization benchmark corresponding to the disturbance magnitude is:

[0096] in, In this embodiment, the perturbation magnitude is that of a two-factor perturbation experiment. , The confidence level of the traffic flow simulation model. This represents the confidence level of the sensor simulation model.

[0097] Therefore, the second-order cross-term coefficients between the traffic flow simulation model and the sensor simulation model are:

[0098] in, These are the second-order cross-term coefficients between the traffic flow simulation model and the sensor simulation model. Using the same method, two-factor perturbation experiments can be conducted on other model combinations, and the corresponding second-order cross-term coefficients can be calibrated.

[0099] The calibration results of the coefficients of each second-order cross term in this embodiment are shown in the table below:

[0100] In this embodiment, a three-factor perturbation experiment is also used to evaluate the third-order interaction term. Since the confidence values ​​of the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model are all between 0 and 1, the third-order interaction term is determined by the product of the confidence values ​​of the three models and the corresponding third-order interaction coefficient, and its magnitude is further reduced compared to the second-order interaction term. To verify whether the third-order and higher-order interaction terms can be truncated, perturbations are applied to the three models simultaneously and the corresponding three-factor interaction effects are evaluated.

[0101] Through three-factor perturbation experiments, it was verified that the absolute values ​​of the four third-order cross coefficients in this embodiment do not exceed 0.003, the total contribution of the third-order terms does not exceed 0.002, and the impact on the overall system-level confidence is less than 0.3%. For cases where the total contribution of the third-order terms is less than 1% of the calculated overall system-level confidence, third-order and higher-order cross terms can be truncated. In this embodiment, the total contribution of the third-order terms meets the above condition; therefore, in the subsequent calculation of the overall system-level confidence, third-order and higher-order cross terms are truncated, retaining only the first-order linear term and the second-order cross term.

[0102] Using the four model confidence values ​​obtained in Example 1, the first-order linear term was calculated, and the results are shown in the table below:

[0103] The corresponding second-order cross term calculation results are as follows:

[0104] Therefore, the overall system-level confidence level in this embodiment is:

[0105] In this system, the first-order linear terms correspond to the independent contributions of each model's confidence score, while the second-order interaction terms supplement the coupling contributions between different models. In this embodiment, the sum of the first-order linear terms is 0.707, the sum of the second-order interaction terms is +0.041, and the overall system-level confidence score obtained after incorporating the second-order interaction terms is 0.748. Compared to the 0.707 obtained using only the first-order linear terms, the increment brought by the second-order interaction terms is approximately 5.8%, and the total contribution of the six second-order interaction terms is positive. Therefore, the coupling effect between the confidence scores of each model is included in the calculation of the overall system-level confidence score.

[0106] Example 4 This embodiment, based on Embodiment 1, further explains the determination of scenario-level equivalence, behavior-level equivalence, result-level equivalence, and the determination of the substitutable ratio.

[0107] In some embodiments, the real test data may come from physical site testing or real road testing. In this embodiment, 50 left-turn scenario tests at the same intersection are performed in a physical test track, and corresponding simulation test data and real test data are obtained. Scenario-level equivalence determination is used to check the degree of correspondence between the simulation scenario and the real test scenario in terms of test conditions; behavioral-level equivalence determination is used to compare the vehicle's movement trajectory and key decision events in simulation and real tests; and outcome-level equivalence determination is used to compare the test indicators obtained from the two types of tests.

[0108] When performing scenario-level equivalence assessment, the road geometry parameters, traffic participant types and initial states, and environmental conditions in the simulated scenario and the real test scenario are matched. The matching results in this embodiment are shown in the table below:

[0109] There are 46 scenarios where all elements match. Scenario-level equivalence is determined based on the average element matching rate for each scenario. In this embodiment, the element matching rates for each scenario are 0.98, 1, 0.94, and 0.96, respectively; therefore, the scenario-level equivalence is:

[0110] in, For scenario-level equivalence.

[0111] Forty-six test scenarios that passed the scenario-level equivalence assessment were used as the objects of the behavior-level equivalence assessment, and vehicle trajectories and key decision event sequences were extracted from the simulation test and the real test, respectively.

[0112] The trajectory consistency score is determined based on the dynamic time warping distance between the simulated trajectory and the actual test trajectory. Taking test scenario number 3 as an example, a dynamic time warping matrix is ​​constructed using the simulated trajectory sampling point sequence and the actual test trajectory sampling point sequence. The cumulative distance of the optimal matching path is 8.7m, the trajectory length is 180m, and the corresponding trajectory consistency score is:

[0113] The average dynamic time normalization score for the 46 test scenarios was 0.87.

[0114] The decision consistency score is determined based on the edit distance between the key decision event sequences in the simulation test and the real test. Taking test scenario number 3 as an example, the key decision event sequence in the simulation test is [decelerate to approach, stop to yield, start to turn left, accelerate to pass], while the key decision event sequence in the real test is [decelerate to approach, stop to yield, start to turn left, fine-tune direction, accelerate to pass]. The edit distance between the two event sequences is 1, and the sequence length is 5. The corresponding decision consistency score is:

[0115] The average edit distance normalized score for the 46 test scenarios was 0.83. Weighted by a score of 0.6 for trajectory consistency and 0.4 for decision consistency, behavioral equivalence was obtained.

[0116] in, For behavioral equivalence.

[0117] When determining the equivalence at the outcome level, a two-sided and one-sided equivalence test is performed on the simulation test indicators and the real test indicators for all 50 test scenarios. In this embodiment, collision time margin, passage time, and minimum collision distance are selected as test indicators, and the equivalence of each test scenario is determined based on the results of the two-sided and one-sided equivalence test for each test scenario.

[0118] The significance level for the two-sided and one-sided equivalence test was set at 1. The following section uses travel time as an example to illustrate the process of bilateral and unilateral equivalence testing.

[0119] Set the equivalence threshold for passage time For each of the 50 test scenarios, the difference between the simulated travel time and the actual travel time was calculated, forming a difference sequence. The mean of the difference sequence Standard deviation Number of test scenarios .

[0120] Calculate the first one-sided test statistic based on the mean, standard deviation, and equivalence threshold of the difference sequence. :

[0121] Calculate the second one-sided test statistic based on the mean, standard deviation, and equivalence threshold of the difference sequence. :

[0122] in, This is the first one-sided test statistic. This is the second one-sided test statistic. This represents the average difference between simulated and actual travel times across 50 test scenarios. This serves as the equivalence threshold for travel time. The standard deviation of the difference sequence is... This represents the number of test scenarios.

[0123] In this embodiment, degrees of freedom Determined according to the following formula:

[0124] in, For degrees of freedom. At the significance level. Degrees of freedom Under the conditions, check The distribution table yields the critical value for the one-tailed test:

[0125] because:

[0126] and:

[0127] Therefore, the travel time passed the two-sided equivalence test.

[0128] The same method was used to conduct two-sided equivalence tests on collision time margin and minimum collision distance. Ultimately, in 38 out of 50 test scenarios, all three test metrics—collision time margin, passage time, and minimum collision distance—passed the two-sided equivalence tests. Therefore, outcome-level equivalence is achieved. for:

[0129] in, For outcome-level equivalence.

[0130] The overall system-level confidence level obtained in conjunction with Example 3 The substitutable ratio for simulation testing is determined according to the following formula:

[0131] in, As a replaceable ratio, For outcome-level equivalence, the value range is [0,1]. This represents the overall confidence level at the system level, with a value range of [0,1].

[0132] Substitute the data into this embodiment:

[0133] Therefore, under the test conditions corresponding to this embodiment, the result-level equivalence is 0.760, and combined with the overall system-level confidence level of 0.748, the substitution ratio of the simulation test is determined to be 56.8%.

[0134] Example 5 This embodiment, in conjunction with a complete test process, explains the confidence transfer, system-level overall confidence calculation, and equivalence determination results between simulation and real tests in the above embodiments. For the specific calculation and calibration processes of the confidence transfer coefficient, nonlinear coupling transfer equation, and equivalence determination, please refer to Embodiments 2 to 4. This embodiment further explains the confidence quantification process of the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model, and provides a complete explanation of the test results.

[0135] This embodiment uses a Level 4 autonomous driving system performing a left turn at an unsignalized intersection in an urban area as the test object. The autonomous driving system under test is the perception-decision-control software stack to be verified. The sensor configuration includes cameras and LiDAR. The cameras include a front-facing camera, a left-side camera, and a right-side camera. The LiDAR is a 128-line LiDAR mounted on the roof of the vehicle.

[0136] The autonomous driving simulation test system includes a scene reconstruction model, a traffic flow simulation model, a sensor simulation model, and a vehicle dynamics model. The scene reconstruction model constructs a 3D road environment based on OpenDRIVE high-precision maps; the traffic flow simulation model generates oncoming straight-going vehicles, pedestrians, and non-motorized vehicles based on game theory agents; the sensor simulation model generates camera images using ray tracing rendering and LiDAR point clouds using ray projection; the vehicle dynamics model uses a 7-DOF vehicle dynamics model to simulate longitudinal motion, lateral motion, yaw motion, and the rotation of the four wheels. The autonomous driving system under test runs in the closed-loop test environment of the autonomous driving simulation test system, and the corresponding real test results come from physical site tests at the same type of intersection.

[0137] In this embodiment, the confidence values ​​of the four models can be quantified according to the simulation characteristics of the corresponding models. Among them, the confidence of the scene reconstruction model is obtained by weighting the structural similarity component, semantic consistency component, physical rationality component, and distribution coverage component. Each component is calculated using extended structural similarity, entity annotation matching rate, kinematic constraint satisfaction, and maximum mean difference, respectively, and the weights are calibrated using the analytic hierarchy process (AHP).

[0138] In this embodiment, based on the importance of each component to the scene reconstruction fidelity, the Saaty 1-9 scale is used to compare the structural similarity component, semantic consistency component, physical plausibility component, and distribution coverage component pairwise. Structural similarity and physical plausibility directly affect the visual realism and kinematic correctness of the simulation environment and are therefore highly important; semantic consistency affects scene understanding; and distribution coverage is used to characterize scene diversity. The confidence judgment matrix of the constructed scene reconstruction model is shown in the table below:

[0139] Determine the sum of the columns of the matrix as follows:

[0140] Divide each element in the judgment matrix by the sum of its column, and then average the results across the normalized rows to obtain the weights of the structural similarity component, semantic consistency component, physical rationality component, and distribution coverage component, respectively:

[0141]

[0142]

[0143]

[0144] in, , , and These represent the weights for the structural similarity component, semantic consistency component, physical plausibility component, and distribution coverage component, respectively. Therefore, the weight vector corresponding to the confidence score of the scene reconstruction model is:

[0145] Perform a consistency check on the judgment matrix. The largest eigenvalue of the judgment matrix is:

[0146] The consistency index for judging the matrix is:

[0147] in, To determine the consistency index of a matrix, To determine the largest eigenvalue of a matrix, In this embodiment, to determine the order of a matrix, When the order of the judgment matrix is ​​4, the random consistency index... Therefore, the consistency ratio is:

[0148] in, To determine the consistency ratio of the matrix, This is used as an index to judge the random consistency of a matrix. Because... The consistency of the judgment matrix meets the requirements.

[0149] Continuing to calculate each component, the calculation results of each component of the scene reconstruction model confidence in this embodiment are shown in the table below:

[0150] Based on the above weights and the measured values ​​of each component, the confidence level of the scene reconstruction model is:

[0151] in, The confidence level of the scene reconstruction model.

[0152] The confidence score of the traffic flow simulation model is obtained by weighting the game behavior authenticity component, the interaction response consistency component, and the traffic flow statistical characteristic component. Specifically, the game behavior authenticity component is calculated based on the difference between the distribution of simulated traffic participant behavior and the distribution of real driver behavior data; the interaction response consistency component is calculated based on the dynamic time warping distance of traffic participant behavior sequences in simulation and real tests under the same initial conditions; and the traffic flow statistical characteristic component is calculated based on the comparison between the simulated traffic flow flow, density, and speed distribution and the real traffic flow statistical data.

[0153] In this embodiment, based on the characteristics of the intersection scenario, the Saaty 1-9 scale is used to compare the game behavior authenticity component, the interaction response consistency component, and the traffic flow statistical characteristic component pairwise. Game behavior authenticity directly affects whether oncoming vehicles correctly yield or overtake, and its importance is the highest; interaction response consistency affects the timing of multi-vehicle interactions, and its importance is the next highest; traffic flow statistical characteristics are used to assist in the evaluation. The 3×3 judgment matrix for the confidence level of the traffic flow simulation model is constructed as shown in the table below:

[0154] The sum of the columns of the judgment matrix is:

[0155] The judgment matrix is ​​column-normalized, and the average of each normalized row is taken to obtain the weights of the game behavior authenticity component, the interaction response consistency component, and the traffic flow statistical characteristic component, respectively:

[0156]

[0157]

[0158] in, , and These represent the weights for the game behavior authenticity component, the interaction response consistency component, and the traffic flow statistical characteristics component, respectively. The weight vector corresponding to the confidence level of the traffic flow simulation model is:

[0159] A consistency check is performed on the judgment matrix, and the largest eigenvalue of the judgment matrix is:

[0160] The consistency index for judging the matrix is:

[0161] in, To determine the consistency index of a matrix, To determine the largest eigenvalue of a matrix, In this embodiment, to determine the order of a matrix, The random consistency index of the judgment matrix is ​​obtained by looking up the table, and the consistency ratio is calculated:

[0162] in, To determine the consistency ratio of the matrix, This is used as an index to judge the random consistency of a matrix. Because... The consistency of the judgment matrix meets the requirements.

[0163] This embodiment further calculates the game behavior authenticity component, the interaction response consistency component, and the traffic flow statistical characteristic component, and the results are shown in the table below:

[0164] Among them, NGSIM is the natural driving dataset, KL divergence is used to characterize the difference between the distribution of simulated traffic participants’ behavior and the distribution of real drivers’ behavior, and DTW is the dynamic time warping distance.

[0165] Based on the above weights and the measured values ​​of each component, the confidence level of the traffic flow simulation model is calculated according to the following formula:

[0166] in, This represents the confidence level of the traffic flow simulation model.

[0167] The confidence level of the sensor simulation model is obtained by weighting pixel-level fidelity, point cloud-level fidelity, signal-level fidelity, and functional-level fidelity. The weighting coefficients are dynamically adjusted according to the sensor configuration used by the autonomous driving system under test, and the weights for unused sensor types are reset to zero.

[0168] In this embodiment, the initial weights for pixel-level fidelity, point cloud-level fidelity, signal-level fidelity, and functional-level fidelity are 0.30, 0.30, 0.20, and 0.20, respectively. Among these, the millimeter-wave radar-related content accounts for approximately 25% of the signal-level fidelity. Since the tested autonomous driving system in this embodiment does not use millimeter-wave radar, the weights corresponding to the millimeter-wave radar are reset to zero, adjusting the signal-level fidelity weight from 0.20 to 0.15. The released 0.05 weight is then distributed according to the original weight ratio of pixel-level fidelity to functional-level fidelity. The original weight ratio of pixel-level fidelity to functional-level fidelity is 0.30:0.20; therefore, weights are increased by 0.03 and 0.02 respectively, resulting in the adjusted weight vector.

[0169] in, This is the weight vector corresponding to the confidence level of the sensor simulation model, where each element corresponds to pixel-level fidelity, point cloud-level fidelity, signal-level fidelity, and functional-level fidelity, respectively.

[0170] After adjusting the weights, each fidelity component is calculated separately. The calculation results for each component in this embodiment are shown in the table below:

[0171] For point cloud-level fidelity, its normalization calculation is as follows:

[0172] For functional-level fidelity, the consistency between target detection results in simulated images and real images is as follows:

[0173] Based on the adjusted weight vector and the measured values ​​of each component, the confidence level of the sensor simulation model is calculated using the following formula:

[0174] in, This represents the confidence level of the sensor simulation model.

[0175] The confidence level of the vehicle dynamics model is obtained by weighting the state quantity fidelity component, the tire force modeling accuracy component, and the response delay consistency component. Specifically, the state quantity fidelity component is calculated by normalizing the root mean square error between the simulated yaw rate, lateral acceleration, and longitudinal velocity and the actual vehicle test values; the tire force modeling accuracy component is calculated by normalizing the root mean square error between the simulated tire longitudinal force and lateral force and the tire test bench data; and the response delay consistency component is calculated by normalizing the deviation between the rise time of the simulated model's step response and the measured value of the actual system.

[0176] In this embodiment, the Saaty 1-9 scale is used to compare the state fidelity component, tire force modeling accuracy component, and response delay consistency component pairwise. State fidelity and tire force modeling accuracy directly affect the vehicle motion state prediction results, and their importance is equal. Response delay consistency reflects the dynamic response timing between the vehicle dynamics model and the real vehicle, and its importance is lower than the aforementioned two components. The resulting judgment matrix for the vehicle dynamics model confidence is shown in the table below:

[0177] Determine the sum of the columns of the matrix as follows:

[0178] Divide each element in the judgment matrix by the sum of its column, and average the results across the normalized rows to obtain the weights of the state quantity fidelity component, the tire force modeling accuracy component, and the response delay consistency component, respectively:

[0179]

[0180]

[0181] in, , and These represent the weights for the state quantity fidelity component, the tire force modeling accuracy component, and the response delay consistency component, respectively. Therefore, the weight vector corresponding to the vehicle dynamics model confidence is:

[0182] A consistency check is performed on the judgment matrix, and the largest eigenvalue of the judgment matrix is:

[0183] The consistency index for judging the matrix is:

[0184] in, To determine the consistency index of a matrix, To determine the largest eigenvalue of a matrix, In this embodiment, to determine the order of a matrix, The random consistency index of the judgment matrix is ​​obtained by looking up the table. Therefore, the consistency ratio is:

[0185] in, To determine the consistency ratio of the matrix, This is used as an index to judge the random consistency of a matrix. Because... The consistency of the judgment matrix meets the requirements.

[0186] After determining the weights of each component, the state quantity fidelity component, the tire force modeling accuracy component, and the response delay consistency component are calculated separately. The calculation results in this embodiment are shown in the table below:

[0187] Regarding the fidelity of state variables, the normalized results for yaw rate, lateral acceleration, and longitudinal velocity are as follows:

[0188]

[0189]

[0190] Taking the average of the three normalization results above, we get:

[0191] Regarding the accuracy of tire force modeling, the normalized results for longitudinal and lateral forces are as follows:

[0192]

[0193] Taking the average of the two normalization results, we get:

[0194] Regarding response delay consistency, the rise time deviation between the simulation model and the real system is 0.03s. Normalization based on a 0.2s threshold yields:

[0195] Based on the above weights and the measured values ​​of each component, the confidence level of the vehicle dynamics model is calculated according to the following formula:

[0196] in, This represents the confidence level of the vehicle dynamics model.

[0197] Based on the above quantification process, the confidence values ​​of the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model are summarized in the table below:

[0198] The response node of the tested autonomous driving system corresponds to the real autonomous driving algorithm software stack to be verified. It is itself the object under test, not a simulation model. In this embodiment, its confidence level is fixed at 1 and it does not participate in the confidence quantification of the four models mentioned above. Based on the perturbation experiment calibration results between adjacent nodes in the confidence propagation chain, the loop gain of the feedback loop is 0.227, which is less than 1. Therefore, the confidence propagation in the feedback loop is determined to be converged. Under this convergence condition, substituting the confidence values ​​of the four models into the nonlinear coupling propagation equation with calibrated coefficients, the overall system-level confidence level is obtained as 0.748.

[0199] Further simulation and real-world tests were conducted on 50 left-turn scenarios at the same intersection. The results of scenario-level, behavioral-level, and outcome-level equivalence assessments are summarized in the table below:

[0200] The above results indicate that this embodiment does not directly determine the substitutable ratio based solely on the difference between the simulation test results and the actual test results. Instead, it simultaneously considers the overall system-level confidence of the autonomous driving simulation test system itself and the result-level equivalence between the simulation test and the actual test, so that the final substitutable ratio is constrained by both the overall credibility of the autonomous driving simulation test system and the degree of equivalence between the simulation test results and the actual test results.

[0201] In this embodiment, the generated equivalence determination result may also include information on the applicable scope of the corresponding test conditions. For example, the applicable scope corresponding to this test is shown in the table below:

[0202] After the expiration of the validity period, the transfer coefficients and coefficients of the nonlinear coupling transfer equation can be recalibrated. The above scope of application information is used to explain the specific test conditions corresponding to the equivalence determination results obtained in this embodiment, and does not constitute a limitation on the scope of application of the autonomous driving simulation test equivalence determination method of this application.

[0203] Example 6 In one embodiment of this application, such as Figure 3As shown, an equivalence determination system for autonomous driving simulation testing is provided, including a confidence acquisition module, a confidence transfer chain construction module, a closed-loop transfer analysis module, a system-level confidence determination module, and an equivalence determination module.

[0204] The confidence acquisition module is used to obtain the confidence values ​​of the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model in the autonomous driving simulation test system.

[0205] The confidence transfer chain construction module is used to construct the confidence transfer chain, which includes, in sequence, a scene reconstruction node, a traffic flow simulation node, a sensor simulation node, a test autonomous driving system response node, a vehicle dynamics simulation node, and a test result node. The vehicle dynamics simulation node feeds back to the traffic flow simulation node, forming a feedback loop.

[0206] The closed-loop transfer analysis module is used to calibrate the transfer coefficients between adjacent nodes in the confidence transfer chain through perturbation experiments, determine the loop gain based on the transfer coefficients between adjacent nodes in the feedback loop, and determine whether the confidence transfer in the feedback loop has converged based on the loop gain.

[0207] The system-level confidence determination module is used to calibrate the coefficients of the nonlinear coupling transfer equation, and when the confidence transfer in the feedback loop converges, it calculates the overall system-level confidence based on the confidence values ​​of the scene reconstruction model, the traffic flow simulation model, the sensor simulation model, and the vehicle dynamics model, as well as the nonlinear coupling transfer equation.

[0208] The equivalence determination module is used to perform result-level equivalence determination based on simulation test data and real test data, obtain result-level equivalence, determine the substitutable ratio of simulation test according to the overall system-level confidence and the result-level equivalence, and generate equivalence determination results.

[0209] In one implementation, the modules interact via a data bus. The confidence acquisition module receives inputs including the output data of each model and corresponding real test reference data, and outputs confidence values ​​for the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model. The confidence transfer chain construction module constructs a confidence transfer chain based on the confidence values. The closed-loop transfer analysis module outputs the transfer coefficients between adjacent nodes and the confidence transfer convergence determination results in the feedback loop. The system-level confidence determination module obtains the overall system-level confidence based on the confidence values ​​of each model, the nonlinear coupling transfer equation, and the confidence transfer convergence determination results. The equivalence determination module receives inputs including simulation test data, real test data, and the overall system-level confidence, and outputs the substitutable ratio of the simulation test and the equivalence determination results.

[0210] Specific limitations regarding the equivalence determination system for autonomous driving simulation testing can be found in the limitations of the equivalence determination method for autonomous driving simulation testing described above, and the corresponding technical effects can also be obtained accordingly, which will not be repeated here. Each module in the aforementioned equivalence determination system for autonomous driving simulation testing can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in computer-readable storage media in a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0211] Example 7 Figure 4 An internal structural diagram of a computer device is shown in one embodiment. This computer device may specifically be a terminal or a server. Figure 4 As shown, the computer device includes a processor and a computer-readable storage medium, and may also include a network interface, a display, and an input device connected via a system bus, depending on actual needs.

[0212] The processor provides computing and control capabilities and is adapted to execute computer programs. A computer-readable storage medium stores a computer program that, when executed by the processor, implements the aforementioned method for determining the equivalence of autonomous driving simulation tests. The computer-readable storage medium may include a non-volatile storage medium for storing an operating system, computer programs, and related data, or it may include storage units that provide an environment for the computer program to run. A network interface can be used for data communication between the computer device and external devices. The display may be a liquid crystal display or an e-ink display, and the input device may be a touch layer on the display, buttons, a trackball, or a touchpad on the computer device, or an external keyboard, touchpad, or mouse.

[0213] Those skilled in the art will understand that Figure 4 The computer device structure shown is merely an exemplary structure related to the technical solution of this application and does not constitute a limitation on the computer device to which the technical solution of this application is applied. Specific computer devices may include, for example, […]. Figure 4 The diagram shows more or fewer components, some of which can be combined, or different component arrangements can be used.

[0214] In one embodiment, a computer device is provided, including a processor and a computer-readable storage medium; the processor is adapted to execute a computer program; the computer-readable storage medium stores a computer program, which, when executed by the processor, implements the above-described method for determining the equivalence of autonomous driving simulation tests.

[0215] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described method for determining the equivalence of autonomous driving simulation tests.

[0216] In summary, the autonomous driving simulation test equivalence determination method, system, computer equipment, and computer-readable storage medium provided in this application construct a confidence transmission chain containing feedback loops, analyze the transmission relationship between adjacent nodes and the convergence of confidence transmission in the feedback loops, and obtain the overall system-level confidence by combining nonlinear coupling transmission equations. Based on this, the substitutable ratio of simulation tests is determined by combining the result-level equivalence of simulation tests and real tests, thereby quantitatively determining the degree of equivalence between autonomous driving simulation tests and real tests.

[0217] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. Especially for system embodiments, since they correspond to method embodiments, the relevant processing procedures can be found in the corresponding descriptions in the method embodiments. It should be noted that the technical features in the above embodiments can be combined without contradiction. For the sake of brevity, not all possible combinations have been described one by one.

[0218] The embodiments described above are only some implementation methods of this application and are not intended to limit the scope of protection of this application. For those skilled in the art, corresponding modifications, equivalent substitutions or improvements can be made without departing from the technical principles of this application.

Claims

1. A method for determining the equivalence of autonomous driving simulation tests, characterized in that, include: Obtain the confidence values ​​of the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model in the autonomous driving simulation test system; A confidence transfer chain is constructed, which sequentially includes a scene reconstruction node, a traffic flow simulation node, a sensor simulation node, a test autonomous driving system response node, a vehicle dynamics simulation node, and a test result node. The vehicle dynamics simulation node feeds back to the traffic flow simulation node, forming a feedback loop. The transmission coefficients between adjacent nodes in the confidence transmission chain are calibrated through perturbation experiments. The loop gain is determined based on the transmission coefficients between adjacent nodes in the feedback loop. The convergence of the confidence transmission in the feedback loop is then determined based on the loop gain. The coefficients of the nonlinear coupling transfer equation are calibrated, and when the confidence transfer in the feedback loop converges, the overall system-level confidence is calculated based on the confidence values ​​of the scene reconstruction model, the traffic flow simulation model, the sensor simulation model, and the vehicle dynamics model, as well as the nonlinear coupling transfer equation. The result-level equivalence is determined based on simulation test data and real test data to obtain the result-level equivalence. The substitutable ratio of the simulation test is determined according to the overall system-level confidence and the result-level equivalence, and the equivalence determination result is generated.

2. The method for determining the equivalence of autonomous driving simulation testing according to claim 1, characterized in that, The transfer coefficients include confidence transfer coefficients and equivalent signal transfer coefficients; For adjacent nodes that are not involved in the response node of the tested autonomous driving system, while keeping the input of other nodes unchanged, a perturbation of known magnitude is applied to the output of the upstream node, and the corresponding confidence transfer coefficient is calibrated according to the confidence change of the adjacent downstream nodes. The transmission coefficients between the sensor simulation node and the response node of the autonomous driving system under test, and between the response node of the autonomous driving system under test and the vehicle dynamics simulation node, are equivalent signal transmission coefficients, which are calibrated based on the changes in the response of the autonomous driving system under test caused by sensor output signal disturbances and the changes in the vehicle dynamics simulation output caused by control command disturbances.

3. The method for determining the equivalence of autonomous driving simulation testing according to claim 1, characterized in that, When the third-order and higher-order cross terms are truncated, the nonlinear coupling transfer equation includes a first-order linear term and a second-order cross term, and the overall system-level confidence is calculated according to the following formula: in, The system-level overall confidence level is... and Number the model. and The first The model and the first The confidence score of each model. For the first The coefficients of the first-order linear terms corresponding to each model For the first The model and the first The coefficients of the second-order cross terms corresponding to each model; The coefficients of the first-order linear term were calibrated through a single-factor perturbation experiment, and the coefficients of the second-order cross term were calibrated through a two-factor perturbation experiment.

4. The method for determining the equivalence of autonomous driving simulation testing according to claim 1, characterized in that, The step of determining the loop gain based on the transmission coefficients between adjacent nodes in the feedback loop, and determining whether the confidence propagation in the feedback loop converges based on the loop gain, includes: The absolute value of the product of the transmission coefficients between adjacent nodes in the feedback loop is determined as the loop gain. When the loop gain is less than 1, the confidence propagation in the feedback loop is determined to be converged; when the loop gain is greater than or equal to 1, the confidence propagation in the feedback loop is determined to be divergent.

5. The method for determining the equivalence of autonomous driving simulation testing according to claim 1, characterized in that, Before performing the result-level equivalence determination, scenario-level equivalence determination and behavior-level equivalence determination are also included. The road geometry parameters, traffic participant types and initial states, and environmental conditions in the simulation scenario and the real test scenario are matched, and the scenario-level equivalence is determined based on the average matching rate of scenario elements. The test scenarios that pass the scenario-level equivalence determination are used as the objects of behavior-level equivalence determination. The trajectory consistency score is determined based on the dynamic time warping distance between the simulation trajectory and the real test trajectory, and the decision consistency score is determined based on the edit distance of the key decision event sequences in the simulation test and the real test. The trajectory consistency score and the decision consistency score are weighted to obtain the behavior-level equivalence.

6. The method for determining the equivalence of autonomous driving simulation testing according to claim 1, characterized in that, The result-level equivalence determination includes: A two-sided equivalence test is performed on the simulation test indicators and the real test indicators for each test scenario. The result-level equivalence is determined based on the proportion of the number of test scenarios that are determined to be equivalent in the two-sided equivalence test to the total number of test scenarios.

7. The method for determining the equivalence of autonomous driving simulation testing according to claim 1, characterized in that, The substitutable ratio is calculated according to the following formula: in, The aforementioned alternative ratio, For the aforementioned result-level equivalence, The system-level overall confidence level is given.

8. An equivalence determination system for autonomous driving simulation testing, characterized in that, include: The confidence level acquisition module is used to acquire the confidence level values ​​of the scene reconstruction model, traffic flow simulation model, sensor simulation model, and vehicle dynamics model in the autonomous driving simulation test system. The confidence transfer chain construction module is used to construct the confidence transfer chain, which sequentially includes a scene reconstruction node, a traffic flow simulation node, a sensor simulation node, a test autonomous driving system response node, a vehicle dynamics simulation node, and a test result node. The vehicle dynamics simulation node feeds back to the traffic flow simulation node to form a feedback loop. The closed-loop transfer analysis module is used to calibrate the transfer coefficient between adjacent nodes in the confidence transfer chain through perturbation experiments, determine the loop gain based on the transfer coefficient between each adjacent node in the feedback loop, and determine whether the confidence transfer in the feedback loop has converged based on the loop gain. The system-level confidence determination module is used to calibrate the coefficients of the nonlinear coupling transfer equation, and calculate the overall system-level confidence based on the confidence values ​​of the scene reconstruction model, the traffic flow simulation model, the sensor simulation model, and the vehicle dynamics model, as well as the nonlinear coupling transfer equation, when the confidence transfer in the feedback loop converges. The equivalence determination module is used to perform result-level equivalence determination based on simulation test data and real test data, obtain result-level equivalence, determine the substitutable ratio of simulation test according to the overall system-level confidence and the result-level equivalence, and generate equivalence determination results.

9. A computer device, characterized in that, Includes processors and computer-readable storage media; The processor is adapted to execute computer programs; The computer-readable storage medium stores a computer program, which, when executed by the processor, implements the autonomous driving simulation test equivalence determination method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1 to 7 for determining the equivalence of autonomous driving simulation tests.