An automatic driving virtual-real fusion test credibility evaluation method and system

Abstract

The present application relates to the technical field of automatic driving car simulation test, and particularly relates to a kind of automatic driving virtual-real fusion test credibility evaluation method and system, the present application technical scheme is: the evaluation system comprising scene dynamic generation credibility, signal injection fidelity, closed-loop timing consistency and test result reproducibility four first-level indexes is constructed;Multiple-source data in the test process is collected to calculate the quantitative value of each index;The entropy weight-variation coefficient combination weighting method is used to determine the weight, and the overall credibility of the system is obtained by weighted aggregation and mapping grade;Real-time credibility monitoring, abnormal classification processing and traceability positioning are realized simultaneously in the test process.The present application can evaluate the full-link credibility of the automatic driving virtual-real fusion test system, break through the limitations of existing methods such as replacing the test system credibility evaluation with data consistency comparison, and ensure the effectiveness and reliability of the test results.

Description

Technical Field

[0001] This invention relates to the technical field of autonomous vehicle simulation testing, and in particular to a method and system for evaluating the credibility of virtual-real fusion testing of autonomous vehicles. Background Technology

[0002] With the rapid development of autonomous driving technology, the systematic and efficient safety verification and testing of autonomous driving systems has become a recognized key technical challenge in the industry. The United Nations Economic Commission for Europe (UNECE) Forum for Harmonization of Automated Vehicles (GRVA) has proposed a "multi-pillar approach" for autonomous driving functions, combining review and auditing, virtual testing, track testing, and real-road testing to conduct autonomous driving function verification. Among these, virtual simulation testing has become an indispensable key component due to its advantages such as customizable scenarios, repeatable testing, and controllable risks.

[0003] In the field of virtual simulation testing for autonomous driving, virtual-real fusion testing technology has received widespread attention in recent years. This technology introduces real vehicles or hardware into the simulation environment, maintaining the real dynamic characteristics of the system under test while injecting virtual scenarios to achieve repeatable and accelerated testing of the autonomous driving system. Virtual-real fusion testing technology, particularly for closed test tracks or controlled roads, places the autonomous vehicle in a closed test track or controlled road environment. A test server is directly connected to the autonomous driving domain controller of the vehicle under test to form a closed-loop test system. The test server dynamically simulates and generates various virtual test scenarios according to the test tasks, translates the virtual scenarios into perception layer signals, and injects them into the domain controller. The autonomous vehicle under test implements autonomous driving functions based on the injected signals. The test server then generates the test scenario for the next simulation step in real time based on the functional performance of the autonomous vehicle under test, thus achieving closed-loop testing of virtual-real fusion. In 2024, the team at Chang'an University released the world's first "Pioneer Vehicle-Cloud-Field Integrated Autonomous Driving Virtual-Real Fusion Testing System," marking the transition of this technology from theoretical research to engineering application.

[0004] However, the effectiveness of virtual-real fusion testing fundamentally depends on the reliability of the testing system itself—that is, whether the testing system can reliably complete the entire virtual-real fusion testing process, from dynamic scene generation to test result output. If the testing system has logical flaws in the scene generation stage, imperceptible distortions in the signal injection stage, or timing discrepancies in the closed-loop interaction stage, the test results obtained based on this system may become invalid, and may even lead to misjudgments about the safety of the autonomous driving system of the tested autonomous vehicle. Therefore, assessing the reliability of the autonomous driving virtual-real fusion testing system itself is a necessary prerequisite for ensuring the reliability of virtual-real fusion test results.

[0005] While existing technologies have made some progress in assessing the credibility of simulation tests, they still have the following shortcomings:

[0006] (1) Existing methods mostly focus on the credibility assessment of simulation output data of test systems.

[0007] Some existing methods set the evaluation object as the consistency between simulation output data and real data, which essentially evaluates the "credibility of simulation-generated data" rather than the "credibility of the virtual-real fusion test system itself".

[0008] For example, the Chinese invention patent "A Quantitative Evaluation Method for the Credibility of Vehicle-in-the-Loop Simulation Testing (Publication No. CN120124286A)" constructs multiple test scenarios to conduct parallel tests on simulated and real vehicles, collects simulation datasets and real vehicle test datasets, and calculates simulation test scores based on multi-dimensional error scores and multi-curve trend consistency scores. Its core logic uses the consistency between simulation output and real vehicle test data as the criterion for measuring simulation credibility, focusing on the degree of agreement between virtual and real data, rather than the credibility of the test system itself in performing the virtual-real fusion test process. However, this method has methodological limitations: even if the simulation output is highly consistent with the real vehicle data, if the test system has logical loopholes in the scenario generation stage, systematic deviations in the signal injection stage, or implicit errors in the time delay synchronization stage, the test results may still be unreliable, and this method lacks the ability to identify such process-level defects.

[0009] The paper "AD-VILS: Implementation and Reliability Validation of Vehicle-In-the-Loop Simulation Platform for Evaluating Autonomous Driving Systems (OhT, Ha Y, Yoo D, Yoo J, IEEE Access, 2024)" proposes a reliability verification method for the VILS platform. It verifies platform reliability by statistically and mathematically comparing the consistency and correlation between virtual test and real-vehicle test datasets. This method also uses the comparison between virtual test data and real-vehicle test data as its core evaluation logic, evaluating the reliability of the VILS platform's output data, rather than the credibility of the platform's own test execution process.

[0010] The paper "XILS Credibility Assessment and Scenario Representativeness Methodology Based on Geometric Similarity Analysis for Autonomous Driving Systems" (Han S, Oh T, Lee S, Park S, Yoo J, Applied Sciences, 2025) proposes a XILS credibility assessment method based on geometric similarity analysis. This method combines three techniques: dynamic fidelity assessment, parametric reliability assessment, and scenario reliability assessment, comparing geometric similarity using Procrustes distance, Fréchet distance, and Hausdorff distance. While this method enriches the assessment dimensions from both fidelity and reliability perspectives, its core assessment logic remains the comparison of similarity and consistency between XILS data and real-world data. The assessment object remains the reliability output of the simulation platform, rather than the credibility of the test system itself.

[0011] (2) Existing methods use “data comparison mode” to assess credibility, which cannot cover the credibility of the entire functional link of virtual-real fusion test.

[0012] The full-function chain of virtual-real fusion testing for autonomous driving includes four key stages: "dynamic scene generation → signal translation and injection → closed-loop timing interaction → result output reproduction." Each stage can potentially become a bottleneck in reliability. However, existing methods focus on evaluating whether the simulation output is similar to real data or the performance of subsystems. These "data comparison mode" methods can only answer the question of "whether the simulation output resembles real data," but cannot answer the question of "whether the functional stages of the virtual-real fusion testing system are reliable."

[0013] For example, the Chinese invention patent "A Credibility Evaluation Method for a Site-Based Vehicle-in-the-Loop Simulation System (Publication No. CN121833432A)" constructs two evaluation indicators: the realism of virtual-real interaction and the real-time synchronization performance of the system. The former evaluates the realism of virtual-real interaction by comparing the similarity between virtual and real signals, while the latter evaluates the real-time synchronization capability by checking the consistency of the system's synchronization performance indicators. Both dimensions use the comparison of simulation data with real data or the comparison of system performance indicators with thresholds as the evaluation logic. In this type of method, the evaluation dimensions are limited to two levels: "whether the output resembles real data" and "whether the performance meets the standards." The logical completeness, parameter coverage, and evolutionary coherence of the scene dynamic generation process, as well as the reproducibility of test results, are not within the scope of data comparison and are therefore difficult to include in this type of evaluation method. However, the credibility of the simulation scene is the starting point for virtual-real fusion testing. If the virtual scene of the test system has logical defects, insufficient coverage in the parameter space, or violates physical constraints during the evolution process, then no matter how accurate the subsequent signal injection and closed-loop interaction are, the entire test process will still be unreliable. The reproducibility of test results is the endpoint of virtual-real fusion testing. If test results under the same conditions cannot be reproduced, then the test system itself has uncertainties, and the validity of the test results cannot be guaranteed. The credibility issues of the above two stages cannot be evaluated through the "data comparison mode," and a new credibility evaluation method needs to be constructed from the perspective of functional links.

[0014] The Chinese invention patent "A Credibility Evaluation Method for a Vehicle-in-the-Loop Simulation Test System (Publication No. CN121807693A)" determines credibility metrics from multiple dimensions at the subsystem level of the vehicle-in-the-loop simulation test system. Although there are many metrics, its evaluation logic is still based on the comparison of the output data or performance parameters of each subsystem with the reference value. It belongs to the "data comparison mode" at the subsystem level and cannot cover the credibility of the functional links in the dynamic generation of scenarios and the reproduction of results.

[0015] The paper "Model-Centric Quantitative Methodology for Evaluating Simulation Reliability in Autonomous-Vehicle Testing (Lee S, Oh T, Han S, Won J, Yoo J, IEEE Access, 2025)" decomposes the simulation architecture into four types of model components: vehicle dynamics, control, environment, and perception sensors. It then performs parameter-level fidelity evaluation on each model component, comparing the output of each component with corresponding reference data. While this method refines the granularity of data comparison at the model component level, the evaluation mode remains a "data comparison mode." The reliability of dynamically generated scenes is a functional-level evaluation problem, involving the logical completeness of scene states and the coverage of the parameter space, which cannot be indirectly reflected by comparing model component outputs with reference data. Similarly, the reproducibility of test results involves the overall stability of the virtual-real fusion testing system, which also cannot be covered by data comparison at the model component level.

[0016] (3) Existing methods cannot monitor the credibility status of the test system in real time during the test run and lack online evaluation capabilities.

[0017] Virtual-real fusion testing is a dynamic process. During operation, the reliability of the testing system may degrade due to factors such as competition for computing resources, communication link fluctuations, and model parameter drift. If reliability anomalies are not detected in a timely manner during testing, it will lead to a large number of invalid test results, wasting testing time and resources. In severe cases, it may even lead to incorrect judgments on the safety of autonomous driving based on unreliable test results. Therefore, online reliability monitoring and alarm capabilities are of great significance for virtual-real fusion testing systems.

[0018] Chinese invention patent "Method, Apparatus, Storage Medium, and Electronic Device for Evaluating the Credibility of a Simulation Testing System (Publication No. CN118445182A)" obtains the credibility of the target sub-model in the simulation testing system at multiple development stages, including the conceptual design stage, software design stage, and code design stage, and calculates the target credibility value of the simulation testing system based on the credibility of each stage. This method belongs to the development process credibility evaluation, and the evaluation is completed after the simulation testing system is developed, without involving online credibility monitoring during the test operation process.

[0019] Chinese invention patents “A Credibility Evaluation Method for a Vehicle-in-the-Loop Simulation Test System Based on a Site (Publication No. CN121807693A),” “A Credibility Evaluation Method for a Vehicle-in-the-Loop Simulation Test System Based on a Site (Publication No. CN121833432A),” and “A Quantitative Evaluation Method for the Credibility of Vehicle-in-the-Loop Simulation Test (Publication No. CN120124286A)” all adopt a post-evaluation model, that is, they perform offline analysis on the collected data after the test is completed to calculate the credibility value. These methods cannot perceive changes in the credibility status of the test system in real time during the test operation.

[0020] The Chinese invention patent "An Online Assessment and Assurance Method for Confidence of Semi-Physical Simulation System Testing (Publication No. CN121742243A)" achieves online confidence assessment by calculating local confidence weakening factors through real-time recording of asynchronous abnormal events and performing a decreasing reorganization of the overall test confidence. While this method possesses online assessment capabilities, its assessment logic is based on a confidence weakening model of asynchronous abnormal events, focusing on the impact of asynchronous anomalies on confidence, rather than a systematic confidence monitoring approach for the functional links of virtual-real fusion testing. This method lacks an online monitoring indicator system covering dimensions such as scene dynamic generation, signal injection fidelity, closed-loop timing consistency, and test result reproducibility involved in autonomous driving virtual-real fusion testing, thus failing to comprehensively reflect the confidence status of each functional link of the test system. Furthermore, the method employs a decreasing reorganization strategy for handling abnormal events, focusing on confidence weakening calculations and lacking the ability to predict trends in gradual anomalies and trace the source of abrupt anomalies.

[0021] The papers “AD-VILS: Implementation and Reliability Validation of Vehicle-In-the-Loop Simulation Platform for Evaluating Autonomous Driving Systems (OhT, Ha Y, Yoo D, Yoo J, IEEE Access, 2024)” and “XILS Credibility Assessment and Scenario Representativeness Methodology Based on Geometric Similarity Analysis for Autonomous Driving Systems (Han S, Oh T, Lee S, Park S, Yoo J, Applied Sciences, 2025)” both adopt a post-test offline evaluation mode, assessing credibility through comparative analysis of data after the test is completed, and do not have online monitoring capabilities.

[0022] (4) The evaluation granularity of existing methods cannot effectively locate the specific links of the credibility problem, and there is a lack of a systematic evaluation framework for the virtual-real fusion test function link.

[0023] In the functional chain of virtual-real fusion testing—"dynamic scene generation → signal translation and injection → closed-loop timing interaction → result output reproduction"—reliability issues at each stage have different characteristics and causes. For example, problems in the scene generation stage may stem from scene logic defects or insufficient parameter coverage; problems in the signal injection stage may arise from signal translation model distortion or excessive injection latency; problems in the closed-loop interaction stage may originate from timing synchronization issues or excessive jitter; and problems in the result reproduction stage may stem from system randomness or parameter inconsistencies. An effective reliability assessment framework should be able to map assessment results to specific functional stages, providing clear guidance for the improvement and optimization of the test system.

[0024] For example, the Chinese invention patent "A Credibility Evaluation Method for a Vehicle-in-the-Loop Simulation Test System (Publication No. CN121807693A)" evaluates the system at the subsystem level, breaking it down into hardware / software subsystems such as a perception subsystem, a decision-making subsystem, and a control subsystem. While the performance indicators at each subsystem level can reflect the subsystem's performance, they do not correspond one-to-one with the functional aspects of virtual-real fusion testing, such as scene generation, signal injection, closed-loop interaction, and result reproduction. Therefore, it is impossible to directly pinpoint problems at the functional level. For instance, low signal injection fidelity may involve both the perception and communication subsystems.

[0025] The paper "Model-Centric Quantitative Methodology for Evaluating Simulation Reliability in Autonomous-Vehicle Testing (Lee S, Oh T, Han S, Won J, Yoo J, IEEE Access, 2025)" performs parameter-level evaluation at the model component level, decomposing the simulation architecture into four types of model components. Although the classification of model components is systematic, there is a many-to-many cross relationship between model components and functional elements. For example, the sensing sensor model involves both signal injection and closed-loop interaction, making it difficult to directly map the evaluation results to the functional element level, resulting in insufficient problem localization.

[0026] The paper "XILS Credibility Assessment and Scenario Representativeness Methodology Based on Geometric Similarity Analysis for Autonomous Driving Systems" (Han S, Oh T, Lee S, Park S, Yoo J, Applied Sciences, 2025) employs geometric similarity analysis to assess overall credibility through radar chart visualization and shape comparison. While this method focuses on overall geometric similarity calculation and can reflect the overall difference between XILS and real-world testing, it cannot break down these differences into specific functional components, thus limiting its ability to pinpoint problems.

[0027] In summary, existing technologies suffer from shortcomings such as bias in the evaluation object, incomplete evaluation dimensions, lack of online evaluation capabilities, and lack of a systematic evaluation framework oriented towards functional links. There is an urgent need for a credibility evaluation method for autonomous driving virtual-real fusion testing systems that covers the entire functional link and can support online monitoring and problem localization. Summary of the Invention

[0028] The purpose of this invention is to provide a method and system for evaluating the credibility of virtual-real fusion testing for autonomous driving, aiming to solve the problem that existing simulation testing credibility evaluation methods mentioned in the background art cannot effectively evaluate the credibility of the autonomous driving virtual-real fusion testing system itself.

[0029] To achieve the above objectives, the present invention provides the following technical solution: a method for evaluating the credibility of virtual-real fusion testing in autonomous driving, used to evaluate the credibility of virtual-real fusion testing implemented by an autonomous driving virtual-real fusion testing system, wherein the autonomous driving virtual-real fusion testing system includes a test server and an autonomous driving vehicle under test, the test server being directly connected to the autonomous driving domain controller of the autonomous driving vehicle under test to form a test closed loop, comprising the following steps:

[0030] S1. Construct a credibility evaluation index system for the virtual-real fusion testing system.

[0031] To address the functional characteristics of the virtual-real fusion testing system for autonomous driving, a credibility evaluation index system is constructed, comprising four primary indicators: credibility of dynamic scene generation, fidelity of signal injection, consistency of closed-loop timing, and reproducibility of test results.

[0032] The scenario dynamic generation credibility is used to evaluate the reliability of the test server's ability to dynamically generate virtual test scenarios based on test tasks. The evaluation focuses on whether the test system can correctly, completely, and consistently generate virtual test scenarios. This primary indicator includes three secondary indicators:

[0033] Scenario logical completeness: This assesses whether the virtual test scenarios generated by the test server logically cover all scenario states and legal state transition relationships required by the test task. Specifically, the virtual test scenario is parsed into a scenario state transition graph, compared with the preset test task scenario requirement model for graph isomorphism, and the scenario state coverage rate and illegal transition rate are calculated to obtain the overall scenario logical completeness index value.

[0034] The method for calculating the completeness of scene logic is as follows:

[0035] For a given test task, the virtual test scenario generated by the test server is parsed into a scenario state transition graph, where nodes in the scenario state transition graph represent key scenarios and edges represent state transition relationships.

[0036] The scenario state transition graph is compared with the preset test task scenario requirement model using graph isomorphism comparison. The scenario state coverage rate and illegal transition rate are calculated, and the scenario logic completeness index is calculated according to the following formula:

[0037]

[0038] in, The number of covered demand scenario states. This represents the total number of scenario states in the demand model. For illegally transferred amounts, This represents the total number of transitions in the scene state transition graph. and Weighting coefficients

[0039] Scene parameter coverage: This evaluates the extent to which the virtual scene parameter vectors generated by the test server in each simulation step cover the parameter space of the test task. Specifically, the parameter space of the test task is defined as a multi-dimensional parameter vector space. The parameter space is sampled hierarchically, and a dense sampling grid is set in the boundary and critical regions of this parameter space. The coarse-grained coverage, boundary region coverage, and critical region coverage are comprehensively calculated and weighted to obtain the scene parameter coverage.

[0040] The method for calculating scene parameter coverage is as follows:

[0041] The parameter space of the test task is defined as a multidimensional parameter vector space:

[0042]

[0043] in, Let be the range of values ​​for the i-th scene parameter;

[0044] The parameter space is sampled in layers, and the virtual scene parameter vectors generated by the test server in each simulation step are extracted. The ratio of the volume of the covered parameter area to the total volume of the parameter space is calculated as the coarse-grained coverage.

[0045] Furthermore, a denser sampling grid is set in the boundary and critical regions of the parameter space, and the coverage rates of the boundary and critical regions are calculated. The scene parameter coverage is then calculated using the following formula:

[0046]

[0047] in, For coarse-grained coverage, For the coverage rate of the boundary area, This represents the coverage rate of the critical area. , , These are the weighting coefficients. .

[0048] Scene evolution coherence: This evaluates whether the virtual scene states generated by the test server between consecutive simulation steps satisfy physical constraints and maintain coherence. Specifically, the virtual scene sequence generated by consecutive simulation steps is extracted, and kinematic constraint consistency and environmental element continuity are checked for adjacent scene states. The mean value of physical constraint violation is calculated to obtain the scene evolution coherence index value.

[0049] The method for calculating the continuity of scene evolution is as follows:

[0050] Extract the virtual scene sequence generated by the test server in consecutive simulation steps. The physical rationality between adjacent scene states is verified, including kinematic constraint consistency verification and environmental element continuity verification.

[0051] Calculate the physical constraint violation degree between adjacent scene states The continuity of scene evolution is calculated according to the following formula:

[0052]

[0053] in, The normalized physical constraint violation degree has a value range of 1. .

[0054] The signal injection fidelity is used to evaluate the fidelity of the test server's translation of the virtual test scenario into perception layer signals and injection into the domain controller. Its evaluation focuses on whether the virtual perception signals injected by the test system truly reflect the semantic information of the virtual scenario. This primary metric includes three secondary metrics:

[0055] Sensing signal morphological similarity: This evaluates the temporal morphological similarity between the virtual sensing signal injected into the domain controller by the test server and its corresponding reference signal. The reference signal is obtained as follows: when a real sensor measurement signal exists in the same scene, it is used as the reference signal; when no real sensor measurement signal exists, the theoretical value of the sensing signal corresponding to the virtual scene is calculated based on the semantic description and physical model of the virtual scene and used as the reference signal. After time alignment of the virtual sensing signal and the corresponding reference signal, the morphological similarity is calculated using a dynamic time warping method. When no real measurement signal is available, this metric employs a scene semantic self-consistency check logic, i.e., checking whether the "semantics of the virtual scene generated by the test system" and "its translated output sensing signal" are physically consistent. This is the core method for evaluating the reliability of the test system's own signal translation process.

[0056] The method for calculating the morphological similarity of the perceived signals is as follows:

[0057] In the same test scenario, virtual sensing signals injected into the domain controller by the test server were collected respectively. and corresponding reference signal The reference signal is obtained as follows:

[0058] When there are real sensor measured signals in the same scenario, the measured signals are used as reference signals.

[0059] When there are no real sensor measured signals, the theoretical value of the sensing signal corresponding to the virtual scene is calculated as a reference signal based on the semantic description and physical model of the virtual scene, so as to realize the scene semantic self-consistency test between the scene generation link and the signal translation link inside the test system.

[0060] After aligning the virtual sensing signal and the corresponding reference signal in time, the morphological similarity is calculated using a dynamic time warping method.

[0061]

[0062] in, L represents the dynamic time warping distance between the virtual sensing signal sequence and the reference signal sequence, and L is the length of the signal sequence.

[0063] Signal Spectrum Consistency: Evaluates the degree of consistency between the virtual sensing signal and the corresponding reference signal in the frequency domain characteristics. Fast Fourier Transform is performed on both signals, and the mean logarithmic deviation of the power spectral density ratios for each frequency band is calculated to obtain the spectrum consistency index. Signal Injection Delay: Evaluates the time delay between the test server generating the virtual scene and injecting the sensing layer signal into the domain controller. The time difference between the time the test server emits the scene signal and the time the domain controller receives the signal is recorded as the signal injection delay.

[0064] The method for calculating signal spectrum consistency is as follows:

[0065] The virtual sensing signal injected into the domain controller of the test server and the corresponding reference signal as described in claim 5 are respectively subjected to fast Fourier transform to obtain the frequency domain representation;

[0066] Calculate the power spectral density ratio of the virtual sensing signal and the corresponding reference signal in each frequency band, and calculate the spectral consistency using the following formula:

[0067]

[0068] in, and , respectively, are the power spectral densities of the virtual sensing signal and the corresponding reference signal in the j-th frequency band, and M is the total number of frequency bands.

[0069] The method for calculating signal injection delay is as follows:

[0070] Record the time when the test server sends out scene signals. The time when the domain controller receives the signal Calculate signal injection delay ;

[0071] The signal injection delay is mapped to a normalized signal injection delay metric value using the following formula:

[0072]

[0073] in, The preset signal injection delay threshold, This is the scale parameter.

[0074] When the sensing layer signal includes multiple signal types, the signal injection delay index value of each signal type is calculated separately, and the comprehensive signal injection delay index value is obtained by weighting the average of each signal type in the current test scenario.

[0075] The closed-loop timing consistency is used to evaluate the reliability of timing coordination between the domain controller's output response and the test server's generation of the next simulation step scenario within the test closed loop. Its evaluation focuses on whether the timing relationships between all links in the entire test closed loop are coordinated, stable, and synchronized, rather than the real-time performance of a single link. This primary metric includes three secondary metrics:

[0076] Closed-loop response delay: Record each key time node in the test closed loop (the time when the test server sends the scene signal, the time when the domain controller receives the signal, the time when the domain controller outputs the decision control signal, the time when the test server receives the feedback signal, and the time when the test server generates the next simulation step scene). Calculate the sum of signal injection delay, decision calculation delay, feedback transmission delay, and scene generation delay as the total closed-loop delay. Map the total closed-loop delay to the closed-loop response delay index value based on the exponential decay function.

[0077] The method for calculating the closed-loop response delay is as follows:

[0078] Record key time points in the test loop, including: the moment the test server sends a scenario signal. When the domain controller receives the signal Domain controller output decision control signal timing The moment the test server receives the feedback signal The test server generates the next simulation step scenario time. ;

[0079] Calculate signal injection delay Decision calculation delay Feedback transmission delay Scene generation delay ;

[0080] The closed-loop response delay index is calculated using the following formula:

[0081]

[0082] in, For the total closed-loop delay, For scale parameters; when hour, .

[0083] Timing jitter rate: Collect closed-loop cycle sequences of multiple consecutive simulation steps, calculate the coefficient of variation of the closed-loop cycle (the ratio of standard deviation to mean) as the timing jitter rate, and then map it to the timing consistency index value.

[0084] The timing jitter rate is calculated as follows:

[0085] During the test run, closed-loop periodic sequences of N consecutive simulation steps are collected. ,in For the first The closed-loop period of each simulation step is calculated; the mean of the closed-loop period is calculated. and standard deviation Calculate the timing jitter rate using the following formula:

[0086]

[0087] The timing consistency index corresponding to the timing jitter rate is calculated using the following formula:

[0088]

[0089] in, This is the preset jitter sensitivity coefficient.

[0090] Step size synchronization deviation: For the test server simulation clock and the domain controller running clock in the test closed loop, the clock step size sequence of both is collected, and the normalized mean of each step size deviation is calculated as the step size synchronization deviation index value.

[0091] The method for calculating the step size synchronization deviation is as follows:

[0092] For the test server simulation clock and the domain controller operating clock in the test closed loop, collect the clock step sequence of both. and ;

[0093] Calculate the step size synchronization deviation using the following formula:

[0094]

[0095] The reproducibility of the test results is used to evaluate the consistency of results obtained from running the test system multiple times under the same test conditions. Its evaluation focuses on the output stability of the test system itself. This primary indicator includes two secondary indicators:

[0096] Deviation rate of multiple runs in the same scenario: Multiple independent test runs are performed in the same test scenario and under the same initial conditions. The average trajectory of each run is used as a reference. The Hausdorff distance between each run trajectory and the average trajectory is calculated and normalized. The average value is taken as the deviation rate of multiple runs in the same scenario.

[0097] The method for calculating the deviation rate in multiple runs of the same scenario is as follows:

[0098] Under the same test scenario and initial conditions, m independent test runs were performed using the aforementioned autonomous driving virtual-real fusion test system, and the trajectory sequence of the autonomous vehicle under test was recorded in each run. ;

[0099] The average trajectory of the m runs For reference, calculate the Hausdorff distance between each run trajectory and the mean trajectory, and calculate the deviation rate of multiple runs in the same scenario using the following formula:

[0100]

[0101] in, For Hausdorf distance, is the path length of the mean trajectory.

[0102] Cross-scenario equivalence reproducibility: Select two sets of test scenarios that are semantically equivalent but have different parameter spaces, run the test in the two sets of scenarios respectively, extract the set of behavior patterns of the autonomous vehicle under test, and calculate the ratio of the intersection to the union of the two sets of behavior pattern sets as the cross-scenario equivalence reproducibility.

[0103] The method for calculating cross-scenario equivalent reproducibility is as follows:

[0104] Two test scenarios, A and B, are selected that are semantically equivalent but have different parameter spaces. Tests are run in both scenarios to obtain the response behavior sequences of the autonomous vehicle under test. and ;

[0105] Semantic behavioral patterns are extracted from the two sets of behavioral sequences. The ratio of the intersection to the union of the behavioral pattern sets is compared, and the cross-scene equivalent reproducibility is calculated using the following formula:

[0106]

[0107] in, and These are sets of behavioral patterns extracted from scenario A and scenario B, respectively.

[0108] S2. Collect multi-source data during the virtual-real fusion test run.

[0109] The multi-source data includes: virtual scene description data generated by the test server, perception layer signal data injected into the domain controller, decision control signal data output by the domain controller, and timestamp data of each stage of the test system.

[0110] S3. Based on the multi-source data, calculate the quantitative values ​​of each secondary indicator. The specific calculation methods for each secondary indicator include the calculation of scene logic completeness, scene parameter coverage, scene evolution coherence, similarity of perceived signal morphology, consistency of signal spectrum, signal injection delay, closed-loop response delay, timing jitter rate, step size synchronization deviation, deviation rate of multiple runs in the same scene, and cross-scene equivalent reproducibility.

[0111] S4. Using a combined weighting method based on entropy weight and coefficient of variation, determine the first-level weight of each secondary indicator relative to its primary indicator, and the second-level weight of each primary indicator relative to the overall credibility of the system.

[0112] The combined weighting method specifically involves the following steps: For multiple secondary indicators under the same primary indicator, based on multiple sets of evaluation sample data, the quantitative values ​​of each secondary indicator are first standardized. Then, the information entropy weight and the coefficient of variation weight are calculated separately. Finally, a combined adjustment factor is used to linearly weight the two to obtain the combined weight. The information entropy weight reflects the amount of information contained in the dispersion of the indicator data, while the coefficient of variation weight reflects the relative volatility of the indicator data. The combination of the two considers both the objective distribution characteristics of the data and the variation information of the indicators, avoiding the bias of a single weighting method.

[0113] Among them, the weighting method based on the combination of entropy weight and coefficient of variation includes:

[0114] For each of the p secondary indicators under the same primary indicator, the information entropy weights are calculated based on the Q-group evaluation sample data. and coefficient of variation weight :

[0115]

[0116]

[0117] in, Let the information entropy of the j-th indicator be , Let be the coefficient of variation of the j-th index;

[0118] Calculate the combination weights using the following formula:

[0119]

[0120] in, As a combined regulatory factor, ;

[0121] The second-level weights of the four primary indicators relative to the overall credibility of the system are determined using the same entropy weight-variance coefficient combination weighting method.

[0122] S5. Based on the quantified values ​​of each secondary indicator and the first-layer weight and the second-layer weight, calculate the overall credibility value of the autonomous driving virtual-real fusion test system through weighted aggregation, map the overall credibility value to a credibility level, and output the credibility evaluation result.

[0123] Specifically, the system first calculates the value of each primary indicator by weighting the quantified values ​​of the secondary indicators under each primary indicator and their corresponding first-level weights; then, it calculates the overall credibility value of the system by weighting the values ​​of each primary indicator and their corresponding second-level weights. The credibility levels include five levels: fully credible, highly credible, basically credible, low credible, and unreliable.

[0124] The method for calculating the overall credibility value of the system using weighted aggregation is as follows:

[0125] First, calculate the value of each primary indicator based on the quantified values ​​of the secondary indicators under each primary indicator and their corresponding first-level weights:

[0126]

[0127] in, For the first Each primary indicator value, For the first The first-level weight of the j-th secondary indicator under each primary indicator. For the corresponding quantization value, For the first The number of secondary indicators under each primary indicator.

[0128] Then, based on the values ​​of each primary indicator and the corresponding secondary weights, the overall system credibility value is calculated:

[0129]

[0130] in, The second-level weight of the k-th primary indicator;

[0131] The overall credibility value The credibility is mapped to a credibility level according to a preset credibility level mapping table. The credibility levels include five levels: fully credible, highly credible, basically credible, low credible, and untrustworthy.

[0132] Furthermore, the present invention also includes step S6, online credibility monitoring and alarm steps.

[0133] During the test run, the quantitative values ​​of each secondary and primary indicator are calculated in real time. When the value of any primary indicator is lower than the preset corresponding alarm threshold, a credibility alarm signal for the corresponding dimension is generated, and the alarm dimension and the location of the abnormal indicator are marked. When the overall credibility value of the system is lower than the preset overall alarm threshold, a test pause command is triggered, and a credibility assessment report and anomaly tracing information are output.

[0134] When a credibility alarm signal is generated, the anomaly type is determined to be either a gradual anomaly or a sudden anomaly based on the changing trend of the quantified value of the secondary indicator corresponding to the alarm dimension. For gradual anomalies, the anomaly start time and evolution process are recorded, and a degradation trend prediction is output. For sudden anomalies, the multi-source data of the most recent simulation steps are traced back to locate the specific link in which the anomaly occurred, and the anomaly source tracing result is output.

[0135] Compared with the prior art, the beneficial effects of the present invention are:

[0136] 1. This invention achieves, for the first time, a fundamental assessment of the reliability of a virtual-real fusion testing system. Existing technologies all focus on the consistency between simulation output data and real data as the core assessment objective, essentially only evaluating the reliability of simulation data. They fail to detect process-level defects in the testing system, such as systemic deviations in scene generation logic, signal injection, and implicit timing discrepancies in the closed loop. This invention explicitly defines the assessment object as the "process reliability of the testing system in executing the entire virtual-real fusion test." It verifies whether the system can reliably complete the test task from the perspective of the entire test execution process. Even if the simulation output appears consistent with the real vehicle data, it can identify potential reliability issues in the process, providing a fundamental guarantee for the validity of the virtual-real fusion test results.

[0137] 2. A four-dimensional systematic evaluation index system covering the entire functional chain has been constructed. This invention accurately corresponds to the four core functional links of virtual-real fusion testing: "dynamic scene generation → signal translation and injection → closed-loop timing interaction → result output reproduction," and constructs a complete evaluation system including 4 primary indicators and 11 secondary indicators. Among them, the reliability of dynamic scene generation solves the problem that existing methods ignore the completeness of scene logic, parameter space coverage, and physical evolution coherence; the reproducibility of test results fills the gap in existing evaluations that focus on the stability of system output. The four dimensions form a closed loop, and the reliability defects of any link can be independently identified, completely solving the problem of the one-sided evaluation dimensions of the traditional "data comparison mode."

[0138] 3. This invention overcomes the bottleneck in evaluating signal injection fidelity in extreme scenarios. It proposes a signal injection evaluation method based on scene semantic self-consistency. In extremely dangerous and rare traffic scenarios where real sensor data is unavailable, the method verifies the internal self-consistency of signal injection fidelity by examining the physical consistency between the "virtual scene semantics generated by the test system" and the "translated and output perception signal." This method eliminates the dependence on external real data, enabling credibility evaluation to cover all test scenarios. It is particularly suitable for high-risk scenarios where credibility verification is most needed but reference data cannot be obtained through real vehicle testing, significantly improving the applicability of the evaluation method.

[0139] 4. Possessing real-time online monitoring and intelligent anomaly handling capabilities, this invention supports real-time reliability calculation and status monitoring during test execution. When the indicator value falls below the alarm threshold, an alarm signal can be generated promptly to prevent invalid test results from continuing after system reliability degrades. Simultaneously, it innovatively categorizes anomalies into two types: gradual and abrupt. For gradual anomalies, it provides degradation trend prediction, reserving an intervention window for testers; for abrupt anomalies, it achieves precise source tracing and localization by tracing back multi-source data. This mechanism significantly improves testing efficiency and reduces the waste of time and resources caused by test system anomalies.

[0140] 5. This invention enables precise identification and targeted improvement of credibility issues. The indicator system of this invention corresponds one-to-one with the functional links of the testing system, and the evaluation results can be directly mapped to specific functional stages. When a certain dimension indicator is low, the problem type can be further refined through secondary indicators. For example, low credibility in dynamic scenario generation can be precisely identified as incomplete logic, insufficient parameter coverage, or inconsistent evolution. This hierarchical positioning capability allows the evaluation results to not only answer the question "Is the system trustworthy?", but also clearly answer "Where is it untrustworthy?" and "Why is it untrustworthy?", providing clear and actionable guidance for the optimization and improvement of the testing system. Attached Figure Description

[0141] Figure 1 A flowchart of the autonomous driving virtual-real fusion test credibility evaluation method provided in the embodiments of the present invention;

[0142] Figure 2 This is a schematic diagram of the architecture of the autonomous driving virtual-real fusion testing system provided in an embodiment of the present invention;

[0143] Figure 3 A diagram illustrating the system architecture of the credibility assessment metrics provided in this embodiment of the invention;

[0144] Figure 4 This is a schematic diagram of the test closed-loop timing relationship provided in an embodiment of the present invention;

[0145] Figure 5 This is a flowchart of online reliability monitoring and alarm provided in an embodiment of the present invention;

[0146] Figure 6 This is a structural block diagram of the autonomous driving virtual-real fusion test credibility evaluation system provided in an embodiment of the present invention. Detailed Implementation

[0147] The present invention will be further described below with reference to the embodiments.

[0148] Example 1

[0149] refer to Figure 1This invention provides a method for evaluating the credibility of virtual-real fusion testing in autonomous driving, including the following steps S1 to S6.

[0150] refer to Figure 2 The architecture of the autonomous driving virtual-real fusion testing system addressed in this invention is as follows: The system includes a test server and an autonomous vehicle under test. The autonomous vehicle under test is deployed in a closed test track or controlled road environment, and is equipped with an autonomous driving domain controller and a vehicle operation status monitoring module. The test server and the autonomous driving domain controller of the autonomous vehicle under test are directly connected via wired or wireless communication links to form a test closed loop. During the test operation, the test server dynamically simulates and generates virtual test scenarios according to the test tasks and test requirements, and translates the virtual test scenarios into perception layer signals (including but not limited to camera image signals, LiDAR point cloud signals, millimeter-wave radar signals, positioning signals, etc.) through a model, and transmits them to the domain controller. The autonomous vehicle under test implements autonomous driving functions based on the received perception layer signals and outputs decision control signals. The test server judges whether the functional performance of the autonomous vehicle under test meets the test expectations based on the data transmitted by the vehicle operation status monitoring module, and generates the test scenario for the next simulation step in real time, thereby realizing the closed-loop test of virtual-real fusion.

[0151] S1. Construct a credibility evaluation index system for virtual-real fusion testing systems.

[0152] refer to Figure 3 In view of the functional link characteristics of the autonomous driving virtual-real fusion test system, a credibility evaluation index system is constructed, which includes four primary indicators: credibility of scene dynamic generation, fidelity of signal injection, consistency of closed-loop timing, and reproducibility of test results.

[0153] The four primary indicators correspond to the four key functional aspects of virtual-real fusion testing:

[0154]

[0155] The secondary indicators under each primary indicator are as follows:

[0156] (1) Credibility of dynamic scene generation, including three secondary indicators:

[0157] Scenario logic completeness: Evaluate whether the virtual test scenarios generated by the test server logically cover all scenario states and legal state transition relationships required by the test task.

[0158] Scene parameter coverage: Evaluate the extent to which the virtual scene parameter vectors generated by the test server in each simulation step cover the test task parameter space.

[0159] Scene evolution coherence: Evaluate whether the virtual scene states generated by the test server between consecutive simulation steps meet physical constraints and remain consistent.

[0160] (2) Signal injection fidelity, including three secondary indicators:

[0161] Sensing signal morphology similarity: Evaluate the similarity in the temporal domain morphology between the virtual sensing signal injected into the domain controller by the test server and the reference signal.

[0162] Signal spectrum consistency: Evaluate the degree of consistency between the virtual sensing signal and the reference signal in the frequency domain characteristics.

[0163] Signal injection latency: Evaluate the time delay between the test server generating the virtual scene and injecting the perception layer signal into the domain controller.

[0164] (3) Closed-loop timing consistency, including three secondary indicators:

[0165] Closed-loop response latency: Evaluate the total latency from when the test server sends a scenario signal to when the next simulation step scenario is generated in the test closed loop.

[0166] Timing jitter rate: Evaluates the stability of the closed-loop cycle of each simulation step in the test closed loop.

[0167] Step size synchronization deviation: Evaluate the degree of step size synchronization between the test server's simulated clock and the domain controller's running clock.

[0168] (4) Reproducibility of test results, including two secondary indicators:

[0169] Deviation rate of multiple runs in the same scenario: Evaluate the deviation of results from multiple independent test runs under the same test scenario and the same initial conditions.

[0170] Cross-scenario equivalence reproducibility: Evaluates the degree of equivalence reproducibility of test results in test scenarios that are semantically equivalent but have different parameter spaces.

[0171] S2. Collect multi-source data during the virtual-real fusion test run.

[0172] During the test run, the following four types of multi-source data were collected simultaneously:

[0173] (a) Test the virtual scene description data generated by the server: including scene state descriptions for each simulation step, scene parameter vectors, scene state transition relationships, etc.;

[0174] (b) Sensing layer signal data injected into the domain controller: including camera image signals, lidar point cloud signals, millimeter-wave radar signals, positioning signals, etc., translated and generated by the test server;

[0175] (c) Decision control signal data output by the domain controller: including control commands such as steering wheel angle, acceleration, and braking pressure;

[0176] (d) Timestamp data of each stage of the test system: including the time when the test server sends out the scene signal, the time when the domain controller receives the signal, the time when the domain controller outputs the decision control signal, the time when the test server receives the feedback signal, and the time when the test server generates the next simulation step scene, etc.

[0177] S3. Based on multi-source data, calculate the quantitative values ​​of each secondary indicator.

[0178] The specific calculation methods for each secondary indicator are as follows:

[0179] 3.1 Completeness of Scene Logic Calculation

[0180] For a given test task, the virtual test scenario generated by the test server is parsed into a scenario state transition diagram. The nodes in the node set V represent key states of the scene, and the edges in the edge set E represent state transition relationships.

[0181] The scenario state transition graph G will be compared with the preset test task scenario requirement model. Perform graph isomorphism comparison, where This refers to the total set of scenario states in the demand model. This is the set of legal state transition relationships in the demand model.

[0182] Count the number of covered demand scenario states: That is, the scenario state transition graph G and the demand model The number of key states in the matched scenario.

[0183] Total number of scenario states in the statistical requirements model: .

[0184] The number of illegal transitions in the state transition graph G of the statistical scenario: That is, G does not belong to the demand model. The number of legal transfer relationships.

[0185] The total number of transitions in the state transition graph G of the statistical scenario: .

[0186] Calculate scene state coverage and illegal transfer rate :

[0187]

[0188]

[0189] The logic completeness of the scenario is calculated using the following formula:

[0190]

[0191] in, and For the weighting coefficients, satisfying and In this embodiment, we take... , .

[0192] when At that time, take ; At that time, take .

[0193] Specific Implementation Example: Suppose the test task requires verifying the three behaviors of an autonomous vehicle at an intersection: left turn, straight ahead, and right turn. The requirement model includes... The virtual scene generated by the test server, after being parsed into a scene state transition graph, contains 11 key scene states (of which...). (matching the demand model) and State transition relations (where (This is an illegal transfer). Therefore:

[0194]

[0195]

[0196]

[0197] 3.2 Scene Parameter Coverage Calculation

[0198] The parameter space of the test task is defined as a k-dimensional parameter vector space:

[0199]

[0200] in The range of values ​​for the i-th scene parameter (such as vehicle speed, distance to the vehicle in front, road curvature, weather conditions, etc.);

[0201] The parameter space is hierarchically sampled, with each parameter dimension uniformly divided into n intervals, forming... Each grid cell is counted. The virtual scene parameter vectors generated by the test server in each simulation step are extracted, and the number of covered grid cells is counted.

[0202] Calculate coarse-grained coverage:

[0203]

[0204] in, The number of grid cells that fall into at least one virtual scene parameter vector. This represents the total number of grid cells.

[0205] Furthermore, a dense sampling grid is set in the boundary region of the parameter space (the upper and lower 10% range of each parameter dimension's value range) and the critical region (the neighborhood of the critical parameter value preset according to the test task). The grid density is m times that of the regular grid (m=3 in this embodiment), and the coverage rate of the boundary region is calculated respectively. and critical area coverage .

[0206] Calculate scene parameter coverage using the following formula:

[0207]

[0208] in, , , These are the weighting coefficients. , , This is to reflect the emphasis on overall coverage of the parameter space, while also taking into account the importance of boundaries and critical regions.

[0209] Specific Implementation Example: Assume the test task involves three scene parameters (vehicle speed 20~120km / h, distance to the vehicle in front 5~150m, road curvature 0~0.01m⁻¹), each parameter is divided into 10 intervals, totaling 1000 grid cells. The virtual scene parameter vector generated by the test server covers 620 grid cells. The boundary region contains 200 encrypted grid cells, covering 110 of them. The critical region contains 150 encrypted grid cells, covering 105 of them. .

[0210]

[0211] 3.3 Scene Evolution Coherence Calculation

[0212] Extract the virtual scene sequence generated by the test server in consecutive simulation steps. For adjacent scene states and The physical plausibility between them is verified, including:

[0213] (a) Kinematic constraint consistency check: Check whether the changes in position, velocity, and acceleration of the vehicle and traffic participants between adjacent scene states satisfy the kinematic constraints (such as maximum acceleration constraints, maximum steering angle rate constraints, etc.).

[0214] (b) Environmental element continuity verification: Check whether static environmental elements such as road geometry, traffic signs, and traffic light status remain consistent or change according to reasonable rules between adjacent scene states.

[0215] For each adjacent scene state pair Calculate the degree of violation of physical constraints ;

[0216]

[0217] in, For the degree of violation of kinematic constraints, The degree of violation of environmental continuity is normalized to interval; and As the weighting coefficient, this embodiment takes... , .

[0218] The coherence of scene evolution is calculated using the following formula:

[0219]

[0220] Specific Implementation Example: Assuming that in 100 consecutive simulation steps, 98 adjacent scene state pairs satisfy physical constraints, and 2 adjacent scene state pairs violate kinematic constraints (violation degrees of 0.3 and 0.5, respectively), then:

[0221]

[0222] 3.4 Sensitive signal morphological similarity Calculation

[0223] In the same test scenario, the virtual sensing signal and the corresponding reference signal injected into the domain controller by the test server were collected respectively.

[0224] The reference signal is obtained in the following way:

[0225] When there are real sensor measured signals in the same scenario, the measured signals are used as reference signals.

[0226] When real sensor measurements are unavailable, theoretical values ​​of the perceived signals corresponding to the virtual scene are calculated as reference signals based on the semantic description and physical model of the virtual scene. Specifically, based on semantic information defined in the virtual scene, such as vehicle pose, traffic participant position / speed / category, road geometry parameters, and lighting / weather conditions, the theoretical signal values ​​that each sensor should output in the virtual scene are calculated using the corresponding sensor physical model (e.g., image generation from a pinhole camera model, LiDAR point cloud generation from a ray tracing model, etc.).

[0227] Virtual sensing signals and corresponding reference signal After time alignment, the morphological similarity is calculated using the Dynamic Time Warping (DTW) method:

[0228]

[0229] in, The dynamic time warping distance between the virtual sensing signal sequence and the reference signal sequence is L, where L is the length of the signal sequence. The range of values ​​is The closer the value is to 1, the higher the morphological similarity.

[0230] Specific Implementation Example: Assuming a scenario where a vehicle cuts into the scene ahead, the test server injects a virtual camera image signal sequence of length L = 200 frames, and the calculated DTW distance is 15.6, then:

[0231]

[0232] 3.5 Signal Spectrum Consistency Calculation

[0233] The virtual sensing signal injected into the domain controller of the test server and the reference signal as described in claim 5 are respectively subjected to Fast Fourier Transform (FFT) to obtain the frequency domain representation.

[0234] The signal frequency band is divided into M frequency bands. The power spectral density ratio of the virtual sensing signal to the reference signal in each frequency band is calculated, and the spectral consistency is calculated using the following formula:

[0235]

[0236] in, and denoted as and respectively, are the power spectral densities of the virtual sensing signal and the reference signal in the j-th frequency band. The range of values ​​is The closer the value is to 1, the higher the spectral consistency. At that time, take .

[0237] Specific implementation example: Assume the signal frequency band is divided into M=8 frequency bands, and each frequency band... Given values ​​of 0.02, 0.05, 0.03, 0.08, 0.04, 0.06, 0.03, and 0.07 respectively, then:

[0238]

[0239] 3.6 Signal Injection Delay Calculation

[0240] Record the time when the test server sends out scene signals. The time when the domain controller receives the signal Calculate signal injection delay ;

[0241] The signal injection delay is mapped to a normalized signal injection delay metric value using the following formula:

[0242]

[0243] in, The preset signal injection delay threshold, This is a scale parameter. In this embodiment, for the camera image signal, we take... , For lidar point cloud signals, take , ;

[0244] For the injection delay of multiple sensing signals, take the signal type... The weighted average value is used as the comprehensive signal injection delay index, and the weights are determined according to the importance of each signal type in the current test scenario.

[0245] Specific Implementation Example: Assuming the camera image signal injection delay is 28ms, then:

[0246]

[0247] If the injection delay of the lidar point cloud signal is 45ms, then:

[0248] (45ms≤50ms threshold)

[0249] If the camera weight is 0.6 and the LiDAR weight is 0.4, the overall result is:

[0250]

[0251] 3.7 Closed-loop response delay Calculation

[0252] refer to Figure 4 Record the key time points in the test loop:

[0253] : The moment when the test server sends out the scene signal;

[0254] The moment the domain controller receives the signal;

[0255] The time when the domain controller outputs the decision control signal;

[0256] : The moment when the test server receives the feedback signal;

[0257] The test server generates the next simulation step scenario time.

[0258] Calculate the delay of each stage:

[0259] Forward injection delay ;

[0260] Decision computation delay ;

[0261] Feedback transmission delay ;

[0262] Scene generation delay .

[0263] Total closed-loop delay: , For scale parameters; in this embodiment, we take... (Corresponding to a simulated step frequency of 10Hz) .

[0264] Specific implementation example: Assume that a certain simulation step measures... , , , ,but .

[0265]

[0266] 3.9 Timing jitter rate Calculation

[0267] During the test run, closed-loop periodic sequences of N consecutive simulation steps were collected. ,in For the first The closed-loop period of each simulation step (i.e., the time interval from when the test server sends out the scene signal in step i to when the test server sends out the scene signal in step i+1).

[0268] Calculate the mean and standard deviation of the closed-loop period.

[0269]

[0270]

[0271] Calculate timing jitter rate:

[0272] Corresponding time series consistency index values:

[0273] in, This is a preset jitter sensitivity coefficient. In this embodiment, it is taken as... When the timing jitter rate reaches 20% It dropped to 0.

[0274] Specific implementation example: Assume that the closed-loop period of 100 simulation steps is collected, and the mean is... Standard deviation ,but:

[0275]

[0276]

[0277] 3.9 Step Size Synchronization Deviation Calculation

[0278] For the test server simulation clock and the domain controller operating clock in the test closed loop, collect the clock step sequence of both. and ,in Let i be the simulation step size of the test server in the i-th simulation step. Let be the running step size of the i-th simulation step domain controller.

[0279] Calculate the step size synchronization deviation using the following formula:

[0280]

[0281] The range of values ​​is The closer the value is to 1, the higher the degree of step size synchronization.

[0282] Specific Implementation Example: Assume the simulation step size of the test server is set to 100ms, and the domain controller's running step size is set to 100ms. Data from 100 simulation steps is collected, and the average step size deviation is measured to be 2.5ms. Then:

[0283]

[0284] 3.10 Deviation rate during multiple runs in the same scenario Calculation

[0285] Under the same test scenario and initial conditions, m independent test runs were performed using the aforementioned autonomous driving virtual-real fusion test system, and the trajectory sequence of the autonomous vehicle under test was recorded in each run.

[0286] Calculate the mean locus:

[0287]

[0288] Calculate the trajectory of j runs with mean locus The distance between Hausdorf :

[0289]

[0290] in, denoted as Euclidean distance between trajectory points.

[0291] Calculate the deviation rate of multiple runs in the same scenario using the following formula:

[0292]

[0293] in, is the path length of the mean trajectory.

[0294] The range of values ​​is The closer the value is to 0, the higher the consistency of multiple runs in the same scenario.

[0295] Corresponding reproducibility index values:

[0296] Where μ is a preset scaling factor. In this embodiment, μ = 10.

[0297] Specific implementation example: Assuming five independent test runs are conducted under the same emergency braking scenario of the preceding vehicle, the average trajectory path length is... The Hausdorff distances between each trajectory and the mean trajectory are 1.2m, 0.8m, 1.5m, 0.6m, and 1.0m, respectively. Then:

[0298] =0.0051

[0299] Corresponding reproducibility index values:

[0300] 3.11 Cross-scenario equivalent reproducibility Calculation

[0301] Two sets of test scenarios, A and B, are selected that are semantically equivalent but have different parameter spaces. "Semantic equivalence" means that the two sets of scenarios describe the same test task at the functional semantic level (e.g., both test the "emergency braking of the vehicle in front" function), but the specific parameters are different (e.g., in scenario A, the vehicle speed is 60 km / h and the vehicle in front decelerates by 5 m / s², while in scenario B, the vehicle speed is 80 km / h and the vehicle in front decelerates by 6 m / s²).

[0302] Tests were run in two separate scenarios to obtain the response behavior sequences of the autonomous vehicle under test. and .

[0303] Semantic-level behavioral pattern extraction is performed on the two sets of behavioral sequences, dividing the continuous behavioral sequences into discrete behavioral pattern segments (such as "normal following", "deceleration response", "emergency braking", "lane change and avoidance"), resulting in a behavioral pattern set. and

[0304] Calculate cross-scenario equivalence reproducibility using the following formula:

[0305]

[0306] Specific Implementation Example: Assume the set of behavioral patterns extracted in scenario A is as follows: The set of behavioral patterns extracted in scenario B is ,but:

[0307] , .

[0308] ,

[0309] .

[0310]

[0311] S4. Determine the weights using a combined weighting method based on entropy weighting and coefficient of variation.

[0312] For p secondary indicators under the same primary indicator, the information entropy weight and the coefficient of variation weight are calculated based on the Q-group evaluation sample data (i.e., the quantitative values ​​of each secondary indicator in the Q-group collected under different test tasks and different test scenarios).

[0313] Step 4.1: Data Standardization

[0314] The quantified value of the j-th secondary indicator in the q-th evaluation sample. Standardization process:

[0315] For positive indicators (the higher the value, the better):

[0316]

[0317] For negative indicators (the smaller the value, the better):

[0318]

[0319] Step 4.2: Calculate the information entropy weight

[0320] Calculate the information entropy of the j-th indicator:

[0321]

[0322] in, .

[0323] Calculate information entropy weights:

[0324]

[0325] Step 4.3: Calculate the weights of the coefficient of variation

[0326] Calculate the coefficient of variation for the j-th indicator:

[0327]

[0328] in, and denoted as the standard deviation and mean of the quantified value of the j-th indicator, respectively.

[0329] Calculate the weights of the coefficient of variation:

[0330]

[0331] Step 4.4: Calculate the combined weights

[0332]

[0333] in, As a combined regulatory factor, In this embodiment, we take That is, a weighted combination of information entropy weight and coefficient of variation weight.

[0334] For the second-level weights of the four primary indicators relative to the overall credibility of the system, the same method as above is used to calculate the weights based on the data of each primary indicator value in different evaluation samples.

[0335] Specific Implementation Example: Assuming that the data for the three secondary indicators (scene logical completeness, scene parameter coverage, and scene evolution coherence) under the scene dynamic generation credibility are calculated in 10 evaluation samples, the information entropy weights are 0.35, 0.40, and 0.25 respectively, and the coefficient of variation weights are 0.30, 0.45, and 0.25 respectively. Then the combined weights are:

[0336]

[0337]

[0338]

[0339] S5. Calculate the overall credibility value of the system using weighted aggregation and map it to the credibility level.

[0340] Step 5.1: Calculate the values ​​of each primary indicator.

[0341] Calculate the value of each primary indicator based on the quantified value of the secondary indicators under each primary indicator and the corresponding first-level weights:

[0342]

[0343] in, For the first One primary indicator value ( =1, 2, 3, 4 correspond to the reliability of dynamic scene generation, the fidelity of signal injection, the consistency of closed-loop timing, and the reproducibility of test results, respectively. For the first The first-level weight of the j-th secondary indicator under each primary indicator. For the corresponding quantization value, For the first The number of secondary indicators under each primary indicator.

[0344] Step 5.2: Calculate the overall system credibility value

[0345] Calculate the overall system credibility value based on the values ​​of each primary indicator and the corresponding secondary weights:

[0346]

[0347] in, For the first The second layer of weights for primary indicators.

[0348] Step 5.3: Credibility Level Mapping

[0349] Overall credibility value Mapped to a confidence level according to a pre-defined confidence level mapping table:

[0350]

[0351] Specific Implementation Example: Assuming that the calculated values ​​of the four primary indicators are as follows:

[0352] Dynamically generated credibility of the scene ;

[0353] Signal injection fidelity ;

[0354] Closed-loop timing consistency ;

[0355] Test result reproducibility .

[0356] The weights for the second layer are as follows: , , , ,but:

[0357]

[0358] Mapped to a "highly trustworthy" level.

[0359] S6, Online Reliability Monitoring and Alarms

[0360] refer to Figure 5 During the test run, the quantitative values ​​of each secondary and primary indicator are calculated in real time to conduct online credibility monitoring and alarms.

[0361] Step 6.1: Real-time indicator calculation

[0362] In each simulation step of the test run or every preset monitoring cycle (every 10 simulation steps in this embodiment), the quantitative values ​​of each secondary and primary indicator are calculated in real time based on the most recently collected multi-source data.

[0363] Step 6.2: Primary indicator alert

[0364] When any level indicator value is lower than the preset corresponding alarm threshold, a confidence alarm signal for the corresponding dimension is generated, and the alarm dimension and abnormal indicator location are marked.

[0365] In this embodiment, the alarm thresholds for each primary indicator are set as follows:

[0366]

[0367] Step 6.3: Overall Alarms and Test Pause

[0368] When the overall system credibility value is lower than the preset overall alarm threshold (0.50 in this embodiment), a test pause command is triggered, and a credibility assessment report and anomaly tracing information are output.

[0369] Step 6.4: Exception Type Determination

[0370] When a credibility alarm signal is generated, the anomaly type is determined based on the changing trend of the quantified values ​​of the secondary indicators corresponding to the alarm dimension:

[0371] Gradual anomaly: When the quantified value of the secondary indicator shows a monotonically decreasing trend within a continuous monitoring period, and the decrease in adjacent monitoring periods does not exceed a preset mutation threshold (0.15 in this embodiment), it is determined to be a gradual anomaly. For gradual anomalies, the onset time and evolution process of the anomaly are recorded, and a degradation trend curve is fitted based on the indicator values ​​of the most recent multiple monitoring periods to output a degradation trend prediction.

[0372] Sudden Anomaly: When the decrease in the quantitative value of a secondary indicator exceeds a preset abrupt change threshold within a single monitoring period, it is determined to be a sudden anomaly. For sudden anomalies, the most recent... One simulation step (in this embodiment, we take...) The system analyzes multi-source data, including scene generation data, signal injection data, and time-series data, step by step, to pinpoint the specific stage where the anomaly occurred (scene generation stage / signal injection stage / closed-loop interaction stage), and outputs the anomaly tracing results.

[0373] Specific Implementation Example: Suppose that during the test run, the signal injection fidelity was monitored to gradually decrease from 0.82 to 0.78, 0.73, 0.68, and 0.62, falling below the alarm threshold of 0.65. Analysis showed that the perceived signal morphology similarity index decreased from 0.90 to 0.70, with the decrease not exceeding 0.15 in adjacent periods, indicating a gradual anomaly. The system records the start time of the anomaly, fits a degradation trend curve, predicts that the signal injection fidelity will drop to 0.57 in the next monitoring period, and outputs a degradation trend warning.

[0374] Suppose that in another test run, the closed-loop timing consistency suddenly drops from 0.85 to 0.40, with a single-cycle decrease of 0.45, exceeding the mutation threshold of 0.15, and is thus identified as a mutation-type anomaly. The system backtracks the multi-source data from the last 50 simulation steps and finds that the... The initial closed-loop response delay suddenly increased from 95ms to 210ms. Further analysis revealed that the feedback transmission delay suddenly increased from 15ms to 130ms. The abnormal link was located as a feedback communication link failure, and the abnormality tracing results were output.

[0375] Example 2

[0376] refer to Figure 6 This invention provides a credibility evaluation system for autonomous driving virtual-real fusion testing, comprising:

[0377] The indicator system construction module 601 is used to construct a credibility evaluation indicator system that includes four primary indicators and corresponding secondary indicators: scene dynamic generation credibility, signal injection fidelity, closed-loop timing consistency, and test result reproducibility.

[0378] The multi-source data acquisition module 602 is used to acquire virtual scene description data generated by the test server, perception layer signal data injected into the domain controller, decision control signal data output by the domain controller, and timestamp data of each stage of the test system.

[0379] The indicator quantification calculation module 603 is used to calculate the quantified values ​​of each secondary indicator based on the multi-source data. This module includes a scene logic completeness calculation unit, a scene parameter coverage calculation unit, a scene evolution coherence calculation unit, a perception signal morphology similarity calculation unit, a signal spectrum consistency calculation unit, a signal injection delay calculation unit, a closed-loop response delay calculation unit, a timing jitter rate calculation unit, a step size synchronization deviation calculation unit, a deviation rate calculation unit for multiple runs in the same scene, and a cross-scene equivalent reproducibility calculation unit.

[0380] The combined weighting module 604 is used to determine the first-level weight of each secondary indicator relative to its primary indicator and the second-level weight of each primary indicator relative to the overall credibility of the system by using a combined weighting method based on entropy weight and coefficient of variation.

[0381] The credibility aggregation output module 605 is used to calculate the overall credibility value of the autonomous driving virtual-real fusion test system based on the quantified values ​​of each secondary indicator and the first-layer weight and the second-layer weight through weighted aggregation, and to map the overall credibility value to a credibility level and output the credibility evaluation result.

[0382] The online monitoring and alarm module 606 is used to calculate the quantitative values ​​of various indicators in real time during the test run. When any level indicator value or the overall system credibility value is lower than the corresponding alarm threshold, a credibility alarm signal is generated or a test pause command is triggered. This module also includes an anomaly type judgment unit to distinguish between gradual anomalies and sudden anomalies, and output degradation trend prediction or anomaly tracing results respectively.

Claims

1. A method for evaluating the credibility of virtual-real fusion testing in autonomous driving, characterized in that, The autonomous driving virtual-real fusion testing system includes a test server and an autonomous vehicle under test. The test server is directly connected to the autonomous driving domain controller of the autonomous vehicle under test to form a test closed loop. The method includes the following steps: S1. Construct a credibility evaluation index system for the virtual-real fusion test system. The index system includes four primary indicators: scene dynamic generation credibility, signal injection fidelity, closed-loop timing consistency, and test result reproducibility. The scene dynamic generation credibility includes three secondary indicators: scene logic completeness, scene parameter coverage, and scene evolution coherence. The signal injection fidelity includes three secondary indicators: perceived signal morphology similarity, signal spectrum consistency, and signal injection delay. The closed-loop timing consistency includes three secondary indicators: closed-loop response delay, timing jitter rate, and step size synchronization deviation. The test result reproducibility includes two secondary indicators: deviation rate of multiple runs in the same scene and cross-scene equivalent reproducibility. S2. Collect multi-source data during the virtual-real fusion test operation, including virtual scene description data generated by the test server, perception layer signal data injected into the autonomous driving domain controller, decision control signal data output by the autonomous driving domain controller, and timestamp data of each stage of the virtual-real fusion test system. S3. Based on the multi-source data, calculate the quantitative value of each secondary indicator respectively; S4. Using the combined weighting method based on entropy weight and coefficient of variation, determine the first-level weight of each secondary indicator relative to its primary indicator, and the second-level weight of each primary indicator relative to the overall credibility of the system. S5. Based on the quantified values ​​of each secondary indicator and the first-layer weight and the second-layer weight, calculate the overall credibility value of the autonomous driving virtual-real fusion test system through weighted aggregation, map the overall credibility value to a credibility level, and output the credibility evaluation result.

2. The method for evaluating the credibility of virtual-real fusion testing in autonomous driving according to claim 1, characterized in that, The method for calculating the secondary index of the credibility of the scene dynamic generation in step S3 is as follows: The scenario logic completeness is as follows: the virtual test scenario generated by the test server is parsed into a scenario state transition graph, compared with the preset test task scenario requirement model, the scenario state coverage and illegal transition rate are calculated, and the scenario logic completeness is obtained by weighting. The scene parameter coverage is as follows: the parameter space of the test task is defined as a multi-dimensional parameter vector space, the parameter space is sampled in layers and a dense sampling grid is set in the boundary region and critical region, the coarse-grained coverage, the boundary region coverage and the critical region coverage are calculated, and the scene parameter coverage is obtained by weighting. The scenario evolution coherence is obtained by extracting the virtual scenario sequence of continuous simulation steps, verifying the kinematic constraint consistency and environmental element continuity of adjacent scenario states, calculating the normalized physical constraint violation degree, and obtaining the scenario evolution coherence.

3. The method for evaluating the credibility of virtual-real fusion testing in autonomous driving according to claim 1, characterized in that, The method for calculating the morphological similarity of the sensing signal in step S3 is as follows: Under the same test scenario, virtual sensing signals and corresponding reference signals are collected respectively. When there is a real sensor measured signal under the same scenario, the measured signal is used as the reference signal. When there is no real sensor measured signal, the theoretical value of the sensing signal is calculated based on the semantic description and physical model of the virtual scenario as the reference signal. This realizes the scene semantic self-consistency test between the scene generation stage and the signal translation stage within the test system. After aligning the virtual sensing signal and the reference signal in time, the morphological similarity is calculated using the dynamic time warping method.

4. The method for evaluating the credibility of virtual-real fusion testing in autonomous driving according to claim 1, characterized in that, The calculation method for the secondary index of closed-loop timing consistency in step S3 is as follows: The closed-loop response delay is calculated by recording the times when the test server sends a scene signal, the domain controller receives a signal, the domain controller outputs a decision control signal, the test server receives a feedback signal, and the test server generates the next simulation step scene. The sum of signal injection delay, decision calculation delay, feedback transmission delay, and scene generation delay is then mapped to obtain the closed-loop response delay index. The timing jitter rate is obtained by collecting the closed-loop cycle sequence of N consecutive simulation steps, calculating the mean and standard deviation of the closed-loop cycle, and mapping it to a timing consistency index. The step size synchronization deviation is obtained by collecting the step size sequences of the test server's simulation clock and the domain controller's running clock, calculating the normalized mean of each step size deviation, and obtaining the step size synchronization deviation.

5. The method for evaluating the credibility of virtual-real fusion testing in autonomous driving according to claim 1, characterized in that, The calculation method for the secondary index of test result reproducibility in step S3 is as follows: The deviation rate of multiple runs in the same scenario is obtained by performing m independent tests in the same test scenario and under the same initial conditions, recording the vehicle trajectory sequence of each run, calculating the Hausdorf distance of each trajectory with the mean trajectory as a reference, and normalizing it. The cross-scenario equivalence reproducibility is obtained by selecting two sets of test scenarios that are semantically equivalent but have different parameter spaces, extracting the behavior pattern sets of the tested vehicles respectively, and calculating the ratio of the intersection to the union of the two sets.

6. The method for evaluating the credibility of virtual-real fusion testing in autonomous driving according to claim 1, characterized in that, The weighting method based on entropy weight and coefficient of variation described in step S4 is as follows: For p secondary indicators under the same primary indicator, based on Q group evaluation sample data, the quantitative values ​​of the indicators are first standardized, and then the information entropy weight and coefficient of variation weight are calculated respectively. The combined weight is obtained by linearly weighting the two by combining adjustment factors. The second-level weights of the four primary indicators relative to the overall credibility of the system are determined using the same method.

7. The method for evaluating the credibility of virtual-real fusion testing in autonomous driving according to claim 1, characterized in that, The weighted aggregation calculation in step S5 is as follows: First, the value of each primary indicator is calculated by weighting the quantified value of the secondary indicator under each primary indicator and the corresponding first-level weight; then, the overall credibility value of the system is calculated by weighting the value of each primary indicator and the corresponding second-level weight; the credibility level is divided into five levels: fully credible, highly credible, basically credible, low credible and untrustworthy.

8. A method for evaluating the credibility of virtual-real fusion testing in autonomous driving according to any one of claims 1-7, characterized in that, The method further includes step S6: During the test run, the quantitative values ​​of each secondary indicator and primary indicator are calculated in real time. When the value of any primary indicator is lower than the preset corresponding alarm threshold, a credibility alarm signal of the corresponding dimension is generated, and the alarm dimension and abnormal indicator position are marked. When the overall credibility value of the system is lower than the preset overall alarm threshold, a test pause command is triggered, and a credibility assessment report and anomaly tracing information are output.

9. The method for evaluating the credibility of virtual-real fusion testing in autonomous driving according to claim 8, characterized in that, Step S6 also includes: When a credibility alarm signal is generated, the anomaly type is determined to be either a gradual anomaly or a sudden anomaly based on the changing trend of the quantified value of the secondary indicator corresponding to the alarm dimension. For gradual anomalies, the system records the anomaly's onset time and evolution process, and outputs a prediction of the degradation trend. For sudden anomalies, the system backtracks the multi-source data from the most recent simulation steps to locate the specific functional link where the anomaly occurred, and outputs the anomaly tracing results.

10. A credibility evaluation system for autonomous driving virtual-real fusion testing, characterized in that, The system is used to perform the autonomous driving virtual-real fusion test credibility evaluation method according to any one of claims 1 to 9, the system comprising: The indicator system construction module is used to construct a credibility evaluation indicator system that includes four primary indicators: scene dynamic generation credibility, signal injection fidelity, closed-loop timing consistency, and test result reproducibility, as well as corresponding secondary indicators. The multi-source data acquisition module is used to acquire virtual scene description data generated by the test server, perception layer signal data injected into the domain controller, decision control signal data output by the domain controller, and timestamp data of each stage of the test system. The indicator quantification calculation module is used to calculate the quantified value of each secondary indicator based on the multi-source data. The combined weighting module is used to determine the first-level weight of each secondary indicator relative to its primary indicator, and the second-level weight of each primary indicator relative to the overall credibility of the system, using a combined weighting method based on entropy weight and coefficient of variation. The credibility aggregation output module is used to calculate the overall credibility value of the autonomous driving virtual-real fusion test system based on the quantified values ​​of each secondary indicator and the first-layer weight and the second-layer weight through weighted aggregation, and to map the overall credibility value to a credibility level and output the credibility evaluation result. The online monitoring and alarm module is used to calculate the quantitative values ​​of each indicator in real time during the test run. When any level indicator value or the overall system credibility value is lower than the corresponding alarm threshold, a credibility alarm signal is generated or a test pause command is triggered.

Images