A multi-index closed-loop evaluation method for a modular end-to-end autonomous driving system

Abstract

The application discloses a kind of multi-index closed-loop evaluation methods for modular end-to-end automatic driving system, comprising: constructing real-time synchronization mechanism in closed-loop simulation, record each frame inference delay, and when delay exceeds simulation step, require simulation car to keep last simulation step control instruction, to simulate control lag;Under the mechanism, drive safety, efficiency and comfort degree and other closed-loop performance indicators are calculated;Based on high-precision energy consumption collector, energy consumption is collected;The CRITIC method is used to determine the index weight, normalize fusion is carried out on performance, delay and energy consumption, and is reflected into collision safety penalty, to build composite evaluation index.The application quantitatively compares the coupling relationship of performance, delay and energy consumption of automatic driving system under unified closed-loop condition, provides basis for model optimization, scheme selection and engineering deployment.Solve the problem that existing method excessively depends on open-loop accuracy, it is difficult to reflect the influence of inference delay on driving behavior, and lack of unified evaluation of engineering indicators such as energy consumption.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving algorithm evaluation, and in particular to a multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems. Background Technology

[0002] In the field of autonomous driving technology, modular end-to-end (M-E2E) algorithms achieve joint optimization of perception, prediction, and planning through shared features and unified training, achieving good results in closed-loop driving performance. However, algorithm evaluation for engineering implementation still faces significant challenges. The engineering value of autonomous driving algorithms depends on their overall performance as a closed-loop control system under real-world operating conditions. Evaluation needs to simultaneously cover multiple dimensions such as algorithm performance, inference latency, and energy consumption. These indicators are nonlinear and strongly coupled; separating them into independent evaluation items will make it difficult for evaluation conclusions to reliably support engineering selection and optimization decisions.

[0003] Existing evaluation methods primarily rely on open-loop offline benchmarks based on datasets such as nuScenes and closed-loop simulation benchmarks represented by CARLA Leaderboard. Both are built upon limited and regularized scenarios and sample distributions, lacking sufficient coverage of long-tail risks, complex interactions, and extreme conditions, thus failing to characterize the robustness and real-world performance of models in real traffic environments. Furthermore, the evaluation objectives emphasize perception and decision-making accuracy indicators while neglecting the quantitative assessment of edge deployment indicators such as latency and energy consumption. More importantly, both types of benchmarks struggle to quantitatively reveal the impact mechanism of latency on closed-loop performance aspects such as safety, comfort, and driving efficiency, thus leaving efficient design at the level of experience-driven performance trade-offs, lacking clear optimization objectives and design basis. Summary of the Invention

[0004] The purpose of this invention is to overcome the problems existing in the evaluation process of end-to-end and modular end-to-end autonomous driving algorithms, such as the single evaluation dimension, the difficulty in characterizing the impact of inference latency and energy consumption on closed-loop behavior, and the lack of unified and comparable basis for the conclusions of multiple indicators.

[0005] By constructing a multi-index evaluation framework, the control update lag caused by real inference delay is explicitly modeled in synchronous closed-loop simulation. Engineering indicators such as safety, efficiency, comfort, and energy consumption are incorporated into a unified quantitative system, thereby achieving system-level and reproducible evaluation of autonomous driving algorithms under real-world operational constraints. This provides a clear basis for optimizing scheme selection and edge-side deployment verification.

[0006] Overall, this invention proposes a multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems. In the closed-loop simulation process, inference latency is collected frame-by-frame and the simulation time is adaptively advanced accordingly to generate a closed-loop trajectory incorporating latency effects. Driving safety, traffic efficiency, and ride comfort indices are calculated on this trajectory, and inference energy consumption is collected through interfaces such as NVML, with a sliding window used to obtain single-frame energy consumption statistics. Furthermore, the multi-dimensional indices are normalized and fused to construct a comprehensive evaluation index including collision penalties. The CRITIC method is used to automatically determine weights based on the comparative strength and correlation of the indices, avoiding subjective weighting and duplicate scoring. This method can characterize the coupled trade-off between performance, latency, and energy consumption under a unified closed-loop condition, improving the stability, interpretability, and cross-scheme comparability of the evaluation conclusions, and providing a reusable technical path for the engineering implementation and iterative optimization of autonomous driving algorithms.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] A multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems includes the following steps:

[0009] S1. Construct a real-time synchronous simulation mechanism (RTS) based on the CARLA simulation framework in closed-loop simulation;

[0010] S2. Model the multi-dimensional indicators of closed-loop driving safety (DS), efficiency (DE), and comfort (DC) under RTS conditions.

[0011] S3. Collect cumulative energy consumption based on NVML and estimate single-frame energy consumption using a sliding window, while simultaneously calculating latency metrics;

[0012] S4. Normalize and integrate performance, latency, and energy consumption, and introduce safety penalties in the event of a collision to construct a composite evaluation index EER. AV ;

[0013] S5. Use CRITIC weighting to determine weights based on the strength and correlation of indicator comparisons and output system-level scores.

[0014] Furthermore, in S1, the real-time synchronous simulation mechanism RTS based on the CARLA simulation framework includes the following steps:

[0015] S101, based on the synchronous simulation mechanism of CARLA Leaderboard, constructs a closed-loop synchronous simulation test process. In each simulation cycle, it calls the tested modular end-to-end autonomous driving system to generate vehicle control commands and updates the simulated vehicle motion state and traffic environment state according to the control commands.

[0016] S102, insert timers at the entry and exit points of the decision function run_step of the modular end-to-end autonomous driving system under test, and measure the single-frame inference delay frame by frame. and will Return to Scene Manager;

[0017] S103, according to and Calculate the number of decision updates n that need to be skipped, where The preset single-frame advance time step for the closed-loop simulation environment represents the fixed time interval at which the simulation clock advances when the simulation environment completes a state update; the calculation method of n satisfies:

[0018]

[0019] S104, when n>0, skip the subsequent n decision updates and keep the control command of the previous frame unchanged in the next n+1 frames. The control command includes steering, throttle and braking. When n=0, advance the simulation by updating the control command once per frame.

[0020] Furthermore, in S2, closed-loop driving safety is calculated under RTS conditions. The method is as follows:

[0021] In an evaluation route pre-given by a closed-loop test dataset, the route is divided into n segments, and the completion rate of the i-th segment is calculated. And assign a penalty coefficient to the j-th type of traffic violation or failure event occurring on the i-th road segment. Safety indicators are calculated by road segment aggregation. ,satisfy:

[0022]

[0023] Furthermore, in S2, speed checkpoints are set in the closed-loop evaluation route to calculate driving efficiency. Specifically:

[0024] Set a fixed number of speed checkpoints M along the route;

[0025] Calculate the vehicle speed at the kth checkpoint Average speed of adjacent vehicles The ratio of these ratios is used to obtain the efficiency index DE, which satisfies the following:

[0026]

[0027] Extreme outliers with efficiency index DE exceeding a set threshold are removed, and test trajectories that fail to pass a certain proportion of the evaluation route given in the closed-loop test dataset are also removed.

[0028] Furthermore, in S2, comfort indices are calculated based on the closed-loop trajectory. Specifically:

[0029] For each frame, six types of vehicle dynamic variables are calculated and smoothness threshold tests are performed on each. The variables include longitudinal acceleration, absolute value of maximum lateral acceleration, yaw rate, yaw acceleration, longitudinal jerk component, and jerk vector magnitude.

[0030] Taking f consecutive frames as an evaluation segment, segments in which all frames within a segment satisfy the threshold test are counted as smooth segments; and the number of smooth segments is used as the evaluation factor. With the total number of segments The ratio is used to calculate the comfort index DC, which satisfies the following:

[0031]

[0032] When the vehicle speed is less than or equal to the set speed threshold And it continues to exceed the set time threshold. When the state is deemed an acceptable state of rest, then... No penalty is imposed; when the total number of frames on the trajectory is less than the frame threshold, the corresponding route will be removed from the current route. Excluded from the statistics.

[0033] Furthermore, S3 specifically includes:

[0034] S301 obtains the cumulative power consumption of the GPU since the driver was loaded via the NVML interface;

[0035] S302 uses a sliding window statistical method to calculate the energy consumption of a single frame:

[0036] A warm-up phase is performed before statistics are collected, with a set warm-up duration.

[0037] Then at the set number of frames Record the cumulative energy consumption difference between the beginning and end of the window. and with As the average energy consumption per frame.

[0038] Furthermore, in S4, the evaluation indicators The construction methods include:

[0039] S401, based on the RTS real-time synchronous simulation framework built in S1, tests and calculates three performance indicators in S2, and denotes them as follows: , , Meanwhile, the single-frame energy consumption of the algorithm is calculated based on the energy consumption calculation method in S3. ;

[0040] S402, for , , , After normalization, we get , , and ,in The smaller the indicator, the better, and its negative contribution is included in the overall score;

[0041] S403, Calculate the overall score for each route i. ,satisfy:

[0042]

[0043] In the formula, C is the collision penalty factor, C=1 when there is no collision and C=0 when a collision occurs; These represent the contribution weights of each indicator;

[0044] S404, the average score of all routes is obtained. , satisfy:

[0045]

[0046] Where L represents the total number of road segments using the dataset.

[0047] Furthermore, in S5, the weighting method based on CRITIC includes:

[0048] S501, Calculate the standard deviation of the i-th evaluation index. To characterize the contrast intensity;

[0049] S502, Calculate the correlation coefficient between the i-th indicator and the j-th evaluation indicator. To characterize conflict;

[0050] S503, based on the contrast strength and representational conflict of a total of m indicators, calculate the information content of the i-th evaluation indicator. ,satisfy:

[0051]

[0052] S504, normalize the information content to obtain the weight of the i-th indicator. ,satisfy:

[0053]

[0054] S505, utilizing the constructed The model takes the weights of each indicator as input and calculates the final comprehensive performance-latency-energy consumption evaluation index.

[0055] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0056] Compared with the prior art, the present invention has the following technical advantages:

[0057] This invention proposes a real-time synchronous closed-loop simulation framework for real-time systems (RTS), which can quantitatively characterize the impact of algorithm inference latency on closed-loop performance. The platform is based on the CARLA Leaderboard synchronous simulation mechanism, dynamically recording the algorithm's forward inference latency in each simulation frame and transmitting it back to the scene management module. The scene management module then calculates the number of simulation steps required based on the latency and simulation step size. This method solves the problem that existing evaluation methods struggle to explicitly map inference latency to closed-loop behavior, thus failing to quantitatively reveal the impact of real-time performance on system performance, and provides a rigorous and consistent evaluation method.

[0058] This invention proposes a multi-dimensional evaluation system for end-to-end autonomous driving algorithms. The platform simultaneously covers safety, efficiency, comfort, and deployment-related latency and energy consumption. Furthermore, it employs CRITIC objective weighting to suppress redundant information and duplicate scoring, thereby obtaining more stable and reproducible comprehensive ranking and trade-off analysis results. This method addresses the problems of existing research that relies heavily on a single dimension or subjective weighting, leading to incomparable conclusions and difficulties in uniformly measuring the performance-latency-energy consumption trade-off. It provides a data-driven comprehensive evaluation basis. Attached Figure Description

[0059] Figure 1 This is a schematic flowchart of the method of the present invention;

[0060] Figure 2 A schematic diagram of a multi-index closed-loop evaluation framework for modular end-to-end autonomous driving systems;

[0061] Figure 3 Security of latency impact on inference performance based on a multi-metric evaluation framework Analysis chart;

[0062] Figure 4 Efficiency of evaluating the impact of latency on inference performance based on a multi-metric evaluation framework and comfort Analysis diagram. Detailed Implementation

[0063] This invention proposes a multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems. Addressing the problem that existing autonomous driving evaluation frameworks have only one evaluation dimension and cannot quantitatively characterize the relationship between engineering deployability and algorithm performance, this invention designs a real-time synchronous simulation mechanism and performs detailed modeling and characterization of multiple dimensions such as driving safety, comfort, efficiency, and energy consumption, ultimately constructing a comprehensive evaluation index. This provides a rigorous quantitative method for evaluating the performance of end-to-end autonomous driving algorithm systems.

[0064] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Component models, material names, connection structures, control methods, algorithms, and other features not explicitly described in this technical solution are considered common technical features disclosed in the prior art.

[0065] Example 1

[0066] This invention provides a multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems, the process of which is as follows: Figure 1 As shown, it includes the following steps:

[0067] S1. Construct a Real-Time Synchronization (RTS) mechanism based on the CARLA simulation framework in the closed-loop simulation to model the control lag effect caused by inference delay in real-world scenarios; the specific process of the RTS simulation mechanism is as follows: Figure 2 As shown.

[0068] S101, based on the synchronous simulation mechanism of CARLA Leaderboard, constructs a closed-loop synchronous simulation test process. In each simulation cycle, it calls the tested modular end-to-end autonomous driving system to generate vehicle control commands and updates the simulated vehicle motion state and traffic environment state according to the control commands.

[0069] S102, high-precision timers are inserted at the entry and exit points of the decision function run_step of the modular end-to-end autonomous driving system under test, and the inference delay of each frame is measured frame by frame. and will Return to Scene Manager;

[0070] S103, according to and Calculate the number of decision updates n that need to be skipped, where The preset single-frame advance time step for the closed-loop simulation environment represents the fixed time interval at which the simulation clock advances when the simulation environment completes a state update; the calculation method of n satisfies:

[0071]

[0072] S104, when n>0, skip the subsequent n decision updates and keep the control commands from the previous frame unchanged for the next n+1 frames. These control commands include at least steering, throttle, and braking to advance the simulation. When n=0, advance the simulation by updating the control commands once per frame. The default value is 0.05s, which is used to reproduce the impact of control update delay caused by inference latency in real-world scenarios during synchronous simulation.

[0073] S2. Model the multi-dimensional indicators of closed-loop driving safety (DS), efficiency (DE), and comfort (DC) under RTS conditions.

[0074] S201, Calculating Driving Safety In an evaluation route pre-given by a closed-loop test dataset, the route is divided into n segments, and the completion rate of the i-th segment is calculated. And assign a penalty coefficient to the j-th type of traffic violation or failure event occurring on the i-th road segment. Safety indicators are calculated by road segment aggregation. ,satisfy:

[0075]

[0076] S202, setting speed checkpoints in the closed-loop evaluation route to calculate driving efficiency. Specifically, a fixed number of speed checkpoints M are set along the route, where M=20; the vehicle speed is calculated at the k-th checkpoint. Average speed of adjacent vehicles The ratio of these ratios is used to obtain the efficiency index DE, which satisfies the following:

[0077]

[0078] S203, remove extreme outliers with an efficiency index DE greater than 1000%, and also remove trajectories that fail to pass the first 5% of the evaluation route given in the closed-loop test dataset.

[0079] S204, Calculating comfort indices based on closed-loop trajectory C, specifically: Calculate at least six types of vehicle dynamic variables for each frame and perform a smoothness threshold test on each. These variables include at least longitudinal acceleration, the absolute value of maximum lateral acceleration, yaw rate, yaw acceleration, longitudinal jerk component, and jerk vector magnitude. Use 20 consecutive frames as an evaluation segment, and count segments where all frames within the segment satisfy the threshold test as smooth segments. The number of smooth segments is then used to determine the smoothness threshold. With the total number of segments The ratio is used to calculate the comfort index DC, which satisfies:

[0080]

[0081] S205, when the vehicle speed is ≤0.1 m / s and remains stationary for more than 60 s, it is considered an acceptable stationary state. No penalty is imposed; when the total number of frames for the trajectory is less than 20, the corresponding route will be changed from... Excluded from statistics;

[0082] S206, wherein the threshold range may be: longitudinal acceleration [-4.05, 2.40], absolute value of maximum lateral acceleration [-4.89, 4.89], yaw rate [-0.95, 0.95], yaw rate acceleration [-1.93, 1.93], longitudinal jerk component [-4.13, 4.13], jerk vector magnitude [-8.37, 8.37].

[0083] S3. Collect cumulative energy consumption based on NVML and estimate single-frame energy consumption using a sliding window, while simultaneously calculating latency metrics.

[0084] S301 obtains the cumulative power consumption of the GPU since the driver was loaded via the NVML interface;

[0085] S302 uses a sliding window statistical method to calculate the energy consumption of a single frame: the cumulative energy consumption difference between the beginning and end of a window is recorded in a window of 100 frames. and with As the average energy consumption per frame.

[0086] In this embodiment, a preheating phase is performed before the formal statistics are collected. The preheating time is 30 seconds to reduce the impact of initial frequency fluctuations on energy consumption statistics.

[0087] S4. Normalize and integrate performance, latency, and energy consumption, and introduce a strong penalty (safety penalty) when a collision occurs to ensure safety priority, thus constructing a composite evaluation index EER. AV .

[0088] S401, based on the RTS real-time synchronous simulation framework built in S1, tests and calculates the three performance indicators introduced in S2, and denotes them as follows: , , Meanwhile, the single-frame energy consumption of the algorithm is calculated based on the energy consumption calculation method in S3. ;

[0089] S402, for , , , After normalization, we get , , and ,in The smaller the value, the better, and the negative contribution is included in the overall score;

[0090] S402, Calculate the overall score for each route i. ,satisfy:

[0091]

[0092] Set the collision penalty factor C: C=1 when there is no collision, and C=0 when a collision occurs; These represent the contribution weights of each indicator;

[0093] S404, the average score of all routes is obtained. ,satisfy:

[0094]

[0095] Where L represents the total number of road segments using the dataset, in this embodiment, .

[0096] S5. CRITIC objective weighting is adopted to determine the weights based on the comparative strength and correlation of the indicators and output the system-level score.

[0097] S501, Calculate the standard deviation of the i-th evaluation index. To characterize the contrast intensity;

[0098] S502, Calculate the correlation coefficient between the i-th indicator and the j-th evaluation indicator. To characterize conflict;

[0099] S503, based on the contrast strength and representational conflict of a total of m indicators, calculate the information content of the i-th evaluation indicator. ,satisfy:

[0100]

[0101] S504, normalize the information content to obtain the weight of the i-th indicator. ,satisfy:

[0102]

[0103] S505, utilizing the constructed The model takes the weights of each indicator as input and calculates the final comprehensive performance-latency-energy consumption evaluation index.

[0104] The following section uses the quantitative evaluation of the relationship between inference latency and algorithm performance of the most advanced modular end-to-end autonomous driving model as an example to specifically illustrate the multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems proposed in this invention.

[0105] Step 1: Algorithm and runtime environment preparation: UniAD, a typical modular end-to-end autonomous driving algorithm, was selected as the basic model. The experimental platform was built based on the CARLA simulation system and loaded with 44 interactive driving scenarios defined by the Bench2Drive dataset, containing a total of 220 closed-loop test routes. The algorithm ran on a computing platform equipped with a 4090 GPU.

[0106] Step 2: Construct an RTS real-time synchronous simulation mechanism based on the CARLA simulation platform: To simulate the control update delay effect caused by "frame-by-frame latency fluctuations" in real deployment, an RTS advancement strategy is introduced into the synchronous simulation: The scenario manager receives the inference latency of each frame. Calculate the number of decision updates to be skipped using the following formula. :

[0107]

[0108] when At that time, the tested modular end-to-end autonomous driving system skipped subsequent... The decision was updated, and in the following... Intra-frame retention of control commands from the previous frame, where The default is 0.05 seconds. The actual inference latency can be obtained frame by frame by inserting high-precision timing at the entry and exit points of the run_step decision function of the modular end-to-end autonomous driving system under test, or it can be set manually.

[0109] Step 3: Determine the test plan: To quantitatively explain the impact of inference latency on the algorithm's closed-loop performance, the inference latency will be manually set. To achieve both low and high frame rate resolutions on the FPS axis, a non-uniform sampling strategy was adopted (dense sampling in the low FPS segment and gradually increasing the sampling interval in the high FPS segment) to sample a total of 14 frame rates, covering a latency range from 1 FPS to 30 FPS.

[0110] Step 4, Performance Evaluation: Statistically analyze security data separately. ,efficiency With comfort Among them, security Aggregate efficiency based on road segment completion rate and violation penalties. Calculate the vehicle / adjacent vehicle speed ratio using M=20 speed checkpoints, and eliminate extreme outliers and the first 5% of the route not traversed, for comfort. The FVS threshold test was performed based on six types of dynamic variables, and the proportion of smoothed segments was statistically analyzed using 20 frames.

[0111] Step 5, Result Analysis: (e.g.) Figure 3 and Figure 4 As shown, inference latency has a significant impact on closed-loop performance. With decreasing latency (increasing FPS), the driving score shows a significant upward trend in the 1-20 FPS range, with an overall improvement of approximately 20.33%. However, the gain tends to saturate when the FPS is around 20–24, and further increases inference speed yield limited marginal benefits and may even experience a slight decline. In the UniAD sub-indicator results, safety... Overall efficiency increases with FPS (e.g., approximately 32.85 at 1 FPS, approximately 39.53 at 20 FPS, and approximately 40.63 at 24 FPS); Overall, there are few changes; comfort At low FPS (<5), the fluctuation was not significant, but it decreased significantly as the FPS continued to increase, showing a decrease of approximately 33.33% at 20 FPS. Further, under RTS settings and the same FPS conditions... The driving score was significantly lower than the fixed inference rate setting; A 24.62% decrease relative to the Bench2Drive baseline (42.35), while The results remain relatively stable, indicating that in addition to average latency, "multi-frame latency stability / long-tail latency" is also a key factor affecting closed-loop performance. The experimental results above demonstrate that the proposed evaluation framework can accurately capture the impact of latency on algorithm performance.

[0112] The above description of the embodiments is provided to enable those skilled in the art to understand and use the invention. It will be apparent to those skilled in the art that various modifications can be made to these embodiments, and the general principles described herein can be applied to other embodiments without inventive effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the invention should be within the protection scope of the present invention.

Claims

1. A multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems, characterized in that, Includes the following steps: S1. Construct a real-time synchronous simulation mechanism (RTS) based on the CARLA simulation framework in closed-loop simulation; S2. Model the multi-dimensional indicators of closed-loop driving safety (DS), efficiency (DE), and comfort (DC) under RTS conditions. S3. Collect cumulative energy consumption based on NVML and estimate single-frame energy consumption using a sliding window, while simultaneously calculating latency metrics; S4. Normalize and integrate performance, latency, and energy consumption, and introduce safety penalties in the event of a collision to construct a composite evaluation index EER. AV ; S5. Use CRITIC weighting to determine weights based on the strength and correlation of indicator comparisons and output system-level scores.

2. The multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems according to claim 1, characterized in that, In S1, the Real-Time Synchronous Simulation (RTS) mechanism based on the CARLA simulation framework includes the following steps: S101, based on the synchronous simulation mechanism of CARLA Leaderboard, constructs a closed-loop synchronous simulation test process. In each simulation cycle, it calls the tested modular end-to-end autonomous driving system to generate vehicle control commands and updates the simulated vehicle motion state and traffic environment state according to the control commands. S102, insert timers at the entry and exit points of the decision function run_step of the modular end-to-end autonomous driving system under test, and measure the single-frame inference delay frame by frame. and will Return to Scene Manager; S103, according to and Calculate the number of decision updates n that need to be skipped, where The preset single-frame advance time step for the closed-loop simulation environment represents the fixed time interval at which the simulation clock advances when the simulation environment completes a state update; the calculation method of n satisfies: ； S104, when n>0, skip the subsequent n decision updates and keep the control command of the previous frame unchanged in the next n+1 frames. The control command includes steering, throttle and braking. When n=0, advance the simulation by updating the control command once per frame.

3. A multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems according to claim 1 or 2, characterized in that, In S2, closed-loop driving safety is calculated under RTS conditions. The method is as follows: In an evaluation route pre-given by a closed-loop test dataset, the route is divided into n segments, and the completion rate of the i-th segment is calculated. And assign a penalty coefficient to the j-th type of traffic violation or failure event occurring on the i-th road segment. Safety indicators are calculated by road segment aggregation. ,satisfy: 。 4. A multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems according to claim 1 or 2, characterized in that, In S2, speed checkpoints are set in the closed-loop evaluation route to calculate driving efficiency. Specifically: Set a fixed number of speed checkpoints M along the route; Calculate the vehicle speed at the kth checkpoint Average speed of adjacent vehicles The ratio of these ratios is used to obtain the efficiency index DE, which satisfies the following: ； Extreme outliers with efficiency index DE exceeding a set threshold are removed, and test trajectories that fail to pass a certain proportion of the evaluation route given in the closed-loop test dataset are also removed.

5. A multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems according to claim 1 or 2, characterized in that, In S2, comfort indices are calculated based on the closed-loop trajectory. Specifically: For each frame, six types of vehicle dynamic variables are calculated and smoothness threshold tests are performed on each. The variables include longitudinal acceleration, absolute value of maximum lateral acceleration, yaw rate, yaw acceleration, longitudinal jerk component, and jerk vector magnitude. Taking f consecutive frames as an evaluation segment, segments in which all frames within a segment satisfy the threshold test are counted as smooth segments; and the number of smooth segments is used as the evaluation factor. With the total number of segments The ratio is used to calculate the comfort index DC, which satisfies the following: ； When the vehicle speed is less than or equal to the set speed threshold And it continues to exceed the set time threshold. When the state is deemed an acceptable state of rest, then... No penalty is imposed; when the total number of frames on the trajectory is less than the frame threshold, the corresponding route will be removed from the current route. Excluded from the statistics.

6. The multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems according to claim 1, characterized in that, S3 specifically includes: S301 obtains the cumulative power consumption of the GPU since the driver was loaded via the NVML interface; S302 uses a sliding window statistical method to calculate the energy consumption of a single frame: A warm-up phase is performed before statistics are collected, with a set warm-up duration. Then at the set number of frames Record the cumulative energy consumption difference between the beginning and end of the window. and with As the average energy consumption per frame.

7. The multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems according to claim 1, characterized in that, In S4, the evaluation indicators The construction methods include: S401, based on the RTS real-time synchronous simulation framework built in S1, tests and calculates three performance indicators in S2, and denotes them as follows: , , Meanwhile, the single-frame energy consumption of the algorithm is calculated based on the energy consumption calculation method in S3. ; S402, for , , , After normalization, we get , , and ,in The smaller the indicator, the better, and its negative contribution is included in the overall score; S403, Calculate the overall score for each route i. ,satisfy: ； In the formula, C is the collision penalty factor, C=1 when there is no collision and C=0 when a collision occurs; These represent the contribution weights of each indicator; S404, the average score of all routes is obtained. ,satisfy: ； Where L represents the total number of road segments using the dataset.

8. A multi-index closed-loop evaluation method for modular end-to-end autonomous driving systems according to claim 1 or 7, characterized in that, In S5, the weighting methods based on CRITIC include: S501, Calculate the standard deviation of the i-th evaluation index. To characterize the contrast intensity; S502, Calculate the correlation coefficient between the i-th indicator and the j-th evaluation indicator. To characterize conflict; S503, based on the contrast strength and representational conflict of a total of m indicators, calculate the information content of the i-th evaluation indicator. ,satisfy: ； S504, normalize the information content to obtain the weight of the i-th indicator. ,satisfy: ； S505, utilizing the constructed The model takes the weights of each indicator as input and calculates the final comprehensive performance-latency-energy consumption evaluation index.

Images