Performance evaluation method, system and equipment of automatic driving simulation platform and medium

Abstract

The invention relates to the technical field of automatic driving simulation, in particular to a performance evaluation method, system and device of an automatic driving simulation platform and a medium, and the method comprises the steps: constructing a simulation environment and a simulation vehicle equipped with a sensor, an actuator and a vehicle-mounted calculation unit simulation model; establishing a multi-dimensional platform evaluation system covering security, efficiency, response time and data processing capability, defining a specific sub-index, an evaluation method and a target data item needing to be collected for each dimension, and setting a comprehensive evaluation rule; running a closed-loop simulation test in the simulation platform, and synchronously collecting all target data; calculating evaluation results of the sub-indexes one by one based on the collected data; and judging whether the overall performance of the platform reaches the standard according to comprehensive evaluation rules. According to the invention, systematization, standardization and quantitative evaluation of the performance of the automatic driving simulation platform are realized, the reliability and comparability of a test result are improved, and an effective basis is provided for optimization of the simulation platform.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving simulation, in particular to a performance evaluation method, system, device and medium of an automatic driving simulation platform. BACKGROUND

[0002] The rapid development of automatic driving technology puts high requirements on the safety and reliability of the system. Under the background of high cost and high risk of real vehicle road testing, using simulation platforms to test and verify automatic driving algorithms on a large scale and with high efficiency has become a key research and development means in the industry. At present, there are various automatic driving simulation software in the industry, which can realize the testing of part of the functions of the automatic driving system by constructing a virtual traffic environment, simulating sensor data and running the tested algorithm. These technical solutions usually focus on the reproduction of specific scenarios or the verification of single performance, providing necessary support for the early development of algorithms.

[0003] The existing simulation testing method can evaluate the specific behavior of the automatic driving system to a certain extent, for example, by presetting the test scene to test the following, lane changing or emergency braking ability of the vehicle, and giving a qualitative conclusion of pass or fail based on the running result of the scene. Some solutions also introduce specific indicators such as trajectory deviation and collision time, trying to quantitatively analyze the simulation results. These methods provide a feasible technical path for how to preliminarily verify the algorithm function without real vehicle testing.

[0004] However, the existing simulation testing method usually focuses on the behavior of the tested automatic driving algorithm itself, and lacks a systematic and quantitative evaluation system for the performance of the simulation platform itself that carries the test. The simulation accuracy, calculation efficiency, comprehensive coverage of the scene and the standardization degree of the test process of the simulation platform itself directly affect the credibility and research and development efficiency of the test results. Due to the lack of a multi-dimensional and quantifiable unified evaluation method for the performance of the simulation platform, it is difficult to objectively compare different test results, and there is also a lack of basis for optimizing the efficiency and reliability of the test process, thereby restricting the simulation test from playing a greater role in the research and development of automatic driving. SUMMARY

[0005] The simulation test method of the existing automatic driving system focuses on evaluating the behavior of the automatic driving algorithm, but lacks a unified system for multi-dimensional, quantifiable and systematic evaluation of the performance of the simulation platform itself, resulting in insufficient reliability and comparability of the test results, and the efficiency and standardization of the test process need to be improved. The present application provides a performance evaluation method, system, device and medium for an automatic driving simulation platform, which realizes objective, quantitative and systematic evaluation of the performance of the simulation platform by constructing a complete simulation platform and an evaluation system covering multiple specific indicators, and defining a standardized process from test execution, data collection to comprehensive judgment, significantly improving the reliability, comparability of the test results, and the efficiency and standardization of the entire test verification link.

[0006] In a first aspect, the present application provides a performance evaluation method for an automatic driving simulation platform, comprising the following steps: S1. Constructing a simulation environment containing static scenes and dynamic traffic flows in the automatic driving simulation platform; Configuring at least one simulation vehicle in the simulation environment; Configuring a sensor simulation model for simulating perception data generation, an actuator simulation model for controlling instruction execution, and a vehicle-mounted computing unit simulation model for controlling the running of the automatic driving algorithm for the simulation vehicle; S2. Constructing a platform evaluation system, the platform evaluation system comprising several evaluation dimensions, the evaluation dimensions including safety, efficiency, response time and data processing capacity; Defining at least one specific sub-indicator for each evaluation dimension; Determining the evaluation method and the target data item to be collected for each specific sub-indicator, the evaluation method including quantitative calculation or qualitative scoring; Setting comprehensive evaluation rules for determining whether the overall performance of the platform meets the standard based on the evaluation results of each specific sub-indicator, the evaluation results being the values obtained using the corresponding evaluation method; S3. Running a simulation test in the automatic driving simulation platform, synchronously collecting data of all target data items during the simulation test, the simulation test comprising: Configuring an automatic driving algorithm in the vehicle-mounted computing unit simulation model, starting a closed-loop simulation, during which perception data is generated by the sensor simulation model, control instructions are generated after decision-making by the automatic driving algorithm, and the actuator simulation model executes the control instructions to control the motion of the simulation vehicle; S4. Based on the data of the target data items, calculating or evaluating the evaluation results of each specific sub-indicator one by one according to the evaluation method of each specific sub-indicator; S5. Based on the comprehensive evaluation rules and the evaluation results of each specific sub-indicator, determining whether the overall performance of the platform meets the standard, and obtaining the evaluation conclusion.

[0007] It should be further explained that step S1, which involves constructing a simulation environment that includes both static scenes and dynamic traffic flow, specifically includes: Import map data to generate a static scene that includes road networks and traffic signs; Dynamic traffic flow is injected based on preset vehicle density, pedestrian density, and behavior models; Configure adjustable weather and lighting systems to simulate different environmental conditions.

[0008] It should be further noted that importing map data specifically refers to importing map data in OpenDRIVE format; The behavioral models include the Intelligent Driver Model (IDM) and the Lane Change Model that minimizes overall braking induction (MOBIL). The system includes adjustable weather and lighting systems, including adjustments for light intensity, precipitation intensity, and fog concentration.

[0009] It should be further noted that configuring a sensor simulation model for generating simulated perception data for the simulated vehicle specifically includes configuring at least one of the following models: The camera simulation model is configured to simulate image acquisition based on ray tracing rendering and apply lens distortion and motion blur effects. The lidar simulation model is configured to simulate point cloud acquisition based on ray projection technology and introduce noise and random point loss. The millimeter-wave radar simulation model is configured to simulate radar signal acquisition and simulate the Doppler effect and clutter based on the electromagnetic wave propagation model. The inertial measurement unit simulation model is configured to generate inertial data based on an error model that includes zero-bias stability parameters.

[0010] It should be further noted that the data output by the camera simulation model is in RAW12 format or H.264 compressed stream; The data output format of the lidar simulation model is either PCAP format or ROS PointCloud2 message; The data output by the millimeter-wave radar simulation model is in CAN FD frame format.

[0011] It should be further noted that configuring the actuator simulation model for the simulated vehicle to execute control commands specifically includes configuring the following models: A simulation model of a steer-by-wire system is used to receive and parse steering wheel angle commands, and to simulate the dynamic response of the actuator through a transfer function. The brake-by-wire system simulation model is used to receive and parse braking pressure commands, simulate hydraulic response, and integrate the functional simulation of the anti-lock braking system (ABS) and the electronic stability control system (ESC).

[0012] It should be further explained that configuring the onboard computing unit simulation model used to control the operation of the autonomous driving algorithm includes implementing the following simulation process: Simulates in-vehicle network communication; Simulate the temporal synchronization and spatial fusion of multi-sensor data; Simulates the computational process of perception algorithms that include target detection and tracking; Simulate a rule-based and algorithm-based decision-making and planning process; Generating lateral and longitudinal control commands for simulated vehicles.

[0013] It should be further noted that, in step S2, the specific sub-indicators of the security dimension include: Collision accident frequency; Traffic violation rate; Success rate in responding to sudden obstacles within the Design Operational Domain (ODD); The specific sub-indicators of the efficiency dimension include: Percentage of autonomous driving mileage; Average vehicle speed; Calculate resource utilization; The specific sub-metrics of the response time dimension include: End-to-end delay from sensor data acquisition to control command generation; Emergency braking response time to sudden obstacles; The specific sub-indicators of the data processing capability dimension include: Data throughput of multi-source sensors; Perceptual model inference frame rate.

[0014] It should be further noted that the calculation formulas for each specific sub-indicator and the target data items to be collected include: Collision accident frequency The calculation formula is:

[0015] The number of collision events in the simulation test is represented by the following statistical standard: during the simulation, the number of events in which the simulated vehicle physically overlaps with an obstacle or road boundary is recorded as the number of collision events, regardless of the collision speed or severity. This represents the total mileage of the simulation test; The target data items to be collected are and ; Traffic violation rate The calculation formula is:

[0016] This indicates the number of traffic violation incidents that occurred during the simulation test; Indicates the total simulation test time; The target data items to be collected are and ; Autonomous driving mileage percentage The calculation formula is:

[0017] This represents the driving mileage in the autonomous driving mode during simulation testing; The target data items to be collected are and ; resource utilization The calculation formula is:

[0018] This represents the average CPU utilization during simulation testing. This represents the average memory usage during simulation testing. The target data items to be collected are and ; End-to-end delay The calculation formula is:

[0019] Indicates the total number of sampling points; Indicates the first The control commands for each sampling point generate timestamps; Indicates the first Timestamps of sensor data from each sampling point; The target data items to be collected are all. and ; Data throughput The calculation formula is:

[0020] This indicates the total amount of sensor data input to the interface; The target data items to be collected are and .

[0021] It should be further explained that the comprehensive evaluation rules include: a comprehensive scenario score based on quantitative calculation results, and then a qualification determination based on the comprehensive scenario score; Overall scene score The calculation formula is:

[0022]

[0023] in, This indicates the initial overall score for the scene. The score represents the security dimension score, which is obtained by normalizing the evaluation results of all specific sub-indicators under the security dimension and then summing them by weight. The efficiency dimension score is obtained by normalizing the evaluation results of all specific sub-indicators under the efficiency dimension and then summing them by weight. The score represents the response time dimension score, which is obtained by normalizing the evaluation results of all specific sub-indicators under the response time dimension and then summing them by weight. The score represents the data processing capability dimension score, which is obtained by normalizing the evaluation results of all specific sub-indicators under the data processing capability dimension and then summing them by weight. This represents the preset weighting coefficients for the security dimension; This represents the preset efficiency dimension weighting coefficient; This represents the preset weighting coefficient for the response time dimension; This represents the preset weight coefficient for the data processing capability dimension; satisfy ; This represents the theoretical maximum value of the overall scene score, which is equal to the initial overall scene score calculated when the evaluation results of all specific sub-indicators take the theoretical optimal value.

[0024] It should be further noted that the weighting coefficients are configured according to the type of simulation test scenario: For security-critical scenarios, configure as , , , ; For efficiency-first scenarios, configure as follows , , , ; For comprehensive driving scenarios, the configuration is as follows: , , , ; Among them, safety-critical scenarios refer to simulation scenarios that are mainly designed to test the response and decision-making capabilities of autonomous driving systems in high-risk situations, including scenarios such as sudden braking of the vehicle in front, pedestrians suddenly appearing out of nowhere, and intersection conflicts. Efficiency-first scenarios refer to simulation scenarios that primarily aim to test the traffic efficiency and economy of autonomous driving systems on smooth roads, including highway cruising scenarios and urban expressway following scenarios. Comprehensive driving scenarios refer to simulation scenarios that integrate various traffic elements and interactions, with the main purpose of testing the comprehensive performance of autonomous driving systems in complex environments. These scenarios include urban road scenarios that include traffic lights, pedestrians, and non-motorized vehicles.

[0025] It should be further explained that the criteria for determining compliance based on a comprehensive scenario score specifically include: when At that time, it was determined to meet the standard; when If the condition is deemed met, the parameters need to be adjusted and the test repeated. when If so, it is judged as not meeting the standard.

[0026] It should be further noted that the comprehensive evaluation rules also include a veto condition. A veto condition is that if any of the following situations occur, the overall performance of the autonomous driving simulation platform will be directly determined to be substandard: A real collision occurred; The system crash rate exceeds 0.5%; End-to-end latency exceeds 100ms; The data loss rate exceeded 2%; The criteria for determining a realistic collision are: the simulated vehicle comes into contact with an obstacle or road boundary and meets any of the following conditions: The collision speed exceeded 5 meters per second; A collision interrupts the dynamics of the simulated vehicle, requiring an emergency stop or system reset. The collision event was recorded in the simulation platform's accident log.

[0027] It should be further explained that in step S3, configuring a benchmark autonomous driving algorithm model as a test load in the autonomous driving simulation platform specifically includes: selecting and loading an autonomous driving algorithm model with a clear interface specification, wherein the model includes at least perception, decision planning and control modules, and interfacing its input and output interfaces with the sensor simulation output and actuator simulation input of the simulation platform.

[0028] It should be further noted that in step S3, the test cases used to run the closed-loop simulation test are automatically generated through the following steps: Receive test requirement descriptions as input in natural language or structured text; Based on predefined scenario description specifications, the test requirement description is parsed into a structured scenario element graph, which includes entities, attributes, and relationships. Instantiate and assign values ​​to the parameters in the scene element graph to generate an executable test script.

[0029] It should be further noted that in step S3, when collecting data from all target data items synchronously, a unified time synchronization mechanism is applied to the data source of all target data items. The time synchronization mechanism adopts the Precise Time Protocol (PTP) or a timestamp injection method based on the simulated master clock to ensure that the timestamp alignment accuracy of the entire link data is better than 1 millisecond.

[0030] It should be further explained that in step S4, the data of the target data item is first cleaned and aligned, and then the evaluation result of each specific sub-indicator is calculated or evaluated one by one according to the evaluation method of each specific sub-indicator. Data cleaning and alignment include: performing packet loss detection and outlier removal on the raw data of the target data items, and aligning data streams of different frequencies to a unified time series using interpolation methods.

[0031] It should be further noted that in the data cleaning and alignment steps, for the data before multi-sensor fusion, an additional spatial calibration verification is performed. That is, the lidar point cloud is projected onto the camera image coordinate system using the known extrinsic parameter matrix, and the reprojection error is calculated. If the average error exceeds 2 pixels, the confidence of the data in that period is marked as reduced.

[0032] It should be further noted that in step S5, after the evaluation conclusion is obtained, a structured test report is automatically generated, which includes: a description of the test scenario, the evaluation results of each specific sub-indicator, the overall scenario score, and the judgment conclusion.

[0033] It should be further noted that the performance evaluation method of this solution is integrated into the continuous integration / continuous deployment (CI / CD) process; The autonomous driving simulation platform automatically executes steps S2 to S5 after each code submission or scheduled task trigger, and feeds back the evaluation conclusions and the evaluation results of each specific sub-indicator to the code repository or project management platform.

[0034] Secondly, this application provides a performance evaluation system for an autonomous driving simulation platform, used to implement the above-mentioned performance evaluation method, including: The simulation platform construction module is used to build an autonomous driving simulation platform, including: building a simulation environment containing static scenes and dynamic traffic flow; configuring at least one simulation vehicle in the simulation environment; configuring a sensor simulation model for simulating perception data generation, an actuator simulation model for executing control commands, and an on-board computing unit simulation model for controlling the operation of autonomous driving algorithms for the simulation vehicle. The evaluation system construction module is used to build the platform evaluation system, which includes several evaluation dimensions, including security, efficiency, response time, and data processing capabilities. It defines at least one specific sub-indicator for each evaluation dimension; determines the evaluation method and target data items to be collected for each specific sub-indicator, with evaluation methods including quantitative calculation or qualitative scoring; and sets comprehensive evaluation rules to determine whether the overall platform performance meets the standards based on the evaluation results of each specific sub-indicator. The evaluation result is the numerical value obtained using the corresponding evaluation method. The simulation test execution module is used to run simulation tests in the autonomous driving simulation platform. During the simulation test, data of all target data items are collected synchronously. The simulation test includes: configuring the autonomous driving algorithm in the vehicle computing unit simulation model, starting the closed-loop simulation, during which the sensor simulation model generates perception data, and after the autonomous driving algorithm makes a decision, it generates control commands. The actuator simulation model executes the control commands to control the movement of the simulated vehicle. The evaluation result generation module is used to calculate or evaluate the evaluation result of each specific sub-indicator based on the data of the target data item and according to the evaluation method of each specific sub-indicator. The evaluation conclusion determination module is used to determine whether the overall performance of the platform meets the standards based on the comprehensive evaluation rules and the evaluation results of each specific sub-indicator, and to draw an evaluation conclusion.

[0035] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described performance evaluation method.

[0036] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described performance evaluation method.

[0037] As can be seen from the above technical solutions, this application has the following advantages: 1. This application constructs a complete autonomous driving simulation platform that includes a simulation environment, vehicle model, sensor and actuator models, and on-board computing unit model. It also establishes a platform evaluation system covering multiple dimensions such as safety, efficiency, response time, and data processing capabilities. Furthermore, it defines a complete evaluation process from data acquisition and index calculation to comprehensive judgment. This solves the problem of the lack of systematic and quantifiable evaluation of the performance of the simulation platform itself in the prior art. It provides a standardized, highly operable, and objective method for evaluating the performance of the simulation platform, which significantly improves the credibility and comparability of the test results.

[0038] 2. This application solves the problems of scattered indicators, vague standards, and subjective conclusions in existing evaluation methods by defining specific sub-indicators, evaluation methods, and target data items to be collected for each evaluation dimension, and setting clear comprehensive evaluation rules. This enables the performance evaluation of the simulation platform in terms of computing resource consumption, data processing throughput, and task response latency to be quantified and refined, providing precise data guidance for platform performance optimization.

[0039] 3. This application addresses the problems of insufficient flexibility in simulation environment construction and difficulty in ensuring scene realism in the prior art by specifying in detail the steps for constructing the simulation environment, including importing maps, injecting parameterized dynamic traffic flow, and configuring an adjustable weather and lighting system. This enables the evaluation system to be based on high-fidelity and diverse test environments, thereby ensuring that the evaluation results fully reflect the platform's performance under different conditions.

[0040] 4. This application solves the problem that existing simulation tests often ignore or simplify the real data processing flow and timing characteristics inside the vehicle computing unit by realizing the time synchronization and spatial fusion of multi-sensor data in the vehicle computing unit simulation model, simulating the complete perception-decision-control algorithm pipeline, and collecting key data such as end-to-end delay. This makes the evaluation more realistically reflect the simulation platform's simulation capability and performance load on the core computing process of the autonomous driving system.

[0041] 5. This application solves the problems of single evaluation conclusions, inability to distinguish scene priorities and tolerance for fatal defects in the prior art by setting a hierarchical comprehensive evaluation rule that includes a veto condition and dynamically configured weights based on scene type. This makes the performance evaluation not only strictly control the safety bottom line, but also flexibly and comprehensively score according to different test objectives, and the evaluation conclusions are more practical and instructive.

[0042] 6. This application solves the problems of low automation and difficulty in closely connecting with the rapid iterative development process in the existing simulation testing process by integrating the performance evaluation method into the continuous integration / continuous deployment process and realizing the automatic generation of test cases and the automatic output of test reports. It realizes the normalized and automated monitoring and feedback of the simulation platform performance, which greatly improves the R&D efficiency and process standardization. Attached Figure Description

[0043] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of a performance evaluation method for an autonomous driving simulation platform in one embodiment of this application.

[0045] Figure 2 This is a schematic block diagram of the performance evaluation system of an autonomous driving simulation platform in one embodiment of this application.

[0046] Figure 3 This is a schematic diagram of the hardware structure of an electronic device in one embodiment of this application. Detailed Implementation

[0047] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] The performance evaluation method involved in this application will be described in detail below. Specific details such as particular system structures and technologies are presented for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details.

[0049] In the performance evaluation methods involved in this application, the term "comprising" indicates the presence of the described feature, whole, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or collections thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0050] To facilitate a clear description of the technical solutions of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0051] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0052] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0053] The performance evaluation method provided in this application embodiment is executed by a computer device, and correspondingly, the performance evaluation system of the autonomous driving simulation platform runs in the computer device.

[0054] Figure 1 This is a flowchart of a performance evaluation method for an autonomous driving simulation platform according to an embodiment of this application. Figure 1 The executing entity can be a performance evaluation system. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0055] like Figure 1 As shown, the performance evaluation method of this autonomous driving simulation platform includes: Step S1: Construct a simulation environment in the autonomous driving simulation platform that includes static scenes and dynamic traffic flow; Configure at least one simulated vehicle in the simulation environment; The simulation vehicle is equipped with sensor simulation models for simulating perception data generation, actuator simulation models for executing control commands, and onboard computing unit simulation models for controlling the operation of autonomous driving algorithms.

[0056] By constructing a simulation environment that includes static scenes and dynamic traffic flow, and configuring simulation models of simulated vehicles and their supporting sensors, actuators, and on-board computing units within it, a high-fidelity, controllable, and complete virtual test field is established for the performance evaluation of the simulation platform. This ensures that the entire evaluation process can be carried out in a benchmark environment that is close to reality and whose parameters can be accurately reproduced. As a result, the final performance index data has a solid foundation and good repeatability, avoiding evaluation bias caused by inconsistent test environments.

[0057] In some specific embodiments, a simulation environment comprising static scenes and dynamic traffic flow is constructed, specifically including: Import map data to generate a static scene that includes road networks and traffic signs; Dynamic traffic flow is injected based on preset vehicle density, pedestrian density, and behavior models; Configure adjustable weather and lighting systems to simulate different environmental conditions.

[0058] By further clarifying that the construction of the simulation environment should include three specific steps—importing map data to generate a static scene, injecting dynamic traffic flow based on preset parameters, and configuring an adjustable weather and lighting system—the construction process of the simulation environment becomes more operable and standardized. This ensures that the constructed test environment not only includes basic road elements but also simulates real traffic interactions and changing environmental conditions. Consequently, it greatly enhances the richness, realism, and challenge of the test scenarios on which the evaluation system relies, enabling the evaluation results to more comprehensively reflect the platform's robustness under different environments.

[0059] In some specific embodiments, importing map data specifically refers to importing map data in OpenDRIVE format; The behavioral models include the Intelligent Driver Model (IDM) and the Lane Change Model that minimizes overall braking induction (MOBIL). The system includes adjustable weather and lighting systems, including adjustments for light intensity, precipitation intensity, and fog concentration.

[0060] By specifying the import of OpenDRIVE format maps, the use of mature behavioral models such as IDM and MOBIL, and the adjustment of specific weather parameters such as light intensity, the system provides industry-standard data interfaces, validated traffic behavior simulation algorithms, and precisely controllable environmental variables for the construction of simulation environments. This enhances the standardization and scientific rigor of simulation environment construction, ensures consistency at the environmental foundation level among different evaluation projects or teams, and makes changes in environmental conditions quantifiable and repeatable, which is beneficial for attribution analysis.

[0061] In some specific embodiments, configuring a sensor simulation model for simulating the generation of perception data for the simulated vehicle specifically includes configuring at least one of the following models: The camera simulation model is configured to simulate image acquisition based on ray tracing rendering and apply lens distortion and motion blur effects. The lidar simulation model is configured to simulate point cloud acquisition based on ray projection technology and introduce noise and random point loss. The millimeter-wave radar simulation model is configured to simulate radar signal acquisition and simulate the Doppler effect and clutter based on the electromagnetic wave propagation model. The inertial measurement unit simulation model is configured to generate inertial data based on an error model that includes zero-bias stability parameters.

[0062] By specifically listing and defining the implementation principles and key simulation characteristics of various mainstream sensor simulation models such as cameras, LiDAR, millimeter-wave radar, and inertial measurement units, a clear technical path is provided for configuring high-fidelity perception data sources for simulated vehicles. This enables the simulation platform to generate realistic sensor data containing simulations of various noises, distortions, and physical effects, thereby ensuring that subsequent evaluations of autonomous driving algorithms and even the platform's own data processing capabilities are based on data inputs that closely resemble the characteristics of real sensors, thus improving the credibility of perception-level testing.

[0063] In some specific embodiments, the data format output by the camera simulation model is RAW12 format or H.264 compressed stream; The data output format of the lidar simulation model is either PCAP format or ROS PointCloud2 message; The data output by the millimeter-wave radar simulation model is in CAN FD frame format.

[0064] By clearly specifying that cameras output RAW12 or H.264 format, LiDAR outputs PCAP or ROSPointCloud2 messages, and millimeter-wave radar outputs CAN FD frame format, the output interfaces of various sensor simulation models are made consistent with the data formats of real vehicle systems or industry-standard middleware data. This ensures that the data streams generated by the simulation can be seamlessly connected and processed by downstream vehicle computing unit simulation models or real algorithm modules, enhancing the compatibility and practicality of the entire simulation testing chain and avoiding additional adaptation costs caused by data format mismatches.

[0065] In some specific embodiments, an actuator simulation model for executing control commands is configured for the simulated vehicle, specifically including configuring the following models: A simulation model of a steer-by-wire system is used to receive and parse steering wheel angle commands, and to simulate the dynamic response of the actuator through a transfer function. The brake-by-wire system simulation model is used to receive and parse braking pressure commands, simulate hydraulic response, and integrate the functional simulation of the anti-lock braking system (ABS) and the electronic stability control system (ESC).

[0066] By specifically defining the requirements for input command parsing, dynamic response simulation, and integrated advanced function simulation such as ABS / ESC in the simulation model of steer-by-wire and brake-by-wire systems, the simulation vehicle is equipped with an execution end that can accurately respond to control commands and simulate the dynamic characteristics and safety functions of real actuators. This allows the control commands generated by the autonomous driving algorithm to be executed and fed back in accordance with physical laws in the simulation, forming a true "vehicle-environment" closed loop. This enables accurate evaluation of the platform's performance in control command transmission, actuator response simulation, and vehicle dynamics closed-loop testing.

[0067] In some specific embodiments, a simulation model of the onboard computing unit used to control the operation of autonomous driving algorithms is configured, including the implementation of the following simulation process: Simulates in-vehicle network communication; Simulate the temporal synchronization and spatial fusion of multi-sensor data; Simulates the computational process of perception algorithms that include target detection and tracking; Simulate a rule-based and algorithm-based decision-making and planning process; Generating lateral and longitudinal control commands for simulated vehicles.

[0068] By requiring the simulation of the entire process in the vehicle computing unit simulation model, from vehicle network communication and spatiotemporal fusion of multi-sensor data to complete perception algorithm calculation, decision planning and control command generation, the simulation of the vehicle computing unit is upgraded from a simple interface forwarding or algorithm black box to a fine-grained simulation of the core data processing and decision logic of intelligent driving. This allows the evaluation system to deeply examine the simulation platform's ability to support complex vehicle computing architecture, heterogeneous data stream processing and real-time decision-making links, making the evaluation dimensions more in-depth and essential.

[0069] Step S2: Construct a platform evaluation system. The platform evaluation system includes several evaluation dimensions, including security, efficiency, response time, and data processing capabilities. Define at least one specific sub-indicator for each evaluation dimension; For each specific sub-indicator, determine its evaluation method and the target data items to be collected. The evaluation method includes quantitative calculation or qualitative scoring. Establish comprehensive evaluation rules to determine whether the overall performance of the platform meets the standards based on the evaluation results of each specific sub-indicator. The evaluation result is the value obtained using the corresponding evaluation method.

[0070] By pre-constructing a platform evaluation system that includes multiple evaluation dimensions such as security, efficiency, response time, and data processing capabilities, and defining specific sub-indicators, evaluation methods, and target data items to be collected for each dimension, and setting comprehensive evaluation rules for overall judgment, the originally potentially vague and subjective understanding of platform performance is transformed into a well-structured, quantitatively quantified, and clearly defined evaluation framework. This allows subsequent testing and data analysis to be targeted and ensures that the final evaluation conclusion is based on predefined and consistent objective standards.

[0071] In some specific embodiments, the specific sub-indicators of the security dimension in step S2 include: Collision accident frequency; Traffic violation rate; Success rate in responding to sudden obstacles within the Design Operational Domain (ODD); The specific sub-indicators of the efficiency dimension include: Percentage of autonomous driving mileage; Average vehicle speed; Calculate resource utilization; The specific sub-metrics of the response time dimension include: End-to-end delay from sensor data acquisition to control command generation; Emergency braking response time to sudden obstacles; The specific sub-indicators of the data processing capability dimension include: Data throughput of multi-source sensors; Perceptual model inference frame rate.

[0072] By further refining and defining a series of specific, operable, and observable sub-indicators for the four core evaluation dimensions of security, efficiency, response time, and data processing capability, the framework of the entire evaluation system becomes more robust and easier to implement. The performance of each dimension can be characterized by one or more specific and clearly defined indicators, greatly enhancing the comprehensiveness, specificity, and measurability of the evaluation system.

[0073] In some specific embodiments, the calculation formulas for each specific sub-indicator and the target data items to be collected include: Collision accident frequency The calculation formula is:

[0074] The number of collision events in the simulation test is represented by the following statistical standard: during the simulation, the number of events in which the simulated vehicle physically overlaps with an obstacle or road boundary is recorded as the number of collision events, regardless of the collision speed or severity. This represents the total mileage of the simulation test; The target data items to be collected are and ; Traffic violation rate The calculation formula is:

[0075] This indicates the number of traffic violation incidents that occurred during the simulation test; Indicates the total simulation test time; The target data items to be collected are and ; Autonomous driving mileage percentage The calculation formula is:

[0076] This represents the driving mileage in the autonomous driving mode during simulation testing; The target data items to be collected are and ; resource utilization The calculation formula is:

[0077] This represents the average CPU utilization during simulation testing. This represents the average memory usage during simulation testing. The target data items to be collected are and ; End-to-end delay The calculation formula is:

[0078] Indicates the total number of sampling points; Indicates the first The control commands for each sampling point generate timestamps; Indicates the first Timestamps of sensor data from each sampling point; The target data items to be collected are all. and ; Data throughput The calculation formula is:

[0079] This indicates the total amount of sensor data input to the interface; The target data items to be collected are and .

[0080] By providing precise mathematical formulas for key sub-indicators such as collision frequency, traffic violation rate, autonomous driving mileage ratio, resource utilization rate, end-to-end latency, and data throughput, and clearly indicating the specific target data items that need to be collected in simulation tests for each variable in the formula, the evaluation process is completely quantified and automated. This ensures that the calculation of each indicator is based entirely on objective data, eliminating the subjectivity of human judgment. At the same time, it provides clear input requirements for the design of the data acquisition system, ensuring the accuracy and repeatability of the evaluation.

[0081] In some specific embodiments, the comprehensive evaluation rules include: performing a comprehensive scenario score based on quantitative calculation results, and then determining compliance based on the comprehensive scenario score; Overall scene score The calculation formula is:

[0082]

[0083] in, This indicates the initial overall score for the scene. The score represents the security dimension score, which is obtained by normalizing the evaluation results of all specific sub-indicators under the security dimension and then summing them by weight. The efficiency dimension score is obtained by normalizing the evaluation results of all specific sub-indicators under the efficiency dimension and then summing them by weight. The score represents the response time dimension score, which is obtained by normalizing the evaluation results of all specific sub-indicators under the response time dimension and then summing them by weight. The score represents the data processing capability dimension score, which is obtained by normalizing the evaluation results of all specific sub-indicators under the data processing capability dimension and then summing them by weight. This represents the preset weighting coefficients for the security dimension; This represents the preset efficiency dimension weighting coefficient; This represents the preset weighting coefficient for the response time dimension; This represents the preset weight coefficient for the data processing capability dimension; satisfy ; This represents the theoretical maximum value of the overall scene score, which is equal to the initial overall scene score calculated when the evaluation results of all specific sub-indicators take the theoretical optimal value.

[0084] By designing a scenario-based comprehensive scoring model based on weighted summation, this model first normalizes and aggregates the evaluation results of sub-indicators under each dimension to obtain scores for each dimension. Then, it assigns weights according to the importance of different dimensions for comprehensive scoring and finally converts them into percentage scores. This provides a mathematical method to scientifically combine the evaluation results of multi-dimensional, heterogeneous, and different-dimension indicators into a single, intuitive comprehensive score, enabling intuitive quantitative comparison and classification of the overall performance of different simulation platforms or different configurations of the same platform.

[0085] In some specific embodiments, the weighting coefficients are configured according to the type of simulation test scenario: For security-critical scenarios, configure as , , , ; For efficiency-first scenarios, configure as follows , , , ; For comprehensive driving scenarios, the configuration is as follows: , , , ; Among them, safety-critical scenarios refer to simulation scenarios that are mainly designed to test the response and decision-making capabilities of autonomous driving systems in high-risk situations, including scenarios such as sudden braking of the vehicle in front, pedestrians suddenly appearing out of nowhere, and intersection conflicts. Efficiency-first scenarios refer to simulation scenarios that primarily aim to test the traffic efficiency and economy of autonomous driving systems on smooth roads, including highway cruising scenarios and urban expressway following scenarios. Comprehensive driving scenarios refer to simulation scenarios that integrate various traffic elements and interactions, with the main purpose of testing the comprehensive performance of autonomous driving systems in complex environments. These scenarios include urban road scenarios that include traffic lights, pedestrians, and non-motorized vehicles.

[0086] By pre-defining and configuring different weight coefficient combinations based on three typical testing objectives—safety-critical scenarios, efficiency-priority scenarios, and comprehensive driving scenarios—the comprehensive evaluation model possesses the ability to adapt to different scenarios. It can flexibly adjust the evaluation focus to match different testing objectives. For example, when testing the safety baseline, the safety indicators are given the highest weight, while when assessing traffic capacity, more attention is paid to efficiency indicators. This makes the evaluation conclusions no longer rigid and singular, but more targeted and practical, and can better serve the diverse R&D testing needs.

[0087] In some specific embodiments, determining compliance based on a comprehensive scenario score specifically includes: when At that time, it was determined to meet the standard; when If the condition is deemed met, the parameters need to be adjusted and the test repeated. when If so, it is judged as not meeting the standard.

[0088] By setting clear threshold ranges for the overall scenario score—75 points or above as meeting the standard, 60 to 75 points as conditionally meeting the standard, and below 60 points as failing the standard—a clear, direct, and operational judgment standard is provided for interpreting the overall score results. This transforms the evaluation conclusion from a vague numerical score into clear decision-making suggestions such as "meets the standard," "needs improvement," or "fails the standard." This greatly facilitates project management and quality control, and provides a direct basis for whether to accept the current simulation platform performance or whether to initiate an optimization process.

[0089] In some specific embodiments, the comprehensive evaluation rules also include a veto condition, which states that if any of the following conditions occur, the overall performance of the autonomous driving simulation platform will be directly determined to be substandard: A real collision occurred; The system crash rate exceeds 0.5%; End-to-end latency exceeds 100ms; The data loss rate exceeded 2%; The criteria for determining a realistic collision are: the simulated vehicle comes into contact with an obstacle or road boundary and meets any of the following conditions: The collision speed exceeded 5 meters per second; A collision interrupts the dynamics of the simulated vehicle, requiring an emergency stop or system reset. The collision event was recorded in the simulation platform's accident log.

[0090] By establishing a series of "one-vote veto" conditions, such as real collisions, excessive system crash rates, excessive end-to-end latency, and excessively high data loss rates, and providing specific judgment criteria for real collisions, a strict safety and reliability red line is superimposed on the comprehensive evaluation system. This ensures that the performance of any simulation platform with fatal defects in key safety items or basic stability will fail the overall evaluation, strengthening the bottom-line thinking and risk control capabilities of the evaluation system, and preventing the comprehensive scoring model from masking extreme but unacceptable risks due to weight allocation.

[0091] Step S3: Run the simulation test in the autonomous driving simulation platform. During the simulation test, data for all target data items are collected synchronously. The simulation test includes: An autonomous driving algorithm is configured in the vehicle computing unit simulation model, and a closed-loop simulation is initiated. During the process, the sensor simulation model generates perception data, which is then used by the autonomous driving algorithm to generate control commands. The actuator simulation model executes the control commands to control the movement of the simulated vehicle.

[0092] By actually running simulation tests in the constructed autonomous driving simulation platform and simultaneously collecting all target data items defined by the evaluation system during the test, while ensuring that the test process runs the real autonomous driving algorithm in the vehicle computing unit simulation model and forms a closed loop of sensor-decision-actuator, the evaluation system is closely integrated with the real platform operating load. This allows the collected performance data to truly and dynamically reflect the actual performance of the simulation platform in handling core autonomous driving tasks, rather than static or isolated parameters, thus enhancing the authenticity and effectiveness of the evaluation results.

[0093] In some specific embodiments, a benchmark autonomous driving algorithm model is configured as a test load in the autonomous driving simulation platform. Specifically, this includes selecting and loading an autonomous driving algorithm model with a clear interface specification. The model includes at least perception, decision planning and control modules, and its input and output interfaces are connected to the sensor simulation output and actuator simulation input of the simulation platform.

[0094] By explicitly specifying a benchmark autonomous driving algorithm model with clear interface specifications as the test load in the simulation platform, a stable and unified algorithm benchmark is provided for performance evaluation. This ensures that the applied computational load and algorithm behavior are consistent when evaluating different simulation platforms or different evaluations. As a result, the performance differences are mainly attributed to the differences in the capabilities of the simulation platforms themselves, rather than the fluctuations of the tested algorithm. This significantly improves the fairness, comparability, and reliability of the performance evaluation conclusions.

[0095] In some specific embodiments, the test cases used to run closed-loop simulation tests are automatically generated through the following steps: Receive test requirement descriptions as input in natural language or structured text; Based on predefined scenario description specifications, the test requirement description is parsed into a structured scenario element graph, which includes entities, attributes, and relationships. Instantiate and assign values ​​to the parameters in the scene element graph to generate an executable test script.

[0096] By describing the automated generation steps from natural language requirements to structured scenario element diagrams, and then to parameter instantiation to generate executable test scripts, this paper provides an efficient and standardized test case construction method. This method greatly reduces the workload of manually designing a large number of test scenarios, reduces human error, and can quickly respond to new test requirements. At the same time, it ensures that the generated test cases conform to predefined specifications, which is conducive to the accumulation, reuse and management of test cases, and improves the preparation efficiency and quality of the entire evaluation process.

[0097] In some specific embodiments, when collecting data for all target data items synchronously, a unified time synchronization mechanism is applied to the data sources of all target data items. The time synchronization mechanism adopts the Precise Time Protocol (PTP) or a timestamp injection method based on the simulated master clock to ensure that the timestamp alignment accuracy of the entire data link is better than 1 millisecond.

[0098] By requiring a unified high-precision time synchronization mechanism when collecting all target data items, and explicitly requiring timestamp alignment accuracy better than 1 millisecond, the system ensures that data streams from different modules and frequencies in the simulation environment have a high degree of consistency in the time dimension. This provides an accurate data foundation for subsequent calculations of indicators that are strictly dependent on time differences, such as end-to-end latency, and avoids calculation errors caused by data time asynchrony, thus fundamentally guaranteeing the accuracy of time-related performance indicator evaluation results.

[0099] Step S4: Based on the data of the target data item, calculate or evaluate the evaluation result of each specific sub-indicator according to the evaluation method of each specific sub-indicator.

[0100] Based on the target data items collected synchronously during simulation testing, the quantitative calculation or qualitative scoring methods defined for each specific sub-indicator in step S2 are strictly followed to process them one by one and obtain their evaluation results. This realizes the transformation of massive amounts of raw operating data into quantitative values ​​or levels that characterize the platform's performance in various aspects and can be directly used for comparison and judgment. This completes the key transformation from data to information and provides accurate and reliable input for the next step of comprehensive value judgment.

[0101] In some specific embodiments, the data of the target data item is first cleaned and aligned, and then the evaluation result of each specific sub-indicator is calculated or evaluated one by one according to the evaluation method of each specific sub-indicator. Data cleaning and alignment include: performing packet loss detection and outlier removal on the raw data of the target data items, and aligning data streams of different frequencies to a unified time series using interpolation methods.

[0102] By introducing data cleaning and alignment steps, including packet loss detection, outlier removal, and time series alignment through interpolation, before calculating the evaluation results of specific sub-indicators, invalid or interfering data generated during the simulation process due to communication jitter, system transient failures, etc., can be effectively removed. All data is unified to a regular timeline, thereby improving the quality and consistency of the dataset used for the final calculation. This makes the performance index results calculated based on this data more stable, reliable, and representative.

[0103] Step S5: Based on the comprehensive evaluation rules and the evaluation results of each specific sub-indicator, determine whether the overall performance of the platform meets the standards and draw an evaluation conclusion.

[0104] Based on the comprehensive evaluation rules pre-set in step S2, and combined with the evaluation results of each specific sub-indicator calculated in step S4, logical operations and judgments are performed to finally arrive at a clear conclusion on whether the overall performance of the platform meets the standards. This completes the final decision of the evaluation process, integrates the scattered and multi-dimensional indicator evaluation results into an overall judgment with clear guiding significance, thereby providing users with clear and direct decision support for performance acceptance or optimization direction, ending the entire evaluation process and outputting core value.

[0105] In some specific embodiments, during the data cleaning and alignment steps, spatial calibration verification is performed on the data before multi-sensor fusion. That is, the lidar point cloud is projected onto the camera image coordinate system using the known extrinsic parameter matrix, and the reprojection error is calculated. If the average error exceeds 2 pixels, the confidence level of the data in that period is marked as reduced.

[0106] By adding the calculation of the reprojection error from the LiDAR point cloud to the camera image using an extrinsic parameter matrix during the cleaning and alignment of data before multi-sensor fusion, and marking the data confidence level according to the error threshold, online verification and data quality assessment of the spatial calibration consistency between sensor simulation models are realized. This can promptly detect spatial inaccuracies in the perception data caused by model parameter drift or errors, and provide higher quality data input for the performance evaluation of subsequent fusion algorithms. It also indirectly evaluates the simulation platform's ability to maintain spatial consistency across multiple sensors.

[0107] In some specific embodiments, after the evaluation conclusion is reached, a structured test report is automatically generated, which includes: a description of the test scenario, the evaluation results of each specific sub-indicator, the overall scenario score, and the judgment conclusion.

[0108] By automatically generating a structured test report containing a description of the test scenario, results of each sub-indicator, comprehensive score, and judgment conclusion after the evaluation conclusion is reached, all key inputs, process data, and final conclusions of the entire evaluation process are fixed in a standardized and complete document format. This not only provides traceability of the evaluation results and facilitates review and analysis, but also saves time on manual compilation and writing by standardizing and automating report generation, improving the efficiency and professionalism of deliverables and making it easier to integrate the evaluation results into the R&D document system.

[0109] In some specific embodiments, this performance evaluation method is integrated into the continuous integration / continuous deployment (CI / CD) process; The autonomous driving simulation platform automatically executes steps S2 to S5 after each code submission or scheduled task trigger, and feeds back the evaluation conclusions and the evaluation results of each specific sub-indicator to the code repository or project management platform.

[0110] By integrating this performance evaluation method into the continuous integration / continuous deployment process and setting it to automatically execute the evaluation process and feed the results back to the development management platform after code submission or scheduled task triggering, the performance evaluation of the simulation platform is no longer an isolated, phased activity, but a normalized and automated quality gatekeeper integrated into the daily development rhythm. It can promptly detect performance regression issues caused by changes in platform code or configuration, and realize continuous monitoring and rapid feedback of the simulation platform's own performance, which strongly supports the agile development and high-quality iteration of the simulation platform software.

[0111] The following are embodiments of the performance evaluation system for the autonomous driving simulation platform provided in this application. This performance evaluation system for the autonomous driving simulation platform belongs to the same inventive concept as the performance evaluation methods in the above embodiments. For details not described in detail in the embodiments of the performance evaluation system, please refer to the embodiments of the performance evaluation methods for the autonomous driving simulation platform described above.

[0112] like Figure 2 As shown, the performance evaluation system of the autonomous driving simulation platform includes: The simulation platform construction module is used to build an autonomous driving simulation platform, including: building a simulation environment containing static scenes and dynamic traffic flow; configuring at least one simulation vehicle in the simulation environment; configuring a sensor simulation model for simulating perception data generation, an actuator simulation model for executing control commands, and an on-board computing unit simulation model for controlling the operation of autonomous driving algorithms for the simulation vehicle. The evaluation system construction module is used to build the platform evaluation system, which includes several evaluation dimensions, including security, efficiency, response time, and data processing capabilities. It defines at least one specific sub-indicator for each evaluation dimension; determines the evaluation method and target data items to be collected for each specific sub-indicator, with evaluation methods including quantitative calculation or qualitative scoring; and sets comprehensive evaluation rules to determine whether the overall platform performance meets the standards based on the evaluation results of each specific sub-indicator. The evaluation result is the numerical value obtained using the corresponding evaluation method. The simulation test execution module is used to run simulation tests in the autonomous driving simulation platform. During the simulation test, data of all target data items are collected synchronously. The simulation test includes: configuring the autonomous driving algorithm in the vehicle computing unit simulation model, starting the closed-loop simulation, during which the sensor simulation model generates perception data, and after the autonomous driving algorithm makes a decision, it generates control commands. The actuator simulation model executes the control commands to control the movement of the simulated vehicle. The evaluation result generation module is used to calculate or evaluate the evaluation result of each specific sub-indicator based on the data of the target data item and according to the evaluation method of each specific sub-indicator. The evaluation conclusion determination module is used to determine whether the overall performance of the platform meets the standards based on the comprehensive evaluation rules and the evaluation results of each specific sub-indicator, and to draw an evaluation conclusion.

[0113] The performance evaluation system in this embodiment is used to implement a performance evaluation method for an autonomous driving simulation platform.

[0114] This application also provides an electronic device for implementing the various embodiments of this application. Figure 3 To illustrate the hardware structure of an electronic device according to various embodiments of this application, as shown in the following diagram... Figure 3 As shown, the electronic device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor.

[0115] Those skilled in the art will understand that the electronic device structure involved in the embodiments of this application does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0116] In embodiments of this application, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0117] In this application embodiment, the processor can be implemented using at least one of an Application-Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such implementations can be implemented within a controller. For software implementations, implementations such as processes or functions can be implemented with separate software modules that allow the performance of at least one function or operation. The software code can be implemented by a software application (or program) written in any suitable programming language, and the software code can be stored in memory and executed by the controller.

[0118] In addition, the electronic device includes some functional modules not shown, which will not be described in detail here.

[0119] Those skilled in the art will understand that the various aspects of the electronic device provided in this application can be implemented as a system, method, or program product. Therefore, the various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0120] This application also provides a storage medium storing a program product capable of implementing a performance evaluation method for an autonomous driving simulation platform. In some possible implementations, various aspects of this application can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this application.

[0121] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0122] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A performance evaluation method for an autonomous driving simulation platform, characterized in that, include: S1. Construct a simulation environment that includes static scenes and dynamic traffic flow in the autonomous driving simulation platform; Configure at least one simulated vehicle in the simulation environment; The simulated vehicle is equipped with a sensor simulation model for simulating perception data generation, an actuator simulation model for executing control commands, and an onboard computing unit simulation model for controlling the operation of autonomous driving algorithms. S2. Construct a platform evaluation system, which includes several evaluation dimensions, including security, efficiency, response time, and data processing capabilities; Define at least one specific sub-indicator for each evaluation dimension; For each specific sub-indicator, determine its evaluation method and the target data items to be collected. The evaluation method includes quantitative calculation or qualitative scoring. Establish comprehensive evaluation rules to determine whether the overall performance of the platform meets the standards based on the evaluation results of each specific sub-indicator. The evaluation result is the value obtained using the corresponding evaluation method. S3. Run simulation tests in the autonomous driving simulation platform. During the simulation tests, data for all target data items are collected synchronously. The simulation tests include: The autonomous driving algorithm is configured in the vehicle computing unit simulation model, and the closed-loop simulation is started. During the process, the sensor simulation model generates perception data, which is then used by the autonomous driving algorithm to generate control commands. The actuator simulation model executes the control commands to control the movement of the simulated vehicle. S4. Based on the data of the target data item, calculate or evaluate the evaluation result of each specific sub-indicator according to the evaluation method of each specific sub-indicator; S5. Based on the comprehensive evaluation rules and the evaluation results of each specific sub-indicator, determine whether the overall performance of the platform meets the standards and draw an evaluation conclusion.

2. The performance evaluation method as described in claim 1, characterized in that, In step S2, the specific sub-indicators of the security dimension include: Collision accident frequency; Traffic violation rate; Success rate in responding to sudden obstacles within the designed operating domain; The specific sub-indicators of the efficiency dimension include: Percentage of autonomous driving mileage; Average vehicle speed; Calculate resource utilization; The specific sub-metrics of the response time dimension include: End-to-end delay from sensor data acquisition to control command generation; Emergency braking response time to sudden obstacles; The specific sub-indicators of the data processing capability dimension include: Data throughput of multi-source sensors; Perceptual model inference frame rate.

3. The performance evaluation method as described in claim 2, characterized in that, The calculation formulas for each specific sub-indicator and the target data items to be collected include: Collision accident frequency The calculation formula is: The number of collision events in the simulation test is represented by the following statistical standard: during the simulation, the number of events in which the simulated vehicle physically overlaps with an obstacle or road boundary is recorded as the number of collision events, regardless of the collision speed or severity. This represents the total mileage of the simulation test; The target data items to be collected are and ; Traffic violation rate The calculation formula is: This indicates the number of traffic violation incidents that occurred during the simulation test; Indicates the total simulation test time; The target data items to be collected are and ; Autonomous driving mileage percentage The calculation formula is: This represents the driving mileage in the autonomous driving mode during simulation testing; The target data items to be collected are and ; resource utilization The calculation formula is: This represents the average CPU utilization during simulation testing. This represents the average memory usage during simulation testing. The target data items to be collected are and ; End-to-end delay The calculation formula is: Indicates the total number of sampling points; Indicates the first The control commands for each sampling point generate timestamps; Indicates the first Timestamps of sensor data from each sampling point; The target data items to be collected are all. and ; Data throughput The calculation formula is: This indicates the total amount of sensor data input to the interface; The target data items to be collected are and .

4. The performance evaluation method as described in claim 1, characterized in that, The comprehensive evaluation rules include: conducting a comprehensive scenario score based on quantitative calculation results, and then determining compliance based on the comprehensive scenario score; Overall scene score The calculation formula is: in, This indicates the initial overall score for the scene. The score represents the security dimension score, which is obtained by normalizing the evaluation results of all specific sub-indicators under the security dimension and then summing them by weight. The efficiency dimension score is obtained by normalizing the evaluation results of all specific sub-indicators under the efficiency dimension and then summing them by weight. The score represents the response time dimension score, which is obtained by normalizing the evaluation results of all specific sub-indicators under the response time dimension and then summing them by weight. The score represents the data processing capability dimension score, which is obtained by normalizing the evaluation results of all specific sub-indicators under the data processing capability dimension and then summing them by weight. This represents the preset weighting coefficients for the security dimension; This represents the preset efficiency dimension weighting coefficient; This represents the preset weighting coefficient for the response time dimension; This represents the preset weight coefficient for the data processing capability dimension; satisfy ; This represents the theoretical maximum value of the overall scene score, which is equal to the initial overall scene score calculated when the evaluation results of all specific sub-indicators take the theoretical optimal value.

5. The performance evaluation method as described in claim 4, characterized in that, The criteria for determining compliance based on a comprehensive scenario-based score include: when At that time, it was determined to meet the standard; when If the condition is deemed met, the parameters need to be adjusted and the test repeated. when If so, it is judged as not meeting the standard.

6. The performance evaluation method as described in claim 4, characterized in that, The comprehensive evaluation rules also include a veto condition. A veto condition is that if any of the following situations occur, the overall performance of the autonomous driving simulation platform will be directly determined to be substandard: A real collision occurred; The system crash rate exceeds 0.5%; End-to-end latency exceeds 100ms; The data loss rate exceeded 2%; The criteria for determining a realistic collision are: the simulated vehicle comes into contact with an obstacle or road boundary and meets any of the following conditions: The collision speed exceeded 5 meters per second; A collision interrupts the dynamics of the simulated vehicle, requiring an emergency stop or system reset. The collision event was recorded in the simulation platform's accident log.

7. The performance evaluation method as described in claim 6, characterized in that, The performance evaluation method is integrated into the continuous integration / continuous deployment process; The autonomous driving simulation platform automatically executes steps S2 to S5 after each code submission or scheduled task trigger, and feeds back the evaluation conclusions and the evaluation results of each specific sub-indicator to the code repository or project management platform.

8. A performance evaluation system for an autonomous driving simulation platform, characterized in that, To implement the performance evaluation method as described in any one of claims 1-7, the method includes: The simulation platform construction module is used to build an autonomous driving simulation platform, including: building a simulation environment containing static scenes and dynamic traffic flow; configuring at least one simulation vehicle in the simulation environment; configuring a sensor simulation model for simulating perception data generation, an actuator simulation model for executing control commands, and an on-board computing unit simulation model for controlling the operation of autonomous driving algorithms for the simulation vehicle. The evaluation system construction module is used to build the platform evaluation system, which includes several evaluation dimensions, including security, efficiency, response time, and data processing capabilities. It defines at least one specific sub-indicator for each evaluation dimension; determines the evaluation method and target data items to be collected for each specific sub-indicator, with evaluation methods including quantitative calculation or qualitative scoring; and sets comprehensive evaluation rules to determine whether the overall platform performance meets the standards based on the evaluation results of each specific sub-indicator. The evaluation result is the numerical value obtained using the corresponding evaluation method. The simulation test execution module is used to run simulation tests in the autonomous driving simulation platform. During the simulation test, data of all target data items are collected synchronously. The simulation test includes: configuring the autonomous driving algorithm in the vehicle computing unit simulation model, starting the closed-loop simulation, during which the sensor simulation model generates perception data, and after the autonomous driving algorithm makes a decision, it generates control commands. The actuator simulation model executes the control commands to control the movement of the simulated vehicle. The evaluation result generation module is used to calculate or evaluate the evaluation result of each specific sub-indicator based on the data of the target data item and according to the evaluation method of each specific sub-indicator. The evaluation conclusion determination module is used to determine whether the overall performance of the platform meets the standards based on the comprehensive evaluation rules and the evaluation results of each specific sub-indicator, and to draw an evaluation conclusion.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the steps of the performance evaluation method as described in any one of claims 1-7 when executing a computer program.

10. A storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the steps of the performance evaluation method as described in any one of claims 1-7.

Images