A real-time evaluation method and system for autonomous driving simulation testing

Abstract

This invention relates to the field of real-time evaluation technology and provides a real-time evaluation method for autonomous driving simulation testing, comprising the following steps: S1: Constructing an evaluation index set containing several evaluation indicators corresponding to the simulation scenario; S2: During the execution of the autonomous driving simulation task based on the simulation scenario, based on the simulation data generated by the continuously updated simulation test progress, when the execution condition of any one of the evaluation indicators is met, and the evaluation of the preceding evaluation indicators on which the current evaluation indicator depends is completed, the current evaluation indicator is evaluated. The above technical solution enables real-time evaluation in autonomous driving simulation testing, effectively reducing the complexity of the evaluation system and improving the efficiency of evaluation, thereby enhancing the overall reliability and efficiency of autonomous driving.

Description

Technical Field

[0001] This invention relates to the technical field of real-time evaluation, and in particular to a real-time evaluation method and system for autonomous driving simulation testing. Background Technology

[0002] Autonomous driving technology is a crucial research area in the field of artificial intelligence. Its goal is to enable vehicles to autonomously perceive their surroundings, understand traffic rules, plan routes, and control their movement without human driver intervention, using technologies such as computer vision, machine learning, and deep learning. The primary requirement for autonomous driving technology to operate on public roads is safety and reliability. This necessitates extensive simulation testing, verification, evaluation, and optimization. Evaluation systems are essential to ensure the safety, compliance, and user trust and acceptance of autonomous driving systems, which is crucial for promoting the development and commercial application of autonomous driving technology. However, due to the complexity and variability of autonomous driving simulation scenarios, current evaluation schemes still require waiting until the simulation is completed and using complete simulation data as the basic evaluation data. Real-time evaluation remains a challenge, and finding an efficient and easily scalable way to integrate with a real-time evaluation system is a pressing issue that needs to be addressed.

[0003] In existing technologies, from a time span perspective, autonomous driving evaluation systems can effectively evaluate the entire simulation test lifecycle, a portion of the simulation test lifecycle, and specific simulation test lifecycles. Existing technologies offer numerous reliable solutions for the entire simulation test lifecycle, which, simply put, rely on the overall evaluation based on the data files or other data structures generated after the simulation test is completed; these are essentially evaluations conducted after the autonomous driving simulation ends. Furthermore, such as... Figure 1 As shown, the evaluation is performed at each simulation time point after the entire simulation test lifecycle has ended. Furthermore, evaluations in existing technologies are generally conducted for pre-defined specific evaluation metrics. The disadvantages of existing evaluation schemes are as follows:

[0004] (1) Poor real-time performance: It is impossible to achieve real-time evaluation of autonomous driving.

[0005] (2) Poor scalability: Most existing technologies are based on specific evaluations to achieve targeted evaluations. They are difficult to expand when faced with diverse evaluation needs, and are prone to redundant calculations.

[0006] (3) Poor realism: Existing evaluation schemes only consider preset evaluation criteria and ignore other challenges that autonomous driving simulation may face in real-world environments. In other words, existing technologies cannot dynamically adjust evaluation indicators.

[0007] (4) Lack of standardization: The lack of a unified standard implementation evaluation method in the existing technology makes it difficult for different combinations of evaluation indicators to be of reference, increasing the difficulty of dynamically adjusting the evaluation scheme. Summary of the Invention

[0008] To address the aforementioned problems, the present invention aims to provide a real-time evaluation method and system for autonomous driving simulation testing. This method enables real-time evaluation during autonomous driving simulation testing, effectively reducing the complexity of the evaluation system and improving evaluation efficiency, thereby enhancing the overall reliability and efficiency of autonomous driving.

[0009] The above-mentioned objective of this invention is achieved through the following technical solutions:

[0010] A real-time evaluation method for autonomous driving simulation testing includes the following steps:

[0011] S1: Construct a set of evaluation metrics corresponding to the simulation scenario, which includes several evaluation indicators;

[0012] S2: During the execution of the autonomous driving simulation task based on the simulation scenario, according to the simulation data generated by the continuously updated simulation test progress, when the evaluation index execution conditions of any one of the evaluation indices are met, and the evaluation of the preceding evaluation index on which the current evaluation index depends is completed, the current evaluation index is evaluated.

[0013] Further, step S2 specifically includes:

[0014] S21: During the execution of the autonomous driving simulation task, after the evaluation subsystem receives the simulation data from the simulation subsystem for the first time, it constructs an evaluation data structure containing all the evaluation indicators based on the set of evaluation indicators corresponding to the simulation scenario. This structure describes the real-time evaluation of the autonomous driving simulation task and includes information such as the dependencies between the evaluation indicators and the execution conditions of the evaluation indicators.

[0015] S22: During the execution of the autonomous driving simulation task, when the evaluation subsystem receives the simulation data from the simulation subsystem for the first time, it updates the simulation data to the data storage unit corresponding to the evaluation data structure, and simultaneously triggers the evaluation process of the simulation scenario based on the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information.

[0016] Furthermore, prior to step S21, the following steps are also included:

[0017] Construct a message queue (MQ) for data transfer between the simulation subsystem and the evaluation subsystem;

[0018] The simulation subsystem asynchronously sends the simulation data, including sensor data, map data, and vehicle posture data, to the message queue MQ. The evaluation subsystem obtains the simulation data by subscribing to the message queue MQ.

[0019] The evaluation submodule asynchronously sends the evaluation result information to the message queue MQ, and the simulation subsystem obtains the evaluation result information by subscribing to the message queue MQ.

[0020] Further, in step S22, the evaluation process of the simulation scenario is run based on the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information, specifically as follows:

[0021] The evaluation process sequentially polls each evaluation indicator in the evaluation data structure, retrieves the data required to evaluate the current evaluation indicator from the data storage unit, determines whether the execution conditions of the current evaluation indicator, including data requirements and other prerequisite requirements, are met, and determines whether the evaluation of the preceding evaluation indicators on which the current evaluation indicator depends has been completed. If so, the evaluation process of the current evaluation indicator is executed, the evaluation result information is obtained, and the evaluation result information is saved to the data storage unit for sharing with other evaluation indicators during evaluation. Otherwise, the current evaluation indicator continues to wait for the next polling.

[0022] Further, in step S21, an evaluation data structure containing all the evaluation indicators is constructed based on the obtained set of evaluation indicators corresponding to the simulation scenario. This structure describes the real-time evaluation of the simulation scenario and includes information such as the dependencies between evaluation indicators and the execution conditions of the evaluation indicators. Specifically:

[0023] Using a behavior tree as the evaluation data structure, all evaluation indicators in the evaluation indicator set are obtained. In the evaluation indicator library, the indicator data, the dependency relationship between indicators and indicators and the indicator calculation information corresponding to each evaluation indicator are obtained. The evaluation indicator library is a predefined database that stores the relevant information of all evaluation indicators.

[0024] Several evaluation subtrees, the same number as the number of evaluation indicators, are defined in parallel under the root node of the behavior tree. In each evaluation subtree, a condition node and a behavior node are defined. The condition node is used to determine whether the execution conditions of the current evaluation indicator, including data requirements and other prerequisite requirements, are met, and whether the evaluation of the prerequisite evaluation indicators on which the current evaluation indicator depends has been completed. The behavior node is used to execute the evaluation process of the current evaluation indicator based on the indicator calculation information after the condition node has passed the judgment.

[0025] A behavior tree blackboard is used as the data storage unit, and the behavior tree blackboard is initialized to define the metadata and storage structure of each evaluation subtree;

[0026] Simultaneously, the feedback data structure for the evaluation result information fed back by the behavior node is initialized.

[0027] Furthermore, when the behavior tree is used as the evaluation data structure and the behavior tree blackboard is used as the data storage unit, the evaluation process of the simulation scenario is run to obtain the evaluation result information, specifically as follows:

[0028] Run the pre-constructed behavior tree;

[0029] Obtain the data required to evaluate all current evaluation metrics from the behavior tree blackboard;

[0030] Each evaluation subtree in the behavior tree is polled in turn. The condition node is used to determine whether the execution conditions of the current evaluation indicator, including data requirements and other prerequisite requirements, are met, and whether the previous evaluation indicators on which the current evaluation indicator depends have been evaluated. After the condition node is judged to be successful, the behavior node is used to execute the evaluation process of the current evaluation indicator based on the indicator calculation information.

[0031] The evaluation result information is output and stored in the behavior tree blackboard using the feedback data structure.

[0032] Furthermore, real-time evaluation methods for autonomous driving simulation testing also include:

[0033] A real-time distributed communication framework, including ROS and Cyber ​​RT, is used to establish communication between the simulation subsystem and the evaluation subsystem.

[0034] A state machine, including a finite state machine (FSM) and a finite state automaton (FSA), is used as the evaluation data structure.

[0035] A real-time evaluation system for autonomous driving simulation testing, used to perform the real-time evaluation method for autonomous driving simulation testing as described above, includes:

[0036] The indicator set construction module is used to construct an evaluation indicator set containing several evaluation indicators corresponding to the simulation scenario.

[0037] The evaluation index evaluation module is used to evaluate the current evaluation index during the execution of the autonomous driving simulation task based on the simulation scenario, based on the simulation data generated by the continuously updated simulation test progress. When the evaluation index execution conditions of any evaluation index are met, and the evaluation of the previous evaluation index on which the current evaluation index depends is completed, the current evaluation index is evaluated.

[0038] A computer device includes a memory and one or more processors, the memory storing computer code that, when executed by the one or more processors, causes the one or more processors to perform the method described above.

[0039] A computer-readable storage medium storing computer code that, when executed, performs the method described above.

[0040] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0041] (1) Improve the efficiency of autonomous driving evaluation: The real-time evaluation method for autonomous driving simulation testing provided by this invention enables real-time evaluation and simulation testing to be carried out simultaneously, which can reduce the overall process time of autonomous driving safety verification.

[0042] (2) Improve the reliability of autonomous driving evaluation: This invention can enhance the real reliability of the system by using real-time evaluation results feedback.

[0043] (3) Improve the adaptability and flexibility of autonomous driving evaluation: This invention can dynamically configure the indicators of autonomous driving evaluation according to different test content and driving environment, thereby improving the adaptability and flexibility of the self-evaluation system.

[0044] (4) Enhance the scalability of autonomous driving assessment: This invention uses behavior tree for real-time assessment. Thanks to the efficient data organization and dynamism of the tree structure, behavior tree can adjust the assessment content in a very efficient and simple way.

[0045] (5) Promote the improvement of autonomous driving evaluation technology: This solution provides a method to achieve real-time evaluation, thereby improving the reliability and efficiency of the evaluation system, assisting the autonomous driving evaluation system to discover and improve its weaknesses more quickly, and thus promoting the improvement of autonomous driving evaluation technology. Attached Figure Description

[0046] Figure 1 This is a schematic diagram illustrating the evaluation of autonomous driving simulation testing in existing technologies;

[0047] Figure 2 This is an overall flowchart of the real-time evaluation method for autonomous driving simulation testing according to the present invention;

[0048] Figure 3 This is a flowchart of the real-time evaluation method for autonomous driving simulation testing according to the present invention.

[0049] Figure 4 This is a diagram illustrating the behavior tree construction process of this invention;

[0050] Figure 5 This is an overall structural diagram of the real-time evaluation system for autonomous driving simulation testing according to the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0053] Explanation of technical terms involved in this invention:

[0054] Behavior Tree: A behavior tree (BT) is a tree-like structure used to describe behavior and manage behavior transitions. It is widely used in artificial intelligence, especially in game development and robot control. A behavior tree represents complex behavioral logic through a series of nodes, which can be conditional statements, action executions, or combinations of multiple sub-behaviors.

[0055] Autonomous driving simulation: Using computer simulation technology to simulate the behavior of autonomous vehicles in different scenarios and situations.

[0056] Combination strategies: These are used within the behavior tree to organize other nodes and form complex behavioral structures. Common combination strategies include sequential completion, selection completion, and conditional completion.

[0057] (1) Complete in sequence

[0058] Definition: Sequential nodes execute their child nodes in sequence until one child node fails or all child nodes succeed.

[0059] Application scenarios: Used for a series of tasks that must be completed in a specific order. For example, an autonomous vehicle needs to slow down before entering an intersection, then detect traffic lights, and finally pass through the intersection.

[0060] (2) Select Complete

[0061] Definition: Select a node and execute its child nodes in sequence until one child node succeeds or all child nodes fail.

[0062] Application scenario: Used for selective task execution. For example, when an autonomous vehicle encounters an obstacle, it can choose to turn left, turn right, or stop; as long as one of these actions is successful, the task is complete.

[0063] (3) Conditions are met

[0064] Definition: A condition node determines whether to execute its child nodes based on specific conditions. A child node will only be executed if the condition is met.

[0065] Application scenarios: Tasks that are performed only under specific conditions. For example, autonomous vehicles only need to slow down when their speed exceeds a certain threshold.

[0066] Message Queues (MQ): A communication pattern used to pass messages between different applications, decoupling dependencies between applications, improving system reliability and scalability, and supporting message persistence and asynchronous transmission. Common message queues include RabbitMQ, Kafka, and Redis.

[0067] Blackboard: Shared memory within the behavior tree used for data exchange. Blackboard data is stored in key-value pairs. Sending and receiving tree nodes communicate by reading and writing the same key.

[0068] Nodes: The basic elements that make up a behavior tree, used to describe behaviors and their organizational structure. Common behavior tree nodes include: behavior nodes, condition nodes, and composite nodes.

[0069] Metadata: Data that describes data. It can be understood as the attributes or characteristics of data and can help to better manage and use data.

[0070] This patented technical solution addresses the pain point of struggling to develop a standardized and efficient real-time evaluation method when faced with complex and diverse evaluation metrics in the field of autonomous driving. Utilizing behavior trees, an efficient data organization method, it provides a standardized and easily scalable real-time evaluation approach. Simultaneously, by employing message queues (MQ) for data transmission, it minimizes the impact on the efficiency of autonomous driving simulation testing while generating evaluation results in real time, enabling the early detection of potential problems in key aspects of the autonomous driving system.

[0071] The following is an illustration through specific examples:

[0072] First Embodiment

[0073] like Figure 2 and 3 As shown, this embodiment provides a real-time evaluation method for autonomous driving simulation testing, including the following steps:

[0074] S1: Construct a set of evaluation metrics corresponding to the simulation scenario, which includes several evaluation indicators.

[0075] Specifically, in this embodiment, after starting the autonomous driving simulation task, the simulation subsystem first uses the scenario building module to build a simulation scenario and a set of evaluation indicators corresponding to the current simulation scenario, which includes several evaluation indicators. Then, it calls the scenario running module in the simulation subsystem and loads the real-time display module of evaluation data in the simulation subsystem.

[0076] Specifically, for the evaluation indicators in the evaluation indicator set, this embodiment can dynamically configure the autonomous driving evaluation indicators according to different test content and driving environment, and adaptively adjust the evaluation indicators, thereby improving the adaptability and flexibility of the evaluation system.

[0077] For example, in a specific implementation, the following four evaluations are set in the evaluation metric set:

[0078] (1) Safety assessment: If there is no collision, perform a safety assessment;

[0079] (2) Reaction time assessment: If there is no collision, perform a safety assessment;

[0080] (3) Path deviation assessment: If the reaction time assessment passes, calculate the path deviation;

[0081] (4) Speed ​​control assessment: If the path deviation assessment passes, check the speed control.

[0082] S2: During the execution of the autonomous driving simulation task based on the simulation scenario, according to the simulation data generated by the continuously updated simulation test progress, when the evaluation index execution conditions of any one of the evaluation indices are met, and the evaluation of the preceding evaluation index on which the current evaluation index depends is completed, the current evaluation index is evaluated.

[0083] Before executing step S21, this embodiment includes: constructing a message queue (MQ) for data transfer between the simulation subsystem and the evaluation subsystem; the simulation subsystem asynchronously sending the simulation data, including sensor data, map data, and vehicle posture data, to the message queue MQ; the evaluation subsystem obtaining the simulation data by subscribing to the message queue MQ; and the evaluation submodule asynchronously sending the evaluation result information to the message queue MQ; the simulation subsystem obtaining the evaluation result information by subscribing to the message queue MQ, and updating and displaying it in real time in the real-time evaluation data display module. Using a message queue MQ minimizes the performance impact of data transmission on the simulation scene. By using MQ as the data interaction between the scene operation module and the evaluation module, performance degradation caused by data transmission is avoided, while reducing the pressure on the evaluation data receiving module. Furthermore, setting up a real-time evaluation data display module allows for the rapid detection of potential problems in the autonomous driving system, improving the reliability of the evaluation system.

[0084] In this embodiment, step S2 specifically includes:

[0085] S21: During the execution of the autonomous driving simulation task, after the evaluation subsystem receives the simulation data from the simulation subsystem for the first time, it constructs an evaluation data structure containing all the evaluation indicators based on the set of evaluation indicators corresponding to the simulation scenario. This structure describes the real-time evaluation of the autonomous driving simulation task and includes information such as the dependencies between the evaluation indicators and the execution conditions of the evaluation indicators.

[0086] Specifically, in this embodiment, after the data receiving module in the evaluation subsystem receives the simulation data from the simulation subsystem for the first time, it automatically constructs an evaluation data structure containing all evaluation indicators.

[0087] One specific implementation involves using a behavior tree to construct the evaluation data structure. When the data receiving module first acquires the simulation data, it forwards it to the evaluation behavior tree construction module. This module then retrieves the corresponding evaluation metrics from the evaluation metric library based on the acquired simulation scenario, along with the relevant metric data and calculation information, and constructs a complete behavior tree. Using a behavior tree as the organizational form for real-time evaluation increases the clarity of the evaluation structure and improves the efficiency and scalability of real-time evaluation. Figure 4 As shown, the specific process is as follows:

[0088] Using a behavior tree as the evaluation data structure, all evaluation indicators in the evaluation indicator set are obtained. In the evaluation indicator library, the indicator data, the dependency relationship between indicators and indicators and the indicator calculation information corresponding to each evaluation indicator are obtained. The evaluation indicator library is a predefined database that stores the relevant information of all evaluation indicators.

[0089] Several evaluation subtrees, the same number as the number of evaluation indicators, are defined in parallel under the root node of the behavior tree. In each evaluation subtree, a condition node and a behavior node are defined. The condition node is used to determine whether the execution conditions of the current evaluation indicator, including data requirements and other prerequisite requirements, are met, and whether the evaluation of the prerequisite evaluation indicators on which the current evaluation indicator depends has been completed. The behavior node is used to execute the evaluation process of the current evaluation indicator based on the indicator calculation information after the condition node has passed the judgment.

[0090] A behavior tree blackboard is used as the data storage unit, and the behavior tree blackboard is initialized to define the metadata and storage structure of each evaluation subtree;

[0091] Simultaneously, the feedback data structure for the evaluation result information fed back by the behavior node is initialized.

[0092] In addition, it is necessary to design solutions for evaluating anomalies, complete testing and debugging, and ensure that the behavior tree construction is free of anomalies.

[0093] To illustrate the process of building a behavior tree, consider this example:

[0094] 1. Analyze and evaluate the needs and dependencies of the indicators.

[0095] Suppose we want to evaluate the performance of autonomous vehicles in urban traffic scenarios, and the evaluation metrics include:

[0096] (1) Safety assessment: If there is no collision, perform a safety assessment;

[0097] (2) Response time assessment: If the safety assessment is passed, perform the response time assessment;

[0098] (3) Path deviation assessment: If the reaction time assessment passes, calculate the path deviation;

[0099] (4) Speed ​​control assessment: If the path deviation assessment passes, check the speed control.

[0100] 2. Construct the evaluation subtree

[0101] A: Data requirements, dependencies, and evaluation behavior for each subtree.

[0102] (1) Security assessment subtree

[0103] Data requirements: Vehicle collision detection data.

[0104] Dependency: Directly dependent on collision detection data.

[0105] Assessment behavior: Check if a collision has occurred.

[0106] (2) Reaction time evaluation subtree

[0107] Data requirements: Obstacle or red light detection data.

[0108] Dependencies: Security assessment passed.

[0109] Evaluation behavior: Calculate the vehicle's reaction time when it detects an obstacle or a red light.

[0110] (3) Path deviation evaluation subtree

[0111] Data requirements: Vehicle location and planned route data.

[0112] Dependencies: Response time assessment passed.

[0113] Evaluation behavior: Calculate the deviation between the vehicle's travel path and the predetermined path.

[0114] (4) Speed ​​control evaluation subtree

[0115] Data requirement: Vehicle speed data.

[0116] Dependency: Path deviation assessment passed.

[0117] Assessment behavior: Check whether the vehicle speed is within the prescribed range.

[0118] B: The final generated behavior tree structure is:

[0119] (1) Security assessment subtree:

[0120] Condition Node 1: Read the collision detection data in the blackboard and check if there is no collision.

[0121] Behavior Node 1: If there is no collision, perform a safety assessment and store the results on the blackboard.

[0122] (2) Response time evaluation subtree (dependency safety evaluation):

[0123] Condition Node 2: Check the security assessment results on the blackboard to ensure that the security test is passed.

[0124] Behavior Node 2: If the security assessment passes, calculate the reaction time and store the result on the blackboard.

[0125] (3) Path deviation evaluation subtree (reaction time dependent evaluation):

[0126] Condition Node 3: Check the reaction time assessment results on the blackboard to ensure that the reaction time is within a reasonable range.

[0127] Behavior Node 3: If the reaction time assessment passes, calculate the path deviation and store the result on the blackboard.

[0128] (4) Speed ​​control evaluation subtree (dependency path deviation evaluation):

[0129] Condition Node 4: Check the path deviation evaluation results on the blackboard to ensure that the path deviation is within a reasonable range.

[0130] Behavior Node 4: If the path deviation assessment passes, check the speed control and store the results on the blackboard.

[0131] 3. Initialize the behavior tree blackboard

[0132] The behavior tree blackboard is a shared memory space used to transfer data between different nodes in the behavior tree. We need to define metadata and storage structure for each evaluation subtree.

[0133] (1) Security assessment subtree:

[0134] Metadata: Collision detection data.

[0135] Storage structure: Stores the most recent collision detection results.

[0136] (2) Reaction time evaluation subtree:

[0137] Metadata: Obstacle or red light detection data.

[0138] Storage structure: Stores the results of the most recent obstacle or red light detection.

[0139] (3) Path deviation evaluation subtree:

[0140] Metadata: Vehicle location and scheduled route data.

[0141] Storage structure: Stores the latest vehicle location and planned route data.

[0142] (4) Speed ​​control evaluation subtree:

[0143] Metadata: Vehicle speed data.

[0144] Storage structure: Stores the most recent vehicle speed data.

[0145] 4. Initialize the evaluation feedback data structure

[0146] The evaluation feedback data structure is used to record the results of each evaluation indicator for subsequent analysis and presentation.

[0147]

[0148] Furthermore, when the behavior tree is used as the evaluation data structure and the behavior tree blackboard is used as the data storage unit, the evaluation process of the simulation scenario is run to obtain the evaluation result information, specifically as follows:

[0149] Run the pre-constructed behavior tree;

[0150] Obtain the data required to evaluate all current evaluation metrics from the behavior tree blackboard;

[0151] Each evaluation subtree in the behavior tree is polled in turn. The condition node is used to determine whether the execution conditions of the current evaluation indicator, including data requirements and other prerequisite requirements, are met, and whether the previous evaluation indicators on which the current evaluation indicator depends have been evaluated. After the condition node is judged to be successful, the behavior node is used to execute the evaluation process of the current evaluation indicator based on the indicator calculation information.

[0152] The evaluation result information is output and stored in the behavior tree blackboard using the feedback data structure.

[0153] S22: During the execution of the autonomous driving simulation task, when the evaluation subsystem receives the simulation data from the simulation subsystem for the first time, it updates the simulation data to the data storage unit corresponding to the evaluation data structure, and simultaneously triggers the evaluation process of the simulation scenario based on the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information.

[0154] In step S22, the evaluation process of the simulation scenario is run based on the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information, specifically:

[0155] The evaluation process sequentially polls each evaluation indicator in the evaluation data structure, retrieves the data required to evaluate the current evaluation indicator from the data storage unit, determines whether the execution conditions of the current evaluation indicator, including data requirements and other prerequisite requirements, are met, and determines whether the evaluation of the preceding evaluation indicators on which the current evaluation indicator depends has been completed. If so, the evaluation process of the current evaluation indicator is executed, the evaluation result information is obtained, and the evaluation result information is saved to the data storage unit for sharing with other evaluation indicators during evaluation. Otherwise, the current evaluation indicator continues to wait for the next polling.

[0156] Furthermore, this invention can also employ a real-time distributed communication framework, including ROS and Cyber ​​RT, to establish communication between the simulation subsystem and the evaluation subsystem, replacing the message queue (MQ) model. ROS (Robot Operating System) is a framework that provides operating system functionality for robot development, supporting hardware abstraction, low-level device control, implementation of common functions, message passing, and packet management. Cyber ​​RT is a high-performance distributed communication framework used in Baidu's Apollo autonomous driving platform, specifically designed for high real-time and high-concurrency scenarios.

[0157] Alternatively, state machines, including finite state machines (FSMs) and finite state automata (FSAs), can be used as the evaluation data structure.

[0158] Example: FSM in an autonomous driving simulation system

[0159] 1. States

[0160] Initialization: The system starts up and prepares to receive data.

[0161] Data reception: Receive simulation data and perform preliminary processing.

[0162] Safety assessment: Assess the safety of the vehicle.

[0163] Reaction time assessment: Assess the vehicle's reaction time.

[0164] Path deviation assessment: assesses the deviation of the vehicle's driving path.

[0165] Speed ​​control assessment: Assess the vehicle's speed control performance.

[0166] Output results: Output the evaluation results and send them to the simulation module.

[0167] (2) Initial State

[0168] initialization

[0169] (3) Events or Inputs

[0170] Received data

[0171] Data valid

[0172] Security assessment completed

[0173] Reaction time assessment completed

[0174] Path deviation assessment completed

[0175] Speed ​​control evaluation completed

[0176] Results output complete.

[0177] (4) State transitions

[0178] (Initialization, data received) → Data reception

[0179] (Data received, data valid) → Security assessment

[0180] (Safety assessment, safety assessment completed) → Response time assessment

[0181] (Reaction time assessment, reaction time assessment completed) → Path deviation assessment

[0182] (Path deviation assessment, path deviation assessment completed) → Speed ​​control assessment

[0183] (Speed ​​control evaluation, speed control evaluation complete) → Results output

[0184] (Result output, result output complete) → Initialization

[0185] (5) Final States

[0186] No specific termination state (system loops)

[0187] Second Embodiment

[0188] like Figure 5 As shown, this embodiment provides a real-time evaluation system for autonomous driving simulation testing, used to perform a real-time evaluation method for autonomous driving simulation testing as described in the first embodiment, comprising:

[0189] Indicator set construction module 1 is used to construct an evaluation indicator set containing several evaluation indicators corresponding to the simulation scenario;

[0190] The evaluation index evaluation module 2 is used to evaluate the current evaluation index during the execution of the autonomous driving simulation task based on the simulation scenario, based on the simulation data generated by the continuously updated simulation test progress, when the evaluation index execution conditions of any evaluation index are met, and the evaluation of the previous evaluation index on which the current evaluation index depends is completed.

[0191] A computer-readable storage medium stores computer code that, when executed, performs the methods described above. Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0192] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

[0193] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0194] It should be noted that the above embodiments can be freely combined as needed. The above description is only a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A real-time evaluation method for autonomous driving simulation testing, characterized in that, Includes the following steps: S1: Construct a set of evaluation metrics corresponding to the simulation scenario, which includes several evaluation indicators; S2: During the execution of the autonomous driving simulation task based on the simulation scenario, according to the simulation data generated by the continuously updated simulation test progress, when the evaluation index execution conditions of any one of the evaluation indices are met, and the evaluation of the previous evaluation index on which the current evaluation index depends is completed, the current evaluation index is evaluated. Step S2, specifically: S21: During the execution of the autonomous driving simulation task, after the evaluation subsystem receives the simulation data from the simulation subsystem for the first time, it constructs an evaluation data structure containing all the evaluation indicators based on the set of evaluation indicators corresponding to the simulation scenario. This structure describes the real-time evaluation of the autonomous driving simulation task and includes information such as the dependencies between the evaluation indicators and the execution conditions of the evaluation indicators. S22: During the execution of the autonomous driving simulation task, when the evaluation subsystem receives the simulation data from the simulation subsystem for the first time, it updates the simulation data to the data storage unit corresponding to the evaluation data structure, and simultaneously triggers the evaluation process of running the simulation scenario based on the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information; In step S21, an evaluation data structure containing all the evaluation indicators is constructed based on the obtained set of evaluation indicators corresponding to the simulation scenario. This structure describes the real-time evaluation of the simulation scenario and includes information such as the dependencies between evaluation indicators and the execution conditions of the evaluation indicators. Specifically: Using a behavior tree as the evaluation data structure, all evaluation indicators in the evaluation indicator set are obtained. In the evaluation indicator library, the indicator data, the dependency relationship between indicators and indicators and the indicator calculation information corresponding to each evaluation indicator are obtained. The evaluation indicator library is a predefined database that stores the relevant information of all evaluation indicators. Several evaluation subtrees, the same number as the number of evaluation indicators, are defined in parallel under the root node of the behavior tree. In each evaluation subtree, a condition node and a behavior node are defined. The condition node is used to determine whether the execution conditions of the current evaluation indicator, including data requirements and other prerequisite requirements, are met, and whether the evaluation of the prerequisite evaluation indicators on which the current evaluation indicator depends has been completed. The behavior node is used to execute the evaluation process of the current evaluation indicator based on the indicator calculation information after the condition node has passed the judgment. A behavior tree blackboard is used as the data storage unit, and the behavior tree blackboard is initialized to define the metadata and storage structure of each evaluation subtree; Simultaneously, the feedback data structure for the evaluation result information fed back by the behavior node is initialized.

2. The real-time evaluation method for autonomous driving simulation testing according to claim 1, characterized in that, Before step S21, the method further includes: Construct a message queue (MQ) for data transfer between the simulation subsystem and the evaluation subsystem; The simulation subsystem asynchronously sends the simulation data, including sensor data, map data, and vehicle posture data, to the message queue MQ. The evaluation subsystem obtains the simulation data by subscribing to the message queue MQ. The evaluation subsystem asynchronously sends the evaluation result information to the message queue MQ, and the simulation subsystem obtains the evaluation result information by subscribing to the message queue MQ.

3. The real-time evaluation method for autonomous driving simulation testing according to claim 1, characterized in that, In step S22, the evaluation process of the simulation scenario is run based on the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information, specifically: The system sequentially polls each evaluation indicator in the evaluation data structure, retrieves the data required to evaluate the current evaluation indicator from the data storage unit, determines whether the execution conditions of the current evaluation indicator, including data requirements and other prerequisite requirements, are met, and determines whether the evaluation of the preceding evaluation indicators on which the current evaluation indicator depends has been completed. If so, the evaluation process of the current evaluation indicator is executed, the evaluation result information is obtained, and the evaluation result information is saved to the data storage unit for sharing with other evaluation indicators during evaluation. Otherwise, the current evaluation indicator continues to wait for the next polling.

4. The real-time evaluation method for autonomous driving simulation testing according to claim 1, characterized in that, When the behavior tree is used as the evaluation data structure and the behavior tree blackboard is used as the data storage unit, the evaluation process of the simulation scenario is run to obtain the evaluation result information, specifically as follows: Run the pre-constructed behavior tree; Obtain the data required to evaluate all current evaluation metrics from the behavior tree blackboard; Each evaluation subtree in the behavior tree is polled in turn. The condition node is used to determine whether the execution conditions of the current evaluation indicator, including data requirements and other prerequisite requirements, are met, and whether the previous evaluation indicators on which the current evaluation indicator depends have been evaluated. After the condition node is judged to be successful, the behavior node is used to execute the evaluation process of the current evaluation indicator based on the indicator calculation information. The evaluation result information is output and stored in the behavior tree blackboard using the feedback data structure.

5. The real-time evaluation method for autonomous driving simulation testing according to claim 1, characterized in that, Also includes: A real-time distributed communication framework, including ROS and Cyber ​​RT, is used to establish communication between the simulation subsystem and the evaluation subsystem. A state machine, including a finite state machine (FSM) and a finite state automaton (FSA), is used as the evaluation data structure.

6. A real-time evaluation system for performing the real-time evaluation method for autonomous driving simulation testing as described in any one of claims 1-5, characterized in that, include: The indicator set construction module is used to construct an evaluation indicator set containing several evaluation indicators corresponding to the simulation scenario. The evaluation index evaluation module is used to evaluate the current evaluation index during the execution of the autonomous driving simulation task based on the simulation scenario, based on the simulation data generated by the continuously updated simulation test progress. When the evaluation index execution conditions of any evaluation index are met, and the evaluation of the previous evaluation index on which the current evaluation index depends is completed, the current evaluation index is evaluated.

7. A computer device comprising a memory and one or more processors, the memory storing computer code that, when executed by the one or more processors, causes the one or more processors to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing computer code, wherein when the computer code is executed, the method of any one of claims 1 to 5 is performed.

Images