Real-time evaluation method and system for automatic driving simulation test

Abstract

The invention relates to the technical field of real-time evaluation, and provides a real-time evaluation method for an automatic driving simulation test, and the method comprises the following steps: S1, constructing an evaluation index set which corresponds to a simulation scene and comprises a plurality of evaluation indexes; s2, in the execution process of the automatic driving simulation task based on the simulation scene, according to simulation data generated by a continuously updated simulation test progress, when the evaluation index execution condition of any evaluation index is satisfied and the evaluation of the pre-posed evaluation index on which the current evaluation index depends is completed, executing the simulation test progress according to the simulation data, and when the evaluation index execution condition of any evaluation index is satisfied and the evaluation of the pre-posed evaluation index depends on the current evaluation index; and evaluating the current evaluation index. According to the technical scheme, evaluation can be carried out in real time in an automatic driving simulation test, the complexity of an evaluation system can be effectively reduced, the evaluation efficiency can be improved, and therefore the overall reliability and efficiency of automatic driving are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time evaluation, and in particular to a real-time evaluation method and system for an autonomous driving simulation test. Background Art

[0002] Autonomous driving technology is an important research direction in the current field of artificial intelligence. Its goal is to enable vehicles to autonomously perceive the surrounding environment, understand traffic rules, plan driving routes and control vehicle driving without the intervention of human drivers through technologies such as computer vision, machine learning and deep learning. The primary requirement for autonomous driving technology to be able to operate on the road is safety and reliability, which requires a lot of simulation test verification and evaluation optimization. The evaluation system is to ensure the safety, compliance and user trust and acceptance of the autonomous driving system, which is of great significance for promoting the development and commercial application of autonomous driving technology. However, due to the complexity and variability of autonomous driving simulation scenarios, the current autonomous driving evaluation scheme still needs to wait until the end of the autonomous driving simulation, and use complete simulation data as basic evaluation data. Real-time evaluation is still a difficulty. How to efficiently and easily expand access to a real-time evaluation system is a matter that needs to be dealt with urgently.

[0003] In the prior art, from the perspective of time span, the autonomous driving evaluation system can effectively evaluate the entire simulation test life cycle, part of the simulation test life cycle, and a specific simulation test life cycle. The prior art provides a large number of reliable solutions for the entire simulation test life cycle. In simple terms, it relies on the overall evaluation of the data files or other data structures generated after the simulation test is completed, which basically belongs to the evaluation after the autonomous driving simulation is completed. In addition, Figure 1 As shown, each simulation time point is evaluated separately after the entire simulation test life cycle. At the same time, the evaluation in the prior art is generally carried out for a specific evaluation index set in advance. The shortcomings of the evaluation scheme in the prior art are as follows:

[0004] (1) Poor real-time performance: Real-time evaluation of autonomous driving cannot be achieved.

[0005] (2) Poor scalability: Most existing technologies are based on specific assessments to achieve targeted assessments. They are difficult to expand when faced with diverse assessment needs and are prone to redundant calculations.

[0006] (3) Poor authenticity: Existing evaluation schemes only consider preset evaluation criteria, while ignoring other challenges that autonomous driving simulation may face in real environments. In other words, existing technologies cannot dynamically adjust evaluation indicators.

[0007] (4) Lack of standardization: The existing technology lacks a unified standard implementation evaluation method, which makes it difficult for different evaluation indicator combinations to be referenced, and increases the difficulty of dynamically adjusting the evaluation plan. Summary of the invention

[0008] In response to the above problems, the purpose of the present invention is to provide a real-time evaluation method and system for autonomous driving simulation tests, which can perform real-time evaluation in autonomous driving simulation tests, effectively reduce the complexity of the evaluation system and improve the efficiency of the evaluation, thereby improving the overall reliability and efficiency of autonomous driving.

[0009] The above-mentioned object of the present invention is achieved through the following technical solutions:

[0010] A real-time evaluation method for an autonomous driving simulation test comprises the following steps:

[0011] S1: Construct an evaluation indicator set containing several evaluation indicators corresponding to the simulation scenario;

[0012] S2: During the execution of the autonomous driving simulation task based on the simulation scenario, according to the simulation data generated by the continuously updated simulation test progress, when the evaluation indicator execution condition of any evaluation indicator is met and the evaluation of the preceding evaluation indicator on which the current evaluation indicator depends is completed, the current evaluation indicator is evaluated.

[0013] Further, step S2 is specifically:

[0014] S21: during the execution of the autonomous driving simulation task, after the evaluation subsystem receives the simulation data in the simulation subsystem for the first time, an evaluation data structure including all the evaluation indicators for describing the real-time evaluation of the autonomous driving simulation task, including the dependency relationship between the evaluation indicators and the execution conditions of the evaluation indicators, is constructed according to the evaluation indicator set corresponding to the simulation scenario;

[0015] S22: During the execution of the autonomous driving simulation task, when the evaluation subsystem receives the simulation data in the simulation subsystem for a non-first time, the simulation data is updated to the data storage unit corresponding to the evaluation data structure, and at the same time, the evaluation process of the simulation scenario is triggered based on the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information.

[0016] Furthermore, before step S21, the method further includes:

[0017] Constructing a message queue MQ for data transmission between the simulation subsystem and the evaluation subsystem;

[0018] The simulation subsystem asynchronously sends the simulation data including sensor data information, map data information, and vehicle body posture data information to the message queue MQ, and the evaluation subsystem obtains the simulation data by subscribing to the message queue MQ;

[0019] And the evaluation submodule asynchronously sends the evaluation result information to the message queue MQ, and the simulation subsystem obtains the evaluation result information by subscribing to the message queue MQ.

[0020] Further, in step S22, the evaluation process of the simulation scenario is run according to the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information, specifically:

[0021] Poll each of the evaluation indicators in the evaluation data structure in turn, obtain the data required for evaluating the current evaluation indicator from the data storage unit, determine whether the evaluation indicator execution conditions including data requirements and other prerequisite requirements of the current evaluation indicator are met, and determine whether the evaluation of the preceding evaluation indicator on which the current evaluation indicator depends is completed. If the evaluation process of the current evaluation indicator is being executed, obtain the evaluation result information, and save the evaluation result information to the data storage unit to share it with other evaluation indicators for use in evaluation. If not, the current evaluation indicator continues to wait for the next polling.

[0022] Further, in step S21, an evaluation data structure containing all the evaluation indicators for describing the real-time evaluation of the simulation scenario, including the dependency relationship between the evaluation indicators and the evaluation indicator execution conditions, is constructed according to the evaluation indicator set corresponding to the acquired simulation scenario, specifically:

[0023] Using a behavior tree as the evaluation data structure, obtaining all the evaluation indicators in the evaluation indicator set, obtaining indicator data corresponding to each evaluation indicator, dependency relationships between indicators, and indicator calculation information in an evaluation indicator library, wherein the evaluation indicator library is a predefined database storing relevant information of all the evaluation indicators;

[0024] A plurality of evaluation subtrees having the same number as the evaluation indicators are defined in parallel under the root node of the behavior tree, and a condition node and a behavior node are defined in each of the evaluation subtrees, wherein the condition node is used to judge whether the evaluation indicator execution conditions including data requirements and other prerequisite requirements of the current evaluation indicator are met, and whether the evaluation of the pre-conditional evaluation indicator on which the current evaluation indicator depends is completed, and the behavior node is used to perform the evaluation process of the current evaluation indicator according to the indicator calculation information after the condition node is judged to be passed;

[0025] Using a behavior tree blackboard as the data storage unit, initializing the behavior tree blackboard, and defining metadata and storage structure of each evaluation subtree;

[0026] At the same time, the feedback data structure of the evaluation result information fed back by the behavior node is initialized.

[0027] Further, when the behavior tree is used as the evaluation data structure and the behavior tree blackboard is used as the data storage unit, the evaluation process of the simulation scenario is run to obtain the evaluation result information, specifically:

[0028] Run the behavior tree that has been constructed;

[0029] Obtain data required for evaluating all current evaluation indicators from the behavior tree blackboard;

[0030] polling each of the evaluation subtrees in the behavior tree in turn, using the condition node to judge whether the evaluation indicator execution conditions including data requirements and other prerequisite requirements of the current evaluation indicator are met, and whether the evaluation of the preceding evaluation indicator on which the current evaluation indicator depends is completed, and using the behavior node to execute the evaluation process of the current evaluation indicator according to the indicator calculation information after the condition node passes the judgment;

[0031] The evaluation result information is output, and the evaluation result information is stored in the behavior tree blackboard using the feedback data structure.

[0032] Furthermore, the real-time evaluation method of the autonomous driving simulation test also includes:

[0033] Using a real-time distributed communication framework including ROS and Cyber ​​RT to establish communication between the simulation subsystem and the evaluation subsystem;

[0034] State machines including finite state machines FSM and finite state automata FSA are used as the evaluation data structure.

[0035] A real-time evaluation system for an autonomous driving simulation test for executing the real-time evaluation method for an autonomous driving simulation test as described above, comprising:

[0036] An indicator set construction module, used to construct an evaluation indicator set including several evaluation indicators corresponding to the simulation scenario;

[0037] An evaluation index evaluation module is used to evaluate the current evaluation index based on the simulation data generated by the continuously updated simulation test progress during the execution of the autonomous driving simulation task based on the simulation scenario, when the evaluation index execution condition of any evaluation index is met and the evaluation of the preceding evaluation index on which the current evaluation index depends is completed.

[0038] A computer device comprises a memory and one or more processors, wherein the memory stores computer codes, and when the computer codes are executed by the one or more processors, the one or more processors execute the above method.

[0039] A computer-readable storage medium stores computer codes. When the computer codes are executed, the above method is executed.

[0040] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0041] (1) Improving the efficiency of autonomous driving evaluation: The real-time evaluation method of autonomous driving simulation test provided by the present invention realizes the simultaneous implementation of real-time evaluation and simulation test, which can reduce the overall process time of autonomous driving safety verification.

[0042] (2) Improving the reliability of autonomous driving evaluation: The present invention can utilize real-time evaluation result feedback to enhance the real reliability of the system.

[0043] (3) Improving the adaptability and flexibility of autonomous driving evaluation: The present invention can dynamically configure the indicators of autonomous driving evaluation according to different test contents and driving environments, thereby improving the adaptability and flexibility of the self-assessment system.

[0044] (4) Enhance the scalability of autonomous driving evaluation: The present invention uses behavior trees for real-time evaluation. Thanks to the efficient data organization and dynamics of the tree structure, the behavior tree can adjust the evaluation content in a very efficient and concise manner.

[0045] (5) Promote the improvement of autonomous driving evaluation technology: This solution provides a method for achieving real-time evaluation, thereby improving the reliability and efficiency of the evaluation system, and assisting the autonomous driving evaluation system to more quickly discover weak links and improve them, thereby promoting the improvement of autonomous driving evaluation technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the evaluation of autonomous driving simulation test in the prior art;

[0047] Figure 2 The overall flow chart of the real-time evaluation method of the autonomous driving simulation test of the present invention;

[0048] Figure 3 This is a flowchart of a real-time evaluation method for an autonomous driving simulation test of the present invention;

[0049] Figure 4 A process diagram for constructing a behavior tree of the present invention;

[0050] Figure 5 This is an overall structural diagram of the real-time evaluation system for the autonomous driving simulation test of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0053] Explanation of technical terms involved in the present invention:

[0054] Behavior Tree: Behavior Tree (BT) is a tree structure used to describe behaviors and manage behavior switching. It is widely used in artificial intelligence, especially in game development and robot control. Behavior tree represents complex behavior logic through a series of nodes. Nodes can be conditional judgments, action executions, or combinations of multiple sub-behaviors.

[0055] Autonomous driving simulation: Using computer simulation technology to simulate the behavior of autonomous vehicles in different scenarios and situations.

[0056] Combination strategy: It is used to organize other nodes in the behavior tree to form a complex behavior structure. Common combination strategies include: sequential completion, selection completion, conditional completion, etc.

[0057] (1) Sequential completion

[0058] Definition: A sequence node executes its child nodes in order until one child node fails or all child nodes succeed.

[0059] Application scenario: used for a series of tasks that must be completed in a specific order. For example, an autonomous vehicle needs to slow down before entering an intersection, then check the traffic lights, and finally pass through the intersection.

[0060] (2) Select Complete

[0061] Definition: A selection node executes its child nodes in order until one child node succeeds or all child nodes fail.

[0062] Application scenario: Used for selective execution of tasks. For example, when an autonomous vehicle encounters an obstacle, it can choose to turn left, turn right, or stop, as long as one of them is successful.

[0063] (3) Conditions completed

[0064] Definition: A conditional node decides whether to execute its child nodes based on a certain condition. Its child nodes will be executed only when the condition is met.

[0065] Application scenario: used for tasks that are performed only under certain conditions. For example, an autonomous vehicle needs to slow down only when its speed exceeds a certain threshold.

[0066] Message Queue MQ: A communication mode used to transmit messages between different applications, decouple the dependencies between applications, improve system reliability and scalability, and support message persistence and asynchronous transmission. Common message queues include RabbitMQ, Kafka, and Redis.

[0067] Blackboard: Shared memory used for data exchange within the behavior tree. The data on the blackboard is stored in the form of "key-value". The sending and receiving tree nodes can communicate with each other by reading and writing the same "key".

[0068] Node: The basic element of a behavior tree, used to describe behaviors and behavior organization structures. Common behavior tree nodes include: behavior nodes, conditional nodes, combination nodes, etc.

[0069] Metadata: Data that describes data, which can be understood as the attributes or characteristics of data, and can help better manage and use data.

[0070] This patented technical solution can solve the pain point of being difficult to form a standard and efficient real-time evaluation method when facing complex and diverse evaluation indicators in the field of autonomous driving. Using the efficient data organization method of behavior tree, a standard and easily scalable real-time evaluation method is provided. At the same time, MQ is used to transmit data, which can avoid affecting the efficiency of autonomous driving simulation tests as much as possible, while producing evaluation results in real time, and discovering the hidden dangers in the key links of the autonomous driving system as early as possible.

[0071] The following is described by specific examples:

[0072] First embodiment

[0073] like Figure 2 and 3 As shown, this embodiment provides a real-time evaluation method for an autonomous driving simulation test, comprising the following steps:

[0074] S1: Construct an evaluation indicator set containing several evaluation indicators corresponding to the simulation scenario.

[0075] Specifically, in this embodiment, after starting the autonomous driving simulation task, the simulation subsystem will first use the scene construction module to build a simulation scene, and build an evaluation indicator set containing several evaluation indicators corresponding to the current simulation scene, and then call the scene running module in the simulation subsystem, and load the evaluation data real-time display module in the simulation subsystem.

[0076] Among them, for the evaluation indicators in the evaluation indicator set, this embodiment can dynamically configure the indicators of autonomous driving evaluation according to different test contents and driving environments, and perform adaptive adjustment of the evaluation indicators, thereby improving the adaptability and flexibility of the evaluation system.

[0077] For example, in a specific embodiment, the following four evaluations are set in the evaluation indicator set:

[0078] (1) Safety assessment: If there is no collision, perform safety assessment;

[0079] (2) Reaction time assessment: If there is no collision, perform safety assessment;

[0080] (3) Path deviation evaluation: If the reaction time evaluation passes, calculate the path deviation;

[0081] (4) Speed ​​control evaluation: If the path deviation evaluation passes, check the speed control.

[0082] S2: During the execution of the autonomous driving simulation task based on the simulation scenario, according to the simulation data generated by the continuously updated simulation test progress, when the evaluation indicator execution condition of any evaluation indicator is met and the evaluation of the preceding evaluation indicator on which the current evaluation indicator depends is completed, the current evaluation indicator is evaluated.

[0083] Before executing step S21, this embodiment includes: constructing a message queue MQ for data transmission between the simulation subsystem and the evaluation subsystem; the simulation subsystem asynchronously sends the simulation data including sensor data information, map data information, and vehicle body posture data information to the message queue MQ, and the evaluation subsystem obtains the simulation data by subscribing to the message queue MQ; and the evaluation submodule asynchronously sends the evaluation result information to the message queue MQ, and the simulation subsystem obtains the evaluation result information by subscribing to the message queue MQ, and updates and displays it in real time in the evaluation data real-time display module. By using the message queue MQ, we can minimize the performance impact of data transmission on the operation of the simulation scene. By using MQ as the data interaction between the scene operation module and the evaluation module, the reduction of simulation performance caused by data transmission is avoided, and the pressure on the evaluation data receiving module is reduced. In addition, a real-time display module for evaluation data is set to discover the hidden dangers of the autonomous driving system as soon as possible and improve the reliability of the evaluation system.

[0084] In this embodiment, step S2 is specifically as follows:

[0085] S21: During the execution of the autonomous driving simulation task, when the evaluation subsystem receives the simulation data in the simulation subsystem for the first time, an evaluation data structure containing all the evaluation indicators is constructed according to the set of evaluation indicators corresponding to the simulation scenario to describe the real-time evaluation of the autonomous driving simulation task, including the dependency relationship between the evaluation indicators and the execution conditions of the evaluation indicators.

[0086] Specifically, in this embodiment, after the data receiving module in the evaluation subsystem receives the simulation data in the simulation subsystem for the first time, it automatically constructs an evaluation data structure including all evaluation indicators.

[0087] A specific implementation method is to use behavior trees to construct the evaluation data structure. When the data receiving module first obtains the simulation data, it will forward the simulation data to the evaluation behavior tree construction module. The evaluation behavior tree construction module obtains the corresponding indicator data and calculation information from the evaluation indicator library through the evaluation indicators corresponding to the obtained simulation scene, and then constructs a complete behavior tree. Using behavior trees as the organizational form of real-time evaluation can increase the clarity of the evaluation structure on the one hand, and improve the efficiency and scalability of real-time evaluation on the other hand. Figure 4 As shown, the specific process is:

[0088] Using a behavior tree as the evaluation data structure, obtaining all the evaluation indicators in the evaluation indicator set, obtaining indicator data corresponding to each evaluation indicator, dependency relationships between indicators, and indicator calculation information in an evaluation indicator library, wherein the evaluation indicator library is a predefined database storing relevant information of all the evaluation indicators;

[0089] A plurality of evaluation subtrees having the same number as the evaluation indicators are defined in parallel under the root node of the behavior tree, and a condition node and a behavior node are defined in each of the evaluation subtrees, wherein the condition node is used to judge whether the evaluation indicator execution conditions including data requirements and other prerequisite requirements of the current evaluation indicator are met, and whether the evaluation of the pre-conditional evaluation indicator on which the current evaluation indicator depends is completed, and the behavior node is used to perform the evaluation process of the current evaluation indicator according to the indicator calculation information after the condition node is judged to be passed;

[0090] Using a behavior tree blackboard as the data storage unit, initializing the behavior tree blackboard, and defining metadata and storage structure of each evaluation subtree;

[0091] At the same time, the feedback data structure of the evaluation result information fed back by the behavior node is initialized.

[0092] In addition, it is necessary to design solutions for evaluation exceptions, complete testing and debugging, and ensure that there are no exceptions in the construction of the behavior tree.

[0093] For example, the construction process of the behavior tree:

[0094] 1. Analyze the needs and dependencies of evaluation indicators

[0095] Suppose we want to evaluate the performance of an autonomous vehicle in an urban traffic scenario. The evaluation metrics include:

[0096] (1) Safety assessment: If there is no collision, perform safety assessment;

[0097] (2) Reaction time assessment: If the safety assessment passes, perform a reaction time assessment;

[0098] (3) Path deviation evaluation: If the reaction time evaluation passes, calculate the path deviation;

[0099] (4) Speed ​​control evaluation: If the path deviation evaluation passes, check the speed control.

[0100] 2. Construct evaluation subtree

[0101] A: Data requirements, dependencies, and evaluation behaviors of each subtree

[0102] (1) Security Assessment Subtree

[0103] Data requirements: Vehicle collision detection data.

[0104] Dependencies: Directly dependent on collision detection data.

[0105] Evaluation behavior: Check if a collision occurred.

[0106] (2) Reaction time evaluation subtree

[0107] Data requirements: Obstacle or red light detection data.

[0108] Dependencies: Security assessment passed.

[0109] Evaluate behavior: Calculate the vehicle's reaction time when it detects an obstacle or a red light.

[0110] (3) Path Deviation Evaluation Subtree

[0111] Data requirements: Vehicle location and planned route data.

[0112] Dependencies: Reaction time evaluation passed.

[0113] Evaluation behavior: Calculate the deviation of the vehicle's path from the planned path.

[0114] (4) Speed ​​Control Evaluation Subtree

[0115] Data requirements: Vehicle speed data.

[0116] Dependencies: Path deviation evaluation passed.

[0117] Assessment behavior: Check whether the vehicle speed is within the specified range.

[0118] B: The final behavior tree structure is:

[0119] (1) Security assessment subtree:

[0120] Conditional node 1: Read the collision detection data in the blackboard to check whether there is no collision.

[0121] Action Node 1: If no collision, perform safety evaluation and store the result in the blackboard.

[0122] (2) Reaction time evaluation subtree (dependency security evaluation):

[0123] Conditional node 2: Check the security assessment results in the blackboard to ensure that the security test passes.

[0124] Behavior Node 2: If the safety assessment passes, calculate the reaction time and store the result in the blackboard.

[0125] (3) Path deviation evaluation subtree (depends on reaction time evaluation):

[0126] Conditional Node 3: Check the reaction time evaluation results in the blackboard to ensure that the reaction time is within a reasonable range.

[0127] Behavior Node 3: If the reaction time evaluation passes, calculate the path deviation and store the result in the blackboard.

[0128] (4) Speed ​​control evaluation subtree (depends on path deviation evaluation):

[0129] Conditional node 4: Check the path deviation evaluation results in the blackboard to ensure that the path deviation is within a reasonable range.

[0130] Action Node 4: If the path deviation evaluation passes, check the speed control and store the result in the blackboard.

[0131] 3. Initialize the behavior tree blackboard

[0132] The behavior tree blackboard is a shared memory used to pass data between different nodes in the behavior tree. We need to define metadata and storage structures for each evaluation subtree.

[0133] (1) Security assessment subtree:

[0134] Metadata: Collision detection data.

[0135] Storage structure: stores the most recent collision detection results.

[0136] (2) Reaction time evaluation subtree:

[0137] Metadata: Obstacle or red light detection data.

[0138] Storage structure: stores the most recent obstacle or red light detection results.

[0139] (3) Path deviation evaluation subtree:

[0140] Metadata: Vehicle location and planned route data.

[0141] Storage structure: stores the most recent vehicle location and scheduled route data.

[0142] (4) Speed ​​control evaluation subtree:

[0143] Metadata: Vehicle speed data.

[0144] Storage structure: stores the most recent vehicle speed data.

[0145] 4. Initialize the evaluation feedback data structure

[0146] The evaluation feedback data structure is used to record the results of each evaluation indicator for subsequent analysis and presentation.

[0147]

[0148] Further, when the behavior tree is used as the evaluation data structure and the behavior tree blackboard is used as the data storage unit, the evaluation process of the simulation scenario is run to obtain the evaluation result information, specifically:

[0149] Run the behavior tree that has been constructed;

[0150] Obtain data required for evaluating all current evaluation indicators from the behavior tree blackboard;

[0151] polling each of the evaluation subtrees in the behavior tree in turn, using the condition node to judge whether the evaluation indicator execution conditions including data requirements and other prerequisite requirements of the current evaluation indicator are met, and whether the evaluation of the preceding evaluation indicator on which the current evaluation indicator depends is completed, and using the behavior node to execute the evaluation process of the current evaluation indicator according to the indicator calculation information after the condition node passes the judgment;

[0152] The evaluation result information is output, and the evaluation result information is stored in the behavior tree blackboard using the feedback data structure.

[0153] S22: During the execution of the autonomous driving simulation task, when the evaluation subsystem receives the simulation data in the simulation subsystem for a non-first time, the simulation data is updated to the data storage unit corresponding to the evaluation data structure, and at the same time, the evaluation process of the simulation scenario is triggered based on the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information.

[0154] In step S22, the evaluation process of the simulation scenario is run according to the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information, specifically:

[0155] Poll each of the evaluation indicators in the evaluation data structure in turn, obtain the data required for evaluating the current evaluation indicator from the data storage unit, determine whether the evaluation indicator execution conditions including data requirements and other prerequisite requirements of the current evaluation indicator are met, and determine whether the evaluation of the preceding evaluation indicator on which the current evaluation indicator depends is completed. If the evaluation process of the current evaluation indicator is being executed, obtain the evaluation result information, and save the evaluation result information to the data storage unit to share it with other evaluation indicators for use in evaluation. If not, the current evaluation indicator continues to wait for the next polling.

[0156] In addition, the present invention can also adopt a real-time distributed communication framework including ROS and Cyber ​​RT to establish communication between the simulation subsystem and the evaluation subsystem, replacing the mode of using message queue MQ. ROS (Robot Operating System) is a framework that provides operating system functions for robot development, supporting hardware abstraction, low-level device control, implementation of common functions, message passing and package management, etc. Cyber ​​RT is a high-performance distributed communication framework used in Baidu Apolo autonomous driving platform, designed for high real-time and high concurrency scenarios.

[0157] A state machine including a finite state machine FSM and a finite state automaton FSA may also be used as the evaluation data structure.

[0158] Example: FSM in an autonomous driving simulation system

[0159] 1. States

[0160] Initialization: The system starts and is ready to receive data.

[0161] Data reception: Receive simulation data and perform preliminary processing.

[0162] Safety Assessment: Evaluate the safety of the vehicle.

[0163] Reaction Time Assessment: Assess the vehicle's reaction time.

[0164] Path Deviation Assessment: Evaluate the deviation of the vehicle's driving path.

[0165] Speed ​​Control Assessment: Assess vehicle speed control.

[0166] Result output: Output the evaluation results and send them to the simulation module.

[0167] (2) Initial State

[0168] initialization

[0169] (3) Events or Inputs

[0170] Receive data

[0171] Data is valid

[0172] Safety assessment completed

[0173] Reaction time assessment completed

[0174] Path deviation assessment completed

[0175] Speed ​​Control Assessment Completed

[0176] Result output completed

[0177] (4) State Transitions

[0178] (Initialization, data received) → Data received

[0179] (Data received, data valid) → Security assessment

[0180] (Safety Assessment, Safety Assessment Completed) → Response Time Assessment

[0181] (Reaction time evaluation, reaction time evaluation completed) → Path deviation evaluation

[0182] (Path Deviation Evaluation, Path Deviation Evaluation Complete) → Speed ​​Control Evaluation

[0183] (Speed ​​control evaluation, speed control evaluation completed) → Result output

[0184] (result output, result output completed) → Initialization

[0185] (5) Final States

[0186] No specific termination state (system loops)

[0187] Second embodiment

[0188] like Figure 5 As shown, this embodiment provides a real-time evaluation system for an autonomous driving simulation test for executing the real-time evaluation method for an autonomous driving simulation test in the first embodiment, comprising:

[0189] An indicator set construction module 1 is used to construct an evaluation indicator set including several evaluation indicators corresponding to the simulation scenario;

[0190] The evaluation index evaluation module 2 is used to evaluate the current evaluation index according to the simulation data generated by the continuously updated simulation test progress during the execution of the autonomous driving simulation task based on the simulation scenario, when the evaluation index execution condition of any evaluation index is met and the evaluation of the preceding evaluation index on which the current evaluation index depends is completed.

[0191] A computer-readable storage medium stores a computer code. When the computer code is executed, the above method is executed. A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0192] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

[0193] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0194] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A real-time evaluation method for autonomous driving simulation testing, characterized in that: The following steps are involved: S1: Construct an evaluation indicator set containing several evaluation indicators corresponding to the simulation scenario; S2: During the execution of the autonomous driving simulation task based on the simulation scenario, according to the simulation data generated by the continuously updated simulation test progress, when the evaluation indicator execution condition of any evaluation indicator is met and the evaluation of the preceding evaluation indicator on which the current evaluation indicator depends is completed, the current evaluation indicator is evaluated.

2. The real-time evaluation method for autonomous driving simulation test according to claim 1, characterized in that: Step S2 is specifically: S21: during the execution of the autonomous driving simulation task, after the evaluation subsystem receives the simulation data in the simulation subsystem for the first time, an evaluation data structure including all the evaluation indicators for describing the real-time evaluation of the autonomous driving simulation task, including the dependency relationship between the evaluation indicators and the execution conditions of the evaluation indicators, is constructed according to the evaluation indicator set corresponding to the simulation scenario; S22: During the execution of the autonomous driving simulation task, when the evaluation subsystem receives the simulation data in the simulation subsystem for a non-first time, the simulation data is updated to the data storage unit corresponding to the evaluation data structure, and at the same time, the evaluation process of the simulation scenario is triggered based on the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information.

3. The real-time evaluation method for autonomous driving simulation test according to claim 2, characterized in that: Before step S21, the method further includes: Constructing a message queue MQ for data transmission between the simulation subsystem and the evaluation subsystem; The simulation subsystem asynchronously sends the simulation data including sensor data information, map data information, and vehicle body posture data information to the message queue MQ, and the evaluation subsystem obtains the simulation data by subscribing to the message queue MQ; And the evaluation submodule asynchronously sends the evaluation result information to the message queue MQ, and the simulation subsystem obtains the evaluation result information by subscribing to the message queue MQ.

4. The real-time evaluation method for autonomous driving simulation test according to claim 2, characterized in that: In step S22, the evaluation process of the simulation scenario is run according to the data stored in the data storage unit and the evaluation data structure to obtain evaluation result information, specifically: Poll each of the evaluation indicators in the evaluation data structure in turn, obtain the data required for evaluating the current evaluation indicator from the data storage unit, determine whether the evaluation indicator execution conditions including data requirements and other prerequisite requirements of the current evaluation indicator are met, and determine whether the evaluation of the preceding evaluation indicator on which the current evaluation indicator depends is completed. If the evaluation process of the current evaluation indicator is being executed, obtain the evaluation result information, and save the evaluation result information to the data storage unit to share it with other evaluation indicators for use in evaluation. If not, the current evaluation indicator continues to wait for the next polling.

5. The real-time evaluation method for autonomous driving simulation test according to claim 2, characterized in that: In step S21, an evaluation data structure containing all the evaluation indicators for describing the real-time evaluation of the simulation scenario, including the dependency relationship between the evaluation indicators and the evaluation indicator execution conditions, is constructed according to the evaluation indicator set corresponding to the obtained simulation scenario, specifically: Using a behavior tree as the evaluation data structure, obtaining all the evaluation indicators in the evaluation indicator set, obtaining indicator data corresponding to each evaluation indicator, dependency relationships between indicators, and indicator calculation information in an evaluation indicator library, wherein the evaluation indicator library is a predefined database storing relevant information of all the evaluation indicators; A plurality of evaluation subtrees having the same number as the evaluation indicators are defined in parallel under the root node of the behavior tree, and a condition node and a behavior node are defined in each of the evaluation subtrees, wherein the condition node is used to judge whether the evaluation indicator execution conditions including data requirements and other prerequisite requirements of the current evaluation indicator are met, and whether the evaluation of the pre-conditional evaluation indicator on which the current evaluation indicator depends is completed, and the behavior node is used to perform the evaluation process of the current evaluation indicator according to the indicator calculation information after the condition node is judged to be passed; Using a behavior tree blackboard as the data storage unit, initializing the behavior tree blackboard, and defining metadata and storage structure of each evaluation subtree; At the same time, the feedback data structure of the evaluation result information fed back by the behavior node is initialized.

6. The real-time evaluation method for autonomous driving simulation test according to claim 5, characterized in that: When the behavior tree is used as the evaluation data structure and the behavior tree blackboard is used as the data storage unit, the evaluation process of the simulation scenario is run to obtain the evaluation result information, specifically: Run the behavior tree that has been constructed; Obtain data required for evaluating all current evaluation indicators from the behavior tree blackboard; polling each of the evaluation subtrees in the behavior tree in turn, using the condition node to judge whether the evaluation indicator execution conditions including data requirements and other prerequisite requirements of the current evaluation indicator are met, and whether the evaluation of the preceding evaluation indicator on which the current evaluation indicator depends is completed, and using the behavior node to execute the evaluation process of the current evaluation indicator according to the indicator calculation information after the condition node passes the judgment; The evaluation result information is output, and the evaluation result information is stored in the behavior tree blackboard using the feedback data structure.

7. The real-time evaluation method for autonomous driving simulation test according to claim 2, characterized in that: Also includes: Using a real-time distributed communication framework including ROS and Cyber ​​RT to establish communication between the simulation subsystem and the evaluation subsystem; State machines including finite state machines FSM and finite state automata FSA are used as the evaluation data structure.

8. A real-time evaluation system for an autonomous driving simulation test for executing the real-time evaluation method for an autonomous driving simulation test according to any one of claims 1 to 7, characterized in that: include: An indicator set construction module is used to construct an evaluation indicator set including several evaluation indicators corresponding to the simulation scenario; An evaluation index evaluation module is used to evaluate the current evaluation index based on the simulation data generated by the continuously updated simulation test progress during the execution of the autonomous driving simulation task based on the simulation scenario, when the evaluation index execution condition of any evaluation index is met and the evaluation of the preceding evaluation index on which the current evaluation index depends is completed.

9. A computer device comprising a memory and one or more processors, wherein the memory stores computer codes, and when the computer codes are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 7. 10 . A computer-readable storage medium storing a computer code. When the computer code is executed, the method according to claim 1 is executed.

Images