Evaluation method and device of autonomous driving safety verification platform, equipment and medium

Abstract

The present disclosure relates to an evaluation method, device, equipment and medium of an automatic driving safety verification platform, which comprises: generating a test task according to a test requirement, the test task comprising scene data, and the test task being sent to a corresponding message queue of a corresponding container group; sending the test task to a corresponding connected message queue according to a container group identifier carried by the test task; each scene control service system acquires the test task in the message queue, calls a UE simulation engine, loads scene data carried in the test task to the UE simulation engine to create a test scene and feeds back to the corresponding scene control service system, then calls a measured system algorithm, loads the measured system algorithm to the corresponding test scene for simulation testing, and evaluates the test result through a preset evaluation index. The present application realizes that multiple measured systems can independently and in parallel perform simulation testing of measured algorithms, and solves the problem of high time cost due to the fact that the evaluation system cannot simultaneously perform multiple test tasks.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to evaluation methods, apparatus, equipment and media for autonomous driving safety verification platforms. Background Technology

[0002] As the level of autonomous driving increases, vehicle systems become more complex. Changing weather conditions, complex traffic environments, diverse driving tasks, and dynamic driving states all present new challenges for testing and evaluating autonomous vehicles. To ensure the effectiveness and safety of advanced autonomous driving functions, their effectiveness and functionality need to be verified throughout the entire research and development phase and after development is completed.

[0003] Scenario-based virtual testing technology offers flexible scenario configuration, high testing efficiency, strong test repeatability, safe testing process, and low testing cost. It can achieve automated and accelerated testing, saving significant manpower and resources. Therefore, scenario-based virtual testing has become an indispensable and important part of the testing and evaluation of autonomous driving systems.

[0004] In autonomous driving simulation testing, autonomous driving systems need to be tested in multiple scenarios to detect their safety, compliance, or comfort. Existing technologies do not have methods or systems for simultaneously testing multiple systems under test in different test scenarios. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides an evaluation method and apparatus for an autonomous driving safety verification platform, which solves the problem that in autonomous driving simulation testing, the existing technology does not have a method and system for simultaneously conducting tests on multiple systems under test in different test scenarios.

[0006] According to a first aspect of the present disclosure, an evaluation method for an autonomous driving safety verification platform is provided, comprising: Test tasks are generated according to test requirements. The test tasks include scenario data and are sent to the message queue corresponding to the corresponding container group. Based on the container group identifier carried by the test task, the test task is sent to the message queue connected to the corresponding container group; the container group includes a first container that encapsulates the scene control service system and a second container that encapsulates the algorithm of the system under test; the first container and the second container are connected to each other through a communication module. Each scene control service system obtains the test task from the corresponding message queue, calls the preset UE simulation engine, loads the scene data carried in the test task into the UE simulation engine to create a test scene, and feeds back the test scene to the corresponding scene control service system; the scene control service system calls the algorithm of the system under test, loads the algorithm of the system under test into the corresponding test scene for simulation testing, and obtains the test results; The test results are evaluated using preset evaluation indicators.

[0007] According to a second aspect of the present disclosure, an evaluation apparatus for an autonomous driving safety verification platform is provided, including a test task generation module, a task distribution module, a simulation test module, and an evaluation module. The test task generation module generates test tasks according to test requirements. The test tasks include scene data and are sent to the message queue corresponding to the container group. The container group includes a first container that encapsulates the scene control service system and a second container that encapsulates the algorithm of the system under test. The first container and the second container are connected to each other through a communication module. The task distribution module is used to send the test task to the message queue connected to the corresponding container group according to the container group identifier carried by the test task. The simulation test module is used by each scene control service system to obtain the test task in the corresponding message queue, call the preset UE simulation engine, load the scene data carried in the test task into the UE simulation engine to create a test scene, and feed the test scene back to the corresponding scene control service system; the scene control service system calls the algorithm of the system under test, loads the algorithm of the system under test into the corresponding test scene for simulation testing, and obtains the test results; The evaluation module is used to evaluate the test results using preset evaluation indicators.

[0008] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the evaluation method of the autonomous driving safety verification platform provided in the first aspect of the present disclosure.

[0009] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the evaluation method of the autonomous driving safety verification platform provided in the first aspect of the present disclosure.

[0010] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: The test task includes scene data and container group identifiers. According to the container group identifiers, the corresponding scene data is distributed to the corresponding message queues, which facilitates the execution of task data obtained by each scene control service system from the corresponding message queues. Then, the algorithm of the system under test is loaded into the required test scene for testing, avoiding the situation where the test scene obtained by each scene control service system is not the scene that the corresponding algorithm of the system under test needs to be tested, making the testing process more efficient. In addition, the scene control service system and the system under test are respectively encapsulated in two communication-connected containers. During the test run, the scene control service system calls the algorithm of the system under test and loads the algorithm of the system under test into the corresponding test scene for simulation testing. This enables multiple systems under test to independently and in parallel perform simulation testing of the algorithm under test, and enables one or more systems under test to perform simulation testing in one or more simulation environments. This solves the problem that the evaluation system cannot perform multiple test tasks at the same time, and different functional tests need to be queued and tested in sequence, resulting in high time costs.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0013] Figure 1 This is a flowchart illustrating an evaluation method for an autonomous driving safety verification platform according to an exemplary embodiment.

[0014] Figure 2 This is a flowchart illustrating the first task operation of an evaluation method for an autonomous driving safety verification platform according to an exemplary embodiment.

[0015] Figure 3 This is a flowchart illustrating the second task operation of an evaluation method for an autonomous driving safety verification platform according to an exemplary embodiment.

[0016] Figure 4 This is a task processing flowchart of a scenario control service system for an evaluation method of an autonomous driving safety verification platform, according to an exemplary embodiment.

[0017] Figure 5 This is a scene control service system and task processing sequence diagram illustrating an evaluation method for an autonomous driving safety verification platform according to an exemplary embodiment.

[0018] Figure 6This is a flowchart illustrating the first evaluation service task processing method of an evaluation method for an autonomous driving safety verification platform according to an exemplary embodiment.

[0019] Figure 7 This is a flowchart illustrating the second evaluation service task processing method of an evaluation method for an autonomous driving safety verification platform according to an exemplary embodiment.

[0020] Figure 8 This is a timing diagram of the evaluation service and task processing of an evaluation method for an autonomous driving safety verification platform, according to an exemplary embodiment.

[0021] Figure 9 This is a block diagram illustrating an evaluation apparatus for an autonomous driving safety verification platform according to an exemplary embodiment.

[0022] Figure 10 This is another structural schematic block diagram of an evaluation device for an autonomous driving safety verification platform, according to an exemplary embodiment.

[0023] Figure 11 This is another structural schematic block diagram of an evaluation device for an autonomous driving safety verification platform, according to an exemplary embodiment.

[0024] Figure 12 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0025] The exemplary embodiments will now be described in detail with reference to the accompanying drawings.

[0026] It should be noted that the relevant embodiments and accompanying drawings are only for describing and illustrating exemplary embodiments provided by this disclosure, and not all embodiments of this disclosure, nor should this disclosure be understood to be limited to the relevant exemplary embodiments.

[0027] It should be noted that the terms "first," "second," etc., used in this disclosure are only used to distinguish different steps, devices, or modules. These terms do not represent any specific technical meaning, nor do they indicate any order or interdependence between them.

[0028] It should be noted that the terms “a,” “a plurality of,” and “at least one” used in this disclosure are illustrative rather than restrictive. Unless otherwise expressly indicated in the context, they should be understood as “one or more.”

[0029] It should be noted that the term "and / or" used in this disclosure is used to describe the relationship between related objects, and generally indicates that there are at least three relationships. For example, A and / or B can at least indicate: the existence of A alone, the existence of both A and B, and the existence of B alone.

[0030] It should be noted that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Unless otherwise specified, the scope of this disclosure is not limited by the order in which the steps are described in the relevant embodiments.

[0031] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0032] Exemplary methods Figure 1 This is a flowchart illustrating an evaluation method for an autonomous driving safety verification platform according to an exemplary embodiment, such as... Figure 1 As shown, the evaluation method for the autonomous driving safety verification platform includes the following steps.

[0033] In step S110, the test task is sent to the message queue corresponding to the container group according to the container group identifier carried by the test task; the container group includes a first container that encapsulates the scene control service system and a second container that encapsulates the algorithm of the system under test; the first container and the second container are connected to each other through a communication module.

[0034] In step S120, each scene control service system obtains the test task from the corresponding message queue, and calls a preset scene rendering (hereinafter referred to as UE) simulation engine through an interface to load the scene data carried in the test task into the UE simulation engine to create a test scene; the test scene is then fed back to the corresponding scene control service system; the scene control service system calls the algorithm of the system under test through an interface, loads the algorithm of the system under test into the corresponding test scene for simulation testing, and obtains the test results; In step S130, the test results are evaluated using preset evaluation indicators.

[0035] The evaluation method of the autonomous driving safety verification platform provided by this invention includes scene data and container group identifiers in each task. Based on the container group identifiers, the corresponding scene data is distributed to the corresponding message queues. This facilitates the execution of task data retrieved from the corresponding message queues by each scene control service system. Then, the algorithm of the system under test is loaded into the required test scenario for testing, avoiding situations where the test scenarios retrieved by each scene control service system are not the scenarios required by the algorithm of the system under test, thus making the testing process more efficient. Furthermore, the scene control service system and the system under test are encapsulated in two communication-connected containers. During test execution, the scene control service system calls the algorithm of the system under test through an interface, loading the algorithm into the corresponding test scenario for simulation testing. This enables multiple systems under test to independently and in parallel perform simulation testing of the algorithm under test, and allows one or more systems under test to perform simulation testing in one or more simulation environments. This solves the problem that the evaluation system cannot perform multiple test tasks simultaneously, and that different functional tests must be performed sequentially, resulting in excessive time costs.

[0036] refer to Figure 2 In some embodiments, a flowchart illustrating the task execution of an autonomous driving safety verification platform provided in this embodiment is shown. This embodiment provides an evaluation method that can complete the test tasks of defining and instantiating platform services through configuration files and script files. For example, test scenario information generation code is compiled into an executable file and added to the test task. Testers only need to write configuration files according to certain specifications and script files according to different services. The test task includes the scenario data required for testing the system under test and the corresponding container group identifier, such as a unique domain DI (domain_id). Multiple message queues A1-An each correspond to a communication connection to a scenario control service system, and each system under test... The test tasks are distributed and processed through n message queues. The system sends task data to the corresponding message queues A1-An based on the domain_id in the test task. This setting adds the required scenario data for the system under test to the task queues before testing. Through one-to-one containerized deployment of each scenario control service system and each algorithm of the system under test, the system under test's algorithms are loaded flawlessly into the required test scenarios during testing. This allows them to run in the virtual scenario built by the autonomous driving environment construction module, controlling the actions and responses of virtual target vehicles to verify the autonomous driving algorithm and obtain test results. The scenario control service system can be a service manager based on a PaaS cloud platform, responsible for allocating and executing virtual test resources and providing runtime data feedback.

[0037] To avoid message propagation and reception confusion caused by inconsistent information interaction between the scene control service system, the algorithm under test, and the simulation engine, which could prevent the corresponding algorithm under test from being loaded into the corresponding test scene for simulation testing, the first container and the second container are connected as communication middleware through separately configured CyberRT communication modules. A scene control service system and a system under test need to communicate through CyberRT. Messages during communication are broadcast within the local area network, and the same domain_id is used during communication to ensure the accuracy of message communication between the two ends and prevent message sending and receiving confusion.

[0038] In some embodiments, multiple UE containers encapsulating the UE emulation engine are configured and managed by a UE agent. The UE agent controls the access permissions of the UE emulation engine in the system, optimizes resource scheduling, and reduces system energy consumption. The UE agent is responsible for load balancing and unified request forwarding of the UE containers, and also acts as an interface to provide access to application requests. (See reference...) Figure 3 This is a flowchart illustrating the task execution of an autonomous driving safety verification platform provided in this embodiment. (Refer to...) Figure 4 This is a flowchart of the task processing of the scene control service system provided in this embodiment. This embodiment provides an evaluation method. In step S120, each scene control service system calls a preset UE simulation engine through an interface, including: Step S1201: Each of the scene control service systems sends a call instruction to the UE agent; Optionally, the invocation instruction may include unique identity information, such as container name, address, or domain_id, to facilitate the UE agent in determining the scene control service system that needs to be invoked from the simulation engine and to feed back the address of the available UE container to the corresponding scene control service system. It may also include an available UE container query command, which is used by the UE agent to query the usage status of multiple UE containers it proxies.

[0039] Step S1202: The UE agent queries the usage status of multiple UE containers that encapsulate the UE simulation engine according to the call instruction, and feeds back the address of the UE container that is in an unused state to the corresponding scene control service system. Optionally, the UE proxy is responsible for load balancing and unified request forwarding for each of the scene control service systems, and also serves as an interface to provide access to application requests. When the UE proxy receives a call instruction, it will access the multiple UE containers it proxies in sequence. When a UE container is in use, it has initiated application isolation and entered an access-restricted state, and the UE proxy cannot access it. If the UE proxy can access one of the UE containers, it means that the UE container is in an unisolated state. In this case, the UE proxy will feed back the UE container address to the corresponding scene control service system according to the unique identity information of the scene control service system in the call instruction.

[0040] Step S1203: The scene control service system calls the simulation engine through the interface based on the UE container address.

[0041] In this embodiment, when running a task, an available UE container address is obtained through the UE agent. The UE agent queries the UE container usage status list to obtain an idle and available UE container, interacts with the UE, and the UE container is used to render the scene based on the scene data and then feeds it back to the scene control service system to optimize the use of multiple UE containers.

[0042] In one specific embodiment, reference is made to Figure 5 This embodiment provides a scene control service system and task processing timing diagram. In step S120, the scene control service system calls a preset UE simulation engine through an interface, and loads the scene data carried in the test task and the algorithm of the system under test into the UE simulation engine for simulation testing, obtaining test results including: Step S1211: Each scene control service system obtains the test task from the corresponding message queue A; Step S1212: The UE agent queries the usage status of multiple UE containers according to the calling instructions of each scenario control service system, and obtains an idle UE container. Step S1213: Set the usage status of the corresponding container to "used"; Step S1214: Return the UE container address to the scene control service system; Step S1215: The scene control service system calls the corresponding UE container according to the received UE container addresses, loads the scene data carried in the test task into the UE simulation engine to create a test scene, and feeds back the test scene to the corresponding scene control service system; the scene control service system calls the algorithm of the system under test through the interface, loads the algorithm of the system under test into the corresponding test scene for simulation testing, and obtains the test results; Step S1216: Feedback the task completion information to the UE agent; Step S1217: The UE agent updates the UE container usage status list; Step S1218: Set the current usage status of the UE container to unused and send the update result back to the scene control service system.

[0043] In some embodiments, reference Figure 6 This embodiment provides a flowchart for processing evaluation service tasks. In step S130, evaluating each of the test results includes: S1301, The respective scenario control service systems will generate test results carrying evaluation containers. site The tasks to be evaluated are sent to the evaluation message queue; Optionally, after obtaining the test results, in order to efficiently evaluate the test results, this embodiment sets up multiple evaluation service systems that can simultaneously evaluate the test results and are respectively encapsulated in the evaluation containers. Each evaluation container also presets evaluation indicators for evaluating the test results, so that each evaluation container can evaluate the preset evaluation tasks. For example, multiple evaluation service systems are set up one-to-one with the multiple container groups, and each evaluation service system is respectively responsible for evaluating the test results of one of the algorithms of the tested system. In this embodiment, to achieve the above objective, testers can add the evaluation container address to the test task when writing the test task. After each scenario control service system completes the simulation test, the evaluation container address is added to the corresponding test result to generate the task to be evaluated and sent to the message queue to be evaluated. Or When deploying the scene control service system, it is configured that after each scene control service system completes simulation testing, it will by default generate a task to be evaluated carrying the address of a specified evaluation container. For example, if the evaluation service system A is configured to send a test task to the evaluation message queue A, then the scene control service system A connected to message queue A will, after completing the simulation test, generate an evaluation container A containing the address of the evaluation service system A. site.

[0044] S1302. The evaluation agent service obtains the task to be evaluated from the message queue to be evaluated, and calls the evaluation container through an interface according to the evaluation container address carried by the task to be evaluated. The evaluation container encapsulates the evaluation service system and evaluation indicators. The evaluation service system obtains the task to be evaluated from the message queue to be evaluated, and evaluates the test results according to the evaluation indicators.

[0045] In this embodiment, multiple evaluation service systems are set up to evaluate the test results. The evaluation container address carried by the task to be evaluated can be added to the test task by the tester when writing the test task. After each scenario control service system completes the simulation test, the evaluation container address is added to the corresponding test result to generate the task to be evaluated and sent to the message queue to be evaluated. The evaluation proxy service calls the corresponding evaluation container according to the evaluation container address in the task to be evaluated to complete the test result evaluation, so that multiple evaluation tasks can be carried out simultaneously and independently in parallel.

[0046] In other embodiments, to achieve flexible scheduling of each evaluation container, the evaluation proxy service uses a multi-container load balancing strategy to perform dynamic operation and maintenance of multiple evaluation containers, dynamically maintaining the number of tasks running in each container, allocating tasks within containers, and reclaiming containers. This improves the utilization rate of each evaluation container while ensuring the continuity and reliability of the evaluation service system. (Reference) Figure 7 Here is another evaluation service task processing flowchart provided in this embodiment. In step S130, evaluating each of the test results includes: Step S1311: The scenario control service system generates an evaluation task carrying evaluation indicators from the obtained test results and sends it to the evaluation message queue. Optionally, in order to achieve flexible scheduling of multiple evaluation containers and to implement dynamic operation and maintenance of multiple evaluation containers using a multi-container load balancing strategy, testers can add the evaluation index information of each test task to the test task when writing the test task. After each scenario control service system completes the simulation test, the evaluation index information carried in the corresponding test task is added to the task to be evaluated and sent to the message queue to be evaluated. Or When deploying the scenario control service system, after each scenario control service system completes simulation testing, it is set to add preset evaluation index information to the task to be evaluated and send it to the message queue to be evaluated by default. Step S1312: The evaluation proxy service queries multiple evaluation containers that encapsulate the evaluation service system, determines the address of the evaluation container that can run the evaluation task, and calls the evaluation service system through an interface according to the evaluation container address. The evaluation service system obtains the task to be evaluated from the message queue to be evaluated and evaluates the test results according to the evaluation index carried by the task to be evaluated.

[0047] In one specific embodiment, reference is made to Figure 8The following is a sequence diagram of the evaluation service system and task processing provided in this embodiment. In step S1312, the evaluation proxy service queries multiple evaluation containers that encapsulate the evaluation service system, determines the address of the evaluation container that can run the evaluation task, and calls the evaluation service system through an interface based on the evaluation container address, including: Step S1320: The evaluation agent service obtains the task to be evaluated from the message queue B to be evaluated; Step S1321: Query the evaluation containers that can run evaluations among the multiple evaluation containers according to the preset threshold for the number of tasks to be evaluated that can be run in each of the evaluation containers; Step S1322: Determine the address of the evaluation container C with the fewest tasks to be evaluated. For example, if an evaluation container is preset to run a maximum of 10 tasks, and 3 tasks have already been run, then the remaining number of runnable tasks is 7. If the current number of runnable tasks in all evaluation containers is 0, it means that there are no evaluation containers with runnable evaluation tasks, then return to step S1321; otherwise, select the container C with the most remaining runnable tasks from all containers and return the address of container C. Step S1323: The evaluation agent service asynchronously calls the evaluation service system interface of the evaluation container C according to the evaluation container address, and at the same time updates the number of running tasks in the evaluation container C by 1, and the evaluation agent service repeats step S1320. Step S1324: When the evaluation service system interface of container C is executed, a task to be evaluated is retrieved from the message queue to be evaluated B to run the evaluation task. Step S1325: After the evaluation task is completed, decrement the number of running tasks in the container where the task is located by 1, and call back the platform service interface to return the corresponding task ID to notify that the evaluation task has been completed.

[0048] In some embodiments, the priority or importance of each test task is determined during the writing process. For example, a queue parameter is set for each test task, and the priority or importance is specified by setting the queue attribute. After each test task is sent to the corresponding message queue, the system will arrange the test task in the corresponding position in the queue according to the set priority or importance. Similarly, after the simulation test is completed, the test results are obtained and sent to the message queue to be evaluated, the system will arrange the evaluation task in the corresponding position in the queue according to the set priority or importance.

[0049] In some embodiments, the evaluation service can evaluate the test results according to the set evaluation indicators, such as comparing the test results with the expected test results of the test algorithm, and finally obtaining an evaluation report on whether the test results are qualified (such as the algorithm's security, compliance, comfort, and other indicators).

[0050] In some embodiments, the testing of the algorithm under test includes, but is not limited to, the security, compliance, accuracy, and limits of the algorithm. For example, setting multiple different levels of scene conditions to test the processing limits of the algorithm, such as testing the accuracy, acceleration performance, and maximum speed of the visual sensor under different traffic environmental factors (weather, vehicle speed, object size, etc.). In this embodiment, the same or different autonomous driving algorithms are deployed in multiple algorithm containers of the system under test. For the deployment of the same autonomous driving algorithm in multiple algorithm containers of the system under test, please refer to the following embodiments. The same or different autonomous driving algorithms are deployed in multiple algorithm containers of the system under test; for the deployment of the same autonomous driving algorithm in multiple algorithm containers of the system under test, please refer to the following example: Specifically, the same autonomous driving algorithm can be deployed in two test system algorithm containers to test the vehicle's front-end sensors' ability to identify obstacles under different visibility and weather conditions, and to test the vehicle's front-end sensors' ability to identify obstacles under different obstacle movement speeds in the same weather conditions. This is used to quickly verify the test results of an autonomous driving algorithm under different scenarios and test parameters.

[0051] Alternatively, testing and evaluating the system under test (SUT) can start from autonomous driving functional scenarios. During the testing of autonomous driving programs or algorithms, the system under test's algorithm identifies corresponding hazardous events and performs SOTIF risk identification and assessment. The assessment evaluates whether the expected functional safety SOTIF indicators meet the SOTIF target specifications. The expected functional safety SOTIF indicators for autonomous driving systems can be as follows: 1. The target scenario was not adequately considered, and the system under test was unable to respond correctly to the environment; 2. The arbitration mechanism and algorithm of the tested system's functional logic are unreasonable, leading to problems in decision-making; 3. The output of the actuator of the tested system deviates from the ideal output, making perfect control difficult - Execution.

[0052] In summary, the technical solution of this invention is applicable to the task processing and container management method of an autonomous driving safety verification platform. Different scene control service systems and systems under test (with different domain_ids) are divided into multiple message queues for the distribution and processing of test tasks. This enables multiple systems under test to independently and in parallel perform simulation tests of the algorithms under test. While one or more systems under test are being simulated in one or more simulation environments, a scene control service system and a system under test are set up to communicate via CyberRT to ensure the accuracy of message communication between the two ends and prevent message sending and receiving chaos. In addition, this embodiment of the method formulates a container selection strategy for multiple UE containers and evaluation containers, and dynamically maintains the container usage status, task allocation in containers, and container recycling through container management and usage methods. This improves the utilization rate of each container while ensuring the continuity and reliability of scene service or evaluation service usage.

[0053] Exemplary device Figure 9 This is a block diagram of an evaluation device for an autonomous driving safety verification platform, according to an exemplary embodiment. (Refer to...) Figure 10 The device 200 includes a test task generation module 210, a task distribution module 220, a simulation test module 230, and an evaluation module 240. The test task generation module 210 generates test tasks according to test requirements. The test tasks include scene data, container group identifiers, and task IDs. The container group includes a first container that encapsulates the scene control service system and a second container that encapsulates the algorithm of the system under test. The first container and the second container are connected to each other through a communication module. The task distribution module 220 is used to send the test task to the message queue connected to the corresponding container group according to the container group identifier carried by the test task. The simulation test module 230 is used by each of the scene control service systems to obtain the test task in the corresponding message queue, and to call a preset UE simulation engine through an interface to load the scene data carried in the test task into the UE simulation engine to create a test scene, and to feed the test scene back to the corresponding scene control service system; the scene control service system calls the algorithm of the system under test through an interface, loads the algorithm of the system under test into the corresponding test scene for simulation testing, and obtains the test results; The evaluation module 240 is used to evaluate the test results using preset evaluation indicators.

[0054] The evaluation device of the autonomous driving safety verification platform provided by this invention includes a test task generation module 210 that generates scene data and container group identifiers for each task. The task distribution module 220 distributes the corresponding scene data to the corresponding message queue according to the container group identifier. This facilitates the simulation test module 230 to load the algorithm of the system under test into the required test scenario before the scene control service system obtains the task data from the corresponding message queue for testing. This avoids the situation where the test scenario obtained by each scene control service system is not the scenario that the algorithm of the system under test needs to be tested, making the testing process more efficient. In addition, the scene control service system and the system under test are encapsulated in two containers with communication connections. During the test operation, the scene control service system calls the algorithm of the system under test through the interface and loads the algorithm of the system under test into the corresponding test scenario for simulation testing. This enables multiple systems under test to independently and in parallel perform simulation testing of the algorithm under test, and enables one or more systems under test to perform simulation testing in one or more simulation environments. This solves the problem that the evaluation system cannot perform multiple test tasks at the same time, and that different functions need to be tested in a queue, resulting in high time costs.

[0055] In some embodiments, this embodiment provides an evaluation device. The test task generation module 210 can complete the definition and instantiation of platform service test tasks through configuration files and script files. For example, the test scenario information generation code is compiled into an executable file and added to the test task. Testers only need to write configuration files according to certain specifications and script files according to different services. The test task includes the scenario data to be tested for the system under test and the corresponding container group identifier, such as a unique domain DI (domain_id). Multiple message queues A1-An correspond to communication connections to a scenario control service system. The test tasks of each system under test are divided into n message queues for task allocation. The task distribution module 220 sends the task data to the corresponding message queue A1-An according to the domain_id in the test task. This setting adds the scenario data required for testing the corresponding system under test to the task queue before testing. Through one-to-one containerized deployment of each scenario control service system and each algorithm of the system under test, the simulation test module 230 loads each algorithm of the system under test into the test scenario that needs to be tested without error during the test process. This allows the system to run in the virtual scenario built by the autonomous driving environment construction module, controlling the actions and responses of the virtual target vehicle, thereby verifying the autonomous driving algorithm and obtaining test results. The scenario control service system can be a service manager based on a PaaS cloud platform, which is responsible for allocating and executing virtual test resources and providing feedback on running data.

[0056] In some embodiments, to avoid message propagation and reception confusion caused by the non-single information interaction between the scene control service system, the algorithm under test, and the simulation engine, which would prevent the corresponding algorithm under test from being loaded into the corresponding test scene for simulation testing, the first container and the second container are connected as communication middleware through separately configured CyberRT communication modules. A scene control service system and a system under test need to communicate through CyberRT. Messages during communication are broadcast within the local area network, and the same domain_id is used during communication to ensure the accuracy of message communication between the two ends and prevent message sending and receiving confusion.

[0057] In some embodiments, multiple UE containers encapsulating the UE emulation engine are configured and managed by a UE agent. The UE agent controls the access permissions of the UE emulation engine in the system, optimizes resource scheduling, and reduces system energy consumption. The UE agent is responsible for load balancing and unified request forwarding of the UE containers, and also acts as an interface to provide access to application requests. (See reference...) Figure 10 This is a block diagram of an evaluation device for another autonomous driving safety verification platform provided in this embodiment. This embodiment provides an evaluation device, which also includes: Each UE container encapsulates the UE emulation engine. UE proxy module 250: Used to manage the usage status of multiple UE containers, query the usage status of each UE container according to the call instructions sent by each scene control service system, and feed back the address of the UE container that is in an unused state to the corresponding scene control service system; The scene control service system calls the simulation engine through an interface based on the UE container address.

[0058] In one embodiment, the UE proxy module 250 queries the usage status of multiple UE containers according to the calling instructions of each scene control service system, feeds back the address of the UE container that is in an unused state to the corresponding scene control service system, and sets the usage status of the corresponding UE container to used; the UE proxy module 250 sets the usage status of the corresponding UE container to unused according to the test task completion information fed back by the scene control service system.

[0059] In some embodiments, reference Figure 11 This is a block diagram of an evaluation device for another autonomous driving safety verification platform provided in this embodiment. This embodiment provides an evaluation device, which also includes: Evaluation Message Queue: Used to receive and store the evaluation tasks generated by each of the scene control service systems; the evaluation task is an evaluation task with evaluation indicators generated by the scene control service system based on the test results; Evaluation module 240: includes evaluation containers that encapsulate multiple evaluation services; Evaluation agent module 260: Used to manage multiple evaluation containers encapsulated with evaluation services, determine the address of the evaluation container that can run the task to be evaluated, and call the evaluation service through an interface according to the evaluation container address. The evaluation service obtains the task to be evaluated from the message queue to be evaluated, and evaluates the test results according to the evaluation indicators carried by the task to be evaluated.

[0060] In this embodiment, the evaluation module 240 sets up multiple evaluation services to evaluate the test results. The evaluation container address carried by the task to be evaluated can be added to the test task by the tester when writing the test task. After each scenario control service system completes the simulation test, the evaluation container address is added to the corresponding test result to generate the task to be evaluated and sent to the message queue to be evaluated. The evaluation agent module 260 calls the corresponding evaluation container according to the evaluation container address in the task to be evaluated to complete the test result evaluation, so that multiple evaluation tasks can be performed simultaneously and independently in parallel.

[0061] In other embodiments, in order to achieve flexible scheduling of each evaluation container, the evaluation agent module 260 uses a multi-container load balancing strategy to perform dynamic operation and maintenance of multiple evaluation containers, dynamically maintain the number of tasks running in each container, allocate tasks in containers and reclaim containers, thereby improving the utilization rate of each evaluation container while ensuring the continuity and reliability of the evaluation service.

[0062] In some embodiments, the evaluation agent module 260 queries the evaluation containers that can run evaluations among multiple evaluation containers according to a preset threshold for the number of tasks to be evaluated that each evaluation container can run, and determines the address of the evaluation container with the fewest tasks to be evaluated at present.

[0063] The device embodiments disclosed herein correspond to the technical solutions of the above-described embodiments of the invention. The specific operation of each module can be understood by referring to the description in the method embodiments. For details, please refer to... Figure 9-11 I understand, so I won't go into details here.

[0064] Exemplary electronic devices Figure 12 This is a block diagram illustrating an electronic device 900 according to an exemplary embodiment. The electronic device 900 may be a vehicle controller, an in-vehicle terminal, an in-vehicle computer, or other types of electronic devices.

[0065] Reference Figure 9The electronic device 900 may include at least one processor 910 and a memory 920. The processor 910 can execute instructions stored in the memory 920. The processor 910 is communicatively connected to the memory 920 via a data bus. In addition to the memory 920, the processor 910 can also be communicatively connected to an input device 930, an output device 940, and a communication device 950 via the data bus.

[0066] Processor 910 can be any conventional processor, such as a commercially available CPU. Processors may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SOCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0067] The memory 920 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0068] In this embodiment of the present disclosure, the memory 920 stores executable instructions, and the processor 910 can read the executable instructions from the memory 920 and execute the instructions to implement all or part of the steps of the evaluation method of the autonomous driving safety verification platform described in any of the exemplary embodiments above.

[0069] Exemplary computer-readable storage media In addition to the methods and apparatus described above, exemplary embodiments of this disclosure may also be a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in any of the methods in the exemplary embodiments described above.

[0070] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0071] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires; electrically erasable programmable read-only memory (EEPROM); erasable programmable read-only memory (EPROM); programmable read-only memory (PROM); read-only memory (ROM); magnetic storage; flash memory; magnetic disk or optical disk; or any suitable combination thereof.

[0072] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0073] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An evaluation method for an autonomous driving safety verification platform, characterized in that, include: Test tasks are generated according to test requirements. The test tasks include scenario data and are sent to the message queue corresponding to the corresponding container group. Based on the container group identifier carried by the test task, the test task is sent to the message queue corresponding to the container group; there are multiple container groups, each of which includes a first container that encapsulates the scene control service system and a second container that encapsulates the algorithm of the system under test; the first container and the second container are connected to each other through a communication module. Each of the scene control service systems obtains the test task in the corresponding message queue, calls the preset scene rendering simulation engine, loads the scene data carried in the test task into the scene rendering simulation engine to create a test scene, and feeds back the test scene to the corresponding scene control service system. The scenario control service system calls the algorithm of the system under test, loads the algorithm of the system under test into the corresponding test scenario for simulation testing, and obtains the test results; The test results are evaluated using preset evaluation indicators.

2. The evaluation method for the autonomous driving safety verification platform according to claim 1, characterized in that, Each scene control service system calls a preset scene rendering simulation engine, including: Each of the aforementioned scene control service systems sends a call instruction to the scene rendering proxy; The scene rendering proxy queries the usage status of multiple scene rendering containers that encapsulate the scene rendering simulation engine according to the calling instruction, and feeds back the address of the scene rendering container that is not in use to the corresponding scene control service system. The scene control service system calls the simulation engine based on the scene rendering container address.

3. The evaluation method for the autonomous driving safety verification platform according to claim 1, characterized in that, The evaluation of the test results includes: Each scenario control service system will generate an evaluation task carrying evaluation indicators from the obtained test results and send it to the evaluation message queue. The evaluation agent service queries multiple evaluation containers that encapsulate the evaluation service system, determines the address of the evaluation container that can run the evaluation task, and calls the evaluation service system according to the evaluation container address. The evaluation service system obtains the task to be evaluated from the message queue to be evaluated, and evaluates the test results according to the evaluation indicators carried by the task to be evaluated.

4. The evaluation method for the autonomous driving safety verification platform according to claim 1, characterized in that, The evaluation of the test results includes: Each scenario control service system will generate an evaluation task carrying the evaluation container address from the test results and send it to the evaluation message queue; The evaluation agent service obtains the task to be evaluated from the message queue to be evaluated, and calls the evaluation container according to the evaluation container address carried by the task to be evaluated. The evaluation container encapsulates the evaluation service system and evaluation indicators. The evaluation service obtains the task to be evaluated from the message queue to be evaluated, and evaluates the test results according to the evaluation indicators.

5. The evaluation method for the autonomous driving safety verification platform according to claim 2, characterized in that, The scene control service system calls a preset scene rendering simulation engine and loads the scene data carried in the test task and the algorithm of the system under test into the scene rendering simulation engine for simulation testing. The test results obtained include: Each of the aforementioned scenario control service systems obtains the test task from the corresponding message queue; The scene rendering proxy queries the usage status of multiple scene rendering containers according to the call instructions of each scene control service system, feeds back the address of the scene rendering container that is in an unused state to the corresponding scene control service system, and sets the usage status of the scene rendering container to used. Each scene control service system calls the corresponding scene rendering container according to the received scene rendering container address, loads the scene data carried in the test task into the scene rendering simulation engine to create a test scene, and feeds back the test scene to the corresponding scene control service system; the scene control service system calls the algorithm of the system under test, loads the algorithm of the system under test into the corresponding test scene for simulation testing, and obtains the test result; it feeds back the task completion information to the scene rendering agent, and the scene rendering agent sets the usage status of the corresponding scene rendering container to unused.

6. The evaluation method for the autonomous driving safety verification platform according to claim 3, characterized in that, The evaluation proxy service queries multiple evaluation containers that encapsulate evaluation service systems, determines the address of the evaluation container capable of running the task to be evaluated, and invokes the evaluation service system based on the evaluation container address, including: The evaluation proxy service queries the evaluation containers that can run evaluations among multiple evaluation containers according to a preset threshold for the number of tasks to be evaluated that each evaluation container can run, and determines the address of the evaluation container with the fewest tasks to be evaluated. The evaluation proxy service then calls the evaluation service system based on the evaluation container address.

7. The evaluation method for the autonomous driving safety verification platform according to claim 3 or 6, characterized in that, The evaluation agent service uses a multi-container load balancing strategy to perform dynamic operation and maintenance of multiple evaluation containers.

8. The evaluation method for the autonomous driving safety verification platform according to claim 1, characterized in that, The first container and the second container are connected via CyberRT communication modules, which are respectively configured as communication middleware.

9. The evaluation method for the autonomous driving safety verification platform according to any one of claims 1-6 or 8, characterized in that, The algorithms of each tested system can be the same or different.

10. An evaluation device for an autonomous driving safety verification platform, characterized in that, It includes a test task generation module, a task distribution module, a simulation test module, and an evaluation module; The test task generation module generates test tasks according to test requirements. The test tasks include scene data and are sent to the message queue corresponding to the container group. The container group includes a first container that encapsulates the scene control service system and a second container that encapsulates the algorithm of the system under test. The first container and the second container are connected to each other through a communication module. The task distribution module is used to send the test task to the message queue connected to the corresponding container group according to the container group identifier carried by the test task. The simulation testing module is used by each scene control service system to obtain the test task in the corresponding message queue, call the preset scene rendering simulation engine, load the scene data carried in the test task into the scene rendering simulation engine to create a test scene, and feed the test scene back to the corresponding scene control service system; the scene control service system calls the algorithm of the system under test, loads the algorithm of the system under test into the corresponding test scene for simulation testing, and obtains the test results; The evaluation module is used to evaluate the test results using preset evaluation indicators.

11. The evaluation apparatus for the autonomous driving safety verification platform according to claim 10, characterized in that, The device further includes: Multiple scene rendering containers are used to encapsulate the scene rendering simulation engine. Scene rendering proxy module: used to manage the usage status of multiple scene rendering containers, query the usage status of each scene rendering container according to the call instructions sent by each scene control service system, and return the address of the scene rendering container that is in an unused state to the corresponding scene control service system; The scene control service system calls the simulation engine based on the scene rendering container address.

12. The evaluation apparatus for the autonomous driving safety verification platform according to claim 10, characterized in that, The device further includes: The first message queue to be evaluated is used to receive and store the tasks to be evaluated generated by each of the scenario control service systems; the task to be evaluated is a task with evaluation indicators generated by the scenario control service system based on the test results. First evaluation module: includes the first evaluation container for multiple encapsulated evaluation service systems; First evaluation agent module: used to manage multiple first evaluation containers encapsulated with evaluation service systems, determine the address of the first evaluation container that can run the task to be evaluated, and call the evaluation service system according to the address of the first evaluation container. The evaluation service system obtains the task to be evaluated from the first message queue to be evaluated, and evaluates the test results according to the evaluation indicators carried by the task to be evaluated.

13. The evaluation apparatus for the autonomous driving safety verification platform according to claim 10, characterized in that, The device further includes: The second message queue to be evaluated is used to receive and store the tasks to be evaluated generated by each of the scenario control service systems; the tasks to be evaluated are the tasks generated by each scenario control service system from the test results and carrying the address of the evaluation container. The second evaluation module includes a second evaluation container containing multiple encapsulated evaluation service systems and evaluation indicators. The second evaluation agent module is used to obtain the task to be evaluated in the second message queue to be evaluated, and to call the second evaluation container according to the address of the second evaluation container carried by the task to be evaluated. The evaluation service system obtains the task to be evaluated in the second message queue to be evaluated, and evaluates the test results according to the evaluation indicators.

14. The evaluation apparatus for the autonomous driving safety verification platform according to claim 11, characterized in that, The scene rendering proxy module queries the usage status of multiple scene rendering containers according to the call instructions of each scene control service system, feeds back the address of the scene rendering container that is in an unused state to the corresponding scene control service system, and sets the usage status of the corresponding scene rendering container to used. The scene rendering proxy module sets the usage status of the corresponding scene rendering container to unused based on the test task completion information fed back by the scene control service system.

15. The evaluation apparatus for the autonomous driving safety verification platform according to claim 12, characterized in that, The evaluation agent module determines the evaluation container address on which the task to be evaluated can run, including: The evaluation agent service queries the evaluation containers that can run evaluations among multiple evaluation containers according to the preset threshold for the number of tasks to be evaluated that each evaluation container can run, and determines the address of the evaluation container with the fewest tasks to be evaluated. The evaluation agent service then calls the evaluation service system based on the evaluation container address.

16. The evaluation apparatus for the autonomous driving safety verification platform according to claim 12 or 13, characterized in that, The evaluation agent service uses a multi-container load balancing strategy to perform dynamic operation and maintenance of multiple evaluation containers.

17. The evaluation apparatus for the autonomous driving safety verification platform according to claim 10, characterized in that, The first container and the second container are connected via CyberRT communication modules, which are respectively configured as communication middleware.

18. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the evaluation method of the autonomous driving safety verification platform according to any one of claims 1-9.

19. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the steps of the evaluation method of the autonomous driving safety verification platform according to any one of claims 1-9.

Images