Autonomous driving simulation test system, method, computer device, readable storage medium and program product

Abstract

The present application relates to an autonomous driving simulation test system, method, computer device, computer-readable storage medium and computer program product. The method includes: a scene player for playing a pre-generated scene script; an image renderer for rendering images based on the environment state and the main vehicle state, and obtaining sensor data based on the rendered image; an end-to-end model module for generating a main vehicle control instruction based on the main vehicle state and sensor data; an engine kernel for calling back the scene player, image renderer, end-to-end model module and simulation detector module based on each pre-set callback frequency, and updating the environment state based on the scene script played by the scene player, and updating the main vehicle state based on the main vehicle control instruction; a simulation detector module for generating simulation test results based on the environment state and the main vehicle state. The use of this method can effectively improve the real-time performance and accuracy of the simulation system.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an autonomous driving simulation test system, method, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the continuous advancement of autonomous driving technology, various intelligent driving algorithms are gradually being developed and applied. Before applying these algorithms to actual driving scenarios, simulation testing is an important means to verify their effectiveness and safety. Simulation testing can not only be carried out in a safe virtual environment, but also greatly save the time and cost required for actual road testing.

[0003] In traditional technologies, during simulation testing, most autonomous driving simulation systems require manual construction of simulation scenarios, and the management between modules is inflexible, resulting in the need to improve the real-time and accuracy of the simulation system. Summary of the invention

[0004] Based on this, it is necessary to provide an autonomous driving simulation test system, method, computer equipment, computer-readable storage medium and computer program product that can effectively improve the real-time and accuracy of the simulation system in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides an autonomous driving simulation test system, the system comprising:

[0006] Scenario player, used to play pre-generated scenario scripts;

[0007] An image renderer, used for rendering an image based on an environment state and a host vehicle state, and obtaining sensor data based on the rendered image;

[0008] An end-to-end model module, used for generating a main vehicle control instruction based on the main vehicle state and the sensor data;

[0009] The engine kernel is used to call back the scene player, the image renderer, the end-to-end model module and the simulation detector module based on the preset callback frequencies, and update the environment state based on the scene script played by the scene player, and update the main vehicle state based on the main vehicle control instruction;

[0010] The simulation detector module is used to generate simulation test results based on the environmental state and the host vehicle state.

[0011] In one embodiment, the engine kernel includes:

[0012] A module scheduler, the module scheduler is used to receive module registration instructions, generate callback functions corresponding to the scene player, image renderer, end-to-end model module and simulation detector module based on the module registration instructions, configure the callback frequency of the callback function, and call the scene player, the image renderer, the end-to-end model module and the simulation detector module through the callback function; and the programming languages ​​corresponding to the scene player, image renderer, end-to-end model module and simulation detector module are the same or different.

[0013] In one embodiment, the scenario script includes at least one of a real scenario script and a virtual scenario script; the real scenario script is generated based on scenario data obtained when a vehicle is traveling in a real scenario; and the virtual scenario script is obtained through editing.

[0014] In one of the embodiments, the scene player is used to load the scene script to obtain an initialized environment state and a behavior tree structure, wherein the behavior tree structure includes a plurality of control flow nodes, condition nodes, and behavior nodes; traverse the behavior tree structure to determine the corresponding control flow node, and make a judgment based on each condition node corresponding to the control flow node to obtain the behavior node to be executed, and execute the behavior corresponding to the corresponding behavior node to obtain the environment data.

[0015] In one embodiment, the conditional node includes at least one of a time trigger node and an event trigger node; the time trigger node detects whether the current time reaches a set time point, and when the current time reaches the set time point, a corresponding behavior node to be executed is obtained; the event trigger node detects whether the input satisfies an event condition, and when the input satisfies the event condition, a corresponding behavior node to be executed is obtained.

[0016] In one of the embodiments, the image renderer includes a server and a client; the server includes rendering programs for different rendering modes; the engine kernel synchronizes the rendering mode identifier, various environmental states and the main vehicle state to the server through a unified API interface, and the server is used to perform image rendering of the environmental state and the main vehicle state based on the rendering program corresponding to the rendering mode identifier; the client is used to obtain sensor data based on the rendered image.

[0017] In one of the embodiments, the end-to-end model module is accessed through a unified adaptation framework, and processes the main vehicle state and the sensor data to obtain input data that meets the model input requirements, performs model inference on the input data to obtain main vehicle control instructions, and returns the main vehicle control instructions to the engine core.

[0018] In one embodiment, the simulation detector module includes at least one simulation detector, and the simulation detector module is used to configure the switch of each simulation detector and / or the detection index corresponding to each simulation detector based on the configuration instruction; each simulation detector is used to obtain the environmental state and the main vehicle state from the engine core in real time when the switch is turned on, and process the environmental state and the main vehicle state based on the configured detection index to obtain the simulation test result.

[0019] In a second aspect, the present application also provides an autonomous driving simulation test method, the method comprising:

[0020] Based on the preset callback frequencies, the callback scene player plays the pre-generated scene script to obtain the driving environment state of the autonomous driving vehicle, the callback image renderer performs image rendering based on the environment state and the main vehicle state, obtains sensor data based on the rendered image, and the callback end-to-end model module generates the main vehicle control instruction based on the main vehicle state and the sensor data;

[0021] Update the main vehicle state based on the main vehicle control instruction;

[0022] A simulation test result is generated based on the environmental state and the host vehicle state by a simulation detector module.

[0023] In one embodiment, the method further comprises:

[0024] Receive a module registration instruction, wherein the programming languages ​​of the scene player, the image renderer, the end-to-end model module, and the simulation detector module corresponding to the module registration instruction are the same or different;

[0025] The callback functions corresponding to the scene player, image renderer, end-to-end model and simulation detector are generated based on the module registration instruction, and the callback frequency of the callback function is configured.

[0026] In one embodiment, the scenario script includes at least one of a real scenario script and a virtual scenario script; the real scenario script is generated based on scenario data obtained when a vehicle travels in a real scenario; and the virtual scenario script is obtained through editing.

[0027] In one embodiment, the callback scene player plays a pre-generated scene script to obtain an environment state, including:

[0028] Calling back the scene player to load the scene script to obtain the initial environment state and behavior tree structure, wherein the behavior tree structure includes a number of control flow nodes, condition nodes, and behavior nodes;

[0029] The callback scene player traverses the behavior tree structure to determine the corresponding control flow node, and makes a judgment based on each condition node corresponding to the control flow node to obtain the behavior node to be executed, and executes the behavior corresponding to the corresponding behavior node to obtain the environment data.

[0030] In one embodiment, the conditional node includes at least one of a time triggering node and an event triggering node; the judgment based on each conditional node corresponding to the control flow node to obtain the behavior node to be executed includes at least one of the following:

[0031] Based on the time trigger node corresponding to the control flow node, detecting whether the current time reaches the set time point; when the current time reaches the set time point, obtaining the corresponding behavior node to be executed; and

[0032] Based on the event triggering node corresponding to the control flow node, it is detected whether the input meets the event condition, and when the input meets the event condition, a corresponding behavior node to be executed is obtained.

[0033] In one embodiment, the callback image renderer performs image rendering based on the environment state and the host vehicle state, and obtains sensor data based on the rendered image, including:

[0034] The rendering mode identifier, various environmental states and main vehicle states are synchronized to the server of the image renderer through a unified API interface. The server is used to perform image rendering of the environmental states and main vehicle states based on a rendering program corresponding to the rendering mode identifier, and obtain sensor data based on the rendered image through the client of the image renderer.

[0035] In one embodiment, the calling back end-to-end model module to generate a main vehicle control instruction based on the main vehicle state and the sensor data includes:

[0036] Calling back an end-to-end model module accessed through a unified adaptation framework, wherein the end-to-end model module is used to process the main vehicle state and the sensor data to obtain input data that meets the model input requirements, and perform model reasoning on the input data to obtain a main vehicle control instruction;

[0037] The main vehicle control instruction is received, and the main vehicle state is updated based on the main vehicle control instruction.

[0038] In one embodiment, before the simulation detector module generates the simulation test result based on the environmental state and the main vehicle state, the method further includes:

[0039] Receive configuration instructions;

[0040] Configure switches of each of the simulation detectors and / or detection indicators corresponding to each of the simulation detectors based on the configuration instructions;

[0041] The generating of the simulation test result based on the environmental state and the host vehicle state by the simulation detector module includes:

[0042] When the switch of the simulation detector is turned on, the environment state and the host vehicle state are acquired from the engine core in real time through the simulation detector;

[0043] The environmental state and the main vehicle state are processed based on the configured detection index to obtain a simulation test result.

[0044] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the above-mentioned embodiments when executing the computer program.

[0045] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method in any one of the above-mentioned embodiments.

[0046] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the method in any one of the above embodiments when executed by a processor.

[0047] The above-mentioned autonomous driving simulation test system, method, computer device, computer-readable storage medium and computer program product, the system includes a scene player, an image renderer, an end-to-end model module, an engine kernel and a simulation detector module; the scene player is used to play a pre-generated scene script; the image renderer is used to render an image based on the environment state and the main vehicle state, and obtain sensor data based on the rendered image; the end-to-end model module is used to generate a main vehicle control instruction based on the main vehicle state and the sensor data; the engine kernel is used to call back the scene player, image renderer, end-to-end model module and simulation detector module based on each preset callback frequency, and update the environment state based on the scene script played by the scene player, and update the main vehicle state based on the main vehicle control instruction; the simulation detector module is used to generate a simulation test result based on the environment state and the main vehicle state, so that the engine kernel calls back the scene player, image renderer, end-to-end model module and simulation detector module based on each preset callback frequency, which can realize real-time update and detection of the simulated scene environment and the main vehicle state, and provide accurate simulation data for autonomous driving effect evaluation through image rendering and model reasoning. This architecture design can effectively improve the real-time and accuracy of the simulation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A framework diagram of an autonomous driving simulation test system in one embodiment;

[0050] Figure 2 A schematic diagram of the callback frequency of each module in an embodiment;

[0051] Figure 3 A schematic diagram of the operation process of a scene player in one embodiment;

[0052] Figure 4 is a workflow diagram of an image renderer in one embodiment;

[0053] Figure 5 is a workflow diagram of an end-to-end model module in one embodiment;

[0054] Figure 6 is a schematic diagram of five situations corresponding to a collision detector in one embodiment;

[0055] Figure 7 A schematic diagram of a flow chart of an autonomous driving simulation test method in one embodiment;

[0056] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] Combination Figure 1 As shown, Figure 1 This is a framework diagram of an autonomous driving simulation test system in an embodiment. The present application provides an autonomous driving simulation test system, including a scene player, an image renderer, an end-to-end model module, an engine kernel, and a simulation detector module.

[0059] The scene player, image renderer, end-to-end model module and simulation detector module are pre-registered in the engine kernel so that the engine kernel can configure the callback frequency of each module to call back the scene player, image renderer, end-to-end model module and simulation detector module.

[0060] The scene player is used to play the pre-generated scene script, obtain the updated information of the environmental elements, and send it to the engine kernel. Specifically, the scene player is responsible for loading and playing the scene script, and updating the environmental status through event-driven and time-driven, including traffic participants such as people, vehicles, and traffic lights.

[0061] The engine kernel is used to update the environment state with the scenario script played by the scenario player, and update the main vehicle state based on the main vehicle control command, and send the updated environment state and main vehicle state to the image renderer. Specifically, the engine kernel includes two units: environment state update and main vehicle state update, which are responsible for receiving the main vehicle control command from the scenario player and the end-to-end model and updating the simulation environment and the physical state of the main vehicle. In addition, the engine kernel is responsible for managing the registration and scheduling of other modules.

[0062] The image renderer is used to render images based on the environment state and the main vehicle state, obtain sensor data based on the rendered image, and send the sensor data to the end-to-end model module. Specifically, the image renderer includes two submodules: rendering element update and image rendering. It is responsible for rendering images according to the real-time state of the main vehicle and the sensor configuration, and outputting the corresponding sensor data.

[0063] The end-to-end model module is used to generate vehicle control instructions based on sensor data and vehicle status, and feed them back to the engine core, so that the engine core can update the vehicle status based on the vehicle control instructions. Specifically, the end-to-end model: is responsible for model reasoning, receives sensor data and outputs vehicle control instructions.

[0064] The simulation detector module is used to generate simulation test results based on the various environmental states and the main vehicle state during the simulation process. Specifically, the simulation detector is responsible for extracting data from the engine core and performing real-time index calculations.

[0065] In this way, through the collaborative work of the above modules, the simulation environment and the status of the main vehicle can be updated and detected in real time, and accurate simulation data can be provided through image rendering and model reasoning to evaluate the effect of autonomous driving. This architecture design can effectively improve the real-time performance and accuracy of the simulation system.

[0066] In the above-mentioned autonomous driving simulation test system, the engine core calls back the scene player, image renderer, end-to-end model module and simulation detector module based on the pre-set callback frequencies, which can realize real-time update and detection of the simulated scene environment and the main vehicle status, and provide accurate simulation data for autonomous driving effect evaluation through image rendering and model reasoning. This architectural design can effectively improve the real-time and accuracy of the simulation system.

[0067] In one of the optional embodiments, the engine kernel includes: a module scheduler, the module scheduler is used to receive module registration instructions, generate callback functions corresponding to the scene player, image renderer, end-to-end model module and simulation detector module based on the module registration instructions, configure the callback frequency of the callback function, and call the scene player, image renderer, end-to-end model module and simulation detector module through the callback function; and the programming languages ​​corresponding to the scene player, image renderer, end-to-end model module and simulation detector module are the same or different.

[0068] Please continue to combine Figure 1 As shown, the engine kernel includes a module scheduler, an environment status update unit and a main vehicle status update unit.

[0069] The module scheduler is used to register and schedule each module. The module scheduler supports high-frequency and high-precision module callback management, and has the ability to simultaneously access modules of multiple programming languages.

[0070] Before the simulation test begins, all external modules complete the callback function registration of each module through the registration interface of the module scheduler and configure the callback frequency of each module. In the subsequent running process, the engine kernel will call back each module according to the specified frequency. Figure 2 As shown, Figure 2 This is a schematic diagram of the callback frequency of each module in an embodiment. In this embodiment, module 1 specifies a module callback frequency of 10hz, and the engine kernel will execute the callback function every 0.1 seconds, that is, at 0ms and 100ms, and other frequencies are similar. Each module calls back based on the pre-configured callback frequency, without interfering with each other, ensuring the accuracy of the simulation time dimension.

[0071] In the above embodiments, strict scheduling logic can ensure that all sub-modules can be accurately scheduled even in a complex simulation environment, thereby ensuring the accuracy of the simulation time dimension.

[0072] The simulation engine uses hybrid programming technology to support C++ / Python and other language modules to register with the module scheduler for easy expansion. That is to say, the programming languages ​​corresponding to the above-mentioned scene player, image renderer, end-to-end model module and simulation detector module are the same or different.

[0073] In summary, the above embodiment supports high-frequency and high-precision callback management of 100 Hz, can easily access multiple language modules, realize efficient real-time data processing and algorithm calculation, and improve the accuracy and efficiency of simulation testing.

[0074] The above-mentioned environment state update unit is responsible for receiving the input of the scene player and updating the state of traffic participants in the environment, including pedestrians, motor vehicles, non-motor vehicles and traffic lights, etc. It is a key unit to realize the dynamics of the simulation system. The scene player can update the state of traffic participants in the environment based on event-driven or time-driven to achieve the update of the environment state. Among them, event-driven can be to determine whether the corresponding traffic participant meets the corresponding conditions, and time-driven can be to check whether the current time reaches the set time point, etc. After the above processing, the state of the traffic participant can be updated.

[0075] The main vehicle state update unit is used to update the main vehicle state. In fact, the main vehicle state is updated based on the main vehicle control instructions of the end-to-end model module. Optionally, the main vehicle state update unit may include a dynamic model, and the main vehicle control instructions may include throttle, brake, steering and other control instructions. The main vehicle control instructions are processed by the dynamic model to update the main vehicle state. The main vehicle state may include position, direction, speed, acceleration, front wheel angle, etc. The main vehicle state update module may support access to different dynamic models to improve flexibility.

[0076] In the above embodiment, firstly, the module scheduler is used to support high-frequency and high-precision callback management for access of multiple language modules, so that seamless access of multiple language modules (such as C++, Python) can be supported. In addition, the setting of the callback frequency can achieve high-frequency callback, thereby improving the efficiency and accuracy of the simulation test.

[0077] In one optional embodiment, the scenario script includes at least one of a real scenario script and a virtual scenario script; the real scenario script is generated based on scenario data obtained when the vehicle is driving in a real scenario; and the virtual scenario script is obtained through editing.

[0078] The scene player supports the playback of scene scripts, which include real scene scripts and virtual scene scripts. The real scene script is generated based on the scene data obtained when the vehicle is driving in the real scene. For example, the scene data obtained when the vehicle is driving in the real scene is recorded in chronological order, so that the playback of the real scene script can be triggered based on time triggering later.

[0079] The virtual scene script supports user editing. For example, the user manually edits the virtual scene script based on the scene to be tested, wherein the virtual scene script can be edited based on a third-party scene editor.

[0080] Therefore, the format of the scene script played by the above-mentioned scene player may include but is not limited to using scene script files in different formats (such as XML, JSON) or integrating a third-party scene editor format.

[0081] In the above embodiment, the scene scripts that can be played by the scene player support real scene scripts and virtual scene scripts, support users to manually edit complex scenes to obtain virtual scene scripts, and are also compatible with playback modes, such as LogSim playback mode, so that real scene scripts can be played back, thereby improving the diversity and practicality of test scenarios.

[0082] In one of the optional embodiments, the scene player is used to load the scene script to obtain the initial environment state and behavior tree structure, the behavior tree structure includes a number of control flow nodes, condition nodes and behavior nodes; traverse the behavior tree structure to determine the corresponding control flow node, and make a judgment based on each condition node corresponding to the control flow node to obtain the behavior node to be executed, and execute the behavior corresponding to the corresponding behavior node to obtain the environment data.

[0083] For ease of understanding, combined Figure 3 As shown, Figure 3 It is a schematic diagram of the operation process of the scene player in an embodiment. In this embodiment, the operation process of the scene player includes an initialization phase and an operation phase.

[0084] The initialization phase is used to load the scenario script to obtain the initialized environment state and behavior tree structure, wherein recording the scenario script refers to loading the corresponding scenario script file, such as loading a scenario script file in JSON format or XML format. Initializing the environment state includes initializing the environment state based on the environment elements in the scenario script, such as initializing the state of each traffic participant, including the initial state of pedestrians, motor vehicles, non-motor vehicles, and traffic lights. It should be noted that only the state of the environment elements is initialized here, and the main vehicle state is initialized by the engine kernel.

[0085] The behavior tree structure is loaded based on the scene script and configures the corresponding nodes, which include control flow nodes, condition nodes and behavior nodes. The behavior tree structure forms a decision-making and action execution process.

[0086] The control flow node is a node of different processes. The control flow node may include multiple behavior nodes and condition nodes. The condition node is a node used to determine whether the corresponding condition is met. The condition node can be a time-triggered node or an event-triggered node. The behavior node corresponds to the branch of the condition node, and different condition nodes correspond to different branches.

[0087] Continue to combine Figure 3 As shown in the figure, the running stage is based on the behavior tree structure. After initialization, the behavior tree structure starts to execute at each time step. Each time the behavior tree structure executes, it traverses the nodes, makes judgments based on the condition nodes, and executes the behaviors corresponding to the corresponding behavior nodes.

[0088] Optionally, the conditional node includes at least one of a time trigger node and an event trigger node. The time trigger node detects whether the current time reaches a set time point, and when the current time reaches the set time point, a corresponding behavior node to be executed is obtained. Specifically, it is triggered at a preset time interval. For example, it is updated once per second. The time trigger node checks whether the current time reaches a set time point, and if the condition is met, the corresponding action or control flow is executed. For ease of understanding, taking a real-scene script as an example, since the real-scene script is recorded based on time, when revisiting the real-scene script, it can be done based on time triggering, so that the simulation scene is consistent with the real scene.

[0089] The event trigger node detects whether the input meets the event condition. When the input meets the event condition, the corresponding behavior node to be executed is obtained. Among them, the event trigger node waits for the occurrence of a specific event. For example, state change, external input, etc. When the event condition is met, the event trigger node is activated and the corresponding action or control flow is executed. For the convenience of understanding, taking the virtual scene script as an example, the virtual scene script can include multiple events, which are edited, so when playing the virtual scene script, it can be based on event triggering.

[0090] Continue to combine Figure 3 As shown, the engine kernel calls back the scene player according to the frequency to update the simulation state; in addition, the scene player can also call the interface of the engine kernel to update the simulation state and output the updated environment state for use by other modules.

[0091] In the above embodiments, the scenario editor supports manual editing of complex test scenarios and is compatible with the LogSim playback mode, thereby enhancing the diversity and practicality of the test scenarios and meeting different test requirements.

[0092] In one of the optional embodiments, the image renderer includes a server and a client; the server includes rendering programs for different rendering modes; the engine kernel synchronizes the rendering mode identifier, various environmental states and the main vehicle state to the server through a unified API interface, and the server is used to perform image rendering of the environmental state and the main vehicle state based on the rendering program corresponding to the rendering mode identifier; the client is used to obtain sensor data based on the rendered image.

[0093] Among them, combined Figure 4 As shown, Figure 4 The figure is a workflow diagram of an image renderer in an embodiment. In this embodiment, the image renderer CS architecture (the client and the server are separated) includes a server and a client; the server includes rendering programs of different rendering modes; the engine kernel and the client interact with the server through a unified API interface.

[0094] Specifically, the server communicates with the client and the engine kernel to exchange data through the unified rendering API (protobuf) definition. The image renderer server includes two units: a state synchronization module and an image rendering module.

[0095] The state synchronization module is for updating the rendering elements. The rendering element update refers to the synchronization of the states of each element, that is, the engine kernel synchronizes the rendering mode identifier, each environment state and the main vehicle state to the server through a unified API interface. The rendering mode identifier is used to characterize the rendering mode of the current frame image. Different rendering modes correspond to different rendering backend programs. For example, the rendering backend program may include a world model (WorldModel) based on a neural network, and a traditional electronic art (CG) solution such as the Unreal5 engine. In other embodiments, the rendering backend program may also include other types, which are not specifically limited here.

[0096] The engine kernel synchronizes the simulation state to the state synchronization module of the renderer server, where the simulation state includes the environment state and the main vehicle state.

[0097] After data synchronization, the image rendering module uses a specific rendering program to render the image. The specific rendering program is selected based on the rendering mode identifier. The server renders the image based on the rendering program, and after serializing the image, it transmits it back to the image renderer client through the unified API interface. The client deserializes and continues to deserialize the image data to obtain the sensor data, where the client can directly use the deserialized image data as sensor data and send the sensor data to the end-to-end model module.

[0098] In the above embodiment, the unified rendering API decouples the specific code implementation of the rendering server and client through the interface, and is a core component that supports the access of multiple different types of image rendering tools. Such a unified rendering API supports flexible access of different types of rendering modules based on gRPC communication, solves the problem of inconsistent rendering interfaces, thereby improving the image rendering effect of the simulation system and the realism of the simulation environment.

[0099] In one of the optional embodiments, the end-to-end model module is accessed through a unified adaptation framework, and the main vehicle status and sensor data are processed to obtain input data that meets the model input requirements, model inference is performed on the input data to obtain the main vehicle control instructions, and the main vehicle control instructions are returned to the engine kernel.

[0100] The end-to-end model module is accessed through a unified adaptation framework. For example, a universal end-to-end model access test API is provided. This can reduce the adaptation work of different types of models and hardware, so that the model to be tested can be accessed using a unified sensor interface and dynamics close to the actual vehicle, solving the problem of complex model access, simplifying the model testing process, and improving testing efficiency.

[0101] For ease of understanding, combined Figure 5 As shown, Figure 5 It is a workflow diagram of an end-to-end model module in an embodiment. In this embodiment, the workflow of the end-to-end model module mainly includes an initialization phase and an operation phase.

[0102] The initialization phase is the phase of initializing the end-to-end model, that is, loading the model according to the model file specified by the user, such as initializing the neural network model.

[0103] The operation phase includes updating the main vehicle data input into the model according to the simulation status, including the main vehicle status update and sensor data update.

[0104] Then, according to the model requirements, the sensor data input to the model is updated, including resetting image parameters such as contrast and grayscale, and adjusting the image size.

[0105] Execute model reasoning and output the planned trajectory. The planned trajectory is post-processed and input into the control module. Through the control algorithm operation, the control instruction at the current moment is output. Finally, the control instruction of this reasoning is returned to the simulation engine core, and the dynamic model built into the main vehicle state update unit of the engine core dynamically deduces the physical state of the main vehicle.

[0106] In the above embodiments, a universal end-to-end model access test API is provided, which can use a unified sensor interface and dynamics close to that of a real vehicle to access the model to be tested, thereby simplifying the model access process and improving test efficiency.

[0107] In one of the optional embodiments, the simulation detector module includes at least one simulation detector, and the simulation detector module is used to configure the switch of each simulation detector and / or the detection index corresponding to each simulation detector based on the configuration instruction; each simulation detector is used to obtain the environmental status and the main vehicle status from the engine core in real time when the switch is turned on, and process the environmental status and the main vehicle status based on the configured detection index to obtain the simulation test result.

[0108] The simulation detector module in the present application includes at least one simulation detector, which is used to configure the switches of each simulation detector and / or the detection indicators corresponding to each simulation detector based on the configuration instructions. This supports users to customize the configuration switches and parameter values, solves the problem of limited types of real-time detectors, and improves the multi-dimensional evaluation capability of the driving level of the intelligent driving module in the simulation.

[0109] The simulation detector module is used to obtain the environmental status and the main vehicle status from the engine core in real time when the switch is on, and process the environmental status and the main vehicle status based on the configured detection indicators to obtain the simulation test results. That is, the simulation detector is responsible for performing multi-dimensional evaluation of the simulation system in real time according to the state of the simulation world using various evaluation indicator calculation formulas. The various evaluation indicator calculation formulas are pre-configured, and whether each simulation detector is working can also be pre-configured.

[0110] For ease of understanding, the collision detector is used as an example. The collision detector obtains the position information of all vehicles in the current simulation, and calculates the outer contour boundary according to the physical properties of each vehicle, such as length, width, and height. Then, the relative position relationship between the other vehicle and the main vehicle is determined based on the outer contour boundary. There are mainly five situations. For details, please refer to Figure 6 As shown, Figure 6 (1) is the main vehicle rear-end collision - the main vehicle is responsible, Figure 6 (2) and (4) are the main vehicle's fault after the collision due to lane change. Figure 6 (3) and (5) in the figure are the collisions with other vehicles after the main vehicle has completed its lane change. The collision results and responsibility judgment are given according to the situation, and the simulation collision evaluation index is output.

[0111] In the above embodiment, custom configuration switches and parameter values ​​are supported, the detector configuration is flexibly adjusted, the driving level of the intelligent driving module is evaluated in multiple dimensions, and comprehensive and accurate test results are provided.

[0112] In an exemplary embodiment, Figure 7 As shown, a method for automatic driving simulation test is provided, which is applied to Figure 1The automatic driving simulation test system in is taken as an example to illustrate, which is specifically applied to the engine kernel, including the following steps 702 to 706. Among them:

[0113] S702: Based on the pre-set callback frequencies, the callback scene player plays the pre-generated scene script to obtain the environmental state of the autonomous driving vehicle, the callback image renderer performs image rendering based on the environmental state and the main vehicle state, and obtains sensor data based on the rendered image, and the callback end-to-end model module generates the main vehicle control instructions based on the main vehicle state and sensor data.

[0114] S704: Update the main vehicle status based on the main vehicle control instruction.

[0115] S706: Generate a simulation test result based on the environment state and the host vehicle state through the simulation detector module.

[0116] Among them, the limitations on the above-mentioned autonomous driving simulation test method can be found in the above limitations on the autonomous driving simulation test system, which will not be repeated here.

[0117] In one of the optional embodiments, the method also includes: receiving module registration instructions, the programming languages ​​of the scene player, image renderer, end-to-end model module and simulation detector module corresponding to the module registration instructions are the same or different; generating callback functions corresponding to the scene player, image renderer, end-to-end model and simulation detector based on the module registration instructions, and configuring the callback frequency of the callback function.

[0118] In one optional embodiment, the scenario script includes at least one of a real scenario script and a virtual scenario script; the real scenario script is generated based on scenario data obtained when the vehicle is driving in a real scenario; and the virtual scenario script is obtained through editing.

[0119] In one of the optional embodiments, the callback scene player plays a pre-generated scene script to obtain the environment state, including: the callback scene player loads the scene script to obtain the initialized environment state and the behavior tree structure, the behavior tree structure includes a number of control flow nodes, condition nodes and behavior nodes; the callback scene player traverses the behavior tree structure to determine the corresponding control flow node, and makes a judgment based on each condition node corresponding to the control flow node to obtain the behavior node to be executed, and executes the behavior corresponding to the corresponding behavior node to obtain the environment data.

[0120] In one of the optional embodiments, the conditional node includes at least one of a time trigger node and an event trigger node; based on each conditional node corresponding to the control flow node, a judgment is made to obtain a behavior node to be executed, including at least one of the following: based on the time trigger node corresponding to the control flow node, detecting whether the current time reaches the set time point; when the current time reaches the set time point, obtaining the corresponding behavior node to be executed; and based on the event trigger node corresponding to the control flow node, detecting whether the input meets the event condition, and when the input meets the event condition, obtaining the corresponding behavior node to be executed.

[0121] In one of the optional embodiments, the callback image renderer performs image rendering based on the environment state and the main vehicle state, and obtains sensor data based on the rendered image, including: synchronizing the rendering mode identifier, each environment state and the main vehicle state to the server of the image renderer through a unified API interface, the server is used to perform image rendering on the environment state and the main vehicle state based on the rendering program corresponding to the rendering mode identifier, and obtain sensor data based on the rendered image through the client of the image renderer.

[0122] In one of the optional embodiments, the end-to-end model module is called back to generate a main vehicle control instruction based on the main vehicle state and sensor data, including: calling back the end-to-end model module connected through a unified adaptation framework, the end-to-end model module is used to process the main vehicle state and sensor data to obtain input data that meets the model input requirements, and perform model reasoning on the input data to obtain the main vehicle control instruction; receiving the main vehicle control instruction, and updating the main vehicle state based on the main vehicle control instruction.

[0123] In one of the optional embodiments, before generating simulation test results based on environmental status and main vehicle status through the simulation detector module, it also includes: receiving configuration instructions; configuring the switch of each simulation detector and / or the detection index corresponding to each simulation detector based on the configuration instructions; generating simulation test results based on environmental status and main vehicle status through the simulation detector module, including: when the switch of the simulation detector is turned on, obtaining the environmental status and main vehicle status from the engine core in real time through the simulation detector; processing the environmental status and main vehicle status based on the configured detection index to obtain the simulation test results.

[0124] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as scene scripts. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for simulating and testing an autonomous driving is implemented.

[0126] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0129] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0131] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0132] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0133] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An automatic driving simulation test system, characterized in that: The system comprises: Scenario player, used to play pre-generated scenario scripts; An image renderer, used for rendering an image based on an environment state and a host vehicle state, and obtaining sensor data based on the rendered image; An end-to-end model module, used for generating a main vehicle control instruction based on the main vehicle state and the sensor data; The engine kernel is used to call back the scene player, the image renderer, the end-to-end model module and the simulation detector module based on the preset callback frequencies, and update the environment state based on the scene script played by the scene player, and update the main vehicle state based on the main vehicle control instruction; The simulation detector module is used to generate simulation test results based on the environmental state and the host vehicle state.

2. The system according to claim 1, characterized in that The engine core includes: A module scheduler, the module scheduler is used to receive module registration instructions, generate callback functions corresponding to the scene player, image renderer, end-to-end model module and simulation detector module based on the module registration instructions, configure the callback frequency of the callback function, and call the scene player, the image renderer, the end-to-end model module and the simulation detector module through the callback function; and the programming languages ​​corresponding to the scene player, image renderer, end-to-end model module and simulation detector module are the same or different.

3. The system according to claim 1, characterized in that The scenario script includes at least one of a real scenario script and a virtual scenario script; the real scenario script is generated based on scenario data obtained when a vehicle travels in a real scenario; and the virtual scenario script is obtained through editing.

4. The system according to claim 1, characterized in that The scene player is used to load the scene script to obtain the initial environment state and behavior tree structure, and the behavior tree structure includes a plurality of control flow nodes, condition nodes and behavior nodes; traverse the behavior tree structure to determine the corresponding control flow node, and make judgments based on the condition nodes corresponding to the control flow node to obtain the behavior node to be executed, and execute the behavior corresponding to the corresponding behavior node to obtain the environment data.

5. The system according to claim 4, characterized in that The conditional node includes at least one of a time trigger node and an event trigger node; the time trigger node detects whether the current time reaches a set time point, and when the current time reaches the set time point, a corresponding behavior node to be executed is obtained; the event trigger node detects whether the input satisfies the event condition, and when the input satisfies the event condition, a corresponding behavior node to be executed is obtained.

6. The system according to claim 1, characterized in that The image renderer includes a server and a client; the server includes rendering programs for different rendering modes; the engine kernel synchronizes the rendering mode identifier, various environmental states and the main vehicle state to the server through a unified API interface, and the server is used to perform image rendering of the environmental state and the main vehicle state based on the rendering program corresponding to the rendering mode identifier; the client is used to obtain sensor data based on the rendered image.

7. The system according to claim 1, characterized in that The end-to-end model module is accessed through a unified adaptation framework, and processes the main vehicle status and the sensor data to obtain input data that meets the model input requirements, performs model inference on the input data to obtain main vehicle control instructions, and returns the main vehicle control instructions to the engine kernel.

8. The system according to claim 1, characterized in that The simulation detector module includes at least one simulation detector, and the simulation detector module is used to configure switches of each simulation detector and / or detection indicators corresponding to each simulation detector based on configuration instructions; Each of the simulation detectors is used to obtain the environmental status and the main vehicle status from the engine core in real time when the switch is turned on, and process the environmental status and the main vehicle status based on the configured detection indicators to obtain a simulation test result.

9. An automatic driving simulation test method, characterized in that: The method comprises: Based on the preset callback frequencies, the callback scene player plays the pre-generated scene script to obtain the driving environment state of the autonomous driving vehicle, the callback image renderer performs image rendering based on the environment state and the main vehicle state, obtains sensor data based on the rendered image, and the callback end-to-end model module generates the main vehicle control instruction based on the main vehicle state and the sensor data; Update the main vehicle state based on the main vehicle control instruction; A simulation test result is generated based on the environment state and the host vehicle state by a simulation detector module.

10. The method according to claim 9, characterized in that The method further comprises: Receive a module registration instruction, wherein the programming languages ​​of the scene player, the image renderer, the end-to-end model module, and the simulation detector module corresponding to the module registration instruction are the same or different; The callback functions corresponding to the scene player, image renderer, end-to-end model and simulation detector are generated based on the module registration instruction, and the callback frequency of the callback function is configured.

11. The method according to claim 9, characterized in that The scenario script includes at least one of a real scenario script and a virtual scenario script; the real scenario script is generated based on scenario data obtained when a vehicle travels in a real scenario; and the virtual scenario script is obtained through editing.

12. The method according to claim 9, characterized in that The callback scene player plays the pre-generated scene script to obtain the environment state, including: Calling back the scene player to load the scene script to obtain the initial environment state and behavior tree structure, wherein the behavior tree structure includes a number of control flow nodes, condition nodes, and behavior nodes; The callback scene player traverses the behavior tree structure to determine the corresponding control flow node, and makes a judgment based on each condition node corresponding to the control flow node to obtain the behavior node to be executed, and executes the behavior corresponding to the corresponding behavior node to obtain the environment data.

13. The method according to claim 12, characterized in that The conditional node includes at least one of a time trigger node and an event trigger node; the judgment based on each conditional node corresponding to the control flow node to obtain a behavior node to be executed includes at least one of the following: Based on the time trigger node corresponding to the control flow node, detecting whether the current time reaches the set time point; When the current time reaches the set time point, a corresponding behavior node to be executed is obtained; and Based on the event triggering node corresponding to the control flow node, it is detected whether the input meets the event condition, and when the input meets the event condition, a corresponding behavior node to be executed is obtained.

14. The method according to claim 9, characterized in that The callback image renderer performs image rendering based on the environment state and the main vehicle state, and obtains sensor data based on the rendered image, including: The rendering mode identifier, various environmental states and main vehicle states are synchronized to the server of the image renderer through a unified API interface. The server is used to perform image rendering of the environmental states and main vehicle states based on a rendering program corresponding to the rendering mode identifier, and obtain sensor data based on the rendered image through the client of the image renderer.

15. The method according to claim 9, characterized in that The calling back end-to-end model module to generate a main vehicle control instruction based on the main vehicle state and the sensor data includes: Calling back an end-to-end model module accessed through a unified adaptation framework, wherein the end-to-end model module is used to process the main vehicle state and the sensor data to obtain input data that meets the model input requirements, and perform model reasoning on the input data to obtain a main vehicle control instruction; The main vehicle control instruction is received, and the main vehicle state is updated based on the main vehicle control instruction.

16. The method according to claim 9, characterized in that Before the simulation detector module generates a simulation test result based on the environmental state and the main vehicle state, the method further includes: Receive configuration instructions; Configure switches of each of the simulation detectors and / or detection indicators corresponding to each of the simulation detectors based on the configuration instructions; The generating of the simulation test result based on the environmental state and the host vehicle state by the simulation detector module includes: When the switch of the simulation detector is turned on, the environment state and the host vehicle state are acquired from the engine core in real time through the simulation detector; The environmental state and the main vehicle state are processed based on the configured detection index to obtain a simulation test result.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 9 to 16 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 9 to 16 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 9 to 16 are implemented.

Images