An automatic driving simulation method and device, electronic equipment and storage medium

Abstract

The application provides an automatic driving simulation method and device, electronic equipment and storage medium. The method comprises the following steps: acquiring a measured data frame collected in an actual driving process of a vehicle, the measured data frame comprising a measured sensor data frame; sending the measured sensor data frame to an automatic driving component for simulation to obtain a simulation vehicle control data frame; performing simulation based on the simulation vehicle control data frame through a sensor simulation component to obtain a target sensor data frame; sending the target sensor data frame to the automatic driving component for simulation to obtain a target simulation data frame; and taking the target simulation data frame as an automatic driving simulation result. The application takes the measured data frame in the actual driving process of the real vehicle as a simulation basis, and constructs a complete closed-loop simulation link, which not only maintains the authenticity advantage of the real vehicle data, but also reflects the dynamic interaction process between the automatic driving system and the environment, thereby improving the effectiveness and reliability of the automatic driving simulation.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to an autonomous driving simulation method, device, electronic device, and storage medium. Background Technology

[0002] Existing autonomous driving simulation technologies are mainly divided into two categories. One category is simulation engines based on general simulation frameworks, which can simulate sensor outputs in a 3D simulation scene using a game engine, thereby generating relatively realistic simulated sensor data. The other category is simulation engines based on real vehicle data feedback. These engines collect data from real vehicles and then use specific hardware and data preprocessing methods to synchronously recreate the test scene.

[0003] However, even with relatively realistic simulated sensor data generated by simulation engines based on general simulation frameworks, the data is still not realistic enough to accurately reproduce the real-world environment during autonomous driving. While simulation engines that use real-vehicle data directly employ recorded data, this data is based on specific scenarios and is difficult to generalize to other scenarios, making it impossible to update sensor data in real time to support closed-loop simulation. Summary of the Invention

[0004] In view of the above problems, embodiments of this application propose an autonomous driving simulation method, apparatus, electronic device, and storage medium.

[0005] In a first aspect of this application, an autonomous driving simulation method is provided, applied to an autonomous driving simulation system, the autonomous driving simulation system including autonomous driving components and sensor simulation components, the method comprising:

[0006] Acquire measured data frames collected during actual vehicle operation; the measured data frames include measured sensor data frames;

[0007] The measured sensor data frame is sent to the autonomous driving component for simulation to obtain the simulated vehicle control data frame;

[0008] The target sensor data frame is obtained by performing simulation based on the simulated vehicle control data frame using the sensor simulation component.

[0009] The target sensor data frame is sent to the autonomous driving component for simulation to obtain a target simulation data frame, which is then used as the autonomous driving simulation result.

[0010] Optionally, the step of obtaining the target sensor data frame by simulating the vehicle control data frame using the sensor simulation component includes:

[0011] The measured vehicle control data frame is updated based on the simulated vehicle control data frame to obtain the virtual vehicle control data frame.

[0012] The virtual vehicle control data frame is sent to the sensor simulation component for simulation to obtain the simulated sensor data frame;

[0013] The measured sensor data frame is updated based on the simulated sensor data frame to obtain the target sensor data frame.

[0014] Optionally, the autonomous driving component includes an end-to-end autonomous driving component and / or a visual language autonomous driving component, and the method further includes:

[0015] Data flow information is acquired and written into the measured data frame; the data flow information is used to indicate that at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component is the data source and / or the receiving end of the measured data frame.

[0016] Acquire data composition information and write it into the measured data frame; the data composition information is used to indicate the preceding data frame required for simulating the measured data frame;

[0017] The measured data frames are sorted according to the acquisition time to form a measured data frame sequence.

[0018] Optionally, the autonomous driving simulation system further includes a process scheduling component, wherein sending the measured sensor data frames to the autonomous driving component for simulation to obtain simulated vehicle control data frames includes:

[0019] Obtain the measured sensor data frames from the measured data frame sequence;

[0020] The measured sensor data frame is encapsulated into a first simulation request message;

[0021] Based on the data flow information in the measured sensor data frame, the first simulation request message is sent to the autonomous driving component for simulation through the process scheduling component to obtain the simulated vehicle control data frame.

[0022] Optionally, the method further includes:

[0023] The simulated vehicle control data frame is encapsulated into a first simulation result response message and sent to the process scheduling component.

[0024] Optionally, updating the measured vehicle control data frame based on the simulated vehicle control data frame to obtain the virtual vehicle control data frame includes:

[0025] The process scheduling component receives and parses the first simulation result response message to obtain the simulated vehicle control data frame.

[0026] The data flow information remains unchanged, and the data composition information is updated according to the simulated vehicle control data frame;

[0027] The virtual vehicle control data frame is obtained by replacing the data in the actual vehicle control data frame with the data in the simulated vehicle control data frame.

[0028] Optionally, the autonomous driving simulation system further includes a process scheduling component, wherein sending the virtual vehicle control data frame to the sensor simulation component for simulation to obtain a simulated sensor data frame includes:

[0029] The virtual vehicle control data frame is encapsulated into a second simulation request message;

[0030] Based on the data flow information in the virtual vehicle control data frame, the second simulation request message is sent to the sensor simulation component for simulation through the process scheduling component to obtain the simulation sensor data frame.

[0031] Optionally, the method further includes:

[0032] The simulated sensor data frame is encapsulated into a second simulation result response message and sent to the process scheduling component.

[0033] Optionally, updating the measured sensor data frame based on the simulated sensor data frame to obtain the target sensor data frame includes:

[0034] The process scheduling component receives and parses the second simulation result response message to obtain the simulation sensor data frame;

[0035] The data flow information remains unchanged, and the data composition information is updated according to the simulated sensor data frame;

[0036] The target sensor data frame is obtained by replacing the data in the actual sensor data frame with the data in the simulated sensor data frame.

[0037] Optionally, the autonomous driving component or the sensor simulation component includes a data buffer pool, a data injection tool, and functional applications corresponding to the autonomous driving component or the sensor simulation component, and the simulation is performed using the following steps:

[0038] The data frame to be simulated is cached in the data buffer pool;

[0039] The preceding data frame required to simulate the functional application is determined based on the data composition information in the data frame to be simulated.

[0040] If the data buffer pool contains the preceding data frame, the data frame to be simulated is processed by the data injection tool to obtain the data frame to be calculated.

[0041] The autonomous driving component or the sensor simulation component is controlled to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining the simulation data frame.

[0042] Optionally, the method further includes:

[0043] The target cache component is determined based on the data flow information in the data frame to be simulated; the target cache component includes at least one of an end-to-end autonomous driving component, a visual language autonomous driving component, and a sensor simulation component; the target cache component is used to cache the data frame to be simulated;

[0044] The data frame to be simulated is cached in the data buffer pool, and a caching success message is generated;

[0045] The cache success information is encapsulated into a cache response message and sent to the process scheduling component.

[0046] Optionally, the method further includes:

[0047] After the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, and the cached response message, the process scheduling component sends the first simulation request message to the autonomous driving component for simulation.

[0048] Alternatively, after the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, or the cached response message, the process scheduling component sends a second simulation request message to the sensor simulation component for simulation.

[0049] Optionally, the autonomous driving component or the sensor simulation component includes a message middleware, or the functional application includes a callback interface. The control of the autonomous driving component or the sensor simulation component drives the functional application to perform simulation based on the data frame to be calculated, to obtain the simulation data frame, including:

[0050] The autonomous driving component or the sensor simulation component calls the callback interface to drive the functional application to perform simulation based on the data frame to be calculated, and obtain the simulation data frame.

[0051] Alternatively, the autonomous driving component or the sensor simulation component may invoke the message middleware to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining the simulation data frame.

[0052] In a second aspect of this application, an autonomous driving simulation device is also provided, applied to an autonomous driving simulation system, the autonomous driving simulation system including autonomous driving components and sensor simulation components, the device comprising:

[0053] The data acquisition module is used to acquire measured data frames collected during the actual driving process of the vehicle; the measured data frames include measured sensor data frames.

[0054] The first simulation module is used to send the measured sensor data frame to the autonomous driving component for simulation to obtain the simulated vehicle control data frame.

[0055] The second simulation module is used to perform simulation based on the simulated vehicle control data frame through the sensor simulation component to obtain the target sensor data frame.

[0056] The third simulation module is used to send the target sensor data frame to the autonomous driving component for simulation, obtain the target simulation data frame, and use the target simulation data frame as the autonomous driving simulation result.

[0057] In a third aspect of this application, an electronic device is also provided, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method described above.

[0058] In a fourth aspect of this application, a storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the method described above.

[0059] The embodiments of this application have the following advantages:

[0060] In this embodiment, measured data frames collected during actual vehicle operation are acquired. These measured data frames include measured sensor data frames. This application uses measured data frames from real vehicle operation as the simulation basis, ensuring the authenticity and reliability of the data source. The measured sensor data frames are sent to the autonomous driving component for simulation, resulting in simulated vehicle control data frames. This application directly drives the autonomous driving component for simulation using measured sensor data frames, effectively preserving the physical characteristics of real sensor data and avoiding data distortion problems caused by virtual rendering in traditional simulation engines. Through the sensor simulation component, simulation is performed based on the simulated vehicle control data frames to obtain target sensor data frames. This application feeds back the simulated vehicle control data frames to the sensor simulation component for secondary simulation, achieving dynamic updates of sensor data and overcoming the limitation of traditional backfeedback schemes that can only statically reproduce recorded data. The target sensor data frames are sent to the autonomous driving component for simulation, resulting in target simulation data frames, which are used as the autonomous driving simulation result. This application constructs a complete closed-loop simulation link, which maintains the advantage of real vehicle data authenticity and gains the ability to dynamically adjust the simulation environment. The final generated target simulation data frame contains the physical characteristics of the real scene and reflects the dynamic interaction process between the autonomous driving system and the environment. It provides an autonomous driving system with a simulation test environment that combines high fidelity and strong generalization ability, significantly improving the effectiveness and reliability of autonomous driving algorithm verification. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0062] Figure 1 This is a flowchart illustrating the steps of an autonomous driving simulation method according to an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of the architecture of an autonomous driving simulation system provided in one embodiment of this application;

[0064] Figure 3 This is a schematic diagram of a data transfer process provided in an embodiment of this application;

[0065] Figure 4 This is a schematic diagram of the structure of a data frame provided in an embodiment of this application;

[0066] Figure 5 This is a timing diagram of a data frame provided in an embodiment of this application;

[0067] Figure 6 This is a logical schematic diagram of a process scheduling component provided in an embodiment of this application.

[0068] Figure 7 This is a logical schematic diagram of a simulation component provided in one embodiment of this application.

[0069] Figure 8 This is a schematic diagram of the structure of an autonomous driving simulation device provided in an embodiment of this application. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of this application to enable readers to better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and updates based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0071] Existing autonomous driving simulation technologies mainly fall into two categories. One category consists of simulation engines based on general simulation frameworks. These engines can simulate sensor outputs in 3D simulation scenes using game engines, thereby generating relatively realistic simulated sensor data. These simulation engines utilize general simulation frameworks such as Carla (Car Learning to Act, a high-precision open-source simulator for autonomous driving) and AirSim (a cross-platform simulation framework) to implement autonomous driving simulations. They typically use game engines like Unity (a cross-platform game engine) and Unreal Engine to construct 3D simulation scenes and simulate sensor outputs, producing highly realistic simulation data. However, for data-driven dual-system autonomous driving systems (fast and slow), this level of realism is still insufficient.

[0072] Another type is simulation engines that use real-vehicle data feedback. These engines collect data from real vehicles and then use specific hardware and data preprocessing techniques to synchronously recreate the test scenario. Because they directly use data recorded from real vehicles, the data fed back into the autonomous driving software does not differ from the real environment. However, the real-vehicle data recorded by this type of simulation engine is based on specific scenarios and is difficult to generalize to other scenarios. It is also difficult to achieve closed-loop simulation and cannot generate new sensor data in real time based on the location updates of the simulated vehicle.

[0073] It should be noted that the fast-slow dual-system autonomous driving system can refer to an autonomous driving system that combines end-to-end processing and Vision-Language Model (VLM). The fast system refers to the end-to-end autonomous driving system, which includes modules with high real-time performance and low latency (such as sensor data processing and vehicle control), typically requiring a response time of 10-100 milliseconds to directly ensure driving safety. The slow system refers to the vision-language autonomous driving system, which includes modules requiring complex computation and semantic understanding (such as VLM-driven scene reasoning and long-term path planning), allowing for processing time of 200 milliseconds to several seconds, focusing on higher-order decision-making. In the embodiments of this application, the autonomous driving simulation system can be an autonomous driving system that combines end-to-end processing and vision-language model. The fast system handles immediate driving actions (such as emergency braking), while the slow system provides semantic-level guidance (such as "lane change suggestion due to congestion ahead"), and the two complement each other through asynchronous communication.

[0074] In this embodiment, the autonomous driving simulation method can be applied to an autonomous driving simulation system, which includes autonomous driving components and sensor simulation components.

[0075] It should be noted that the autonomous driving component can be a collection of algorithms used in a simulation system to simulate real autonomous driving decision-making and control functions. Its core function is to generate vehicle control commands by processing input sensor data. It typically consists of three parts: a perception module, a decision-making and planning module, and a control module. The perception module is responsible for parsing raw sensor data such as camera images and LiDAR point clouds and outputting structured environmental information. The decision-making and planning module formulates driving strategies based on the perception results and map data. The control module translates the decisions into specific steering, throttle, and braking commands. In the embodiments of this application, the autonomous driving component is used to generate vehicle control data based on sensor data.

[0076] The sensor simulation component can be a subsystem specifically designed to generate virtual sensor data within a simulation environment. It replaces actual hardware devices such as cameras and radars by accurately simulating the physical characteristics and working mechanisms of real sensors. Specific implementation methods include using game engine rendering technology to generate realistic camera images, simulating lidar point cloud characteristics through ray collision calculations using a physics engine, or reproducing radar echo characteristics using a data-driven generative model. This component can dynamically adjust its output data based on changes in the simulated vehicle's real-time state parameters, such as position, speed, and environmental conditions. Its simulation accuracy and fidelity directly determine the domain difference between virtual sensor data and real-world acquired data, having a decisive impact on the training and testing of data-driven autonomous driving algorithms and being a key step in ensuring the reliability of simulation results. In this embodiment, the sensor simulation component is used to generate sensor data based on vehicle control data.

[0077] Example 1

[0078] This application provides an autonomous driving simulation method. (Refer to...) Figure 1 The diagram shows a flowchart of the steps of an autonomous driving simulation method provided in an embodiment of this application.

[0079] The method may specifically include the following steps:

[0080] S101, acquire the measured data frame collected during the actual driving process of the vehicle; the measured data frame includes the measured sensor data frame;

[0081] S102, the measured sensor data frame is sent to the autonomous driving component for simulation to obtain the simulated vehicle control data frame;

[0082] S103, using the sensor simulation component, simulation is performed based on the simulated vehicle control data frame to obtain the target sensor data frame;

[0083] S104, the target sensor data frame is sent to the autonomous driving component for simulation to obtain a target simulation data frame, and the target simulation data frame is used as the autonomous driving simulation result.

[0084] In this embodiment of the application, measured data frames collected during the actual driving process of the vehicle can be obtained, and the measured data frames may include measured sensor data frames.

[0085] The measured data frames can be obtained by segmenting data recorded during actual vehicle operation. The measured sensor data can be sensor data collected during actual vehicle operation.

[0086] In practical implementation, sensor data refers to the collection of raw information reflecting the vehicle's surrounding environment and its own operational status, gathered by various onboard sensors in an autonomous driving system. This data can be categorized into several types based on its physical characteristics and functions:

[0087] Visual sensor data (such as RGB images, infrared images, and video streams captured by cameras) is characterized by containing rich texture and color information but is significantly affected by lighting conditions.

[0088] LiDAR point cloud data constructs a three-dimensional environment model by emitting laser beams and receiving reflected signals. It has centimeter-level ranging accuracy but is easily affected by rain and fog.

[0089] Millimeter-wave radar data detects the distance, speed, and azimuth of target objects using radio waves, and has all-weather operation capability, but has low spatial resolution;

[0090] Ultrasonic sensor data is mainly used for short-range obstacle detection and plays an important role in low-speed scenarios such as automatic parking;

[0091] Positioning sensor data (such as GNSS and IMU) provides the vehicle's absolute position and attitude information, which is the basis for path planning and navigation;

[0092] Vehicle bus data (such as CAN signals) includes vehicle dynamic parameters such as vehicle speed, steering angle, and throttle opening.

[0093] These multimodal sensor data are fused and processed based on time synchronization and spatial calibration, collectively forming the input source for the autonomous driving system to perceive the environment. The quality and reliability of this data directly affect the accuracy of subsequent decision-making and control.

[0094] In this embodiment, the measured sensor data frames can be sent to the autonomous driving component for simulation to obtain simulated vehicle control data frames.

[0095] The simulated vehicle control data frame can be obtained by the autonomous driving component based on the measured sensor data frame. The simulated vehicle control data can be vehicle control data obtained during simulated vehicle operation.

[0096] In practical implementation, vehicle control data can be a set of underlying execution instructions output by the autonomous driving system, specifically including core parameters such as steering system control quantities (e.g., steering wheel angle, steering motor torque), longitudinal dynamics control quantities (e.g., throttle opening, braking pressure, target acceleration), and gear control signals.

[0097] Vehicle control data has three typical characteristics: first, high real-time performance, typically updated at a frequency of 10-100Hz to ensure the timeliness of vehicle dynamic control; second, strong constraints, all output commands must conform to the vehicle dynamics envelope (such as maximum steering angular velocity limits and smooth acceleration transitions); and third, multi-level verification mechanisms, requiring multiple safety checks such as actuator health status monitoring and control quantity rationality verification before output. In simulation systems, vehicle control data serves both as the test object (verifying the performance of autonomous driving components) and as a key input for closed-loop simulation (driving sensor simulation components to update environmental states). Its numerical accuracy and time synchronization accuracy directly affect the reliability of simulation results. Autonomous driving components can adopt a hierarchical control architecture. The upper-level decision module outputs path tracking targets (such as target curvature and velocity curves), and the lower-level controller converts these into specific actuator commands, which are then transmitted via a bus (such as CAN / FlexRay) to steering, braking, and drive units, forming a complete control closed loop.

[0098] In this embodiment of the application, the target sensor data frame can be obtained by performing simulation based on the simulated vehicle control data frame using the sensor simulation component.

[0099] The target sensor data frame can be obtained by the sensor simulation component based on the simulated vehicle control data frame. The target sensor data can be sensor data obtained during the simulated vehicle driving process in an environment rendered based on the vehicle control data obtained during the simulated vehicle driving process.

[0100] In this embodiment, the target sensor data frame can be sent to the autonomous driving component for simulation to obtain the target simulation data frame, and the target simulation data frame can be used as the autonomous driving simulation result.

[0101] The target simulation data frame can be obtained by the autonomous driving component through simulation based on the target sensor data frame. The target simulation data frame can be used as the result of the autonomous driving simulation.

[0102] In this embodiment, measured data frames collected during actual vehicle operation are acquired. These measured data frames include measured sensor data frames. This application uses measured data frames from real vehicle operation as the simulation basis, ensuring the authenticity and reliability of the data source. The measured sensor data frames are sent to the autonomous driving component for simulation, resulting in simulated vehicle control data frames. This application directly drives the autonomous driving component for simulation using measured sensor data frames, effectively preserving the physical characteristics of real sensor data and avoiding data distortion problems caused by virtual rendering in traditional simulation engines. Through the sensor simulation component, simulation is performed based on the simulated vehicle control data frames to obtain target sensor data frames. This application feeds back the simulated vehicle control data frames to the sensor simulation component for secondary simulation, achieving dynamic updates of sensor data and overcoming the limitation of traditional backfeedback schemes that can only statically reproduce recorded data. The target sensor data frames are sent to the autonomous driving component for simulation, resulting in target simulation data frames, which are used as the autonomous driving simulation result. This application constructs a complete closed-loop simulation link, which maintains the advantage of real vehicle data authenticity and gains the ability to dynamically adjust the simulation environment. The final generated target simulation data frame contains the physical characteristics of the real scene and reflects the dynamic interaction process between the autonomous driving system and the environment. It provides an autonomous driving system with a simulation test environment that combines high fidelity and strong generalization ability, significantly improving the effectiveness and reliability of autonomous driving algorithm verification.

[0103] Example 2

[0104] This application provides an autonomous driving simulation method.

[0105] The method may specifically include the following steps:

[0106] S201, acquire measured data frames collected during actual vehicle operation; the measured data frames include measured sensor data frames and measured vehicle control data frames;

[0107] S202, the measured sensor data frame is sent to the autonomous driving component for simulation to obtain the simulated vehicle control data frame;

[0108] S203, update the measured vehicle control data frame according to the simulated vehicle control data frame to obtain the virtual vehicle control data frame;

[0109] S204, the virtual vehicle control data frame is sent to the sensor simulation component for simulation to obtain the simulated sensor data frame;

[0110] S205, Update the measured sensor data frame according to the simulated sensor data frame to obtain the target sensor data frame;

[0111] S206, the target sensor data frame is sent to the autonomous driving component for simulation to obtain a target simulation data frame, and the target simulation data frame is used as the autonomous driving simulation result.

[0112] In the embodiments of this application, steps S201, S202 and S206 can be implemented with reference to steps S101, S102 and S104 described above, and will not be repeated here.

[0113] In this embodiment, a virtual vehicle control data frame can be obtained by updating the actual vehicle control data frame based on the simulated vehicle control data frame. The virtual vehicle control data frame is the actual vehicle control data frame after data replacement. The virtual vehicle control data frame retains information such as the timestamp of the actual vehicle control data frame, but the data within it is the data from the simulated vehicle control data frame.

[0114] In this embodiment of the application, virtual vehicle control data frames can be sent to a sensor simulation component for simulation to obtain simulated sensor data frames.

[0115] The simulated sensor data frames can be obtained by the sensor simulation component based on the virtual vehicle control data frames. These simulated sensor data frames can be sensor data from the simulated vehicle driving process, rendered in an environment based on vehicle control data obtained during simulated vehicle driving.

[0116] In this embodiment, the measured sensor data frame can be updated based on the simulated sensor data frame to obtain the target sensor data frame. The target sensor data frame is the measured sensor data frame after data replacement; it retains information such as the timestamp of the measured sensor data frame, but the data within it is the data from the simulated sensor data frame.

[0117] In this embodiment, the target simulation data frame can also be used as the input of the sensor simulation component to continue iterating the simulation until the timestamp of the currently processed measured data frame is the last timestamp of all measured data frames, at which point the simulation is complete.

[0118] This application uses measured sensor data to drive autonomous driving components to generate control commands, and then uses sensor simulation components to convert the control commands into new sensor data, forming a complete closed loop of "perception-decision-environment update". This method retains key information such as the timestamp of the original measured data to ensure temporal consistency, and realizes dynamic generation of sensor data through an iterative replacement mechanism, effectively solving the pain point of traditional backfeed simulation that cannot update environmental data in real time. At the same time, by retaining the physical characteristic framework of the measured data and only replacing the internal data values, it provides scenario generalization ability while ensuring the realism of simulation results, and can support diverse testing needs including extreme conditions. In addition, the iterative design of this method allows for flexible control of simulation depth, which can verify basic functions in a single closed loop or test long-term decision effects through multiple iterations. It provides a verification platform for fast and slow dual-system autonomous driving algorithms that combines realism and scalability, significantly improving the efficiency and reliability of simulation testing.

[0119] In existing technologies, simulation engines based on general simulation frameworks typically only offer strong real-time computation, corresponding to the fast system in a fast-slow dual-system system. Simulation frameworks commonly used in traditional autonomous driving technologies are also optimized only for real-time computation. They cannot effectively integrate the slow system to simulate a fast-slow dual-system autonomous driving system. Furthermore, the aforementioned simulation engines based on real vehicle data feedback cannot generate new sensor data in real time based on the simulated vehicle's position updates; this deficiency will prevent them from supporting the simulation training of fast-slow dual-system autonomous driving systems.

[0120] Example 3

[0121] This application provides an autonomous driving simulation method. The autonomous driving components include an end-to-end autonomous driving component and / or a visual-language autonomous driving component. The autonomous driving simulation system also includes a process scheduling component.

[0122] It should be noted that the end-to-end autonomous driving component can refer to a system, that is, an end-to-end autonomous driving system. In the embodiments of this application, the end-to-end autonomous driving component can be a software container that includes an end-to-end autonomous driving system.

[0123] The visual language autonomous driving component can refer to a slow system, that is, a visual language autonomous driving system. In the embodiments of this application, the visual language autonomous driving component can be a software container that includes a visual language autonomous driving system.

[0124] The process scheduling component can be a component responsible for process scheduling, which can be used to serialize the simulation process and coordinate the operating rhythm of end-to-end autonomous driving components, visual language autonomous driving components, etc., with different operating frequencies. In the embodiments of this application, the process scheduling component can be a flow control scheduler.

[0125] The method may specifically include the following steps:

[0126] S301, acquire measured data frames collected during actual vehicle operation; the measured data frames include measured sensor data frames and measured vehicle control data frames;

[0127] S302, acquire data flow information and write it into the measured data frame; the data flow information is used to indicate that at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component is the data source and / or the receiving end of the measured data frame;

[0128] S303, acquire data composition information and write it into the measured data frame; the data composition information is used to indicate the preceding data frame required for simulating the measured data frame;

[0129] S304, The measured data frames are sorted according to the acquisition time order to form a measured data frame sequence.

[0130] S305, the measured sensor data frame is sent to the autonomous driving component for simulation to obtain the simulated vehicle control data frame;

[0131] S306, Update the measured vehicle control data frame according to the simulated vehicle control data frame to obtain the virtual vehicle control data frame;

[0132] S307, The virtual vehicle control data frame is sent to the sensor simulation component for simulation to obtain the simulated sensor data frame;

[0133] S308, Update the measured sensor data frame according to the simulated sensor data frame to obtain the target sensor data frame;

[0134] S309, the target sensor data frame is sent to the autonomous driving component for simulation to obtain a target simulation data frame, and the target simulation data frame is used as the autonomous driving simulation result.

[0135] In the embodiments of this application, steps S301 and S305-S209 can be implemented with reference to steps S201-S206 described above, and will not be repeated here.

[0136] In this embodiment of the application, data flow information can be obtained and written into the measured data frame; the data flow information is used to indicate that at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component is the data source and / or the receiving end of the measured data frame;

[0137] Among them, at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component is the receiving end of the measured data frame, indicating that the measured data frame needs to be sent to at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component.

[0138] Alternatively, at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component is the data source of the measured data frame, indicating that the measured data frame is generated by any one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component.

[0139] In a practical implementation, data flow information can be obtained by loading and running configurations, and mapping relationships between subscription / publishing topics and end-to-end autonomous driving components, visual language autonomous driving components, and sensor simulation components.

[0140] In this embodiment, data composition information can be obtained and written into the measured data frame; the data composition information is used to indicate the preceding data frame required for simulating the measured data frame. The preceding data frame can be one or more.

[0141] In practical implementation, the data composition information can be process control information.

[0142] In this embodiment of the application, the measured data frames can be sorted according to the acquisition time order to form a measured data frame sequence.

[0143] In practical implementation, the measured data frame sequence can be represented in the form of a message queue.

[0144] The autonomous driving simulation method provided in this application achieves high-fidelity closed-loop simulation through an innovative dynamic data management mechanism. By encapsulating data flow information and data composition information in a structured manner, each data frame carries its own transmission path and processing dependencies, preserving the real data base of traditional feedback schemes while enabling dynamic updates of simulation data. Through the collaborative scheduling of end-to-end autonomous driving components, visual language autonomous driving components, and sensor simulation components, a complete closed loop of "perception-decision-environment update" is constructed. Data flow information and data composition information can be used to coordinate the asynchronous operation of fast and slow systems. A timestamp-sorted message queue mechanism ensures data temporal consistency, while automatic verification and synchronization of preceding data frames are achieved through process control information, significantly improving the simulation stability in complex scenarios.

[0145] In one optional embodiment of this application, step S305 further includes the following steps:

[0146] Obtain the measured sensor data frames from the measured data frame sequence;

[0147] The measured sensor data frame is encapsulated into a first simulation request message;

[0148] Based on the data flow information in the measured sensor data frame, the first simulation request message is sent to the autonomous driving component for simulation through the process scheduling component to obtain the simulated vehicle control data frame.

[0149] In one optional embodiment of this application, the method further includes:

[0150] The simulated vehicle control data frame is encapsulated into a first simulation result response message and sent to the process scheduling component.

[0151] In this embodiment, measured sensor data frames from a measured data frame sequence can be acquired and encapsulated into a first simulation request message. Based on the data flow information within the measured sensor data frames, the data source of the measured sensor data frames can be determined to be either an end-to-end autonomous driving component or a visual-language autonomous driving component. This allows for the determination that the first simulation request message should be sent to either the end-to-end autonomous driving component or the visual-language autonomous driving component. Then, the process scheduling component sends the first simulation request message to the determined end-to-end autonomous driving component or visual-language autonomous driving component for simulation, obtaining simulated vehicle control data frames. These simulated vehicle control data frames are then encapsulated into a first simulation result response message and sent to the process scheduling component. The first simulation request message encapsulates sensor data frames. The first simulation request message is used to request simulation from the autonomous driving component. The first simulation result response message is used to encapsulate the simulation results generated by the autonomous driving component.

[0152] In practical implementation, the process scheduling component can include a scheduling object. The scheduling object encapsulates data frames into messages; specifically, it encapsulates measured sensor data frames into a first simulation request message. Furthermore, both message sending and receiving can be implemented using the scheduling object.

[0153] In one optional embodiment of this application, step S306 further includes the following steps:

[0154] The process scheduling component receives and parses the first simulation result response message to obtain the simulated vehicle control data frame.

[0155] The data flow information remains unchanged, and the data composition information is updated according to the simulated vehicle control data frame;

[0156] The virtual vehicle control data frame is obtained by replacing the data in the actual vehicle control data frame with the data in the simulated vehicle control data frame.

[0157] In this embodiment of the application, the first simulation result response message can be received and parsed by the process scheduling component to obtain the simulated vehicle control data frame. Then, the data flow information can be kept unchanged, the data composition information can be updated according to the simulated vehicle control data frame, and the data in the actual vehicle control data frame can be replaced with the data in the simulated vehicle control data frame to obtain the virtual vehicle control data frame.

[0158] In one optional embodiment of this application, step S307 further includes the following steps:

[0159] The virtual vehicle control data frame is encapsulated into a second simulation request message;

[0160] Based on the data flow information in the virtual vehicle control data frame, the second simulation request message is sent to the sensor simulation component for simulation through the process scheduling component to obtain the simulation sensor data frame.

[0161] In one optional embodiment of this application, the method further includes:

[0162] The simulated sensor data frame is encapsulated into a second simulation result response message and sent to the process scheduling component.

[0163] In this embodiment, the virtual vehicle control data frame can be encapsulated into a second simulation request message. Based on the data flow information in the virtual vehicle control data frame, it can be determined that the data source of the virtual vehicle control data frame is a sensor simulation component. The second simulation request message should then be sent to the sensor simulation component. The process scheduling component then sends the second simulation request message to the sensor simulation component for simulation, obtaining a simulated sensor data frame. This simulated sensor data frame is then encapsulated into a second simulation result response message and sent to the process scheduling component. The second simulation request message encapsulates the vehicle control data frame. The second simulation request message is used to request simulation from the sensor simulation component. The second simulation result response message is used to encapsulate the simulation results generated by the sensor simulation component.

[0164] In practical implementation, the process scheduling component can include a scheduling object. The scheduling object encapsulates data frames into messages; specifically, it encapsulates virtual vehicle control data frames into second simulation request messages. Furthermore, both message sending and receiving can be implemented using the scheduling object.

[0165] In one optional embodiment of this application, step S308 further includes the following steps:

[0166] The process scheduling component receives and parses the second simulation result response message to obtain the simulation sensor data frame;

[0167] The data flow information remains unchanged, and the data composition information is updated according to the simulated sensor data frame;

[0168] The target sensor data frame is obtained by replacing the data in the actual sensor data frame with the data in the simulated sensor data frame.

[0169] In this embodiment of the application, the second simulation result response message can be received and parsed by the process scheduling component to obtain the simulation sensor data frame. Then, the data flow information can be kept unchanged, the data composition information can be updated according to the simulation sensor data frame, and the data in the actual sensor data frame can be replaced with the data in the simulation sensor data frame to obtain the target sensor data frame.

[0170] In one optional embodiment of this application, the autonomous driving component or sensor simulation component includes a data buffer pool, a data injection tool, and a functional application corresponding to the autonomous driving component or sensor simulation component, and the simulation can be performed using the following steps:

[0171] The data frame to be simulated is cached in the data buffer pool;

[0172] The preceding data frame required to simulate the functional application is determined based on the data composition information in the data frame to be simulated.

[0173] If the data buffer pool contains the preceding data frame, the data frame to be simulated is processed by the data injection tool to obtain the data frame to be calculated.

[0174] The autonomous driving component or the sensor simulation component is controlled to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining the simulation data frame; the simulation data frame includes the simulated vehicle control data frame or the simulated sensor data frame.

[0175] In this embodiment, the data frame to be simulated can be cached in a data buffer pool. The data frame to be simulated may include the measured sensor data frame in the first simulation request message or the virtual vehicle control data frame in the second simulation request message. The data buffer pool can be used to temporarily store the data frame to be simulated, such as the sensor data frame or the vehicle control data frame.

[0176] In this embodiment, the preceding data frame required for simulating the functional application can be determined based on the data composition information in the data frame to be simulated. If the data buffer pool contains the preceding data frame, the data frame to be simulated is processed by a data injection tool to obtain the data frame to be calculated. The preceding data frame can be determined based on the data composition information in the data frame to be simulated, and there can be one or more preceding data frames. The data injection tool can be used to dynamically modify the data content to make the data frame to be simulated meet the simulation requirements of autonomous driving components or sensor simulation components.

[0177] In this embodiment, the autonomous driving component or sensor simulation component can be controlled to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining a simulation data frame. The simulation data frame may include a simulated vehicle control data frame or a simulated sensor data frame. The functional application refers to an executable module within the autonomous driving component or sensor simulation component used to implement the core algorithm. In this embodiment, it is manifested as an end-to-end autonomous driving application or a visual language autonomous driving application in the autonomous driving component, and as a graphics-physics engine in the sensor simulation component.

[0178] This application ensures the temporal consistency and format compliance of simulation data through the collaborative operation of a data buffer pool and a data injection tool. It guarantees the fulfillment of simulation conditions by dynamically verifying the completeness of preceding data frames and coordinates the operation of autonomous driving software and sensor simulation components with different operating frequencies through a master-slave scheduling approach. This not only enables the updating of sensor data but also achieves closed-loop simulation of autonomous driving software with a fast and slow dual-system architecture.

[0179] In one optional embodiment of this application, the method further includes:

[0180] The target cache component is determined based on the data flow information in the data frame to be simulated; the target cache component includes at least one of an end-to-end autonomous driving component, a visual language autonomous driving component, and a sensor simulation component; the target cache component is used to cache the data frame to be simulated;

[0181] The data frame to be simulated is cached in the data buffer pool, and a caching success message is generated;

[0182] The cache success information is encapsulated into a cache response message and sent to the process scheduling component.

[0183] In this embodiment of the application, the target cache component can be determined according to the data flow information in the data frame to be simulated. That is, the receiving end of the data frame to be simulated indicated by the data flow information is used as the target cache component. The target cache component includes at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component. The target cache component is used to cache the data frame to be simulated.

[0184] Then, the data frame to be simulated can be cached to the data buffer pool, and a caching success message can be generated. The caching success message is then encapsulated into a caching response message and sent to the process scheduling component.

[0185] Since the data frame to be simulated includes the measured sensor data frame in the first simulation request message or the virtual vehicle control data frame in the second simulation request message, during the data caching process, the first simulation request message is also used to request data caching from the autonomous driving component, and the second simulation request message is also used to request data caching from the sensor simulation component.

[0186] In one optional embodiment of this application, the method further includes:

[0187] After the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, and the cached response message, the process scheduling component sends the first simulation request message to the autonomous driving component for simulation.

[0188] Alternatively, after the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, or the cached response message, the process scheduling component sends a second simulation request message to the sensor simulation component for simulation.

[0189] In this embodiment of the application, after the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, and the cached response message, the process scheduling component may send the first simulation request message to the autonomous driving component for simulation; or, after the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, and the cached response message, the process scheduling component may send the second simulation request message to the sensor simulation component for simulation.

[0190] In the specific implementation, apart from the process scheduling component sending the first simulation request message or the second simulation request message for the first time in the entire simulation process, during the data flow process, the process scheduling component will only send the first simulation request message or the second simulation request message after receiving any one of the first simulation result response message, the second simulation result response message, or the cached response message.

[0191] In one optional embodiment of this application, the autonomous driving component or sensor simulation component includes a message middleware, or the functional application includes a callback interface. The step of controlling the autonomous driving component or sensor simulation component to drive the functional application to perform simulation based on the data frame to be calculated, and obtaining the simulation data frame, further includes the following steps:

[0192] The autonomous driving component or the sensor simulation component calls the callback interface to drive the functional application to perform simulation based on the data frame to be calculated, and obtain the simulation data frame.

[0193] Alternatively, the autonomous driving component or the sensor simulation component may invoke the message middleware to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining the simulation data frame.

[0194] In this embodiment, the autonomous driving component or sensor simulation component calls a callback interface to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining a simulation data frame. The callback interface is a standardized service endpoint exposed by the functional application.

[0195] Alternatively, in this embodiment, if the functional application does not provide a standardized service endpoint to the outside world, a message middleware can be set in the autonomous driving component or the sensor simulation component. The autonomous driving component or the sensor simulation component calls the message middleware to drive the functional application to perform simulation based on the data frame to be calculated, and obtain the simulation data frame.

[0196] Example 4

[0197] This application provides an autonomous driving simulation method. The autonomous driving component includes an end-to-end autonomous driving component and / or a visual language autonomous driving component. The autonomous driving simulation system also includes a process scheduling component. The autonomous driving component or sensor simulation component includes a data buffer pool, a data injection tool, and functional applications corresponding to the autonomous driving component or sensor simulation component. The autonomous driving component or sensor simulation component includes a message middleware, and the functional applications include callback interfaces.

[0198] The method may specifically include the following steps:

[0199] S401, acquire measured data frames collected during actual vehicle operation; the measured data frames include measured sensor data frames and measured vehicle control data frames;

[0200] S402, acquire data flow information and write it into the measured data frame; the data flow information is used to indicate that at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component is the data source and / or the receiving end of the measured data frame;

[0201] S403, acquire data composition information and write it into the measured data frame; the data composition information is used to indicate the preceding data frame required for simulating the measured data frame;

[0202] S404, Sort the measured data frames according to the acquisition time order to form a measured data frame sequence;

[0203] S405, acquire the measured sensor data frames in the measured data frame sequence;

[0204] S406, the measured sensor data frame is encapsulated into a first simulation request message;

[0205] S407, based on the data flow information in the measured sensor data frame, the first simulation request message is sent to the autonomous driving component for simulation through the process scheduling component to obtain the simulated vehicle control data frame;

[0206] S408, the simulated vehicle control data frame is encapsulated into a first simulation result response message and sent to the process scheduling component;

[0207] S409, the process scheduling component receives and parses the first simulation result response message to obtain the simulation vehicle control data frame;

[0208] S410, the data flow information remains unchanged, and the data composition information is updated according to the simulated vehicle control data frame;

[0209] S411, replace the data in the actual vehicle control data frame with the data in the simulated vehicle control data frame to obtain the virtual vehicle control data frame;

[0210] S412, the virtual vehicle control data frame is encapsulated into a second simulation request message;

[0211] S413, based on the data flow information in the virtual vehicle control data frame, the second simulation request message is sent to the sensor simulation component for simulation through the process scheduling component to obtain the simulation sensor data frame;

[0212] S414, the simulated sensor data frame is encapsulated into a second simulation result response message and sent to the process scheduling component;

[0213] S415, the process scheduling component receives and parses the second simulation result response message to obtain the simulation sensor data frame;

[0214] S416, the data flow information remains unchanged, and the data composition information is updated according to the simulated sensor data frame;

[0215] S417, Replace the data in the measured sensor data frame with the data in the simulated sensor data frame to obtain the target sensor data frame;

[0216] S418, cache the data frame to be simulated to the data buffer pool; the data frame to be simulated includes the measured sensor data frame in the first simulation request message or the virtual vehicle control data frame in the second simulation request message;

[0217] S419, determine the preceding data frame required to simulate the functional application based on the data composition information in the data frame to be simulated;

[0218] S420, if the data buffer pool contains the preceding data frame, then the data frame to be simulated is processed by the data injection tool to obtain the data frame to be calculated;

[0219] S421, the autonomous driving component or the sensor simulation component calls the callback interface to drive the functional application to perform simulation based on the data frame to be calculated, and obtain the simulation data frame; the simulation data frame includes the simulated vehicle control data frame or the simulated sensor data frame.

[0220] Alternatively, in step S422, the autonomous driving component or the sensor simulation component invokes the message middleware to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining the simulation data frame. The simulation data frame includes the simulated vehicle control data frame or the simulated sensor data frame.

[0221] S423, determine the target cache component based on the data flow information in the data frame to be simulated; the target cache component includes at least one of an end-to-end autonomous driving component, a visual language autonomous driving component, and a sensor simulation component; the target cache component is used to cache the data frame to be simulated;

[0222] S424, cache the data frame to be simulated to the data buffer pool and generate a caching success message;

[0223] S425, the cache success information is encapsulated into a cache response message and sent to the process scheduling component;

[0224] S426, after the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, and the cache response message, the process scheduling component sends the first simulation request message to the autonomous driving component for simulation;

[0225] Alternatively, S427, after the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, and the cache response message, the process scheduling component sends the second simulation request message to the sensor simulation component for simulation.

[0226] S428, the target sensor data frame is sent to the autonomous driving component for simulation to obtain a target simulation data frame, and the target simulation data frame is used as the autonomous driving simulation result.

[0227] In this embodiment of the application, measured data frames collected during the actual driving process of the vehicle can be obtained, and the measured data frames may include measured sensor data frames.

[0228] The measured data frames can be obtained by segmenting data recorded during actual vehicle operation. The measured sensor data can be sensor data collected during actual vehicle operation.

[0229] In this embodiment of the application, data flow information can be obtained and written into the measured data frame; the data flow information is used to indicate that at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component is the data source and / or the receiving end of the measured data frame.

[0230] Among them, at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component is the receiving end of the measured data frame, indicating that the measured data frame needs to be sent to at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component.

[0231] Alternatively, at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component is the data source of the measured data frame, indicating that the measured data frame is generated by any one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component.

[0232] In this embodiment, data composition information can be obtained and written into the measured data frame; the data composition information is used to indicate the preceding data frame required for simulating the measured data frame. The preceding data frame can be one or more.

[0233] In this embodiment of the application, the measured data frames can be sorted according to the acquisition time order to form a measured data frame sequence.

[0234] In this embodiment, measured sensor data frames from a measured data frame sequence can be acquired and encapsulated into a first simulation request message. Based on the data flow information within the measured sensor data frames, it can be determined whether the data source is an end-to-end autonomous driving component or a visual-language autonomous driving component. This allows it to be determined that the first simulation request message should be sent to either the end-to-end autonomous driving component or the visual-language autonomous driving component. Then, the process scheduling component sends the first simulation request message to the determined end-to-end autonomous driving component or visual-language autonomous driving component for simulation, resulting in a simulated vehicle control data frame.

[0235] In this embodiment, simulated vehicle control data frames can be encapsulated into a first simulation result response message and sent to the process scheduling component. The first simulation request message encapsulates sensor data frames. The first simulation request message is used to request simulation from the autonomous driving component. The first simulation result response message is used to encapsulate the simulation results generated by the autonomous driving component.

[0236] In this embodiment, the first simulation result response message can be received and parsed by the process scheduling component to obtain the simulated vehicle control data frame. Then, the data flow information can be kept unchanged, the data composition information can be updated according to the simulated vehicle control data frame, and the data in the actual vehicle control data frame can be replaced with the data in the simulated vehicle control data frame to obtain the virtual vehicle control data frame.

[0237] In this embodiment, the virtual vehicle control data frame can be encapsulated into a second simulation request message. Based on the data flow information in the virtual vehicle control data frame, it can be determined that the data source of the virtual vehicle control data frame is a sensor simulation component. The second simulation request message should be sent to the sensor simulation component, and then the process scheduling component sends the second simulation request message to the sensor simulation component for simulation to obtain a simulated sensor data frame.

[0238] In this embodiment, the simulated sensor data frame can be encapsulated into a second simulation result response message and sent to the process scheduling component. The second simulation request message encapsulates a vehicle control data frame. The second simulation request message is used to request simulation from the sensor simulation component. The second simulation result response message is used to encapsulate the simulation results generated by the sensor simulation component.

[0239] In this embodiment, the second simulation result response message can be received and parsed by the process scheduling component to obtain the simulation sensor data frame. Then, the data flow information can be kept unchanged, the data composition information can be updated according to the simulation sensor data frame, and the data in the measured sensor data frame can be replaced with the data in the simulation sensor data frame to obtain the target sensor data frame.

[0240] In this embodiment, the data frame to be simulated can be cached in a data buffer pool. The data frame to be simulated may include the measured sensor data frame in the first simulation request message or the virtual vehicle control data frame in the second simulation request message. The data buffer pool can be used to temporarily store the data frame to be simulated, such as the sensor data frame or the vehicle control data frame.

[0241] In this embodiment, the preceding data frame required for simulating the functional application can be determined based on the data composition information in the data frame to be simulated. If the data buffer pool contains the preceding data frame, the data frame to be simulated is processed by a data injection tool to obtain the data frame to be calculated. The preceding data frame can be determined based on the data composition information in the data frame to be simulated, and there can be one or more preceding data frames. The data injection tool can be used to dynamically modify the data content to make the data frame to be simulated meet the simulation requirements of autonomous driving components or sensor simulation components.

[0242] In this embodiment, the autonomous driving component or sensor simulation component can be controlled to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining a simulation data frame. The simulation data frame may include a simulated vehicle control data frame or a simulated sensor data frame. The functional application refers to an executable module within the autonomous driving component or sensor simulation component used to implement the core algorithm. In this embodiment, it is manifested as an end-to-end autonomous driving application or a visual language autonomous driving application in the autonomous driving component, and as a graphics-physics engine in the sensor simulation component.

[0243] In this embodiment, the autonomous driving component or sensor simulation component calls a callback interface to control the autonomous driving component or sensor simulation component to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining the simulation data frame. The functional application can refer to the executable module within the autonomous driving component or sensor simulation component used to implement the core algorithm. In this embodiment, it is manifested as an end-to-end autonomous driving application or a visual language autonomous driving application in the autonomous driving component, and as a graphics-physics engine in the sensor simulation component. The callback interface is a standardized service endpoint exposed by the functional application.

[0244] Alternatively, in this embodiment, if the functional application does not provide a standardized service endpoint to the outside world, a message middleware can be set in the autonomous driving component or the sensor simulation component. The autonomous driving component or the sensor simulation component calls the message middleware to control the autonomous driving component or the sensor simulation component to drive the functional application to perform simulation based on the data frame to be calculated, and obtain the simulation data frame.

[0245] In this embodiment of the application, the target cache component can be determined according to the data flow information in the data frame to be simulated. That is, the receiving end of the data frame to be simulated indicated by the data flow information is used as the target cache component. The target cache component includes at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component. The target cache component is used to cache the data frame to be simulated.

[0246] Then, the data frame to be simulated can be cached to the data buffer pool, and a caching success message can be generated. The caching success message is then encapsulated into a caching response message and sent to the process scheduling component.

[0247] In this embodiment of the application, after the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, and the cached response message, the process scheduling component may send the first simulation request message to the autonomous driving component for simulation; or, after the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, and the cached response message, the process scheduling component may send the second simulation request message to the sensor simulation component for simulation.

[0248] In the specific implementation, apart from the process scheduling component sending the first simulation request message or the second simulation request message for the first time in the entire simulation process, during the data flow process, the process scheduling component will only send the first simulation request message or the second simulation request message after receiving any one of the first simulation result response message, the second simulation result response message, or the cached response message.

[0249] In this embodiment, the target sensor data frame can be sent to the autonomous driving component for simulation to obtain the target simulation data frame, and the target simulation data frame can be used as the result of the autonomous driving simulation.

[0250] The target simulation data frame can be obtained by the autonomous driving component through simulation based on the target sensor data frame. The target simulation data frame can be used as the result of the autonomous driving simulation.

[0251] In this embodiment, measured data frames collected during actual vehicle operation are acquired. These measured data frames include measured sensor data frames. This application uses measured data frames from real vehicle operation as the simulation basis, ensuring the authenticity and reliability of the data source. The measured sensor data frames are sent to the autonomous driving component for simulation, resulting in simulated vehicle control data frames. This application directly drives the autonomous driving component for simulation using measured sensor data frames, effectively preserving the physical characteristics of real sensor data and avoiding data distortion problems caused by virtual rendering in traditional simulation engines. The simulated vehicle control data frames are sent to the sensor simulation component for simulation, resulting in target sensor data frames. This application feeds back the simulated vehicle control data frames to the sensor simulation component for secondary simulation, achieving dynamic updates of sensor data and overcoming the limitation of traditional backfeedback schemes that can only statically reproduce recorded data. The target sensor data frames are sent to the autonomous driving component for simulation, resulting in target simulation data frames, which are used as the autonomous driving simulation result. This application constructs a complete closed-loop simulation link, which maintains the advantage of real vehicle data authenticity and gains the ability to dynamically adjust the simulation environment. The final generated target simulation data frame contains the physical characteristics of the real scene and reflects the dynamic interaction process between the autonomous driving system and the environment. It provides an autonomous driving system with a simulation test environment that combines high fidelity and strong generalization ability, significantly improving the effectiveness and reliability of autonomous driving algorithm verification.

[0252] Reference Figure 2 This diagram illustrates the architecture of an autonomous driving simulation system according to an embodiment of this application. In specific implementations, this architecture can be used as a reference to construct an autonomous driving simulation system.

[0253] The autonomous driving simulation system includes a process scheduling component 201, an end-to-end autonomous driving component 215, a visual language autonomous driving component 216, and a sensor simulation component 217.

[0254] The process scheduling component 201 includes a message queue 202 and scheduling objects 203, 204 and 205.

[0255] The end-to-end autonomous driving component 215 includes a data buffer pool 206, a data injection tool 209, and an end-to-end autonomous driving application 212.

[0256] The visual language autonomous driving component 216 includes a data buffer pool 207, a data injection tool 210, and a visual language autonomous driving application 214.

[0257] The sensor simulation component 217 includes a data buffer pool 208, a data injection tool 211, and a graphics physics engine 214.

[0258] In a specific implementation, the process scheduling component 201 can be a flow control scheduler. The process scheduling component 201 is responsible for serializing the simulation process and coordinating the running rhythm of software with different operating frequencies.

[0259] The end-to-end autonomous driving component 215, the visual language autonomous driving component 216, and the sensor simulation component 217 can all be in the form of software containers. The software containers are responsible for responding to scheduling, driving the fast and slow dual-system autonomous driving software to run, and driving the graphics physics engine to generate sensor data.

[0260] The entire simulation process achieves closed-loop simulation based on real vehicle data under the master-slave scheduling of the flow control scheduler and software container.

[0261] Within the process scheduling component 201, there is a message queue 202 and three scheduling objects: scheduling object 203, scheduling object 204, and scheduling object 205. These three scheduling objects correspond to the software containers that load the end-to-end autonomous driving application 212, the visual language autonomous driving application 213, and the graphics physics engine 214, respectively. That is, scheduling object 203 corresponds to the end-to-end autonomous driving component 215, scheduling object 204 corresponds to the visual language autonomous driving component 216, and scheduling object 205 corresponds to the sensor simulation component 217.

[0262] Message queue 202 is responsible for loading recorded data and sorting it according to the timestamps recorded in the data. In this embodiment, the recorded data can be measured data frames, and message queue 202 can be a sequence of measured data frames. After sorting according to the timestamps recorded in the data, the measured data frame sequence in this embodiment can be obtained.

[0263] During simulation, message queue 202 can sequentially send data from message queue 202 to the corresponding software containers via scheduling objects, based on the topics of interest to end-to-end autonomous driving application 212 and visual language autonomous driving application 213, thereby driving the corresponding autonomous driving software to complete calculations. In this embodiment, the topics of interest to end-to-end autonomous driving application 212 and visual language autonomous driving application 213 can refer to data flow information.

[0264] Meanwhile, message queue 202 also collects new messages generated by end-to-end autonomous driving application 212 and visual language autonomous driving application 213, replaces the original measured data, and sends the replaced messages to the graphics engine and physics engine to complete the rendering of the new scene and generate sensor data. Similarly, these newly generated sensor data are collected back to replace the original measured data and sent to end-to-end autonomous driving application 212 and visual language autonomous driving application 213 in the subsequent simulation process to realize closed-loop simulation of autonomous driving software with fast and slow dual system architecture.

[0265] Scheduler objects 203, 204, and 205 are responsible for communicating with their corresponding software containers via remote calls. Each remote call is blocking; the message queue is only triggered to continue distributing messages after the software container returns a message. This mechanism ensures that the entire simulation process is serialized, with only one software container running at a time.

[0266] Remote procedure calls (RPCs) are a key method for the process scheduling component 201 to serialize the simulation process. RPCs are implemented using RPCs, a common application-layer network communication method typically implemented with a server / client architecture. The client initiates a request, encapsulating the parameters and data required for a calculation in a request message and sending it to the server. The server receives the request message, completes the corresponding calculation, and returns the result in a response message to the client. Each client request receives a response from the server; the call process is blocking, and the client's calling thread only continues running upon receiving a response message. If there is no response for an extended period, the client should throw a timeout exception. The server and client can be deployed on different machines, transmitting data via TCP / UDP transport layer protocols. Alternatively, they can be deployed on the same machine, transmitting data via inter-process communication.

[0267] In this embodiment, the end-to-end autonomous driving component 215, the visual language autonomous driving component 216, and the sensor simulation component 217 are remote call servers that can provide a feedback message service. Within the end-to-end autonomous driving component 215, the visual language autonomous driving component 216, and the sensor simulation component 217, data buffer pools 206, 207, and 208, and data injection tools 209, 210, and 211 isolate the loaded end-to-end autonomous driving application 212, the visual language autonomous driving application 214, and the graphics physics engine 214, providing them with independent operating environments and communication channels.

[0268] Based on the data source and / or receiver of the measured data frame, the data received by the end-to-end autonomous driving component 215, the visual language autonomous driving component 216, and the sensor simulation component 217 can be divided into two categories. One category is input messages received by the receiver of the measured data frame from other functional software, and the other category is output messages published by the functional applications loaded within the measured data frame. In this embodiment, both input and output messages can be represented by simulation request messages.

[0269] Whenever an input message is received, the end-to-end autonomous driving component 215, the visual language autonomous driving component 216, and the sensor simulation component 217 store it in the flow control message buffer pool and notify the process scheduling component 201 that the message has been fully received via a cached response message. When an output message is received, the end-to-end autonomous driving component 215, the visual language autonomous driving component 216, and the sensor simulation component 217 check the data composition information of the preceding data frames required to indicate the simulation of the measured data frame. Once all the preceding data frames for the measured data frame have been collected in the data buffer pools 206, 207, and 208 of the corresponding components, these preceding data frames are retrieved from the data buffer pools 206, 207, and 208. Then, after making the necessary modifications through data injection tools 209, 210 and 211, the data is passed to the end-to-end autonomous driving application 212, the visual language autonomous driving application 214 and the graphics physics engine 214, driving the end-to-end autonomous driving application 212, the visual language autonomous driving application 214 and the graphics physics engine 214 to complete a simulation calculation, and the new data produced is fed back to the process scheduling component 201 through the simulation response message of remote call.

[0270] Furthermore, for the sensor simulation component 217 equipped with the graphics physics engine 214, the input messages are vehicle control data issued by the end-to-end autonomous driving component 215 and the visual language autonomous driving component 216. The graphics physics engine 214 can render environmental information based on the vehicle control data, and the output messages of the sensor simulation component 217 are simulated sensor data generated based on the vehicle control data and environmental information. The interface of the graphics physics engine 214 is encapsulated through the data buffer pool 208 and the data injection tool 211. This allows for the application of generalization techniques based on real-world data such as world models to generate highly realistic scenes. This provides closed-loop data support for the simulation scheme of the fast and slow dual-system autonomous driving system.

[0271] Furthermore, the way the end-to-end autonomous driving component 215, the visual language autonomous driving component 216, and the sensor simulation component 217 drive the execution of their loaded end-to-end autonomous driving application 212, visual language autonomous driving application 214, and graphics physics engine 214 depends on the message middleware and application interfaces used by the end-to-end autonomous driving application 212, visual language autonomous driving application 214, and graphics physics engine 214. If the application interface directly provides callback interfaces for messages and timers, then the end-to-end autonomous driving application 212, visual language autonomous driving application 214, and graphics physics engine 214 can be driven to run directly by loading them through dynamic link libraries.

[0272] If the application interface does not directly provide a callback interface, then an additional message middleware needs to be set up in the end-to-end autonomous driving component 215, the visual language autonomous driving component 216, and the sensor simulation component 217. The logic of the message middleware needs to be set before the data buffer pool 206, the data buffer pool 207, and the data buffer pool 208. Then, the message middleware is encapsulated, and the messages are still provided to the end-to-end autonomous driving application 212, the visual language autonomous driving application 214, and the graphics physics engine 214 through the subscription / publishing mode inside the end-to-end autonomous driving component 215, the visual language autonomous driving component 216, and the sensor simulation component 217.

[0273] In addition, communication needs to be isolated from the end-to-end autonomous driving component 215, the visual language autonomous driving component 216, and the sensor simulation component 217 to prevent them from communicating with each other.

[0274] Reference Figure 3 This illustration shows a schematic diagram of a data flow process provided in an embodiment of this application. In a specific implementation, the transmission of some data frames and... Figure 3 The process is similar.

[0275] To coordinate the operational rhythms of end-to-end autonomous driving applications and visual language autonomous driving applications with different operating frequencies, and to support closed-loop simulation of autonomous driving systems with a fast and slow dual-system architecture, process control information is introduced. In this embodiment, the process control information can be data composition information. Furthermore, after the end-to-end autonomous driving application, the visual language autonomous driving application, and the graphics physics engine complete their calculations, the timestamps or frame numbers of the preceding data involved in this calculation are recorded in the generated data.

[0276] like Figure 3 As shown, there are two functional applications 305 and 306. In this embodiment, the functional application can be any one of an end-to-end autonomous driving application, a visual language autonomous driving application, and a graphics physics engine.

[0277] Functional application 305 generates frame e based on the input data from three sensors: frame a of sensor 301, frame b of sensor 302, and frame c of sensor 303, and outputs it to downstream functional application 306.

[0278] Function application 306 receives the e-th frame data from function application 305 and the d-th frame data from sensor 304, and produces the f-th frame data from function application 306.

[0279] Reference Figure 4 This diagram illustrates the structure of a data frame according to an embodiment of this application. In specific implementations, the data frame structure can be referred to... Figure 4 set up.

[0280] In the e-th frame data of functional application 305 and the f-th frame data of functional application 306 produced by functional application 305 and functional application 306 respectively, a field specifically set is set to reflect the data composition information, which is used to record their upstream process control information, that is, to record the preceding data of the current frame data. In the embodiments of this application, the data frame is set with data composition information for indicating the preceding data frame required for simulating the measured data frame.

[0281] like Figure 4 As shown, the data in frame e of function application 305 includes data composition information: data from frame a of sensor 301, data from frame b of sensor 302, and data from frame c of sensor 303. Frame number e. Timestamp: system time at the time of publication. And other necessary data.

[0282] The data in frame f of function application 306 includes data composition information: data in frame e of function application 305, and data in frame d of sensor 304. Frame number f. Timestamp: system time at the time of publication. And other necessary data.

[0283] In the embodiments of the present application, other necessary data may be vehicle control data and sensor data.

[0284] with reference to Figure 5 , a timing schematic diagram of a data frame provided by an embodiment of the present application is shown. In a specific implementation, the timing of the data frame may refer to Figure 5 for configuration.

[0285] The process scheduling component 201 may sort the data frames according to the timestamps recorded in the data frames. Due to the causal relationship of generating the data frames, if the data frame output by the functional application is set as the current data frame, the time in the timestamp of the data frame serving as the precondition data will definitely be earlier than the time in the timestamp of the current data frame.

[0286] as shown in Figure 5 , the a-th frame data of the sensor 301, the b-th frame data of the sensor 302, the c-th frame data of the sensor 303, the e-th frame data of the functional application 305, the d-th frame data of the sensor 304, and the f-th frame data of the functional application 306. The timing sequence between these data is {t1,t2,t3}<t4, {t4,t5}<t6. It can be ensured that before the functional application generates data through simulation, the precondition data required for the simulation has been sent to the data buffer pool of the component where the corresponding functional application is located. In the embodiments of the present application, the functional application may be any one of an end-to-end autonomous driving application, a visual-language autonomous driving application, and a graphics physics engine.

[0287] with reference to Figure 6 , a logic schematic diagram of a process scheduling component provided by an embodiment of the present application is shown. In a specific implementation, the logic of the process scheduling component may refer to Figure 6 for configuration.

[0288] First, the process scheduling component constructs a mapping relationship between the subscribe / publish topics and the functional software by loading the operation configuration. In the embodiments of the present application, the mapping relationship between the subscribe / publish topics and the functional software may be represented by data flow information.

[0289] Then, the process scheduling component sorts the messages according to the timestamps recorded in the messages and writes them into the message queue. In the embodiments of the present application, the message queue may be represented by a data frame sequence.

[0290] Next, the process scheduler will sequentially scan all the measured messages starting from the first frame of data in the message queue. If the simulation component is the data source of the data frame, the data frame is directly sent to the corresponding simulation component through RPC. In the embodiments of the present application, the simulation component is any one of an end-to-end autonomous driving component, a visual-language autonomous driving component, and a sensor simulation component.

[0291] The simulation component drives end-to-end autonomous driving applications, visual language autonomous driving applications, and the graphics physics engine to generate new data and returns the new data through simulation response messages.

[0292] The process scheduling component will replace the original measured data with the returned new data. After completing the necessary message replacement, the flow control scheduler will send the replaced new data to other simulation components that are acting as data receivers, allowing these simulation components to cache the new data.

[0293] After traversing all messages in the message queue, the simulation exits normally. If a timeout or exception occurs, primarily a communication timeout, or the software container returns an exception message, the simulation exits abnormally and terminates.

[0294] Reference Figure 7 This diagram illustrates a logic schematic of a simulation component according to an embodiment of this application. In a specific implementation, the logic of the simulation component can be referred to... Figure 7 set up.

[0295] The operation process of the simulation component in this invention is as follows: Figure 7 As shown. In this embodiment, the simulation component is any one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component.

[0296] First, the runtime configuration is loaded, and the mapping relationship between subscribed / published topics and functional software is obtained. In this embodiment, the mapping relationship between subscribed / published topics and functional software can be reflected by data flow information.

[0297] Then, start the remote call service and receive request messages.

[0298] Without terminating the simulation, if the current simulation component is the data source for this data frame, check the data buffer pool to confirm whether all preceding data frames have been cached. If any messages are missing, it indicates that the operating conditions of the current functional application are not met. An exception response message should be constructed and returned, and the component's abnormal exit logic should be executed.

[0299] If all preceding data frames have been cached, simulation data is injected using a data injection tool. The resulting functional application then completes a computation, filling the simulation response message with the new output. The functional application can be any of the following: an end-to-end autonomous driving application, a visual-language autonomous driving application, or a graphics-physics engine.

[0300] If the current simulation component is the receiver of this data frame, and receives an input message, it saves it to the data buffer pool, constructs and returns a cached response message.

[0301] If a scheduling instruction to exit the simulation task is received, the component will enter the normal exit process, and the simulation will end.

[0302] Reference Figure 8 This illustration shows a schematic diagram of the structure of an autonomous driving simulation device according to an embodiment of this application, applied to an autonomous driving simulation system. The autonomous driving simulation system includes autonomous driving components and sensor simulation components. The device includes:

[0303] The data acquisition module 801 is used to acquire measured data frames collected during the actual driving process of the vehicle; the measured data frames include measured sensor data frames.

[0304] The first simulation module 802 is used to send the measured sensor data frame to the autonomous driving component for simulation to obtain the simulated vehicle control data frame.

[0305] The second simulation module 803 is used to perform simulation based on the simulated vehicle control data frame through the sensor simulation component to obtain the target sensor data frame.

[0306] The third simulation module 804 is used to send the target sensor data frame to the autonomous driving component for simulation, obtain the target simulation data frame, and use the target simulation data frame as the autonomous driving simulation result.

[0307] In one optional embodiment of this application, the measured data frame includes a measured vehicle control data frame, and the second simulation module 803 includes:

[0308] The first update submodule is used to update the measured vehicle control data frame according to the simulated vehicle control data frame to obtain a virtual vehicle control data frame.

[0309] The first simulation submodule is used to send the virtual vehicle control data frame to the sensor simulation component for simulation to obtain the simulated sensor data frame.

[0310] The second update submodule is used to update the measured sensor data frame according to the simulated sensor data frame to obtain the target sensor data frame.

[0311] In one optional embodiment of this application, the autonomous driving component includes an end-to-end autonomous driving component and / or a visual language autonomous driving component, and the device further includes:

[0312] The data flow information acquisition module is used to acquire data flow information and write it into the measured data frame; the data flow information is used to indicate that at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component is the data source and / or the receiving end of the measured data frame.

[0313] A data composition information acquisition module is used to acquire data composition information and write it into the measured data frame; the data composition information is used to indicate the preceding data frame required for simulating the measured data frame.

[0314] The data frame sequence acquisition module is used to sort the measured data frames according to the acquisition time order to form a measured data frame sequence.

[0315] In one optional embodiment of this application, the autonomous driving simulation system further includes a process scheduling component, wherein the first simulation module 802 includes:

[0316] The measured sensor data acquisition submodule is used to acquire the measured sensor data frames in the measured data frame sequence;

[0317] The first message encapsulation submodule is used to encapsulate the measured sensor data frame into a first simulation request message;

[0318] The second simulation submodule is used to send the first simulation request message to the autonomous driving component for simulation through the process scheduling component based on the data flow information in the measured sensor data frame, so as to obtain the simulated vehicle control data frame.

[0319] In one optional embodiment of this application, the apparatus further includes:

[0320] The second message encapsulation submodule is used to encapsulate the simulated vehicle control data frame into a first simulation result response message and send it to the process scheduling component.

[0321] In one optional embodiment of this application, the first update submodule includes:

[0322] The first message parsing unit is used to receive and parse the first simulation result response message through the process scheduling component to obtain the simulation vehicle control data frame.

[0323] The first data update unit is used to update the data composition information according to the simulated vehicle control data frame while keeping the data flow information unchanged.

[0324] The second data update unit is used to replace the data in the actual vehicle control data frame with the data in the simulated vehicle control data frame to obtain the virtual vehicle control data frame.

[0325] In one optional embodiment of this application, the second simulation module 803 includes:

[0326] The third message encapsulation submodule is used to encapsulate the virtual vehicle control data frame into a second simulation request message;

[0327] The third simulation submodule is used to send the second simulation request message to the sensor simulation component for simulation through the process scheduling component based on the data flow information in the virtual vehicle control data frame, so as to obtain the simulation sensor data frame.

[0328] In one optional embodiment of this application, the apparatus further includes:

[0329] The fourth message encapsulation submodule is used to encapsulate the simulation sensor data frame into a second simulation result response message and send it to the process scheduling component.

[0330] In one optional embodiment of this application, the second update submodule includes:

[0331] The second message parsing unit is used to receive and parse the second simulation result response message through the process scheduling component to obtain the simulation sensor data frame.

[0332] The third data update unit is used to update the data composition information based on the simulation sensor data frame while keeping the data flow information unchanged.

[0333] The fourth data update unit is used to replace the data in the measured sensor data frame with the data in the simulated sensor data frame to obtain the target sensor data frame.

[0334] In one optional embodiment of this application, the autonomous driving component or the sensor simulation component includes a data buffer pool, a data injection tool, and a functional application corresponding to the autonomous driving component or the sensor simulation component, and the simulation is performed using the following steps:

[0335] The data caching module is used to cache the data frame to be simulated to the data buffer pool; the data frame to be simulated includes the measured sensor data frame in the first simulation request message or the virtual vehicle control data frame in the second simulation request message;

[0336] The pre-data confirmation module is used to determine the pre-data frame required to simulate the functional application based on the data composition information in the data frame to be simulated.

[0337] The data injection module is used to process the data frame to be simulated through the data injection tool to obtain the data frame to be calculated if the data buffer pool contains the preceding data frame.

[0338] The fourth simulation module is used to control the autonomous driving component or the sensor simulation component to drive the functional application to perform simulation based on the data frame to be calculated, so as to obtain the simulation data frame; the simulation data frame includes the simulated vehicle control data frame or the simulated sensor data frame.

[0339] In one optional embodiment of this application, the apparatus further includes:

[0340] A target cache component determination module is used to determine a target cache component based on the data flow information in the data frame to be simulated; the target cache component includes at least one of an end-to-end autonomous driving component, a visual language autonomous driving component, and a sensor simulation component; the target cache component is used to cache the data frame to be simulated;

[0341] The cache success information generation module is used to cache the data frame to be simulated to the data buffer pool and generate cache success information;

[0342] The cache response message encapsulation module is used to encapsulate the cache success information into a cache response message and send it to the process scheduling component.

[0343] In one optional embodiment of this application, the apparatus further includes:

[0344] The first message scheduling module is used to send the first simulation request message to the autonomous driving component for simulation after the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, and the cached response message;

[0345] Alternatively, a second message scheduling module is configured to, after the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, and the cached response message, send a second simulation request message to the sensor simulation component for simulation.

[0346] In one optional embodiment of this application, the autonomous driving component or the sensor simulation component includes a message middleware, or the functional application includes a callback interface, and the fourth simulation module includes:

[0347] The callback interface calling module is used by the autonomous driving component or the sensor simulation component to call the callback interface, drive the functional application to perform simulation based on the data frame to be calculated, and obtain the simulation data frame.

[0348] Alternatively, a message middleware invocation module may be used by the autonomous driving component or the sensor simulation component to invoke the message middleware to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining the simulation data frame.

[0349] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.

[0350] An embodiment of this application also provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the method described above.

[0351] An embodiment of this application also provides a storage medium, which may be a computer-readable storage medium, on which a computer program may be stored, and which, when executed by a processor, implements the method described above.

[0352] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0353] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0354] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0355] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0356] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0357] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0358] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other modifications and updates to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all modifications and updates falling within the scope of the embodiments of the present application.

[0359] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the aforementioned element.

[0360] The above provides a detailed description of the autonomous driving simulation method, device, electronic device, and storage medium. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An autonomous driving simulation method, characterized in that, Applied to an autonomous driving simulation system, the autonomous driving simulation system including autonomous driving components and sensor simulation components, the method includes: Acquire measured data frames collected during actual vehicle operation; the measured data frames include measured sensor data frames; The measured sensor data frame is sent to the autonomous driving component for simulation to obtain the simulated vehicle control data frame; The target sensor data frame is obtained by performing simulation based on the simulated vehicle control data frame using the sensor simulation component. The target sensor data frame is sent to the autonomous driving component for simulation to obtain a target simulation data frame, which is then used as the autonomous driving simulation result.

2. The method according to claim 1, characterized in that, The measured data frame includes a measured vehicle control data frame. The step of simulating the target sensor data frame using the sensor simulation component based on the simulated vehicle control data frame includes: The measured vehicle control data frame is updated based on the simulated vehicle control data frame to obtain the virtual vehicle control data frame. The virtual vehicle control data frame is sent to the sensor simulation component for simulation to obtain the simulated sensor data frame; The measured sensor data frame is updated based on the simulated sensor data frame to obtain the target sensor data frame.

3. The method according to claim 1 or 2, characterized in that, The autonomous driving component includes an end-to-end autonomous driving component and / or a visual language autonomous driving component, and the method further includes: Data flow information is acquired and written into the measured data frame; the data flow information is used to indicate that at least one of the end-to-end autonomous driving component, the visual language autonomous driving component, and the sensor simulation component is the data source and / or the receiving end of the measured data frame. Acquire data composition information and write it into the measured data frame; the data composition information is used to indicate the preceding data frame required for simulating the measured data frame; The measured data frames are sorted according to the acquisition time to form a measured data frame sequence.

4. The method according to claim 3, characterized in that, The autonomous driving simulation system also includes a process scheduling component. The step of sending the measured sensor data frames to the autonomous driving component for simulation to obtain simulated vehicle control data frames includes: Obtain the measured sensor data frames from the measured data frame sequence; The measured sensor data frame is encapsulated into a first simulation request message; Based on the data flow information in the measured sensor data frame, the first simulation request message is sent to the autonomous driving component for simulation through the process scheduling component to obtain the simulated vehicle control data frame.

5. The method according to claim 4, characterized in that, The method further includes: The simulated vehicle control data frame is encapsulated into a first simulation result response message and sent to the process scheduling component.

6. The method according to claim 5, characterized in that, The step of updating the measured vehicle control data frame according to the simulated vehicle control data frame to obtain the virtual vehicle control data frame includes: The process scheduling component receives and parses the first simulation result response message to obtain the simulated vehicle control data frame. The data flow information remains unchanged, and the data composition information is updated according to the simulated vehicle control data frame; The virtual vehicle control data frame is obtained by replacing the data in the actual vehicle control data frame with the data in the simulated vehicle control data frame.

7. The method according to claim 2, characterized in that, The autonomous driving simulation system further includes a process scheduling component, wherein sending the virtual vehicle control data frame to the sensor simulation component for simulation to obtain a simulated sensor data frame includes: The virtual vehicle control data frame is encapsulated into a second simulation request message; Based on the data flow information in the virtual vehicle control data frame, the second simulation request message is sent to the sensor simulation component for simulation through the process scheduling component to obtain the simulation sensor data frame.

8. The method according to claim 7, characterized in that, The method further includes: The simulated sensor data frame is encapsulated into a second simulation result response message and sent to the process scheduling component.

9. The method according to claim 8, characterized in that, The step of updating the measured sensor data frame based on the simulated sensor data frame to obtain the target sensor data frame includes: The process scheduling component receives and parses the second simulation result response message to obtain the simulation sensor data frame; The data flow information remains unchanged, and the data composition information is updated according to the simulated sensor data frame; The target sensor data frame is obtained by replacing the data in the actual sensor data frame with the data in the simulated sensor data frame.

10. The method according to any one of claims 2-9, characterized in that, The autonomous driving component or the sensor simulation component includes a data buffer pool, a data injection tool, and functional applications corresponding to the autonomous driving component or the sensor simulation component. The simulation is performed using the following steps: The data frame to be simulated is cached in the data buffer pool; The preceding data frame required to simulate the functional application is determined based on the data composition information in the data frame to be simulated. If the data buffer pool contains the preceding data frame, the data frame to be simulated is processed by the data injection tool to obtain the data frame to be calculated. The autonomous driving component or the sensor simulation component is controlled to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining the simulation data frame.

11. The method according to claim 10, characterized in that, The method further includes: The target cache component is determined based on the data flow information in the data frame to be simulated; the target cache component includes at least one of an end-to-end autonomous driving component, a visual language autonomous driving component, and a sensor simulation component; the target cache component is used to cache the data frame to be simulated; The data frame to be simulated is cached in the data buffer pool, and a caching success message is generated; The cache success information is encapsulated into a cache response message and sent to the process scheduling component.

12. The method according to any one of claims 10-11, characterized in that, The method further includes: After the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, and the cached response message, the process scheduling component sends the first simulation request message to the autonomous driving component for simulation. Alternatively, after the process scheduling component receives any one of the first simulation result response message, the second simulation result response message, or the cached response message, the process scheduling component sends a second simulation request message to the sensor simulation component for simulation.

13. The method according to any one of claims 10-12, characterized in that, The autonomous driving component or the sensor simulation component includes a message middleware, or the functional application includes a callback interface. The control of the autonomous driving component or the sensor simulation component drives the functional application to perform simulation based on the data frame to be calculated, to obtain the simulation data frame, including: The autonomous driving component or the sensor simulation component calls the callback interface to drive the functional application to perform simulation based on the data frame to be calculated, and obtain the simulation data frame. Alternatively, the autonomous driving component or the sensor simulation component may invoke the message middleware to drive the functional application to perform simulation based on the data frame to be calculated, thereby obtaining the simulation data frame.

14. An autonomous driving simulation device, characterized in that, Applied to an autonomous driving simulation system, the autonomous driving simulation system includes autonomous driving components and sensor simulation components, the device includes: The data acquisition module is used to acquire measured data frames collected during the actual driving process of the vehicle; the measured data frames include measured sensor data frames. The first simulation module is used to send the measured sensor data frame to the autonomous driving component for simulation to obtain the simulated vehicle control data frame. The second simulation module is used to perform simulation based on the simulated vehicle control data frame through the sensor simulation component to obtain the target sensor data frame. The third simulation module is used to send the target sensor data frame to the autonomous driving component for simulation, obtain the target simulation data frame, and use the target simulation data frame as the autonomous driving simulation result.

15. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1-13.

16. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-13.

Images