Simulation method and apparatus, and computing device cluster

Abstract

A simulation method, comprising: acquiring an input map of a simulation environment (S701); on the basis of the input map, generating road network information of the simulation environment (S702); on the basis of the road network information and trajectory information of a vehicle, obtaining motion trajectory information of at least one object within a first time period (S703); and within the first time period of executing an autonomous driving simulation task, on the basis of the road network information or on the basis of the road network information and the motion trajectory information of the at least one object within the first time period, performing simulation to obtain simulation data of at least one virtual sensor of the vehicle within the first time period (S704).

Description

A simulation method, apparatus, and cluster of computing devices

[0001] This application claims priority to Chinese Patent Application No. 202410796433.6, filed on June 19, 2024, entitled "An End-to-End Controllable Simulation System for Autonomous Driving Environment", and to Chinese Patent Application No. 202410796433.X, filed on September 19, 2024, entitled "A Simulation Method, Apparatus and Computing Device Cluster", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to the field of autonomous driving technology, and in particular to a simulation method, apparatus, and computing device cluster. Background Technology

[0003] Autonomous driving technology refers to technologies that enable vehicles to navigate and drive without direct human intervention. It relies on advanced sensors, computer vision, and artificial intelligence algorithms to allow vehicles to perceive their surroundings, make decisions, and execute actions. Autonomous driving technology is considered a crucial direction for the future of transportation, promising to improve traffic efficiency, reduce accidents, and enhance driver comfort and convenience.

[0004] In autonomous driving technology, the perception module and the planning and control module are the two most crucial algorithm modules. The perception module is responsible for acquiring information about the surrounding environment through sensors (such as radar, cameras, and lidar). This information includes roads, obstacles, pedestrians, and signs. The perception module accurately identifies and understands this information. Based on the information perceived by the perception module, the planning and control module formulates and executes the vehicle's driving path and actions to ensure safe and efficient driving. However, in autonomous driving technology, the perception module, especially, typically requires extensive data training to achieve sufficient performance. Data from extreme cases (corner cases) that are difficult to collect during conventional road surveys significantly impacts the perception module's capabilities under these extreme conditions. Summary of the Invention

[0005] To address the aforementioned problems, embodiments of this application provide a simulation method that can improve the realism of simulation data and reduce the structural complexity of the simulation module. Furthermore, this application also provides a simulation apparatus and computing device cluster corresponding to this simulation method.

[0006] Therefore, the following technical solutions are adopted in the embodiments of this application:

[0007] In a first aspect, this application provides a simulation method applied to a driving simulation platform for executing driving simulation tasks, the driving simulation tasks including driving tasks of at least one vehicle in a simulation environment. The method includes: acquiring an input map of the simulation environment; generating road network information of the simulation environment based on the input map; and, during a first time period of executing the autonomous driving simulation task, simulating simulation data of at least one virtual sensor of the vehicle during the first time period based on the road network information; the simulation data being used to analyze environmental information of the vehicle's surrounding environment during the first time period, the environmental information being used to analyze control commands for the vehicle during the first time period, the control commands being used to control the driving behavior of the vehicle, the surrounding environment including at least one object, and the environmental information including the position of the at least one object.

[0008] In this embodiment, the method only needs to generate road network information during the generation of simulation data from various virtual sensors, without needing to construct an environmental model around the vehicle, thus reducing the structural complexity of the simulation module. Furthermore, this method can directly simulate the simulation data from various virtual sensors based on the road network information, which not only improves the realism of the simulation data but also eliminates the need for a physics engine and graphics engine, further reducing the structural complexity of the simulation module.

[0009] In one embodiment, before simulating simulation data of at least one virtual sensor of the vehicle during the first time period of executing the autonomous driving simulation task based on the road network information, the method further includes: determining at least one object in the surrounding environment of the vehicle during the first time period; obtaining motion trajectory information of the at least one object during the first time period based on the road network information and the trajectory information of the vehicle; the step of simulating simulation data of at least one virtual sensor of the vehicle during the first time period of executing the autonomous driving simulation task based on the road network information specifically includes: simulating simulation data of at least one virtual sensor of the vehicle during the first time period of executing the autonomous driving simulation task based on the road network information and the motion trajectory information of the at least one object during the first time period.

[0010] In this embodiment, after identifying objects around the vehicle, the method can predict the trajectory of at least one object around the vehicle. Then, based on road network information and the trajectory of at least one object around the vehicle, it can directly simulate the simulation data of various virtual sensors including at least one object around the vehicle, which can further improve the realism of the simulation data.

[0011] In one implementation, determining at least one object in the vehicle's surrounding environment during the first time period specifically includes: determining the number of objects and the type of objects in the vehicle's surrounding environment during the first time period based on preset conditions and / or prediction instructions input by the user.

[0012] In this embodiment, the method can simulate the simulation data of various virtual sensors of the vehicle in different scenarios based on preset conditions and / or prediction instructions input by the user, so as to improve the application scenarios of autonomous driving technology.

[0013] In one embodiment, the motion trajectory information of the at least one object during the first time period is represented by a three-dimensional bounding box.

[0014] In this embodiment, the method represents each position in the form of a 3D bounding box, so that when generating video, the position of the object is represented by the 3D bounding box, allowing the driver to perceive the position of the vehicle more intuitively.

[0015] In one implementation, generating road network information of the simulation environment based on the input map specifically includes: inputting the input map into the simulator to obtain road network information of the vehicle's surrounding environment during the first time period. The road network information includes one or more of the following: road shape, road connectivity, traffic signs, lane markings, and traffic signals.

[0016] In this embodiment, the method utilizes a simulator to simulate road network information of the vehicle's surrounding environment on an input map, including road shapes, connections, traffic signs, lane markings, traffic signals, and other road information. Compared to methods that generate 3D environment models, this method generates road network information in a simpler process, requires fewer related algorithms, and thus simplifies the structure of the simulation module.

[0017] In one embodiment, when the at least one virtual sensor includes a virtual camera, the step of simulating simulation data of at least one virtual sensor of the vehicle during the first time period of performing the autonomous driving simulation task, based on the road network information, specifically includes: inputting the road network information into a video generation model to obtain a video of the vehicle's surrounding environment during the first time period, wherein the video generation model is a neural network model, and the simulation data includes the video.

[0018] In this implementation, the method utilizes generation algorithms, such as video generation technology based on a diffusion model, to directly simulate the video from a virtual camera. Compared to using a 3D environment model, the simulation data generated by this method has higher realism. Moreover, this method only requires the generation algorithm and does not require additional physics and graphics engines, further reducing the structural complexity of the simulation module.

[0019] In one embodiment, before inputting the road network information into the video generation model to obtain a video of the vehicle's surrounding environment in the first time period, the method further includes: training a basic neural network model using training data to obtain the video generation model, wherein the training data includes feature data and label data, the feature data being road network information training data, and the label data being the video corresponding to the road network information training data.

[0020] Secondly, this application provides a simulation device, comprising: an environment simulation unit for acquiring an input map of a simulation environment; the environment simulation unit is further configured to generate road network information of the simulation environment based on the input map; a data simulation unit for simulating simulation data of at least one virtual sensor of the vehicle during a first time period of executing the autonomous driving simulation task, based on the road network information; a perception module for analyzing environmental information of the vehicle's surrounding environment during the first time period based on the simulation data; and a planning and control module for analyzing control instructions for the vehicle during the first time period based on the environmental information, the control instructions being used to control the driving behavior of the vehicle, the surrounding environment including at least one object, and the environmental information including the position of the at least one object.

[0021] In one embodiment, the trajectory simulation unit is used to determine at least one object in the surrounding environment of the vehicle during the first time period; and to obtain the motion trajectory information of the at least one object during the first time period based on the road network information and the trajectory information of the vehicle; the data simulation unit is specifically used to simulate simulation data of at least one virtual sensor of the vehicle during the first time period based on the road network information and the motion trajectory information of the at least one object during the first time period when performing the autonomous driving simulation task.

[0022] In one embodiment, the trajectory simulation unit is specifically used to determine the number of objects in the vehicle's surrounding environment during the first time period and the types of objects in the vehicle's surrounding environment during the first time period based on preset conditions and / or prediction instructions input by the user.

[0023] In one embodiment, the motion trajectory information of the at least one object during the first time period is represented by a three-dimensional bounding box.

[0024] In one embodiment, the environment simulation unit is specifically used to input the input map into the simulator to obtain road network information of the vehicle's surrounding environment during the first time period. The road network information includes one or more of the following: road shape, road connection relationship, traffic signs, lane markings, and traffic signals.

[0025] In one embodiment, when the at least one virtual sensor includes a virtual camera, the data simulation unit is specifically used to input the road network information into a video generation model to obtain a video of the vehicle's surrounding environment during the first time period. The video generation model is a neural network model, and the simulation data includes the video.

[0026] In one embodiment, before inputting the road network information into the video generation model to obtain a video of the vehicle's surrounding environment in the first time period, the data simulation unit is further configured to train a basic neural network model using training data to obtain the video generation model. The training data includes feature data and label data, wherein the feature data is road network information training data and the label data is the video corresponding to the road network information training data.

[0027] Thirdly, embodiments of this application provide a computing device, including: at least one memory; and at least one processor, the processor being configured to execute instructions stored in the memory to cause the computing device to perform the various possible implementations of the first aspect.

[0028] Fourthly, embodiments of this application provide a computer-readable storage medium including computer program instructions, which, when executed by a computing device, perform the various possible implementations of the first aspect.

[0029] Fifthly, this application provides a computer program product containing instructions, characterized in that the computer program product stores instructions that, when executed by a computing device, cause the computing device to implement various possible implementations of the first aspect.

[0030] In a sixth aspect, embodiments of this application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the various possible implementations of the first aspect.

[0031] In a seventh aspect, embodiments of this application provide a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the various possible implementations of the first aspect.

[0032] Eighthly, this application provides a computer program product containing instructions, characterized in that the computer program product stores instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to implement various possible implementations of the first aspect. Attached Figure Description

[0033] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0034] Figure 1 is a schematic diagram of the structure of an autonomous driving system in the related technology;

[0035] Figure 2 is a schematic diagram of the structure of an autonomous driving system provided in an embodiment of this application;

[0036] Figure 3 is a schematic diagram of road network information in BEV map format provided in the embodiments of this application;

[0037] Figure 4 is a schematic diagram of the position of the 3D bounding box provided in the embodiment of this application;

[0038] Figure 5 is a schematic diagram of an image captured by a virtual camera provided in an embodiment of this application;

[0039] Figure 6 is a schematic diagram of a user using an autonomous driving system provided in an embodiment of this application;

[0040] Figure 7 is a flowchart illustrating a simulation method provided in an embodiment of this application;

[0041] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0042] Figure 9 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application;

[0043] Figure 10 is a schematic diagram of another computing device cluster architecture provided in the embodiments of this application. Detailed Implementation

[0044] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0045] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0046] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0047] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0048] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0049] Next, the technical solution provided in this application will be introduced.

[0050] Generally, when autonomous driving technology predicts trajectories, it can use a simulation module to simulate the environment around the vehicle, generating simulation data from various virtual sensors that can be recognized by the perception module. This makes the simulation data input to the perception module easier to perceive and makes the trajectory predicted by the planning and control module more accurate.

[0051] For example, as shown in Figure 1, the simulation module can construct a three-dimensional (3D) environment model of the vehicle's surroundings using data collected from high-precision maps, cameras, radars, and other sensors on the vehicle, or by having the user manually input a 3D environment model of the vehicle's surroundings.

[0052] Then, after receiving the vehicle's trajectory information from the planning and control module, the simulation module can predict the trajectories of other vehicles around the vehicle. The simulation module can use a physics engine to appropriately adjust the attitude, position, and other information in the predicted trajectories of other vehicles around the vehicle. The simulation module can also use a graphics engine to render the 3D environment model, simulating data from virtual cameras, radar, GPS, and IMU sensors. In other words, the simulation module can generate simulation data collected by virtual cameras and virtual radar based on the motion trajectories of the vehicle and other vehicles around it in the 3D environment model.

[0053] In this scheme, the simulation module needs to build a 3D environment model, which requires configuring a series of algorithms for building the 3D environment model, as well as additional configuration of the physics engine and graphics engine, resulting in a relatively complex structure for the simulation module. The realism of the simulation data output by the simulation module heavily depends on the realism of the 3D environment model, which is related to the number of training parameters and the rendering time of the model. If the number of training parameters is insufficient or the rendering time is short, the realism of the simulation data output by the simulation module will be relatively poor.

[0054] In view of this, this application provides an autonomous driving system in which the simulation module, in the process of generating simulation data from various virtual sensors, only needs to generate road network information and does not need to construct an environmental model around the vehicle, thus reducing the structural complexity of the simulation module. This autonomous driving system can utilize generation algorithms to directly simulate simulation data from various virtual sensors based on road network information, or road network information and the motion trajectory of at least one object around the vehicle. This not only improves the realism of the simulation data but also eliminates the need for a physics engine and graphics engine, further reducing the structural complexity of the simulation module.

[0055] Figure 2 is a schematic diagram of the structure of an autonomous driving system provided in an embodiment of this application. As shown in Figure 2, the autonomous driving system 200 may include a simulation module 210, a perception module 220, and a planning and control module 230.

[0056] The simulation module 210, upon receiving the input map and vehicle trajectory information, can simulate various virtual sensor data based on this information. The simulation module 210 can be divided into an environment simulation unit 211, a trajectory simulation unit 212, and a data simulation unit 213, based on its execution functions.

[0057] The environment simulation unit 211, trajectory simulation unit 212, and data simulation unit 213 can all be implemented in software or in hardware. For example, the implementation of the environment simulation unit 211 will be described below. Similarly, the implementation of the trajectory simulation unit 212 and the data simulation unit 213 can refer to the implementation of the environment simulation unit 211.

[0058] As an example of a software functional unit, the environment simulation unit 211 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the environment simulation unit 211 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0059] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0060] As an example of a hardware functional unit, the environment simulation unit 211 may include at least one computing device, such as a server. Alternatively, the environment simulation unit 211 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0061] The multiple computing devices included in the environment simulation unit 211 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the environment simulation unit 211 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the environment simulation unit 211 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0062] It should be noted that, in other embodiments, the environment simulation unit 211 can be used to execute any step in the simulation method, the trajectory simulation unit 212 can be used to execute any step in the simulation method, and the data simulation unit 213 can be used to execute any step in the simulation method. The steps implemented by the environment simulation unit 211, the trajectory simulation unit 212, and the data simulation unit 213 can be specified as needed. By implementing different steps in the simulation method through the environment simulation unit 211, the trajectory simulation unit 212, and the data simulation unit 213, all functions of the simulation module 210 can be realized.

[0063] The environment simulation unit 211 receives an input map in real time or periodically and generates road network information about the vehicle's surrounding environment based on the received input map. The input map is a map that includes information about the vehicle's surrounding environment and can be a map provided by a third-party map company. The input map can be a user-created custom map, such as one from RoadRunner. Alternatively, the input map can be a map automatically generated by a map-drawing component on the vehicle during driving, such as one from MetaDrive.

[0064] Road network information refers to road network data of a region or city, including detailed information such as road shape, connectivity, traffic signs, lane markings, and traffic signals. In this embodiment, the environment simulation unit 211 can use an open-source simulator (such as CARLA) to simulate the road shape, connectivity, traffic signs, lane markings, and traffic signals in the corresponding vehicle surrounding environment on the input map, obtaining simulated road information. This simulated road information is then used to construct the road network information of the vehicle's surrounding environment. Compared to generating 3D environment models, the process of generating road network information by the environment simulation unit 211 is simpler, requires fewer related algorithms, and makes the structure of the simulation module 210 simpler.

[0065] The road network information surrounding the vehicle can refer to the road network information of the surrounding environment within a defined area around the vehicle. Optionally, the defined area can be a circular area with the vehicle's position as the center and a defined length as the radius.

[0066] Road network information can be presented in the form of semantically meaningful scenarios such as bird's-eye view (BEV) maps and high-precision maps. For example, as shown in Figure 3, the road network information is presented in the form of a BEV map. The dark gray lines on the outer edges of this BEV map represent the curbs. The slightly lighter dark gray lines in the middle of the road represent lane lines. The light gray areas at road bends represent traffic signals.

[0067] The trajectory simulation unit 212 is used to obtain the motion trajectory information of at least one object around the vehicle during a set time period based on the road network information of the vehicle's surrounding environment and the vehicle's trajectory information.

[0068] The vehicle's trajectory information can be the vehicle's motion trajectory predicted by the planning and control module 230 beforehand, or it can be the motion trajectory calculated based on data collected by the vehicle's vision sensors, GPS, IMU, and other sensors. The vehicle's trajectory information can be data composed of time and location, data composed of a series of coordinate positions, or other types of data.

[0069] The set time period can be a period of time set by the user, or it can refer to the time period during which the autonomous driving system 200 generates control commands. That is, the set time period is greater than or equal to the time between when the simulation module 210 receives the input map and when the planning control module 230 outputs the control commands.

[0070] For example, when simulating the motion trajectory of an object, the trajectory simulation unit 212 can determine the number and type of objects to be simulated, based on preset conditions or prediction instructions input by the user, as objects surrounding the vehicle. Preset conditions may refer to randomly generating the number and type of objects to be simulated. Prediction instructions are actively input by the user, indicating the number and type of objects to be simulated selected by the user. The number of objects can be a positive integer greater than or equal to 0. The type of object can be vehicles, pedestrians, trees, or other objects that affect the normal driving of the vehicle.

[0071] If the number of objects is 0, the trajectory simulation unit 212 does not need to simulate the motion trajectory of the objects. If the number of objects is a positive integer greater than 0, the trajectory simulation unit 212 can use a motion prediction algorithm to predict the motion trajectory of at least one object around the vehicle within a set time period based on the vehicle's trajectory information and the road network information of the surrounding environment.

[0072] A motion trajectory is generally composed of different positions at different times, that is, a series of positions. Each position in the motion trajectory can be represented by a 3D bounding box or other methods that can describe the position. In this embodiment, the trajectory simulation unit 212 represents each position in the form of a 3D bounding box, so that when generating video, the position of the object is represented by a 3D bounding box, allowing the driver to perceive the position of the vehicle more intuitively.

[0073] In one embodiment, the location is represented by a 3D bounding box. As shown in Figure 4, each solid 3D bounding box represents the position of an object at a certain moment. Each dashed 3D bounding box represents the position of an object at the next moment. The relative orientation between the solid and dashed 3D bounding boxes of the same object represents the trajectory of the object between the two moments.

[0074] The data simulation unit 213 is used to generate simulation data of at least one virtual sensor of the vehicle during the first time period by using a generation algorithm, based on the road network information of the vehicle's surrounding environment, or based on the road network information of the vehicle's surrounding environment and the motion trajectory information of at least one object in the vehicle's surrounding environment during the first time period.

[0075] Virtual sensors can be virtual cameras, virtual radar, or other similar sensors. Taking a virtual camera as an example, the generation algorithm can be video generation technology. Video generation technology refers to a series of techniques capable of creating, editing, or enhancing video content for automated creation. Current state-of-the-art video generation technologies are primarily based on diffusion models. The combination of diffusion and transformer models demonstrates significant potential in video generation. The transformer model possesses powerful sequence modeling capabilities, providing strong support for the diffusion process, resulting in not only improved video quality but also the ability to handle longer video sequences.

[0076] For example, the process by which the data simulation unit 213 generates the simulated video captured by the virtual camera is as follows:

[0077] First, the data simulation unit 213 can design a diffusion model suitable for processing vehicle motion trajectories and road network information. Optionally, the data simulation unit 213 can train the neural network model using training data to obtain a video generation model, i.e., a diffusion model. The training data can include feature data and label data. The feature data is historical road network information training data. The label data is the video corresponding to the historical road network information training data. The historical road network information can be road network information generated by the environment simulation unit 211 before this moment, road network information generated by other vehicles before this moment, and other road network information.

[0078] Then, the data simulation unit 213 inputs the road network information of the vehicle's surrounding environment output by the environment simulation unit 211 and the motion trajectory information of at least one object around the vehicle during a set time period output by the trajectory simulation unit 212 into the diffusion model. The data simulation unit 213 can use a large amount of road network information and trajectory data of various objects around the vehicle to reason about the diffusion model, allowing the diffusion model to learn and generate video content that conforms to real road conditions.

[0079] Next, the data simulation unit 213 can start with random noise and have the diffusion model generate video frames of the motion trajectories of objects around the vehicle on the road.

[0080] Finally, the data simulation unit 213 can synthesize the generated video frames into a complete video sequence to obtain a preliminary simulation video. The data simulation unit 213 can use subjective and objective evaluation methods to detect the quality and realism of the preliminary simulation video, and then optimize the preliminary simulation video to obtain the final simulation video, thereby improving the quality and realism of the simulation video.

[0081] If the trajectory simulation unit 212 fails to simulate the motion trajectory of objects around the vehicle, the data simulation unit 213 can simulate a virtual video taken by a virtual camera without any objects, based on the road network information of the vehicle's surrounding environment. In this case, the process by which the data simulation unit 213 generates the virtual video taken by the virtual camera is similar to the process described above for simulating a virtual video taken by a virtual camera with objects, except that the input "motion trajectory information of at least one object around the vehicle and road network information of the vehicle's surrounding environment" is changed to "road network information of the vehicle's surrounding environment".

[0082] The simulation video generated by the data simulation unit 213 can have any angle, such as the front view of the vehicle, the rear view of the vehicle, the left view of the vehicle, the right view of the vehicle, etc. Taking the front view of the vehicle as an example, this view can be the shooting angle of a camera installed on the vehicle to capture the front view of the vehicle.

[0083] Assume the objects surrounding the vehicle include a car and two pedestrians. As shown in Figure 5, in a frame of a simulated video captured by a virtual camera, the car and pedestrians are positioned within solid 3D bounding boxes, representing their locations when that frame was captured. The dashed 3D bounding boxes represent their locations when the next frame is captured.

[0084] In this embodiment, the data simulation unit 213 can utilize generation algorithms, such as video generation technology based on a diffusion model, to directly simulate simulation data from various virtual sensors. Compared to using a 3D environment model, the simulation data generated by the data simulation unit 213 has higher realism. Moreover, the data simulation unit 213 only needs to use generation algorithms and does not require additional physics and graphics engines, which can further reduce the structural complexity of the simulation module 210.

[0085] The perception module 220 is used to identify and understand the simulation data of at least one virtual sensor of the vehicle during a set time period after receiving the simulation data of the vehicle during a set time period output by the simulation module 210, and to analyze the environmental information of the vehicle's surrounding environment during the set time period.

[0086] The planning and control module 230, upon receiving environmental information about the vehicle's surroundings during a set time period from the perception module 220, predicts the vehicle's trajectory based on this information. The planning and control module 230 then generates a series of control commands based on the vehicle's trajectory and sends these commands to the vehicle's various sensors, instructing the vehicle to follow the predicted trajectory. Finally, the planning and control module 230 sends the vehicle's trajectory to the trajectory simulation unit 212 in the simulation module 210, enabling the trajectory simulation unit 212 to predict the trajectories of objects around the vehicle during the next set time period.

[0087] In the aforementioned autonomous driving system 200, the simulation module 210, perception module 220, and planning and control module 230 can be deployed together. For example, all three modules of the autonomous driving system 200 can be deployed on the vehicle itself, along with a cloud server that communicates with the vehicle. Alternatively, the three modules of the autonomous driving system 200 can be deployed separately. For example, the simulation module 210 can be deployed on a cloud server that communicates with the vehicle, while the perception module 220 and planning and control module 230 can both be deployed on the vehicle. Another example is that the simulation module 210 and perception module 220 can both be deployed on a cloud server that communicates with the vehicle, while the planning and control module 230 can be deployed on the vehicle. And other possible scenarios exist.

[0088] It should be understood that the functional modules and components involved in the aforementioned autonomous driving system 200 can all be implemented through software or hardware, depending on the actual situation, and are not limited here. Furthermore, the functional modules and components involved in the aforementioned autonomous driving system 200 can be arranged individually or integrated, and are not limited here.

[0089] The above is a description of the autonomous driving system 200 provided in the embodiments of this application. It is understood that the autonomous driving system 200 can be configured on a cloud computing platform, for example, deployed on at least one instance such as a virtual machine or container, so that the cloud computing platform can provide model training services. Of course, the autonomous driving system 200 can also be configured on nodes other than the cloud computing platform, for example, deployed in at least one data center or on at least one server, depending on the actual situation, and is not limited here. The cloud computing platform can provide pages related to public cloud services for users to remotely access public cloud services. In this embodiment, users can pre-purchase the autonomous driving services provided by the autonomous driving system 200 on the cloud computing platform. For ease of understanding, the interaction between the user and the cloud computing platform is described below.

[0090] As shown in Figure 6, the interaction between the user and the cloud computing platform mainly includes: the user logs into the cloud computing platform 600 through a client (car) webpage, selects and purchases cloud services (i.e., autonomous driving services) related to the autonomous driving system 200 on the cloud computing platform 600, and after purchase, the user can generate the autonomous driving system 200 on the cloud computing platform 600 based on the functions provided by the autonomous driving service. The cloud computing platform 600 is primarily used to manage the infrastructure for running the autonomous driving service. For example, the infrastructure for the autonomous driving service may include multiple data centers located in different regions, each data center including multiple servers. Data centers can provide basic resources for the autonomous driving service, such as computing resources and storage resources. Therefore, when purchasing and using model training services, the user mainly pays for the resources used. When using the autonomous driving service, the user can input their autonomous driving requirements through the configuration interface, application program interface (API), or user interaction interface provided by the cloud computing platform 600. The cloud computing platform 600 can then generate an autonomous driving system matching the user's (or other software / hardware, etc.) input requirements.

[0091] Alternatively, some modules in the autonomous driving system 200 can be configured on the cloud side and others on the edge side, thereby achieving autonomous driving through edge-cloud collaboration. Furthermore, the autonomous driving system 200 can also be entirely configured on the edge side, depending on the specific circumstances; no limitation is made here.

[0092] The above is an introduction to the autonomous driving system provided in the embodiments of this application. Next, based on the above content, the simulation method provided in the embodiments of this application will be introduced.

[0093] For example, Figure 7 shows a schematic flowchart of a simulation method provided in an embodiment of this application. It can be understood that this simulation method can be executed by the simulation module 210 described above, and the specific implementation process is as follows:

[0094] Step S701: Obtain the input map of the simulation environment.

[0095] Step S702: Generate road network information for the simulation environment based on the input map.

[0096] The input map is a map that includes information about the vehicle's surrounding environment. It can be a map provided by a third-party map company or a custom map created manually by the user. The input map can also be a map automatically generated by the map drawing component on the vehicle during the vehicle's movement.

[0097] Specifically, the simulation module 210 can utilize an open-source simulator to simulate road information such as the shape, connectivity, traffic signs, lane markings, and traffic signals of roads in the vehicle's surrounding environment on the input map, thereby obtaining simulated road information. This simulated road information is then used to construct the road network information of the vehicle's surrounding environment. Compared to methods for generating 3D environment models, the process of generating road network information using the simulation module 210 is simpler, requiring fewer related algorithms, thus simplifying the structure of the simulation module 210.

[0098] The road network information surrounding the vehicle can refer to the road network information of the surrounding environment within a defined area around the vehicle. Optionally, the defined area can be a circular area with the vehicle's position as the center and a defined length as the radius.

[0099] Step S703: Based on the road network information and vehicle trajectory information, obtain the motion trajectory information of at least one object in the first time period.

[0100] The vehicle's trajectory information can be the vehicle's motion trajectory predicted by the planning and control module 230 beforehand, or it can be the motion trajectory calculated based on data collected by the vehicle's vision sensors, GPS, IMU, and other sensors.

[0101] The first time period, also known as the set time period mentioned above, can be a period of time set by the user or the time period during which the autonomous driving system 200 generates control commands. In other words, the set time period is greater than or equal to the time between the time the simulation module 210 receives the input map and the time the planning control module 230 outputs the control commands.

[0102] Specifically, when simulating the motion trajectory of objects, the simulation module 210 can determine the number and type of objects to be simulated, based on preset conditions or user-inputted prediction commands, as objects surrounding the vehicle. Preset conditions can refer to randomly generating the number and type of objects to be simulated. Prediction commands are actively input by the user, indicating the number and type of objects to be simulated selected by the user. The number of objects can be a positive integer greater than or equal to 0. The type of object can be vehicles, pedestrians, trees, or other objects that affect the normal driving of the vehicle.

[0103] If the number of objects is 0, the simulation module 210 does not need to simulate the motion trajectory of the objects. If the number of objects is a positive integer greater than 0, the simulation module 210 can use a motion prediction algorithm to predict the motion trajectory of at least one object around the vehicle within a set time period, based on the vehicle's trajectory information and the road network information of the surrounding environment.

[0104] A motion trajectory is generally composed of different positions at different times, that is, a series of positions. Each position in the motion trajectory can be represented by a 3D bounding box or other methods that can describe the position. The simulation module 210 represents each position in the form of a 3D bounding box, so that when generating video, the position of the object is represented by a 3D bounding box, allowing the driver to perceive the position of the vehicle more intuitively.

[0105] Step S704: During the first time period of performing the autonomous driving simulation task, simulation data of at least one virtual sensor of the vehicle during the first time period is obtained based on road network information, or based on road network information and the motion trajectory information of at least one object during the first time period.

[0106] Virtual sensors can be virtual cameras, virtual radar, or other similar sensors. Taking a virtual camera as an example, the generation algorithm can be video generation technology. Video generation technology refers to a series of techniques capable of creating, editing, or enhancing video content for automated creation. Currently, the latest video generation technologies are primarily based on diffusion models. The combination of diffusion and transformer models demonstrates significant potential in video generation. The transformer model possesses powerful sequence modeling capabilities, providing strong support for the diffusion process, resulting in not only improved video quality but also the ability to handle longer video sequences.

[0107] For example, taking a virtual camera as a virtual sensor. First, the simulation module 210 can design a diffusion model suitable for processing vehicle motion trajectories and road network information. Then, if the simulation module 210 does not generate motion trajectory information of at least one object around the vehicle in the first time period, the road network information of the surrounding environment can be input into the diffusion model for inference, allowing the diffusion model to learn and generate video content that conforms to real road conditions. If the simulation module 210 generates motion trajectory information of at least one object around the vehicle in the first time period, the road network information of the surrounding environment and the motion trajectory information of at least one object around the vehicle in the first time period can be input into the diffusion model for inference, allowing the diffusion model to learn and generate video content that conforms to real road conditions. Next, the simulation module 210 can start with random noise and allow the diffusion model to generate video frames without objects, or video frames that generate motion trajectories of objects around the vehicle on the road. Finally, the simulation module 210 can synthesize the generated video frames into a complete video sequence to obtain a preliminary simulation video. The simulation module 210 can use subjective and objective evaluation methods to detect the quality and realism of the preliminary simulation video, and then optimize the preliminary simulation video to obtain the final simulation video, thereby improving the quality and realism of the simulation video.

[0108] In this embodiment, the simulation module 210 only needs to generate road network information during the generation of simulation data from various virtual sensors, without needing to construct an environmental model around the vehicle, thus reducing the structural complexity of the simulation module. Furthermore, the simulation module 210 can directly simulate the simulation data from various virtual sensors based on road network information, or road network information and vehicle trajectory information. This not only improves the realism of the simulation data but also eliminates the need for a physics engine and graphics engine, further reducing the structural complexity of the simulation module.

[0109] This application provides a simulation device that, based on its execution functions, is divided into an environment simulation unit 211, a trajectory simulation unit 212, and a data simulation unit 213, as shown in Figure 2. The simulation device can be an application program, software code, etc., deployed on a computing device or a cluster of computing devices. The computing device can be a server, computer, laptop, tablet, smartphone, etc. The simulation device can be deployed on a cloud server. If the simulation device is deployed on a cloud service, designers can use local devices to call the cloud server to use the simulation device to complete corresponding tasks. The specific implementation process of the simulation device is as follows:

[0110] The environment simulation unit 211 is used to acquire the input map of the simulation environment. The environment simulation unit 211 is also used to generate road network information of the simulation environment based on the input map; the data simulation unit 213 is used to simulate and obtain simulation data from at least one virtual sensor of the vehicle during the first time period of executing the autonomous driving simulation task, based on the road network information; the perception module is used to analyze the environmental information of the vehicle's surrounding environment during the first time period based on the simulation data; the planning control module is used to analyze the control commands of the vehicle during the first time period based on the environmental information. The control commands are used to control the driving behavior of the vehicle. The surrounding environment includes at least one object, and the environmental information includes the position of at least one object.

[0111] In one embodiment, the trajectory simulation unit 212 is used to determine at least one object in the surrounding environment of the vehicle during a first time period; and to obtain the motion trajectory information of at least one object during the first time period based on road network information and the vehicle's trajectory information; the data simulation unit 213 is specifically used to simulate and obtain simulation data of at least one virtual sensor of the vehicle during the first time period based on road network information and the motion trajectory information of at least one object during the first time period when performing an autonomous driving simulation task.

[0112] In one embodiment, the trajectory simulation unit 212 is specifically used to determine the number of objects in the vehicle's surrounding environment and the types of objects in the vehicle's surrounding environment during the first time period, based on preset conditions and / or prediction instructions input by the user.

[0113] In one implementation, the motion trajectory information of at least one object in the first time period is represented by a three-dimensional bounding box.

[0114] In one embodiment, the environment simulation unit 211 is specifically used to input the input map into the simulator to obtain the road network information of the vehicle's surrounding environment in the first time period. The road network information includes one or more of the following: road shape, road connection relationship, traffic signs, lane markings, and traffic signals.

[0115] In one embodiment, when at least one virtual sensor includes a virtual camera, the data simulation unit 213 is specifically used to input road network information into a video generation model to obtain a video of the vehicle's surrounding environment in the first time period. The video generation model is a neural network model, and the simulation data includes video.

[0116] In one implementation, before inputting road network information into the video generation model to obtain a video of the vehicle's surrounding environment in the first time period, the data simulation unit 213 is also used to train a basic neural network model using training data to obtain a video generation model. The training data includes feature data and label data. The feature data is road network information training data, and the label data is the video corresponding to the road network information training data.

[0117] Figure 8 is a schematic diagram of a computing device provided in an embodiment of this application. As shown in Figure 8, the computing device 800 includes a bus 810, a processor 820, a memory 830, and a communication interface 840. The processor 820, the memory 830, and the communication interface 840 communicate with each other via the bus 810. The computing device 800 can be a server, computer, laptop, server rack, etc. It should be understood that this application does not limit the number of processors and memories in the computing device 800.

[0118] Bus 810 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 8, but this does not imply that there is only one bus or one type of bus. Bus 810 can include pathways for transmitting information between various components of computing device 800 (e.g., processor 820, memory 830, communication interface 840).

[0119] The processor 820 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0120] The memory 830 may include volatile memory, such as random access memory (RAM). The memory 830 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0121] The memory 830 stores executable program code, which the processor 820 executes to implement the functions of the aforementioned modules, such as the environment simulation unit 211, the trajectory simulation unit 212, and the data simulation unit 213, thereby realizing the simulation method. In other words, the memory 830 stores instructions for executing the simulation method.

[0122] Alternatively, the memory 830 stores executable code, which the processor 820 executes to implement the functions of the aforementioned modules, thereby realizing the simulation method. That is, the memory 830 stores instructions for executing the simulation method.

[0123] The communication interface 840 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 800 and other devices or communication networks.

[0124] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0125] As shown in Figure 9, the computing device cluster includes at least one computing device 800. The memory 830 of one or more computing devices 800 in the computing device cluster may store the same instructions for executing the simulation method.

[0126] In some possible implementations, the memory 830 of one or more computing devices 800 in the computing device cluster may also store a portion of the instructions for executing the simulation method. In other words, a combination of one or more computing devices 100 can jointly execute the instructions for executing the simulation method.

[0127] It should be noted that the memory 830 in different computing devices 800 within the computing device cluster can store different instructions, which are used to execute some of the functions of the aforementioned environment simulation unit 211, trajectory simulation unit 212, and data simulation unit 213. That is, the instructions stored in the memory 830 of different computing devices 800 can implement the functions of one or more modules in the aforementioned environment simulation unit 211, trajectory simulation unit 212, and data simulation unit 213.

[0128] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 illustrates one possible implementation. As shown in Figure 10, two computing devices, computing device 800A and computing device 800B, are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 830 in computing device 800A stores instructions for performing the functions of some modules in the aforementioned environment simulation unit 211, trajectory simulation unit 212, and data simulation unit 213. Simultaneously, the memory 830 in computing device 800B stores instructions for performing the functions of another portion of the aforementioned environment simulation unit 211, trajectory simulation unit 212, and data simulation unit 213.

[0129] The connection method between the computing device clusters shown in Figure 10 can be considered as follows: taking into account that the simulation method provided in this application requires a large amount of data storage, it is considered that the functions implemented by another part of the modules in the above-mentioned environment simulation unit 211, trajectory simulation unit 212 and data simulation unit 213 are handed over to the computing device 800B for execution.

[0130] It should be understood that the functions of computing device 800A shown in Figure 10 can also be performed by multiple computing devices 800. Similarly, the functions of computing device 800B can also be performed by multiple computing devices 800.

[0131] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method of the computing device cluster described in Figures 8 and 9. The difference is that the memory 830 of one or more computing devices 800 in this computing device cluster can store the same instructions for executing the simulation method.

[0132] In some possible implementations, the memory 830 of one or more computing devices 800 in the computing device cluster may also store a portion of the instructions for executing the simulation method. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the simulation method.

[0133] It should be noted that the memory 830 in different computing devices 800 within the computing device cluster can store different instructions to execute some functions of the computing device 800. That is, the instructions stored in the memory 830 of different computing devices 800 can implement the functions of one or more modules in the aforementioned environment simulation unit 211, trajectory simulation unit 212, and data simulation unit 213.

[0134] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a simulation method.

[0135] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a simulation method.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A simulation method, characterized by, The method is applied to a driving simulation platform for performing a driving simulation task, the driving simulation task including a driving task of at least one vehicle in a simulation environment, and the method includes: obtaining an input map of the simulation environment; generating road network information of the simulation environment according to the input map; during a first period of performing the automatic driving simulation task, simulating simulation data of at least one virtual sensor of the vehicle in the first period according to the road network information; the simulation data is used to analyze environmental information of the surrounding environment of the vehicle in the first period, the environmental information is used to analyze control instructions of the vehicle in the first period, the control instructions are used to control the driving behavior of the vehicle, and the surrounding environment includes at least one object, and the environmental information includes the position of the at least one object.

2. The method of claim 1, wherein, Before the step of, during the first period of performing the automatic driving simulation task, simulating simulation data of at least one virtual sensor of the vehicle in the first period according to the road network information, the method further includes: determining at least one object in the surrounding environment of the vehicle in the first period; obtaining motion trajectory information of the at least one object in the first period according to the road network information and trajectory information of the vehicle; The step of, during the first period of performing the automatic driving simulation task, simulating simulation data of at least one virtual sensor of the vehicle in the first period according to the road network information, specifically includes: during the first period of performing the automatic driving simulation task, simulating simulation data of at least one virtual sensor of the vehicle in the first period according to the road network information and the motion trajectory information of the at least one object in the first period.

3. The method of claim 2, wherein, The step of determining at least one object in the surrounding environment of the vehicle in the first period specifically includes: determining the number of objects in the surrounding environment of the vehicle in the first period and the type of objects in the surrounding environment of the vehicle in the first period according to a preset condition and / or a prediction instruction input by a user.

4. The method according to claim 2 or 3, characterized in that, The motion trajectory information of the at least one object in the first period is represented by a three-dimensional bounding box.

5. The method according to any one of claims 1 to 4, characterized in that, The step of generating road network information of the simulation environment according to the input map specifically includes: inputting the input map into a simulator to obtain road network information of the surrounding environment of the vehicle in the first period, the road network information including one or more of the shape of the road, the connection relationship of the road, the traffic sign, the lane marking and the traffic signal.

6. The method according to any one of claims 1 to 5, characterized in that, When the at least one virtual sensor includes a virtual camera, The step of, during the first period of performing the automatic driving simulation task, simulating simulation data of at least one virtual sensor of the vehicle in the first period according to the road network information, specifically includes: inputting the road network information into a video generation model to obtain a video of the surrounding environment of the vehicle in the first period, the video generation model being a neural network model, and the simulation data including the video.

7. The method of claim 6, wherein, Before the road network information is input into the video generation model to obtain a video of the surrounding environment of the vehicle in the first time period, the method further comprises: training a basic neural network model using training data to obtain the video generation model, the training data comprising feature data and label data, the feature data being road network information training data, and the label data being a video corresponding to the road network information training data.

8. An emulation apparatus, characterized by comprising: comprise: an environment simulation unit, configured to obtain an input map of a simulation environment; the environment simulation unit is further configured to generate road network information of the simulation environment according to the input map; a data simulation unit, configured to simulate, in a first time period in which the automatic driving simulation task is performed, simulation data of at least one virtual sensor of the vehicle in the first time period according to the road network information; a perception module, configured to analyze environmental information of the surrounding environment of the vehicle in the first time period according to the simulation data; and a planning and control module, configured to analyze a control instruction of the vehicle in the first time period according to the environmental information, the control instruction being used to control a driving behavior of the vehicle, the surrounding environment comprising at least one object, and the environmental information comprising a position of the at least one object.

9. The apparatus of claim 8, wherein, Further comprise a trajectory simulation unit, the trajectory simulation unit is configured to determine at least one object in the surrounding environment of the vehicle in the first time period; and obtain motion trajectory information of the at least one object in the first time period according to the road network information and trajectory information of the vehicle; the data simulation unit is specifically configured to simulate, in the first time period in which the automatic driving simulation task is performed, the simulation data of the at least one virtual sensor of the vehicle in the first time period according to the road network information and the motion trajectory information of the at least one object in the first time period.

10. The apparatus of claim 9, wherein the trajectory simulation unit is specifically configured to determine a number of objects in the surrounding environment of the vehicle in the first time period and a type of the objects in the surrounding environment of the vehicle in the first time period according to a preset condition and / or a prediction instruction input by a user.

11. The apparatus of claim 9 or 10, wherein, The motion trajectory information of the at least one object in the first time period is represented in a three-dimensional bounding box.

12. The apparatus of any one of claims 8-11, wherein the environment simulation unit is specifically configured to input the input map into a simulator to obtain the road network information of the surrounding environment of the vehicle in the first time period, the road network information comprising one or more of a shape of a road, a connection relationship of the road, a traffic sign, a lane marking, and a traffic signal.

13. The apparatus of any of claims 8-12, wherein, When the at least one virtual sensor comprises a virtual camera, the data simulation unit is specifically configured to input the road network information into a video generation model to obtain a video of the surrounding environment of the vehicle in the first time period, the video generation model being a neural network model, and the simulation data comprising the video.

14. The apparatus of claim 13, wherein The data simulation unit, before the inputting of the road network information into the video generation model to obtain the video of the surrounding environment of the vehicle in the first time period, is further configured to training a basic neural network model using training data to obtain the video generation model, the training data comprising feature data and label data, the feature data being road network information training data, and the label data being a video corresponding to the road network information training data.

15. A cluster of computing devices, characterized in that, comprising: at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method of any one of claims 1-7.

16. A computer readable storage medium characterized by: comprising computer program instructions that, when executed by a computing device, cause the computing device to perform the method of any one of claims 1-7.

17. A computer program product comprising instructions, characterized in that, the computer program product stores instructions that, when executed by a computing device, cause the computing device to implement the method of any one of claims 1-7.

Images