Automatic driving real scene simulator based on three-dimensional Gaussian sputtering and simulation method

Abstract

The invention provides an automatic driving live-action simulator based on three-dimensional Gaussian sputtering, and the simulator comprises a general model interface module which achieves the integration of different Gaussian models, sensor models and vehicle kinematics models through providing a standardized interface, and obtains basic elements and self-vehicle elements in a simulation scene; the internal scene representation module is used for defining and organizing the basic elements in a structured mode to obtain environment elements; the collision detection module is used for distinguishing foreground objects and backgrounds of the environment elements, respectively carrying out foreground collision detection and background collision detection, and carrying out collision judgment on the self-vehicle elements and the environment elements in the simulation scene; and the simulation control module is used for performing dynamic response of the simulator to the external control signal based on the internal scene representation module and the collision detection module. The method can meet the requirements of the automatic driving algorithm for massive high-quality data, and is suitable for training and testing of the automatic driving algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving simulation, in particular to a high-fidelity multi-modal general automatic driving real scene simulator based on three-dimensional Gaussian splatting (3DGS) and a simulation method. BACKGROUND

[0002] With the rapid development of automatic driving technology, the demand for large-scale and high-quality training and testing data of algorithms is increasingly urgent. Simulation technology, with the advantages of low cost, strong repeatability, controllable scene, etc., has become a key means to generate massive data and plays an important role in the training, testing and verification process of automatic driving algorithms.

[0003] In the aspect of simulation scene construction, the existing technology mainly falls into two categories: one is the traditional manual modeling method, which builds a virtual environment by manually designing scene geometry and specifying textures. However, this method not only requires a large amount of manpower, but also the constructed scene has certain gaps with the real world, which is difficult to meet the demand of high-fidelity simulation; the other is to reconstruct a virtual environment from the real world based on three-dimensional reconstruction technology, which can significantly improve the authenticity of simulation data. In recent years, three-dimensional Gaussian splatting (3DGS) technology has attracted widespread attention in the field of three-dimensional scene reconstruction due to its efficient real-time rendering capability and high-quality visual fidelity, providing a new technical path for automatic driving simulation.

[0004] However, the existing simulation based on 3DGS still has some deficiencies, for example: the support for data generation of non-visual sensors such as laser radar and millimeter wave radar is insufficient, which cannot meet the testing needs of automatic driving algorithms for multi-source perception data; most frameworks can only realize open-loop simulation based on original trajectories, and cannot respond to control signals output by automatic driving algorithms, lacking closed-loop capability; there is a lack of collision detection method for foreground objects and background environment, which cannot simulate the real physical interaction process; the compatibility of different improved GS algorithms, sensor models and vehicle kinematics models is poor, and users cannot customize and integrate modules according to their needs.

[0005] Therefore, in view of the above-mentioned defects of the existing automatic driving simulation framework based on 3DGS technology, it is urgent to construct a multi-modal expandable general closed-loop simulation architecture to provide a more efficient testing tool for algorithm development. At present, no similar technology or report has been found, and no similar data has been collected at home and abroad. SUMMARY

[0006] The present application provides an automatic driving real scene simulator based on three-dimensional Gaussian splatting and a simulation method to solve the above-mentioned problems in the prior art.

[0007] According to one aspect of the present application, an automatic driving real scene simulator based on three-dimensional Gaussian sputtering is provided, comprising:

[0008] A general model interface module, which realizes the integration of different Gaussian models, sensor models and vehicle kinematics models by providing a standardized interface, and obtains basic elements and ego elements in a simulation scene;

[0009] An internal scene representation module, which defines and organizes the basic elements in a structured manner to obtain environmental elements;

[0010] A collision detection module, which is used to distinguish foreground objects from backgrounds of the environmental elements, and respectively performs foreground collision detection and background collision detection to determine collisions between ego elements and environmental elements in the simulation scene;

[0011] A simulation control module, which performs dynamic response of the simulator to external control signals based on the internal scene representation module and the collision detection module, and realizes end-to-end test control of automatic driving.

[0012] Preferably, the simulation control module comprises:

[0013] A model loading unit, which loads the structured representation of the basic elements in the simulation scene based on the internal scene representation module, and initializes the initial pose [x0, y0, θ0] and velocity v0 of the ego element, wherein the pose and the velocity constitute an ego state vector x(t) = [x, y, θ, v]. T T ;

[0014] A simulation input unit, which generates multi-modal perception data as perception input of the automatic driving algorithm by a rendering method according to the current ego state x(t), sensor internal and external parameters and the structured organization relationship of the basic elements in the simulation scene;

[0015] A control unit, which is used to obtain an external control quantity u(t);

[0016] A state calculation unit, which calculates the ego state at the next time x(t+Δt) = f(x(t), u(t), Δt) by a vehicle kinematics model; in the formula, f(·) is a preset vehicle kinematics model, the input is the current state x(t), the control quantity u(t) and the time step Δt, and the output is the updated pose [x+Δx, y+Δy, θ+Δθ] and velocity v+Δv; T

[0017] A simulation output unit, which calls the collision detection module, and if a collision is determined, the simulation is terminated and a warning is output; otherwise, the simulation input unit is returned and the next round of loop is entered. ​​

[0018] Preferably, the control unit, the external control quantity u(t) is obtained by any one of the following ways:

[0019] An external controller is adopted to output the control quantity u(t) based on the perception data, forming a closed-loop control mode;

[0020] Based on the recorded historical trajectory, the ego motion is reproduced to output the control quantity u(t), forming an open-loop control mode;

[0021] According to another aspect of the present application, an automatic driving real scene simulation method based on three-dimensional Gaussian sputtering is provided, comprising:

[0022] A preset Gaussian model, a sensor model and a vehicle kinematics model are obtained as basic elements and ego elements in a simulation scene;

[0023] The basic elements in the simulation scene are defined and organized in a structured manner to obtain environment elements;

[0024] According to the current ego state, the sensor internal and external parameters and the structured organization relationship of the basic elements in the simulation scene, multi-modal perception data are generated as the perception input of the automatic driving algorithm;

[0025] The control quantity is obtained, and the next time ego state is calculated through the vehicle kinematics model based on the current ego state and the control quantity;

[0026] The environment elements are distinguished into foreground objects and background regions, and foreground collision detection and background collision detection are respectively performed, and collision judgment is performed on the ego elements and the environment elements in the simulation scene; if collision is judged, the simulation is terminated and a warning is output; otherwise, the step of perception input is returned to enter the next round of circulation.

[0027] In some preferred embodiments, the above method further comprises:

[0028] The external control quantity u(t) is obtained by any one of the following ways:

[0029] An external controller is adopted to output the control quantity u(t) based on the perception data, forming a closed-loop control mode;

[0030] Based on the recorded historical trajectory, the ego motion is reproduced to output the control quantity u(t), forming an open-loop control mode.

[0031] Compared with the prior art, the present application has at least one of the following beneficial effects due to the adoption of the above technical solutions:

[0032] The application provides an automatic driving real scene simulator and a simulation method based on three-dimensional Gaussian sputtering, and realizes an automatic driving real scene simulation architecture based on Gaussian sputtering.

[0033] The automatic driving real scene simulator and the simulation method based on three-dimensional Gaussian sputtering provided by the application provide various module interfaces such as a rendering model interface, a vehicle model interface, a collision detection algorithm interface and a sensor model interface, thereby providing an extensible general simulation framework, and facilitating integration and development of custom modules by users.

[0034] The automatic driving real scene simulator and the simulation method based on three-dimensional Gaussian sputtering provided by the application design a visual operation interface including functions such as path visualization, a log window, multi-modal perception data display and parameter configuration, effectively reduce the use threshold, and improve the ease of use of the simulation technology.

[0035] The automatic driving real scene simulator and the simulation method based on three-dimensional Gaussian sputtering provided by the application can realize large-scale data generation through the general rendering interface and the structured scene representation of the simulator, can meet the demand of automatic driving algorithm research and development for massive high-quality data, and is suitable for training and testing of automatic driving algorithms. BRIEF DESCRIPTION OF DRAWINGS

[0036] Other characteristics, objects and advantages of the application will become more apparent through reading the following detailed description of non-restrictive embodiments with reference to the accompanying drawings:

[0037] Figure 1 The figure is a working architecture diagram of the automatic driving real scene simulator based on three-dimensional Gaussian sputtering in an embodiment of the application.

[0038] Figure 2 The figure is a coordinate system relationship diagram in a typical automatic driving scene in a preferred embodiment of the application.

[0039] Figure 3 The figure is a collision detection flowchart in a preferred embodiment of the application.

[0040] Figure 4 The figure is a general model interface diagram in a preferred embodiment of the application.

[0041] Figure 5 The figure is a visual interface diagram in a preferred embodiment of the application.

[0042] Figure 6A closed-loop simulation architecture diagram of an automatic driving real scene simulator based on three-dimensional Gaussian sputtering in a preferred embodiment of the present application.

[0043] Figure 7 A simulation method workflow diagram of an automatic driving real scene simulator based on three-dimensional Gaussian sputtering in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The embodiments of the present application are described in detail below: The embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation manners and specific operation processes are given. It should be pointed out that, for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

[0045] The existing simulation based on 3DGS lacks a unified evaluation framework, and has serious deficiencies in multi-modal support and scalability. In view of the above problems, an embodiment of the present application provides an automatic driving real scene simulator based on three-dimensional Gaussian sputtering, which realizes a multi-modal scalable general closed-loop simulation architecture.

[0046] Specifically, as shown in Figure 1 , the automatic driving real scene simulator based on three-dimensional Gaussian sputtering provided by the embodiment can include:

[0047] A general model interface module, which realizes the integration of different Gaussian models, sensor models and vehicle kinematics models by providing a standardized interface, and obtains basic elements and ego elements in a simulation scene;

[0048] An internal scene representation module, which defines and organizes the basic elements in a structured manner to obtain environmental elements;

[0049] A collision detection module, which is used to distinguish foreground objects and backgrounds of the environmental elements, and respectively performs foreground collision detection and background collision detection, and collision determination of the ego elements and the environmental elements in the simulation scene;

[0050] A simulation control module, which performs dynamic response of the simulator to external control signals based on the internal scene representation module and the collision detection module, and realizes end-to-end test control of automatic driving.

[0051] In some preferred embodiments, as shown in Figure 4 , the above general model interface module can further include a rendering model interface, a sensor interface and a vehicle model interface; wherein:

[0052] A rendering model interface is used to load a Gaussian model and a custom renderer matched with the Gaussian model, and to provide a renderable element of a basic element in a simulation scene; wherein the rendering model interface comprises a model information class, a model loader and a custom renderer; the model information class is used to store related parameters of a Gaussian sputtering algorithm model, including basic parameters such as a center point position (mean value), a scaling coefficient, a rotation matrix, a color and an opacity; for different Gaussian sputtering algorithm models, specific extension parameters can also be included, and the specific parameters depend on the algorithm model; the model loader is used to read a Gaussian model file trained by a user, and to parse the Gaussian model file into corresponding model information, so as to realize conversion from an external model file to an internal scene representation of the simulator; the custom renderer is used to access a custom renderer matched with the Gaussian sputtering algorithm model information after the model information is loaded, so as to complete a rendering process.

[0053] A sensor model interface is used to add a sensor model, and to provide a sensor element of a basic element in a simulation scene. The sensor model comprises a camera, a lidar and a millimeter wave radar, and corresponding information classes; wherein the camera information class comprises related parameters of a pinhole camera model, mainly including internal and external parameters, and a unique identifier for reading of multiple cameras; for some time-varying Gaussian sputtering algorithms, timestamp information is also included; the lidar or millimeter wave radar information class comprises a horizontal field of view, a horizontal resolution, a vertical field of view and a vertical resolution; a user can also select a virtual sensor to visualize a lidar or millimeter wave radar point cloud in a data set, so as to perfect the entire rendering process.

[0054] A vehicle model interface is used to load different vehicle kinematic models, to adapt to different vehicle types, and to provide a self-vehicle element in a simulation scene.

[0055] In some preferred embodiments, the above-mentioned general model interface module can further comprise:

[0056] A collision detection algorithm interface is used to load a preset collision detection algorithm model, and to provide the collision detection algorithm model to a collision detection module; the collision detection module can directly apply the preset algorithm model, or can perform collision detection through each functional unit of the collision detection model.

[0057] In some preferred embodiments, the above-mentioned internal scene representation module, which defines and organizes the basic elements in a structured manner and performs rendering, can further comprise:

[0058] The basic elements comprise renderable elements and sensor elements; wherein the renderable elements comprise a background S and a foreground object V k , both of which are represented by Gaussian models; a basic parameter set of each Gaussian body in the Gaussian model is G i, including mean, scaling matrix, rotation matrix, color and opacity; the parameters in the parameter set describing the pose are defined in the local coordinate system of the element itself; thus, for a renderable element, the attribute set is represented as:

[0059] BACKGROUND where N is the number of background Gaussians, is the coordinate transformation matrix of the background local coordinate system to its parent node P;

[0060] Foreground object: where M k is the number of Gaussians of the kth foreground object, is the coordinate transformation matrix of the foreground local coordinate system to its parent node P;

[0061] Sensor element includes camera, lidar and millimeter wave radar, and the attributes include unique identifier id, intrinsic parameters and coordinate transformation matrix of the local coordinate system to its parent node, and the attribute set is represented as:

[0062] Camera: where is the camera intrinsic parameter, is the coordinate transformation matrix of the camera coordinate system to its parent node P;

[0063] Lidar: where is the lidar parameter, is the coordinate transformation matrix of the lidar coordinate system to its parent node P;

[0064] Millimeter wave radar: where is the millimeter wave radar parameter, is the coordinate transformation matrix of the millimeter wave radar coordinate system to its parent node P;

[0065] The transformation matrix of the base element from the world coordinate W to the canonical coordinate W0 is:

[0066]

[0067] where is the transformation matrix of the base element from the world coordinate W to the canonical coordinate W0, λ is the normalized scaling factor, is the transformation matrix without scaling; the actual representation of the renderable element and the sensor element includes the canonical transformation matrix, that is, the structured representation of the base element is obtained as:

[0068] BACKGROUND

[0069] Foreground object:

[0070] Camera:

[0071] Lidar:

[0072] Millimeter-wave radar:

[0073] When rendering, the sensor elements and the foreground objects are transformed to a unified canonical coordinate system, then:

[0074] For the sensor elements, the extrinsic parameters in the canonical coordinate system are:

[0075]

[0076] The attribute set of the vehicle in the foreground object in the canonical coordinate system is represented as:

[0077]

[0078] In the formula, is the original Gaussian parameter set G i In which, the mean and the rotation matrix become:

[0079]

[0080] to form a new parameter set;

[0081] Based on the structured representation of the above basic elements, the rendered new view image or point cloud is represented as:

[0082] X = C m , L n , R l

[0083] wherein, is the rendered image or point cloud, K X is the intrinsic parameter of the sensor X, is the transformation matrix of the sensor X to the canonical world coordinate system W0, X T(X = C m , L n , R l ) is the manual adjustment of the user to the sensor element, that is, the pose transformation of the sensor element in its own coordinate system.

[0084] In some preferred embodiments, as shown in Figure 3 the above collision detection module distinguishes the foreground objects and the background of the environment elements, and performs foreground collision detection and background collision detection respectively, and can further include: a foreground collision detection unit and a background collision detection unit; wherein:

[0085] The foreground collision detection unit comprises: for collision detection of the ego element V1 and other foreground objects V k , a bounding box generation two-step strategy combined with the separation axis theorem is adopted; wherein:

[0086] The bounding box generation subunit, for a foreground object composed of N Gaussian bodies, each Gaussian body parameter includes mean μ i , opacity α i and covariance Σ i , wherein α i satisfies A directed bounding box is generated by the following sub-modules:

[0087] The Gaussian center calculation submodule is used to calculate the overall center of the Gaussian model is:

[0088] The covariance calculation submodule calculates the overall covariance matrix Σ as:

[0089] The eigenvalue decomposition submodule is used to perform eigenvalue decomposition on the overall covariance matrix Σ to obtain the orthogonal principal direction {u1, u2, u3} as the bounding box principal axis;

[0090] The bounding box edge length calculation submodule is used to calculate the bounding box edge length L j in each principal axis direction as: In the formula, k is an adjustable parameter;

[0091] The bounding box center calculation submodule is used to calculate the center c of the bounding box in the original coordinate system as:

[0092] The separation axis theorem judgment subunit judges the foreground collision through the separation axis theorem, comprising:

[0093] For the bounding boxes of two foreground objects, whether there is a collision is judged by checking 15 potential separation axes; if there is an axis, the projection intervals of the two boxes on the axis have no overlap, then there is no collision; otherwise, it is determined as a collision;

[0094] The background collision detection unit comprises: for collision detection of the ego element and the background, a composite detection strategy combining Gaussian body probability and laser radar point cloud is adopted; wherein:

[0095] The collision risk index R of the Gaussian body and the laser radar is defined, R ∈ [0, 1], when R→1, it indicates that a collision occurs.

[0096] In some preferred embodiments, the collision risk indicator includes: Gaussian body-based collision risk, laser radar point cloud-based collision risk, and fusion risk. Among them:

[0097] The Gaussian body-based collision risk includes:

[0098] The collision risk indicator within the ego vehicle bounding box Ω is defined as:

[0099]

[0100] In the formula, V is the volume of the bounding box, M is the number of Monte Carlo sampling points, k G is a sensitivity parameter; a detection area Ω0 slightly larger than the area enclosed by the ego vehicle bounding box Ω is selected, and only the Gaussian body with the center point falling into Ω0 is subjected to collision detection;

[0101] The laser radar point cloud-based collision risk includes:

[0102] After ground point removal of the laser radar point cloud, the number of points falling into the ego vehicle bounding box Ω N L is counted, and the risk indicator is defined as:

[0103] R L = 1-exp(-k L N L )

[0104] In the formula, k L is a sensitivity parameter;

[0105] The fusion risk includes:

[0106] The basic radar risk weight w0 ∈ [0, 1] is selected, and the Gaussian body risk weight is 1-w0; the actual fusion weight of R G and R L is defined as:

[0107]

[0108] Then the final comprehensive risk value is:

[0109] R F = w L R L + w G R G

[0110] When R G ≈ R L , the original defined basic weight value is maintained; and when the difference between R G and R L tends to 1, the detection value of the laser radar is preferred.

[0111] In some preferred embodiments, the simulation control module, based on the internal scene representation module and the collision detection module, performs dynamic response of the simulator to external control signals, and can further comprise:

[0112] a model loading unit, which loads the structured representation of the basic elements in the simulation scene based on the internal scene representation module, initializes the initial pose [x0, y0, θ0] and velocity v0 of the ego vehicle element, wherein the pose and the velocity constitute the ego vehicle state vector x(t) = [x, y, θ, v] T ; T ;

[0113] a simulation input unit, which generates multi-modal perception data as the perception input of the autonomous driving algorithm through a rendering method according to the current ego vehicle state x(t), the internal and external parameters of the sensor, and the structured organization relationship of the basic elements in the simulation scene;

[0114] a control unit, which is configured to obtain an external control quantity u(t);

[0115] a state calculation unit, which calculates the ego vehicle state at the next time point x(t+Δt) = f(x(t), u(t), Δt) through a vehicle kinematics model; in the formula, f(·) is a preset vehicle kinematics model, the input is the current state x(t), the control quantity u(t) and the time step Δt, and the output is the updated pose [x+Δx, y+Δy, θ+Δθ] T and velocity v+Δv;

[0116] a simulation output unit, which calls the collision detection module, and if a collision is determined, the simulation is terminated and a warning is output; otherwise, the simulation input unit is returned and the next round of loop is entered.

[0117] In some preferred embodiments, the control unit can obtain the external control quantity u(t) in any of the following ways:

[0118] an external controller is used to output the control quantity u(t) based on the perception data, forming a closed-loop control mode; the closed-loop control loop of the simulation control module is as shown in Figure 6 ;

[0119] the ego vehicle motion is reproduced based on the recorded historical trajectory, and the control quantity u(t) is output, forming an open-loop control mode.

[0120] In some preferred embodiments, the simulation control module can further comprise:

[0121] a clock unit, which is driven by a global timer and controls the simulation loop time in a fixed time step Δt.

[0122] In some preferred embodiments, the simulator described above can further comprise:

[0123] a visual operation interface module that realizes human-computer interaction by providing a visual interface.

[0124] In some preferred embodiments, as shown in Figure 5 the visual operation interface module described above can further comprise:

[0125] a path visualization unit for visualizing the trajectory of the vehicle;

[0126] a log window unit for printing simulation information, including the total simulation time at the end of the simulation and collision warning information for collisions;

[0127] an image visualization window unit for visualizing the rendering results of the camera images;

[0128] a point cloud visualization window unit for visualizing the bird's eye view of the laser radar or millimeter wave radar point cloud rendering results;

[0129] a model setting interface unit for selecting and setting simulation parameters, including loading model type selection, loading model path selection, simulation step setting, simulation mode setting, and simulator generated data output setting;

[0130] a sensor pose adjustment interface unit for adjusting the layout and configuration of sensor elements;

[0131] a other setting interface unit for other selections and configurations, including collision detection type selection, point cloud bird's eye view range adjustment, laser radar ground point removal, and virtual sensor loading.

[0132] The automatic driving real scene simulator based on three-dimensional Gaussian sputtering provided by the above embodiments of the present application is used for low-cost and high-fidelity large-scale data generation in automatic driving algorithm research, wherein: the universal interface and rendering process of the simulator support multi-modal data generation; since the poses of the scene elements are dynamically controlled by the transformation matrix, on the one hand, the user can manually adjust the sensor pose to generate new perspective data; on the other hand, the user can add foreground objects or dynamically modify the positions of the foreground objects to generate new scene data; the structured scene representation of the simulator enables the user to configure the number of sensors, supporting simultaneous generation of multi-perspective data. The above three parts enable the simulator to generate flexible and diverse large-scale data.

[0133] Based on the same inventive concept, an embodiment of the present application further provides an automatic driving real scene simulation method based on three-dimensional Gaussian sputtering.

[0134] Specifically, asFigure 7 As shown, the simulation method provided in this embodiment can be implemented by using the automatic driving real scene simulator based on three-dimensional Gaussian sputtering provided in any of the above embodiments of the application, and can include the following steps:

[0135] S1, obtaining a preset Gaussian model, a sensor model and a vehicle kinematics model as basic elements and ego elements in a simulation scene;

[0136] S2, defining and organizing the basic elements in the simulation scene in a structured manner to obtain environment elements;

[0137] S3, generating multi-modal perception data as the perception input of the automatic driving algorithm according to the current ego state, the sensor internal and external parameters and the structured organization relationship of the basic elements in the simulation scene;

[0138] S4, obtaining an external control quantity, calculating the ego state at the next time based on the current ego state and the control quantity through the vehicle kinematics model;

[0139] S5, distinguishing the foreground objects from the background, and performing foreground collision detection and background collision detection respectively, and performing collision judgment on the ego elements and the environment elements in the simulation scene; if collision is judged, the simulation is terminated and a warning is output; otherwise, returning to S3 for the next round of loop.

[0140] In some preferred embodiments, the S4 of obtaining an external control quantity can also be obtained in any of the following ways:

[0141] S41, using an external controller to output a control quantity based on the perception data to form a closed-loop control mode;

[0142] S42, based on the recorded historical trajectory, the ego motion is reproduced, and the control quantity is output to form an open-loop control mode.

[0143] In some preferred embodiments, the method can further include:

[0144] S6, driven by a global timer, the simulation method is executed in a fixed time step Δt.

[0145] It should be noted that the steps in the method provided by the application can be implemented by using the corresponding components in the simulator, and those skilled in the art can refer to the technical solution of the simulator to implement the step flow of the method, or refer to the technical solution of the method to implement the composition of the simulator, that is, the embodiments in the simulator and the embodiments in the method can be understood as preferred examples, which will not be repeated here.

[0146] The technical solutions provided by the above embodiments of the application will be further described in detail in combination with a specific application example.

[0147] The simulator involved in this specific application instance mainly includes the following five parts: structured scene representation and rendering strategy, collision detection mechanism, general model interface design, visual operation interface and closed-loop simulation process design, and the overall working architecture is as shown in Figure 1 .

[0148] I. Structured scene representation and rendering strategy

[0149] This part aims to define and organize the elements in the simulation scene in a structured way, and to realize efficient rendering based on the structure, including scene element composition, coordinate transformation and rendering strategy.

[0150] The elements of the simulation scene are divided into two categories: renderable elements and sensors. The renderable elements include background S and foreground objects V k , which are represented by Gaussian models. The basic parameter set of each Gaussian body in the Gaussian model is G i , including mean, scaling matrix, rotation matrix, color and opacity. The parameters describing the pose in the parameter set are defined in the local coordinate system of the element itself. Therefore, for a renderable element, its attribute set is represented as:

[0151] Background: where N is the number of background Gaussian bodies, is the coordinate transformation matrix of the background local coordinate system S to its parent node P;

[0152] Foreground objects: where M k is the number of Gaussian bodies of the kth foreground object, is the coordinate transformation matrix of the foreground object local coordinate system to its parent node.

[0153] Sensors include cameras, lidar and millimeter wave radar, and attributes include unique identifier id, intrinsic parameters and coordinate transformation matrix of local coordinate system to parent node, and the attribute set can be represented as:

[0154] Camera: where is the camera intrinsic parameter, is the coordinate transformation matrix of the camera coordinate system to its parent node;

[0155] Lidar: where is the lidar parameter, is the coordinate transformation matrix of the lidar coordinate system to its parent node;

[0156] Millimeter wave radar: where is the millimeter wave radar parameter, is the coordinate transformation matrix of the mmWave radar coordinate system to its parent node P.

[0157] Scene elements form a tree-like hierarchy extending from the world coordinate system W, which is the global reference frame, through parent-child relationships and coordinate transformation matrices. In addition, some GS algorithms introduce a canonical coordinate system W0 to improve the convergence speed of training. The mean, scaling matrix, and rotation matrix of the Gaussian body are represented in this canonical coordinate system. The transformation matrix from the element world coordinate to the canonical coordinate is:

[0158]

[0159] where λ is the normalized scaling factor, is the transformation matrix without scaling. Therefore, the actual representation of elements and sensors that can be rendered needs to include the canonical transformation matrix, i.e.,

[0160] Background:

[0161] Foreground objects:

[0162] Camera:

[0163] Laser radar:

[0164] mmWave radar:

[0165] For foreground objects V k , if separated from the original data set (denoted as V k1 ), its canonical transformation matrix is If obtained from other different data sets or foreground asset libraries (denoted as V k2 ), the canonical transformation matrix of its source data set is adopted

[0166] When rendering, the sensors and foreground objects need to be transformed to a unified canonical coordinate system. Taking a simple autonomous driving scene as an example, a vehicle V1 equipped with a camera, a laser radar, and a mmWave radar is driving on the road, as shown in Figure 2 The external parameters of the sensors in the canonical coordinate system are:

[0167]

[0168] The attribute set of the foreground vehicle in the canonical coordinate system can be represented as:

[0169]

[0170] where The original Gaussian parameter set G i The mean and rotation matrix become:

[0171]

[0172] The new parameter set is formed. Based on the above scene representation, the process of rendering a new view image or point cloud is:

[0173] X = C m , L n , R l

[0174] wherein, is the rendered image or point cloud, K X is the intrinsic parameter of sensor X, is the transformation matrix from sensor X to the canonical world coordinate system W0, X T(X = C m , L n , R l ) is the manual adjustment of the user to the sensor element, i.e., the pose transformation of the sensor element in its own coordinate system.

[0175] II. Collision detection mechanism

[0176] This part distinguishes foreground objects from background environment, respectively designs targeted detection methods, realizes the collision judgment of the ego vehicle and the environment elements in the simulation scene, specifically includes foreground collision detection and background collision detection, as shown in Figure 3 .

[0177] The foreground collision detection is the collision detection of the ego vehicle V1 and other foreground objects V k , which adopts a two-step strategy of bounding box generation combined with the separation axis theorem. For a foreground object composed of N Gaussian bodies, each Gaussian body parameter includes mean μ i , opacity α i and covariance Σ i , wherein α i satisfies The following steps are used to generate a directed bounding box:

[0178] Calculate the overall center of the Gaussian model:

[0179] Calculate the overall covariance matrix:

[0180] Perform eigenvalue decomposition on Σ to obtain 3 orthogonal principal directions {u1, u2, u3} as the principal axes of the bounding box;

[0181] Calculate the bounding box edge length in each principal axis direction: wherein k is an adjustable parameter;

[0182] Calculate the center of the bounding box in the original coordinate system:

[0183] Next, the separating axis theorem can be used to determine the foreground collision. For the bounding boxes of two foreground objects, check 15 potential separating axes (i.e. 6 from the 3 main directions of the two boxes, and 9 from the cross product of the 6 directions) to determine whether there is a collision. If there is an axis on which the projection intervals of the two boxes have no overlap, there is no collision; otherwise, it is determined to be a collision.

[0184] For collision detection between the ego vehicle and the background environment, a composite detection strategy combining Gaussian body probability and lidar point cloud is adopted. A collision risk indicator R ∈ [0, 1] is defined, where R → 1 indicates a collision. The risk indicator includes three types: Gaussian body-based collision risk, lidar point cloud-based collision risk, and fusion risk.

[0185] Gaussian body-based collision risk: define the collision risk indicator within the bounding box Ω of the ego vehicle:

[0186]

[0187] where V is the volume of the bounding box, M is the number of Monte Carlo sampling points, k G is a sensitivity parameter. In actual operation, since the number of Gaussian bodies N in the scene is large, and most of the Gaussian bodies far from the bounding box have small integral values in the bounding box, a detection region Ω0 slightly larger than the region enclosed by the bounding box can be selected, and only the Gaussian bodies whose center points fall within Ω0 are subjected to collision detection.

[0188] Lidar point cloud-based collision risk: after ground point removal of the lidar point cloud, the number of points N L falling into the bounding box of the ego vehicle is counted, and the risk indicator is defined as:

[0189] R L = 1-exp(-k L N L )

[0190] where k L is a sensitivity parameter.

[0191] Fusion risk: select a basic radar risk weight w0 ∈ [0, 1], and the Gaussian body risk weight is 1-w0. Since the collision detection based on lidar information is more accurate, if R G and R L are significantly different, more weight is given to the risk value R detected by the lidar.L The actual fusion weight of both is defined as:

[0192]

[0193] The final integrated risk value is:

[0194] R F = w L R L + w G R G

[0195] In this way, when R G ≈ R L , the original defined base weight value is maintained; and when the difference between R G and R L tends to 1, the detection value of the laser radar is preferred, and the detection reliability is improved.

[0196] In addition, the user can also test other collision detection algorithms through the collision detection algorithm interface.

[0197] III. General model interface design

[0198] This part realizes the integration of different GS algorithms and sensor models by designing standardized interfaces, and improves the compatibility and expansibility of the simulator.

[0199] The key model interfaces include rendering model interfaces and sensor model interfaces, as shown in Figure 4 , the functions of each part are as follows:

[0200] Rendering model interface: includes model information class, model loader and custom renderer. Model information is used to store Gaussian body related parameters, including center point position (mean), scaling factor, rotation matrix, color, opacity and other basic parameters; for different GS algorithms, it can also include its own extended parameters, the specific parameters depend on the algorithm model. The model loader is responsible for reading the user's trained Gaussian model file, supports ply, pth and other formats, and parses it into corresponding model information, realizing the conversion from external model file to internal scene representation of the simulator. After the model information is loaded, the custom renderer needs to access the custom renderer matched with the GS algorithm model information to complete the rendering process.

[0201] Sensor model interface: mainly used to add other types of cameras, lidar and millimeter wave radar. Camera information class contains pinhole camera model related parameters, mainly internal and external parameters, and a unique identifier for multi-camera reading. For some time-varying GS algorithms, it also contains timestamp information. Lidar or millimeter wave radar information class contains horizontal field of view, horizontal resolution, vertical field of view, vertical resolution, etc. Users can also choose virtual sensors to visualize lidar or millimeter wave radar point clouds in the dataset to complete the entire rendering process.

[0202] Four, visualization operation interface

[0203] This part further facilitates user use by designing a visualization operation interface, as shown in Figure 5 The functions of the visualization operation interface are described as follows.

[0204] ①Path visualization: used to visualize the trajectory of the vehicle. The gray trajectory is the vehicle trajectory during data collection, and the green trajectory is the vehicle trajectory during real-time simulation. The module also contains simulation process control buttons, and users can start, pause or stop the simulation at any time. In addition, the module also displays some simulation data in real time, such as simulation time, rendering frame rate and vehicle speed, etc.

[0205] ②Log window: used to print simulation information, such as printing the total simulation time at the end of the simulation, and printing collision warning information when a collision occurs;

[0206] ③Image visualization window: used to visualize the rendering results of camera images. If the user configures multi-camera simulation, the selection box in the upper left corner can select the camera to be rendered, and the selection box in the upper right corner is used to select the modal of model rendering, such as RGB image, depth map, etc.

[0207] ④Point cloud visualization window: visualizes the bird's eye view of the rendering results of lidar or millimeter wave radar point clouds;

[0208] ⑤Model setting interface: including model type selection, model path selection, simulation step size setting, simulation mode setting (used to specify the controller) and simulator data output setting. After the user accesses a new model in the format specified by the model general interface, the simulator can automatically detect the newly accessed model and load it in the selection box;

[0209] ⑥Sensor pose adjustment interface: including three-axis translation and rotation adjustment of cameras, lidar and millimeter wave radar. Users can conveniently configure the sensor layout through the sliding bar;

[0210] ⑦ Other setting interface: including collision detection type selection, users can choose Gaussian-based, laser radar-based or fusion-based collision detection algorithm. In addition, it also includes point cloud bird's eye view range adjustment, remove laser radar ground point selection and load virtual sensor option. Loading virtual sensors means rendering laser radar or millimeter wave radar point cloud ground truth during data collection, mainly used for open-loop simulation, and evaluation of laser radar or millimeter wave radar rendering model.

[0211] V. Closed-loop simulation process design

[0212] This part realizes the dynamic response of the simulator to external control signals through the constructed closed-loop control loop, as shown in Figure 6 , which supports end-to-end testing of autonomous driving algorithms.

[0213] The closed-loop simulation process is driven by a global timer and executes the following steps in fixed time steps Δt:

[0214] M1, load the Gaussian representation model of the scene (including background S and foreground objects V k ) and sensor parameters (camera C m , laser radar L n and millimeter wave radar R l ), initialize the initial pose [x0, y0, θ0] T and speed v0 of the ego vehicle, where the pose and speed constitute the ego vehicle state vector x(t) = [x, y, θ, v] T ;

[0215] M2, based on the current ego vehicle state x(t), sensor internal and external parameters and the structured organization relationship of basic elements in the simulation scene, the rendering method based on Gaussian sputtering or the rendering model interface provided by the simulator is used to build the Gaussian sputtering rendering method, and the rendering method is called to generate multi-modal perception data as the perception input of the autonomous driving algorithm; where the multi-modal perception data can include RGB images, depth images, semantic images, point clouds, etc., and the specific modal is determined by the rendering model accessed by the user;

[0216] M3, the external controller outputs control quantity u(t) based on the perception data;

[0217] M4, the next time ego vehicle state x(t+Δt) is calculated through the vehicle kinematics model: x(t+Δt) = f(x(t), u(t), Δt). Where f(·) is a pre-set vehicle kinematics model, inputting the current state x(t), control quantity u(t) and time step Δt, outputting the updated pose [x+Δx, y+Δy, θ+Δθ] T and speed v+Δv;

[0218] M5, call the collision detection mechanism, if collision is determined, terminate the simulation and output a warning; otherwise, return to step M2 and enter the next round of loop.

[0219] In addition, the simulator also supports switching between closed-loop and open-loop simulation. When connected to an external controller, the simulator is in closed-loop mode, enabling the complete process described above, supporting real-time adjustment of the ego vehicle's trajectory by control commands. When not connected to a controller, the simulator is in open-loop mode, reproducing the ego vehicle's motion according to the historical trajectory recorded in the data set.

[0220] Through the vehicle model interface, users can replace the kinematic model f(·), which can adapt to the characteristics of different vehicle models and improve the compatibility of the process for diverse testing needs.

[0221] The automatic driving real scene simulator and simulation method based on three-dimensional Gaussian sputtering provided by the above embodiments of the present application realize a multi-modal scalable closed-loop autonomous driving simulation framework based on Gaussian sputtering. In view of the problems that the existing Gaussian sputtering-based autonomous driving simulation method lacks a unified evaluation framework and has deficiencies in multi-modal support and scalability, the framework supports simulation of various sensor modalities such as cameras, laser radars and millimeter wave radars, and the modal type is flexibly determined by the rendering model accessed by the user; to ensure scalability, the framework provides a rendering model interface, a vehicle model interface, a sensor model interface and / or a collision detection algorithm interface to facilitate user integration and development of custom modules; based on the above framework, a complete end-to-end closed-loop or open-loop simulation is constructed; at the same time, the framework is also equipped with an intuitive visualization interface for configuring simulation parameters and displaying simulation data. The present application can meet the demand of autonomous driving algorithm research and development for massive high-quality data, and is suitable for training and testing of autonomous driving algorithms.

[0222] The above embodiments of the present application are described. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various modifications or modifications within the scope of the claims, which does not affect the essential content of the present application.

[0223] The above describes specific embodiments of the present application. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various modifications or modifications within the scope of the claims, which does not affect the essential content of the present application.

Claims

1. An automated driving real scene simulator based on three-dimensional Gaussian sputtering, characterized in that, The application relates to a simulation system for automatic driving, which comprises the following modules: a general model interface module, which realizes the integration of different Gaussian models, sensor models and vehicle kinematic models by providing a standardized interface, and obtains basic elements and ego elements in a simulation scene; an internal scene representation module, which defines and organizes the basic elements in a structured manner, and obtains environment elements; a collision detection module, which is used for distinguishing foreground objects from backgrounds of the environment elements, and respectively performing foreground collision detection and background collision detection, and performing collision determination on the ego elements and the environment elements in the simulation scene; a simulation control module, which performs dynamic response of a simulator to external control signals based on the internal scene representation module and the collision detection module, and realizes end-to-end test control of automatic driving.

2. The three-dimensional Gaussian-spraying based automated driving real scene simulator according to claim 1, wherein, The general model interface module comprises a rendering model interface, a sensor interface and a vehicle model interface; wherein: the rendering model interface is used for loading Gaussian models and custom renderers matched with the Gaussian models, and providing renderable elements of the basic elements in the simulation scene; the sensor model interface is used for adding sensor models, and providing sensor elements of the basic elements in the simulation scene; the vehicle model interface is used for loading different vehicle kinematic models, and adapting different vehicle types, and providing the ego elements in the simulation scene.

3. The three-dimensional Gaussian sputter-based autonomous driving real scene simulator of claim 1, wherein, The internal scene representation module defines and organizes the basic elements in a structured manner, and comprises: the basic elements comprise renderable elements and sensor elements; wherein: The renderable elements include a background S and a foreground object V k , both of which are represented by Gaussian models; for a renderable element, its attribute set is represented as: BACKGROUND where N is the number of background Gaussians, is the coordinate transformation matrix of the background local coordinate system to its parent node P; Foreground object: where M k is the number of Gaussian bodies of the kth foreground object, is the coordinate transformation matrix from the local coordinate system of the foreground object to its parent node P. the sensor elements comprise cameras, laser radars and millimeter wave radars, and attribute sets of the sensor elements are represented as: Camera: wherein, is the camera intrinsic parameter, is the coordinate transformation matrix from the camera coordinate system to its parent node P; Lidar: wherein, is a lidar parameter, is a coordinate transformation matrix of the lidar coordinate system to its parent node P; Millimeter wave radar: wherein, is a millimeter wave radar parameter, is a coordinate transformation matrix of the millimeter wave radar coordinate system to its parent node P; a transformation matrix of the basic elements from a world coordinate W to a canonical coordinate W0 is: wherein is the transformation matrix from world coordinates W to canonical coordinates Wo, and λ is a normalizing scale factor, is the transformation matrix without scaling; a representation of the basic elements contains a canonical transformation matrix, and a structured representation of the basic elements is: BACKGROUND foreground object: Camera: Laser radar: Millimeter wave radar: when rendering, the sensor elements and the foreground objects are transformed to a unified canonical coordinate system, and then: for the sensor elements, an external parameter in the canonical coordinate system is: an attribute set of a vehicle in the foreground objects in the canonical coordinate system is: wherein is the original set of Gaussian parameters G i In this case, the mean and rotation matrix become: a new parameter set is formed; a rendered new view image or a point cloud based on the structured representation of the basic elements is: wherein, is the rendered image or point cloud, K X is the intrinsic of sensor X, is the transformation matrix from sensor X to canonical world coordinate system W0, X T(X=C m , L n , R l is the manual adjustment of the user to the sensor element, i.e. the pose transformation of the sensor element in its own coordinate system.

4. The three-dimensional Gaussian sputter-based autonomous driving real scene simulator of claim 1, wherein, The collision detection module comprises a foreground collision detection unit and a background collision detection unit; wherein: The foreground collision detection unit adopts a two-step strategy collision detection algorithm combined with the separation axis theorem for collision detection between the ego element V1 and other foreground objects V k , which includes: A bounding box generation subunit, which, for a foreground object consisting of N Gaussian bodies, each Gaussian body parameter includes mean μ i , opacity α i And covariance Σ i , Where α i Satisfy A directed bounding box is generated by the following submodule: a Gaussian center calculation submodule, which is configured to calculate the overall center of the Gaussian model is: a covariance calculation submodule that calculates the population covariance matrix∑ as: a feature decomposition submodule is used for performing feature decomposition on a total covariance matrix Sigma, and obtaining orthogonal principal directions {u1, u2, u3} as main axes of bounding boxes; The bounding box side length calculation submodule is configured to calculate the bounding box side length L in each main axis direction j For: In the formula, k is an adjustable parameter; A surrounding frame center calculating sub-module is configured to calculate the center c of the surrounding frame in the original coordinate system as: a separation axis theorem judgment subunit is used for judging foreground collision through the separation axis theorem; wherein: for bounding boxes of two foreground objects, whether collision exists is judged by checking 15 potential separation axes; if there is an axis, and the projection intervals of the two boxes on the axis have no overlap, then no collision exists; otherwise, collision is determined. The background collision detection unit adopts a composite detection strategy collision detection algorithm of Gaussian body probability combined with laser radar point cloud for collision detection of the ego elements and the background; wherein: a collision risk index R of the Gaussian body and the laser radar is defined, and when R approaches 1, it indicates that collision occurs.

5. The three-dimensional Gaussian-spraying based automated driving real scene simulator of claim 4, wherein, The collision risk indicators include a Gaussian body-based collision risk, a laser radar point cloud-based collision risk, and a fusion risk. The Gaussian body-based collision risk includes: The collision risk indicators within the ego vehicle bounding box Ω are defined as: In the formula, V is the bounding box volume, M is the number of Monte Carlo sampling points, k G is a sensitivity parameter; A detection region Ω0 slightly larger than the region surrounded by the ego vehicle bounding box Ω is selected, and collision detection is only performed on the Gaussian bodies whose center points fall within Ω0; The laser radar point cloud-based collision risk includes: After the ground point removal of the lidar point cloud, the number N of points falling into the ego bounding box Ω is counted L The risk index is defined as: R L = 1 - exp(-k L N L ) where k L is a sensitivity parameter; The fusion risk includes: Select the basic radar risk weight w0∈[0,1], and the Gaussian body risk weight is 1-w0; define R G With R L The actual fusion weight of the two is: The final comprehensive risk value is: R F = w L R L + w G R G When R G ≈R L , the original defined base weight value is maintained; and when the difference between R G and R L tends to 1, the detection value of the laser radar is preferred.

6. The three-dimensional Gaussian sputter-based autonomous driving real scene simulator of claim 1, wherein, The simulation control module, based on the internal scene representation module and the collision detection module, performs dynamic response of the simulator to external control signals, including: a model loading unit, based on the internal scene representation module, loading a structured representation of the base elements in the simulated scene, initializing an initial pose [x0, y0, θ0] of the ego element T and a velocity v0, wherein the pose and the velocity constitute the ego state vector x(t) = [x, y, θ, v] T ; A simulation input unit generates multi-modal perception data as the perception input of the automatic driving algorithm through a rendering method according to the current ego vehicle state x(t), the sensor internal and external parameters, and the structured organizational relationship of the basic elements in the simulation scene; A control unit is configured to obtain an external control variable u(t); The state calculation unit calculates the vehicle's state at the next moment using a vehicle kinematics model: x(t+Δt)=f(x(t),u(t),Δt); where f(·) is a preset vehicle kinematics model, the inputs are the current state x(t), the control variable u(t), and the time step Δt, and the output is the updated pose [x+Δx,y+Δy,θ+Δθ]. T and velocity v + Δv; A simulation output unit calls the collision detection module, and if a collision is determined, the simulation is terminated and a warning is output; otherwise, the simulation input unit is returned, and the next round of loop is entered.

7. The three-dimensional Gaussian-spraying based automated driving real scene simulator according to claim 6, wherein, Further comprising: The control unit obtains the external control variable u(t) in any of the following ways: An external controller is used to output the control variable u(t) based on the perception data, forming a closed-loop control mode; Based on the recorded historical trajectory, the ego vehicle motion is reproduced, and the control variable u(t) is output, forming an open-loop control mode; And / or The simulation control module further comprises: A clock unit driven by a global timer controls the simulation loop time at a fixed time step Δt; and / or a visual operation interface module provides a visual interface to realize human-computer interaction; And / or The general model interface module further comprises: A collision detection algorithm interface is used to load a preset collision detection algorithm model and provide it to the collision detection module.

8. The three-dimensional Gaussian sputter-based autonomous driving real scene simulator of claim 7, wherein, The visual operation interface module comprises: A path visualization unit is configured to visually display the trajectory of the vehicle; A log window unit is configured to print simulation information, including the total simulation time and collision warning information when the simulation ends; An image visualization window unit is configured to visualize the rendering result of the camera image; A point cloud visualization window unit is configured to visualize the bird's eye view of the laser radar or millimeter wave radar point cloud rendering result; A model setting interface unit is configured to select and set simulation parameters, including loading model type selection, loading model path selection, simulation step size setting, simulation mode setting, and simulator generated data output setting; A sensor pose adjustment interface unit is configured to adjust the layout and configuration of the sensor elements; Other setting interface units are configured to make other selections and configurations, including collision detection type selection, point cloud bird's eye view range adjustment, laser radar ground point removal, and virtual sensor loading.

9. An automatic driving real scene simulation method based on three-dimensional Gaussian sputtering, characterized in that, Including: The preset Gaussian model, sensor model, and vehicle kinematics model are obtained as the basic elements in the simulation scene and the ego vehicle elements; The basic elements in the simulation scene are defined and organized in a structured manner to obtain environment elements; According to the current ego state, the sensor internal and external parameters, and the structured organization relationship of the basic elements in the simulation scene, multi-modal perception data is generated as the perception input of the automatic driving algorithm; An external control variable is obtained, and the next time ego state is calculated based on the current ego state and the control variable through a vehicle kinematics model; The environment elements are distinguished into foreground objects and background areas, and foreground collision detection and background collision detection are performed respectively to determine the collision between the ego elements and the environment elements in the simulation scene; If collision is determined, the simulation is terminated and a warning is output; otherwise, the step of perception input is returned to enter the next round of loop.

10. The three-dimensional Gaussian sputter-based autonomous driving real scene simulation method according to claim 9, characterized in that, Further comprising: Driven by a global timer as the core, the simulation method is executed in a fixed time step Δt loop; And / or The external control variable is obtained by any of the following methods: An external controller is used to output the control variable based on the perception data to form a closed-loop control mode; Based on the recorded historical trajectory, the ego motion is reproduced to output the control variable to form an open-loop control mode.

Images