A method for physical simulation of driving scenes based on neural rendering

Abstract

This invention discloses a driving scene physics simulation method based on neural rendering. It includes the following steps: First, a training dataset corresponding to each driving scene is acquired. A motion structure recovery module is used to extract the pose corresponding to the training dataset of the current driving scene. After training the hybrid rendering pipeline, the neural radiation field pipeline outputs the depth map of the current driving scene and obtains a trained Gaussian splash pipeline. Then, an interactive collision mesh space is generated based on the depth map. Finally, the target pose is input into the trained Gaussian splash pipeline to obtain the scene color observed at the target pose. A reinforcement learning-based agent explores and provides feedback in the interactive collision mesh space based on the scene color observed at the target pose. This invention can significantly improve the problem of unrealistic texture details in current driving simulation software while ensuring real-time interactive performance, reducing the gap between simulation software and the real world.

Description

Technical Field

[0001] This invention relates to a neural rendering method in the field of driving scene simulation, specifically to a driving scene physical simulation method based on neural rendering. Background Technology

[0002] Simulation technology is a key component in the field of autonomous driving, which can significantly improve the iteration efficiency of the system, reduce the cost pressure of real vehicle testing, and shorten the time for business deployment.

[0003] While commonly used simulators such as CARLA, Baidu Apollo, and Tencent TAD Sim generate data that provides a testing environment for driver assistance systems, the data produced by these traditional rendering pipelines differs significantly from real-world scenes, especially in terms of texture and detail, and cannot fully meet the needs of real-vehicle testing. Using this low-fidelity scene data for training perception models or vehicle simulation may lead to overfitting of the model to the simulation environment, resulting in a decline in perception performance in real-world scenarios.

[0004] Techniques based on neural radiation fields or Gaussian splashing methods have achieved some success in synthesizing realistic scenes from multiple perspectives, but their applications are mainly focused on multi-view synthesis. They cannot provide driving scene-level collision sensing and physical simulation, nor can they build an interactive exploration environment for assisted driving systems to learn driving rules.

[0005] One current technology is the method described by Alexey et al. in their paper "CARLA: An Open Urban Driving Simulator". Although the CARLA platform utilizes a physically based rendering engine to create data assets and model physical scenes, the limitations of the rendering engine mean that the 3D simulation scenes generated in real-time still differ significantly from the real world, making it difficult to meet practical needs. Using such low-fidelity scene data for perception model training or whole-vehicle simulation may lead to overfitting of the model to the simulation environment, resulting in decreased perception performance in real-world environments.

[0006] The second existing technology is the method described by Matthew et al. in their paper "Block-NeRF: Scalable Large Scene Neural View Synthesis". Block-NeRF is an extension of neural radiation fields designed to represent large-scale environments. This method divides the urban scene into multiple modules and independently assigns a neural radiation field to each module for rendering, dynamically combining these radiation fields during prediction. This decomposition process effectively decouples rendering time from scene size, enabling rendering to scale to arbitrary sizes and supporting block-by-block environment updates. However, this method cannot effectively predict the location of geometric information, resulting in blurry renderings or the creation of floating objects in the air, failing to provide realistic driving scenes.

[0007] The third existing technology is the method presented by Zhou et al. in their paper "DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes". Addressing the presence of moving objects in complex scenes, this method first employs an incremental static Gaussian model to sequentially and progressively simulate the static background of the entire scene. Next, it utilizes a composite dynamic Gaussian map to process multiple moving objects, reconstructing them individually and accurately restoring their positions and occlusion relationships within the scene. While this method provides high-fidelity and multi-camera consistency in realistic surround view synthesis, it only considers synthesizing a realistic view without considering interaction with the environment and lacks collision sensing design, which is detrimental to the rule learning of the assisted driving system. Summary of the Invention

[0008] To effectively address the issues of poor realism and lack of collision sensing in existing methods for rendering driving simulation scenarios, this invention proposes a neural rendering-based physical simulation method for driving scenes. This invention can render highly realistic driving scenes in real time, reducing the gap between the simulation system and the real world. Furthermore, this invention introduces collision distance to simulate collisions in the physical world, helping the driver assistance system learn driving rules.

[0009] The technical solution adopted in this invention is:

[0010] I. A Physical Simulation Method for Driving Scenes Based on Neural Rendering

[0011] S1: Construct a driving scene rendering network, which includes a connected motion structure recovery module and a hybrid rendering pipeline, wherein the hybrid rendering pipeline includes a connected neural radiation field pipeline and a Gaussian splashing pipeline.

[0012] S2: The training dataset for each driving scene is formed by the sequence of images containing the same driving simulation scene. After training the hybrid rendering pipeline in the driving scene rendering network using the training dataset of the current driving scene, the neural radiation field pipeline outputs the depth map of the current driving scene and obtains the trained Gaussian splash pipeline.

[0013] S3: After processing the depth map of the current driving scene using the truncation symbol function, the interactive collision mesh space is obtained;

[0014] S4: Input the target pose into the trained Gaussian splash pipe to obtain the scene color observed at the target pose. The reinforcement learning-based agent explores and provides feedback in the interactive collision grid space by combining the scene color observed at the target pose.

[0015] The aforementioned method for physical simulation of driving scenes based on neural rendering further includes the following steps:

[0016] S5: Obtain sequence images of different driving simulation scenarios, repeat S2-S4, and complete the simulation exploration in different driving simulation scenarios.

[0017] In S1, the motion structure recovery module of the driving scene rendering network is used to extract the pose of the input image. The pose of the input image and the position and orientation in the three-dimensional space of the scene are used as the input of the neural radiation field pipeline. The neural radiation field pipeline outputs the depth map, and the Gaussian splash pipeline outputs the scene color.

[0018] The motion structure recovery module includes a feature extraction module, a feature matching module, and a pose estimation module connected in sequence.

[0019] In step S2, the hybrid rendering pipeline in the driving scene rendering network is trained using the training dataset of the current driving scene, specifically including:

[0020] The pose estimated by the motion structure recovery module for each input image serves as the input pose for the neural radiation field pipeline. Based on the input pose and the current position and orientation in the 3D space of the scene, the neural radiation field pipeline predicts the color and density of points in the current 3D space, outputting voxel color rendering results, a depth map, and scene geometry information. The Gaussian splash pipeline learns Gaussian sphere properties based on the scene geometry information, thus outputting scene color and optimized scene geometry information. It updates the current position in the 3D space of the scene using the optimized scene geometry information. The network parameters in the neural radiation field pipeline are optimized and adjusted based on the voxel color rendering results and the first loss function calculated from the current input image, and the parameters in the Gaussian splash pipeline are optimized and adjusted based on the scene color and the second loss function calculated from the current input image, completing one training iteration. Next, the estimated poses corresponding to other input images in the current training dataset are sequentially input into the neural radiation field pipeline, and both the neural radiation field pipeline and the Gaussian splash pipeline are trained multiple times, while simultaneously updating the scene's 3D space, thus obtaining the trained neural radiation field pipeline and Gaussian splash pipeline. The trained neural radiation field pipeline outputs a depth map.

[0021] The neural radiation field includes a color estimation module and a geometry estimation module. The orientation in the three-dimensional space of the scene is spherically harmonic encoded and then input into the color estimation module, which outputs the voxel color rendering result. The position in the three-dimensional space of the scene is hash encoded and then input into the geometry estimation module, which outputs density information. The density information is then converted into a depth map and then the depth map is densified to obtain the scene geometry information.

[0022] The nerve radiation field conduit also includes:

[0023] The depth map output by the geometry estimation module is smoothed and regularized to obtain a processed depth map. The processed depth map is then subjected to point compaction to obtain scene geometry information, which is used as input to the Gaussian splash pipeline.

[0024] In the reinforcement learning-based agent, the final reward function of the agent satisfies the following formula:

[0025]

[0026] r basic (s t a t )=0.1*(lap(s t )-u)

[0027]

[0028] Where r(s) t a t) represents the final reward function value, r basic (s t a t ) represents the basic reward function value, s t a represents the state of the agent's interaction and exploration with the environment at time t. t Let f(a, b) represent the action of the agent, f(a, b) represent the collision distance, and lap(s) represent the collision distance. t ) indicates that the quality of the rendered image is estimated using the Laplacian operator, and u represents the threshold for whether the observed scene colors are sharp; x a ,y a ,z a These are the center coordinates of the vehicle in the world coordinate system, x and x. b ,y b ,z b These are the center coordinates of the colliding object's mesh in the world coordinate system, a and b. r Let b be the radius of the vehicle. r The radius of the object's mesh during the collision.

[0029] The color estimation module consists of two interconnected fully connected layers; the geometry estimation module consists of one interconnected fully connected layer.

[0030] The depth map is smoothed and regularized according to the following formula:

[0031]

[0032] in, Let θ represent the loss function for the depth map, and r represent the learned parameters. ij Let r represent the ray passing through pixel (i, j). i+1j+1 Let r represent the ray passing through pixel (i+1, j+1). i+1j Let r represent the ray passing through pixel (i+1, j). ij+1 Let r represent the ray passing through pixel (i, j+1). ij S represents the ray passing through pixel (i, j). patch This indicates the size of the rendered image patch, which is the region centered on ray r. This represents the set of rays sampled from the camera pose. The purpose of smoothing regularization is to minimize the depth difference between adjacent pixels in a specific image sub-block.

[0033] In step S3, the depth map of the current driving scene is processed using the following formula:

[0034]

[0035] tsdf(x)=max[-1, min(1, sdf(x) / λ)]

[0036] in, The depth map representing the current driving scene, cam z (x) represents the depth of voxel x relative to the camera, λ represents the threshold of the difference between voxel x and the depth of the object cross section, and tsdf(x) represents the truncation sign function operation on voxel x.

[0037] In the reinforcement learning-based agent, the state s of the agent's interaction with the environment at time t is... t Satisfy the following formula:

[0038] s t =(C t h t )

[0039] Among them, h t C represents the color of the historical scene observed at time t. t This represents the scene color observed by the agent at time t.

[0040] The action space of the intelligent agent is the driving behavior in the world coordinate system, which specifically includes four displacement actions and two rotation actions.

[0041] II. A computer device

[0042] The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the driving scene physics simulation method based on neural rendering.

[0043] III. A computer-readable storage medium

[0044] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the driving scene physics simulation method based on neural rendering.

[0045] IV. A computer program product

[0046] The computer program product includes a computer program / instruction that, when executed by a processor, implements the steps of the neural rendering-based driving scene physics simulation method.

[0047] The beneficial effects of this invention are:

[0048] By adopting the above technical solutions, the method proposed in this invention can synthesize a highly realistic driving simulation environment through a driving scene rendering network, providing a rendering scene with clear texture details.

[0049] The hybrid renderer proposed in this invention learns scene depth through hybrid optimization, thereby constructing an interactive and collision-enabled mesh space, which can solve the problem of physical sensing that current rendering methods cannot provide.

[0050] This invention utilizes reinforcement learning technology, enabling driver assistance systems to explore and collect vast amounts of data in a simulated environment through trial and error. This allows them to learn and simulate perception and decision-making in real-world environments, without relying on traditional rule-based methods. This invention improves upon the unrealistic texture details in existing driving simulation software while maintaining real-time interactive performance, narrowing the gap between simulation and the real world. It also provides collision distances and learns driving rules in highly realistic simulated environments; this not only enhances the robustness of driver assistance systems in complex driving environments but also addresses the scarcity of real-world data. Furthermore, the real-time interactive performance provides the agent with real-time feedback for learning and error correction. Attached Figure Description

[0051] Figure 1 This is an overall flowchart of a driving scene physics simulation method based on neural rendering, according to an embodiment of the present invention.

[0052] Figure 2 This is the framework of the driving scene physics simulation method based on neural rendering in this invention.

[0053] Figure 3 This is a flowchart of the neural radiation field in the hybrid renderer of this invention.

[0054] Figure 4 This is a comparison result between the driving scenario simulation of this invention and a traditional driving simulation engine.

[0055] Figure 5 This is a highly realistic visualization result of the hybrid renderer rendering from different angles during driving scene simulation according to an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0057] This invention proposes a method for physical simulation of driving scenes based on neural rendering, such as... Figure 1 As shown, the method includes the following steps:

[0058] S1: Construct a driving scene rendering network, such as Figure 2 As shown, the driving scene rendering network includes a connected motion structure recovery module and a hybrid rendering pipeline; the hybrid rendering pipeline includes a connected neural radiation field pipeline and a Gaussian splashing pipeline.

[0059] The motion structure recovery module of the driving scene rendering network is used to extract the pose of the input image. The pose of the input image and the position and orientation in the three-dimensional space of the scene are used as inputs to the neural radiation field pipeline. The neural radiation field pipeline outputs the depth map, and the Gaussian splash pipeline outputs the scene color.

[0060] The motion structure recovery module includes a feature extraction module, a feature matching module, and a pose estimation module connected in sequence. The input image is input into the feature extraction module, and the final pose estimation module outputs the initial pose corresponding to the input image. The pose represents the position and orientation of the object in the coordinate system.

[0061] S2: The training dataset for each driving scene is formed by the sequence of images containing the same driving simulation scene. After end-to-end training of the hybrid rendering pipeline in the driving scene rendering network using the training dataset of the current driving scene, the neural radiation field pipeline outputs the depth map of the current driving scene and obtains the trained Gaussian splash pipeline.

[0062] In this embodiment, three common datasets—Waymo, Argoverse, and Kitti—were selected. These datasets all contain images captured under good field of view and include diverse driving scenarios such as city streets, alleys, and highways. The collected datasets were divided into multi-camera scenarios and single-camera scenarios based on the number of cameras. The datasets were then further divided into training and test sets based on the number of frames, with every 10 frames used as the test set and the remainder as the training set.

[0063] In S2, the hybrid rendering pipeline in the driving scene rendering network is trained end-to-end using the training dataset of the current driving scene, specifically including:

[0064] The pose estimated by the motion structure recovery module for each input image serves as the input pose for the neural radiation field pipeline. Based on the input pose and the current position and orientation in the 3D space of the scene, the neural radiation field pipeline predicts the color and density of points in the current 3D space, outputting voxel color rendering results, a depth map, and scene geometric information (calculated from density). The initial position and orientation in the scene's 3D space are generated; the position is determined by the sampling point location traversed by a ray originating from the camera origin and passing through each pixel in the corresponding input image, while the orientation refers to the direction of the ray. The Gaussian splash pipeline learns the properties of a Gaussian sphere based on the scene geometry, such as its shape, size, and color, thus outputting scene color and optimized scene geometry. The optimization method for scene geometry in the Gaussian splash pipeline employs differentiable point cloud splitting and cloning techniques. The optimized scene geometry information is used to update the current position in the 3D space of the scene, thereby improving the accuracy of the reconstructed geometric space. The network parameters in the neural radiation field pipeline are optimized and adjusted based on the voxel color rendering results and the first loss function calculated using the current input image. Similarly, the parameters in the Gaussian splash pipeline are optimized and adjusted based on the scene color and the current input image using the second loss function, thus completing one training iteration. The specific training process of the Gaussian splash pipeline follows the formula below. c i The color of the Gaussian sphere, α i The opacity of Gaussian sphere i is α. j The opacity of Gaussian sphere j is given by N, where N is the number of Gaussian spheres and C is the number of Gaussian spheres. t The output scene color is represented by the first layer. Next, the estimated poses of other input images from the current training dataset are sequentially input into the neural radiation field pipeline. The neural radiation field pipeline and the Gaussian splash pipeline are trained multiple times, while simultaneously updating the scene's 3D space. This results in trained neural radiation field pipelines and Gaussian splash pipelines. The trained neural radiation field pipeline outputs a depth map, used to calculate the subsequent collision mesh space; the Gaussian splash pipeline outputs the scene color, used for subsequent observation by the agent. Specifically, the positions in the updated scene's 3D space are hashed to obtain a new depth map. Then, a point compaction method is used to transform and obtain new scene geometric information x. This new scene geometric information x contains new positions used to fill in problematic areas.

[0065] like Figure 3As shown, the neural radiation field includes a color estimation module and a geometry estimation module. The orientation in the scene's 3D space is spherically harmonic-encoded and then input into the color estimation module, which outputs the voxel color rendering result. Similarly, the position in the scene's 3D space is hash-encoded and then input into the geometry estimation module, which outputs density information. This density information is then converted into a depth map, which is subsequently densified to obtain the scene's geometric information.

[0066] c θ , σ θ =f θ (γ(x), γ(d))

[0067] Among them, c θ For color, σ θ For density, f θ The neural radiation field is parameterized, where x is the position, d is the direction, and γ represents a predefined spherical harmonic code or positional code. Given the neural radiation field, pixels are rendered by projecting a ray r(u) = o + ud from the camera center o along direction d through the pixel. The rendered pixels are used to represent the simulated driving scene, while a depth map still needs to be learned to represent the mesh space.

[0068] Depth maps satisfy the following formula:

[0069]

[0070] in, u represents the depth value corresponding to ray r. n and u f These are the predefined near and far planes used for rendering, respectively. Q(u) represents the cumulative transmittance along the ray, and σ... θ The value represents the opacity of the neural radiation field estimation; r(u) represents the ray at the sampling point, and u represents the location of the sampling point.

[0071] Specifically, the following formula is used to perform point densification on the depth map:

[0072]

[0073] Where x represents the scene geometry, o is the origin of the ray, and d r The direction of the ray.

[0074] The color estimation module consists of two fully connected layers with 64 channels.

[0075] The geometry estimation module consists of a fully connected layer with 64 channels connected in one layer.

[0076] The nerve radiation field conduit also includes:

[0077] The depth map output by the geometry estimation module is smoothed and regularized to obtain a processed depth map. The processed depth map is then subjected to point compaction to obtain scene geometry information, which is used as input to the Gaussian splash pipeline.

[0078] The optimization of driving scene depth is achieved through multi-view Figure 1 Consistency and depth map smoothing regularization are composed of:

[0079] The depth map obtained by hybrid optimization using neural radiation field pipelines and Gaussian splashing pipelines is optimized through multi-view processing to ensure 3D consistency. Figure 1 Consistency regularization: For each pose, two camera depth maps are selected from neighboring viewpoints, and the keypoints of the current viewpoint are converted to camera coordinates. These points are projected back to the current view to measure pixel-by-pixel differences.

[0080] The depth map is smoothed and regularized according to the following formula:

[0081]

[0082] in, Let θ represent the loss function for the depth map, and r represent the learned parameters. ij Let r represent the ray passing through pixel (i, j). ij Let r represent the ray passing through pixel (i, j). i+1j+1 Let r represent the ray passing through pixel (i+1, j+1). i+1j Let r represent the ray passing through pixel (i+1, j). ij+1 Let r represent the ray passing through pixel (i, j+1). ij S represents the ray passing through pixel (i, j). patch This indicates the size of the rendered image patch, which is the region centered on ray r. This represents the set of rays sampled from the camera pose. The purpose of smoothing regularization is to minimize the depth difference between adjacent pixels in a specific image sub-block.

[0083] S3: After processing the depth map of the current driving scene using the truncation symbol function, the interactive collision mesh space is obtained, which is used to simulate physical collisions between vehicles or interactions between vehicles and the environment.

[0084] Specifically, the depth map of the current driving scene is processed using the following formula:

[0085]

[0086] tsdf(x)=max[-1, min(1, sdf(x) / λ)]

[0087] in, The depth map representing the current driving scene, cam z (x) represents the depth of voxel x relative to the camera, λ represents the threshold of the difference between voxel x and the depth of the object cross section, and tsdf(x) represents the truncation sign function operation on voxel x.

[0088] If tsdf(x) is close to 0, it means the voxel is close to the surface. Conversely, when tsdf(x) is close to 1 or -1, it means the voxel is far from the surface.

[0089] S4: Utilize reinforcement learning-based agents to explore the interactive collision grid space.

[0090] In a reinforcement learning-based agent, the state s of the agent's interaction with the environment at time t is... t Satisfy the following formula:

[0091] s t =(C t h t )

[0092] Among them, h t C represents the color of the historical scene observed at time t. t This represents the scene color observed by the agent at time t.

[0093] like Figure 4 As shown, the Gaussian splash pipeline in the hybrid rendering pipeline outputs the scene's colors, which are then combined with historical image frames for the agent's observation. Compared to traditional simulators, the rendering results of this invention are more realistic in texture, reducing the gap with the real world. Furthermore, the hybrid rendering pipeline proposed in this invention only requires driving data from the forward-looking perspective to render different perspective renderings for the agent's observation, such as... Figure 5 As shown in the figure. Compared with existing methods, this avoids the need for repeated collection of large amounts of data.

[0094] The agent's action space is the driving behavior in the world coordinate system, specifically including four displacement actions and two rotation actions. The four displacement actions are forward, backward, left, and right; the two rotation actions are rotating 10° clockwise and 10° counterclockwise.

[0095] The agent learns the value of actions Q(s) through repeated exploration. t a t By selecting the action that maximizes the value in the current state, the driver learns real-world driving rules. Specifically, given an initial state s... t The intelligent agent determines the action with the highest value in the current environment through a value function. * =argmaxQ(s t a t The agent executes the optimal policy a.* Then, transition to the next state s. t+1 Through a continuous sequential decision-making process (a t a t+1 a t+2 a t+3 a t+4 a t+4… The intelligent agent eventually learns the optimal driving strategy.

[0096] The agent's final reward function satisfies the following formula:

[0097]

[0098] r basic (s t a t )=0.1*(lap(s t )-u)

[0099]

[0100] Where r(s) t a t ) represents the final reward function value, r basic (s t a t ) represents the basic reward function value, s t a represents the state of the agent's interaction and exploration with the environment at time t. t Let f(a, b) represent the action of the agent, f(a, b) represent the collision distance, and lap(s) represent the collision distance. t The expression indicates that the Laplacian operator is used to estimate the quality of the rendered image, and u represents the threshold for whether the observed scene colors are sharp, set to 100; x a y a , z a These are the center coordinates of the vehicle in the world coordinate system, x and x. b y b , z b These are the center coordinates of the colliding object's mesh in the world coordinate system, a and b. r Let b be the radius of the vehicle. r The radius of the object's mesh during the collision.

[0101] The exploration of agent-environment interaction is mainly based on reinforcement learning techniques with value functions. By obtaining the positions of objects such as vehicles and houses through interactive collision grid space, it can determine whether the actions performed by the agent will receive positive or negative feedback. In this way, it learns the driving rules of the assisted driving system in complex environments and improves its robustness in complex driving environments.

[0102] S5: Obtain sequence images of different driving simulation scenarios, repeat S2-S4, and complete the simulation exploration in different driving simulation scenarios.

[0103] The method proposed in this invention can improve the problem of unrealistic texture details in existing driving simulation software while ensuring real-time interactive performance, thus narrowing the gap between simulation and the real world. Furthermore, the real-time interactive performance can provide the intelligent agent with real-time feedback information for learning and error correction.

[0104] The present invention also proposes a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of a driving scene physics simulation method based on neural rendering.

[0105] The present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a driving scene physics simulation method based on neural rendering.

[0106] The present invention also proposes a computer program product comprising a computer program / instruction that, when executed by a processor, implements the steps of a driving scene physics simulation method based on neural rendering.

[0107] This invention proposes a driving scene physics simulation method based on neural rendering. Through the design of a hybrid neural renderer, it can render highly realistic driving scenes in real time. This invention utilizes multi-view... Figure 1 Consistency regularization and smoothing regularization optimize the depth map, thereby generating a high-quality mesh space that provides accurate collision sensing for the agent, enabling it to learn rules based on the driving simulation scenario. This invention will further facilitate the practical application of driving simulation methods in the field of autonomous driving.

[0108] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for physical simulation of driving scenes based on neural rendering, characterized in that, Includes the following steps: S1: Construct a driving scene rendering network, which includes a connected motion structure recovery module and a hybrid rendering pipeline, wherein the hybrid rendering pipeline includes a connected neural radiation field pipeline and a Gaussian splashing pipeline. S2: The training dataset for each driving scene is formed by the sequence of images containing the same driving simulation scene. After training the hybrid rendering pipeline in the driving scene rendering network using the training dataset of the current driving scene, the neural radiation field pipeline outputs the depth map of the current driving scene and obtains the trained Gaussian splash pipeline. S3: After processing the depth map of the current driving scene using the truncation symbol function, the interactive collision mesh space is obtained; S4: Input the target pose into the trained Gaussian splash pipe to obtain the scene color observed at the target pose. The reinforcement learning-based agent explores and provides feedback in the interactive collision grid space by combining the scene color observed at the target pose.

2. The driving scene physics simulation method based on neural rendering according to claim 1, characterized in that, The method further includes the following steps: S5: Obtain sequence images of different driving simulation scenarios, repeat S2-S4, and complete the simulation exploration in different driving simulation scenarios.

3. The driving scene physics simulation method based on neural rendering according to claim 1, characterized in that, In S1, the motion structure recovery module of the driving scene rendering network is used to extract the pose of the input image. The pose of the input image and the position and orientation in the three-dimensional space of the scene are used as the input of the neural radiation field pipeline. The neural radiation field pipeline outputs the depth map, and the Gaussian splash pipeline outputs the scene color.

4. The driving scene physics simulation method based on neural rendering according to claim 1, characterized in that, The motion structure recovery module includes a feature extraction module, a feature matching module, and a pose estimation module connected in sequence.

5. The driving scene physics simulation method based on neural rendering according to claim 1, characterized in that, In step S2, the hybrid rendering pipeline in the driving scene rendering network is trained using the training dataset of the current driving scene, specifically including: The pose estimated by the motion structure recovery module for each input image serves as the input pose for the neural radiation field pipeline. Based on the input pose and the current position and orientation in the 3D space of the scene, the neural radiation field pipeline predicts the color and density of points in the current 3D space, outputting voxel color rendering results, a depth map, and scene geometry information. The Gaussian splash pipeline learns Gaussian sphere properties based on the scene geometry information, thus outputting scene color and optimized scene geometry information. It updates the current position in the 3D space of the scene using the optimized scene geometry information. The network parameters in the neural radiation field pipeline are optimized and adjusted based on the voxel color rendering results and the first loss function calculated from the current input image, and the parameters in the Gaussian splash pipeline are optimized and adjusted based on the scene color and the second loss function calculated from the current input image, completing one training iteration. Next, the estimated poses corresponding to other input images in the current training dataset are sequentially input into the neural radiation field pipeline, and both the neural radiation field pipeline and the Gaussian splash pipeline are trained multiple times, while simultaneously updating the scene's 3D space, thus obtaining the trained neural radiation field pipeline and Gaussian splash pipeline. The trained neural radiation field pipeline outputs a depth map.

6. The driving scene physics simulation method based on neural rendering according to claim 5, characterized in that, The neural radiation field includes a color estimation module and a geometric estimation module. The direction in the three-dimensional space of the scene is spherically harmonic encoded and then input into the color estimation module. The color estimation module outputs the voxel color rendering result. The location in the 3D space of the scene is hash-encoded and then input into the geometry estimation module. The geometry estimation module outputs density information, which is then converted into a depth map and then the depth map is densified to obtain the scene geometry information.

7. The driving scene physics simulation method based on neural rendering according to claim 5, characterized in that, The nerve radiation field conduit also includes: The depth map output by the geometry estimation module is smoothed and regularized to obtain a processed depth map. The processed depth map is then subjected to point compaction to obtain scene geometry information, which is used as input to the Gaussian splash pipeline.

8. The driving scene physics simulation method based on neural rendering according to claim 1, characterized in that, In the reinforcement learning-based agent, the final reward function of the agent satisfies the following formula: r basic (s t ,a t )=0.1*(lap(s t )-u) Where r(s) t a t ) represents the final reward function value, r basic (s t ,a t ) represents the basic reward function value, s t a represents the state of the agent's interaction and exploration with the environment at time t. t Let f(a, b) represent the action of the agent, f(a, b) represent the collision distance, and lap(s) represent the collision distance. t ) indicates that the quality of the rendered image is estimated using the Laplacian operator, and u represents the threshold for whether the observed scene colors are sharp; x a ,y a ,z a These are the center coordinates of the vehicle in the world coordinate system, x and x. b y b , z b These are the center coordinates of the colliding object's mesh in the world coordinate system, a and b. r Let b be the radius of the vehicle. r The radius of the object's mesh during the collision.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the driving scene physics simulation method based on neural rendering as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the driving scene physics simulation method based on neural rendering as described in any one of claims 1 to 8.

Images