Automatic driving scene neural radiation field reconstruction method based on Bayesian rays

Abstract

The invention discloses an uncertainty perception simulation method based on a neural radiation field (NeRF), and aims to solve the problem of insufficient reality sense of a dynamic scene in an automatic driving scene. Aiming at the problems of rendering artifacts and uncertainty generated when a reflecting material and a shielding area are processed by a traditional simulation method, the invention provides a new framework fusing Bayesian uncertainty quantization and dynamic scene optimization. On the basis of an MARS simulator, the frame remarkably improves the visual quality and reliability of a simulation scene through a special rendering loss function and a physics-based rendering model. Experimental results show that the method is superior to a baseline MARS simulator in KITTI and VKITTI data sets, and particularly shows higher application value and generalization ability in the aspects of reducing artifacts and improving rendering fidelity, and provides a more reliable test environment for automatic driving simulation.

Description

TECHNICAL FIELD

[0001] The present application relates to a large scene three-dimensional reconstruction method based on neural radiance field (NeRF) in the field of automatic driving, in particular to a neural radiance field reconstruction method based on Bayesian rays. BACKGROUND

[0002] In recent years, significant progress has been made in autonomous driving technology, and vehicles are now able to use advanced mapping and sensing technology to travel in complex urban environments. However, developing fully autonomous driving systems still needs to overcome major challenges, especially when dealing with rare and unexpected situations that can pose significant risks to passengers and other road users. In order to reduce these risks, the research community is increasingly relying on high-fidelity simulations as an important tool for training and validating autonomous driving algorithms in controlled environments.

[0003] Traditional simulation methods, such as CARLA, rely on graphics engines and manually curated assets, which limit their ability to capture the complexity of real-world scenarios, especially under different lighting conditions and dynamic objects. This often results in rendering artifacts, such as "ghosting" in reflections, which can mislead the algorithms being tested. While new approaches like GeoSim attempt to enhance realism through data-driven image enhancement, they still struggle to fully replicate the nuances of dynamic real-world environments.

[0004] Urban-NeRF methods such as UrbanIR and LightSim focus on urban scene reconstruction. UrbanIR infers shape, albedo, and visibility from a single video, uses a visibility loss to estimate shadows, and supports relighting and object insertion. LightSim constructs a "light-aware" digital twin by combining physical fundamentals and learnable rendering to achieve urban scene relighting. These methods demonstrate the potential of NeRF in urban autonomous driving environments, but have limited capabilities in handling dynamic scenes.

[0005] The MARS simulator represents a significant advancement in this field by achieving high fidelity through the use of neural radiance fields (NeRF) to model static and dynamic elements. Its instance-awareness and modular design contribute to flexible and realistic simulations, setting a new benchmark for photo-realism. However, MARS also faces specific challenges, as dynamic highlights can produce artifacts when rendering reflective materials such as car windows or water surfaces. Additionally, the inherent uncertainty in occluded areas and the interactions between transient objects such as pedestrians and the environment further complicate the simulation, affecting its overall reliability. SUMMARY

[0006] The present application aims to provide a neural radiance field reconstruction method for autonomous driving scenes based on Bayesian rays.

[0007] The technical solution adopted by this invention is to propose a Bayesian ray-based neural radiation field reconstruction method for autonomous driving scenarios, combining the uncertainty quantification strategy of Bayesian ray with the dynamic modeling capability of MARS. The specific steps are as follows:

[0008] 1. NeRF dynamic modeling: The input sequence is segmented and scene flow predicted using MARS to separate static background from dynamic foreground objects, providing accurate prior information for NeRF dynamic modeling.

[0009] 2. Uncertainty Field Generation: Bayes’Rays applies local spatial perturbations to dynamic regions (e.g., vehicle surfaces) in pre-trained MARS. The Bayesian Laplace approximation estimates the change in reconstruction loss due to the perturbations and generates a volumetric uncertainty heatmap.

[0010] 3. Artifact filtering and scene optimization: Under the guidance of the uncertainty heat map, an adaptive threshold algorithm is used to filter out low-confidence areas and retain geometrically consistent areas to optimize the final scene.

[0011] The input in step 1 is defined as follows: Define a multi-view dynamic sequence consisting of RGB images from the vehicle side or roadside sensor Sensor posture and object trajectories (3D bounding boxes Category {type ij} N×M and instance ID {idx ij} N×M ), where N is the number of frames and M is the number of tracking instances. The radiation field modeling uses σ(x) = f θ (x) represents the volume density, f(x,d)=g φ (x, d) denotes the view-dependent color, and parameters θ and φ are conditioned on the segmentation data from MARS.

[0012] In step 1, we use the instance-aware design of MARS to pre-process the dynamic scene using instance segmentation and scene flow prediction. This provides information for the NeRF model to separate the static and dynamic components. For a given 3D position and viewing direction d∈S 2 , the radiation field is encoded as:

[0013] σ(x)=f θ (x),f(x,d)=g φ (x,d),

[0014] Among them, θ and φ are optimized using the foreground-background separation representation of MARS, where θ and φ represent the foreground and background respectively, thereby enhancing the control of instance trajectory and appearance.

[0015] Step two is inspired by Bayesian rays, introducing spatial perturbations into dynamic regions. For a point x, the perturbed position is x + δ, where δ is a small displacement. The reconstruction loss L(x) is defined as the photometric error between the predicted pixel color and the ground truth pixel color. The loss change due to the perturbation is approximated by a second-order Taylor expansion:

[0016]

[0017] where is the Hessian matrix. Using the Laplace approximation, we estimate the uncertainty as the inverse of the diagonal elements of the approximated Hessian matrix:

[0018]

[0019] where represents the uncertainty estimate at position x, diag(―H(0)) ―1 represents the inverse of the Hessian matrix diagonal elements, effectively calculating high-uncertainty regions (e.g., reflective surfaces) by backpropagating the Jacobian, highlighted in the heatmap, guiding the subsequent optimization.

[0020] Step three integrates a physics-based rendering layer using the Cook-Torrance microfacet model for handling reflective surfaces (e.g., vehicle glass) that decompose the radiance into diffuse and specular components:

[0021] I(x, d) = I diffuse (x) + I specular (x, d),

[0022] x represents the 3D position, d represents the viewing direction, I diffuse is the diffuse component, and I diffuse is the specular component, where the specular term is:

[0023]

[0024] where D(h) is the microfacet distribution, F(h) is the Fresnel term, G(x, d) is the geometric shadowing, n as the surface normal, l and v as the light and view directions. We adopt the self-calibration of HDR-Plenoxels to correct the high dynamic range (HDR) reflections, optimizing the composite rendering loss:

[0025]

[0026] where λ1 balances the pixel-level color accuracy, λ2 balances the texture smoothness, I pred represents the model processed image, and I gtrepresent a real image.

[0027] Compared with the prior art, the present application has the following advantages:

[0028] (1) By integrating the neural radiance field and the Bayesian ray uncertainty quantification technology, the present application significantly improves the visual fidelity of the simulation environment in the automatic driving scene reconstruction. NeRF can accurately capture the three-dimensional geometry and lighting information of complex scenes through deep learning based on volume rendering, and the Bayesian ray uncertainty quantification further introduces statistical modeling to optimize the uncertainty estimation in the process of ray propagation. This combined method not only effectively reduces the common artifacts in traditional NeRF models, but also truly restores the complex lighting effects in dynamic scenes through a physically-based rendering model. Compared with traditional grid-based rendering methods or simple volume rendering techniques, the scenes generated by us are closer to the real world in terms of texture details, dynamic object trajectories, and lighting changes, providing a higher quality virtual environment for automatic driving simulation.

[0029] (2) Traditional NeRF models often suffer from performance degradation when dealing with dynamic highlights, complex occlusions, or non-uniform lighting due to the lack of uncertainty modeling. However, the present application can dynamically evaluate the confidence of each pixel or voxel in the scene through Bayesian ray uncertainty quantification, thereby prioritizing high-uncertainty areas during optimization. This strategy not only improves the model's adaptability to complex scenes, but also enhances its generalization performance in unknown environments. For example, in the face of weather changes (such as rainy or foggy days) or different lighting conditions at different times, our reconstruction results can maintain stable high-quality output.

[0030] (3) Bayesian ray uncertainty quantification not only generates high-fidelity three-dimensional scenes, but also provides confidence distributions for each reconstructed area, clearly identifying low-confidence areas. This uncertainty information is crucial for the training and validation of automatic driving algorithms, as it allows developers to identify potential model weaknesses and optimize perception, planning, and decision-making modules accordingly. Compared with traditional methods such as LiDAR-based point cloud reconstruction or ordinary NeRF models, our uncertainty estimation can provide more reliable assessments for safety-critical scenarios such as intersections, pedestrian-dense areas, or low-visibility environments. This capability makes the present application have significant advantages in automatic driving algorithm validation, virtual testing, and actual deployment, providing strong technical support for achieving safer and more reliable automatic driving technology. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a schematic diagram of the overall architecture of the system;

[0032] Figure 2 is a schematic diagram of the rendering composition;

[0033] Figure 3 Schematic diagram of specular reflection and diffuse reflection;

[0034] Figure 4 Qualitative image reconstruction experimental results for the KITTI dataset;

[0035] Figure 5 This is the artifact filtering experiment result;

[0036] Figure 6 Experimental results for the KITTI dataset image reconstruction task;

[0037] Figure 7 Experimental results on new view synthesis for the KITTI dataset;

[0038] Figure 8 Experimental results on new view synthesis for the VKITTI dataset; DETAILED DESCRIPTION

[0039] The present invention will be further described below with reference to the accompanying drawings.

[0040] To address the issue of dynamic highlights generating artifacts when rendering reflective materials, we proposed an autonomous driving simulation framework that combines Bayesian uncertainty quantization with dynamic NeRF to extend the capabilities of the MARS architecture. We developed a specialized rendering loss function that takes into account the characteristics of reflective materials, effectively reducing ghosting artifacts and enhancing the fidelity of the simulated scene. This also enables an efficient post-processing pipeline that is fully compatible with the MARS modular structure, ensuring ease of integration and use. The overall system architecture is shown in the figure below. Figure 1 : Taking multi-view dynamic sequences as input, MARS is used for instance segmentation and scene flow prediction to separate static background from dynamic foreground objects. Bayes'Rays is used to apply local spatial perturbations to dynamic regions in pre-trained MARS to generate volumetric uncertainty heatmaps. Under the guidance of the uncertainty heatmap, an adaptive threshold algorithm can filter out low-confidence regions and retain geometrically consistent regions to optimize the final scene.

[0041] (1) Technical details of uncertainty quantification dynamic scene optimization

[0042] like Figure 1 As shown in the middle part, the background and foreground objects rendered by the MARS baseline are input into the uncertainty field to generate an uncertainty heat map reflecting local reliability. Design a threshold adaptive algorithm based on Filtering. We define a dynamic threshold τ, which is updated per scene based on lighting, motion, and reflection intensity:

[0043]

[0044] This approach improves overall scene quality by removing low-confidence areas (e.g., overexposed reflections) while retaining high-fidelity geometry and textures.

[0045] (2) Technical details of rendering composition

[0046] like Figure 2 As shown, the four parts of the image are the original input image (upper left), the background rendered image (upper right), the foreground rendered image (lower left) and the combined rendering result (lower right).

[0047] MARS achieves foreground and background separation through instance segmentation and scene flow prediction. It processes input data such as RGB images, sensor poses, and object trajectories as described above, uses instance segmentation to identify dynamic objects (here, vehicles), and tracks their motion through scene flow prediction. Static backgrounds such as roads and buildings are modeled separately. Detailed expansion: The input includes RGB images (I), sensor poses (P), and object trajectories (3D bounding boxes, categories, and instance IDs). Object trajectories are provided by vehicle-side or roadside sensors, with the number of frames being T and the number of tracking instances being N. The input sequence is instance segmented, and based on this, the motion trajectory of dynamic objects is tracked through scene flow prediction. Scene flow prediction estimates the 3D displacement and deformation of objects between consecutive frames, and is suitable for handling non-rigid motion and occlusion. This process provides temporal consistency for NeRF dynamic modeling, ensuring accurate modeling of dynamic foreground.

[0048] For a given 3D position x and view direction d, the radiance field is encoded as: σ(x, d) and f(x, d), where σ(x, d) represents the volume density at position x and direction d, and f(x, d) represents the view-dependent color at the same position and direction. Parameters θ and φ are optimized using MARS's decomposed foreground and background representations. These parameters are conditioned on segmentation data, ensuring that static background and dynamic foreground are modeled independently but can be recombined for rendering. The optimization process can be expressed as:

[0049] Radiance Field=f(θ,φ∣Segmentation Data),

[0050] Where f is the neural network function of NeRF, and the conditioned input includes the results of instance segmentation and scene flow prediction.

[0051] The separated components are then combined through a rendering pipeline to produce the final scene. Static background and dynamic foreground objects are rendered separately and then composited into the final image. Figure 2 This process is shown, including the ground truth image, the extracted background (background), the isolated object (foreground), and the combined image. The separation process can be conceptually represented as:

[0052]

[0053] where Background is the static radiance field, Foreground i represents the radiance field of the ith dynamic object, and N is the number of tracked instances. This decomposition ensures that each component can be optimized independently while maintaining the consistency of the scene as a whole.

[0054] (3) Details of Specular and Diffuse Reflection

[0055] As shown in Figure 3 , specular reflection occurs on smooth surfaces (such as car windows or water surfaces), where light is reflected at a specific angle, producing highlights or mirror effects; diffuse reflection occurs on rough surfaces (such as roads or walls), where light is scattered uniformly and does not depend on the viewing angle. Specular reflection uses the Cook-Torrance model:

[0056]

[0057] where D represents the microfacet distribution, F is the Fresnel term, G is the geometric occlusion term, and n, v, l are the surface normal, view direction, and light source direction vectors, respectively.

[0058] Diffuse reflection uses the Lambertian model:

[0059] L diffuse ∝max(0,n·l)

[0060] The final color is calculated in combination with the surface albedo.

[0061] The total radiance is the sum of specular and diffuse reflection:

[0062] L=L diffuse +L specular

[0063] By decomposing the radiance into specular and diffuse reflection components, combined with high dynamic range (HDR) calibration and a specialized rendering loss function, the reconstruction system can significantly reduce artifacts and improve the visual fidelity of autonomous driving simulation, especially when dealing with complex reflective surfaces.

[0064] (4) Presentation of Scene Reconstruction Experimental Results

[0065] To evaluate the effectiveness of integrating Bayes' Rays into the MARS simulator to mitigate artifacts, we conducted a series of experiments. These experiments aimed to assess the impact of uncertainty quantification on rendering quality and artifact reduction. The MARS simulator was configured with its default instance-aware Neural Radiance Field (NeRF) architecture, modeling background and foreground instances independently. Bayes' Rays were implemented as a post-processing module to compute the volume uncertainty field. The integration included training NeRF models for background and foreground instances, followed by uncertainty-guided rendering adjustments, such as increasing the sampling density in high-uncertainty regions.

[0066] We used two widely recognized autonomous driving datasets: KITTI and VKITTI (Virtual KITTI2). Each scene was pre-processed to extract multi-view images, depth maps, and semantic annotations as inputs to the simulator. The models were trained using the default settings for MARS: 50,000 iterations, batch size 1024, Adam optimizer (learning rate 0.0001). The uncertainty field in Bayes' Rays used a sparse Hessian approximation, reducing the overhead to approximately 15% of the rendering time. In high-uncertainty regions (above the 90th percentile), the sampling density was increased by 20%, while other regions maintained the default density. The output included RGB images, depth maps, and semantic masks achieved through volume rendering. As Figure 6 shown, our results on the KITTI dataset for the image reconstruction task outperformed other baseline methods, with a PSNR value of 30.31, an improvement of 1.25 over the previously best-performing MARS, and an SSIM value of 0.894, an improvement of 0.009. Figure 5 As

[0067] As Figure 4 shown, we qualitatively and quantitatively compared our method to other advanced methods and baselines: NeRF, NeRF with timestamped input (denoted as NeRF+Time), NSG, SUDS, and the MARS baseline. The results showed that the proposed method modeled higher smoothness, and for glass materials or reflective car bodies, the reconstruction results were significantly better than other models.

[0068] Referring to the MARS training setting, in the image reconstruction setting, the training and test image sets are the same, while in the new perspective synthesis (NVS) task, frames that are not included in the training data are rendered. That is, every 4th, 2nd, and 4th frame are retained, and only one frame out of every four frames is used for training, i.e. 25%, 50%, and 75%. Following the standard evaluation protocol in image synthesis, we report quantitative evaluations of peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learning-perceptual image patch similarity (LPIPS) with our default settings. The results are shown in Figure 6 The image reconstruction task and Figure 7 and Figure 8 The NVS task in

[15] shows that our method outperforms the baseline method. The results show that when using 75% of the training data, our method achieves a PSNR of 27.91, an SSIM of 0.894, and a LPIPS of 0.056 on the KITTI dataset. Compared with the previous best method, the PSNR and SSIM are improved by 3.68 and 0.049 respectively, and the LPIPS is reduced by 0.104. Similarly, when using 75% of the training data, the method achieves a PSNR of 33.39, an SSIM of 0.953, and a LPIPS of 0.035 on the VKITTI dataset. Compared with the previous best method, the PSNR and SSIM are improved by 3.60 and 0.036 respectively, and the LPIPS is reduced by 0.053.

Claims

1. A Bayesian ray-based neural radiation field reconstruction method for autonomous driving scenes, characterized by: The following steps are involved: 1) Dynamic Modeling: Multi-view dynamic sequences are input into the MARS framework for instance segmentation and scene flow prediction, separating static background from dynamic foreground objects and constructing a conditional neural radiance field model. 2) Local spatial perturbations are applied to dynamic regions, and the changes in reconstruction loss are calculated using the Bayesian Laplace approximation to generate a volumetric uncertainty heatmap. 3) Based on this uncertainty heatmap, an adaptive thresholding algorithm is used to filter low-confidence regions, retaining geometrically consistent regions to optimize the final scene rendering, and combining it with a physically based rendering model to decompose the radiance component.

2. The Bayesian ray-based neural radiation field reconstruction method for autonomous driving scenes according to claim 1 is characterized in that: The multi-view dynamic sequence defined in step 1 includes RGB images from vehicle side or roadside sensors Sensor posture and object trajectories (3D bounding boxes Category {type ij } N×M and instance ID {idx ij } N×M ), where N is the number of frames and M is the number of tracking instances. The radiation field modeling uses σ(x) = f θ (x) represents the volume density, f(x,d)=g φ (x, d) denotes the view-dependent color, and parameters θ and φ are conditioned on the segmentation data from MARS.

3. The processing of multi-view dynamic sequence according to claim 1, characterized in that In step 1, dynamic scenes are preprocessed using instance segmentation and scene flow prediction to provide information for the NeRF model to separate static and dynamic components. and viewing direction d∈S 2 , the radiation field is encoded as: σ(x)=f θ (x),f(x,d)=g φ (x,d), Where σ represents the volume density, g φ Represents view-dependent color, θ and φ are optimized using the foreground-background separation representation of MARS, where θ represents the foreground and φ represents the background, enhancing control over instance trajectory and appearance.

4. The Bayesian ray-based neural radiation field reconstruction method for autonomous driving scenes according to claim 1 is characterized in that: In step 2, spatial perturbations are introduced into the dynamic region. For a point x, the perturbation position is x + δ, where δ is a small displacement. The reconstruction loss L(x) is defined as the photometric error between the predicted pixel color and the true pixel color. The change in loss due to perturbation is approximated using a second-order Taylor expansion: in is the Hessian matrix. Using the Laplace approximation, the uncertainty is estimated as the inverse of the diagonal of the approximate Hessian matrix: in represents the uncertainty estimate of position x, diag(―H(0)) ―1 It represents the inverse of the diagonal elements of the Hessian matrix, and efficiently calculates high uncertainty areas through backpropagation of the Jacobian term, which are highlighted in the heat map to guide subsequent optimization.

5. The method of displaying high uncertainty areas in a heat map according to claim 4, wherein: A physically based rendering layer was integrated for reflective surfaces using the Cook-Torrance microfacet model, with radiosity decomposed into diffuse and specular components: I(x,d)=I diffuse (x)+I specular (x,d), Where x represents the 3D position, d represents the viewing direction, and I diffuse is the diffuse reflection component, I diffuse is the specular reflection component. Specifically, the specular reflection term is: Where D(h) is the microfacet distribution, F(h) is the Fresnel term, G(x,d) is the geometric shadow, n is the surface normal, l and v are the light and view directions. We use the self-calibration of HDR-Plenoxels to correct the high dynamic range (HDR) reflections to optimize the composite rendering loss: Where λ1 is the balance pixel-level color accuracy, λ2 is the texture smoothness, and I pred Represents the model processing image, I gt Represents a real image.

6. The Bayesian ray-based neural radiation field reconstruction method for autonomous driving scenes according to claim 1 is characterized in that: In step 3, we generate an uncertainty heat map reflecting local reliability. Design a threshold adaptive algorithm based on For screening, we define a dynamic threshold τ, which is updated per scene based on lighting, motion, and reflection intensity: Delete low-confidence areas, retain high-fidelity geometry and textures, and improve the overall scene quality. Filterregion represents the filter area and Retain region represents the retained area.

7. The model generation for background and foreground objects according to claim 3, characterized in that: Static background and dynamic foreground are modeled independently but can be recombined for rendering. The optimization process can be expressed as: RadianceField=f(θ,φ∣SegmentationData), Where f is the NeRF neural network function, θ represents the foreground, φ represents the background, SegmentationData is the segmentation data, and the conditioned input includes the results of instance segmentation and scene flow prediction. The separated components are combined through the rendering pipeline to generate the final scene. The static background and dynamic foreground objects are rendered separately and then composited into the final image. Including the extracted background (background), isolated objects (foreground), and the combined image, the separation process can be conceptually represented as: Among them: Background is the static radiation field, Foreground i represents the radiance field of the i-th dynamic object, and N is the number of tracking instances.

8. The Bayesian ray-based neural radiation field reconstruction method for autonomous driving scenes according to claim 1 is characterized in that: The texture smoothness constraint adopts a block processing strategy in GPU parallel computing: the scene is divided into 32×32×16 voxel blocks, an independent CUDA thread warp is assigned to each block, and the gradient update amount is accumulated through atomic operations.

Images