A light-robust 3D GS method for novel view image synthesis

Abstract

The present application relates to the technical field of computer image processing, and particularly relates to a light robustness 3DGS method for new view image synthesis, which can generate high-quality new view images under extreme or complex lighting conditions, and considering that direct enhancement of individual images in two-dimensional image space may lead to view inconsistency, the present application introduces a bilateral grid in the training stage to simulate the image processing flow, and removes it in the inference stage to effectively eliminate view inconsistency.In addition, the present application introduces and loss to generate new view images under normal lighting conditions with higher quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer image processing, and particularly relates to a light robustness 3DGS method for new view image synthesis. BACKGROUND

[0002] Extracting accurate three-dimensional scene representation from a series of two-dimensional images and generating new views is a core challenge in computer vision, which has wide applications in multiple fields, including autonomous driving, medical imaging, robotics, and virtual reality. The emergence of NeRF (Neural Radiance Fields) has significantly accelerated the development of three-dimensional reconstruction and new view synthesis, effectively interpreting three-dimensional scenes from two-dimensional images and generating realistic new views, injecting new vitality into these research fields. NeRF uses multi-layer perceptron to implicitly encode volume density and color, achieving high-quality new view generation. This breakthrough technology has made significant progress in multiple fields, driving the development of the intersection of computer vision and computer graphics.

[0003] Recently, 3DGS has shown comparable expressiveness to NeRF while being hundreds of times faster in training and inference, showing its potential to be a better choice than NeRF. Unlike NeRF's implicit surface representation, 3DGS adopts an explicit three-dimensional representation, efficiently constructing three-dimensional space through a simplified and logically clear framework. 3DGS not only achieves high-quality reconstruction and rendering, but also provides super-real-time rendering and inference speed, effectively addressing the significant problem of NeRF in time efficiency. Despite the significant progress of 3DGS in new view synthesis, its output quality still highly depends on the quality of input images. In extreme lighting conditions such as low light or overexposure, the reduction of visible information and the lack of details in input images pose a huge challenge to accurately recover geometric structures. A simple solution is to perform low-light enhancement or overexposure correction in two-dimensional image space to enrich image details. However, this approach introduces a key problem: existing two-dimensional processing methods usually enhance each view independently, leading to inconsistencies between views. Therefore, when observing the same location from different perspectives, contradictory results may occur, resulting in artifacts in the reconstructed image and significantly reducing the quality of synthesized new view images.

[0004] In this work, we propose a new method that integrates two-dimensional image processing techniques to enhance visibility while maintaining multi-view consistency, effectively bridging the gap between two-dimensional space and three-dimensional representation. SUMMARY

[0005] The present application provides a light-robust 3DGS method for new-view image synthesis to make up for the deficiencies in the prior art and achieve the above functions.

[0006] The present application is realized by the following technical solutions:

[0007] A light-robust 3DGS method for new-view image synthesis includes the following steps:

[0008] Capture N images under extreme lighting conditions, and use low-light enhancement or overexposure correction to obtain a processed image set according to scene requirements , wherein represents a multi-view image set enhanced separately;

[0009] Subsequently, these images are processed by software COLMAP to generate an initial point cloud as a starting result;

[0010] Subsequently, N bilateral grids are initialized for processing each training image in the pixel level .

[0011] For a specific pixel (u,v) in the Lth training image, wherein it obtains the rendered evaluation color in 3DGS using a slice-based differentiable rasterization rendering algorithm , the slice operation expression is as follows:

[0012] ,

[0013] ,

[0014] wherein, for the guide function g(C), a logarithmic luminance formula is adopted to provide compression of the luminance dynamic range, aiming to process scenes with high illumination intensity, thereby achieving a more natural visual effect; set the value of c to 6, and the guide function g(C) is:

[0015] ,

[0016] Apply affine transformation to obtain the processed color after slicing.

[0017] Further, in order to better realize the present application, the bilateral grid is compatible with the neural network, and the bilateral grid maps image data into a four-dimensional tensor , wherein W, H and M represent the resolution of the grid, 12 corresponds to the length of the 3x4 affine color transformation matrix, and the color value of each pixel used to locate the corresponding color transform position in the bilateral grid, which is achieved by computing a weighted sum based on color values and spatial positions;

[0018] the weight of the weighting is derived from the hat function a linear interpolation kernel calculation, thus ensuring that only color-similar neighboring pixels will have an impact on the current pixel;

[0019] Finally, the processed color is generated by multiplying the extracted affine transform matrix with the color value of the pixel.

[0020] Further, in order to better achieve the present application, a total variation regularization term is introduced to control the smoothness of the bilateral grid, and the final objective function is represented as:

[0021] ,

[0022] ,

[0023] representing the finite difference operator, set as a parameter 10 to obtain the best practical application results.

[0024] Further, in order to better achieve the present application, a color consistency loss function is used:

[0025] ,

[0026] to optimize the bilateral grid for processing and subsequent removal from which the color balance and overall appearance saturation in the reconstructed image are restored.

[0027] Further, in order to better achieve the present application, the predicted normal illumination image is enhanced by a contrast loss function to balance the balance between local details and overall contrast, where: ,

[0028] C represents the number of channels, denotes the variance of channel C, which is set to 0.3 for overexposed scenes and 0.4 for low-light scenes.

[0029] Further, in order to better achieve the present application, the color consistency loss and the contrast loss are solved by optimizing the loss function, and the loss function is composed of three parts: the rendering loss and two unsupervised brightness correction losses and ; the overall training loss is represented as:

[0030] ,

[0031] .

[0032] The beneficial effects of the present application are:

[0033] The present application can generate high-quality novel view images under extreme or complex lighting conditions, considering that direct enhancement of individual images in two-dimensional image space may lead to view inconsistency, the present application introduces a bilateral grid in the training stage to simulate the image processing flow, and removes it in the inference stage to effectively eliminate view inconsistency. In addition, the present application introduces and to generate novel view images under normal lighting conditions with higher quality. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a schematic diagram of the principle of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0036] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0037] (1) About 3D Gaussian sputtering:

[0038] 3DGS is prior art, all the following descriptions are knowledge background to help understand this technology.

[0039] When the density of point clouds is high enough, the geometry and appearance of a three-dimensional object can be described from any viewing angle. However, when describing complex or smooth regions using relatively sparse data, points in the point cloud need to be endowed with a certain volume. If each point is regarded as a shape following a certain distribution, projecting these shapes onto a two-dimensional plane can generate high-quality images. 3DGS provides a solution in which each point is constructed as the center of a shape conforming to a three-dimensional Gaussian distribution to describe a three-dimensional object. Real-time rendering is achieved through differentiable rasterization. Each 3DGS primitive {G} is defined by the following parameters: Gaussian center μ, opacity α, scaling ratio described by covariance matrix Σ, and color c modeled using spherical harmonics. The basis function of the Gaussian primitive can be represented by the following formula:

[0040] ,

[0041] To represent a multivariate Gaussian distribution, only spatial coordinates and a covariance matrix are required. The covariance matrix can be represented as where R represents a rotation matrix and S is a scaling matrix. For a three-dimensional space point, only a 3D scaling vector s and a quaternion q need to be stored. In the rendering stage, a block-based differentiable rasterization algorithm is adopted. Before rendering, 3D Gaussian points are pre-ordered according to their spatial depth. The final color is calculated through alpha blending, as follows:

[0042] ,

[0043] where, is the learned color, represents the opacity of the i-th two-dimensional Gaussian distribution probability density weighted by the target pixel position . The loss function is composed of the weighted combination of the L1 loss of color reconstruction and the D-SSIM loss of the reconstructed image, and the specific form is as follows:

[0044] .

[0045] (2) Color Prior:

[0046] Referring to the description in (1), 3DGS constructs a shape conforming to a three-dimensional Gaussian distribution centered on each point, which requires processing multiple view images to generate an initial point cloud. However, it is challenging to accurately estimate camera parameters and generate a complete initial point cloud for images taken under extreme lighting conditions. Some NeRF-based research, such as AlethNeRF and LLNeRF, improves the neural radiance field under poor lighting conditions and achieves good results. However, these methods assume that accurate camera poses can be obtained, and usually require paired images to work effectively, which is often difficult to achieve in real-world scenarios. In particular, under low-light conditions, the extreme lack of visibility can result in a highly sparse point cloud. Using existing 2D image processing methods to preliminarily enhance the images and then using COLMAP to generate a point cloud significantly improves the point cloud and its structure. Although there is a deviation compared to the camera pose and the number of point clouds obtained under normal lighting, the impact on the reconstruction result is small.

[0047] (3) Reference Figure 1 The present application comprises the following steps:

[0048] Capture N images under extreme lighting conditions, according to the scene requirements, use low-light enhancement or overexposure correction to obtain a set of processed images wherein represents a set of multi-view images after individual enhancement;

[0049] Subsequently, these images are processed by software COLMAP to generate an initial point cloud as a starting result;

[0050] Subsequently, N bilateral grids are initialized for processing at the pixel level for each training image in .

[0051] For a specific pixel (u, v) in the Lth training image, where it obtains the rendered evaluation color in 3DGS using a slice-based differentiable rasterization rendering algorithm,

[0052] ,

[0053] ,

[0054] wherein, for the guide function g(C), a logarithmic luminance formula is adopted to provide compression of the luminance dynamic range, aiming to handle scenes with high lighting intensity, thereby achieving a more natural visual effect; set the value of c to 6, and the guide function g(C) is: ,

[0055] Apply affine transformation After slicing, the processed color is obtained .

[0056] The bilateral grid is compatible with neural networks, and the bilateral grid maps the image data into a four-dimensional tensor where W, H and M represent the resolution of the grid, 12 corresponds to the length of the 3x4 affine color transformation matrix, and the color value of each pixel is used to locate the corresponding color transformation position in the bilateral grid, and this process is achieved by calculating a weighted sum based on color values and spatial positions;

[0057] The weight of the weighting is calculated by a linear interpolation kernel derived from the hat function , so as to ensure that only color-similar neighboring pixels will have an impact on the current pixel;

[0058] Finally, the processed color is generated by multiplying the extracted affine transformation matrix with the color value of the pixel.

[0059] A total variation regularization term is introduced to control the smoothness of the bilateral grid, and the final objective function is represented as:

[0060] ,

[0061] to train,

[0062] representing the finite difference operator, set as a parameter 10 to obtain the best practical application results;

[0063] The color consistency loss function is used:

[0064] to optimize the bilateral grid for processing and then remove it, and the color balance and overall appearance saturation in the reconstructed image.

[0065] The predicted normal illumination image is enhanced by the contrast loss function to balance the balance between local details and overall contrast, where: ,

[0066] C represents the number of channels, represents the variance of channel C, and the overexposed scene is set to 0.3 and the low-light scene is set to 0.4.

[0067] The color consistency loss and contrast loss are solved by optimizing the loss function, and the loss function consists of three parts: rendering loss and two unsupervised brightness correction losses and ; the overall training loss is represented as:

[0068] ,

[0069] .

[0070] The above introduces a bilateral grid simulation enhancement process during training, the bilateral grid is removed during testing to eliminate inconsistent enhancement.

[0071] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it, and other modifications or equivalent replacements of the technical solutions of the present application made by those skilled in the art should be covered in the scope of the claims of the present application as long as they do not deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A lighting-robust 3D GS method for novel view image synthesis, characterized in that: The method comprises the following steps: Capture N images under extreme lighting conditions, use low light enhancement or overexposure correction to obtain a processed set of images according to the scene requirements wherein denotes a set of multi-view images enhanced individually; Subsequently, the images are processed by software COLMAP to generate an initial point cloud as a starting result; Subsequently, N bilateral grids are initialized for processing each training image in the set at the pixel level; For a particular pixel (u, v) in the Lth training image, where it is rendered using a slice-based differentiable rasterization rendering algorithm in 3DGS to obtain the evaluated color The slice operation expression is as follows: ， ， Wherein, for the guide function g(C), a logarithmic brightness formula is adopted to provide compression of the brightness dynamic range, aiming to process scenes with high illumination intensity, so as to achieve a more natural visual effect; the c value is set to 6, and the guide function g(C) is: ， Applying an affine transformation After slicing, the processed color is obtained .

2. The illumination-robust 3D GS method for novel view image synthesis of claim 1, wherein: The bilateral grid is compatible with a neural network, the bilateral grid maps image data as a four-dimensional tensor where W, H and M represent the resolution of the grid, 12 corresponds to the length of the 3x4 affine color transformation matrix, the color value of each pixel is used to locate the corresponding color transformation position in the bilateral grid, this process is achieved by calculating a weighted sum based on the color value and the spatial position; Weighted weights The linear interpolation kernel is derived from the hat function The linear interpolation kernel is derived from the hat function Finally, the processed color is generated by multiplying the extracted affine transformation matrix with the color value of the pixel.

3. The illumination-robust 3D GS method for novel view image synthesis of claim 2, wherein: A total variation regularization term is introduced to control the smoothness of the bilateral grid, and the final target function is represented as: ， ， representing a finite difference operator, and are set as parameters 10 to obtain the best practical application results.

4. The illumination-robust 3D GS method for novel view image synthesis of claim 3, wherein: The color consistency loss function is utilized: ， When the bilateral grid is optimized and then deleted from it, the color balance and overall appearance saturation in the reconstructed image are processed.

5. The illumination-robust 3D GS method for novel view image synthesis of claim 4, wherein: Predicted normal illumination image, enhanced by a contrast loss function to enhance the balance between local details and overall contrast, wherein: , C represents the number of channels, C represents the variance of the channel, which is set to 0.3 for overexposed scenes and 0.4 for low light scenes.

6. The illumination-robust 3D GS method for novel view image synthesis of claim 5, wherein: Color consistency loss and contrast loss are solved by optimizing the loss function, which consists of three parts: rendering loss and two unsupervised brightness correction losses and ; the overall training loss is represented as: ， 。

Images