Three-dimensional scene reconstruction method based on camera luminosity optimization

Abstract

The invention discloses a three-dimensional scene reconstruction method based on camera luminosity optimization. According to the method, firstly, multiple images of the same scene are represented through a radiation field model, then a depth regularization method is introduced, points obviously deviating from the surface of an object are inhibited, the overfitting problem is relieved, and a projection image at a corresponding view angle is obtained. Secondly, an external luminosity model is established by considering the distortion of a camera lens caused by the influence of dust, stains, other interferents or atomization, and an internal luminosity model is established by considering the internal distortion problem influenced by halation and non-uniformity of a sensor; by optimizing the radiation field characterization parameters and the camera luminosity model parameters, the camera luminosity parameters can be estimated in the three-dimensional reconstruction process, so that compared with the prior art, the method can separate and obtain a high-quality three-dimensional scene with higher camera luminosity distortion and robustness under the condition of camera luminosity distortion, especially under the condition of imaging degradation, the three-dimensional scene with higher camera luminosity distortion and robustness can be obtained. And high-quality 3D scene representation can also be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a method for representing a scene from multi-view images, and in particular to a three-dimensional scene reconstruction method based on camera photometric optimization. Background Art

[0002] In computer vision, from multiple views Figure 2 Reconstructing 3D scenes from 2D images is a critical task with a wide range of applications. Using various representation methods including point clouds, meshes, and radiance fields based on 2D images to reconstruct 3D scene representations is the basis for a series of tasks such as new perspective synthesis, simultaneous localization and mapping, and structure recovery from motion. Typically, these methods build models for the scenes to be reconstructed and the cameras that capture these scenes, respectively, and regard each input image as a projection from the 3D scene to the camera. Based on this camera and scene model, the neural radiance field method and the explicit radiance field method have made significant progress in learning scene radiance fields, achieving high-fidelity 3D scene representations under ideal data.

[0003] However, most existing reconstruction methods only focus on geometric parameters. Without considering the photometric parameters in the complete image formation process, the inherent photometric distortion in the camera imaging process can significantly degrade the image quality, such as lens attenuation (vignetting) and lens stains. If the photometric modeling of the camera itself is ignored when restoring the radiation field, the imaging quality will be affected in the presence of photometric distortion, resulting in poor reconstructed 3D scenes. At the same time, since the data used for radiation field reconstruction is usually captured by consumer-grade cameras, there is usually limited control over camera conditions in real-world scenes, so the problem of photometric distortion is ubiquitous.

[0004] For the camera photometric model, such as Figure 1 As shown in , after removing the geometric distortion, existing radiation field rendering methods usually assume that the camera is an ideal pinhole camera model, in which the light emitted from a point in space propagates along a straight line and is directly projected onto the sensor imaging plane to form an image. Figure 2 As shown in Figure 1, in a complete image formation pipeline, light passes through various components including lenses and image sensors, each of which affects the final imaging result. Factors such as lens stains, lens attenuation (vignetting), and non-uniform sensor response usually degrade image quality and introduce photometric distortion.

[0005] To solve this problem, some radiation field-based solutions simplify the radiation field through isotropic regularization and use affine brightness parameters to explain different exposures to improve photometric consistency. Although some photometric inconsistencies are related to the scene light source and can be fitted by the isotropic inconsistency of the 3D model, the problems caused by the camera itself often produce incorrect fitting results.

[0006] Cameras that have undergone initial factory calibration can compensate for inherent photometric distortion through the image signal processor, but changes in camera conditions over time may introduce parameter uncertainties. At the same time, modifying camera parameters will also introduce varying degrees of photometric effects. Figure 3 Comparison of vignetting under different aperture settings, where f refers to the camera focal length, T is the exposure time, and ISO is the sensitivity. It can be seen that the camera's inherent luminosity impact is different under different parameters.

[0007] Some methods start with the image itself to perform image restoration, defect removal, and vignette removal. These methods usually learn scene texture priors from large datasets and extract features to reconstruct or complete the image to the desired quality. However, these techniques are mainly designed for single-view image correction. When applied to 3D scene reconstruction, they may cause inconsistencies in multi-view tasks and ultimately affect the results of the 3D representation model. In addition, network-based methods may perform poorly when dealing with unseen scenes and objects, which limits their application in diverse scenarios, especially in image restoration effects. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention proposes a 3D scene reconstruction method based on camera photometric optimization, introduces camera internal and external photometric models, performs camera characterization, and effectively separates scene-irrelevant information from the 3D scene representation. At the same time, deep regularization is used to prevent the 3D scene representation from fitting irrelevant information. This helps to achieve high-quality 3D scene representation under imaging degradation conditions.

[0009] A three-dimensional scene reconstruction method based on camera photometric optimization specifically comprises the following steps:

[0010] Step 1: Characterize multiple images of the same scene using a three-dimensional Gaussian model to obtain the radiation value R(x) corresponding to the pixel point x in the image.

[0011] Step 2: A deep regularization scheme is designed. For each pixel, the inverse depth of the associated three-dimensional GS based on the mean and variance is calculated, and then a Gaussian function is applied to suppress points outside the range of 1 standard deviation and points that deviate significantly from the surface of the object to alleviate the overfitting problem.

[0012]

[0013] where c n , α n They represent the weighted value and opacity of the spherical harmonic color coefficient of the nth Gaussian sphere, respectively. ID represents the inverse depth (inv-depth) at the pixel value. ID n represents the inverse depth of the nth Gaussian sphere, α m Indicates the opacity of the mth Gaussian ball, m<n. N is the number of Gaussian balls, μ id :R→[0,1] and σ id :R→[0,1] is the mean and standard deviation of the inverse depth of the 3D GS in the line of sight direction.

[0014] Step 3: Establish a camera photometric model, including an internal photometric model and an external photometric model.

[0015] The external photometric model takes into account the distortion caused by dust, stains, other interferences or fogging of the camera lens:

[0016] B i (x) = S α (x)R(π(p))+S β (x)

[0017] x=π(p)

[0018] Where i represents the input image index, B i (x) is the radiation entering the camera, S α :R 2 →[0,1] represents the attenuation factor of the interferent, S β :R 2 →R simulates the radiation emitted by the interferer itself. p represents the three-dimensional position of the observation point, π:R 3 →R 2 is the projection function that maps p to pixel coordinate x in the pinhole camera model.

[0019] The internal photometric model takes into account the internal distortion issues affected by vignetting and sensor non-uniformity:

[0020] I i (x) = G (x) V (x) B i (x)

[0021] Among them I i (x) represents the pixel intensity observed from the i-th image, V: R 2 →[0,1] represents the lens attenuation function, G∶R 2 →[0,1] represents the response function.

[0022] The radiation value B obtained by the external photometric model iSubstituting (x) into the internal photometric model, we obtain the following camera photometric model:

[0023] I i (x) = G(x)V(x)S α (x)R(x)+G(x)V(x)S β (x)

[0024] =M α (x)R(x)+M β (x)

[0025] Among them, M α (x) = G(x)V(x)S α (x) represents the comprehensive attenuation factor, M β (x) = G(x)V(x)S β (x) represents the comprehensive bias term.

[0026] Step 4: Use a loss function based on direct image supervision Optimize the radiation field characterization parameters in step 1:

[0027]

[0028] in represents the pixel intensity of the input image, λ is the weight value of different loss terms, Represents the structural similarity loss function term.

[0029] The comprehensive attenuation factor and comprehensive bias term of the camera photometric model are optimized using a balanced loss function:

[0030]

[0031] Among them I α and I β M α (x) and M β The output of (x), I′ is the output of the internal photometric model with all 1s as input. and Respectively represent I′, I α and I β Image after Gaussian smoothing, ‖·‖2 is the binary norm loss.

[0032] Step 5: Input cameras and corresponding projection images of multiple viewpoints in a scene. For each viewpoint, first perform 3D scene representation, and then obtain the corresponding projection image after depth regularization. Input the projection images of multiple viewpoints into the camera photometric model trained in step 4, and output the 3D reconstruction result.

[0033] The present invention has the following beneficial effects:

[0034] 1. By introducing the camera characterization module, the camera photometric parameters can be estimated during the three-dimensional reconstruction process, so that compared with the existing technology, this method can separate the camera photometric distortion and obtain a high-quality three-dimensional scene with stronger robustness.

[0035] 2. Through the deep regularization method, a new and efficient scheme that can constrain the geometric distribution of the scene is proposed. Compared with the existing technology, it can efficiently separate the noise caused by the camera photometric error without affecting the scene fitting ability to meet the estimation of the camera photometric parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of imaging of a traditional camera model;

[0037] Figure 2 Schematic diagram of imaging of a camera model with photometric distortion;

[0038] Figure 3 Comparison of vignetting at different aperture settings;

[0039] Figure 4 The flowchart of the 3D scene reconstruction method based on camera photometric optimization is shown below;

[0040] Figure 5 It is a schematic diagram of Gaussian sphere distribution in the single pixel projection direction;

[0041] Figure 6 It is a display of input data under different interferences in the embodiment. DETAILED DESCRIPTION

[0042] The present invention is further explained below in conjunction with the accompanying drawings.

[0043] Combining the process of scene and camera representation training, a 3D scene reconstruction method based on camera photometric optimization is proposed. Figure 4 As shown, the red arrow represents the camera representation training and the black arrow represents the scene representation training. During the camera representation training process, a deep regularized projection is used to assist the camera fitting, and multiple view cameras are captured to obtain a set of projected images. This set of images is paired with the input image to adapt the camera representation, where the gradient is propagated only for the camera representation. In the model representation training, the camera representation is fixed, a single view is selected for projection, and the lens-specific photometric distortion is supplemented on the 3D Gaussian model to fit the input image. The specific steps are as follows:

[0044] Step 1: Characterize the radiation field of multiple images of the same scene. Characterize the scene as a set of three-dimensional Gaussian balls. Each three-dimensional Gaussian is characterized by a covariance matrix Σ and a center point μ in three-dimensional space. The radiation value R(x) corresponding to the pixel point x in the image is obtained:

[0045]

[0046] where c n , α n Respectively represent the weighted value and opacity of the spherical harmonic color coefficient of the nth Gaussian sphere. m Indicates the opacity of the mth Gaussian sphere, m<n. N is the number of Gaussian spheres.

[0047] Step 2: In order to alleviate the overfitting problem of the model, a deep regularization technique based on prior knowledge is proposed to reduce the interference of floating points by minimizing the contribution of overfitting points in the projected image.

[0048] Figure 5 The Gaussian sphere distribution in the projection direction of a single pixel shows that there is a certain distribution relationship in the space around the object surface. Since the radiation field is represented by multiple three-dimensional Gaussian spheres, the opacity near the object surface should roughly follow the Gaussian distribution during the projection process. Based on this feature, a Gaussian distribution-based depth regularization is designed: for each pixel, the inverse depth of the associated three-dimensional GS based on the mean and variance is calculated, and then the Gaussian function is applied to suppress points outside the range of 1 standard deviation:

[0049]

[0050] Among them, μ id :R→[0,1] and σ id :R→[0,1] is the mean and standard deviation of the inverse depth of the 3D GS in the line of sight direction. n Represents the inverse depth of the nth Gaussian sphere.

[0051] The camera photometric parameters are optimized by adjusting the opacity according to the Gaussian function value, which effectively and dynamically suppresses points that significantly deviate from the object surface, thereby alleviating the overfitting problem and making the camera model learn a cleaner model, but it does not essentially reduce the model's fitting ability.

[0052] Step 3: For the geometric part, use the pinhole camera model to simplify the geometric camera model and project the 3D points onto the 2D image:

[0053] x=π(p)

[0054] Among them, p represents the three-dimensional position of the observation point, π:R 3 →R 2is the projection function that maps p to pixel coordinate x. A camera photometric model is established to map the actual radiance omitted from the position in front of the lens to its corresponding intensity value.

[0055] Since photometric distortion may come from inside and outside the camera, the camera photometric model is divided into internal photometric model and external photometric model.

[0056] Camera lenses may accumulate dust, stains, or other interferences over time, or may be affected by unexpected problems such as lens fogging or obstructions on the lens surface, which can introduce distortion and degrade image quality. These surface distortions may appear as irregular patterns or local variations in image intensity, which can be thought of as partial loss of scene radiance information as it passes through the interference, while the interference itself emits additional radiation to the camera. Incorporating surface distortion parameters into the camera model, the external photometric model is expressed as:

[0057] B i (x) = S α (x)R(π(p))+S β (x)

[0058] Among them, B i (x) is the radiation entering the camera, and i represents the input image index. α :R 2 →[0,1] represents the attenuation factor of the interferent, S β :R 2 →R simulates the radiation emitted by the interferer itself, characterizes the radiance of the scene radiation field with the observation direction, and represents the radiation information emitted by the spatial point. Therefore, the effect of photometric distortion on the scene radiation received by the camera is captured by the external photometric model. It should be noted that although the external photometric model includes the observation direction related to the camera geometric extrinsics, this information only involves sampling from the radiation field and does not depend on the camera geometric parameters.

[0059] The main factors causing the camera's internal photometric distortion are vignetting and sensor non-uniformity. The internal photometric model is characterized as:

[0060] I i (x) = G (x) V (x) B i (x)

[0061] Among them I i (x) represents the pixel intensity observed from image i, V:R 2 →[0,1] represents the lens attenuation function, G∶R 2 →[0,1] represents the response function, which reflects the changes in the lens and sensor response at different pixels. Because the radiation scalar entering the camera is unknown, the response function G maps this value to the interval [0,1] to characterize the degree of radiation response.

[0062] The radiation value B obtained by the external photometric model i Substituting (x) into the internal photometric model, we obtain the following camera photometric model:

[0063] I i (x) = G(x)V(x)S α (x)R(p,σ)+G(x)V(x)S β (x)

[0064] Simplify the camera photometric model:

[0065] I i (x) = M α (x)R(p,σ)+M β (x)

[0066] Among them, M α (x) = G(x)V(x)S α (x) represents the comprehensive attenuation factor, M β (x) = G(x)V(x)S β (x) Characterizes the comprehensive bias term. The simplified camera photometric model shows that although there are multiple photometric distortion factors, they can be integrated into a single linear transformation model for each pixel. The simplified model not only simplifies the photometric modeling process, but also simplifies the design of subsequent 3D scene-camera characterization methods.

[0067] In actual camera imaging, the camera mainly focuses on the captured object, that is, focuses on the scene, so the stains on the lens usually appear blurred. In addition, the vignetting of the lens and the non-uniform response of the image sensor are usually smooth, and the photometric parameters of adjacent pixels are correlated in the final captured image. Therefore, a shallow multi-layer perceptron model mlp is used respectively. α and mlp β To fit the camera photometric model M α (x) and M β (x):

[0068] I i (x) = mlp α (x)R(x)+mlp β (x)

[0069] Among them, mlp α (x) and mlp β (x) represent the outputs of two multi-layer perceptron models respectively. The multi-layer perceptron model is used to model various photometric distortion factors that affect the input radiation and complete scaling and bias correction.

[0070] Step 4: Use different loss functions to optimize the three-dimensional Gaussian scene parameters and camera photometric model parameters in step 1.

[0071] First, we use a loss function based on direct image supervision Optimize the radiation field characterization parameters in step 1:

[0072]

[0073] in represents the pixel intensity of the input image, λ is the weight value of different loss terms, Represents the structural similarity loss function term.

[0074] Second, since photometric distortions usually exhibit smooth properties, a smoothness loss is used To constrain the camera photometric parameters during the optimization process:

[0075]

[0076] Among them I α and I β M α (x) and M β The output of (x), I′ is the output of the internal photometric model with a white image as input. and Respectively represent I′, I α and I β Image after Gaussian smoothing, ‖·‖2 is the binary norm loss.

[0077] The camera photometric model requires multiple views as input for fitting, while the 3D reconstruction model usually uses a single image with specific camera coordinates and angles. Therefore, the optimization process uses an iterative training method to optimize the camera photometric parameters and the scene photometric parameters in turn. For the camera model, a set of viewpoints is selected according to the current state of the 3D reconstruction model, the scene is projected to obtain a rendered image, and the loss between the final rendered image obtained by the camera model and the corresponding input image is calculated. In each iteration, the camera model is first updated and trained, and then the update is paused to continue the training of the 3D reconstruction model.

[0078] Step 5: Using a Sony camera, we captured 40 to 60 images of different scenes, including different types of photometric distortion, such as camera vignetting, fingerprints, dirt, and water droplets on the lens surface. Figure 6 As shown in the figure. Introducing different types of photometric distortion can evaluate their impact on rendering quality and comprehensively evaluate model performance. In addition, corresponding data is supplemented by mobile phone lenses and logical wide-angle lenses to evaluate the generalization ability of the method for different lenses.

[0079] In this embodiment, multiple baseline models are selected for comparative experiments, where implicit representations include mip-NeRF, INGP, Rip-NeRF and NeuRBF, and explicit representations include 3DGS and 3DGS-mcmc. Peak signal-to-noise ratio (PSNR) and structure similarity index (SSIM) are selected as evaluation indicators. Table 1 shows the comparison results of 3D reconstruction of different baseline models under different external photometric distortions, and Table 2 shows the comparison results of 3D reconstruction of different baseline models under multiple scenes with internal photometric distortion (vignetting):

[0080]

[0081]

[0082] Table 1

[0083]

[0084] Table 2

[0085] The results in Table 1 show that this method is more robust and effective than other methods in the presence of mixed photometric distortions, especially in complex photometric distortions such as fingerprints and water droplets, and outperforms other state-of-the-art methods. At the same time, according to the results in Table 2, it can be seen that this method can effectively reduce the impact of subtle vignetting on three-dimensional representation, even in scenes that are difficult for human observers to detect. Therefore, the experimental results prove the significant advantages of this method in various scenes and photometric distortions. And for various scenes, this method has a certain improvement over the baseline method (3DGS-mcmc).

Claims

1. A three-dimensional scene reconstruction method based on camera photometric optimization, characterized in that: The specific steps include: Step 1: multiple images of the same scene are characterized by a radiation field model to obtain the radiation value R(x) corresponding to the pixel point x in the image; Step 2: Establish a camera photometric model, including an internal photometric model and an external photometric model; The external photometric model is: B i (x)=S α (x)R(x)+S β (x) Where i represents the input image index, B i (x) is the radiation entering the camera, S α :R 2 →[0,1] represents the attenuation factor of the interferent, S β :R 2 →R simulates the radiation emitted by the interferer itself; The internal photometric model is: I i (x)=G(x)V(x)B i (x) Among them I i (x) represents the pixel intensity observed from the i-th image, V: R 2 →[0,1] represents the lens attenuation function, G∶R 2 →[0,1] represents the response function; The radiation value B obtained by the external photometric model i Substituting (x) into the internal photometric model, we obtain the following camera photometric model: I i (x)=G(x)V(x)S α (x)R(x)+G(x)V(x)S β (x) =M α (x)R(x)+M β (x) Among them, M α (x) = G(x)V(x)S α (x) represents the comprehensive attenuation factor, M β (x) = G(x)V(x)S β (x) represents the comprehensive bias term; Step 3: Use a loss function based on direct image supervision Optimize the radiation field characterization parameters in step 1; use the balanced loss function Optimize the comprehensive attenuation factor and comprehensive bias term of the camera photometric model; Step 4: Input cameras with multiple viewpoints and corresponding projection images in a scene; for each viewpoint, perform three-dimensional scene representation and obtain the corresponding projection image; input the projection images of multiple viewpoints into the camera photometric model trained in step 3, and output the three-dimensional reconstruction result.

2. A three-dimensional scene reconstruction method based on camera photometric optimization as claimed in claim 1, characterized in that: Use the three-dimensional Gaussian model to characterize the image and obtain the radiation value R(x) corresponding to the pixel point x in the image: where c n , α n Respectively represent the weighted value and opacity of the spherical harmonic color coefficient of the nth Gaussian sphere; α m Indicates the opacity of the mth Gaussian sphere, m<n; N is the number of Gaussian spheres.

3. A three-dimensional scene reconstruction method based on camera photometric optimization as claimed in claim 2, characterized in that: A deep regularization scheme is designed. For each pixel, the inverse depth of the associated three-dimensional GS based on the mean and variance is calculated, and then a Gaussian function is applied to suppress points outside the range of 1 standard deviation, suppress points that deviate significantly from the surface of the object, and alleviate the overfitting problem: where c n , α n Respectively represent the weighted value and opacity of the spherical harmonic color coefficient of the nth Gaussian sphere, id represents the inverse depth, id n represents the inverse depth of the nth Gaussian sphere, α m Indicates the opacity of the mth Gaussian ball, m<n; N is the number of Gaussian balls, μ id :R→[0,1] and σ id :R→[0,1] is the mean and standard deviation of the inverse depth of the 3D GS in the line of sight direction.

4. A three-dimensional scene reconstruction method based on camera photometric optimization as claimed in claim 1, characterized in that: Map p to pixel coordinate x using the pinhole camera model: x=π(p) p represents the three-dimensional position of the observation point, π:R 3 →R 2 is the projection function that maps p to pixel coordinate x.

5. A three-dimensional scene reconstruction method based on camera photometric optimization as claimed in claim 1, characterized in that: Use shallow multi-layer perceptron model mlp respectively α and mlp β To fit the camera photometric model M α (x) and M β (x): I i (x)=mlp α (x)R(x)+mlp β (x) Among them, mlp α (x) and mlp β (x) represent the outputs of two multi-layer perceptron models respectively. The multi-layer perceptron model is used to model various photometric distortion factors that affect the input radiation and complete scaling and bias correction.

6. A three-dimensional scene reconstruction method based on camera photometric optimization as claimed in claim 1, characterized in that: The loss function based on direct image supervision for: in represents the pixel intensity of the input image, λ is the weight value of different loss terms, Represents the structural similarity loss function term.

7. A three-dimensional scene reconstruction method based on camera photometric optimization as claimed in claim 1, characterized in that: The balance loss function for: Among them I α and I β M α (x) and M β The output of (x), I′ is the output of the internal photometric model with all 1s as input; and Respectively represent I′, I α and I β Image after Gaussian smoothing, ‖·‖2 is the binary norm loss.

Images