A method for three-dimensional scene reconstruction based on camera photometric optimization

Abstract

The application discloses a kind of three-dimensional scene reconstruction methods based on camera photometric optimization.The method is first characterized by radiation field model to multiple images of the same scene, then introduce depth regularization method, inhibit the point that significantly deviates from object surface, relieve overfitting problem, obtain the projection image under corresponding view angle.Then consider that the distortion caused by dust, stain, he interference or atomization of camera lens is established external photometric model, consider the internal distortion problem of being affected by vignetting and sensor non-uniformity and establish internal photometric model.Through the optimization of radiation field characterization parameter and camera photometric model parameter, camera photometric parameter can be estimated in three-dimensional reconstruction process, so that the method compared with prior art can separate camera photometric distortion and more robust high-quality three-dimensional scene under camera photometric distortion, especially under imaging degradation condition, high-quality 3D scene representation can also be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and relates to a method for representing a scene from multi-view images, in particular to a three-dimensional scene reconstruction method based on camera photometric optimization. BACKGROUND

[0002] In computer vision, reconstructing a three-dimensional scene from multi-view images is a key task with a wide range of applications. Various representation methods including point clouds, meshes, and radiance fields are used to reconstruct a three-dimensional scene representation based on two-dimensional images, which is the basis for a series of tasks such as novel view synthesis, simultaneous localization and mapping, and structure from motion. Generally, these methods will construct a model for the scene to be reconstructed and the camera that captures the scene, and each input image is regarded as a projection from the three-dimensional scene to the camera. Based on this camera and scene model, methods of neural radiance fields and explicit radiance fields have made significant progress in learning scene radiance fields, achieving high-fidelity three-dimensional scene representation under ideal data. Figure Two However, most existing reconstruction methods only focus on geometric parameters. Without considering photometric parameters in the complete image formation process, the inherent photometric distortion in the camera imaging process will significantly reduce the image quality, such as lens attenuation (vignetting) and lens stains. If the photometric modeling of the camera itself is ignored when recovering the radiance field, it will affect the imaging quality in the presence of photometric distortion, and further lead to poor results of the reconstructed three-dimensional scene. At the same time, since the data used for radiance field reconstruction is usually captured by consumer-grade cameras, there is usually limited control over camera conditions in real-world scenes, so the problem of photometric distortion is ubiquitous.

[0003] For camera photometric models, as shown in

[0004] After removing geometric distortion, existing radiance field rendering methods usually assume that the camera is an ideal pinhole camera model, in which light rays emitted from a point in space propagate along a straight line and are directly projected onto the imaging plane of the sensor to form an image. However, as shown in Figure One In the complete image formation pipeline, light rays pass through various components including lenses and image sensors, and each component affects the final imaging result. Factors such as lens stains, lens attenuation (vignetting), and sensor response non-uniformity often reduce image quality and introduce photometric distortion. Figure Two

[0005] ​To solve this problem, some radiation field-based schemes simplify the radiation field through isotropic regularization and use affine intensity parameters to explain different exposures to improve photometric consistency. Although part of the photometric inconsistency is related to the scene light source and can be fitted by the anisotropy of the three-dimensional model, the problems caused by the camera itself often lead to false fitting results.

[0006] The camera that has been preliminarily calibrated in the factory can compensate for the inherent photometric distortion through the image signal processor, but the change of the camera condition over time may introduce parameter uncertainty, and at the same time, the modification of the parameters of the camera will also introduce different degrees of photometric influence, Figure Three The comparison of the vignetting under different apertures can be seen, where f refers to the focal length of the camera, T is the exposure time, and ISO is the sensitivity. It can be seen that the inherent photometric influence of the camera under different parameters is also different.

[0007] Some methods start from the image itself and perform image restoration, defect removal and vignetting removal. These methods usually learn scene texture priors from large data sets and extract features to reconstruct or complete the image to the required quality. However, these techniques are mainly designed for single-view image correction, and when they are applied to three-dimensional scene reconstruction, they may cause inconsistency in multi-view tasks and ultimately affect the results of the three-dimensional representation model. In addition, network-based methods may not perform well when dealing with unseen scenes and objects, which limits their application in diversified scenes, especially in terms of image restoration effect. SUMMARY

[0008] In view of the shortcomings of the prior art, the present application proposes a three-dimensional scene reconstruction method based on camera photometric optimization, which introduces camera internal and external photometric models for camera representation, effectively separating scene-independent information from three-dimensional scene representation. At the same time, deep regularization is used to prevent three-dimensional scene representation from fitting irrelevant information. It is helpful to achieve high-quality three-dimensional scene representation under imaging degradation conditions.

[0009] A three-dimensional scene reconstruction method based on camera photometric optimization, specifically comprising the following steps:

[0010] Step 1, representing multiple images of the same scene through a three-dimensional Gaussian model to obtain the radiation value R(x) corresponding to the pixel point x in the image.

[0011] Step 2, a deep regularization scheme is designed, which calculates the inverse depth of the associated three-dimensional GS based on the mean and variance for each pixel point, and then applies a Gaussian function to suppress points located outside the range of 1 standard deviation, suppress points that deviate significantly from the surface of the object, and alleviate the overfitting problem.

[0012]

[0013] where c n , a n represent the spherical harmonic color coefficient weighted value and opacity of the nth Gaussian sphere, respectively, id represents the inv-depth at the pixel value, id n represents the inv-depth of the nth Gaussian sphere, a m represents the opacity of the mth Gaussian sphere, m < n. N is the number of Gaussian spheres, μ id : R → [0, 1] and σ id : R → [0, 1] are the mean and standard deviation of the inv-depth of the 3D GS along the line-of-sight.

[0014] Step 3, establishing a camera photometric model, including an internal photometric model and an external photometric model.

[0015] The external photometric model considers the distortion caused by the influence of dust, stains, other interference objects or fogging on the camera lens:

[0016] B i (x) = S α (x) R (π (p)) + S β (x)

[0017] x = π (p)

[0018] where i represents the input image index, B i (x) is the radiation entering the camera, S α : R 2 → [0, 1] represents the attenuation factor of the interference object, S β : R 2 → R simulates the radiation emitted by the interference object itself. p represents the three-dimensional position of the observation point, π: R 3 → R 2 is a projection function that maps p to pixel coordinates x in the pinhole camera model.

[0019] The internal photometric model considers the internal distortion problem affected by vignetting and sensor non-uniformity:

[0020] I i (x) = G (x) V (x) B i (x)

[0021] where I i (x) represents the observed pixel intensity from the ith image, V: R 2 → [0, 1] represents the lens attenuation function, G: R 2 → [0, 1] represents the response function.

[0022] The radiation value B i(x) is substituted into the internal photometric model to obtain a camera photometric model as follows:

[0023] I i (x) = G(x) V(x) S α (x) = G(x) V(x) S β (x)

[0024] = M α (x) = G(x) V(x) S β (x)

[0025] wherein M α (x) = G(x) V(x) S α (x) represents a comprehensive attenuation factor, and M β (x) = G(x) V(x) S β (x) represents a comprehensive bias term.

[0026] Step 4, using a loss function based on direct image supervision Optimizing the radiation field representation parameters in step 1:

[0027]

[0028] wherein I represents the pixel intensity of the input image, and λ is the weight value of different loss terms, represents a structural similarity loss function term.

[0029] Using a balanced loss function to optimize the comprehensive attenuation factor and the comprehensive bias term of the camera photometric model:

[0030]

[0031] wherein I α and I β are outputs of M α (x) and M β (x) respectively, and I' is an output result obtained by taking all 1 as input by the internal photometric model. and respectively represent I', I α and I β images after Gaussian smoothing, and ‖·‖2 is a two-norm loss.

[0032] Step 5, input multiple camera views and corresponding projection images in a scene. For each view, first perform three-dimensional scene representation, and then obtain the corresponding projection image after depth regularization processing. Input the projection images of multiple views into the camera photometric model trained in step 4, and output the three-dimensional reconstruction result.

[0033] The present application has the following beneficial effects:

[0034] 1. By introducing a camera characterization module, the camera photometric parameters can be estimated during the 3D reconstruction process. Compared with existing technologies, this method can separate the camera photometric distortion and obtain high-quality 3D scenes with stronger robustness.

[0035] 2. Through the deep regularization method, a new and efficient scheme that can constrain the geometric distribution of the scene is proposed. Compared with the existing technology, it can efficiently separate the noise caused by the camera photometric error without affecting the scene fitting ability, so as to meet the estimation of the camera photometric parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure One This is a schematic diagram of imaging of a traditional camera model;

[0037] Figure Two Schematic diagram of imaging of a camera model with photometric distortion;

[0038] Figure Three Comparison of vignetting at different aperture settings;

[0039] Figure Four This is a flow chart of the 3D scene reconstruction method based on camera photometric optimization;

[0040] Figure Five Schematic diagram of Gaussian sphere distribution in the single pixel projection direction;

[0041] Figure Six This is a display of input data under different interferences in the embodiment. DETAILED DESCRIPTION

[0042] The present invention will be further explained below with reference to the accompanying drawings.

[0043] Combining the process of scene and camera representation training, a 3D scene reconstruction method based on camera photometric optimization is proposed. Figure Four As shown in the figure, the red arrow represents the camera representation training and the black arrow represents the scene representation training. During the camera representation training process, deep regularized projection is used to assist camera fitting, and multiple view cameras are captured to obtain a set of projected images. This set of images is paired with the input image to adapt the camera representation, where the gradient is propagated only for the camera representation. In the model representation training, the camera representation is fixed, a single view is selected for projection, and lens-specific photometric distortion is added to the 3D Gaussian model to fit the input image. The specific steps are as follows:

[0044] Step 1, the radiation field of multiple images of the same scene is characterized, and the scene is characterized as a set of three-dimensional Gaussian balls, each three-dimensional Gaussian is characterized by a covariance matrix Σ and a center point μ in three-dimensional space, and the radiation value R(x) corresponding to the pixel point x in the image is obtained:

[0045]

[0046] where c n , α n respectively represent the spherical harmonic color coefficient weighted value and the opacity of the nth Gaussian ball. α m represents the opacity of the mth Gaussian ball, m < n. N is the number of Gaussian balls.

[0047] Step 2, in order to alleviate the overfitting problem of the model, a depth regularization technique based on prior knowledge is proposed, which reduces the interference of floating points by minimizing the contribution of overfitting points in the projection image.

[0048] Figure Five For the Gaussian ball distribution in the projection direction of a single pixel, it can be seen that there is actually a certain distribution relationship in the space around the object surface. Since the radiation field is represented by multiple three-dimensional Gaussian balls, the opacity near the object surface should generally follow a Gaussian distribution during projection. Based on this characteristic, a depth regularization based on Gaussian distribution is designed: for each pixel, the inverse depth of the associated three-dimensional GS based on the mean and variance is calculated, and then a Gaussian function is applied to suppress points located outside the range of 1 standard deviation:

[0049]

[0050] where μ id : R→[0,1] and σ id : R→[0,1] are the mean and standard deviation of the inverse depth of the three-dimensional GS in the line-of-sight direction. id n represents the inverse depth of the nth Gaussian ball.

[0051] According to the Gaussian function value, the opacity is adjusted to optimize the camera photometric parameters, effectively and dynamically suppressing points that deviate significantly from the object surface, thereby alleviating the overfitting problem, so that the camera model learns a relatively clean model, but essentially does not reduce the fitting ability of the model.

[0052] Step 3, for the geometric part, a pinhole camera model is used to simplify the geometric camera model, and the three-dimensional point is projected onto the two-dimensional image:

[0053] x = π(p)

[0054] where p represents the three-dimensional position of the observed point, π: R 3 → R 2is the projection function that maps p to pixel coordinates x. A camera photometric model is established to map the actual radiance at a position in front of the lens to its corresponding intensity value, omitting the position.

[0055] Since photometric distortions can come from both inside and outside the camera, the camera photometric model is divided into an internal photometric model and an external photometric model.

[0056] A camera lens can accumulate dust, stains or other disturbances over time, or can be affected by sudden problems such as lens fogging or lens surface obstacles, which can introduce distortions and reduce image quality. These surface distortions can manifest as irregular patterns or local variations in image intensity, which can be thought of as partial loss of scene radiance information as it passes through the disturbance, while the disturbance itself emits additional radiance to the camera. The surface distortion parameters are incorporated into the camera model, and the external photometric model is expressed as:

[0057] B i (x) = S α (x) R(p) + S β (x)

[0058] where B i (x) is the radiance entering the camera, i denotes the input image index. S α : R 2 → [0, 1] represents the attenuation factor of the disturbance, S β : R 2 → R simulates the radiance emitted by the disturbance itself, represents the radiance of the scene radiance field with observation direction, and represents the radiance information emitted by a spatial point. Therefore, the influence of photometric distortion on the scene radiance received by the camera is captured through the external photometric model. It should be noted that although the external photometric model includes the observation direction related to the camera geometric extrinsic parameters, these information only involves sampling from the radiance field and does not depend on the camera geometric parameters.

[0059] The main factors causing camera internal photometric distortion are vignetting and sensor non-uniformity, and the internal photometric model is expressed as:

[0060] I i (x) = G(x) V(x) B i (x)

[0061] where I i (x) represents the pixel intensity observed from image i, V: R 2 → [0, 1] represents the lens attenuation function, and G: R 2 → [0, 1] represents the response function, reflecting the changes in lens and sensor response on different pixels. Because the radiance scalar entering the camera is unknown, the degree of radiance response is characterized by mapping this value to the interval [0, 1] through the response function G.

[0062] Radiance values B i (x) into the internal photometric model to obtain the following camera photometric model:

[0063] I i (x) = G(x)V(x)S α (x)R(p,σ) + G(x)V(x)S β (x)

[0064] Simplify the camera photometric model:

[0065] I i (x) = M α (x)R(p,σ) + M β (x)

[0066] where M α (x) = G(x)V(x)S α (x) represents a comprehensive attenuation factor, and M β (x) = G(x)V(x)S β (x) represents a comprehensive bias term. As can be seen from the simplified camera photometric model, although there are multiple photometric distortion factors, they can be integrated into a single linear transformation model for each pixel. The simplified model not only simplifies the photometric modeling process, but also simplifies the design of subsequent three-dimensional scene-camera representation methods.

[0067] In actual camera imaging, the camera mainly focuses on the captured object, i.e., focuses on the scene, so the stains on the lens usually appear blurred. In addition, the vignetting of the lens and the non-uniform response of the image sensor are usually smooth, and the photometric parameters of adjacent pixels are related on the final captured image. Therefore, the shallow multi-layer perception machine model mlp α and mlp β are used to fit M α (x) and M β (x) in the camera photometric model, respectively:

[0068] I i (x) = mlp α (x)R(x) + mlp β (x)

[0069] where mlp α (x) and mlp β (x) represent the outputs of the two multi-layer perception machine models, respectively, which model various photometric distortion factors affecting the input radiance through multi-layer perception machine models, complete scaling and bias correction.

[0070] Step 4, Different loss functions are used to optimize the 3D Gaussian scene parameters and camera photometric model parameters respectively in step 1.

[0071] First, a loss function based on direct image supervision is used Optimization of the radiance field representation parameters in step 1:

[0072]

[0073] where represents the pixel intensity of the input image, λ is the weight value of different loss terms, represents the structural similarity loss function term.

[0074] Second, since photometric distortion usually exhibits smooth properties, a smooth loss is used to constrain the camera photometric parameters in the optimization process:

[0075]

[0076] where I α and I β are the outputs of M α (x) and M β (x), respectively, and I' is the output result obtained by the internal photometric model with a white picture as input. and represent the images after Gaussian smoothing of I', I α and I β , respectively, and ‖·‖2 is the two-norm loss.

[0077] The camera photometric model requires multiple views as input for fitting, while the 3D reconstruction model usually uses a single image with specific camera coordinates and angles. Therefore, the optimization process uses an iterative training method to optimize the camera photometric parameters and scene photometric parameters in turn. For the camera model, a set of viewpoints is selected according to the current state of the 3D reconstruction model, the scene is projected to obtain a rendered image, and the loss between the final rendered image obtained by the camera model and the corresponding input image is calculated. In each iteration, the camera model is first updated and trained, and then the update is paused to continue the training of the 3D reconstruction model.

[0078] Step 5, Handheld shooting was performed using a Sony camera, capturing 40 to 60 images in different scenes, including different types of photometric distortion such as camera vignetting, fingerprints, dirt and water droplets on the lens surface, as shown in Figure Six The introduction of different types of photometric distortion can evaluate their impact on rendering quality and comprehensively evaluate model performance. In addition, corresponding data was supplemented through a mobile phone lens and a logical wide-angle lens to evaluate the generalization ability of the method to different lenses.

[0079] In this embodiment, multiple baseline models are selected for comparative experiments, where the implicit representation includes mip-NeRF, INGP, Rip-NeRF and NeuRBF, and the explicit representation includes 3DGS and 3DGS-mcmc. The peak signal-to-noise ratio (PSNR) and the structure similarity index (SSIM) are selected as evaluation indicators. Table 1 is the comparison results of different baseline models for three-dimensional reconstruction under different external photometric distortion, and Table 2 is the comparison results of different baseline models for three-dimensional reconstruction under multiple scenes with internal photometric distortion (vignetting):

[0080]

[0081]

[0082] Table 1

[0083]

[0084] Table 2

[0085] The results of Table 1 show that the present method has superior robustness and effectiveness under mixed photometric distortion, especially under complex photometric distortion such as fingerprints and water droplets, and is superior to other advanced methods. According to the results of Table 2, it can be seen that the present method can effectively reduce the influence of subtle vignetting on three-dimensional representation, even in scenes that are difficult for human observers to detect. Therefore, the experimental results prove the significant advantages of the present method under various scenes and photometric distortions. And for various scenes, the present method has certain improvement compared with the baseline method (3DGS-mcmc).

Claims

1. A 3D scene reconstruction method based on camera photometric optimization, characterized by: The specific steps include: Step 1: Use the 3D Gaussian model to characterize multiple images of the same scene and obtain the radiation value R(x) corresponding to the pixel point x in the image: where c n , α n Respectively represent the weighted value and opacity of the spherical harmonic color coefficient of the nth Gaussian sphere; α m Indicates the opacity of the mth Gaussian sphere, m<n; N is the number of Gaussian spheres; A deep regularization scheme is designed. For each pixel, the inverse depth of the associated three-dimensional GS based on the mean and variance is calculated, and then a Gaussian function is applied to suppress points outside the range of 1 standard deviation, suppressing points that deviate significantly from the object surface, and alleviating the overfitting problem: where c n , α n Represents the weighted value and opacity of the spherical harmonic color coefficient of the nth Gaussian sphere, id represents the inverse depth, id n represents the inverse depth of the nth Gaussian sphere, α m Indicates the opacity of the mth Gaussian ball, m<n; N is the number of Gaussian balls, μ id :R→[0,1] and σ id :R→[0,1] is the mean and standard deviation of the inverse depth of the 3D GS in the line of sight direction; Step 2: Establish a camera photometric model, including an internal photometric model and an external photometric model; The external photometric model is: B i (x)=S α (x)R(x)+S β (x) Where i represents the input image index, B i (x) is the radiation entering the camera, S α :R 2 →[0,1] represents the attenuation factor of the interferent, S β :R 2 →R simulates the radiation emitted by the interferer itself; The internal photometric model is: I i (x)=G(x)V(x)B i (x) Among them I i (x) represents the pixel intensity observed from the i-th image, V:R 2 →[0,1] represents the lens attenuation function, G∶R 2 →[0,1] represents the response function; The radiation value B obtained by the external photometric model i Substituting (x) into the internal photometric model, we obtain the following camera photometric model: I i (x)=G(x)V(x)S α (x)R(x)+G(x)V(x)S β (x) =M α (x)R(x)+M β (x) Among them, M α (x)=G(x)V(x)S α (x) represents the comprehensive attenuation factor, M β (x)=G(x)V(x)S β (x) represents the comprehensive bias term; respectively use the shallow multi-layer perceptron model mlp α and mlp β To fit the camera photometric model M α (x) and M β (x): I i (x)=mlp α (x)R(x)+mlp β (x) Among them mlp α (x) and mlp β (x) represents the output of two multi-layer perceptron models, which are used to model various photometric distortion factors affecting the input radiation and complete scaling and bias correction; Step 3: Use a loss function based on direct image supervision Optimize the radiation field characterization parameters in step 1; use the balanced loss function Optimize the comprehensive attenuation factor and comprehensive bias term of the camera photometric model; Step 4: Input cameras with multiple viewpoints and corresponding projection images of a scene; for each viewpoint, perform 3D scene representation and obtain the corresponding projection image; input the projection images of multiple viewpoints into the camera photometric model trained in step 3, and output the 3D reconstruction result.

2. The three-dimensional scene reconstruction method based on camera photometric optimization according to claim 1, characterized in that: Map p to pixel coordinate x using the pinhole camera model: x=π(p) p represents the three-dimensional position of the observation point, π:R 3 →R 2 is the projection function that maps p to pixel coordinate x.

3. The three-dimensional scene reconstruction method based on camera photometric optimization according to claim 1, characterized in that: The loss function based on direct image supervision for: in represents the pixel intensity of the input image, λ is the weight value of different loss terms, Represents the structural similarity loss function term.

4. The three-dimensional scene reconstruction method based on camera photometric optimization according to claim 1, characterized in that: The balance loss function for: Among them I α and I β M α (x) and M β The output of (x), I′ is the output of the internal photometric model with all 1s as input; and Represent I′, I α and I β The image is smoothed by Gaussian, and ‖·‖2 is the two-norm loss.

Images