Three-dimensional reconstruction method and device based on sparse view inline prior

Abstract

The embodiment of the invention discloses a three-dimensional reconstruction method and device based on sparse view inline prior. A specific embodiment of the method comprises the following steps: determining a sparse point cloud of a three-dimensional scene from a sparse view angle image; according to the sparse point cloud, initializing the three-dimensional Gaussian point cloud to obtain an initialized three-dimensional Gaussian point cloud; performing random sampling in a preset range of each sparse view angle in each sparse view angle of the sparse view angle image so as to generate a pseudo view angle and obtain a pseudo view angle set; determining a plurality of optimization regular terms according to each sparse view angle and pseudo view angle set; determining a loss function according to the plurality of optimization regular terms; and according to the loss function, optimizing parameters of the initialized three-dimensional Gaussian point cloud so as to carry out sparse visual angle three-dimensional reconstruction. According to the embodiment, mode offset is inhibited, and the three-dimensional reconstruction quality under the sparse view angle is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a three-dimensional reconstruction method and device based on sparse view inline priors. Background Art

[0002] 3D Gaussian sputtering methods based on gradient optimization and differentiable rendering are one of the research hotspots in 3D reconstruction (also known as new view synthesis) tasks. Such methods often require a large number of dense sparse views for training. In the case of sparse training views, most 3D Gaussian sputtering methods will have serious overfitting and performance degradation problems, and it is difficult to synthesize high-quality new view views. To address this problem, most researchers choose to introduce external priors to supervise the optimization of sparse view 3D reconstruction, such as semantic information priors, monocular depth priors, and diffusion model visual priors. For diffusion priors, some researchers use a lot of computing resources to fine-tune diffusion model priors or pre-train image encoders with external 3D data. Some researchers did not fine-tune and pre-train when introducing diffusion model visual prior guidance, but it is difficult to directly extract diffusion model visual prior knowledge, and thus it is difficult to effectively supplement the missing visual information of sparse views. Although the diffusion model visual prior has shown excellent capabilities in text-3D model generation and image-3D model generation tasks through fractional distillation sampling, it performs poorly in sparse view 3D reconstruction tasks. This is due to the essential difference between sparse views and text cues: for unseen views, unlike text cues, the ideal rendered image supervision information in sparse views is not completely absent. Due to the consistency of 3D geometry and structure, the intrinsic prior supervision information exists in a given sparse view. Due to the domain shift between a specific scene and the diffusion model prior, and the inherent suboptimality of the rendered image distribution under sparse views, if fractional distillation sampling is directly used, it is easy to deviate from the target mode of the diffusion model prior distribution domain. Therefore, a large amount of external 3D annotation data and computing resources are often required for domain correction, without fine-tuning the diffusion model and training the encoder to promote the diffusion model prior domain correction, thereby effectively extracting the 3D knowledge in the diffusion model.

[0003] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose a three-dimensional reconstruction method based on sparse view inline prior to solve one or more of the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a three-dimensional reconstruction method based on sparse view inline prior, the method comprising: determining a sparse point cloud of a three-dimensional scene from a sparse perspective image; initializing a three-dimensional Gaussian point cloud based on the sparse point cloud to obtain an initialized three-dimensional Gaussian point cloud; randomly sampling within a preset range of each sparse perspective of each sparse perspective of the sparse perspective image to generate a pseudo perspective to obtain a pseudo perspective set; determining multiple optimization regularization terms based on the above sparse perspectives and the above pseudo perspective set; determining a loss function based on the above multiple optimization regularization terms; and optimizing the parameters of the initialized three-dimensional Gaussian point cloud based on the loss function to perform sparse perspective three-dimensional reconstruction.

[0007] In a second aspect, some embodiments of the present disclosure provide a three-dimensional reconstruction device based on sparse view inline prior, the device comprising: a first determination unit, configured to determine a sparse point cloud of a three-dimensional scene from a sparse perspective image; an initialization unit, configured to initialize a three-dimensional Gaussian point cloud based on the sparse point cloud to obtain an initialized three-dimensional Gaussian point cloud; a sampling unit, configured to randomly sample within a preset range of each sparse perspective of each sparse perspective of the sparse perspective image to generate a pseudo perspective and obtain a pseudo perspective set; a second determination unit, configured to determine a plurality of optimization regularization terms based on the sparse perspectives and the pseudo perspective set; a third determination unit, configured to determine a loss function based on the plurality of optimization regularization terms; an optimization unit, configured to optimize the parameters of the initialized three-dimensional Gaussian point cloud based on the loss function to perform sparse perspective three-dimensional reconstruction.

[0008] The present invention corrects the distribution of rendered images by inline priors, decomposes the optimization goal of fractional distillation sampling into two sub-goals, and the corrected distribution, as the intermediate state of the optimization goal, plays a role in controlling the direction of pattern search. On this basis, three-dimensional Gaussian sputtering is used as an explicit three-dimensional representation, and deep regularization is introduced to support inline priors and geometric consistency regularization to reduce the difference between pixel-level rendered image distribution and corrected distribution, thereby suppressing pattern deviation and improving the quality of three-dimensional reconstruction under sparse perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0010] Figure 1is a flowchart of some embodiments of a 3D reconstruction method based on sparse view inline prior according to the present disclosure;

[0011] Figure 2 is a schematic structural diagram of some embodiments of a three-dimensional reconstruction device based on sparse view inline prior according to the present disclosure; DETAILED DESCRIPTION

[0012] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0013] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0014] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0015] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0016] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0017] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0018] Figure 1 This is a process 100 of some embodiments of the 3D reconstruction method based on sparse view inline prior of the present disclosure. The 3D reconstruction method based on sparse view inline prior comprises the following steps:

[0019] Step 101 : determining a sparse point cloud of a three-dimensional scene from a sparse view image.

[0020] In some embodiments, an execution subject (eg, a computing device) of a 3D reconstruction method based on sparse view inline priors may determine a sparse point cloud of a 3D scene from a sparse view image.

[0021] Here, the above-mentioned sparse perspective image may refer to an image covering a portion of the perspective.

[0022] As an example, the execution subject may use the structure-from-motion method to determine a sparse point cloud of a three-dimensional scene from a sparse view image through feature extraction and matching techniques.

[0023] It should be noted that the above-mentioned computing device can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. For example, the computing device can be the above-mentioned target terminal. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0024] Step 102 : Initialize the three-dimensional Gaussian point cloud according to the sparse point cloud to obtain an initialized three-dimensional Gaussian point cloud.

[0025] In some embodiments, the execution entity may initialize the three-dimensional Gaussian point cloud according to the sparse point cloud to obtain an initialized three-dimensional Gaussian point cloud.

[0026] Here, the above three-dimensional Gaussian point cloud can represent a set of Gaussian point sets. The three-dimensional position of the three-dimensional Gaussian point cloud is expressed as:

[0027]

[0028] Among them, μ n represents the position of the three-dimensional Gaussian point cloud, ∑ n represents the covariance matrix, c n Indicates color, α n represents opacity, n represents the number of Gaussian points, x represents a random variable, and T represents transpose. Represents the inverse of the covariance matrix. The above parameters are optimized during the training process, among which the covariance matrix ∑ n Use the rotation matrix R n and the expansion matrix S n Represented to ensure positive definiteness during optimization: Among them, R n represents the rotation matrix. S n Represents the scaling matrix. Represents the inverse of the scaling matrix. Represents the inverse of a rotation matrix.

[0029] Optionally, after initializing the three-dimensional Gaussian point cloud according to the sparse point cloud to obtain the initialized three-dimensional Gaussian point cloud, the method further includes:

[0030] The first step is to render the initialized three-dimensional Gaussian point cloud to obtain a two-dimensional image.

[0031] As an example, the above execution subject can be n =JW∑ n W T J T , according to the perspective transformation matrix W and the affine approximation Jacobian matrix J of the projection matrix, determine the covariance matrix ∑′ under the camera perspective n , as a 2D image. T denotes transpose.

[0032] The second step is to sort the Gaussian point cloud corresponding to the above two-dimensional image to obtain the sorted Gaussian point cloud.

[0033] As an example, the execution entity may sort the Gaussian points that contribute to pixel rendering in the two-dimensional image to obtain a sorted Gaussian point cloud.

[0034] The third step is to render the sorted Gaussian point cloud to obtain the corresponding color of the rendered image.

[0035] As an example, the execution subject can render the sorted Gaussian point cloud by the following formula to obtain the corresponding color of the rendered image: in, It is a pseudo-perspective rendered image In pixel p j The rendering color on is the two-dimensional covariance matrix ∑′ of the two-dimensional projected Gaussian points n and the opacity α of the Gaussian point itself n The obtained two-dimensional projection Gaussian point is at pixel p j The subscript n represents the ordinal sign of the Gaussian point, and the subscript m represents the ordinal sign less than n. N represents the maximum ordinal value of the Gaussian point. The above j represents the identifier of the pseudo-view.

[0036] Step 103 : Randomly sample within a preset range of each sparse perspective of each sparse perspective of the sparse perspective image to generate a pseudo perspective, thereby obtaining a pseudo perspective set.

[0037] In some embodiments, the execution entity may randomly sample within a preset range of each sparse perspective of each sparse perspective of the sparse perspective image to generate a pseudo perspective and obtain a pseudo perspective set.

[0038] Here, the preset range may refer to a preset circular range. For example, the preset range may refer to a preset circular range with the sparse viewing angle as the center and 1 as the radius.

[0039] Optionally, the execution subject may perform random sampling within a preset range of each sparse perspective of each sparse perspective of the sparse perspective image through the following steps to generate a pseudo perspective and obtain a pseudo perspective set:

[0040] The first step is to set the training image sets corresponding to each sparse perspective.

[0041] Here, the above training image set can be represented as The above sparse perspectives can represent v i .

[0042] The second step is to construct the transformation function.

[0043] Here, the above transformation function can be expressed as Among them, R j→i represents the relative pose transformation between two views, D j Denotes the pseudo-view set v j Rendering depth.

[0044] The third step is to reversely transform the above training image set from each sparse perspective to a pseudo perspective set through the above transformation function.

[0045] Here, each pixel p in the pseudo-viewing set j The calculation formula is: Among them, d n is the z-buffer of Gaussian point n. j (p j ) represents the pseudo-view set v j The rendering depth D j Each pixel p on j In the transformation process, for the pseudo-view set v j The pixel p j Corresponding visible angle v j The pixel p j→i , the transformation relationship calculation formula is: j→j ~KR j→i D j (p j )K -1 p j . Where K is the camera internal parameter. Thus, each sparse view v can be obtained i The training image set To the pseudo-viewpoint set v j Transformation image Transform each pixel p of the image jThe calculation formula is:

[0046] Among them, Sampler(·) is the nearest neighbor sampling operator.

[0047] Step 104: Determine a plurality of optimization regularization terms according to the sparse perspectives and the pseudo perspective set.

[0048] In some embodiments, the execution entity may determine a plurality of optimization regularization items according to the sparse perspectives and the pseudo perspective set.

[0049] Optionally, the execution subject may determine a plurality of optimization regularization items according to the sparse perspectives and the pseudo perspective set through the following steps:

[0050] The first step is to determine the pixel loss and structural consistency loss of the ground truth image and the rendered image corresponding to each of the above sparse perspectives.

[0051] As an example, the above execution entity can be Determine the pixel loss and structural consistency loss of the ground truth image corresponding to each of the sparse perspectives and the rendered image. Indicates pixel loss. Indicates the loss of structural consistency. Represents a rendered image. Represents the ground truth image corresponding to each sparse perspective, that is, the training image set.

[0052] In the second step, the sparse-view monocular depth estimation of the ground truth image corresponding to each of the above sparse viewpoints is determined through the external monocular depth prior.

[0053] As an example, the above execution entity can be Through the external monocular depth prior, the sparse view monocular depth estimation of the ground truth image corresponding to each sparse view is determined. Mono(·) is the external monocular depth estimator. Represents sparse-view monocular depth estimation. The subscript m stands for monocular.

[0054] In the third step, the sparse perspective depth Pearson correlation loss is determined based on the sparse perspective monocular depth estimation and the rendering depth corresponding to each sparse perspective.

[0055] As an example, the above execution entity can be According to the sparse perspective monocular depth estimation and the rendering depth corresponding to each sparse perspective, the sparse perspective depth Pearson correlation loss is determined. Represents the sparse view depth Pearson correlation loss. Indicates the rendering depth corresponding to each sparse perspective. Corr(·) is the Pearson correlation operator, and the calculation formula is:

[0056] Above D r Indicates the rendering depth. The subscript r stands for render. m Represents monocular depth estimation.

[0057] The fourth step is to determine the pseudo-perspective monocular depth estimation corresponding to the above pseudo-perspective set through the external monocular depth prior.

[0058] As an example, the above execution entity can be Through the external monocular depth prior, the pseudo-perspective monocular depth estimation corresponding to the above pseudo-perspective set is determined. Represents pseudo-view monocular depth estimation.

[0059] In the fifth step, the pseudo-perspective depth Pearson correlation loss is determined according to the rendering depth corresponding to the pseudo-perspective set and the pseudo-perspective monocular depth estimation.

[0060] As an example, the above execution entity can be According to the rendering depth corresponding to the pseudo-perspective set and the pseudo-perspective monocular depth estimation, the pseudo-perspective depth Pearson correlation loss is determined. represents the pseudo-perspective depth Pearson correlation loss. Indicates the rendering depth corresponding to the pseudo perspective set.

[0061] The sixth step is to determine the transformation image and consistency mask from the ground truth image corresponding to each of the above sparse perspectives to the above pseudo perspective set to obtain the geometric consistency loss.

[0062] As an example, the execution subject may first determine the training image set corresponding to each sparse perspective. To pseudo-perspective v j Transformation image and consistency mask M i→j Then, the 1-norm loss of the mask is determined to obtain the geometric consistency loss, i.e.

[0063]

[0064] In the seventh step, noise is added to the above rendered image according to the time step to obtain a noisy rendered image distribution.

[0065] As an example, the above execution entity can be used to render an image Adding noise according to the time step t can get the distribution of noisy rendered images The distribution of noisy rendered images follows a Gaussian distribution: in, represents the degradation coefficient, I represents the identity matrix, and θ represents the parameters of the three-dimensional Gaussian point cloud. Represents a rendered image Rendered image after adding noise at time step t.

[0066] In the eighth step, according to the above distribution of noisy rendered images, the score matching loss based on inline prior guidance is determined.

[0067] As an example, the above implementation can introduce an inline prior to distribute the noisy rendered image Corrected to Use transformed images from each sparse viewpoint To guide The sampling trajectory of , where φ represents the parameter introduced to construct the modified distribution. Represents a noisy rendered image. represents the modified distribution. Therefore, the optimization objective of the score matching based on the inline prior guidance can be expressed as:

[0068]

[0069] Among them, ω(t) is the weight function about t, D KL is the KL divergence, η r Adjustment coefficients for the two sub-optimization objectives. In practice, the diffusion repair model Used to represent the modified distribution. Diffusion model Used to represent the prior distribution of the noisy diffusion model. represents the diffusion prior distribution. Therefore, the gradient corresponding to the optimization objective of score matching guided by the inline prior can be obtained as:

[0070]

[0071] Here, the above γ(t) represents Above It is subject to the diffusion prior distribution. Therefore, the score matching loss based on the inline prior guidance can be expressed as:

[0072]

[0073] In the ninth step, the above pixel loss and structural consistency loss, the above sparse view depth Pearson correlation loss, the above pseudo view depth Pearson correlation loss, the above geometric consistency loss and the above score matching loss based on inline prior guidance are determined as multiple optimization regularization terms.

[0074] The first to ninth steps above construct a corrected distribution as an intermediate state between the noisy diffusion model prior distribution and the noisy rendered image distribution, guide the pattern search behavior of fractional distillation, control the optimization trajectory of 3D representation parameters, and effectively extract the prior visual knowledge of the diffusion model, thereby improving the quality of sparse perspective 3D reconstruction.

[0075] Step 105: determine a loss function based on the above multiple optimization regularization terms.

[0076] In some embodiments, the execution entity may determine a loss function based on the multiple optimization regularization terms.

[0077] As an example, the execution entity may determine the loss function according to the above multiple optimization regularization terms through the following formula: . The above λ represents the regularization parameter.

[0078] Step 106: Optimize the parameters of the initialized three-dimensional Gaussian point cloud according to the loss function to perform sparse perspective three-dimensional reconstruction.

[0079] In some embodiments, the execution entity may optimize the parameters of the initialized three-dimensional Gaussian point cloud according to the loss function to perform sparse perspective three-dimensional reconstruction.

[0080] As an example, the above-mentioned execution entity can supervise each sparse perspective and the above-mentioned pseudo-perspective set based on the loss function, optimize and update the three-dimensional Gaussian point cloud parameters through differentiable rendering and gradient descent algorithm, so as to achieve three-dimensional reconstruction under sparse perspective.

[0081] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a 3D reconstruction method based on sparse view inline priors. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0082] like Figure 2As shown, a 3D reconstruction apparatus 200 based on sparse view inline prior in some embodiments includes: a first determination unit 201 , an initialization unit 202 , a sampling unit 203 , a second determination unit 204 , a third determination unit 205 and an optimization unit 206 . Among them, the first determination unit 201 is configured to determine the sparse point cloud of the three-dimensional scene from the sparse perspective image; the initialization unit 202 is configured to initialize the three-dimensional Gaussian point cloud according to the above sparse point cloud to obtain the initialized three-dimensional Gaussian point cloud; the sampling unit 203 is configured to randomly sample within a preset range of each sparse perspective of each sparse perspective of the above sparse perspective image to generate a pseudo perspective and obtain a pseudo perspective set; the second determination unit 204 is configured to determine multiple optimization regularization terms according to the above sparse perspectives and the above pseudo perspective set; the third determination unit 205 is configured to determine the loss function according to the above multiple optimization regularization terms; the optimization unit 206 is configured to optimize the parameters of the initialized three-dimensional Gaussian point cloud according to the above loss function to perform sparse perspective three-dimensional reconstruction.

[0083] It is understood that the units described in the device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the sparse view inline prior based three-dimensional reconstruction device 200 and the units included therein, and will not be described in detail here.

Claims

1. A 3D reconstruction method based on sparse view inline prior, comprising: Determine a sparse point cloud of a three-dimensional scene from sparse viewpoint images; Initializing the three-dimensional Gaussian point cloud according to the sparse point cloud to obtain an initialized three-dimensional Gaussian point cloud; Randomly sampling within a preset range of each sparse perspective of each sparse perspective of the sparse perspective image to generate a pseudo perspective, thereby obtaining a pseudo perspective set; Determining a plurality of optimization regularization items according to the sparse perspectives and the pseudo perspective set; Determining a loss function according to the multiple optimization regularization terms; According to the loss function, the parameters of the initialized three-dimensional Gaussian point cloud are optimized to perform sparse perspective three-dimensional reconstruction.

2. The method according to claim 1, wherein: The randomly sampling within a preset range of each sparse perspective of each sparse perspective of the sparse perspective image to generate a pseudo perspective to obtain a pseudo perspective set includes: Set the training image sets corresponding to each sparse perspective; Construct transformation functions; The training image set is inversely transformed from each sparse perspective to a pseudo perspective set through the transformation function.

3. The method according to claim 1, wherein: After initializing the three-dimensional Gaussian point cloud according to the sparse point cloud to obtain the initialized three-dimensional Gaussian point cloud, the method further includes: Rendering the initialized three-dimensional Gaussian point cloud to obtain a two-dimensional image; Sorting the Gaussian point cloud corresponding to the two-dimensional image to obtain a sorted Gaussian point cloud; The sorted Gaussian point cloud is rendered to obtain a color corresponding to the rendered image.

4. A three-dimensional reconstruction device based on sparse view inline prior, comprising: A first determining unit is configured to determine a sparse point cloud of a three-dimensional scene from the sparse view image; An initialization unit is configured to initialize the three-dimensional Gaussian point cloud according to the sparse point cloud to obtain an initialized three-dimensional Gaussian point cloud; a sampling unit configured to randomly sample within a preset range of each sparse perspective of each sparse perspective of the sparse perspective image to generate a pseudo perspective and obtain a pseudo perspective set; A second determining unit is configured to determine a plurality of optimization regularization items according to the sparse perspectives and the pseudo perspective set; A third determining unit is configured to determine a loss function according to the multiple optimization regularization terms; The optimization unit is configured to optimize the parameters of the initialized three-dimensional Gaussian point cloud according to the loss function to perform sparse perspective three-dimensional reconstruction.

5. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

6. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images