Single-view three-dimensional reconstruction method based on dynamic mixed diffusion model

Abstract

The invention discloses a single-view three-dimensional reconstruction method based on a dynamic mixed diffusion model, and relates to the technical field of computer vision and graphics, and the method comprises the following steps: receiving a single two-dimensional RGB image, and carrying out the preprocessing of the image, thereby obtaining a foreground image, a foreground mask, a pseudo-real depth map, and a semantic text description; taking the foreground image, the foreground mask and the pseudo-true depth map as supervision signals, and optimizing neural radiation field representation by using a loss function only containing 3D diffusion prior to obtain a low-resolution three-dimensional model with a preliminary geometric structure; and converting the neural radiation field representation into a micro tetrahedral mesh representation, optimizing the micro tetrahedral mesh representation based on dynamic mixed diffusion prior loss, and outputting a high-quality three-dimensional model. According to the method, texture fidelity and geometric consistency are dynamically balanced, so that the problems of overfitting, insufficient details and inconsistent visual angles caused by a fixed guide scale in a traditional method are effectively solved, and a high-quality three-dimensional model is generated from a single image.

Description

Technical Field

[0001] This invention relates to the field of computer vision and graphics technology, and more specifically, to a single-view 3D reconstruction method based on a dynamic hybrid diffusion model. Background Technology

[0002] With the development of computer graphics and technologies such as virtual reality and augmented reality, the demand for 3D content generation in fields such as game development, film and television production, and digital simulation is constantly increasing. Traditional 3D model construction methods mostly rely on manual modeling or multi-view acquisition, which suffers from high costs and low efficiency. In contrast, reconstructing 3D models from a single 2D image has attracted widespread attention due to its ease of data acquisition and flexible application scenarios. In recent years, deep learning technology has driven the development of single-view 3D reconstruction methods, especially the use of diffusion models pre-trained on large-scale 2D image datasets as prior constraints to alleviate the problem of scarce 3D labeled data, which has gradually become an important research direction in this field.

[0003] Existing single-view 3D reconstruction methods based on diffusion priors mainly include 2D diffusion priors, 3D diffusion priors, and hybrid prior schemes combining the two. Among them, 2D diffusion prior methods have advantages in texture generation but lack 3D geometric perception capabilities and are prone to inconsistencies across multiple views. 3D diffusion prior methods perform well in terms of geometric consistency, but their texture detail quality is limited by the scale of training data. Hybrid diffusion prior methods attempt to combine the above advantages, but they usually use a fixed diffusion guidance scale, which is difficult to adapt to the needs of different optimization stages, easily leading to texture overfitting or insufficient detail. At the same time, there is still a difficult technical contradiction to balance between multi-view consistency and reference view fidelity, affecting the stability and overall quality of 3D reconstruction results.

[0004] To address the above problems, this invention proposes a solution. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a single-view 3D reconstruction method based on a dynamic hybrid diffusion model, in order to solve the problem that existing hybrid diffusion prior methods are difficult to balance the consistency of multiple views and the fidelity of the reference view due to the use of a fixed guiding scale, resulting in unstable reconstruction results.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] The single-view 3D reconstruction method based on a dynamic hybrid diffusion model includes the following steps:

[0008] Receive a single 2D RGB image and preprocess it to obtain a foreground image, a foreground mask, a pseudo-real depth map, and a semantic text description;

[0009] Using the foreground image, foreground mask, and pseudo-real depth map as supervision signals, the neural radiation field representation is optimized using a loss function that only includes 3D diffusion priors, resulting in a low-resolution 3D model with preliminary geometric structure.

[0010] The neural radiation field representation is converted into a differentiable tetrahedral mesh representation and optimized based on dynamic hybrid diffusion prior loss to output a high-quality 3D model.

[0011] The dynamic hybrid diffusion prior loss is composed of dynamic fusion of 2D diffusion prior loss and 3D diffusion prior loss, and its guiding scale is adjusted according to the semantic similarity change rate between the current rendering view and the reference view, as well as the camera perspective of the rendering view.

[0012] In a preferred embodiment, receiving a single two-dimensional RGB image and preprocessing it includes:

[0013] Use a pre-trained image segmentation model to extract the foreground image and its corresponding foreground mask;

[0014] Use a pre-trained depth estimation model to estimate a pseudo-real depth map from a reference perspective;

[0015] Use an image description model to generate semantic text descriptions of images.

[0016] In a preferred embodiment, the loss function that includes only 3D diffusion prior includes:

[0017] The neural radiation field representation is used as a 3D representation, and a color image, image mask, and depth map of the reference viewpoint are rendered based on camera parameters.

[0018] Calculate the reconstruction loss of the rendering result from the reference viewpoint and the input data, including RGB loss, mask loss and depth loss;

[0019] For a non-referenced viewpoint with random sampling, fractional distillation sampling loss is calculated using a pre-trained 3D diffusion model and a fixed guiding strength.

[0020] In a preferred embodiment, converting the neural radiation field representation into a differentiable tetrahedral mesh representation specifically includes:

[0021] From the density field of the neural radiation field, an isosurface characterizing the surface of the object is extracted;

[0022] A set of three-dimensional vertices is sampled on the isosurface, and their tetrahedral topological connections are constructed;

[0023] Based on the neural radiation field, the symbolic distance function value is calculated for each vertex, and the vertex position offset is initialized.

[0024] In a preferred embodiment, the dynamic mixing-diffusion prior loss is calculated using the following formula:

[0025]

[0026] in, For 3D diffusion prior loss, The loss is 2D diffusion prior, where T represents the total time step and t represents the current time step.

[0027] In a preferred embodiment, the guiding scale is determined by the following steps:

[0028] Render a set of multi-view rendered images from the current 3D model representation;

[0029] Calculate the average CLIP semantic similarity between the current multi-view rendered image set and the reference image;

[0030] Based on the average CLIP semantic similarity, the average change in similarity within a preset sliding window is calculated;

[0031] Based on the average similarity change, the benchmark guiding scale is calculated using a nonlinear mapping function;

[0032] Based on the camera azimuth angle of the rendered view, the reference guide scale is linearly scaled according to the viewpoint to obtain the final guide scale applied to that viewpoint.

[0033] In a preferred embodiment, the nonlinear mapping function is specifically:

[0034]

[0035] in, Based on the guiding scale and Hyperparameters set by the user This represents the average change in similarity.

[0036] In a preferred embodiment, the specific method of the view-related linear scaling is as follows:

[0037]

[0038] in, Based on the guiding scale and Hyperparameters set by the user This represents the azimuth deviation between the camera's azimuth angle and the reference viewing angle.

[0039] The technical effects and advantages of the single-view 3D reconstruction method based on the dynamic hybrid diffusion model of this invention are as follows:

[0040] 1. This invention designs a progressive two-stage reconstruction framework by converting the neural radiation field representation into a differentiable tetrahedral mesh representation and optimizing it based on a dynamic hybrid diffusion prior loss. The method first optimizes the neural radiation field representation using a loss function that only includes a 3D diffusion prior to establish a robust preliminary geometric structure, effectively avoiding geometric distortion introduced by early optimization. Subsequently, it is converted to a differentiable tetrahedral mesh for refined optimization, significantly improving the stability of the reconstruction process and the geometric accuracy of the final 3D model.

[0041] 2. This invention innovatively achieves dynamic and intelligent control of the fusion weights of 2D and 3D diffusion priors by adaptively adjusting the guiding scale of the dynamic fusion diffusion prior loss based on the semantic similarity change rate between the current rendered view and the reference view, as well as the camera perspective of the rendered view. This mechanism can automatically balance "multi-view consistency" and "reference view fidelity," adaptively preventing texture overfitting or insufficient detail during the optimization process, thereby generating a texture-rich and visually high-quality 3D model. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the single-view 3D reconstruction method based on a dynamic hybrid diffusion model according to the present invention.

[0043] Figure 2 The figure shows the experimental results of this invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0045] Example 1, Figure 1 The present invention provides a single-view 3D reconstruction method based on a dynamic hybrid diffusion model, comprising the following steps:

[0046] S1 receives a single 2D RGB image and preprocesses it to obtain a foreground image, a foreground mask, a pseudo-real depth map, and a semantic text description.

[0047] In this embodiment, receiving a single two-dimensional RGB image and preprocessing it includes:

[0048] Use a pre-trained image segmentation model to extract the foreground image and its corresponding foreground mask;

[0049] Use a pre-trained depth estimation model to estimate a pseudo-real depth map from a reference perspective;

[0050] Use an image description model to generate semantic text descriptions of images.

[0051] The preprocessing stage receives a single 2D RGB image I_input as user input, which may contain complex backgrounds. To reduce background interference and provide targeted supervision, this stage executes the following three sub-steps sequentially:

[0052] S101, Foreground Segmentation: Using a pre-trained image segmentation model on the input image The process involves precise separation of the target object. In a preferred embodiment, a Segment Anything Model (SAM) based on the Transformer architecture is employed. By providing an image to the SAM model and leveraging its automatic instance segmentation capabilities or through simple point / box cues, the binary foreground mask of the target object can be extracted. Using this foreground mask With input image By performing pixel-by-pixel multiplication, a clean foreground image with background interference removed can be obtained. Its mathematical expression is:

[0053]

[0054] in, This indicates pixel-wise multiplication. It should be noted that instance segmentation models such as Mask R-CNN can also serve as equivalent alternatives to the image segmentation model used in this step.

[0055] S102, Depth Estimation: To obtain initial geometric constraints, the depth information of the target object needs to be estimated from the reference viewpoint (i.e., the input image viewpoint). This invention uses a pre-trained monocular depth estimation model to process the foreground image. In a preferred embodiment, the MiDaS v3.0 model is used. Input the model, and output a map that is similar to... Relative depth map at the same resolution To transform the relative depth into a scale suitable for subsequent NeRF supervision, it is normalized, ultimately yielding a pseudo-real depth map. Specifically, only for the foreground mask. The depth value of the indicated foreground region is min-max normalized so that its range falls within the [0,1] interval, while the depth value of the background region is set to 0.

[0056] S103, Text Generation: To provide strong semantic conditions for the subsequent 2D diffusion model, it is necessary to generate text describing the input image. This invention uses a Visual-Language Pre-trained (VLP) model to automatically generate the description. In a preferred embodiment, the BLIP-2 model is used. The original input image... (or foreground image) The image encoder of BLIP-2 is input, and its text decoder generates one or more descriptive texts, such as "a detailed sculpture of a dragon". The final semantic text description is denoted as e. This semantic text description e will serve as the text conditional input for the 2D diffusion prior (such as StableDiffusion) in the refinement stage, guiding the direction and content of texture generation.

[0057] Preprocessed foreground target image Foreground mask Reference View Depth Map and semantic text description These data will serve as supervisory inputs for subsequent stages.

[0058] S2, using the foreground image, foreground mask and pseudo-real depth map as supervision signals, optimize the neural radiation field representation using a loss function that only includes 3D diffusion priors to obtain a low-resolution 3D model with preliminary geometric structure.

[0059] In this embodiment,

[0060] S201, Neural Radiation Field Representation and Initialization. A neural radiation field composed of a multilayer perceptron is used as the 3D representation. This network takes 3D spatial coordinates (x, y, z) and the view direction (θ, φ) as input and outputs the volume density σ and view-dependent color values ​​(r, g, b) at that point. The network weights are initialized to a standard normal distribution. The scene bounding box is determined based on the foreground mask. And estimating pseudo-true depth maps Make a rough estimate to ensure that the optimization process focuses on the space occupied by the target object.

[0061] S202, Reference View Supervision. To ensure the correctness of the reconstructed model under the input viewpoint, the foreground image obtained in the preprocessing stage is used. Foreground mask and pseudo-real depth map As direct supervision, in each optimization iteration, the corresponding predicted color image is rendered from the reference viewpoint (i.e., the camera viewpoint of the input image, whose pitch and azimuth angles are usually set to (0,0)). Mask image and depth map .

[0062] Calculate the rendering result from the reference viewpoint and the reconstruction loss from the input data, including:

[0063] RGB loss To ensure the similarity between the generated front view and the input view, and to constrain the consistency between the rendered colors and the actual colors, mean squared error is used for calculation.

[0064]

[0065] in, This indicates that the NeRF rendering process is based on camera parameters. The rendered reference view image, The reference angle is a 2-dimensional vector, typically the pitch angle and azimuth angle. The reference angle is uniformly set to (0, 0).

[0066] Mask loss : Used to constrain the accuracy of object contours, also employing mean square error:

[0067]

[0068] in, This represents the masking function. This indicates that the NeRF rendering process is based on camera parameters. The rendered mask image from the reference viewpoint.

[0069] Deep loss Due to scale uncertainty in monocular depth estimation, normalized cross-correlation is used to constrain the consistency between the predicted depth and the estimated depth distribution within the foreground region.

[0070]

[0071] in, Represents the covariance function. Represents the standard deviation function. A mask image representing the depth map rendered from the reference viewpoint.

[0072] The above three losses together constitute the reconstruction loss from the reference perspective. :

[0073]

[0074] in, , and These are hyperparameters specified by the user, typically set to 1000, 100, and 10 respectively.

[0075] S203, New Perspective Geometric Constraint: For randomly sampled non-reference perspectives, a pre-trained 3D diffusion model (Stable Zero-1-to-3) and a fixed guiding strength (default 5) are used to calculate the fractional distillation sampling (SDS) loss. To guide the overall optimization of the 3D model:

[0076]

[0077] in, Represents the time step of the diffusion process and controls the noise intensity. This represents the latent representation with noise, which is derived from the current rendered image. Obtained by adding noise. The image representing the reference view, i.e. . and This represents the camera's rotation and translation parameters, used to tell the diffusion model the viewing angle corresponding to the currently rendered image. This represents a time-step dependent weighting function used to adjust the contribution of the loss function at different noise levels t. and These represent the predicted noise and the actual noise, respectively. These are learnable parameters of the neural radiation field.

[0078] S204, Calculate and optimize the total loss. :

[0079]

[0080] in, and The balancing hyperparameters specified by the user.

[0081] The parameters of NeRF were optimized using the Adam optimizer for approximately 5000 iterations. After this optimization phase, we obtained a low-resolution 3D neural radiation field representation with a reasonable overall geometry but relatively coarse texture. This model lays a solid geometric foundation for the next stage of refined texture synthesis.

[0082] S3, the neural radiation field representation is converted into a differentiable tetrahedral mesh representation, and optimized based on dynamic hybrid diffusion prior loss to output a high-quality three-dimensional model;

[0083] In this embodiment, converting the neural radiation field representation into a differentiable tetrahedral mesh representation specifically includes:

[0084] The density field is extracted from the optimized neural radiation field. A density threshold (e.g., 0.5) is set, and points in space with a density value equal to this threshold are connected to form a continuous triangular mesh. This mesh serves as the isosurface representing the object's surface. A set of three-dimensional vertices is uniformly sampled on the extracted isosurface. Based on these vertices, a tetrahedral background mesh is constructed in three-dimensional space, and the vertex indices of the tetrahedrons are recorded to clarify their topological connections. This tetrahedral mesh constitutes the basic structure capable of slight deformation.

[0085] Each tetrahedral vertex is assigned a signed distance function value, which defines the signed distance from the vertex to the object surface (negative for the interior, positive for the exterior). The initial SDF value of the vertex is approximated by querying its value in the neural radiation density field and subtracting a density threshold. Simultaneously, a learnable vertex displacement vector is initialized for each vertex, initially set to zero. This displacement vector will be adjusted in subsequent optimizations to deform the mesh surface and fit finer geometric and texture details.

[0086] In this embodiment, the dynamic hybrid diffusion prior jointly optimizes the geometry and texture of the micro tetrahedral mesh representation (DMTet). In each optimization iteration, the geometry and texture of the DMTet are jointly optimized, and the loss function is:

[0087]

[0088] in, The reference perspective is used to supervise the loss (the calculation method is the same as S202). This is the prior loss for dynamic mixing and diffusion.

[0089] The dynamic hybrid diffusion prior loss is constructed by dynamically fusing the 2D diffusion prior loss and the 3D diffusion prior loss, and is calculated using the following formula:

[0090]

[0091] in, For 3D diffusion prior loss, For 2D diffusion prior loss, The weights are dynamic and range from [0,1].

[0092] 2D diffusion prior loss The specific calculation formula is as follows:

[0093]

[0094] in, The noise predicted by the two-dimensional diffusion model with the text e and time step t as conditions. This is the derivative of the image encoder with respect to the input image. The derivative of the rendered image I with respect to the DMTet parameters θ (vertex position, displacement, etc.).

[0095] In this embodiment, the guiding scale is adjusted based on the semantic similarity change rate between the current rendered view and the reference view, as well as the camera perspective of the rendered view, specifically as follows:

[0096] In each iteration k, a set of two-dimensional images from multiple perspectives is rendered from the current differentiable tetrahedral mesh. The image set is computed using a pre-trained CLIP model (such as the image encoder ε_CLIP) and a reference image. Average semantic similarity between :

[0097]

[0098] in, (·) indicates the CLIP image encoder. and These represent the CLIP encoding of the image set rendered by DMtet and the reference image, respectively. Represents a set The quantity in.

[0099] To avoid the influence of noise from a single evaluation, a sliding window of length L is used. The average change in similarity within the window is calculated. As a stable indicator of model optimization progress:

[0100]

[0101] in, This indicates the current iteration step.

[0102] Change in average similarity Input a nonlinear mapping function to calculate the baseline guiding scale. :

[0103]

[0104] Among them, the natural logarithm function acts as a "soft saturation" mechanism, preventing excessive saturation in the early stages of optimization. When the scale is very large, excessively large guidance can cause model collapse. Furthermore, to provide smooth decay during optimization, the ReLU function is used to prevent this. Negative numbers may occur. The basic guiding scale is the minimum guiding scale of the diffusion model, used to ensure the most basic generative capability. In this invention, The value range is [5, 10]. and Hyperparameters set by the user.

[0105] To balance the fidelity of the frontal view with the freedom of generating the rear view, a view-dependent scaling is introduced. Based on the camera azimuth angle of the rendered view, the baseline guiding scale is linearly scaled according to the viewpoint to obtain the final guiding scale applied to that viewpoint. :

[0106]

[0107] in, Based on the guiding scale and To control the scaling factor and the hyperparameters of the curve, The azimuth deviation between the camera azimuth and the reference viewpoint is... When =0 (front view), at this time = At this point, the model is forced to conform the generated texture to the reference image.

[0108] when = At this time (rear view), the guiding intensity reaches its maximum value, making full use of the texture generation capability of the 2D diffusion model to supplement the details on the back.

[0109] Dynamic hybrid weights w and final guiding scale Negative correlation. One specific implementation is:

[0110]

[0111] in, The normalization constant is This is an optional global attenuation factor. When As w increases, w decreases. This means that the loss is significant when the back view or optimization progresses rapidly. 2D diffusion prior loss weight This will increase, thereby driving the model to generate richer details.

[0112] Using the above definition The mechanism involves dynamically calculating weights w, and iteratively optimizing the vertex positions, displacements, and texture attributes of the DMTet using a differentiable renderer and gradient descent methods (such as the Adam optimizer). After sufficient optimization, a final high-quality, geometrically accurate, and texture-rich 3D model can be extracted from the DMTet.

[0113] Example 2, as Figure 2As shown, following the complete process of the single-view 3D reconstruction method based on the dynamic hybrid diffusion model in Example 1, an RGB image containing a sparrow is processed, and finally a high-precision 3D sparrow model is output.

[0114] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0115] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0116] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0118] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0119] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A single-view 3D reconstruction method based on a dynamic hybrid diffusion model, characterized in that, Includes the following steps: Receive a single 2D RGB image and preprocess it to obtain a foreground image, a foreground mask, a pseudo-real depth map, and a semantic text description; Using the foreground image, foreground mask, and pseudo-real depth map as supervision signals, the neural radiation field representation is optimized using a loss function that only includes 3D diffusion priors, resulting in a low-resolution 3D model with preliminary geometric structure. The neural radiation field representation is converted into a differentiable tetrahedral mesh representation and optimized based on dynamic hybrid diffusion prior loss to output a high-quality 3D model. The dynamic hybrid diffusion prior loss is composed of dynamic fusion of 2D diffusion prior loss and 3D diffusion prior loss, and its guiding scale is adjusted according to the semantic similarity change rate between the current rendering view and the reference view, as well as the camera perspective of the rendering view.

2. The single-view 3D reconstruction method based on a dynamic hybrid diffusion model according to claim 1, characterized in that, The receiving of a single two-dimensional RGB image and its preprocessing include: Use a pre-trained image segmentation model to extract the foreground image and its corresponding foreground mask; Use a pre-trained depth estimation model to estimate a pseudo-real depth map from a reference perspective; Use an image description model to generate semantic text descriptions of images.

3. The single-view 3D reconstruction method based on a dynamic hybrid diffusion model according to claim 2, characterized in that, The loss function that includes only 3D diffusion prior includes: The neural radiation field representation is used as a 3D representation, and a color image, image mask, and depth map of the reference viewpoint are rendered based on camera parameters. Calculate the reconstruction loss of the rendering result from the reference viewpoint and the input data, including RGB loss, mask loss and depth loss; For a non-referenced viewpoint with random sampling, fractional distillation sampling loss is calculated using a pre-trained 3D diffusion model and a fixed guiding strength.

4. The single-view 3D reconstruction method based on a dynamic hybrid diffusion model according to claim 3, characterized in that, The conversion of the neural radiation field representation into a differentiable tetrahedral mesh representation specifically includes: From the density field of the neural radiation field, an isosurface characterizing the surface of the object is extracted; A set of three-dimensional vertices is sampled on the isosurface, and their tetrahedral topological connections are constructed; Based on the neural radiation field, the symbolic distance function value is calculated for each vertex, and the vertex position offset is initialized.

5. The single-view 3D reconstruction method based on a dynamic hybrid diffusion model according to claim 4, characterized in that, The dynamic mixing-diffusion prior loss is calculated using the following formula:

6. The single-view 3D reconstruction method based on a dynamic hybrid diffusion model according to claim 5, characterized in that, The guiding scale is determined through the following steps: Render a set of multi-view rendered images from the current 3D model representation; Calculate the average CLIP semantic similarity between the current multi-view rendered image set and the reference image; Based on the average CLIP semantic similarity, the average change in similarity within a preset sliding window is calculated; Based on the average similarity change, the benchmark guiding scale is calculated using a nonlinear mapping function; Based on the camera azimuth angle of the rendered view, the reference guide scale is linearly scaled according to the viewpoint to obtain the final guide scale applied to that viewpoint.

7. The single-view 3D reconstruction method based on a dynamic hybrid diffusion model according to claim 6, characterized in that, The nonlinear mapping function is specifically:

8. The single-view 3D reconstruction method based on a dynamic hybrid diffusion model according to claim 7, characterized in that, The specific method of the viewpoint-dependent linear scaling is as follows:

Images