Self-supervised diffusion method and system for generating three-dimensional Gaussian with consistent views

Abstract

The invention provides a self-supervised diffusion method and system for generating three-dimensional Gaussian with consistent views, and relates to the technical field of image recognition, and the method comprises the steps: obtaining an input image, carrying out the geometric consistent multi-view generation of the input image, and obtaining a refined multi-view image; generating a geometric correlation priori based on the input image and the multi-view image; taking the geometric correlation priori as geometric supervision of a preset three-dimensional Gaussian model, and optimizing the preset three-dimensional Gaussian model to obtain a final three-dimensional Gaussian model with consistent views; and rendering the final three-dimensional Gaussian model to obtain a high-fidelity image result. The invention aims to develop an efficient solution to realize three-dimensional generation with consistent views and high quality.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method for generating a visual Figure 1 Consistent three-dimensional Gaussian self-supervised diffusion method and system. Background Art

[0002] Self-supervised diffusion of 3D Gaussians aims to recover high-quality and high-consistency 3D scene representations from multi-view images through a self-supervised learning mechanism. With the development of deep learning technology, 3D reconstruction technology can automatically process multi-view image inputs and generate high-quality and high-consistency 3D scene representations. Therefore, these representations are widely used in entertainment, education, virtual reality, navigation and other fields.

[0003] In recent years, some studies have used large models for view-consistent 3D generation. These models are trained on large-scale 3D datasets. Although the inference speed is fast within seconds, they require huge computational consumption, and the diversity of the generated results is limited by the distribution of the training data. Another series of works explored optimization-based methods to optimize the generation of 3D objects, in which fractional distillation sampling extracts 3D priors from powerful 2D generative models. However, since 2D generative models cannot provide view-consistent 3D priors, the results of these methods usually show Figure 1 Inconsistent viewing angles and blurred textures are shown. In addition, previous methods used neural radiance fields as 3D representations, which resulted in long optimization times, such as more than half an hour for each example. To address the above issues, some advanced works have adopted 3D Gaussian scatter as 3D representations, significantly reducing the optimization time by about 2 minutes. However, they still face problems such as blurred textures outside the reference image view. Recently, the development of multi-view diffusion models has made it possible to generate multi-view images of the same object based on a single reference image. These models provide texture prior information from new views for 3D reconstruction and bring bright prospects for solving the blurred texture problem. However, directly using multi-view diffusion models may lead to Figure 2 The results shown are that new artifacts are introduced due to the inconsistency between multiple views, resulting in low quality of the 3D generated views. Summary of the invention

[0004] In order to solve the problems of inconsistency of multiple views and low quality of three-dimensional generated views in the above-mentioned prior art, the present invention proposes a method for generating a three-dimensional view. Figure 1 A self-supervised diffusion method based on a consistent 3D Gaussian is proposed to develop an efficient solution to achieve visual Figure 1 Consistent and high-quality 3D generation.

[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0006] A generated vision Figure 1 The self-supervised diffusion method of the three-dimensional Gaussian includes the following steps:

[0007] S1. Obtain an input image, generate a geometrically consistent multi-view of the input image, and obtain a refined multi-view image;

[0008] S2. generating a geometric correlation prior based on the input image and the multi-view image;

[0009] S3. Using the geometric correlation prior as the geometric supervision of the preset three-dimensional Gaussian model, optimizing the preset three-dimensional Gaussian model, and obtaining the visual Figure 1 The final three-dimensional Gaussian model is consistent;

[0010] S4. Render the final three-dimensional Gaussian model to obtain a high-fidelity image result.

[0011] Preferably, generating a geometrically consistent multi-view of the input image to obtain a refined multi-view image includes:

[0012] S11. Inputting the input image into a preset multi-view diffusion model, and outputting a plurality of supplementary multi-views under camera postures;

[0013] S12. Based on the input image and the supplementary multi-view, the preset rough three-dimensional Gaussian model is optimized to obtain an optimized rough three-dimensional Gaussian model, and the optimized rough three-dimensional Gaussian model is rendered to obtain a new rendered image;

[0014] S13. Optimize the new rendered image to obtain a refined multi-view image under the camera posture.

[0015] Preferably, the optimizing the preset rough three-dimensional Gaussian model includes:

[0016] S121. Using the input image I0 and the supplementary multi-view Construct the first score distillation sampling loss function;

[0017] S122. Construct the rough three-dimensional Gaussian model by minimizing the first score distillation sampling loss function Get the optimized rough three-dimensional Gaussian model

[0018] Preferably, the calculation expression of the first score distillation sampling loss function is as follows:

[0019]

[0020] in, represents the first score distillation sampling loss function, Θ represents the rough 3D Gaussian model , t represents a randomly sampled time step, p represents a randomly sampled camera pose, ∈ represents random Gaussian noise, ω(t) represents the weight associated with the time step, ∈ φ (·) represents the pre-trained noise predictor, x represents the rough 3D Gaussian model under camera pose p Rendered image, x r Represents the image set Selected images.

[0021] Preferably, the optimizing the new rendered image comprises: gradually denoising the new rendered image to generate a refined multi-view image with consistent geometry and clear texture.

[0022] Preferably, generating a geometric correlation prior based on the input image and the multi-view image comprises:

[0023] S21. Using the input image I0 and the multi-view image Preset refined 3D Gaussian model Optimize and obtain an optimized and refined three-dimensional Gaussian model

[0024] S22. Based on the input image I0 and the multi-view image The camera pose is optimized for the refined 3D Gaussian model Render and get the depth map and object mask And respectively combine the input image I0 and the multi-view image The supplementary multi-view With the multi-view image The pixel-level correspondence is converted into a visibility map

[0025] S23. The depth map The object mask and the visibility graph Denoted as the geometric correlation prior.

[0026] Preferably, the preset refined three-dimensional Gaussian model Optimizations include:

[0027] S211. Using the input image I0 and the multi-view image Construct the second score distillation sampling loss function;

[0028] S212. Constructing the refined three-dimensional Gaussian model by minimizing the second score distillation sampling loss function Get an optimized and refined three-dimensional Gaussian model

[0029] Preferably, the calculation expression of the second score distillation sampling loss function is as follows:

[0030]

[0031] in, represents the second score distillation sampling loss function, Θ′ represents the refined three-dimensional Gaussian model , t represents a randomly sampled time step, p represents a randomly sampled camera pose, ∈ represents random Gaussian noise, ω(t) represents the weight associated with the time step, ∈ φ (·) represents the pre-trained noise predictor, and x′ represents the refined 3D Gaussian model under camera pose p. Rendered image, x r ′ represents the image set Selected images.

[0032] Preferably, the depth map in the geometric correlation prior is used Object Mask and visibility graph The preset three-dimensional Gaussian model is optimized in terms of geometric constraints and adaptive texture to obtain the final three-dimensional Gaussian model.

[0033] The present invention also proposes a method for generating a visual Figure 1 A consistent three-dimensional Gaussian self-supervised diffusion system, including:

[0034] A multi-view generation module, used to obtain an input image, perform geometrically consistent multi-view generation on the input image, and obtain a refined multi-view image;

[0035] A geometric correlation prior generation module, used to generate a geometric correlation prior according to the input image and the multi-view image;

[0036] A Gaussian optimization module is used to use the geometric correlation prior as the geometric supervision of a preset three-dimensional Gaussian model, optimize the preset three-dimensional Gaussian model, and obtain a visual Figure 1 The final three-dimensional Gaussian model is consistent;

[0037] The output module is used to render the final three-dimensional Gaussian model to obtain a high-fidelity image result.

[0038] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0039] The present invention proposes a method for generating a visual Figure 1A self-supervised diffusion method and system for consistent three-dimensional Gaussian. Since it is challenging to jointly optimize geometric shape and texture from a single image input, the method adopts a three-stage framework. First, in the first stage, geometrically consistent multi-view generation is performed on the input image to obtain a refined multi-view image; secondly, in the second stage, geometric correlation priors are generated based on the input image and the multi-view image; then in the third stage, the geometric correlation priors are used as geometric supervision of a preset three-dimensional Gaussian model, and the visual Figure 1 Finally, the final 3D Gaussian model is rendered to obtain a high-fidelity image result to achieve visual Figure 1 Consistent and high-quality 3D generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram showing a prior art method for optimizing and generating three-dimensional objects proposed in the background technology of the present invention;

[0041] Figure 2 It represents the prior art of generating multi-view principle diagram by using multi-view diffusion model proposed by the background technology of the present invention;

[0042] Figure 3 A method for generating a visual image is provided in an embodiment of the present invention. Figure 1 Flow chart of the self-supervised diffusion method for consistent three-dimensional Gaussian;

[0043] Figure 4 A method for generating a visual image is provided in an embodiment of the present invention. Figure 1 Schematic diagram of the principle of the self-supervised diffusion method of consistent three-dimensional Gaussian;

[0044] Figure 5 A schematic diagram showing the principle of a method for calculating a visibility graph proposed in an embodiment of the present invention;

[0045] Figure 6 A method for generating a visual image is provided in an embodiment of the present invention. Figure 1 Another flowchart of the self-supervised diffusion method for three-dimensional Gaussian;

[0046] Figure 7 A diagram showing a qualitative comparison result on the Google Scanned Object dataset proposed in an embodiment of the present invention;

[0047] Figure 8 A qualitative comparison result diagram showing a qualitative comparison on the OmniObject3D dataset proposed in an embodiment of the present invention;

[0048] Fig. 9 A method for generating a visual image is provided in an embodiment of the present invention. Figure 1 The structural block diagram of the consistent three-dimensional Gaussian self-supervised diffusion system. DETAILED DESCRIPTION

[0049] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0050] It is understandable to those skilled in the art that some well-known contents may be omitted in the drawings;

[0051] In order to facilitate understanding of this embodiment, first, the prior art information of this embodiment is introduced as follows:

[0052] 3D Generative Models: With the development of image generative models and large-scale image-text datasets, it is now possible to synthesize high-quality and diverse objects and scenes. More specifically, the diffusion model demonstrated that generative models can learn scalable image generators through a denoising process. However, the original diffusion model is limited to two dimensions and cannot represent three-dimensional scenes or objects. Extending it to the three-dimensional domain requires a large amount of 3D data that is difficult to collect. An alternative approach to lift the generation task into the three-dimensional domain is to leverage the generative power of large-scale 2D diffusion models and combine them with powerful 3D representations such as neural radiance fields and Gaussian scatter methods. The task then turns to predicting new viewpoints from a single image or text input using neural radiance fields or Gaussian scatter methods as 3D representations. CLIP contrasts language-image pre-trained models for text- or single-image-guided 3D reconstruction, while some methods replace CLIP with a score distillation sampling loss to leverage the generative power in 2D diffusion models to generate high-fidelity 3D objects and scenes from text input.

[0053] Single-view object generation: Due to the lack of unobserved view information, single-view Figure 3 There is uncertainty in the reconstruction process. Figure 3 3D reconstruction is viewed as a conditional generation task. Many works adopt different variants of diffusion models to synthesize new views from single view input. Large 3D reconstruction models like GRM and LGM can facilitate high-quality 3D generation from single view input or text. Single-view image to consistent multi-view image generation is used as the first stage of single-image to 3D generation. This is followed by a sparse view reconstruction module that can generate high-quality 3D models in 1 second.

[0054] Sparse View Reconstruction: Sparse view reconstruction methods using neural radiance fields have been widely explored in the field of 3D reconstruction. The optimization method RegNeRF introduces a deep smoothing regularization between the training views and interpolated views rendered from the neural radiance fields during training. The color correction module trained from large-scale object data is used to enforce color consistency between sparse views. On the other hand, P-NeRF uses pre-computed visual priors to guide the neural radiance fields to analyze the input views pairwise. TensoRF can decode unseen views in 360-degree scenes. PixelNeRF provides a learning framework to adjust the neural radiance fields based on one to a few image inputs in a fully convolutional manner. PixelNeRF provides a new paradigm for controlling the generation module using pixel alignment, while Reconfusion further enhances the sparse scene reconstruction capability by introducing a denoising diffusion model controlled by features generated by CLIP and pixel-nerf.

[0055] SyncDreamer uses 3D convolution and deep attention to maintain multi-view Figure 1 Consistency, generate images from 12 new camera perspectives, and use NeuS for sparse view reconstruction of three-dimensional objects. SyncDreamer can generate smooth geometric shapes, but often lacks details. In addition to using cross-attention to achieve information interaction between different views, Wonder3D also realizes cross-domain information interaction, learns the common distribution of normal vector maps and RGB maps, and uses these normal maps and RGB images for 3D generation to improve the alignment of 3D object geometry and texture. However, due to the inconsistency within the sparse multi-view images, Wonder3D sometimes produces unreasonable geometric shapes. DreamGaussian is implemented in two stages. In the first stage, the reconstruction loss optimizes the input perspective generation, and the fractional distillation sampling loss optimizes the new perspective generation. In the second stage, the mesh is extracted from the three-dimensional Gaussian model and the texture of the model is optimized. DreamGaussian can generate a 3D model that matches the reference image, but because the new perspective information is only supplemented by fractional distillation sampling optimization, blurred textures will be generated in the new perspective area. However, most existing methods have problems with multi-view inconsistency and low quality of three-dimensional generated views. Therefore, in an embodiment, it is intended to develop an efficient solution to achieve visual Figure 1 Consistent and high-quality 3D generation.

[0056] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0057] Example 1

[0058] like Figure 3 and Figure 4 As shown, this embodiment proposes a method for generating a visual Figure 1The self-supervised diffusion method of the three-dimensional Gaussian includes the following steps:

[0059] S1. Obtain an input image, generate a geometrically consistent multi-view of the input image, and obtain a refined multi-view image;

[0060] Although existing multi-view diffusion models can generate complementary images to alleviate the problem of limited input views, there are differences in the geometric distribution between their training data and the given reference image, which inevitably leads to geometric inconsistencies in the generated images. For example, the camera position of the reference image may be significantly different from the camera position used for training. Therefore, using the output of these diffusion models in a normal way (for example, using the generated multi-view as the real image) is prone to exacerbating artifacts in the reconstructed Gaussian image. To solve this problem, S1 proposes geometrically consistent multi-view generation of the input image. The key idea is to construct a rough three-dimensional Gaussian model from a given reference image and perform diffusion inversion on the image rendered by the rough three-dimensional Gaussian model. All images rendered by the same rough three-dimensional Gaussian model are geometrically consistent. In addition, diffusion inversion is actually a process of destroying the image by gradually adding noise. Therefore, even rendered images with imperfect textures can still be inverted back to the latent space of these diffusion models, and refined images can be generated by denoising according to the same geometric distribution of these models. Wherein the input image is geometrically consistent with multi-view generation, and the refined multi-view image is obtained, including:

[0061] S11. Input the input image I0 into a preset multi-view diffusion model, and output supplementary multi-views under six camera postures Where I1,…,I6 are supplementary multi-view An image in;

[0062] In S11, the multi-view diffusion model is specifically Zero123++, and the optimization of the preset rough three-dimensional Gaussian model includes:

[0063] S121. Using the input image I0 and the supplementary multi-view Construct a first score distillation sampling loss function; wherein the calculation expression of the first score distillation sampling loss function is as follows:

[0064]

[0065] in, represents the first score distillation sampling loss function, Θ represents the rough 3D Gaussian model , t represents a randomly sampled time step, p represents a randomly sampled camera pose, ∈ represents random Gaussian noise, ω(t) represents the weight associated with the time step, ∈φ (·) represents the pre-trained noise predictor, x represents the rough 3D Gaussian model under camera pose p Rendered image, x r Represents the image set The image whose camera pose is most similar to p is selected from , where the similarity between different camera poses is measured by the cosine similarity between their embeddings.

[0066] S122. Construct the rough three-dimensional Gaussian model by minimizing the first score distillation sampling loss function Get the optimized rough three-dimensional Gaussian model

[0067] S12. Based on the input image and the supplementary multi-view, the preset rough 3D Gaussian model is optimized using the score distillation sampling loss to obtain an optimized rough 3D Gaussian model, and the optimized rough 3D Gaussian model is rendered to obtain a new rendered image; wherein in 3D computer graphics, rendering is the process of converting a 3D scene into a 2D image; although the new rendered image has high consistency, the texture details are relatively fuzzy; therefore, the new rendered image needs to be optimized;

[0068] S13. Optimize the new rendered image to obtain a refined multi-view image under the camera posture.

[0069] In S13, the optimizing the new rendered image includes: gradually denoising the new rendered image to generate a refined multi-view image with consistent geometry and clear texture. The new rendered image is gradually denoised, including: obtaining a new rendered image under the camera posture defined by Zero123++, performing diffusion inversion on the new rendered image to improve the quality, and using the inverse diffusion process of Zero123++ to obtain a multi-view image with consistent geometry and clear texture Among them, I1 ′ ,…,I6 ′ Multi-view images An image in;

[0070] S2. generating a geometric correlation prior based on the input image and the multi-view image;

[0071] The step of generating a geometric correlation prior based on the input image and the multi-view image comprises:

[0072] S21. Using the input image I0 and the multi-view image Preset refined 3D Gaussian model Optimize and obtain an optimized and refined three-dimensional Gaussian model

[0073] The preset refined three-dimensional Gaussian model Optimizations include:

[0074] S211. Using the input image I0 and the multi-view image A second score distillation sampling loss function is constructed; wherein the calculation expression of the second score distillation sampling loss function is as follows:

[0075]

[0076] in, represents the second score distillation sampling loss function, Θ′ represents the refined three-dimensional Gaussian model , t represents a randomly sampled time step, p represents a randomly sampled camera pose, ∈ represents random Gaussian noise, ω(t) represents the weight associated with the time step, ∈ φ (·) represents the pre-trained noise predictor, and x′ represents the refined 3D Gaussian model under camera pose p. Rendered image, x r ′ represents the image set The image whose camera pose is most similar to p is selected from , where the similarity between different camera poses is measured by the cosine similarity between their embeddings.

[0077] S212. Constructing the refined three-dimensional Gaussian model by minimizing the second score distillation sampling loss function Get an optimized and refined three-dimensional Gaussian model

[0078] S22. Based on the input image I0 and the multi-view image The camera pose is optimized for the refined 3D Gaussian model Render and get the depth map and object mask Where D0,…,d6 are depth maps An image in , M0,…,M6 are object masks respectively A mask image data of; and respectively the input image I0 and the multi-view image The supplementary multi-view With the multi-view image The pixel-level correspondence is converted into a visibility map

[0079] In S22, the depth map is a grayscale image that records the distance of each pixel in the scene from the camera. During the rendering process, the depth channel is used to store this information. The depth value of each pixel represents the straight-line distance from the camera to the scene point represented by the pixel. These depth values ​​are then converted to grayscale values ​​to obtain a depth map. For object masks, when rendering 3D scenes, the depth buffer can be used to generate object masks. Each pixel value in the depth buffer represents the distance of the object surface from the camera. By comparing the depth values, it can be determined whether a pixel is blocked by other objects, thereby generating an object mask. More importantly, since 3DGS is a rasterization-like method, the pixel-level correspondence between any two adjacent images can be calculated, combined with the depth map to obtain a visibility map that describes which areas in the current image are visible in its adjacent images.

[0080] See also Figure 5 , Figure 5 Demonstrates the computation of visibility graph method, in Figure 5 In , the adjacent relationship is described by a one-way arrow, where the end of the arrow points to the current image and the starting point of the arrow points to its adjacent image. ′ 1 as an example, I0 is regarded as its adjacent image. Given the camera pose and depth map D0 of I0, each pixel in I0 can be lifted to the three-dimensional space; then, these three-dimensional points are projected onto the image plane of I′1 using the camera pose of I′1, thereby obtaining the pixel-level correspondence between I0 and I′1. Such a correspondence can be further converted into a visibility map V1. The calculation expression of V1 is as follows:

[0081]

[0082] where ⊙ represents the element-wise product, and Represent the lifting and projection operations respectively; the calculation formulas of the remaining 6 visibility maps are basically similar to the calculation expression of V1 above, and will not be repeated here. From the above calculation formula, it can be known that the visibility map of the current image is calculated based on the depth information, mask information of the adjacent image I0 and the mask information of the current image; according to Figure 5 The mapping relationship in , the adjacent image of I2 is I1, the adjacent image of I3 is I2, the adjacent image of I4 is I5, the adjacent image of I5 is I6, and the adjacent image of I6 is I0; it should be noted that the generated three-dimensional Gaussian is expected to be consistent with the input image, so the main part of the input image should be invisible from other perspectives, and its visibility map value is all 0; the depth map D and object mask M here are the depth map D and object mask M obtained above; calculate all seven visibility maps and represent them as V = {V0, ..., V6};

[0083] S23. The depth map The object mask and the visibility graph Denoted as the geometric correlation prior; it will be used for the final optimization in S3.

[0084] S3. Using the geometric correlation prior as the geometric supervision of the preset three-dimensional Gaussian model, optimizing the preset three-dimensional Gaussian model, and obtaining the visual Figure 1 The final three-dimensional Gaussian model is consistent;

[0085] It should be noted that S1-S3 is a step-by-step optimization process, and each step serves the next step. S1 is used to obtain the refined multi-view image for S2 to generate a more consistent refined 3D Gaussian model. Using this more consistent refined 3D Gaussian model, the depth map, object mask and visibility map can be obtained, which are used for the geometric constraints and adaptive texture optimization of the preset 3D Gaussian model in S3.

[0086] S4. Render the final three-dimensional Gaussian model to obtain a high-fidelity image result.

[0087] In this example, since it is challenging to jointly optimize geometry and texture from a single image input, see Figure 6 The method adopts a three-stage framework. First, in the first stage, geometrically consistent multi-view generation is performed on the input image to obtain a refined multi-view image. Secondly, in the second stage, geometric correlation priors are generated based on the input image and the multi-view image. Then, in the third stage, the geometric correlation priors are used as geometric supervision of a preset 3D Gaussian model and combined with an optimization algorithm to perform visual Figure 1 Finally, the final 3D Gaussian model is rendered to obtain a high-fidelity image result to achieve visual Figure 1 Consistent and high-quality 3D generation.

[0088] Example 2

[0089] This embodiment further illustrates the optimization of the preset three-dimensional Gaussian model. S3 aims to improve the expressiveness of the preset three-dimensional Gaussian model, especially in terms of texture details. There is cross-view content inconsistency between them, and the supervision of the same region in multiple views may be different, which hinders the optimization of the 3D Gaussian model. In order to alleviate the side effects of cross-view content inconsistency and make better use of The complementary texture information in the depth map of the geometric correlation prior is used Object Mask and visibility graph The preset three-dimensional Gaussian model is optimized in terms of geometric constraints and adaptive texture to obtain the final three-dimensional Gaussian model.

[0090] Among them, the optimization of geometric constraints and adaptive texture of the preset three-dimensional Gaussian model includes the simultaneous use of adaptive visibility reconstruction loss, adaptive texture complexity score distillation sampling loss, and depth map and object mask guided geometric loss. These methods can make full use of the texture and geometric guidance provided by the multi-view diffusion model to generate high-quality rendering results.

[0091] First, in the process of optimizing the geometric constraints and adaptive textures of the preset three-dimensional Gaussian model, an adaptive visibility reconstruction loss is introduced. The key is to reduce the weight of the mean square error in the visible area. For example, in S2, the visibility maps under seven viewing angles are calculated. The area with a value of 1 represents the visible area, and the area with a value of 0 represents the invisible area. When calculating the mean square error under one viewing angle, the mean square error of the invisible area is calculated normally, and the mean square error of the visible area is multiplied by a weight less than 1 to reduce the impact of inconsistent areas. This is because the loss on the visible area will be repeatedly calculated between adjacent views, which means that the wrong guidance caused by inconsistent content across views will be amplified.

[0092] Then, unlike the reconstruction loss, the score distillation sampling loss of adaptive texture complexity uses the generative prior of the two-dimensional diffusion model to supervise the learning of the preset three-dimensional Gaussian model, which tends to smooth texture details. Therefore, the weight of the score distillation sampling loss is reduced for areas with high texture complexity. To this end, a score distillation sampling loss of adaptive texture complexity optimized for Gaussian is introduced. In order to detect areas with high texture complexity, Laplace sharpening is performed on each rendered image, where the rendered image refers to multiple iterative optimizations in the process of optimizing the preset three-dimensional Gaussian model. In each iteration, the preset three-dimensional Gaussian model is rendered to obtain a rough three-dimensional Gaussian model under the camera posture p. The rendered image x uses the difference between the sharpened image and the rendered image to locate complex areas, because smooth and flat areas are less affected by the Laplacian filter. For the score distillation sampling loss of complex areas, a smaller weight is added to enhance the texture clarity of the complex texture area.

[0093] Further, in terms of the geometric loss guided by the depth map and object mask, the geometry-related loss is used to constrain the geometric learning of the preset 3D Gaussian model, including: a refined 3D Gaussian model with blurred texture but high-quality geometry is generated and saved in S2 In the iterative optimization of S3, the preset 3D Gaussian model in the current optimization process is rendered to obtain the depth map and object mask under the random camera perspective, and the mean square error between the depth map and object mask of the two stages is calculated. Back propagation is performed to constrain the geometric learning of the preset 3D Gaussian model. These losses are determined by the geometric map in S2 (i.e. and ) and the geometric image rendered by the preset 3D Gaussian model.

[0094] In this embodiment, in the final optimization stage, adaptive visibility reconstruction loss, adaptive texture complexity score distillation sampling loss, and depth map and object mask guided geometric loss are used simultaneously. These methods can make full use of the texture and geometric guidance provided by the multi-view diffusion model to generate high-quality rendering results. Figure 1 The final 3D Gaussian model can be applied to entertainment, education and other fields. Game modeling usually requires professional technicians to spend a lot of time to make it carefully. Figure 1 The final 3D Gaussian model can quickly generate high-quality 3D images from a single image, reducing the cost of game modeling. Figure 1 The final 3D Gaussian model has no strict requirements on the input, only a single image is needed. In the teaching process, educators can efficiently and conveniently use visual Figure 1 The final consistent 3D Gaussian model transforms images into 3D objects with richer information, making teaching more interesting and helping students understand new knowledge clearly and comprehensively.

[0095] Example 3

[0096] This embodiment is a method for generating a visual Figure 1 The verification effect achieved by the self-supervised diffusion method of the three-dimensional Gaussian is further explained.

[0097] Two 3D datasets were selected to evaluate the method of generating a visual image proposed in the above embodiment. Figure 1The verification effect achieved by the self-supervised diffusion method of the consistent three-dimensional Gaussian is shown in the following two 3D datasets, including the Google Scanned Object dataset and the OmniObject3D dataset, which contain a variety of objects. Regarding the Google Scanned Object dataset, 30 objects provided by SyncDreamer are used, which cover a wide range from daily necessities to animals. In addition, 50 objects are randomly selected from OmniObject3D to ensure the diversity of each category, thereby verifying the versatility and high performance of the method of the present invention. For each object, a single image with a resolution of 320x 320 is rendered as the input of the invention and the existing methods. At the same time, in order to calculate the metric, 16 viewing angles are rendered, and the azimuth angles of these viewing angles are evenly distributed from 0 to 360 degrees, and the elevation angle is set to -30 degrees.

[0098] Table 1: Quantitative comparison results of the present invention and the existing method

[0099]

[0100] Table 1 shows the quantitative comparison results of the proposed method and existing methods. In Table 1, key indicators such as PSNR, SSIM and LPIPS are reported to measure the quality of the generated 3D models.

[0101] These three indicators are often used to evaluate image quality. PSNR evaluates the quality of generated images by calculating the peak signal-to-noise ratio between the generated image and the real image. The higher the PSNR, the better the quality of the generated image. SSIM structural loss is an indicator used to measure the similarity between two images. It not only considers the brightness and contrast of the image, but also emphasizes the structural properties of the image. It evaluates the overall similarity of two images by comparing their similarities in brightness, contrast and structure. The higher the SSIM, the more similar the two images are. LPIPS perceptual loss evaluates the similarity between two images. The smaller the value, the more similar they are. It is a deep learning-based evaluation method that is more in line with human perception.

[0102] Figure 7 and Figure 8The qualitative comparison results of the proposed method with other works on the Google Scanned Object dataset and the OmniObject3D dataset are shown respectively. In the image comparison, the first row is the input single image, the second to fifth rows are SyncDreamer, Wonder3D, DreamerGaussian, and the 3D results generated by the proposed method (VC-Gaussian), and the last row is the real result. SyncDreamer often generates smooth geometric shapes, but usually lacks finer details. Compared with SyncDreamer, Wonder3D can capture more detailed textures, but sometimes produces unrealistic geometric shapes due to inconsistencies in sparse multi-view images. DreamGaussian can generate 3D models that match the reference image, but often produces blurred textures in unseen areas. This is because it only supplements the information of the new perspective through SDS optimization. The randomness of the diffusion model leads to blurred textures. Our method not only generates high-quality 3D Gaussian models that are aligned with the input image and maintain consistency between different perspectives, but also performs well in generating richer details and clearer textures.

[0103] The proposed method shows better performance than existing methods in all these indicators. In quantitative and qualitative comparisons, the proposed method stands out, not only generating a high-quality final 3D Gaussian model that is aligned with the input image and maintains consistency across different viewpoints, but also performing well in producing richer details and clearer textures.

[0104] Example 4

[0105] See also Fig. 9 This embodiment proposes a method for generating a visual Figure 1 A consistent three-dimensional Gaussian self-supervised diffusion system, including:

[0106] A multi-view generation module, used to obtain an input image, perform geometrically consistent multi-view generation on the input image, and obtain a refined multi-view image;

[0107] A geometric correlation prior generation module, used to generate a geometric correlation prior according to the input image and the multi-view image;

[0108] A Gaussian optimization module is used to use the geometric correlation prior as the geometric supervision of a preset three-dimensional Gaussian model, optimize the preset three-dimensional Gaussian model, and obtain a visual Figure 1 The final three-dimensional Gaussian model is consistent;

[0109] The output module is used to render the final three-dimensional Gaussian model to obtain a high-fidelity image result.

[0110] In this embodiment, a three-stage framework is adopted. First, in the first stage, geometrically consistent multi-view generation is performed on the input image to obtain a refined multi-view image. Secondly, in the second stage, a geometric correlation prior is generated based on the input image and the multi-view image. Then, in the third stage, the geometric correlation prior is used as the geometric supervision of the preset three-dimensional Gaussian model, and the optimization algorithm is combined to perform visual Figure 1 Finally, the final 3D Gaussian model is rendered to obtain a high-fidelity image result to achieve visual Figure 1 Consistent and high-quality 3D generation.

[0111] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A self-supervised diffusion method for generating view-consistent 3D Gaussians, characterized in that: The following steps are involved: S1. Obtain an input image, generate a geometrically consistent multi-view of the input image, and obtain a refined multi-view image; S2. generating a geometric correlation prior based on the input image and the multi-view image; S3. Using the geometric correlation prior as geometric supervision of a preset three-dimensional Gaussian model, optimizing the preset three-dimensional Gaussian model to obtain a final three-dimensional Gaussian model consistent with the view; S4. Render the final three-dimensional Gaussian model to obtain a high-fidelity image result.

2. The self-supervised diffusion method for generating view-consistent 3D Gaussians according to claim 1, characterized in that: The step of generating a geometrically consistent multi-view image for the input image to obtain a refined multi-view image includes: S11. Inputting the input image into a preset multi-view diffusion model, and outputting a plurality of supplementary multi-views under camera postures; S12. Based on the input image and the supplementary multi-view, the preset rough three-dimensional Gaussian model is optimized to obtain an optimized rough three-dimensional Gaussian model, and the optimized rough three-dimensional Gaussian model is rendered to obtain a new rendered image; S13. Optimize the new rendered image to obtain a refined multi-view image under the camera posture.

3. The self-supervised diffusion method for generating view-consistent 3D Gaussians according to claim 2, characterized in that: The optimization of the preset rough three-dimensional Gaussian model includes: S121. Using the input image I0 and the supplementary multi-view Construct the first score distillation sampling loss function; S122. Construct the rough three-dimensional Gaussian model by minimizing the first score distillation sampling loss function Get the optimized rough three-dimensional Gaussian model 4. The self-supervised diffusion method for generating view-consistent 3D Gaussians according to claim 3, characterized in that: The calculation expression of the first score distillation sampling loss function is as follows: in, represents the first score distillation sampling loss function, Θ represents the rough 3D Gaussian model , t represents a randomly sampled time step, p represents a randomly sampled camera pose, ∈ represents random Gaussian noise, ω(t) represents the weight associated with the time step, ∈ φ (·) represents the pre-trained noise predictor, x represents the rough 3D Gaussian model under camera pose p Rendered image, x r Represents an image set Selected images.

5. The self-supervised diffusion method for generating view-consistent 3D Gaussians according to claim 2, characterized in that: The optimizing of the new rendered image comprises: gradually denoising the new rendered image to generate a refined multi-view image with consistent geometry and clear texture.

6. The self-supervised diffusion method for generating view-consistent 3D Gaussians according to claim 5, characterized in that: The step of generating a geometric correlation prior based on the input image and the multi-view image comprises: S21. Using the input image I0 and the multi-view image Preset refined 3D Gaussian model Optimize and obtain an optimized and refined three-dimensional Gaussian model S22. Based on the input image I0 and the multi-view image The camera pose is optimized for the refined 3D Gaussian model Render and get the depth map and object mask And respectively combine the input image I0 and the multi-view image The supplementary multi-view With the multi-view image The pixel-level correspondence is converted into a visibility map S23. The depth map The object mask and the visibility graph Denoted as the geometric correlation prior.

7. The self-supervised diffusion method for generating view-consistent 3D Gaussians according to claim 6, characterized in that: The preset refined three-dimensional Gaussian model Optimizations include: S211. Using the input image I0 and the multi-view image Construct the second score distillation sampling loss function; S212. Constructing the refined three-dimensional Gaussian model by minimizing the second score distillation sampling loss function Get an optimized and refined three-dimensional Gaussian model 8. The self-supervised diffusion method for generating view-consistent 3D Gaussians according to claim 7, characterized in that: The calculation expression of the second score distillation sampling loss function is as follows: in, represents the second score distillation sampling loss function, Θ′ represents the refined three-dimensional Gaussian model , t represents a randomly sampled time step, p represents a randomly sampled camera pose, ∈ represents random Gaussian noise, ω(t) represents the weight associated with the time step, ∈ φ (·) represents the pre-trained noise predictor, and x′ represents the refined 3D Gaussian model under camera pose p. Rendered image, x r ′ represents the image set Selected images.

9. The self-supervised diffusion method for generating view-consistent 3D Gaussians according to claim 6, characterized in that: Using the depth map in the geometric correlation prior Object Mask and visibility graph The preset three-dimensional Gaussian model is optimized in terms of geometric constraints and adaptive texture to obtain the final three-dimensional Gaussian model.

10. A self-supervised diffusion system for generating view-consistent 3D Gaussians, characterized in that include: A multi-view generation module, used to obtain an input image, perform geometrically consistent multi-view generation on the input image, and obtain a refined multi-view image; A geometric correlation prior generation module, used to generate a geometric correlation prior according to the input image and the multi-view image; A Gaussian optimization module, configured to use the geometric correlation prior as geometric supervision of a preset three-dimensional Gaussian model, optimize the preset three-dimensional Gaussian model, and obtain a final three-dimensional Gaussian model with consistent views; The output module is used to render the final three-dimensional Gaussian model to obtain a high-fidelity image result.

Images