Three-dimensional scene generation method based on diffusion model and gaussian splatting and related device

Abstract

The application discloses a three-dimensional scene generation method based on a diffusion model and Gaussian splashing and related equipment, and belongs to the field of computer vision. The method comprises a point cloud construction stage and a three-dimensional Gaussian optimization stage. In the point cloud construction stage, initial point clouds are extracted from input initial images, and the point clouds are expanded through a Stable Diffusion image restoration model, a monocular depth estimation model and a boundary-aware depth alignment module to create a scene. In the three-dimensional Gaussian optimization stage, a three-dimensional Gaussian is initialized according to the obtained point clouds, the entire scene is represented by using the three-dimensional Gaussian, and the parameters of the three-dimensional Gaussian are optimized by using multi-view images as a supervision signal, so that an immersive three-dimensional scene is obtained after optimization. The application can generate a high-fidelity three-dimensional scene with geometric accuracy and natural appearance without training any three-dimensional data, solves the problem of dependence on three-dimensional scene data, has a small demand for computing resources, and reduces the production cost of the three-dimensional scene.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and related equipment for generating three-dimensional scenes based on a diffusion model and Gaussian splashing. Background Technology

[0002] With the booming development of industries such as virtual reality (VR), augmented reality (AR), gaming, and film, the demand for 3D content creation, especially 3D scene generation technology, is becoming increasingly strong. Traditional 3D scene generation requires modelers to manually create the entire 3D scene, which is then imported into relevant modeling software for rendering, resulting in high costs and labor consumption. Currently, most deep learning-based 3D scene generation methods require training with 3D scene data. Due to the scarcity of 3D data, these methods typically lack generalization ability and are limited to generating specific 3D scenes. Summary of the Invention

[0003] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of this invention is to provide a three-dimensional scene generation method and related equipment based on diffusion model and Gaussian splashing.

[0004] The first technical solution adopted in this invention is:

[0005] A 3D scene generation method based on diffusion model and Gaussian splashing includes a point cloud construction stage and a 3D Gaussian optimization stage:

[0006] In the point cloud construction stage, an initial point cloud is extracted from the input initial image, and the point cloud is expanded through a Stable Diffusion image inpainting model, a monocular depth estimation model, and a boundary-aware depth alignment module to create a scene.

[0007] In the 3D Gaussian optimization stage, a 3D Gaussian is initialized based on the obtained point cloud, and the entire scene is represented by the 3D Gaussian. Multi-view images are used as supervision signals to optimize the parameters of the 3D Gaussian. After optimization, an immersive 3D scene is obtained.

[0008] Furthermore, the point cloud construction stage includes:

[0009] The initial image is input into a monocular depth estimation model to obtain a depth map, and the initial point cloud is obtained from the initial image using the RGB-D boosting method.

[0010] Construct an instruction set for Stable Diffusion image inpainting, input the image and instruction set into a multimodal large model (such as GPT-4o), and obtain the corresponding prompts;

[0011] Using preset camera parameters and trajectories, the following process is repeated iteratively until preset conditions are met to obtain the final scene point cloud:

[0012] The point cloud is rendered to another viewpoint using a renderer to obtain the image to be repaired, the mask of the area to be repaired, and a partial depth map.

[0013] The prompt, the image to be repaired, and the mask of the area to be repaired are input into the StableDiffusion image inpainting model to obtain the complete image.

[0014] The image is input into the monocular depth estimation model to obtain the corresponding depth map. This depth map and the rendered partial depth map are then input into the boundary-aware depth alignment module to obtain the final depth map.

[0015] The final depth map and the complete image are upscaled to a 3D point cloud using the RGB-D upscaling method, and then stitched together with the previously obtained point cloud.

[0016] Further, the extraction of the initial point cloud from the input initial image includes:

[0017] The initial image is defined as I0∈R 3×H×W The initial depth map is defined as D0∈R H×W H and W represent the height and width of the image; the camera's intrinsic parameters are defined as K, and the camera's initial extrinsic parameters are defined as E0, where K and E0 are preset parameter values;

[0018] From the RGB-D image [I0,D0], pixels are uplifted to the initial point cloud P0 using the following method:

[0019] D0 = d(I0)

[0020] P0 = f 2→3 ([I0,D0],K,E0)

[0021] In the formula, d() represents the monocular depth estimation model, and f 2→3 This represents the back projection function from 2D to 3D.

[0022] Furthermore, the entire 3D scene point cloud is generated by sequentially aggregating the point clouds onto the initial point cloud:

[0023] Set a camera trajectory of length N, using E i This represents the extrinsic parameters of the i-th camera. The i-th iteration steps are as follows:

[0024] Rendering point clouds to E using the PyTorch3D renderer i The rendered image obtained from the corresponding camera viewpoint Rendered depth and the rendered mask

[0025]

[0026] In the formula, π represents the renderer;

[0027] The rendered image Mask indicating the repair area The prompt word T is input into the StableDiffusion image inpainting model g to obtain the inpainted image I. i+1 :

[0028]

[0029] In the formula, Indicates will The mask obtained by inverting the pointer indicates the repair area;

[0030] The repaired image I i+1 The data is input into a monocular depth estimation model to obtain a depth map. Will and The input is aligned to the boundary-aware depth alignment module to obtain the final depth map D. i+1 ;

[0031] The final depth map D i+1 And the complete image I i+1 Upgraded to 3D point cloud using RGB-D enhancement method Then compare it with the point cloud P obtained earlier. i Aggregation yields P i+1 :

[0032]

[0033] In the formula, ∪ represents the union operation, and the final point cloud P is obtained through iterative operations. N .

[0034] Furthermore, the boundary-aware depth alignment module aligns in the following manner. and

[0035] The boundary map is obtained using the following method.

[0036]

[0037] In the formula, ∩ and ∩ represent the dilation and intersection operations, respectively, and B represents the structuring element used for the dilation operation; to obtain the coarse alignment depth Model the alignment problem as a least-squares problem:

[0038]

[0039] In the formula, ⊙ represents the Hadhamada product, which is scaled and shifted using α and β. This increases the depth of the boundary region rendering. Depth estimated by monoculars Alignment, to obtain the coarse alignment depth.

[0040]

[0041] Use the Navier-Stokes deep repair algorithm to repair get Next and The final depth D is obtained by performing linear weighted mixing. i+1 :

[0042]

[0043] In the formula, w i+1 It is a mixed weight.

[0044] Furthermore, the three-dimensional Gaussian optimization stage includes:

[0045] The mean μ of the 3D Gaussian ellipsoid is initialized using the point cloud obtained in the point cloud construction stage. The generated point cloud is then rendered using a renderer on a preset additional camera trajectory to obtain M additional images as supervision signals to supervise the optimization of the 3D Gaussian splashing:

[0046]

[0047] In the formula, π represents the renderer;

[0048] During the optimization process, only supervision is performed. The pixels; the loss function is as follows:

[0049] L=(1-λ)L1+λL D-SSIM

[0050] In the formula, L1 represents the L1 loss of pixels between the rendered image and the real image, L D-SSIM This represents the D-SSIM loss of pixels between the rendered image and the real image, where λ is the scaling factor.

[0051] Furthermore, in the neural radiation field optimization stage, the scene in the three-dimensional space is represented as a density field and a color field using neural radiation field (NeRF). The model parameters are optimized through volume rendering, the three-dimensional scene is reconstructed using multi-view images as supervision signals, and new viewpoints are synthesized.

[0052] The second technical solution adopted in this invention is:

[0053] A 3D scene generation device based on a diffusion model and Gaussian splashing, comprising:

[0054] The point cloud building module is used to extract an initial point cloud from the input initial image and expand the point cloud through the StableDiffusion image inpainting model, monocular depth estimation model and boundary-aware depth alignment module to create a scene;

[0055] The 3D Gaussian optimization module is used to initialize a 3D Gaussian based on the obtained point cloud, represent the entire scene using a 3D Gaussian, and use multi-view images as supervision signals to optimize the parameters of the 3D Gaussian. After optimization, an immersive 3D scene is obtained.

[0056] The third technical solution adopted in this invention is:

[0057] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a three-dimensional scene generation method based on a diffusion model and Gaussian splashing as described above.

[0058] The fourth technical solution adopted in this invention is:

[0059] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a three-dimensional scene generation method based on a diffusion model and Gaussian splashing as described above.

[0060] The fifth technical solution adopted in this invention is:

[0061] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method for generating a 3D scene based on a diffusion model and Gaussian splashing.

[0062] The beneficial effects of this invention are: this invention can generate geometrically accurate and naturally-looking high-fidelity 3D scenes without any 3D data for training, effectively solving the problem of dependence on 3D scene data, and requiring little computing power, thus greatly reducing the production cost of 3D scenes. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart illustrating the workflow of a three-dimensional scene generation method based on a diffusion model and Gaussian splashing in an embodiment of the present invention.

[0065] Figure 2 This is a flowchart of the boundary-aware depth alignment module in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the quantitative results in the first scenario of this invention.

[0067] Figure 4 This is a schematic diagram of the qualitative results in the first scenario of this invention embodiment;

[0068] Figure 5 This is a schematic diagram of the quantitative results in the second scenario of this invention.

[0069] Figure 6 This is a schematic diagram of the qualitative results in the second scenario of this invention embodiment;

[0070] Figure 7 This is a schematic diagram of the quantitative results in the third scenario of this invention.

[0071] Figure 8 This is a schematic diagram of the qualitative results in the third scenario of this invention embodiment;

[0072] Figure 9 This is a comparison chart of the qualitative results of the method provided by this invention and the LucidDreamer method. Detailed Implementation

[0073] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0074] The terminology used in the embodiments of this application is for the purpose of describing specific embodiments only and is not intended to limit the embodiments of this application. The singular forms "a," "described," and "the" used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. Furthermore, unless otherwise expressly limited, terms such as "set," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0075] In the description of this application, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0076] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0077] In the description of this application, "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship.

[0078] Terminology Explanation:

[0079] Diffusion models: Diffusion models are a type of generative model based on deep learning. Their core idea is to generate data by simulating the "diffusion" process of data over time, adding or removing noise from the data.

[0080] Stable Diffusion Inpainting (SDI) is an image inpainting model that takes as input an image to be inpainted, a mask (marking the parts that need to be inpainted), and a corresponding prompt. It gradually removes noise within the masked area while using the unmasked area (known parts) and the corresponding inpainting text as conditions to guide the inpainting process.

[0081] Monocular depth estimation: Monocular depth estimation refers to a technique that uses only a single digital image (or a still frame from an image sequence) as input information, and predicts and outputs the distance (depth) values ​​of each pixel in a scene relative to the camera's optical center (or reference plane) using computer vision and / or deep learning algorithms. By inputting an RGB image, the monocular depth estimation model predicts and outputs the depth values ​​of each pixel in the scene relative to the camera's optical center (or reference plane). It can be divided into relative depth estimation and absolute depth estimation. Relative depth estimation outputs the relative order of depth information, rather than specific depth values. Absolute depth estimation outputs the depth values ​​relative to the camera's imaging plane. Existing methods all use absolute depth estimation models.

[0082] Depth alignment: Depth alignment is a method for aligning depth maps with overlapping regions from different viewpoints. For RGB images with different viewpoints and overlapping viewpoints, estimating depth using a monocular depth estimation model and directly upscaling them to 3D space using RGB-D upscaling methods can lead to discontinuities in the depth of overlapping regions. Existing methods upscale RGB images from different viewpoints to point clouds and then optimize the L2 distance of the point clouds in the overlapping regions to alleviate these discontinuities.

[0083] Gaussian Splatting: Gaussian splatting is a rendering method that represents a 3D scene as a point cloud defined by a Gaussian distribution. It achieves efficient image generation and rendering by projecting these Gaussian points onto a 2D image plane.

[0084] 3D scene reconstruction based on Gaussian splashing: By initializing the 3D point cloud, the 3D Gaussian parameter is parameterized as mean, covariance, opacity and color. Multi-view images are used as supervision signals to optimize the parameters of the 3D Gaussian, so as to achieve efficient and fast 3D scene reconstruction and new perspective synthesis.

[0085] Multimodal understanding and generation: Multimodal understanding and generation refers to using a large multimodal model to process image and text command data, and output corresponding image understanding content, thereby achieving efficient and accurate image content understanding.

[0086] GPT-4o: A multimodal large model released by OpenAI that can perform inference on audio, vision, and text in real time.

[0087] Currently, the existing technology still has the following technical problems:

[0088] (1) Training paradigm using 3D scene data

[0089] a) Dependence on 3D scene data: In reality, 3D scene data is usually obtained through 3D scanners. Compared with image data, 3D scene data is difficult to obtain and is very scarce. Due to the scarcity of data, the generalization ability of existing methods trained based on 3D scene data is insufficient, and they can only generate specific 3D scenes.

[0090] b) Requires a lot of computing resources: 3D data is more complex than 2D data. Directly using 3D scene data to train deep neural networks usually requires a lot of computing resources, which hinders the democratization of 3D content creation.

[0091] (2) Prompt word engineering for diffusion model image inpainting

[0092] a) Time-consuming, labor-intensive, and inefficient: The prompts input into the diffusion model in the existing methods are all given manually, which requires manual mining of image content and style, and is a time-consuming and labor-intensive task.

[0093] b) Insufficient understanding of image content and style: Existing methods rely on manually specified diffusion model prompts, which make it difficult to grasp the overall content and style of the image from a global perspective. The prompts are localized, leading to content offset issues during the restoration process, which greatly affects subsequent restoration processes.

[0094] (3) Depth alignment of different overlapping viewpoints

[0095] a) Poor alignment: Existing methods alleviate the discontinuity of overlapping region depth by optimizing the L2 distance of the overlapping region point cloud. However, the problem of breakage at the edge of the projection region of the overlapping region point cloud still exists. Poor alignment at the edge leads to distortion of the point cloud geometry.

[0096] b) Unsuitable for real-time application: Existing methods employ online test-time training techniques or iterative distance optimization methods to promote consistency between the predicted depth map of the current frame and the geometry of the synthesized scene. However, these techniques require a certain amount of time to achieve depth alignment, making them unsuitable for real-time applications.

[0097] To address the problems of existing technologies, this invention proposes an innovative method for generating 3D scenes based on a diffusion model and Gaussian splashing. This method can generate geometrically accurate and naturally appearing high-fidelity 3D scenes without requiring any 3D data for training. This method solves the problem of dependence on 3D scene data, requires minimal computing resources, and significantly reduces the production cost of 3D scenes.

[0098] Specifically, this method first constructs an instruction for image understanding, innovatively introducing GPT-4o for deep understanding of image content and style to obtain structured prompts for diffusion model image inpainting. Second, it employs viewpoint projection and diffusion model image inpainting to acquire images from new perspectives. It innovatively invents a boundary-aware depth alignment method to obtain continuous depth at the projection boundaries of the image, and uses RGB-D upscaling to upscale the image space to a 3D point cloud space. Repeating the above process yields the final 3D scene point cloud. Finally, Gaussian splashing is used as the scene representation for subsequent rendering. By fully utilizing the image generation prior of the diffusion model, it avoids directly training neural networks with 3D data, reducing the production cost of 3D scenes and democratizing the creation of 3D content.

[0099] The technical solution of this invention can solve the following problems:

[0100] (1) Generation of 3D scenes based on prior knowledge generated by diffusion model

[0101] By using an iterative optimization method, the problem of dependence on large amounts of 3D data and computing resources was solved, and the generalization ability of 3D scene generation was improved.

[0102] (2) Image Inpainting with Prompt Text Based on Multimodal Large Model

[0103] a) An automated prompt generation workflow was built, which avoided manual intervention in the prompt engineering process, greatly improved the efficiency of prompt engineering, and solved the problems of time-consuming, labor-intensive and inefficient prompt engineering.

[0104] b) Designed an accurate instruction set to guide multimodal large models, and introduced GPT-4o to grasp the overall image content and style, solving the limitations of prompt words and the content offset problem in the image restoration process.

[0105] (3) Boundary-aware depth alignment technology

[0106] a) The boundary pre-alignment technique based on least squares and the depth weighted fusion technique solve the problem of depth discontinuity at the projection edge in the existing methods, so that the point cloud is better aligned at the projection edge.

[0107] b) It eliminates the need for time-consuming operations such as iterative optimization of point cloud distances or online testing training techniques. Instead, it directly employs a boundary pre-alignment technique based on the least squares method, thus solving the problem of real-time application of depth alignment algorithms.

[0108] Example 1

[0109] like Figure 1As shown, this embodiment provides a 3D scene generation method based on a diffusion model and Gaussian splashing. The input is an image, and the output is the corresponding 3D scene. This allows users to immerse themselves in the 3D scene and obtain the corresponding point cloud data for downstream processing. The method specifically includes a point cloud construction stage and a 3D Gaussian optimization stage.

[0110] In the point cloud construction stage, an initial point cloud is extracted from the input initial image, and the point cloud is expanded through a Stable Diffusion image inpainting model, a monocular depth estimation model, and a boundary-aware depth alignment module to create a scene.

[0111] In the 3D Gaussian optimization stage, a 3D Gaussian is initialized based on the obtained point cloud. The entire scene is represented by the 3D Gaussian, and multi-view images are used as supervision signals to optimize the parameters of the 3D Gaussian. After optimization, an immersive 3D scene can be obtained.

[0112] As another implementation, a neural radiation field optimization stage is used instead of a three-dimensional Gaussian optimization stage, that is, a neural radiation field (NeRF) is used instead of a three-dimensional Gaussian splashing stage.

[0113] In the neural radiation field optimization stage, the scene in 3D space is represented as a density field and a color field using neural radiation fields. Model parameters are optimized through volume rendering, and multi-view images are used as supervisory signals to reconstruct the 3D scene, followed by new viewpoint synthesis. The advantage of this alternative is that NeRF can better model high-frequency features of the scene, such as edge information, which can further improve rendering quality.

[0114] The above method will be explained in detail below with reference to the accompanying drawings and specific embodiments.

[0115] (1) Point cloud construction stage

[0116] In order to generate multiple views Figure 1 To obtain an aligned point cloud, this embodiment first inputs the initial image into a monocular depth estimation model to acquire a depth map, and then uses the RGB-D upscaling method to obtain the initial point cloud from the initial image. Next, a command set for Stable Diffusion image inpainting is constructed, and the image and command set are input into GPT-4o to obtain the corresponding prompts. Using preset camera parameters and trajectories, the following process is iteratively repeated:

[0117] 1) Use a renderer to render the point cloud to another viewpoint to obtain the image to be repaired, the mask of the area to be repaired, and a partial depth map;

[0118] 2) Input the prompt, the image to be repaired, and the mask of the area to be repaired into the StableDiffusion image inpainting model to obtain the complete image;

[0119] 3) Input the image into the monocular depth estimation model to obtain the corresponding depth map. Input this depth map and the rendered partial depth map into the boundary-aware depth alignment module to obtain the final depth map.

[0120] 4) Upgrade the final depth map and the complete image to a 3D point cloud using the RGB-D upscaling method, and then stitch them together with the previously obtained point cloud;

[0121] After the above process is completed, the final scene point cloud can be obtained.

[0122] (1.1) Point cloud initialization

[0123] In some embodiments, the MoGe monocular depth estimation model (denoted by d) is used to estimate the depth of the image. The initial image is defined as I0∈R. 3×H×W The initial depth map is defined as D0∈R H×W H and W represent the height and width of the image. The camera's intrinsic parameters are defined as K, and the camera's initial extrinsic parameters are defined as E0, where K and E0 are preset parameter values.

[0124] From the RGB-D image [I0, D0], pixels are uplifted to the initial point cloud P0 using the following method:

[0125] D0 = d(I0)

[0126] P0 = f 2→3 ([I0,D0],K,E0)

[0127] As an alternative implementation, a monocular depth estimation model such as DepthAnything or Marigold can be used instead of MoGe. The advantage of this alternative is that monocular depth estimation models such as DepthAnything or Marigold can produce better depth data in unbounded regions such as the sky.

[0128] (1.2) Point cloud iterative aggregation

[0129] This embodiment uses a method of sequentially aggregating point clouds onto an initial point cloud to generate the entire 3D scene point cloud.

[0130] Specifically, a camera trajectory of length N was set, using E i This represents the extrinsic parameters of the i-th camera. Taking the i-th iteration step as an example:

[0131] First, use the PyTorch3d renderer (represented by π) to render the point cloud to E. i The rendered image obtained from the corresponding camera viewpoint Rendered depth and the rendered mask

[0132]

[0133] Second, the rendered image Mask indicating the repair area The prompt word T is input into the Stable Diffusion image restoration model g to obtain the restored image I. i+1 :

[0134]

[0135] In the formula, This refers to The mask obtained by inverting the pointer indicates the repair area.

[0136] Third, the repaired image I i+1 Input into MoGe to obtain a depth map. D i+1 and The input is fed into the boundary-aware depth alignment module to obtain the final depth map D. i+1 :

[0137]

[0138] It is worth noting that, and Overlapping areas may not align. To ensure seamless integration of the generated point cloud with existing point clouds, alignment is necessary. and This avoids discontinuities in the point cloud at the projection boundaries and rendering artifacts.

[0139] Fourth, the final depth map D i+1 And the complete image I i+1 Upgraded to 3D point cloud using RGB-D enhancement method And the point cloud P obtained previously i Aggregation yields P i+1 :

[0140]

[0141] ∪ represents the union operation. By iteratively repeating the above operation, the final point cloud P is obtained. N .

[0142] (1.3) Boundary-aware depth alignment module

[0143] To avoid discontinuities in the point cloud and rendering artifacts at the projection boundaries, this embodiment proposes a boundary-aware depth alignment module, such as... Figure 2 As shown, its working principle is as follows:

[0144] First, obtain the boundary map using the following method.

[0145]

[0146] ∩ and ∩ represent the dilation and intersection operations, respectively, and B represents the structuring element used for the dilation operation. To obtain the coarse alignment depth... This embodiment models the alignment problem as a least-squares problem:

[0147]

[0148] ⊙ represents the Hadramad product, using α and β to scale and move. This increases the depth of the boundary region rendering. Depth estimated by monoculars Alignment. This allows us to obtain the coarse alignment depth.

[0149]

[0150] To ensure smoother connections at the edges, this embodiment further proposes a deep blending technique. Specifically, this embodiment first uses the Navier-Stokes deep inpainting algorithm to repair... get Next and The final depth D is obtained by performing linear weighted mixing. i+1 :

[0151]

[0152] (2) Three-dimensional Gaussian splashing optimization stage

[0153] In this embodiment, the point cloud obtained during the point cloud construction stage is used to initialize the mean μ of the 3D Gaussian ellipsoid. The generated point cloud is then rendered using PyTorch3D on a preset additional camera trajectory to obtain additional M images, which are used as supervision signals to supervise the optimization of the 3D Gaussian splashing.

[0154]

[0155] During the optimization process, only supervision is performed. The number of pixels helps prevent the model from learning incorrect details. The loss function is as follows:

[0156] L=(1-λ)L1+λL D-S1IM

[0157] In the formula, L1 represents the L1 loss of pixels between the rendered image and the real image, L D-SSIM This represents the D-SSIM loss of pixels between the rendered image and the real image.

[0158] (2.1) 3D Gaussian Splatting

[0159] 3D Gaussian splashing uses a 3D Gaussian ellipsoid as a volumetric representation of the radiation field, enabling high-quality real-time rendering. This method initializes the scene point cloud as anisotropic 3D Gaussian ellipsoids and uses multi-view images as supervision signals to optimize the parameters of each 3D Gaussian ellipsoid: mean μ, scaling matrix S, rotation matrix R, opacity α, and color c. Each Gaussian ellipsoid is defined by a covariance matrix centered at the mean μ.

[0160]

[0161] It is worth noting that the covariance matrix only has physical meaning when it is positive semidefinite, and this condition cannot be guaranteed when using gradient descent for optimization. Therefore, Kerbl et al. defined a parametric ellipse with a scaling matrix S and a rotation matrix R, and then constructed the covariance matrix as follows:

[0162] Σ=RSS T R T

[0163] Therefore, a three-dimensional Gaussian ellipsoid can be parameterized as a mean. Scaling vector and quaternions To represent this. In the embodiment, a rotation matrix is ​​used. This is used to represent the corresponding rotation.

[0164] During rendering, the color C of a pixel is calculated by mixing all 3D Gaussian splatter points that overlap with the pixel:

[0165]

[0166] Where c i This represents the color of each point, which is indicated by the spherical harmonic coefficient. d i This is the value obtained by projecting the center of a 3D Gaussian ellipsoid onto the z-axis of the camera coordinate system. Mixing weight α ′The opacity α of each point is determined by multiplying the projection of the three-dimensional Gaussian ellipsoid onto the two-dimensional plane. To maintain the visibility order, the Gaussian splatter points are sorted according to depth before blending.

[0167] (3) Experimental Results

[0168] This embodiment uses LucidDreamer as the baseline model and demonstrates quantitative and qualitative results across three scenarios. (Higher CLIP-Score, CLIP-IQA Quality, CLIP-IQA Colorful, and CLIP-IQA Sharp are better, while lower BRISQUE and NIQE are better.)

[0169] (3.1) Scene 1: Christmas

[0170] Quantitative results such as Figure 3 As shown. According to Figure 3 As can be seen, the method in this embodiment outperforms LucidDreamer in most metrics, thanks to the multi-image inpainting prompt word engineering based on a multimodal large model and the boundary-aware alignment method.

[0171] Qualitative results such as Figure 4 As shown in the rendering, the method in this embodiment is able to better follow the content and style of the input image, and the lighting and atmosphere are reproduced more faithfully, while LucidDreamer exhibits a style shift problem.

[0172] (3.2) Scene 2: Kitchen

[0173] Quantitative results such as Figure 5 As shown. According to Figure 5 As can be seen, the method in this embodiment outperforms LucidDreamer across all metrics, thanks to the multi-image inpainting prompt word engineering based on a multimodal large model and the boundary-aware alignment method.

[0174] Qualitative results such as Figure 6 As shown in the image, the method in this embodiment better adheres to the content and style of the input image, resulting in higher quality rendered images.

[0175] (3.3) Scenario 3: Fruit Supermarket

[0176] Quantitative results such as Figure 7 As shown. According to Figure 7 As can be seen, the method in this embodiment outperforms LucidDreamer in most metrics, thanks to the multi-image inpainting prompt word engineering based on a multimodal large model and the boundary-aware alignment method.

[0177] Qualitative results such as Figure 8 As shown in the rendered image, the method in this embodiment better follows the content and style of the input image, without any content offset issues. In contrast, LucidDreamer exhibits content offset problems, losing the main content of the input image during the image restoration process.

[0178] The running time of this invention in the depth alignment module is compared with the quantitative results of LucidDreamer (the comparison is the time required to run the depth alignment module once), as shown in Table 1:

[0179] Table 1

[0180] LucidDreamer 0.827s Ours 0.068s

[0181] As can be seen from Table 1, the depth alignment module in this embodiment can run in real time, and its running speed is more than 10 times faster than the previous method.

[0182] Regarding the edge alignment and fidelity of the generated point cloud, the method in this embodiment is compared with the qualitative results of LucidDreamer. Figure 9 As shown, point clouds for three different scenarios are presented. Qualitative results indicate that the method in this embodiment produces point clouds with better structure and fewer flying points compared to those generated by LucidDreamer; the connections at the edges are more natural and continuous, significantly improving the fidelity of the point cloud.

[0183] (4) Beneficial effects

[0184] In summary, compared with the prior art, the present invention has at least the following advantages and beneficial effects:

[0185] 1) This invention employs an iterative optimization method for generating 3D scenes, distilling large-scale prior knowledge of generation from a diffusion model. This avoids the problem of directly training neural networks using 3D scene data, improving the model's generalization ability and the diversity of generation. The entire system can run on a single NVIDIA RTX 3090 graphics card, saving significant computing resources compared to previous methods that used 3D data training (requiring an NVIDIA A100 graphics card).

[0186] 2) This invention adopts a multi-image restoration prompt word engineering based on a multimodal large model. Compared with the previous method, it greatly follows the content and style of the input image, and improves the fidelity and accuracy of the 3D scene generation. There is no need to manually provide prompt words for image restoration, making the whole system more automated.

[0187] 3) This invention adopts a boundary-aware depth alignment method, which solves the problem of depth discontinuity at the projection edge in existing methods, resulting in better alignment of point clouds at the projection edge and obtaining a better point cloud structure; it also solves the problem that previous depth alignment methods could not respond in real time, and the boundary-aware depth alignment method can align depths in real time.

[0188] 4) In terms of CLIP-Score, CLIP-IQA Quality, CLIP-IQA Colorful, CLIP-IQA Sharp, BRISQUE, and NIQE, the present invention outperforms previous methods.

[0189] 5) In virtual reality (VR) and augmented reality (AR), this invention can generate immersive 3D scenes that allow users to freely navigate and browse. For example, a user can select a favorite image and use this method to generate a corresponding 3D scene for navigation and browsing.

[0190] 6) In the gaming field, this invention can generate a 3D scene for immersive experience from a game-related scene image, saving game scene modeling time and improving game scene modeling efficiency.

[0191] 7) In the film and television industry, this invention can quickly generate point clouds of three-dimensional scenes required for movies. By using a mesh reconstruction method, the point clouds can be converted into editable three-dimensional scene models, shortening the design cycle from several weeks to several days and improving the efficiency of film and television production.

[0192] In summary, the technology of this invention has broad application potential in multiple fields such as virtual reality (VR), augmented reality (AR), games, and film and television, greatly reducing the production cost of 3D scenes and improving production efficiency.

[0193] Example 2

[0194] This embodiment provides a 3D scene generation device based on a diffusion model and Gaussian splashing, including:

[0195] The point cloud building module is used to extract an initial point cloud from the input initial image and expand the point cloud through the StableDiffusion image inpainting model, monocular depth estimation model and boundary-aware depth alignment module to create a scene;

[0196] The 3D Gaussian optimization module is used to initialize a 3D Gaussian based on the obtained point cloud, represent the entire scene using a 3D Gaussian, and use multi-view images as supervision signals to optimize the parameters of the 3D Gaussian. After optimization, an immersive 3D scene is obtained.

[0197] Since this device is a 3D scene generation device based on diffusion model and Gaussian splashing according to an embodiment of the present invention, and the principle of solving the problem by this device is similar to that of the method, the implementation of this device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0198] Example 3

[0199] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 1 This paper presents a method for generating 3D scenes based on a diffusion model and Gaussian splashing.

[0200] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0201] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0202] Since this electronic device corresponds to a three-dimensional scene generation method based on diffusion model and Gaussian splashing in an embodiment of the present invention, and the principle of solving the problem by this electronic device is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0203] Example 4

[0204] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 1 This paper presents a method for generating 3D scenes based on a diffusion model and Gaussian splashing.

[0205] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0206] Since this storage medium is the storage medium corresponding to a three-dimensional scene generation method based on diffusion model and Gaussian splashing in an embodiment of the present invention, and the principle of solving the problem by this storage medium is similar to that of this method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0207] Example 5

[0208] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of a three-dimensional scene generation method based on a diffusion model and Gaussian splashing according to various exemplary embodiments of the present application described above. The executable computer program code or "code" for performing the various embodiments can be written in high-level programming languages ​​such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0209] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0210] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0211] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for generating 3D scenes based on a diffusion model and Gaussian splashing, characterized in that, Includes the point cloud construction stage and the 3D Gaussian optimization stage: In the point cloud construction stage, an initial point cloud is extracted from the input initial image, and the point cloud is expanded through a Stable Diffusion image inpainting model, a monocular depth estimation model, and a boundary-aware depth alignment module to create a scene. In the 3D Gaussian optimization stage, a 3D Gaussian is initialized based on the obtained point cloud, and the entire scene is represented by the 3D Gaussian. Multi-view images are used as supervision signals to optimize the parameters of the 3D Gaussian. After optimization, an immersive 3D scene is obtained. The point cloud construction phase includes: The initial image is input into a monocular depth estimation model to obtain a depth map, and the initial point cloud is obtained from the initial image using the RGB-D boosting method. Construct an instruction set for Stable Diffusion image restoration, input the image and instruction set into a multimodal large model, and obtain the corresponding prompt words; Using preset camera parameters and trajectories, the following process is repeated iteratively until preset conditions are met to obtain the final scene point cloud: The point cloud is rendered to another viewpoint using a renderer to obtain the image to be repaired, the mask of the area to be repaired, and a partial depth map. The prompt, the image to be repaired, and the mask of the area to be repaired are input into the Stable Diffusion image inpainting model to obtain the complete image; The image is input into the monocular depth estimation model to obtain the corresponding depth map. This depth map and the rendered partial depth map are then input into the boundary-aware depth alignment module to obtain the final depth map. The final depth map and the complete image are upscaled to a 3D point cloud using the RGB-D upscaling method, and then stitched together with the previously obtained point cloud. The boundary-aware depth alignment module aligns in the following way and : The boundary map is obtained using the following method. : In the formula, and These represent the dilation and intersection operations, respectively. This represents the structuring element used for the dilation operation; to obtain the coarse alignment depth. The alignment problem is modeled as a least squares problem: In the formula, The term "Hadamardi" is used to represent the product of the Hadamardi product. and To zoom and move This makes the rendering depth of the boundary region... Depth estimated by monoculars Alignment, to obtain the coarse alignment depth. : Use deep repair algorithms to repair ,get Next, and The final depth is obtained by performing linear weighted mixing. : In the formula, It is a mixed weight.

2. The method for generating a 3D scene based on a diffusion model and Gaussian splashing as described in claim 1, characterized in that, The step of extracting the initial point cloud from the input initial image includes: Define the initial image as Define the initial depth map as , and This represents the height and width of the image; the camera's intrinsic parameters are defined as follows: The initial extrinsic parameters of the camera are defined as follows: ,in and All of these are preset parameter values; From RGB-D images [ In the [reference], pixels are upscaled to the initial point cloud using the following method. : In the formula, This represents a monocular depth estimation model. This represents the back projection function from 2D to 3D.

3. The method for generating a 3D scene based on a diffusion model and Gaussian splashing as described in claim 1, characterized in that, The entire 3D scene point cloud is generated by sequentially aggregating point clouds onto the initial point cloud: Set the length to The camera trajectory, using Indicates the first The external parameters of the camera, the first The iterative steps are as follows: Render the point cloud using a renderer The rendered image obtained from the corresponding camera viewpoint Rendered depth and the rendered mask : In the formula, Indicates the renderer; The rendered image , mask indicating the repair area and prompt words Input into the StableDiffusion image inpainting model In the process, the repaired image is obtained. : In the formula, By Obtained by inverting; The repaired image The data is input into a monocular depth estimation model to obtain a depth map. ,Will and The input is aligned to the boundary-aware depth alignment module to obtain the final depth map. ; The final depth map and complete image Upgraded to 3D point cloud using RGB-D enhancement method Then, combined with the point cloud obtained earlier Aggregation is performed to obtain : In the formula, This represents the union operation, which is used to obtain the final point cloud through iterative operations. .

4. The method for generating a 3D scene based on a diffusion model and Gaussian splashing as described in claim 3, characterized in that, The three-dimensional Gaussian optimization stage includes: The mean of the 3D Gaussian ellipsoid is initialized using the point cloud obtained during the point cloud construction phase. The generated point cloud is rendered using a renderer along a preset additional camera trajectory to obtain additional... The images are used as supervision signals to supervise the optimization of 3D Gaussian splashing: In the formula, Indicates the renderer; During the optimization process, only supervision is performed. The pixels; the loss function is as follows: In the formula, Represents the pixel difference between the rendered image and the real image. loss, Represents the pixel difference between the rendered image and the real image. loss, This is the proportionality coefficient.

5. The method for generating a 3D scene based on a diffusion model and Gaussian splashing as described in claim 1, characterized in that, The neural radiation field optimization stage is used instead of the three-dimensional Gaussian optimization stage. In the neural radiation field optimization stage, the scene in three-dimensional space is represented as a density field and a color field using neural radiation fields. The model parameters are optimized through volume rendering, and the three-dimensional scene is reconstructed using multi-view images as supervision signals. New viewpoints are then synthesized.

6. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method as described in any one of claims 1 to 5.

8. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a processor, are used to perform the method as described in any one of claims 1 to 5.

Images