Non-domain limitation text three-dimensional scene method based on stable diffusion model

By combining stable diffusion model and three-dimensional Gaussian splashing technology, using point clouds and depth maps to generate consistent three-dimensional scenes in multiple views, solving the problems of inconsistency and insufficient details in the existing technology, achieving high quality and efficient Three-dimensional scene generation.

CN119048680APending Publication Date: 2024-11-29GUANGDONG BOHUA UHD INNOVATION CENT CO LTD
View PDF 0 Cites 8 Cited by

Patent Information

Application Number
CN202411179965.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The prior art is difficult to maintain consistency of three-dimensional scenes between multiple views, and the generated three-dimensional scene details are insufficient to generate high-quality continuous three-dimensional scenes.

Method used

Combining the stable diffusion model with three-dimensional Gaussian splash, through point cloud initialization, camera trajectory setting, generation and alignment processes, the potential diffusion model is used to generate images and estimate depth maps using the monocular depth estimation model to repair and render three-dimensional scenes to ensure multiple Consistency and high-quality rendering of views.

Benefits of technology

It improves the diversity and quality of Wensheng's three-dimensional scenes, improves the generation efficiency, realizes image consistency in different views, and accelerates the rendering process, solving the problems of insufficient details and slow rendering speed in the three-dimensional scene.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048680A_ABST
    Figure CN119048680A_ABST
Patent Text Reader

Abstract

The invention provides a domain-limitation-free text three-dimensional scene method based on a stable diffusion model, and the method comprises the following steps: S1, point cloud initialization and depth estimation: enabling an input text to generate an image related to a given text through a potential diffusion model, and estimating a depth map through a monocular depth estimation model; s2, setting a camera track; dividing the initial point cloud into 8 areas of 2 * 2 * 2, and setting a camera track with the length of # imgabs0 #; s3, generating a new point cloud; s4, aligning the point cloud and the restored image; and S5, performing Gaussian splash rendering to obtain a rendered three-dimensional scene. According to the method, the diversity and quality of the text three-dimensional scene are effectively improved, and the generation efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a domain-free three-dimensional scene generation method based on a stable diffusion model. Background Art

[0002] Generative AI and 3D models are very important in many fields today and have a wide range of application scenarios, such as virtual reality, game development, and video production. 3D Gaussian Splatter (3DGS) was proposed in 2023 and is the most revolutionary innovation in the field of 3D reconstruction in recent years. Compared with the implicit expression of the previous NeRF-based method, 3DGS draws on the advantages of explicit expression from the point cloud-based 3D reconstruction method. Compared with NeRF, which needs to query the neural network to calculate the scene in real time, 3DGS has faster rendering speed and shorter training time.

[0003] With the rapid innovation of 3D graphics technology, high-quality 3D scene generation has become one of the most important problems in computer vision. This requires the ability to create diverse and realistic 3D scenes from various types of inputs (such as text, images). There are some attempts to use diffusion models to directly generate 3D objects and scenes in voxels, point clouds, and implicit neural representations, but due to the limitations of 3D scan-based training data, the result is that neither diversity nor quality is improved. One way to deal with this problem is to leverage the power of pre-trained image generation diffusion models such as Stable Diffusion to create diverse, high-quality 3D scenes. Although this large model can create reasonable images using data-driven knowledge learned from large-scale training data, it does not guarantee multi-view between generated images. Figure 1 Consistency.

[0004] Therefore, in the prior art, the generated three-dimensional scene often cannot maintain consistency between multiple views, and the three-dimensional scene details are insufficient, and the prior art is difficult to generate high-quality continuous three-dimensional scenes.

[0005] The difficulty of solving the above problems and defects is to combine the stable diffusion model with the three-dimensional Gaussian splash, which requires using the stable diffusion model to operate the point cloud so that the three-dimensional Gaussian splash can render a three-dimensional scene.

[0006] The significance of solving the above problems and defects is: it provides a feasible idea of ​​Vincent three-dimensional scene, and is a successful attempt to combine the stable diffusion model with three-dimensional Gaussian splashing. Summary of the invention

[0007] The present invention provides a domain-free Vincent three-dimensional scene method based on a stable diffusion model, which effectively improves the diversity and quality of Vincent three-dimensional scenes and enhances the generation efficiency.

[0008] The technical solution of the present invention is as follows: The method for generating a three-dimensional scene without domain restrictions based on a stable diffusion model of the present invention comprises the following steps: S1. initializing point cloud and estimating depth; S2. setting camera trajectory; S3. generating; S4. aligning point cloud and repairing image; S5. using Gaussian splash rendering to obtain a rendered three-dimensional scene.

[0009] Optionally, in the above-mentioned domain-free text-generated 3D scene method based on the stable diffusion model, in step S1, first, the user's text input is obtained, and an image related to the given text is generated using the potential diffusion model; then the depth map is estimated using the monocular depth estimation model, and the generated RGB image and depth map are respectively represented as and ,in and are the height and width of the image, respectively, generated from the latent diffusion model and , according to the image size to determine the camera intrinsic parameter matrix and the external parameter matrix ; For the input RGBD image [ I 0 , D 0 ] , initialize it as a point cloud in space; the initial point cloud generated using the first image is defined as , whose expression is: P 0 = ϕ 2→3 ([ I 0 , D 0 ], K , R 0 | t 0 ) in, It is a function that upgrades the RGBD image into a point cloud.

[0010] Optionally, in the above-mentioned domain-free Vincent 3D scene method based on the stable diffusion model, in step S2, the initial point cloud is divided into 8 regions of 2*2*2, and the length is set to The camera trajectory is set Indicates that the camera is The position and posture at the index The operation of the step is as follows: first, the camera is moved from the previous position Move and rotate to , transform the coordinates from the world coordinate system to the current camera coordinate system, and use the intrinsic matrix and the external parameter matrix Projected onto the camera plane.

[0011] Optionally, in the above-mentioned domain-free Vincent 3D scene method based on the stable diffusion model, in step S3, the image is projected at the camera position, and a mask is used to distinguish filled and unfilled areas; the image is repaired using the stable diffusion model, and the corresponding depth map is calculated using the same monocular depth estimation network as in step S1; and the optimal depth scaling coefficient is estimated to minimize the distance between the 3D points of the new image and the corresponding points in the initial point cloud.

[0012] Optionally, in the above-mentioned domain-free Vincent 3D scene method based on the stable diffusion model, in step S3, the camera The projected image at , define the mask To distinguish the area that has been filled by the existing points from the unfilled area, if the corresponding pixel has been filled, then The value is 1, otherwise it is 0; use the stable diffusion repair model Incomplete image and mask Generate real images ;Use a monocular depth estimation network Estimate the corresponding depth map ; Estimate the optimal depth scaling factor , so that the distance between the 3D points of the new image and the corresponding points in the initial point cloud is minimized; then the coefficients are Multiply by the estimated depth map To calculate the actual depth map , d i = arg min ( ∑ M i =1 ϕ 2→3 ([ I i , d D ̃ i ], K , R i | t i )- P i - 1 ) This means that the distance between point pairs in the overlap region is used to estimate the optimal depth scaling factor , Using image and corresponding depth map [ I i , D i ] Upgrading to three-dimensional space, in the New point cloud generated by step It can be calculated as: P ̃ i = ϕ 2→3 ([ I i , D i | M i =0], K , R i | t i ) in [ I i , D i | M i =0] Represents the repaired area in the RGBD image, that is, the unfilled area.

[0013] Optionally, in the above-mentioned domain-free 3D scene method based on the stable diffusion model, in step S4, the newly generated point is moved in the 3D space to smoothly connect the point cloud; the area where the mask value changes is extracted, the displacement vector from the new point cloud to the initial point cloud is calculated, and the overall shape of the initial point cloud is maintained by an interpolation algorithm, and the first point cloud is moved in the 3D space. New point cloud generated by step points to smoothly connect the two point clouds and , repeat the process To construct the final point cloud .

[0014] Optionally, in the above-mentioned domain-free Vincent 3D scene method based on the stable diffusion model, in step S4, moving in the 3D space points to smoothly connect the two point clouds and ; Extract the area where the mask value changes, that is , to find and The point corresponding to the area in arrive The displacement vector.

[0015] Optionally, in the above-mentioned domain-free Vincent 3D scene method based on the stable diffusion model, in step S4, it is mandatory Each point in moves along a ray from the camera center to the corresponding pixel; find the distance along the ray The nearest point to the corresponding point in the 3D space is found, and the depth change caused by the movement is recorded; through this constraint, the RGB image is preserved when the point is moved in 3D space. ; then, for points that have no real corresponding points ( ), how the depth value of each pixel should change is calculated by linear interpolation; the mismatch between pixels caused by violent movement is alleviated by smooth interpolation; the aligned point cloud is combined with the initial point cloud to be expressed as: The calculation of movement and interpolation is expressed as , repeat the process To construct the final point cloud .

[0016] Optionally, in the above-mentioned domain-free Vincent 3D scene method based on the stable diffusion model, in step S5, in the final point cloud After creation, the point cloud and projection image are used to train the 3D Gaussian splash model. The center point of the 3D Gaussian is initialized according to the input point cloud, and the volume and position of each point are changed through the supervision of the input real projection image. In order to train the image of the 3D Gaussian splash model, an additional images and the point cloud images, and finally get the rendered 3D scene.

[0017] According to the technical solution of the present invention, the beneficial effects produced are: The present invention implements a Vincent 3D scene method without domain restriction based on a stable diffusion model, and utilizes a diffusion model, 3D Gaussian splashing, and depth map estimation to realize a Vincent 3D scene algorithm without domain restriction. The present invention implements a Vincent 3D scene method without domain restriction based on a stable diffusion model, and in view of the problems of low quality and domain restriction of scenes generated by the existing Vincent 3D scene method, a 3D scene generation model based on 3D Gaussian splashing is constructed, which effectively improves the diversity and quality of Vincent 3D scenes and improves the generation efficiency.

[0018] The method of the present invention achieves image consistency under different views by using the initial point cloud as a geometric guide for subsequent image generation. The present invention uses a divide-and-conquer strategy to synchronously perform point cloud repair in multiple directions and smoothly connect point clouds in three-dimensional space, making the operation faster and smoother.

[0019] The present invention provides a method for solving the problem that three-dimensional scenes can only be generated within a limited field in a Vincent three-dimensional scene task, and the diversity and quality are not high. The method can generate three-dimensional scenes with high diversity and high quality, and can also ensure the consistency between the generated multi-view images. After obtaining the initial point cloud, the camera trajectory is adjusted to repair the missing pixels, and the image is projected at the camera position, and the initial point cloud is used as a geometric guide for generating each image; the image is repaired using a stable diffusion model, and the corresponding depth map is calculated using a monocular depth estimation network.

[0020] The present invention proposes an effective point cloud alignment and rendering method, which solves the problem of insufficient details of the three-dimensional scene generated by the prior art. Finally, the point cloud and the projected image are used to train the three-dimensional Gaussian splash model, which solves the problems of slow generation speed and insufficient details.

[0021] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0023] Figure 1 It is a flow chart of the method for producing three-dimensional scenes without domain restrictions based on a stable diffusion model of the present invention; Figure 2 It is a comparison of input and output of the Vincent 3D scene method of the present invention and the existing model. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0025] The invention provides a domain-free Vincent 3D scene method based on a stable diffusion model, which has improved diversity and quality, and uses a picture generated by a potential diffusion model as input to generate an initial point cloud, and sets the point cloud as a geometric guide for image generation, and uses a stable diffusion model to repair and monocular depth estimation to expand the point cloud area to create a large scene. Then, the point cloud and the reprojected image are used to optimize the Gaussian splash, and the scene is represented by a three-dimensional Gaussian splash.

[0026] The principles of the method of the present invention are: 1) generating diverse high-quality 3D scenes through a stable diffusion model and three-dimensional Gaussian splashing; 2) generating text as an image as input through a potential diffusion model to generate an initial point cloud as a geometric guide for subsequent image generation; 3) dividing the initial point cloud into 8 areas of 2*2*2, and using a stable diffusion model to perform simultaneous repair and monocular depth estimation in 8 directions to expand the point cloud; 4) using the expanded point cloud for three-dimensional Gaussian splashing.

[0027] The method for generating a three-dimensional scene without domain restrictions based on a stable diffusion model of the present invention includes: point cloud construction: the diffusion model will be used to generate an image related to a given text, generate a point cloud and estimate a depth map using a monocular depth estimation model. Alternating generation and alignment: projecting the image at the camera position and using a mask to distinguish between filled and unfilled areas; repairing the image using a stable diffusion model and calculating the corresponding depth map using a monocular depth estimation network; estimating the optimal depth scaling factor to minimize the distance between the three-dimensional points of the new image and the corresponding points in the initial point cloud; moving the newly generated points in three-dimensional space to smoothly connect the point clouds; extracting the area where the mask value changes, calculating the displacement vector from the new point cloud to the initial point cloud, and maintaining the overall shape of the initial point cloud through an interpolation algorithm. Three-dimensional Gaussian splash rendering: training a three-dimensional Gaussian splash model using the final point cloud and the projected image.

[0028] like Figure 1 As shown, the method for generating a three-dimensional scene without domain restrictions based on a stable diffusion model of the present invention has the following specific steps from the beginning to the end: S1. Point cloud initialization and depth estimation. The input text is passed through the potential diffusion model to generate an image related to the given text, and the monocular depth estimation model is used to estimate the depth map. The image and depth map are represented as and , and define the initial point cloud generated by the first image as .

[0029] Specifically, in step S1, first, the user's text input is obtained, and the image related to the given text is generated using the potential diffusion model; then the depth map is estimated using the monocular depth estimation model. The generated RGB image and depth map are represented as and ,in and are the height and width of the image respectively. According to the latent diffusion model generated and , according to the image size to determine the camera intrinsic parameter matrix and the external parameter matrix For the input RGBD image [ I 0 , D 0 ] , initialize it as a point cloud in space. The initial point cloud generated using the first image is defined as , whose expression is:

[0030] P 0 = ϕ 2→3 ([ I 0 , D 0 ], K , R 0 | t 0 ) in, It is a function that upgrades the RGBD image into a point cloud.

[0031] S2. Set the camera trajectory. Divide the initial point cloud into 8 regions of 2*2*2, and set the length to The camera trajectory is used to generate a new point cloud in step S3.

[0032] Specifically, in step S2, the length is set to The camera trajectory is set Indicates that the camera is The position and posture at the index is the external parameter matrix.

[0033] In the The operation of the step is as follows: first, the camera is moved from the previous position Move and rotate to ; Convert the coordinates from the world coordinate system to the current camera coordinate system, and then use the intrinsic matrix and Projected onto the camera plane.

[0034] S3. Generate a new point cloud. Project the image at the camera position and use a mask to distinguish between filled and unfilled areas; inpaint the image using a stable diffusion model and calculate the corresponding depth map using the same monocular depth estimation network as step S1; estimate the optimal depth scaling factor to minimize the distance between the 3D points of the new image and the corresponding points in the initial point cloud. The projected image at . Define the mask To distinguish between areas that have been filled by existing points and areas that have not been filled. Use a stable diffusion repair model Incomplete image and mask Generate real images Using a monocular depth estimation network Estimate the corresponding depth map . Estimating the optimal depth scaling factor , so that the distance between the 3D points of the new image and the corresponding points in the initial point cloud is minimized. Then, the coefficients Multiply by the estimated depth map To calculate the actual depth map .

[0035] In step S3, the camera The projected image at Due to the change in the position and pose of the camera, some areas cannot be filled from the existing point cloud. Define the mask To distinguish between areas that have been filled with existing points and areas that have not been filled. Specifically, if the corresponding pixel has been filled, then The value of is 1, otherwise it is 0. Use stable diffusion to repair the model Incomplete image and mask Generate real images Using a monocular depth estimation network Estimate the corresponding depth map Here, the monocular depth estimation model can only estimate the relative depth, and the depth coefficient from the relative depth to the actual depth may be different between different images. If the depth coefficient is different, the 3D point cloud in the newly generated image will not be connected and there will be a certain gap. Estimating the optimal depth scaling coefficient , so that the distance between the 3D point cloud of the new image and the corresponding points in the initial point cloud is minimized. Then, the coefficients Multiply by the estimated depth map To calculate the actual depth map .

[0036] d i = arg min ( ∑ M i =1 ϕ 2→3 ([ I i , d D ̃ i ], K , R i | t i )- P i - 1 1 ) here, This means that the distance between point pairs in the overlap region is used to estimate the optimal depth scaling factor .

[0037] Using image and corresponding depth map [ I i , D i ] Upgrade to three-dimensional space. New point cloud generated by step It can be calculated as:

[0038] P ̃ i = ϕ 2→3 ([ I i , D i | M i =0], K , R i | t i ) in [ I i , D i | M i =0] Represents the repaired area in the RGBD image, that is, the unfilled area.

[0039] Step S3 is one of the core invention points of the present invention.

[0040] S4. Align point clouds and repair images. Move the new point cloud in 3D space to smoothly connect the point clouds; extract the area where the mask value changes, calculate the displacement vector from the new point cloud to the initial point cloud, and maintain the overall shape of the initial point cloud through the interpolation algorithm. Move in 3D space in the first New point cloud generated by step points to smoothly connect the two point clouds and Repeat the process To construct the final point cloud .

[0041] In step S4, move in 3D space New point cloud generated by step points to smoothly connect the two point clouds and Specifically, extract the region where the mask value changes (i.e. ), to find and Then, calculate the point corresponding to the area from arrive Displacement vectors of the initial point cloud. Simply moving these points may distort the shape of the final point cloud and cause misalignment issues between the point cloud and the inpainted image. This problem is mitigated by imposing constraints on point movement and using an interpolation algorithm to maintain the overall shape of the initial point cloud.

[0042] First, force Each point in moves along a ray from the camera center to the corresponding pixel. Find the distance along the ray The nearest point to the corresponding point in the 3D space is found and the depth change caused by the movement is recorded. With this constraint, the RGB image is preserved when moving points in 3D space. Then, for points that have no real corresponding points ( ), and how the depth value of each pixel should change is calculated by linear interpolation. Through smooth interpolation, the mismatch between pixels caused by violent movement is alleviated. The aligned point cloud is combined with the initial point cloud to be expressed as:

[0043] The calculation of movement and interpolation is expressed as Repeat the process To construct the final point cloud Step S4 is also one of the core inventions of the present invention.

[0044] S5. Rendering using Gaussian splatting. After the final point cloud is created, use the point cloud and the projected images to train a 3D Gaussian splatter model. Use the final point cloud as the initial SfM points and use additional images for training.

[0045] In step S5, in the final point cloud After creation, the point cloud and projection image are used to train the 3D Gaussian splatter model. The center point of the 3D Gaussian is initialized based on the input point cloud, and the volume and position of each point are changed through the supervision of the input real projection image. Using the final constructed point cloud As the initial SfM point, this can improve the convergence speed and let the model focus on generating details. To train the images of the 3D Gaussian splatter model, an additional images and the point cloud images, because the initial The images are not enough to train the network to generate reasonable output. Finally, we get a rendered 3D scene.

[0046] The invention of the present invention is that: by using a stable diffusion model, depth estimation and explicit three-dimensional representation, better Vincent three-dimensional scene generation is achieved. Point cloud is used as a geometric guide for the diffusion model to repair the image, and the point clouds generated successively are smoothly connected so that the generated three-dimensional scene is consistent under multiple views. Three-dimensional Gaussian splashing is used as the rendering method of the three-dimensional scene to improve the rendering speed.

[0047] The method of the present invention uses the initial point cloud as a geometric guide for the diffusion model to repair the image, and smoothly connects the point clouds generated successively, so that the generated three-dimensional scene remains consistent under multiple views. Using three-dimensional Gaussian splashing as the rendering method of the three-dimensional scene improves the rendering speed, thereby solving the problems of discontinuity, low rendering quality and slow rendering speed of the three-dimensional scene in multiple views in the existing method. The method of the present invention projects a part of the point cloud into the required view and uses the projection as a guide for generating the model to repair the image. The repaired image is lifted into the three-dimensional space through the estimated depth map to form new points. Secondly, in order to aggregate the newly generated points into the three-dimensional scene, the present invention proposes an alignment algorithm that can integrate the newly generated three-dimensional scene part. The three-dimensional scene finally obtained is used as the initial point for optimizing the Gaussian splashing, thereby effectively solving the problem that the old method can only focus on the generation of three-dimensional scenes in a limited field.

[0048] The comparison results of the scenes generated by the present invention and the existing Wensheng 3D scene method are shown in Table 1 and Figure 2 As shown, Figure 2 The left side shows Rgbd2 and the input of the present invention, and the right side shows the output of Rgbd2 and the present invention.

[0049] Through Table 1 and Figure 2 It can be seen that the three-dimensional scene generated by the present invention is better in quality, color richness and sharpness than that of RGBD2. Since the present invention uses the initial point cloud as a geometric guide for the diffusion model to repair the image, multiple steps of new point cloud generation are performed when generating the three-dimensional scene. These newly generated point clouds will be aligned with the initial point cloud to complete the task of repairing the image. Therefore, compared with RGBD2's simple use of the diffusion model to expand the initial image and then render it, the present invention can make the generated three-dimensional scene smoother and more complete, and will not have the problem of small-scale scene missing in RGBD2. The present invention uses three-dimensional Gaussian splashing for rendering. Compared with RGBD2's rendering based on neural radiation fields, the present invention has faster rendering speed and higher rendering quality. Specifically, the CLIP-Score value achieves better results. CLIP-Score is an indicator used to evaluate the matching degree between text and image. The higher the score, the better the matching degree between text and image. Judging from the CLIP-IQA value, the overall visual quality of the image, including clarity and noise level, is better in the present invention than in RGBD2; the color of the image is more vivid and balanced than in RGBD2; the edge clarity and detail clarity of the image of the present invention are also better than in RGBD2.

[0050] Table 1 Comparison results of parameters involved in scenes generated by the Wensheng 3D scene method of the present invention and the existing model

[0051] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A domain-free three-dimensional scene method based on a stable diffusion model, characterized in that: The following steps are involved: S1. Point cloud initialization and depth estimation; S2. Set the camera trajectory; S3. Generate a new point cloud; S4. Align point cloud and repair image; S5. Use Gaussian splash rendering to obtain a rendered three-dimensional scene.

2. The method for generating a three-dimensional scene without domain restrictions based on a stable diffusion model according to claim 1, characterized in that: In step S1, first, the user's text input is obtained, and the image related to the given text is generated using the potential diffusion model; then the depth map is estimated using the monocular depth estimation model, and the generated RGB image and depth map are represented as and ,in and are the height and width of the image, respectively, generated from the latent diffusion model and , according to the image size to determine the camera intrinsic parameter matrix and the external parameter matrix ; For the input RGBD image , initialize it as a point cloud in space; the initial point cloud generated using the first image is defined as , whose expression is: in, It is a function that upgrades the RGBD image into a point cloud.

3. The method for generating a three-dimensional scene without domain restrictions based on a stable diffusion model according to claim 1, characterized in that: In step S2, the initial point cloud is divided into 8 regions of 2*2*2, and the length is set to The camera trajectory is set Indicates that the camera is The position and posture at the index The operation of the step is as follows: first, the camera is moved from the previous position Move and rotate to , transform the coordinates from the world coordinate system to the current camera coordinate system, and use the intrinsic matrix and the external parameter matrix Projected onto the camera plane.

4. The domain-free three-dimensional scene method based on a stable diffusion model according to claim 1 is characterized in that: In step S3, the image is projected at the camera position and a mask is used to distinguish between filled and unfilled areas; the image is inpainted using a stable diffusion model and the corresponding depth map is calculated using the same monocular depth estimation network as in step S1; and the optimal depth scaling factor is estimated to minimize the distance between the 3D points of the new image and the corresponding points in the initial point cloud.

5. The method for generating a three-dimensional scene without domain restriction based on a stable diffusion model according to claim 4 is characterized in that: In step S3, the camera The projected image at , define the mask To distinguish the area that has been filled by the existing points from the unfilled area, if the corresponding pixel has been filled, then The value of is 1, otherwise it is 0; Repair model using stable diffusion Incomplete image and mask Generate real images ;Use a monocular depth estimation network Estimate the corresponding depth map ; Estimate the optimal depth scaling factor , so that the distance between the 3D points of the new image and the corresponding points in the initial point cloud is minimized; then the coefficients are Multiply by the estimated depth map To calculate the actual depth map , This means that the distance between point pairs in the overlap region is used to estimate the optimal depth scaling factor , Using image and corresponding depth map Upgrading to three-dimensional space, in the New point cloud generated by step It can be calculated as: in Represents the repaired area in the RGBD image, that is, the unfilled area.

6. The method for generating a three-dimensional scene without domain restrictions based on a stable diffusion model according to claim 5, characterized in that: In step S4, the new point cloud is moved in the three-dimensional space to smoothly connect the point clouds; the area where the mask value changes is extracted, the displacement vector from the new point cloud to the initial point cloud is calculated, and the overall shape of the initial point cloud is maintained by the interpolation algorithm, and the point cloud is moved in the 3D space. New point cloud generated by step points to smoothly connect the two point clouds and , repeat the process To construct the final point cloud .

7. The domain-free Vincent 3D scene method based on a stable diffusion model according to claim 6, characterized in that: In step S4, move in the three-dimensional space New point cloud generated by step points to smoothly connect the two point clouds and ; Extract the area where the mask value changes, that is , to find and The point corresponding to the area in arrive The displacement vector.

8. The method for generating a three-dimensional scene without domain restrictions based on a stable diffusion model according to claim 7, characterized in that: In step S4, the mandatory Each point in moves along a ray from the camera center to the corresponding pixel; find the distance along the ray The nearest point to the corresponding point in the 3D space is found, and the depth change caused by the movement is recorded; through this constraint, the RGB image is preserved when the point is moved in 3D space. ; then, for points that have no real corresponding points ( ), how the depth value of each pixel should change is calculated by linear interpolation; the mismatch between pixels caused by violent movement is alleviated by smooth interpolation; the aligned point cloud is combined with the initial point cloud to be expressed as: The calculation of movement and interpolation is expressed as , repeat the process To construct the final point cloud .

9. The method for generating a three-dimensional scene without domain restrictions based on a stable diffusion model according to claim 1, characterized in that: In step S5, in the final point cloud After creation, the three-dimensional Gaussian splash model is trained using the point cloud and the projection image. The center point of the three-dimensional Gaussian is initialized according to the input point cloud, and the volume and position of each point are changed by the supervision of the input real projection image. In order to train the image of the three-dimensional Gaussian splash model, an additional images and the point cloud images, and finally get the rendered 3D scene.

Citation Information

Cited By

  • Layered guidance method for 2D image generation of target fine control based on trajectory

    CN119648568A

  • Absolute scale camera track image-to-video generation method suitable for real scene

    CN119919496A

  • Slope crack evolution analysis method fusing unmanned aerial vehicle image and laser point cloud

    CN120235856A

  • Efficient three-dimensional modeling method for repaired building colored drawing

    CN120431266A

  • High-efficiency three-dimensional modeling method after repair of architectural fresco

    CN120431266B