Image adjustment 3D content creation architecture based on 2D diffusion
By using a diffusion model and neural radiation field based on the U-Net structure, multi-view images are generated using monocular images and external camera information, solving the under-constraint problem of creating 3D geometric models from monocular images and achieving efficient 3D content creation.
Patent Information
- Application Number
- CN202510943847.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-09
- Publication Date
- 2025-10-28
AI Technical Summary
In existing technologies, the creation of 3D geometric models from monocular images suffers from severe underconstraints. Directly applying the vanilla diffusion model requires billions of 3D pairing data points and has high training costs, which limits the scalability and versatility of the diffusion model.
A multi-cross attention block diffusion model based on the U-Net structure is adopted, which combines a control encoder and a neural radiation field to generate multi-view images using monocular images and external camera information. The neural radiation field is optimized by a multi-resolution mesh encoder to reduce computational resource requirements and generate a 3D geometric model.
It can generate realistic 3D geometric models without requiring a large amount of 3D training data, reducing training time from 5 hours to 30 minutes, and achieving efficient 3D content creation.
Smart Images
Figure CN120852656A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of diffusion model technology, specifically a 3D content creation architecture for image adjustment based on 2D diffusion. Background Technology
[0002] Creating a 3D geometric model from a single monocular image is a severely underconstrained problem because a single image is insufficient to determine the scale and position of an object.
[0003] Over the past few years, diffusion models have seen unprecedented improvements, achieving high-fidelity results. However, directly applying vanilla diffusion models to 3D content creation is impractical, requiring billions of 3D pairings. Vanilla diffusion models have been validated across various modalities, demonstrating impressive results for image, text, and audio generation. However, a significant challenge is that each modality typically requires extensive, specialized training data to achieve optimal performance. This reliance on large, modality-specific datasets limits the scalability and versatility of diffusion models, as collecting and annotating data for certain modalities can be both expensive and time-consuming. Therefore, there is an urgent need to develop an image-modulated 3D content creation architecture based on 2D diffusion to address these technical challenges. Summary of the Invention
[0004] The purpose of this invention is to provide an image adjustment 3D content creation architecture based on 2D diffusion to solve the problems mentioned in the background art.
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] A 2D diffusion-based image conditioning 3D content creation architecture includes: a diffusion model, which consists of multiple cross-attention blocks within a U-Net structure, supporting the effective fusion of multiple modalities such as text, image, and camera parameters;
[0007] Diffusion models trained on a large number of images achieve efficient learning of natural image distributions. However, vanilla, latent, and diffusion models lack support for native view control. Therefore, to synthesize novel images from specific camera perspectives, a control encoder is incorporated into the latent diffusion model, and camera externalities are provided as input to the conditional encoder to instruct the synthesis process. The downsampling and intermediate layers of the vanilla diffusion model are replicated as the control encoder. The output of the conditional control network is first processed by 1x1 convolutions, and then combined with the intermediate and upsampling layers of the latent diffusion model. Cross-lookup blocks are embedded in the U-Net structure to form the control encoder, which effectively fuses different modal inputs. The control encoder allows us to set camera externalities as additional inputs to facilitate novel view synthesis. The entire noise reduction process is supervised by the following:
[0008]
[0009] in It is the time step, z t Here, x is the latent feature, P is the camera extrinsic function, and c is the control encoder. θ It is a noise reduction network;
[0010] The neural radiation field represents the scene as a 5D continuous function that takes 3D position and 2D view direction as input and generates color and volume density fields. To approximate the 5D representation, a multilayer perceptron network is used and the network weights are optimized to convert the 5D input into color and volume density fields, implicitly learning the structure of the scene without 3D supervision. To render the image, a certain number of points are sampled from the camera, pixel-centric, and then the following formula is calculated:
[0011]
[0012] Where c i , σ i These represent the color and density values for each sample, s i Indicates distance;
[0013] This rendering function is differentiable between adjacent samples and is easy to optimize. However, the original neural radiation field uses a parameter-heavy multilayer perceptron network to represent the scene, which is resource-intensive and slow. To improve training speed and save computational resources, a multi-resolution grid encoder is used instead of the multilayer perceptron. The grid encoder stores the learnable latent code at the coordinates of multiple feature grids at various resolutions. This formula minimizes the need for a large number of matrix operations, thereby effectively accelerating the training process. Based on the multi-view images generated by the diffusion model, we use the following loss function to optimize the neural radiation field:
[0014] Reconstruction loss ensures that the generated 3D object accurately reflects the input 2D image;
[0015]
[0016] in, is the projection of the rendered 3D object, I is the input image, and depth smoothness loss is typically used to regularize the depth values in the neural radiation field to promote the spatial smoothness of the depth values. This constraint is used to penalize abrupt changes in depth between adjacent points to promote more coherent and realistic 3D reconstruction.
[0017]
[0018] Where, d i d is the depth value of the i-th sampling point. j These are the depth values of the adjacent points of the j sampling points;
[0019] Near-view consistency loss is used to normalize appearance changes between nearby views, ensuring that the appearance of an object or scene does not change abruptly between adjacent viewpoints. This loss is particularly important in neural radiation fields, where maintaining consistent visual features between adjacent views is crucial.
[0020]
[0021] in, It is the appearance image in viewpoint v. It is the appearance of the near-viewpoint. After field optimization, the three-dimensional geometric model is derived using the traveling cube algorithm.
[0022] As a further aspect of the invention: the reconstructed results are evaluated using the chamfer distance and the distance of the Earth movement device.
[0023]
[0024] Where, p i and q j These are points in point clouds P and q, respectively, and π. ij It refers to the transmission of traffic from P to j.
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] This invention focuses on creating 3D content using a latent diffusion model. Instead of directly generating 3D geometry through the latent diffusion model, a two-stage formula is employed. First, a latent diffusion model of view-conditioned reflexes is organized, using a monocular image and camera exterior as input to synthesize multi-view images. Then, a neural radiation field is trained using the synthesized multi-view images; since volumetric rendering is differentiable, it is easily optimized. After training the neural radiation field, a 3D geometric model is generated using a traveling cubes algorithm applied to the density field. This framework eliminates the requirement for paired 3D training data and requires minimal computational resources. Attached Figure Description
[0027] Figure 1 A schematic diagram of the architecture for creating 3D content based on 2D diffusion images.
[0028] Figure 2 Create an architecture for adjusting 3D content based on 2D diffusion images by reconstructing geometry on field images.
[0029] Figure 3 Create an architecture for adjusting 3D content based on 2D diffusion by reconstructing geometry on a synthetic image. Detailed Implementation
[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0031] Example 1
[0032] An image modulation 3D content creation architecture based on 2D diffusion includes: a diffusion model, which consists of multiple cross-attention blocks within a U-Net structure, supporting the effective fusion of multiple modalities such as text, image, and camera parameters.
[0033] Figure 1The graphical representation of our framework is illustrated. The diffusion model trained on a large number of images achieves efficient learning of natural image distributions. However, vanilla, latent, and diffusion models lack support for native view control. Therefore, to synthesize novel images in specific camera perspectives, a control encoder is incorporated into the latent diffusion model, and camera exteriors are provided as input to the conditional encoder to instruct the synthesis process. The downsampling and intermediate layers of the vanilla diffusion model are replicated as the control encoder. The output of the conditional control network is first processed by 1x1 convolutions, and then combined with the intermediate and upsampling layers of the latent diffusion model. Cross-lookup blocks are embedded in the U-Net structure to form the control encoder, which effectively fuses different modal inputs. The control encoder allows us to set camera exteriors as additional inputs to facilitate novel view synthesis. The entire noise reduction process is supervised by the following:
[0034]
[0035] in It is the time step, z t Here, x is the latent feature, P is the camera extrinsic function, and c is the control encoder. θ It is a noise reduction network;
[0036] The neural radiation field represents the scene as a 5D continuous function that takes 3D position and 2D view direction as input and generates color and volume density fields. To approximate the 5D representation, a multilayer perceptron network is used and the network weights are optimized to convert the 5D input into color and volume density fields, implicitly learning the structure of the scene without 3D supervision. To render the image, a certain number of points are sampled from the camera, pixel-centric, and then the following formula is calculated:
[0037]
[0038] Where c i , σ i These represent the color and density values for each sample, s i Indicates distance;
[0039] This rendering function is differentiable between adjacent samples and is easy to optimize. However, the original neural radiation field uses a parameter-heavy multilayer perceptron network to represent the scene, which is resource-intensive and slow. To improve training speed and save computational resources, a multi-resolution grid encoder is used instead of the multilayer perceptron. The grid encoder stores the learnable latent code at the coordinates of multiple feature grids at various resolutions. This formula minimizes the need for a large number of matrix operations, thereby effectively accelerating the training process. Based on the multi-view images generated by the diffusion model, we use the following loss function to optimize the neural radiation field:
[0040] Reconstruction loss ensures that the generated 3D object accurately reflects the input 2D image;
[0041]
[0042] in, is the projection of the rendered 3D object, I is the input image, and depth smoothness loss is typically used to regularize the depth values in the neural radiation field to promote the spatial smoothness of the depth values. This constraint is used to penalize abrupt changes in depth between adjacent points to promote more coherent and realistic 3D reconstruction.
[0043]
[0044] Where, d i d is the depth value of the i-th sampling point. j These are the depth values of the adjacent points of the j sampling points;
[0045] Near-view consistency loss is used to normalize appearance changes between nearby views, ensuring that the appearance of an object or scene does not change abruptly between adjacent viewpoints. This loss is particularly important in neural radiation fields, where maintaining consistent visual features between adjacent views is crucial.
[0046]
[0047] in, It is the appearance image in viewpoint v. It is the appearance of the near-viewpoint. After field optimization, the three-dimensional geometric model is derived using the traveling cube algorithm.
[0048] For novel view synthesis, a pre-trained diffusion model was employed. The view-conditional diffusion model provides an efficient method for 3D reconstruction utilizing rich 2D appearance priors. The experimental setup included two classes of input images: one was real-world captured photographs, and the other was synthesized images rendered from 3D textured geometry models. Real-world photographs were sourced from the internet. Textured geometry models were obtained from the Free3d website.
[0049] The entire experiment was conducted on an NVIDIA GeForce RTX 4090 with 24GB of RAM. The ADAM optimizer was used to train the neural radiation field for 10,000 iterations. The learning rate during training was set from 5×10⁻⁶. -2 Initially, it decreases exponentially until it reaches a final value of 1×10. -4 With the support of a multi-resolution mesh encoder, training time has been reduced from 5 hours to only about 30 minutes.
[0050] Figure 2The reconstruction results of the field images are shown. Figure 2 The leftmost element of each block is the input image. Figure 2 The first and second rows of each block display the synthesized multi-view image and the reconstructed geometry, respectively. For example... Figure 2 As shown, the framework effectively recovers the geometry associated with the input image and generates plausible geometric structures from different viewpoints. Despite the lack of some geometric details, fully reconstructing detailed geometry from a single monocular RGB image remains a significant challenge. Figure 3 This demonstrates the reconstructed geometry on a synthetic image. The input image is displayed... Figure 3 The far left. Figure 3 The first and second rows of each block describe the ground truth geometry and the reconstructed geometry, respectively. The qualitative results are shown in Table 1.
[0051] The results of the reconstruction were evaluated using chamfer distance and Earth movement distance:
[0052]
[0053] Where, p i and q j These are points in point clouds P and q, respectively, and π. ij It refers to the transmission of traffic from P to j.
[0054] Quantitative and qualitative results show that the reconstructed geometry deviates slightly from the true geometry. Given that only a single image is provided as input, accurate geometry estimation is not feasible. However, our method still produces a reasonable reconstruction that fits very well with the input image.
[0055]
[0056] Table 1: Qualitative results of the synthesized images
[0057] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.
[0058] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
Claims
1. An image modulation 3D content creation architecture based on 2D diffusion, comprising: The diffusion model is characterized by being composed of multiple cross-attention blocks within a U-Net structure, supporting the effective fusion of multiple modalities including text, image, and camera parameters. Diffusion models trained on a large number of images achieve efficient learning of natural image distributions. However, vanilla, latent, and diffusion models lack support for native view control. Therefore, to synthesize novel images from specific camera perspectives, a control encoder is incorporated into the latent diffusion model, and camera externalities are provided as input to the conditional encoder to instruct the synthesis process. The downsampling and intermediate layers of the vanilla diffusion model are replicated as the control encoder. The output of the conditional control network is first processed by 1x1 convolutions, and then combined with the intermediate and upsampling layers of the latent diffusion model. Cross-lookup blocks are embedded in the U-Net structure to form the control encoder, which effectively fuses different modal inputs. The control encoder allows us to set camera externalities as additional inputs to facilitate novel view synthesis. The entire noise reduction process is supervised by the following: in It is the time step, z t Here, x is the latent feature, P is the camera extrinsic function, and c is the control encoder. θ It is a noise reduction network; The neural radiation field represents the scene as a 5D continuous function that takes 3D position and 2D view direction as input and generates color and volume density fields. To approximate the 5D representation, a multilayer perceptron network is used and the network weights are optimized to convert the 5D input into color and volume density fields, implicitly learning the structure of the scene without 3D supervision. To render the image, a certain number of points are sampled from the camera, pixel-centric, and then the following formula is calculated: Where c i , σ i These represent the color and density values for each sample, s i Indicates distance; This rendering function is differentiable between adjacent samples and is easy to optimize. However, the original neural radiation field uses a parameter-heavy multilayer perceptron network to represent the scene, which is resource-intensive and slow. To improve training speed and save computational resources, a multi-resolution grid encoder is used instead of the multilayer perceptron. The grid encoder stores the learnable latent code at the coordinates of multiple feature grids at various resolutions. This formula minimizes the need for a large number of matrix operations, thereby effectively accelerating the training process. Based on the multi-view images generated by the diffusion model, we use the following loss function to optimize the neural radiation field: Reconstruction loss ensures that the generated 3D object accurately reflects the input 2D image; in, is the projection of the rendered 3D object, I is the input image, and depth smoothness loss is typically used to regularize the depth values in the neural radiation field to promote the spatial smoothness of the depth values. This constraint is used to penalize abrupt changes in depth between adjacent points to promote more coherent and realistic 3D reconstruction. Where, d i d is the depth value of the i-th sampling point. j These are the depth values of the adjacent points of the j sampling points; Near-view consistency loss is used to normalize appearance changes between nearby views, ensuring that the appearance of an object or scene does not change abruptly between adjacent viewpoints. This loss is particularly important in neural radiation fields, where maintaining consistent visual features between adjacent views is crucial. in, It is the appearance image in viewpoint v. It is the appearance of the near-viewpoint. After field optimization, the three-dimensional geometric model is derived using the traveling cube algorithm.
2. The image adjustment 3D content creation architecture based on 2D diffusion according to claim 1, characterized in that, The results of the reconstruction were evaluated using chamfer distance and Earth movement distance: Where, p i and q j These are points in point clouds P and q, respectively, and π. ij It refers to the transmission of traffic from P to j.