Single-view three-dimensional modeling method and system based on diffusion model

By combining multi-view diffusion model and sparse view large-scale reconstruction model in three-dimensional modeling, using ViT encoder and FlexiCubes module, high-quality 3D mesh is quickly generated from a single image, solving the complex image acquisition and calculation in the existing technology. The problems of high resources and insufficient reconstruction results have achieved efficient and diversified three-dimensional modeling effects.

CN119068144APending Publication Date: 2024-12-03YIKA (BEIJING) TECH CO LTD
View PDF 0 Cites 7 Cited by

Patent Information

Application Number
CN202411122527.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing three-dimensional modeling methods are complex and time-consuming when acquiring multi-view images and processing sparse views, which are difficult to meet the needs of real-time applications, and the diffusion model and LRM-based methods are insufficient in computing resources and reconstruction results diversity.

Method used

Using a single-view 3D modeling method based on diffusion model, the coordinated work of multi-view diffusion model and sparse view large-scale reconstruction model can be used to quickly generate high-quality 3D mesh from a single image, and the ViT encoder and FlexiCubes module can be used to improve the image to The conversion efficiency and quality of 3D representation.

Benefits of technology

It realizes the generation of diversified 3D assets within 10 seconds, significantly improving the efficiency and quality of 3D modeling, reducing computing resource requirements, enhancing geometric details and accuracy, and simplifying the operation process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068144A_ABST
    Figure CN119068144A_ABST
Patent Text Reader

Abstract

The invention discloses a single-view three-dimensional modeling method and system based on a diffusion model. The method comprises the steps of firstly obtaining a single image as input; the received images are input into a multi-view diffusion model, then the generated multi-view images are input into a ViT encoder, and each view image is converted into an image token through the ViT encoder; inputting the image token into a three-plane decoder to generate a three-plane representation; finally, contour surface extraction is carried out on the three-plane representation by using FlexiCubes, so that the three-plane representation is converted into grid representation; and 3D grid rendering: rendering the generated grid representation to generate a 3D model view. Through cooperative work of the multi-view diffusion model and the sparse view large-scale reconstruction model, rapid three-dimensional modeling from a single image is achieved, and the method has the advantages of being high in generation speed, high in model quality, rich in geometric details and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a single-view method based on a diffusion model. Figure 3 Dimensional modeling method and system. Background Art

[0002] With the continuous development of computer vision and graphics technology, 3D modeling is increasingly used in various fields, such as virtual reality, game development, film and television production, and architectural design. Traditional 3D modeling methods usually rely on professional modeling software and manual operation, which is not only time-consuming and labor-intensive, but also requires high professional skills of operators. In order to improve the efficiency and quality of 3D modeling, researchers and engineers are constantly exploring new automated modeling methods.

[0003] At present, image-based 3D reconstruction technology has made significant progress. Common methods include Multi-View Stereo (MVS) and Structured Light Scanning. These methods usually require multiple images from different perspectives to reconstruct the 3D model. Although they can generate high-quality 3D meshes, in practical applications, the process of obtaining multi-view images is complicated and time-consuming. In addition, existing 3D reconstruction methods often have difficulty in ensuring the accuracy and integrity of the reconstruction results when processing sparse views.

[0004] In recent years, the application of deep learning technology in the field of computer vision has made breakthrough progress, especially the emergence of diffusion models and large reconstruction models (LRM), which provide new ideas for 3D modeling. Diffusion models generate high-quality images through a gradual denoising process and have performed well in tasks such as image generation and image restoration. The sparse view reconstruction model based on the LRM architecture can reconstruct high-quality 3D models under sparse view conditions by using the powerful representation ability of deep learning networks.

[0005] However, the existing 3D modeling methods based on diffusion model and LRM still have some shortcomings. For example, the diffusion model usually requires a lot of computing resources and time when generating 3D models, which is difficult to meet the needs of real-time applications; and the sparse view reconstruction model based on LRM still needs to improve the diversity and detail of the reconstruction results when processing a single image. Summary of the invention

[0006] Based on this, the embodiment of the present application provides a single-view method based on a diffusion model. Figure 3The 3D modeling method and system combines the advantages of the existing multi-view diffusion model and the sparse view reconstruction model based on the LRM architecture to provide a feed-forward framework that can quickly generate high-quality 3D meshes from a single image. It can create diverse 3D assets within 10 seconds, significantly improving the efficiency and quality of 3D modeling.

[0007] First, a single-view method based on a diffusion model is provided. Figure 3 dimensional modeling method, the method comprising:

[0008] Input image: Get a single image as input;

[0009] Multi-view generation: The received image is input into the multi-view diffusion model to generate 6 views at different angles;

[0010] Image token generation: The generated multi-view images are input into the ViT encoder, and each view image is converted into an image token by the ViT encoder;

[0011] Three-plane decoding: Input the image token into the three-plane decoder to generate a three-plane representation;

[0012] Isosurface extraction: Use FlexiCubes to extract isosurfaces from the three-plane representation and convert the three-plane representation into a mesh representation;

[0013] 3D mesh rendering: Render the generated mesh representation to generate a 3D model view.

[0014] Optionally, taking a single image as input includes, further comprising:

[0015] Perform preprocessing operations on the image; wherein the preprocessing operations include scaling, cropping and denoising.

[0016] Optionally, the received image is input into a multi-view diffusion model to generate 6 views at different angles, specifically including:

[0017] The multi-view diffusion model uses convolutional neural networks in deep learning to capture the spatial structure information of the image and generate multiple views at different angles; where the views correspond to fixed camera positions.

[0018] Optionally, the generated multi-view image is input into a ViT encoder, and each view image is converted into an image token by the ViT encoder, specifically including:

[0019] Each view in the generated multi-view image is divided into image blocks of fixed size;

[0020] Each image block is flattened into a one-dimensional vector, mapped to the same dimensional space through a linear transformation, and positionally encoded to obtain an image token;

[0021] For each view image generated by the multi-view diffusion model, the image segmentation and position encoding process is repeated to obtain a set of image tokens for each view;

[0022] The image tokens of all views are input into the ViT encoder, which consists of multiple stacked transformer layers, each of which contains a self-attention mechanism and a feed-forward neural network.

[0023] After being processed by the ViT encoder, each image token is updated to contain more contextual information and semantic features; finally, the ViT encoder outputs the updated image token.

[0024] Optionally, the image token is input into a three-plane decoder to generate a three-plane representation, specifically comprising:

[0025] After receiving the image tokens output by the ViT encoder, the triplane decoder converts these tokens into a triplane representation; where a triplane is three orthogonal planes.

[0026] Optionally, the three-plane representation is converted into a mesh representation by performing isosurface extraction on the three-plane representation using FlexiCubes, including:

[0027] The FlexiCubes module first receives the three-plane representation output by the three-plane decoder as input and initializes an empty 128*128*128 three-dimensional grid;

[0028] During the isosurface extraction process, the FlexiCubes module traverses each sample point in the three-plane representation, calculates the position of these points in three-dimensional space, and checks whether these points meet the threshold conditions; the threshold conditions are used to define the boundaries of the isosurface;

[0029] When a sample point meets the isosurface condition, its position is mapped to the initialized three-dimensional grid and the grid point is marked as belonging to the isosurface;

[0030] Through the isosurface extraction process, multiple isosurface points are formed on the three-dimensional grid, and these points are connected by mesh refinement technology to generate a continuous mesh surface; wherein the mesh refinement technology includes interpolation and smoothing.

[0031] Secondly, a single-view diffusion model is provided. Figure 3 dimensional modeling system, the system comprising:

[0032] Input image module: used to obtain a single image as input;

[0033] Multi-view generation module: used to input the received image into the multi-view diffusion model to generate 6 views at different angles;

[0034] Image token generation module: used to input the generated multi-view image into the ViT encoder, and convert each view image into an image token through the ViT encoder;

[0035] Three-plane decoding module: used to input the image token into the three-plane decoder to generate a three-plane representation;

[0036] Isosurface extraction module: used to extract isosurfaces from three-plane representations using FlexiCubes to convert the three-plane representations into mesh representations;

[0037] 3D mesh rendering module: used to render the generated mesh representation and generate a 3D model view.

[0038] In a third aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the single-view video according to any one of the first aspects is implemented. Figure 3 Dimensional modeling approach.

[0039] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the single-view video device described in any one of the first aspects is implemented. Figure 3 Dimensional modeling approach.

[0040] In a fifth aspect, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the single-view method described in any one of the first aspects. Figure 3 Dimensional modeling approach.

[0041] The present invention uses an innovative multi-view diffusion model and a sparse view large-scale reconstruction model to quickly generate a high-quality 3D mesh model from a single input image, which has the following significant advantages and effects:

[0042] (1) Fast generation speed: The overall framework of the present invention can complete the generation process from image to 3D model in about 10 seconds. Compared with traditional 3D reconstruction methods, the present invention significantly improves the generation speed and meets the real-time requirements.

[0043] (2) High model quality: By generating multi-view images with high consistency through a multi-view diffusion model and combining the sparse view large-scale reconstruction model and the geometric supervision of the FlexiCubes module, the 3D mesh model generated by the present invention has high precision and high quality, and is rich in geometric details.

[0044] (3) Rich geometric details: The FlexiCubes module integrates geometric supervision such as depth and normals during the isosurface extraction process to ensure that the generated 3D mesh model has rich geometric details and accuracy.

[0045] (4) System structure optimization: Through modular design, the present invention organically combines the steps of multi-view generation, image token generation, three-plane decoding and isosurface extraction to form an efficient image-to-3D model generation process, simplifying the operation and control process.

[0046] (5) Resource saving: The multi-view diffusion model is used to generate multi-view images, which avoids the resource consumption of multiple shots or acquisitions of multi-view images in traditional methods, saving computing resources and storage space.

[0047] (6) Easy operation: The framework of the present invention is simple in design. Users only need to input a single image to generate a high-quality 3D mesh model. The operation is simple and the usage threshold is lowered. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0049] Figure 1 A single-view image based on a diffusion model provided in an embodiment of the present application Figure 3 Flowchart of steps in the dimensional modeling method;

[0050] Figure 2 The single-view diffusion model based on the embodiment of the present application is provided Figure 3 Block diagram of the dimensional modeling system;

[0051] Figure 3 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] In the description of the present invention, the terms "comprises", "has" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units added based on further optimization schemes conceived by the present invention.

[0054] Although the existing image-based 3D reconstruction technology has made significant progress, it still has some shortcomings in practical applications, which are specifically manifested in the following aspects:

[0055] 1. Limitations of Multi-view Stereo (MVS) Methods:

[0056] Complex image acquisition process: MVS methods usually require taking a large number of images from multiple different perspectives to ensure the accuracy and completeness of the reconstruction results. However, in practical applications, the process of acquiring multi-view images is complex and time-consuming, especially in dynamic scenes or inaccessible environments, where it becomes more difficult to obtain images from sufficient perspectives.

[0057] High consumption of computing resources: The MVS method needs to process a large amount of image data for feature matching and 3D point cloud generation, which places high demands on computing resources and is difficult to meet the needs of real-time applications.

[0058] 2. Limitations of structured light scanning method:

[0059] High equipment cost: The structured light scanning method relies on dedicated hardware devices, such as projectors and high-resolution cameras, which are expensive and limit its popularity among ordinary users.

[0060] Greatly affected by ambient light: The structured light scanning method has high requirements for ambient light. In strong or weak light environments, the accuracy and quality of the scanning results will be significantly affected.

[0061] 3. Limitations of the 3D modeling method based on the diffusion model:

[0062] High computing resource requirements: The diffusion model generates high-quality images through a step-by-step denoising process, which usually requires a lot of computing resources and time, making it difficult to meet the needs of real-time applications.

[0063] Slow generation speed: Due to the gradual generation characteristics of the diffusion model, it usually takes a long time to generate a high-quality 3D model, which cannot quickly respond to user needs.

[0064] 4. Limitations of the sparse view reconstruction model based on the LRM architecture:

[0065] Insufficient diversity and details of reconstruction results: When processing a single image, although the LRM-based sparse view reconstruction model can utilize the powerful representation ability of the deep learning network to generate a 3D model, the diversity and details of the reconstruction results still need to be improved, and it is difficult to meet the needs of high-quality 3D modeling.

[0066] Dependence on a single view: When only a single image is available, it is difficult for the reconstructed model to fully utilize the information in the image, resulting in insufficient details and accuracy of the generated 3D model.

[0067] In summary, the existing 3D modeling methods have many deficiencies in terms of process and performance, mainly reflected in the complex image acquisition process, high computing resource requirements, slow generation speed, and lack of diversity and details of reconstruction results. These defects limit the widespread application of 3D modeling technology in real-time applications, and a new method that can quickly generate high-quality 3D models is urgently needed to solve the above problems.

[0068] The purpose of this invention is to propose a single-view Figure 3 The three-dimensional modeling method aims to overcome the various defects of the existing technology in the three-dimensional modeling process, including:

[0069] 1. Simplify the image acquisition process: By generating high-quality 3D models with only a single image, the complex and time-consuming multi-view image acquisition process in the multi-view stereo (MVS) method is avoided, thereby improving the convenience and practicality of 3D modeling.

[0070] 2. Reduce computing resource requirements: Combining the advantages of the existing multi-view diffusion model and the sparse view reconstruction model based on the LRM architecture, a feed-forward framework is designed, which significantly reduces the computing resource requirements and makes the 3D modeling process more efficient. It can generate diverse 3D assets within 10 seconds to meet the needs of real-time applications.

[0071] 3. Improve the generation speed: By optimizing the generation process of the diffusion model, reducing the calculation steps of the step-by-step denoising, and speeding up the generation speed of the three-dimensional model, the problem of slow generation speed of the existing diffusion model is solved.

[0072] 4. Enhance the diversity and detail of reconstruction results: Utilize the powerful representation ability of deep learning networks to generate 3D models with high details and diversity under the condition of a single image, overcoming the defects of the sparse view reconstruction model based on the LRM architecture in terms of diversity and insufficient details of reconstruction results when processing a single image.

[0073] Through the above improvements, the present invention provides a feed-forward framework that can quickly generate high-quality 3D meshes from a single image, significantly improving the efficiency and quality of 3D modeling, solving the shortcomings of the prior art in terms of image acquisition, computing resources, generation speed and reconstruction results, and meeting the needs of virtual reality, game development, film and television production, and architectural design for efficient 3D modeling.

[0074] This paper proposes a new single-view based diffusion model Figure 3 3D modeling method for generating high-quality 3D mesh models from a single input image. The framework mainly consists of two core modules: multi-view diffusion model and sparse view large-scale reconstruction model. Through the collaborative work of these two modules, the generation process from image to 3D model can be completed in about 10 seconds. Please refer to Figure 1 , which shows a single-view based diffusion model provided by an embodiment of the present application Figure 3 A flowchart of a dimensional modeling method, the method may include the following steps:

[0075] S1) Input image: Get a single image as input.

[0076] In this step, a single image is input and preprocessed before being input into the multi-view diffusion model. The preprocessing operations include scaling, cropping, and denoising.

[0077] S2) Multi-view generation: The received image is input into the multi-view diffusion model to generate 6 views at different angles.

[0078] The technical principle of the multi-view diffusion model in this step includes: inputting a single image and generating 6 views with fixed camera positions through the multi-view diffusion model. The model generates multi-view images using the diffusion process to ensure that the generated views are consistent and of high quality.

[0079] When the received image is input into the multi-view diffusion model, the specific process of generating 6 views at different angles can be:

[0080] Through the multi-view diffusion model architecture and algorithm, the model is able to generate multiple views at different angles based on the image after the noise is introduced. These views correspond to fixed camera positions, ensuring consistency and high quality between views.

[0081] In the process of generating multi-view images, the model may use convolutional neural networks (CNNs) in deep learning or other suitable model structures to capture the spatial structure information of the image. This can help the model generate views from different angles more accurately.

[0082] Finally, the model outputs 6 views at different angles. These views should be consistent and high quality, accurately reflecting the appearance and shape of the original image at different viewpoints. The quality of the generated multi-view images can be verified by visual inspection or quantitative evaluation metrics.

[0083] S3) Image token generation: The generated multi-view images are input into the ViT encoder, and each view image is converted into an image token by the ViT encoder.

[0084] In the specific implementation details of image token generation, the encoder based on the Vision Transformer (ViT) plays a key role. The following are the specific implementation details of the improved image token generation:

[0085] Image Blocking (Patch Embedding): First, each view in the input multi-view image is divided into fixed-size image patches (Patches). The size of these image patches is usually a hyperparameter in model design, such as 16x16 pixels.

[0086] Next, each image block is flattened into a one-dimensional vector and mapped to the same dimensional space through a linear transformation (LinearProjection) to obtain the so-called "Patch Embedding". This step is similar to the process of converting words into word embeddings in natural language processing.

[0087] Positional Encoding: Since the Transformer structure itself does not have position information, additional positional encoding is needed to identify the position of each patch in the image. Positional encoding can be learnable or fixed, such as generated by sine and cosine functions. Adding positional encoding to Patch Embedding obtains the final representation of the image block, namely Image Tokens.

[0088] Multi-view image tokenization: For each view image generated by the multi-view diffusion model, the above image segmentation and position encoding process is repeated to obtain a set of image tokens for each view.

[0089] ViT Encoder: The image tokens of all views are input into the ViT Encoder. The ViT Encoder consists of multiple stacked Transformer Layers, each of which contains a Self-Attention Mechanism and a Feed-Forward Neural Network.

[0090] The self-attention mechanism allows the model to capture global dependencies between image tokens, while the feedforward neural network provides the ability of nonlinear transformation and feature extraction. After being processed by the ViT encoder, each image token is updated to contain more contextual information and semantic features.

[0091] Output image tokens: Finally, the ViT encoder outputs updated image tokens, which will serve as the input of the subsequent three-plane decoder.

[0092] S4) Three-plane decoding: The image token is input into the three-plane decoder to generate a three-plane representation.

[0093] After receiving the image tokens output by the ViT encoder, the triplane decoder converts these tokens into triplane representations. Triplanes usually refer to three orthogonal planes (such as XY, XZ, and YZ planes), which can jointly describe geometric information in three-dimensional space.

[0094] The three-plane decoder may contain multiple convolutional layers, upsampling layers, or other types of network structures to gradually convert image tokens into high-resolution three-plane representations. The three-plane representation is compact and easy to process, which can effectively improve the efficiency and quality of the conversion from image to 3D representation.

[0095] S5) Isosurface extraction: Use FlexiCubes to perform isosurface extraction on the three-plane representation and convert the three-plane representation into a mesh representation.

[0096] In the above process, the sparse view large-scale reconstruction model is implemented. Specifically, the generated multi-view images are input into the transformer-based sparse view large-scale reconstruction model, which first converts the images into image tokens (ImageTokens) through the ViT encoder. Then, the image tokens are converted into a triplane representation (Triplane) through the triplane decoder (TriplaneDecoder). Finally, the isosurface is extracted through the FlexiCubes module to generate a 128*128*128 grid, and the final 3D mesh model is rendered.

[0097] Isosurface extraction of the tri-planar representation using the FlexiCubes module to convert the tri-planar representation into a 128x128x128 grid, specific implementation details of the isosurface extraction generation:

[0098] Three-plane representation preparation: The FlexiCubes module first receives the three-plane representation output by the three-plane decoder as input. These three-plane representations contain the 3D geometric information extracted from the input image.

[0099] Mesh initialization: FlexiCubes module first initializes an empty 128x128x128 3D mesh. This mesh will serve as the basic structure of the final output 3D model.

[0100] Isosurface extraction: During the isosurface extraction process, the FlexiCubes module traverses each point (or sample point) in the three-plane representation and calculates the position of these points in three-dimensional space. Then, the FlexiCubes module checks whether these points meet certain threshold conditions (such as density, color, depth, etc.), which define the boundaries of the isosurface. When a sample point meets the isosurface condition, the FlexiCubes module maps its position to the initialized three-dimensional grid and marks the grid point as belonging to the isosurface.

[0101] Mesh refinement: Through the isosurface extraction process, the FlexiCubes module forms a series of isosurface points on the 3D mesh. However, these points are usually discrete and insufficient to form a smooth mesh surface. Therefore, the FlexiCubes module uses interpolation, smoothing or other mesh refinement techniques to connect these points and generate a continuous mesh surface.

[0102] Geometric supervision: The FlexiCubes module supports the direct application of geometric supervision on the mesh representation. This means that during the training process, known geometric information such as depth maps and normal maps can be used to guide the generation of the mesh. Specifically, the module calculates the differences between the generated mesh and the real geometric data (such as depth difference, normal difference, etc.), and uses these differences as part of the loss function to optimize the model parameters. In this way, the FlexiCubes module is able to learn more accurate geometric structures, thereby generating 3D mesh models with higher accuracy and higher quality.

[0103] Mesh output: After isosurface extraction and mesh refinement, the FlexiCubes module will output a complete 128x128x128 3D mesh model. This model can be directly used for visualization, rendering or other 3D application scenarios.

[0104] Through the above steps, the FlexiCubes module can efficiently convert the three-plane representation into a high-precision and high-quality 3D mesh model, providing strong support for subsequent 3D modeling and rendering tasks.

[0105] S6) 3D mesh rendering: Rendering the generated mesh representation to generate a 3D model view.

[0106] The final generated 3D mesh model is rendered through a rendering module to obtain a high-quality 3D model.

[0107] Among them, technical measures and improvements include:

[0108] 1. Introduction of multi-view diffusion model:

[0109] The multi-view diffusion model is used to generate multi-view images with high consistency, which solves the problem of insufficient information when generating 3D models from a single view.

[0110] 2. Transformer-based sparse view large-scale reconstruction model:

[0111] The combination of the ViT encoder and the tri-plane decoder improves the efficiency and quality of the conversion of images to 3D representations.

[0112] 3. Application of FlexiCubes modules:

[0113] Isosurface extraction and geometry supervision are performed through the FlexiCubes module to ensure that the generated 3D mesh model has high accuracy and quality.

[0114] The data processing process includes:

[0115] 1. Input image preprocessing:

[0116] Normalize and resize the input image.

[0117] 2. Multi-view generation process:

[0118] Multi-view image generation via diffusion model.

[0119] 3. Image token generation process:

[0120] The ViT encoder converts multi-view images into image tokens.

[0121] 4. Three-plane decoding process:

[0122] The tri-plane decoder converts the image token into a tri-plane representation.

[0123] 5. Isosurface extraction process:

[0124] The FlexiCubes module performs isosurface extraction and generates 1283 grids.

[0125] 6.3D mesh rendering process:

[0126] The rendering module renders the generated 3D mesh to obtain the final 3D model.

[0127] The algorithm improvements include:

[0128] 1. Improvements to the multi-view diffusion model:

[0129] High-quality multi-view images are generated through a diffusion process, ensuring consistency between views.

[0130] 2. Improvements in large-scale reconstruction models with sparse views:

[0131] The combination of the ViT encoder and the tri-plane decoder improves the efficiency and quality of the conversion of images to 3D representations.

[0132] 3. Improvements to the FlexiCubes module:

[0133] Through isosurface extraction and geometric supervision, the generated 3D mesh model is ensured to have high accuracy and quality.

[0134] In summary, the present invention realizes a technical solution for quickly generating a high-quality 3D mesh model from a single image through the collaborative work of a multi-view diffusion model and a sparse view large-scale reconstruction model.

[0135] It can be seen that the key point of the present invention is to realize the technical solution of quickly generating a high-quality 3D mesh model from a single input image through an innovative multi-view diffusion model and a sparse view large-scale reconstruction model. The specific key points are as follows:

[0136] 1. Introduction of multi-view diffusion model:

[0137] The present invention uses a multi-view diffusion model to generate six views with fixed camera positions from a single input image. The model generates multi-view images through a diffusion process, ensuring that the generated views are consistent and high quality, thus solving the problem of insufficient information when generating 3D models from a single view.

[0138] 2. Transformer-based sparse view large-scale reconstruction model:

[0139] The present invention adopts a sparse view large-scale reconstruction model based on transformer (ViT encoder) to convert the generated multi-view images into image tokens. The image tokens are converted into three-plane representations through a three-plane decoder, which improves the conversion efficiency and quality of images to 3D representations.

[0140] 3. Application of FlexiCubes modules:

[0141] The present invention uses the FlexiCubes module to extract isosurfaces and convert the three-plane representation into a 128*128*128 grid. The module can also apply geometric supervision such as depth and normals directly on the grid representation to further enhance the results and ensure that the generated 3D grid model has high accuracy and quality.

[0142] 4. Rapidly generate high-quality 3D mesh models:

[0143] The overall framework of the present invention can complete the generation process from image to 3D model in about 10 seconds, which significantly improves the generation speed and meets the real-time requirements.

[0144] 5. Integration of Geometric Supervision:

[0145] The present invention integrates geometric supervision in the FlexiCubes module, such as direct application of depth and normals, which enhances the geometric details and accuracy of the generated 3D mesh models.

[0146] 6. Optimization of system structure:

[0147] The present invention organically combines the steps of multi-view generation, image token generation, three-plane decoding and isosurface extraction through modular design to form an efficient image-to-3D model generation process.

[0148] In summary, the present invention realizes a technical solution for quickly generating a high-quality 3D mesh model from a single image through the collaborative work of a multi-view diffusion model and a sparse view large-scale reconstruction model, which has the advantages of fast generation speed, high model quality, and rich geometric details.

[0149] Please refer to Figure 2 , which shows a single-view based diffusion model provided by an embodiment of the present application Figure 3 A block diagram of a dimensional modeling system. The system may include:

[0150] Input image module: used to obtain a single image as input;

[0151] Multi-view generation module: used to input the received image into the multi-view diffusion model to generate 6 views at different angles;

[0152] Image token generation module: used to input the generated multi-view image into the ViT encoder, and convert each view image into an image token through the ViT encoder;

[0153] Three-plane decoding module: used to input the image token into the three-plane decoder to generate a three-plane representation;

[0154] Isosurface extraction module: used to extract isosurfaces from three-plane representations using FlexiCubes to convert the three-plane representations into mesh representations;

[0155] 3D mesh rendering module: used to render the generated mesh representation and generate a 3D model view.

[0156] On Single Vision Based on Diffusion Model Figure 3 The specific definition of the dimensional modeling system can be found in the above description of the single-view diffusion model. Figure 3 The limitations of the dimensional modeling method are not described here. Figure 3 Each module in the dimensional modeling system can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of the above modules.

[0157] In one embodiment, an electronic device is provided. The electronic device may be a computer, and its internal structure diagram may be as follows: Figure 3 As shown. The electronic device includes a processor, a memory and a network interface connected via a system bus. The processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for single-view based on the diffusion model. Figure 3 The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a single-view method based on a diffusion model is implemented. Figure 3 Dimensional modeling approach.

[0158] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] In one embodiment of the present application, a computer-readable storage medium is provided on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned single-view based on diffusion model is implemented. Figure 3 Steps of the dimensional modeling method.

[0160] The computer-readable storage medium provided in this embodiment has similar implementation principles and technical effects to those of the above method embodiments, and will not be described in detail here.

[0161] In one embodiment of the present application, a computer program product is provided, including a computer program / instruction, which when executed implements the above-mentioned single-view based on diffusion model Figure 3 Steps of the dimensional modeling method.

[0162] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0163] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A single-view 3D modeling method based on a diffusion model, characterized in that: The method comprises: Input image: Get a single image as input; Multi-view generation: The received image is input into the multi-view diffusion model to generate 6 views at different angles; Image token generation: The generated multi-view images are input into the ViT encoder, and each view image is converted into an image token by the ViT encoder; Three-plane decoding: Input the image token into the three-plane decoder to generate a three-plane representation; Isosurface extraction: Use FlexiCubes to extract isosurfaces from the three-plane representation and convert the three-plane representation into a mesh representation; 3D mesh rendering: Render the generated mesh representation to generate a 3D model view.

2. The single-view 3D modeling method according to claim 1, characterized in that: Taking a single image as input includes, among other things: Perform preprocessing operations on the image; wherein the preprocessing operations include scaling, cropping and denoising.

3. The single-view 3D modeling method according to claim 1, characterized in that: The received image is input into the multi-view diffusion model to generate 6 views at different angles, including: The multi-view diffusion model uses convolutional neural networks in deep learning to capture the spatial structure information of the image and generate multiple views at different angles; where the views correspond to fixed camera positions.

4. The single-view 3D modeling method according to claim 1, characterized in that: The generated multi-view images are input into the ViT encoder, and each view image is converted into an image token by the ViT encoder, specifically including: Each view in the generated multi-view image is divided into image blocks of fixed size; Each image block is flattened into a one-dimensional vector, mapped to the same dimensional space through a linear transformation, and positionally encoded to obtain an image token; For each view image generated by the multi-view diffusion model, the image segmentation and position encoding process is repeated to obtain a set of image tokens for each view; The image tokens of all views are input into the ViT encoder, which consists of multiple stacked transformer layers, each of which contains a self-attention mechanism and a feed-forward neural network. After being processed by the ViT encoder, each image token is updated to contain more contextual information and semantic features; finally, the ViT encoder outputs the updated image token.

5. The single-view 3D modeling method according to claim 1, characterized in that: The image token is input into the three-plane decoder to generate a three-plane representation, which includes: After receiving the image tokens output by the ViT encoder, the triplane decoder converts these tokens into a triplane representation; where a triplane is three orthogonal planes.

6. The single-view 3D modeling method according to claim 1, characterized in that: Use FlexiCubes to extract isosurfaces from the three-plane representation and convert the three-plane representation into a mesh representation, including: The FlexiCubes module first receives the three-plane representation output by the three-plane decoder as input and initializes an empty 128*128*128 three-dimensional grid; During the isosurface extraction process, the FlexiCubes module traverses each sample point in the three-plane representation, calculates the position of these points in three-dimensional space, and checks whether these points meet the threshold conditions; the threshold conditions are used to define the boundaries of the isosurface; When a sample point meets the isosurface condition, its position is mapped to the initialized three-dimensional grid and the grid point is marked as belonging to the isosurface; Through the isosurface extraction process, multiple isosurface points are formed on the three-dimensional grid, and these points are connected by mesh refinement technology to generate a continuous mesh surface; wherein the mesh refinement technology includes interpolation and smoothing.

7. A single-view 3D modeling system based on a diffusion model, characterized in that: The system comprises: Input image module: used to obtain a single image as input; Multi-view production module: used to input the received image into the multi-view diffusion model to generate 6 views at different angles; Image token generation module: used to input the generated multi-view image into the ViT encoder, and convert each view image into an image token through the ViT encoder; Three-plane decoding module: used to input the image token into the three-plane decoder to generate a three-plane representation; Isosurface extraction module: used to extract isosurfaces from three-plane representations using FlexiCubes and convert the three-plane representations into mesh representations; 3D mesh rendering module: used to render the generated mesh representation and generate a 3D model view.

8. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the single-view three-dimensional modeling method as claimed in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the single-view three-dimensional modeling method as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the single-view three-dimensional modeling method as described in any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Three-dimensional object model generation method and device, storage medium and computer equipment

    CN119762681A

  • Methods, apparatus, storage media and computer equipment for generating 3D object models

    CN119762681B

  • Single-view 3D scene reconstruction method and device based on diffusion model, and medium

    CN120259571A

  • A single-view 3D scene reconstruction method, device and medium based on diffusion model

    CN120259571B

  • Method, device and equipment for generating three-dimensional eyeball model based on single fundus photo

    CN120931819A