Monocular three-dimensional model generation method and device based on undirected distance function

Through a monocular reconstruction method based on undirected distance function, combined with two-dimensional diffusion model and neural radiation field model, the problem of difficulty in dealing with open surfaces in the existing technology is solved, high-fidelity generation of open surfaces is achieved, and the three-dimensional model is improved Generate quality and detail capture capability.

CN119006745APending Publication Date: 2024-11-22INST OF COMPUTING TECH CHINESE ACAD OF SCI
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411114733.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing three-dimensional generative models are difficult to deal with open surfaces, especially when training data is scarce and complex, and cannot generate high-fidelity open surface models, limiting their feasibility in real-world applications.

Method used

Using monocular reconstruction and three-dimensional model generation methods based on undirected distance function, a joint framework is constructed by combining the two-dimensional diffusion model and the neural radiation field model, using the data prior and unsigned distance field reconstruction of the multi-view diffusion model. Generate high-quality 3D object models with arbitrary topology from a single image.

Benefits of technology

High-fidelity generation of open surfaces is achieved, which goes beyond the limitations of traditional methods that can only generate closed surfaces, significantly improves the generation quality and detail capture capabilities of open surface models, and achieves better generation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006745A_ABST
    Figure CN119006745A_ABST
Patent Text Reader

Abstract

The invention provides a monocular three-dimensional model generation method and device based on an undirected distance function. Data priori from a multi-view diffusion model and geometric priori brought by an unsigned distance field UDF reconstructor are combined. Specifically, we utilize a united framework, the framework consisting of: a generation module that optimizes a shallow density network using data priori brought by a diffusion model, thereby providing consistent rendering from any view; and the reconstruction module introduces geometric priori to regularize the deep UDF network, and the network extracts the rendering result of the generation module into an accurate geometry. And the field coupler is used for establishing positive feedback from UDF to rendering in the diffusion process, so that the geometric perception capability of the generation module is obviously improved, and the UDF distribution is not influenced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical fields of artificial intelligence, computer graphics and three-dimensional modeling. Background Art

[0002] 3D Generative Models. Advances in Generative Adversarial Networks (GANs) have enabled significant progress in the field of 2D image synthesis in recent years. Recently, there has been growing interest in leveraging 3D-aware GAN techniques to acquire knowledge about 3D shapes from 2D images alone, without explicit 3D data. However, it is worth mentioning that GAN-based methods still face challenges in terms of training stability and ability to handle diverse objects. Diffusion models, which have been previously studied in 2D novel view synthesis, open up new avenues of exploration for their application in 3D generation research. This is achieved through their integration with Neural Radiance Fields (NeRFs), feature grids, and tri-planes.

[0003] 3D Generation Based on 2D Diffusion Models: Recent progress in 2D-lifting methods has demonstrated the potential of pre-trained 2D generative models to generate 3D data from 2D images. Based on these 2D priors, fractional distillation sampling (SDS) or variational score distillation (VSD) techniques have been widely used in various studies to build 3D models based on textual or visual inputs. Zero123 was proposed to generate images from different viewpoints based on reference images and positions as conditioning factors. The utilization of this prior can be used for optimization of neural radiance fields through the score distillation sampling technique. Magic123 combines 2D and 3D diffusion priors to improve the quality of the reconstruction process. Wonder3D and SyncDreamer utilize multi-view diffusion models to generate consistent multi-view images. However, these models mainly represent surfaces in the form of signed distance functions (SDFs) or density fields, thus limiting their performance on open structures.

[0004] Neural Reconstruction of 3D Open Surfaces: In current computer graphics research, a lot of effort has been put into building a general representation that can model not only closed surfaces but also open surfaces. Recent developments in the field of neural implicit representations based on multi-view images have set a new standard for 3D modeling and reconstruction without the need for 3D data priors, as evidenced by breakthroughs in this field. However, the above algorithms can only reconstruct watertight shapes because they rely on signed distance functions (SDFs).

[0005] In summary, current 3D generative models have the following limitations: A major limitation faced by 3D generative models is the scarcity of the amount and variety of training data. Existing data often contains simple structures, limiting their ability to generate a variety of real-world objects. In addition, unlike closed surfaces with clear internal spaces, open surfaces with difficult-to-distinguish internal spaces (such as leaves, flowers, and clothing) pose a huge challenge due to their complex details. The lack of complex training data prevents the model from capturing these nuances, ultimately limiting its application and real-world feasibility.

[0006] Limitations of 3D generation based on 2D diffusion models: Methods such as DreamFusion attempt to bridge the 3D data gap with 2D data through NeRF and SDS loss functions, but they tend to focus more on rendering quality rather than accurate geometry, especially for objects with open surfaces. Therefore, this method cannot achieve true open surface generation, which makes it limited in real-world applications.

[0007] Limitations of Neural Reconstruction of 3D Open Surfaces: Existing 3D reconstruction methods for handling open surfaces typically rely on dense and accurate inputs, such as multiple images from different viewpoints. In contrast, 3D generative models that aim to create objects from scratch typically use only limited inputs, such as a single image. This limitation poses a huge challenge in capturing the details of open surfaces, as open surfaces are inherently more difficult to reconstruct using limited information. Summary of the invention

[0008] The purpose of the present invention is to solve the problem that the prior art can only process closed surfaces, and propose a monocular reconstruction and three-dimensional model generation method and system based on an undirected distance function. Specifically, Fig.11 As shown, the monocular 3D model generation method based on undirected distance function proposed by the present invention includes:

[0009] Initial step S1, obtaining a monocular training image, wherein the monocular image contains a target object to be three-dimensionalized; inputting the monocular training image into the image macro model to obtain a target multi-view image of the target object as a data prior;

[0010] Generating step S2, constructing a generating module composed of a two-dimensional diffusion model and a neural radiation field model, inputting the monocular training image into the generating module to obtain a generated multi-view image, and constructing a first loss function according to the generated multi-view image and the target multi-view image to train and update the generating module;

[0011] Reconstruction step S3, constructing a reconstruction module composed of an unsigned distance field model and a renderer, inputting the generated multi-view image into the reconstruction module, obtaining a reconstructed multi-view image and a triangular mesh model of the target object, and constructing a second loss function according to the reconstructed multi-view image and the generated multi-view image to train and update the reconstruction module;

[0012] Iteration step S4, looping and iterating the generation step and the reconstruction step until the first loss function and the second loss function converge or reach a preset number of loop iterations, and saving the current generation module and the reconstruction module as the generation module and the reconstruction module after training respectively;

[0013] Apply step S5, input the monocular image containing the target object into the trained generation module, and input the generated multi-view image into the trained reconstruction module to obtain the triangular mesh model of the target object in the monocular image as the three-dimensional model generation result.

[0014] The monocular 3D model generation method based on undirected distance function, wherein the generation step comprises:

[0015] The spatial position is hash-encoded, and the encoding result is input into the neural radiation field model to obtain a first encoding feature representing the density of each point in the space. The neural radiation field model renders the first encoding feature into an image of a specified perspective to obtain the generated multi-perspective image.

[0016] The monocular 3D model generation method based on undirected distance function, wherein the first coding feature also includes the undirected distance value output by the unsigned distance field model, specifically, the undirected distance value is fused with the density output by the neural radiation field model through the following formula:

[0017]

[0018] Where σ(p) represents the final rendering density of the neural radiation field of the three-dimensional point p in the space, σ g (ε hash (p)) is the density of the output of the neural radiation field model, σ u is the spatial density offset;

[0019]

[0020] Where (1-|p|) is the initial density deviation, |p| represents the distance to the center of the space, and s b ,s d and u Scaling factor representing density deviation;

[0021] is the logistic distribution function: where k is the standard deviation parameter.

[0022] The monocular three-dimensional model generation method based on the undirected distance function, wherein the reconstruction step also includes: constructing a third loss function based on the reconstructed multi-view image and the target multi-view image, and the second loss function and the third loss function are trained together to update the reconstruction module.

[0023] The present invention also proposes a Fig.12 The monocular 3D model generation device based on the undirected distance function shown in the figure comprises:

[0024] The initial module M1 obtains a monocular training image, wherein the monocular image contains a target object to be three-dimensionalized; the monocular training image is input into the image macro model to obtain a target multi-view image of the target object as a data prior;

[0025] A generation module M2 is used to construct a generation module composed of a two-dimensional diffusion model and a neural radiation field model, input the monocular training image into the generation module to obtain a generated multi-view image, and construct a first loss function according to the generated multi-view image and the target multi-view image to train and update the generation module;

[0026] A reconstruction module M3 is configured to construct a reconstruction module composed of an unsigned distance field model and a renderer, input the generated multi-view image into the reconstruction module, obtain a reconstructed multi-view image and a triangular mesh model of the target object, and construct a second loss function according to the reconstructed multi-view image and the generated multi-view image to train and update the reconstruction module;

[0027] Iteration module M4, loop iteratively executes the generation module and the reconstruction module until the first loss function and the second loss function converge or reach a preset number of loop iterations, and saves the current generation module and reconstruction module as the generation module and reconstruction module after training respectively;

[0028] Application module M5 inputs the monocular image containing the target object into the trained generation module, and inputs the generated multi-view image into the trained reconstruction module to obtain the triangular mesh model of the target object in the monocular image as the 3D model generation result.

[0029] The monocular 3D model generation device based on undirected distance function, wherein the generation module comprises:

[0030] The spatial position is hash-encoded, and the encoding result is input into the neural radiation field model to obtain a first encoding feature representing the density of each point in the space. The neural radiation field model renders the first encoding feature into an image of a specified perspective to obtain the generated multi-perspective image.

[0031] The monocular three-dimensional model generation device based on the undirected distance function, wherein the first coding feature also includes the undirected distance value output by the unsigned distance field model, specifically, the undirected distance value is fused with the density output by the neural radiation field model through the following formula:

[0032]

[0033] Where σ(p) represents the final rendering density of the neural radiation field of the three-dimensional point p in the space, σ g (ε hash (p)) is the density of the output of the neural radiation field model, σ u is the spatial density offset;

[0034]

[0035] Where (1-|p|) is the initial density deviation, |p| represents the distance to the center of the space, and s b ,s d and u Scaling factor representing density deviation;

[0036] is the logistic distribution function: where k is the standard deviation parameter.

[0037] The monocular three-dimensional model generation device based on the undirected distance function, wherein the reconstruction module also includes: constructing a third loss function according to the reconstructed multi-view image and the target multi-view image, and the second loss function and the third loss function are trained and updated together to update the reconstruction module.

[0038] The present invention also proposes an electronic device, which includes the monocular three-dimensional model generation device described above. The electronic device may be connected to an information display device, and the information display device is used to display the three-dimensional model generation result according to display parameters and attributes set by the user or through an artificial intelligence model.

[0039] The present invention also proposes a storage medium for storing a computer program for executing the monocular three-dimensional model generation method.

[0040] It can be seen from the above scheme that the advantages of the present invention are:

[0041] The method of the present invention can generate high-fidelity open surfaces, while previous methods cannot be extended to open surfaces. Table 1 below shows a quantitative comparison of the generation quality on the open surface dataset DF3D. The method of the present invention achieves the smallest chamfer error and achieves better generation results (33.36). Fig. 9 and Fig.10It is shown that the method of the present invention can generate shapes with open boundaries, while other methods can only generate closed shapes.

[0042]

[0043] Table 1: Quantitative comparison of build quality BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flow chart for generating a three-dimensional model based on three planes;

[0045] Figure 2 Generate a flow chart for 3D based 2D diffusion model;

[0046] Figure 3 Flowchart of neural reconstruction for three-dimensional open surface;

[0047] Figure 4 It is the algorithm flow chart of the present invention;

[0048] Figure 5 To generate a schematic diagram of the module;

[0049] Figure 6 This is a schematic diagram of the reconstruction module;

[0050] Figure 7 This is a schematic diagram of L1;

[0051] Figure 8 Schematic diagram of the spatial density offset in the field coupler in one dimension;

[0052] Fig. 9 This is a comparison diagram of the DF3D dataset;

[0053] Fig.10 Comparison diagrams on open boundary structures (butterfly, four-leaf clover, sunny doll) and closed shapes (teddy bear from GSO dataset and toys from RealFusion dataset);

[0054] Fig.11 is a flow chart of the method of the present invention;

[0055] Fig.12 It is a module diagram of the device of the present invention;

[0056] Fig.13 This is a schematic diagram of the structure of a first electronic device of the present invention;

[0057] Fig.14 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0058] Fig.15 It is a schematic diagram of the structure of a second electronic device of the present invention.

[0059] Reference numerals:

[0060] A-First electronic device;

[0061] B-monocular 3D model generation device;

[0062] C-data acquisition equipment;

[0063] D-information display device;

[0064] 1000 - second electronic device;

[0065] Ⅰ-computational unit;

[0066] II-ROM;

[0067] III-RAM;

[0068] IV-bus;

[0069] V-interface;

[0070] VI - input unit;

[0071] VII-output unit;

[0072] VIII- Storage medium;

[0073] Ⅸ-Communication unit. DETAILED DESCRIPTION

[0074] It should be noted that the processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0075] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0076] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0077] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0078] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0079] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0080] It should also be understood that the term "and / or" in the present invention is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship, which can be understood by referring to the context.

[0081] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0082] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0083] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0086] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0087] When the inventors were studying the monocular 3D model generation method, they found that the defect in the prior art was caused by only considering closed surfaces and not considering open surfaces. After research, the inventors found that the defect could be solved by a monocular reconstruction and 3D model generation method based on an undirected distance function. The joint framework newly proposed by the present invention can generate a high-quality 3D object model with arbitrary topological structure from a single image. In order to solve the challenge of generating correct topology based on sparse and fuzzy observations, the present invention combines the data prior from the multi-view diffusion model and the geometric prior brought by the unsigned distance field (UDF) reconstructor. Specifically, the present invention utilizes a joint framework consisting of the following parts: 1) a generation module for generating a neural radiation field that can be realistically rendered from any view; 2) a reconstruction module for extracting a learnable radiation field into a surface with arbitrary topological structure. Thus, the present invention can simultaneously utilize the prior of a two-dimensional diffusion model and a three-dimensional geometric constraint. The present invention further introduces a field coupler that bridges the radiation field and the unsigned distance field UDF under a novel optimization scheme. This enables the generation module and the reconstruction module to promote each other during the training process.

[0088] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0089] This paper proposes a new framework for generating high-quality 3D objects with arbitrary topology from a single image. To address the challenge of obtaining accurate topology in sparse images, we combine the data priors from the multi-view diffusion model and the geometric priors brought by the unsigned distance field (UDF) reconstructor. Specifically, we utilize a joint framework consisting of the following components: Generation module: Utilizes the data priors brought by the diffusion model to optimize the shallow density network to provide consistent rendering from arbitrary views. Reconstruction module: Introduces geometric priors to regularize the deep UDF network, which extracts the rendering results of the generation module into accurate geometry. Field coupler: Establishes positive feedback from UDF to rendering during the diffusion process, thereby significantly improving the geometric perception of the generation module without affecting the UDF distribution.

[0090] Data priors from the generative module:

[0091] The goal of the generation module is to use multi-view Figure 2 The 3D diffusion model and 3D data prior can effectively generate visualization representations from different viewpoints without geometric constraints such as closure.

[0092] For a given image I in and its corresponding predicted background I mask , we adopt a multi-view cross-domain diffusion scheme to obtain sparse six-view pseudo-reference color I i And the normal direction n of the i-th viewing angle i The generated colors have a fixed viewing angle c i , and used as anchor views in the generation module. It should be noted that we choose this multi-view cross-domain diffusion scheme to obtain consistency between the sparsely generated color and normal maps. The normal map will be used to handle geometric regularization in the reconstruction module. The use of anchor views is conducive to the initial estimation of the radiation field in the next stage, thereby reducing the complexity of the optimization process.

[0093] To improve the quality and consistency of renderings from different viewpoints, we perform the following steps:

[0094] 1) Sample a new perspective c n ;

[0095] 2) From c n Perspective rendering of the neural radiation field to obtain image I n ;

[0096] 3) Calculate the nearest anchor view c a and the corresponding anchor image;

[0097] 4) Using SDS loss function to optimize rendering I n :

[0098]

[0099] To calculate the SDS loss, we anchor the view image I a and anchor view angle c a , and from angle c n The sampled new view image I n , and input them together with the Zero123-based paradigm for denoising. Specifically, we use the stableZero123 model to extract the new view c n and the nearest anchor view c a The angle {R,T} between them is the condition, and the latent space is based on I n The denoising process is carried out: The final formula for SDS loss is:

[0100]

[0101] Geometric priors from the reconstruction module:

[0102] The reconstruction module aims to transform the rendering results from the generation module into surfaces with open boundaries and can flexibly handle topological structures. In this section, we adopt unsigned distance functions (UDFs) to represent surfaces with open and closed surfaces. With a set of rendering results from the generation module, we extract the rendering results into the UDF network using UDF-based volume rendering weights:

[0103]

[0104] The unsigned distance function (UDF) representation inherently contains a geometric prior that models the shape as a surface rather than a spatial volume. Eikonal regularization further regularizes the spatial region of the UDF and prevents the appearance of spurious surface patches:

[0105]

[0106] In addition, the UDF reconstructor will overfit to generate inconsistent areas in the rendering results, resulting in unclear zero isosurfaces of the UDF. To solve this problem, we propose a surface regularization loss Assume that the sampling point r in the UDF domain u At distance t u It intersects with the surface S at . In the local area, the ray r u and the surface normal n Su The angle between them can be regarded as a constant value <r u ,n Su >=θ. According to the characteristics of UDF, we expect that within a small interval around the first surface intersection point ( Figure 6The red marked area in the middle shows a "V"-shaped local UDF distribution:

[0107]

[0108] Based on this observation, the present invention adopts a surface regularization method based on estimated normals and rendering weight distribution to regularize the UDF gradient. For a sampling ray r, select the two nearest normal maps, for example, n1 and n2. For each sampling point r(t) on ray r, we project r(t) onto n1 and n2 to obtain the estimated normal n1(i t ,j t ),n2(i t ,j t )(where i t ,i j is the two-dimensional index of the sampling point). Then, we calculate the normal similarity cos of each r(t) <n1(i t ,j t ),n2(i t ,j t )>, where<a,b> represents the angle between vectors a and b. Based on the similarity and the rendering weight w(t) along the ray r(t), we find the sampling point that is most likely to be a surface point:

[0109] t s =argmin t cos <n1(i t ,j t ),n2(i t ,j t )>,t∈{t|w r (t)>δ w ,cos <n1(i t ,j t ),n2(i t ,j t )>>δ c}

[0110] The candidate sampling point set {t|w r (t)>δ w ,cos <n1(i t ,j t ),n2(i t ,j t )>>δ c} identifies the potential region for regularization. Initially, this set contains most of the sample points with high cosine similarity, which are widely distributed along the ray. During training, as the rendering weights w r Concentrating in a smaller area, the set of candidate sampling points will converge to a set of points with similar spatial locations.

[0111] Then, we select the selected sample point (denoted as r(t s ))’s 8 nearest neighboring sampling points, denoted as r(t s-4 ),r(t s-3 ),r(t s-1 ),r(t s+1 ),…,r(t s+4 ), which is used for surface regularization. The regularization formula is as follows:

[0112]

[0113] Where d represents the direction of the sampling ray, and a1 and a2 represent the azimuth values ​​of n1 and n2. Aims to use finite differences to regularize both the value and direction of the UDF gradient. Aims to s ) is regularized by the analytical gradient at . Aims to reduce the UDF value at the beginning of training.

[0114] Aligning data and geometric priors:

[0115] We use geometric priors and data priors to optimize two independent networks, namely, Figure 1 A shallow density network for consistent rendering and a deep UDF network for modeling realistic geometry. Figure 1 To extract the consistent rendering results into an accurate UDF representation, we adopt the existing UDF reconstruction work NeUDF. To improve the geometric perception of the density representation, we introduce the Field Coupler technique, which uses the UDF values ​​derived from the reconstruction module to enhance the generative rendering process. This is achieved by introducing a spatial density offset before the final activation layer of the density network, thereby coupling the density field and UDF:

[0116]

[0117] Where σ(p) represents the final rendering density of the neural radiation field of the 3D point p, σ g (ε hash (p)) is composed of a two-layer shallow network σ g The predicted density, is the undirected distance value output by the UDF network. u is the spatial density offset (e.g. Figure 8 shown):

[0118]

[0119] Where (1-|p|) adds an initial density bias to encourage the optimization process to focus on the central region of the unit space. Here |p| represents the distance to the center of the three-dimensional space. Then, and It is used to convert a "V" shaped UDF function into a bell shaped function, thus further directing the optimization to the surface of the geometry. b ,s d and u Scaling factor representing the density bias.

[0120] Specifically, is a logistic distribution function: where k is the standard deviation parameter, controlling the width of the bell-shaped function.

[0121] Offset σ u It has the following characteristics: 1) it peaks at the zero point of the UDF, 2) it decreases as the UDF value increases, possibly becoming negative, and 3) it increases slightly as it approaches the center of the field. The degree of increase and decrease is determined by s u ,s b It should be noted that when σ u When higher values ​​are obtained, the gradient of the density network increases: σ u It ensures that spatial points with lower UDF values ​​(close to the surface) have a higher probability of being activated in the density field, while points with higher UDF values ​​are more like frozen states. When calculating the rendering density σ, the UDF network The parameters are fixed.

[0122] The field coupler enables the UDF network to enhance the geometry awareness of the density network. The field coupler also ensures that the rendering results of the density network are roughly consistent with the geometry of the UDF network, while still leaving flexibility for the generation module to refine the rendering results without negatively affecting the UDF distribution in the stage where geometric regularization is not used.

[0123] Multi-stage training that reinforces each other:

[0124] To avoid the excessive memory overhead caused by training multiple modules and prior models simultaneously, we propose a novel multi-stage training scheme that can mutually improve the performance of the generation module and the reconstruction module. The entire training process is divided into three stages: Stage 1: A coarse UDF is generated using the reconstruction module and six anchor views and their corresponding normal maps. Stage 2: The neural rendering network is initialized with the coarse UDF values. Then, the rendering network is used to generate new views and its parameters are updated using the data prior from the stable Zero123. Stage 3: We adopt an alternating training strategy to perform rendering updates and surface distillation in a mutually reinforcing manner. Specifically, with the support of a fixed diffusion data prior, better multi-view rendering can improve the reconstruction module to generate higher-fidelity explicit surfaces. After the training of the reconstruction module converges, it is fixed and used as an enhanced shape prior to guide the optimization of the diffusion prior through a field coupler. Experiments show that our specially designed training strategy can mutually improve the two modules during training.

[0125] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0126] The present invention also proposes a Fig.12 The monocular 3D model generation device based on the undirected distance function shown in the figure comprises:

[0127] The initial module M1 obtains a monocular training image, wherein the monocular image contains a target object to be three-dimensionalized; the monocular training image is input into the image macro model to obtain a target multi-view image of the target object as a data prior;

[0128] A generation module M2 is used to construct a generation module composed of a two-dimensional diffusion model and a neural radiation field model, input the monocular training image into the generation module to obtain a generated multi-view image, and construct a first loss function according to the generated multi-view image and the target multi-view image to train and update the generation module;

[0129] A reconstruction module M3 is configured to construct a reconstruction module composed of an unsigned distance field model and a renderer, input the generated multi-view image into the reconstruction module, obtain a reconstructed multi-view image and a triangular mesh model of the target object, and construct a second loss function according to the reconstructed multi-view image and the generated multi-view image to train and update the reconstruction module;

[0130] Iteration module M4, loop iteratively executes the generation module and the reconstruction module until the first loss function and the second loss function converge or reach a preset number of loop iterations, and saves the current generation module and reconstruction module as the generation module and reconstruction module after training respectively;

[0131] Application module M5 inputs the monocular image containing the target object into the trained generation module, and inputs the generated multi-view image into the trained reconstruction module to obtain the triangular mesh model of the target object in the monocular image as the 3D model generation result.

[0132] The monocular 3D model generation device based on undirected distance function, wherein the generation module comprises:

[0133] The spatial position is hash-encoded, and the encoding result is input into the neural radiation field model to obtain a first encoding feature representing the density of each point in the space. The neural radiation field model renders the first encoding feature into an image of a specified perspective to obtain the generated multi-perspective image.

[0134] The monocular three-dimensional model generation device based on the undirected distance function, wherein the first coding feature also includes the undirected distance value output by the unsigned distance field model, specifically, the undirected distance value is fused with the density output by the neural radiation field model through the following formula:

[0135]

[0136] Where σ(p) represents the final rendering density of the neural radiation field of the three-dimensional point p in the space, σ g (ε hash (p)) is the density of the output of the neural radiation field model, σ u is the spatial density offset;

[0137]

[0138] Where (1-|p|) is the initial density deviation, |p| represents the distance to the center of the space, and s b ,s d and u Scaling factor representing density deviation;

[0139] is the logistic distribution function: where k is the standard deviation parameter.

[0140] The monocular three-dimensional model generation device based on the undirected distance function, wherein the reconstruction module also includes: constructing a third loss function according to the reconstructed multi-view image and the target multi-view image, and the second loss function and the third loss function are trained and updated together to update the reconstruction module.

[0141] The present invention also proposes an electronic device, which includes the monocular three-dimensional model generation device described above. The electronic device may be connected to an information display device, and the information display device is used to display the three-dimensional model generation result according to display parameters and attributes set by the user or through an artificial intelligence model.

[0142] like Fig.13 As shown, the present invention further proposes, in another embodiment, a first electronic device A including the monocular three-dimensional model generating device.

[0143] like Fig.14 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect and acquire monocular images, and the information display device D is used to display the triangular mesh model obtained by the analysis of the present invention.

[0144] The information display device D can sort and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. The user is presented with the key information specified by the user, and the user can understand the information more promptly without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present rich and necessary key information to the user.

[0145] The present invention also proposes a storage medium VIII in another embodiment for storing a computer program for executing the monocular three-dimensional model generation method. It should be understood that the storage medium in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0146] Fig.15 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown in the present invention, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present invention described and / or required in the present invention. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0147] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0148] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a disk, an optical disk, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0149] The computing unit I may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S5. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM II and / or a communication unit IX. When the computer program is loaded into RAM III and executed by the computing unit I, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit I may be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0150] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A monocular 3D model generation method based on undirected distance function, It is characterized in that include: An initial step is to obtain a monocular training image, wherein the monocular image contains a target object to be three-dimensionalized; The monocular training image is input into the image macro model to obtain a target multi-view image of the target object as a data prior; A generation step, constructing a generation module composed of a two-dimensional diffusion model and a neural radiation field model, inputting the monocular training image into the generation module to obtain a generated multi-view image, and constructing a first loss function according to the generated multi-view image and the target multi-view image to train and update the generation module; A reconstruction step, constructing a reconstruction module composed of an unsigned distance field model and a renderer, inputting the generated multi-view image into the reconstruction module, obtaining a reconstructed multi-view image and a triangular mesh model of the target object, and constructing a second loss function according to the reconstructed multi-view image and the generated multi-view image to train and update the reconstruction module; An iteration step, looping and iterating the generation step and the reconstruction step until the first loss function and the second loss function converge or reach a preset number of loop iterations, and saving the current generation module and the reconstruction module as the generation module and the reconstruction module after training respectively; The application step is to input a monocular image containing a target object into a trained generation module, and to input the generated multi-view image into a trained reconstruction module to obtain a triangular mesh model of the target object in the monocular image as a three-dimensional model generation result.

2. The method for generating a monocular three-dimensional model based on an undirected distance function as claimed in claim 1, It is characterized in that The generation steps include: The spatial position is hash-encoded, and the encoding result is input into the neural radiation field model to obtain a first encoding feature representing the density of each point in the space. The neural radiation field model renders the first encoding feature into an image of a specified perspective to obtain the generated multi-perspective image.

3. The method for generating a monocular three-dimensional model based on an undirected distance function as claimed in claim 2, It is characterized in that The first encoding feature also includes an undirected distance value output by the unsigned distance field model. Specifically, the undirected distance value is fused with the density output by the neural radiation field model through the following formula: Where σ(p) represents the final rendering density of the neural radiation field of the three-dimensional point p in the space, σ g (ε hash (p)) is the density of the output of the neural radiation field model, σ u is the spatial density offset; Where (1-|p|) is the initial density deviation, |p| represents the distance to the center of the space, and s b ,s d and u Scaling factor representing density deviation; is the logistic distribution function: where k is the standard deviation parameter.

4. The method for generating a monocular three-dimensional model based on an undirected distance function as claimed in claim 2, It is characterized in that The reconstruction step also includes: constructing a third loss function based on the reconstructed multi-view image and the target multi-view image, and the second loss function and the third loss function are trained together to update the reconstruction module.

5. A monocular 3D model generation device based on undirected distance function, It is characterized in that include: An initial module obtains a monocular training image, wherein the monocular image contains a target object to be three-dimensionalized; The monocular training image is input into the image macro model to obtain a target multi-view image of the target object as a data prior; A generation module is constructed, which is composed of a two-dimensional diffusion model and a neural radiation field model, the monocular training image is input into the generation module to obtain a generated multi-view image, and a first loss function is constructed according to the generated multi-view image and the target multi-view image to train and update the generation module; A reconstruction module is constructed, which is composed of an unsigned distance field model and a renderer, and the generated multi-view image is input into the reconstruction module to obtain a reconstructed multi-view image and a triangular mesh model of the target object, and a second loss function is constructed according to the reconstructed multi-view image and the generated multi-view image to train and update the reconstruction module; An iteration module, which executes the generation module and the reconstruction module in a loop and iterative manner until the first loss function and the second loss function converge or reach a preset number of loop iterations, and saves the current generation module and reconstruction module as the generation module and reconstruction module after training is completed, respectively; The application module inputs the monocular image containing the target object into the trained generation module, and inputs the generated multi-view image into the trained reconstruction module to obtain the triangular mesh model of the target object in the monocular image as the 3D model generation result.

6. The monocular 3D model generation device based on undirected distance function according to claim 1, It is characterized in that The generation module includes: The spatial position is hash-encoded, and the encoding result is input into the neural radiation field model to obtain a first encoding feature representing the density of each point in the space. The neural radiation field model renders the first encoding feature into an image of a specified perspective to obtain the generated multi-perspective image.

7. The monocular three-dimensional model generation device based on undirected distance function according to claim 2, It is characterized in that The first encoding feature also includes an undirected distance value output by the unsigned distance field model. Specifically, the undirected distance value is fused with the density output by the neural radiation field model through the following formula: Where σ(p) represents the final rendering density of the neural radiation field of the three-dimensional point p in the space, σ g (ε hash (p)) is the density of the output of the neural radiation field model, σ u is the spatial density offset; Where (1-|p|) is the initial density deviation, |p| represents the distance to the center of the space, and s b ,s d and u Scaling factor representing density deviation; is the logistic distribution function: where k is the standard deviation parameter.

8. The monocular three-dimensional model generation device based on undirected distance function according to claim 2, It is characterized in that The reconstruction module also includes: constructing a third loss function according to the reconstructed multi-view image and the target multi-view image, and the second loss function and the third loss function are trained together to update the reconstruction module.

9. An electronic device, It is characterized in that It includes a monocular three-dimensional model generation device as described in claims 5-8, and the electronic device is connected to an information display device, and the information display device is used to display the three-dimensional model generation result according to the display parameters and attributes set by the user or through an artificial intelligence model.

10. A storage medium for storing a computer program for executing the monocular three-dimensional model generation method according to claims 1-4.

Citation Information

Cited By

  • Multi-dimensional feature and three-plane representation enhanced three-dimensional grid reconstruction method and device

    CN119540497A

  • Three-dimensional model generation method compatible with open surface, electronic equipment and storage medium

    CN120707748A

  • Methods for generating 3D models compatible with open surfaces, electronic devices and storage media

    CN120707748B