Method and device for generating three-dimensional object from two-dimensional image

Abstract

The invention discloses a method for generating a three-dimensional object from a two-dimensional image, and the method comprises the steps: employing a cross-domain diffusion model constructed based on a multi-view cross-domain attention mechanism for the preprocessed two-dimensional image data, and generating a multi-view color image and a normal graph; based on the normal graph, a spatial point projection algorithm with normal constraint is adopted for projection, and an initialized training data set of the 3D Gaussian model is generated; according to the initialized training data set, a multi-view color image is used as supervision, a 3D Gaussian model is trained as representation of a three-dimensional object, normal constraint is added in the training process, and the 3D Gaussian model is trained; and inputting a two-dimensional image to be generated into the trained 3D Gaussian model to generate a three-dimensional object. According to the invention, the two-dimensional image can be rendered into a high-quality model picture in real time from the 3D Gaussian model representation.

Description

Technical Field

[0001] The present application relates to the field of computer graphics methods, and in particular to a method and device for generating a three-dimensional object from a two-dimensional image based on a 3D Gaussian model. Background Art

[0002] Currently, with the development of AIGC, artificial intelligence content generation technology is widely used in various computer graphics fields. However, due to the lack of three-dimensional data and the complexity of training, it is difficult to directly expand the two-dimensional artificial intelligence generation method to three dimensions. How to efficiently generate three-dimensional models from two-dimensional image data generated by artificial intelligence is an important task that needs to be solved in current computer graphics.

[0003] Currently, three types of methods for generating 3D objects are proposed in the prior art:

[0004] The first type is to directly use 3D data to train the diffusion model. However, due to the small amount of 3D data, it is not easy to train and the more complex structure of 3D objects, most of the current methods can only generate some relatively simple objects.

[0005] The second category is the method of generating 3D models based on NeRF (Neural Radiance Field), but this method cannot ensure consistent generation under multiple perspectives (multi-faceted Janus problem) and cannot improve the synthesis and rendering speed;

[0006] The third category is a method for generating 3D models based on 3D-GS (three-dimensional Gaussian Splatting), but the quality of the model generated by this method is not high and the consistency problem of generation under multiple perspectives still exists.

[0007] In order to solve the technical problems existing in the above methods, it is urgent to propose a method for generating a three-dimensional model from a single image based on Gaussian sputtering to solve the multi-faceted Janus problem that is easily generated in the existing three-dimensional object generation technology and the problem of low quality of the generated model, and to be able to render high-quality model images in real time from the 3D-GS representation. Summary of the invention

[0008] In order to solve the multi-faceted Janus problem and the technical problem of low quality of generated models in the above-mentioned prior art, a method and device for generating three-dimensional objects from two-dimensional images are realized.

[0009] In a first aspect, an embodiment of the present application provides a method for generating a three-dimensional object from a two-dimensional image, comprising:

[0010] Multi-view generation step: For the pre-processed two-dimensional image data, a cross-domain diffusion model based on a multi-view cross-domain attention mechanism is used to generate multi-view color images and normal maps, where the multi-view color images and normal maps maintain spatial geometric consistency;

[0011] Projection initialization step: Based on the normal map, a spatial point projection algorithm with normal constraints is used for projection to generate an initialization training data set for the 3D Gaussian model;

[0012] Model training steps: Based on the initial training data set, use multi-view color images as supervision, train a 3D Gaussian model as a representation of three-dimensional objects, add normal constraints during the training process, and train the 3D Gaussian model;

[0013] Three-dimensional object generation step: Input the two-dimensional image to be generated into the trained 3D Gaussian model to generate a three-dimensional object.

[0014] As an optional implementation, in the first aspect of the present invention, the method for generating a three-dimensional object from a two-dimensional image further includes:

[0015] 3D object restoration steps: Use the distance-aware sampling algorithm to sample the generated 3D objects based on the new camera perspective, and use the images generated by the diffusion model for supervision to optimize the fitting quality of the 3D objects.

[0016] As an optional implementation manner, in the first aspect of the present invention, the multi-view generating step further includes:

[0017] Dataset preparation steps: Generate a 2D image from a pre-built dataset and use image segmentation methods to extract the mask of the central object in the 2D image.

[0018] Steps for constructing a cross-domain diffusion model: For multiple domains, a cross-domain attention mechanism is used to construct a cross-domain diffusion model, where the domains include: normal domain and color image domain. The cross-domain attention mechanism is constructed based on the integrated cross-domain attention layer in the two-dimensional stable diffusion model, where the cross-domain self-attention layer is set before the cross self-attention layer; in the cross-domain self-attention layer, the keys from the normal image domain and the values ​​from the color image domain are combined and processed through self-attention operations.

[0019] As an optional implementation, in the first aspect of the present invention, the projection initialization step further includes:

[0020] Center point calculation steps: Set multiple camera perspectives around the three-dimensional object, and calculate the center point position of the camera based on each camera perspective;

[0021] Sampling steps: Create a bounding box of a preset size with the center point as the center, perform linear sampling within the bounding box, and obtain a rectangular point set;

[0022] Projection step: Take any point from the point set, project it onto the pixel plane through perspective projection to obtain the projected point set.

[0023] As an optional implementation, in the first aspect of the present invention, the projecting step further includes:

[0024] Screening step: based on the point set data, delete the points that are not on the pixel plane after projection, delete the points that are not in any mask map area, delete the points inside the 3D object, and retain the points near the surface of the 3D object;

[0025] Point cloud computing steps: calculate the center of the three-dimensional object by taking the mean of the projected point set, create a bounding box based on the center of the three-dimensional object, repeat the sampling and filtering steps, and generate a 3D Gaussian sphere set composed of point clouds after filtering.

[0026] As an optional implementation, in the first aspect of the present invention, the above-mentioned model training step further includes:

[0027] 3D Gaussian model training steps: Based on the 3D Gaussian ball set, the 3D Gaussian model is trained under the sparse input perspective;

[0028] Model supervision steps: Use normal maps, depth maps, and mask maps to supervise the training of the 3D Gaussian model and improve the model, calculate the total loss value during the 3D Gaussian model training process, and iteratively optimize the 3D Gaussian model. When the iterative calculation makes the total loss value reach the preset minimum value, the quality of fitting of the three-dimensional Gaussian ball model at a given viewing angle is the best.

[0029] As an optional implementation, in the first aspect of the present invention, the three-dimensional object repairing step further includes:

[0030] Training path establishment steps: Establish a path with the training view according to the set camera Figure 1 The circular path includes: a reference path and a repair path, the reference path is an arc near the camera, and the repair path is an arc outside the reference path;

[0031] New sampling step: resample in the repair path to obtain a new camera perspective, use the new camera perspective to render the 3D Gaussian model scene, and obtain the blurred image that needs to be optimized;

[0032] Optimize the image generation steps: find the nearest fixed camera perspective of the blurred image, put the training image under the fixed camera perspective and the difference between the new camera perspective and the fixed camera perspective into the diffusion model, generate an optimized image under the fixed camera perspective, compare the blurred image with the optimized image, and improve the quality of the 3D model under the new perspective.

[0033] In a second aspect, an embodiment of the present application provides a device for generating a three-dimensional object from a two-dimensional image, using the method for generating a three-dimensional object from a two-dimensional image as described above, the device includes:

[0034] Multi-view generation module: It is used to generate multi-view color images and normal maps based on the pre-processed two-dimensional image data using a cross-domain diffusion model built based on a multi-view cross-domain attention mechanism. The multi-view color images and normal maps maintain spatial geometric consistency.

[0035] Projection initialization module: Based on the normal map, a spatial point projection algorithm with normal constraints is used for projection to generate an initialization training data set for the 3D Gaussian model;

[0036] Model training module: Based on the initial training data set, multi-view color images are used as supervision to train a 3D Gaussian model as a representation of a three-dimensional object. Normal constraints are added during the training process to train the 3D Gaussian model.

[0037] 3D object generation module: used to input the 2D image to be generated into the trained 3D Gaussian model to generate a 3D object;

[0038] 3D object repair module: a distance-aware sampling algorithm is used to sample the generated 3D object based on a new camera perspective, and pictures generated by a diffusion model are used for supervision to optimize the fitting quality of the 3D object.

[0039] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for generating a three-dimensional object from a two-dimensional image.

[0040] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for generating a three-dimensional object from a two-dimensional image as described above are implemented.

[0041] Compared with the related prior art, it has the following outstanding beneficial effects:

[0042] 1. The method of the present invention uses a multi-view cross-domain 2D diffusion model to predict a fixed multi-view normal map and an RGB image based on an input image; it can generate a multi-view image and a normal map under a fixed view based on the input image;

[0043] 2. The method of the present invention designs a spatial point projection method with normal constraints as the initialization of the three-dimensional model; it can extract more reasonable point cloud features from sparse perspectives and use the point cloud as the initialization of 3D-GS;

[0044] 3. The method of the present invention designs a 3D Gaussian training method with normal constraints, adding normals as supervision during the 3D Gaussian sputtering training process; improving the performance of the 3D-GS method under sparse view input conditions;

[0045] 4. The method of the present invention uses a Gaussian restoration model based on a multi-view diffusion model; through the prior information of the diffusion model, it is possible to eliminate defects such as artifacts caused by insufficient viewing angle in the 3D Gaussian sputtering method, thereby further improving the rendering quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 A schematic diagram of a method for generating a three-dimensional object from a two-dimensional image according to an embodiment of the present invention;

[0048] Figure 2 Schematic diagram of an original image and a mask image after segmentation in a data set according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of estimating a depth map of a single image according to an embodiment of the present invention;

[0050] Figure 4 Schematic diagram of the structure of a multi-view cross-domain transformer block according to an embodiment of the present invention;

[0051] Figure 5 Schematic diagram of a spatial point projection algorithm with normal according to an embodiment of the present invention;

[0052] Figure 6 A schematic diagram of a reference path and a repair path according to an embodiment of the present invention;

[0053] Figure 7 A schematic diagram of a device for generating a three-dimensional object from a two-dimensional image according to an embodiment of the present invention;

[0054] Figure 8 It is a schematic diagram of the computer hardware of the present invention;

[0055] Fig. 9 A schematic diagram of the comparison between the results generated by the present invention and the prior art.

[0056] In the above picture:

[0057] 101 multi-view generation module 102 projection initialization module

[0058] 103 model training module 104 three-dimensional object generation module

[0059] 105 3D object repair module. DETAILED DESCRIPTION

[0060] It should be noted that the processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0061] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0062] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0063] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0064] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0065] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0066] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0067] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can be represented by: a, b, c, ab, ac, bc or abc, where a, b, c can be single or plural.

[0068] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0069] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0070] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0071] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0072] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0073] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0074] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0075] The method and device of the present invention are intended to propose a method for generating a 3D model from a single image based on Gaussian sputtering, aiming to solve the multi-faceted Janus problem that is easily generated in the existing 3D object generation technology and the problem of low quality of the generated model. The method can render a high-quality model image in real time from a 3D-GS representation.

[0076] The method of the present invention is mainly divided into the following four parts: First, the method of the present invention uses a multi-view cross-domain attention mechanism to generate multi-view images of objects and corresponding normal maps. Secondly, since the sparse view images cannot estimate the position of 3D points through the traditional motion structure recovery algorithm, the method of the present invention designs a spatial point projection algorithm with normal constraints as the initialization of 3D Gaussians, which can provide a better initial geometric prior for training 3D Gaussian sets. Then, the method of the present invention uses multi-view images as supervision to train a 3D Gaussian set as a representation of three-dimensional objects, and adds normal constraints during the training process. Finally, the method of the present invention designs a distance-aware sampling algorithm to sample new viewpoints, and uses the pictures generated by Stable Zero123 as supervision to optimize the quality of objects at new viewpoints.

[0077] In order to reasonably evaluate the robustness and quality of the model, the method of the present invention was compared with the single image generation method under the same data set based on multiple indicators such as PSNR, SSIM, LPIPS, etc. The experimental results show that: compared with the comparison method, the method of the present invention has certain improvements in most examples.

[0078] Embodiment 1

[0079] like Figure 1 As shown, a method for generating a three-dimensional object from a two-dimensional image in an embodiment of the present application includes:

[0080] Multi-view generation step S101: for the pre-processed two-dimensional image data, a cross-domain diffusion model based on a multi-view cross-domain attention mechanism is used to generate a multi-view color image and a normal map;

[0081] The method of the present invention uses a cross-domain diffusion model in the Wonder3D method (https: / / arxiv.org / abs / 2310.15008) in a specific embodiment, which allows connections to other views within the attention layer by extending the self-attention layer in the transformer module. The keys and values ​​of different views are interconnected to facilitate information exchange. By sharing information between different views within the attention layer, the diffusion model can perceive multi-view correlations and generate geometrically consistent multi-view color images and normal maps.

[0082] The multi-view generating step S101 further includes:

[0083] Data set preparation step S1011: Generate a two-dimensional image from a pre-constructed data set, and use an image segmentation method to extract a mask image of a central object in the original two-dimensional image;

[0084] 1. Dataset Introduction

[0085] The method of the present invention is mainly tested on some examples in the OpenIllumination dataset (OpenIllumination is a commonly used dataset for object generation, which mainly contains real pictures of multiple objects under different lighting conditions. The pictures with the best lighting conditions are selected for testing in the specific embodiment of the method. The download link of the OpenIllumination dataset is: https: / / arxiv.org / abs / 2309.07921). At the same time, some examples outside the dataset are also tested to demonstrate the robustness of the method. Due to the different formats of different datasets and the different resolutions of the pictures, the method of the present invention also performs some preprocessing operations on all the data used.

[0086] 2. Data processing

[0087] In the OpenIllumination dataset, in the specific embodiment of the method of the present invention, a single image was generated on examples with the best lighting conditions, such as pumpkin vines and fabric toys. During the experiment, in the specific embodiment of the method of the present invention, only the reconstruction of the central object in the scene was considered, and the background of the scene was not reconstructed. Therefore, the method of the present invention needs to use some image segmentation algorithms to extract the mask of the central object in each dataset. In the specific embodiment of the present invention, the segment-anything image segmentation method developed by Meta is used, which can accurately extract the mask of the central object, but the present invention is not limited to this. Figure 2 As shown, the original image in the dataset and the mask image after segmentation.

[0088] In the process of 3D-GS training, depth map constraints can be optionally added. Therefore, in the specific embodiment of the method of the present invention, it is also necessary to use ZoeDepth, a monocular depth estimation framework, to perform depth estimation on the images in the data set, but the present invention is not limited to this. The results are as follows Figure 3 shown.

[0089] Cross-domain diffusion model construction step S1012: for multiple domains, a cross-domain diffusion model is constructed using a cross-domain attention mechanism, wherein the domains include: a normal domain and a color image domain, and the cross-domain attention mechanism is constructed based on an integrated cross-domain attention layer in a two-dimensional stable diffusion model, wherein a cross-domain self-attention layer is arranged before a cross self-attention layer; in the cross-domain self-attention layer, keys from the normal image domain and values ​​from the color image domain are combined and processed through a self-attention operation;

[0090] Furthermore, in a specific embodiment of the present invention, the cross-domain diffusion model is based on a pre-trained two-dimensional stable diffusion model (2D Stable Diffuison, hereinafter referred to as SD). However, existing two-dimensional diffusion models are designed for a single domain and cannot generate images of multiple domains (normal domain and color image domain) at the same time. To overcome this difficulty, the cross-domain diffusion model of the present invention designs a domain switcher, denoted as s. The switcher s is a one-dimensional vector (the switcher is mainly suitable for marking color images and normal images. In a specific embodiment of the present invention, 0|1 markings can be used, but the present invention is not limited to this, and other vectors can also be used), which is used to mark the color image domain and the normal image domain. After position encoding and time embedding connection, a multi-layer perceptron is used to realize position encoding and time embedding, and then the vectors output by the multi-layer perceptron are spliced ​​and injected into the U-Net network of the SD model to realize switching between the two domains.

[0091] By adding the domain switcher flag mentioned above, the SD model can switch freely between generating normal maps and color images. However, for the same viewing angle, the above method cannot guarantee that the generated color image and normal map are geometrically consistent. To solve this problem, the cross-domain diffusion model introduces a cross-domain attention mechanism to promote information exchange between the two domains. The cross-domain attention layer maintains the same structure as the original SD model self-attention layer and is integrated before the cross-attention layer in each Transformer block of the U-Net, as shown in Figure 4 In the cross-domain attention layer, the keys from the normal image domain and the values ​​from the color image domain are combined and processed through the self-attention operation, where self-attention is a mechanism used in neural networks to capture the dependencies between elements within the input sequence. For each element x in the input image sequence, _i , generate the query vector q through linear transformation _i , key vector k _i Sum value vector v _i , for each element in the sequence, calculate its similarity score with all other elements in the sequence. This is usually done by taking the dot product of the query vector and the key vector. The elements in each image sequence are updated by the similarity score to achieve the optimization purpose. The design of the method of the present invention ensures that the generation of color images and normal maps is closely related, thereby promoting geometric consistency between the two domains, i.e., the predicted normal map and color map and the normal map of the 3D geometric model itself and the rendered color map. Figure 1 Sample.

[0092] The method of the present invention has the following positive technical effects: a multi-view cross-domain 2D diffusion model is used to predict a fixed multi-view normal map and an RGB image based on an input image; a multi-view image and a normal image under a fixed view can be generated based on the input image.

[0093] Projection initialization step S102: Based on the normal map, a spatial point projection algorithm with normal constraints is used for projection to generate an initialization training data set for a 3D Gaussian model;

[0094] The above projection initialization step S102 further includes:

[0095] Center point calculation step S1021: setting multiple camera perspectives around the three-dimensional object, and calculating the center point position of the camera based on each camera perspective, wherein the cameras are set as a circular path and the cameras are spaced at the same angle;

[0096] Sampling step S1022: creating a bounding box of a preset size with the center point as the center, and performing linear sampling in the bounding box to obtain a rectangular point set;

[0097] Projection step S1023: select any point from the point set, and perform perspective projection on the pixel plane to obtain a projected point set.

[0098] The overall algorithm flow of the specific embodiment of the present invention is as follows: Figure 5 As shown, in the specific embodiment of the present invention, after using Wonder3D to generate the fixed six camera perspectives {π1, ..., π6} of the image (relative coordinates are used between the cameras), the center point o (x o ,y o ,z o ), with point o as the center, create a bounding box (Bounding Box) of a specified size of 400*400. In the specific embodiment of the present invention, the specified bounding box range is 1-1000, but the present invention is not limited thereto. A bounding box value greater than 1000 can also be taken according to the situation. A linear sampling operation is performed in the bounding box to obtain a rectangular point set P = {p1, ..., p n}, select any point p from the point set P i ∈P, p i After perspective projection onto the pixel plane (points in three-dimensional space are projected onto the pixel plane to form an image), the projected point set P′={p′1,...,p′ n};

[0099] As an optional implementation, in the first aspect of the present invention, the projection step S1023 further includes:

[0100] 1. Screening step: Based on the point set data, delete the points that are not on the pixel plane after projection, delete the points that are not in any mask map area, delete the points inside the 3D object, and retain the points near the surface of the 3D object;

[0101] In the embodiment of the present invention, the elements in the sets P and P' are screened as follows:

[0102] First, we need to delete those points that are not on the pixel plane after projection. 3D-GS uses the Colmap camera coordinate system, in which the z-axis faces forward, the x-axis faces right, and the y-axis faces downward. Therefore, the projected point p′ i (x′ i ,y′ i ,z′ i )∈P′, if z′ i ≤0, it means that the point is invisible from the camera's perspective and needs to be deleted. i According to z′ i Normalize to get point p i Coordinate p″ on the pixel plane i(u i ,v i ), for u i <0∨u i >image_width and v i <0∨v i >image_height (image_width and image_height are the image width and height in the camera intrinsic parameters) (that is, points outside the pixel plane) also need to be deleted.

[0103] Secondly, we need to delete those points that are not in any mask area. According to the coordinates p″ of the point on the pixel plane i (u i ,v i ), sampling is performed on the mask images under the six viewing angles, and a total of six vectors with sampling values ​​of 0 or 1 are obtained 1 means that the point is within the mask area. In other words, the point is inside the object. On the contrary, 0 means that the point is not within the mask area and needs to be deleted.

[0104] Finally, we need to delete the points inside the object and keep only the points near the surface of the object. Similar to the previous method of deleting points that are not in any mask area, we use the coordinates p″ of the point on the pixel plane to i (u i ,v i ), at a distance point p i Sampling is performed on the two nearest normal maps to obtain point p i The two normals and The cosine similarity between two normals is calculated using the following formula:

[0105]

[0106] Set a normal filtering threshold normal_threshold, delete all points below the threshold, and retain the points near the surface of the object.

[0107] 2. Point cloud computing steps: Calculate the center of the three-dimensional object by taking the mean of the projected point set, create a bounding box based on the center of the three-dimensional object, repeat the sampling and filtering steps, and generate a 3D Gaussian sphere set composed of point clouds after filtering.

[0108] In a specific embodiment of the present invention, since the center of the camera is not necessarily the center of the object in most cases, after performing the above-mentioned screening operation, the center of the object can be calculated by taking the average of the obtained point cloud, and then a bounding box is created according to the center of the object. After repeating the above-mentioned sampling steps and screening steps, a point cloud with a better geometric structure can be obtained as the initialization of the 3D-GS training process.

[0109] The positive technical effects of the method of the present invention are as follows: the present invention designs a spatial point projection method with normal constraints as the initialization of the three-dimensional model; it can extract more reasonable point cloud features from sparse perspectives, and use the point cloud as the initialization of 3D-GS;

[0110] Model training step S103: according to the initialization training data set, using the multi-view color image as supervision, training the 3D Gaussian model as the representation of the three-dimensional object, adding normal constraints during the training process, and training the 3D Gaussian model;

[0111] The above-mentioned model training step S103 further includes:

[0112] 3D Gaussian model training step S1031: Based on the 3D Gaussian ball set, the 3D Gaussian model is trained under the input sparse view angle; in the specific embodiment of the present invention, after obtaining the point cloud with a good geometric structure, the Gaussian ball set G (g1, ..., g n ), train 3D-GS under sparse input viewpoints;

[0113] Model supervision step S1032: Use the normal map, depth map and mask map to supervise the training of the 3D Gaussian model and improve the model, calculate the total loss value during the 3D Gaussian model training process, and iteratively optimize the 3D Gaussian model. When the iterative calculation makes the total loss value reach the preset minimum value, the quality of fitting of the three-dimensional Gaussian ball model at a given viewing angle is the best.

[0114] In a specific embodiment of the present invention, during the training process, in addition to using the color image and the rendered image for difference calculation loss, the normal map, the depth map and the mask map are also used for supervision.

[0115] In a specific embodiment of the present invention, there is a scaling vector S(s x ,s y ,s z) is used to control the scaling ratio of each axis of the Gaussian sphere, and the present invention selects the smallest axis of the three scaling components as the normal of the Gaussian, similar to the alpha-blending strategy when 3D-GS renders colors. The normal information of each Gaussian is also passed into the differentiable rasterizer of 3D-GS as a color, and the normal map of the corresponding viewing angle is rendered and compared with the normal map generated by the cross-domain diffusion model. The loss function of the normal is defined as follows:

[0116] L Normal =1-normal_cosine_similarity(n render ,n gt ).mean()

[0117] L normal is the normal loss function defined, which is obtained by taking the cosine similarity between the rendered normal and the true normal and taking the average value, L normal The smaller the value, the closer the model's normal is to the true normal. render is the rendered normal map, n gt is the true normal map.

[0118] In addition, the specific embodiment of the present invention also uses the monocular depth estimation framework ZoeDepth to perform monocular depth estimation on the six input images. In addition, the depth of each Gaussian is sorted during the 3D-GS rasterization process, and the depth map can be directly rendered according to the depth value. The depth loss function during the training process is defined as follows:

[0119] L Depth =1-pearson_corrcoef(d render ,d gt )

[0120] L Depth is the depth loss function, which is mainly obtained by comparing the rendered depth map with the real depth map. render is the rendered depth map, d gt is the real depth map. pearson_corrcoef is the Pearson correlation coefficient, a statistic used to measure the linear correlation between two variables. It can calculate the correlation between the two depth maps. The calculation formula is as follows:

[0121]

[0122] Therefore, the total loss calculation formula during the improved 3D-GS training process is as follows:

[0123] L total =L1+L D-SSIM +L Depth +L Normal

[0124] L1 loss is a common loss function, which is obtained by directly taking the absolute average of the RGB values ​​of two images. D-SSIM To calculate the (dis)similarity between two or more images using an algorithm that approximates human vision (https: / / ece.uwaterloo.ca / ~z70wang / research / ssim / ).

[0125] Positive technical effects of the method of the present invention: The present invention designs a 3D Gaussian training method with normal constraints, adds normals as supervision during the 3D Gaussian sputtering training process, and improves the performance of the 3D-GS method in the case of sparse visual input.

[0126] Three-dimensional object generation step S104: inputting the two-dimensional image to be generated into the trained 3D Gaussian model to generate a three-dimensional object.

[0127] Furthermore, the method of the present invention also includes:

[0128] Three-dimensional object repair step S105: adopt a distance-aware sampling algorithm to sample the generated three-dimensional object based on the new camera perspective, and use the image generated by the diffusion model for supervision to achieve the optimization of the fitting quality of the three-dimensional object.

[0129] The above three-dimensional object repairing step S105 further includes:

[0130] Training path establishment step S1051: Establish a path corresponding to the training view according to the set camera. Figure 1 The circular path includes: a reference path and a repair path, the reference path is an arc near the camera, and the repair path is an arc outside the reference path;

[0131] In the specific embodiment of the present invention, after training 3D-GS for a certain number of rounds under sparse viewing angles, the quality of fitting of the three-dimensional Gaussian sphere model under a given viewing angle is already good. At the same time, the quality under a new viewing angle that is closer to the given viewing angle is also good, and no further optimization is required. Therefore, Figure 6 As shown, the method of the present invention establishes a line with the training view according to the input camera Π{π1,…,π6} Figure 1 The circular path is input to the camera π i The nearby arc is defined as the reference path Arc ref , the reference path can render high-quality images without further optimization. The arc outside the reference path constitutes the repair path Arc repair . Reference pathArc ref The length of is determined by the hyperparameter d (parameter d is manually specified).

[0132] New sampling step S1052: resampling is performed in the repair path to obtain a new camera perspective, and the 3D Gaussian model scene is rendered using the new camera perspective to obtain a blurred image that needs to be optimized;

[0133] Optimized image generation step S1053: find the nearest fixed camera perspective of the blurred image, put the training image under the fixed camera perspective, and the difference between the new camera perspective and the fixed camera perspective into the diffusion model, generate an optimized image under the fixed camera perspective, compare the blurred image with the optimized image, and improve the quality of the 3D model under the new perspective.

[0134] In a specific embodiment of the present invention, in the repair path Arc repair Random sampling is performed in to obtain a new camera angle π nov , use the new camera perspective to render the 3D-GS scene and get the blurred image x that needs to be optimized nov Next, the method of the present invention finds π nov The nearest fixed viewing angle π i ∈Π, π i The training image under and the difference between the two perspectives are put into StableZero123 to generate π nov Stable Zero123 is a variant of the diffusion model, which can generate an image x at the target perspective based on the input image and the camera parameters of the target perspective. gen . Render the new perspective image x nov And generate image x gen By making comparisons, the quality of 3D-GS under new perspectives can be improved. The loss function in the Gaussian restoration process is defined as follows:

[0135]

[0136] Where E represents the mean, w(t) and λ(π nov ) are the diffusion model denoising weight function and distance weight function, respectively, x nov is a picture rendered from a new perspective, x gen This is the image restored by zero123. p It is to calculate the LPIPS similarity between two images.

[0137] Positive technical effects of the method of the present invention: the present invention uses a Gaussian restoration model based on a multi-view diffusion model; through the prior information of the diffusion model, it is possible to eliminate defects such as artifacts caused by insufficient viewing angle in the 3D Gaussian sputtering method, thereby further improving the rendering quality.

[0138] Embodiment 2

[0139] The embodiment of the present application provides a device for generating a three-dimensional object from a two-dimensional image, using the method for generating a three-dimensional object from a two-dimensional image as described above, such as Figure 7 As shown, the device comprises:

[0140] The multi-view generation module 101 is used to generate a multi-view color image and a normal map based on the pre-processed two-dimensional image data by using a cross-domain diffusion model constructed based on a multi-view cross-domain attention mechanism, wherein the multi-view color image and the normal map maintain spatial geometric consistency;

[0141] Projection initialization module 102: Based on the normal map, a spatial point projection algorithm with normal constraints is used for projection to generate an initialization training data set for a 3D Gaussian model;

[0142] Model training module 103: according to the initialization training data set, using the multi-view color image as supervision, training the 3D Gaussian model as the representation of the three-dimensional object, adding normal constraints during the training process, and training the 3D Gaussian model;

[0143] The three-dimensional object generation module 104 is used to input the two-dimensional image to be generated into the trained 3D Gaussian model to generate a three-dimensional object;

[0144] 3D Gaussian model repair module 105: adopts a distance-aware sampling algorithm to sample the generated three-dimensional object based on a new camera perspective, and uses a picture generated by a diffusion model for supervision to achieve optimization of the fitting quality of the three-dimensional object.

[0145] Embodiment 3

[0146] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for generating a three-dimensional object from a two-dimensional image.

[0147] Embodiment 4

[0148] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for generating a three-dimensional object from a two-dimensional image as described above are implemented.

[0149] In addition, combined Figure 1 The method for generating a three-dimensional object from a two-dimensional image according to the embodiment of the present application can be implemented by an electronic device, such as a computer device. Figure 8 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present application.

[0150] In some of these embodiments, the computer device may further include a communication interface 83 and a bus 80. Figure 8As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0151] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0152] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0153] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any one of the methods for generating a three-dimensional object from a two-dimensional image in the above embodiments.

[0154] In summary, compared with the prior art, the method of the present invention has the following beneficial effects:

[0155] In order to reasonably evaluate the robustness and quality of the model, the present invention conducted a comparative experiment with the single image generation method under the same data set based on multiple indicators such as PSNR, SSIM, LPIPS, etc. The experimental results show that the present invention method has certain improvements over the comparison method in most examples.

[0156] Effect comparison display

[0157] In the embodiment of the present invention, the method of the present invention is used on some examples in some data sets to perform a process of generating a three-dimensional object from a single image, and is compared with other generation methods, such as Fig. 9 As shown, some of the generated results are given.

[0158] It can be seen that in the two examples tested, the method of the present invention is obviously better than the DreamGaussian method, which also generates a 3D model for a single image. Compared with the random initialization of the DreamGaussian method within a sphere, the present invention uses a spatial point projection technology with normal constraints to initialize 3D-GS, and has good geometric shapes and approximate color information under various viewing angles before starting training. After training, the method of the present invention is better than DreamGaussian in detail expression. In the example of a toy cat (catstatue), the method of the present invention has a more obvious stripe effect on the back, and the overall details of the model are clearer.

[0159] The present invention takes a single picture as input and compares the method of the present invention with the existing method of generating a three-dimensional model from a single image. The compared methods include DreamGaussian and the original 3D-GS. The numerical results of the comparison are shown in Table 1 below.

[0160]

[0161] Table 1 Comparison of numerical values ​​of 3D objects generated from a single image

[0162] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for generating a three-dimensional object from a two-dimensional image, characterized in that: The method comprises: Multi-view generation step: for the pre-processed two-dimensional image data, a cross-domain diffusion model based on a multi-view cross-domain attention mechanism is used to generate a multi-view color image and a normal map, wherein the multi-view color image and the normal map maintain spatial geometric consistency; Projection initialization step: Based on the normal map, a spatial point projection algorithm with normal constraints is used for projection to generate an initialization training data set for the 3D Gaussian model; Model training step: according to the initialized training data set, using the multi-view color image as supervision, training the 3D Gaussian model as a representation of the three-dimensional object, adding normal constraints during the training process, and training the 3D Gaussian model; Three-dimensional object generation step: input the two-dimensional image to be generated into the trained 3D Gaussian model to generate a three-dimensional object.

2. The method for generating a three-dimensional object from a two-dimensional image according to claim 1, characterized in that: The method further comprises: Three-dimensional object repair step: use a distance-aware sampling algorithm to sample the generated three-dimensional object based on a new camera perspective, and use images generated by a diffusion model for supervision to optimize the fitting quality of the three-dimensional object.

3. The method for generating a three-dimensional object from a two-dimensional image according to claim 1 or 2, characterized in that: The multi-view generating step further includes: Dataset preparation step: generating a two-dimensional image from a pre-constructed dataset, and extracting a mask image of a central object in the two-dimensional image original image using an image segmentation method; Cross-domain diffusion model construction steps: For multiple domains, a cross-domain attention mechanism is used to construct the cross-domain diffusion model, wherein the domains include: a normal domain and a color image domain, and the cross-domain attention mechanism is constructed according to integrating a cross-domain attention layer in a two-dimensional stable diffusion model, wherein the cross-domain self-attention layer is arranged before the cross self-attention layer; in the cross-domain self-attention layer, the keys from the normal image domain and the values ​​from the color image domain are combined and processed through self-attention operations.

4. The method for generating a three-dimensional object from a two-dimensional image according to claim 1 or 2, characterized in that: The projection initialization step further includes: Center point calculation step: setting a plurality of camera viewing angles around the three-dimensional object, and calculating the center point position of the camera based on each of the camera viewing angles; Sampling step: creating a bounding box of a preset size with the center point as the center, and performing linear sampling in the bounding box to obtain a rectangular point set; Projection step: select any point from the point set, and perform perspective projection of the arbitrary point onto the pixel plane to obtain a projected point set.

5. The method for generating a three-dimensional object from a two-dimensional image according to claim 4, characterized in that: The projecting step further includes: Screening step: based on the point set data, deleting points that are not on the pixel plane after projection, deleting points that are not in any area of ​​the mask map, deleting points inside the three-dimensional object, and retaining points near the surface of the three-dimensional object; Point cloud computing step: calculate the center of the three-dimensional object by taking the mean of the projected point set, create a bounding box based on the center of the three-dimensional object, repeat the sampling and screening steps, and generate a 3D Gaussian sphere set composed of point clouds after screening.

6. The method for generating a three-dimensional object from a two-dimensional image according to claim 5, characterized in that: The model training step further includes: 3D Gaussian model training step: based on the 3D Gaussian ball set, training the 3D Gaussian model under the input sparse perspective; Model supervision step: Use the normal map, depth map and mask map to supervise the training of the 3D Gaussian model and improve the model, calculate the total loss value during the training process of the 3D Gaussian model, and iteratively optimize the 3D Gaussian model. When the iterative calculation makes the total loss value reach a preset minimum value, the quality of fitting of the three-dimensional Gaussian ball model at a given viewing angle is the best.

7. The method for generating a three-dimensional object from a two-dimensional image according to claim 2, characterized in that: The three-dimensional object repairing step further includes: A training path establishment step: establishing a circular path consistent with the training view according to the set camera, wherein the circular path includes a reference path and a repair path, wherein the reference path is an arc near the camera, and the repair path is an arc outside the reference path; New sampling step: resample in the repair path to obtain a new camera perspective, use the new camera perspective to render the 3D Gaussian model scene, and obtain the blurred image that needs to be optimized; Optimized image generation step: find the nearest fixed camera perspective of the blurred image, put the training image under the fixed camera perspective and the difference between the new camera perspective and the fixed camera perspective into the diffusion model, generate an optimized image under the fixed camera perspective, compare the blurred image with the optimized image, and improve the quality of the 3D model under the new perspective.

8. A device for generating a three-dimensional object from a two-dimensional image, using the method for generating a three-dimensional object from a two-dimensional image as claimed in any one of claims 1 to 7, characterized in that: The device comprises: Multi-view generation module: used for generating multi-view color images and normal maps based on the pre-processed two-dimensional image data by using a cross-domain diffusion model based on a multi-view cross-domain attention mechanism, wherein the multi-view color images and the normal maps maintain spatial geometric consistency; Projection initialization module: Based on the normal map, a spatial point projection algorithm with normal constraints is used for projection to generate an initialization training data set for a 3D Gaussian model; Model training module: according to the initialized training data set, using the multi-view color image as supervision, training the 3D Gaussian model as a representation of the three-dimensional object, adding normal constraints during the training process, and training the 3D Gaussian model; A three-dimensional object generation module: used for inputting the two-dimensional image to be generated into the trained 3D Gaussian model to generate a three-dimensional object; 3D object repair module: a distance-aware sampling algorithm is used to sample the generated 3D object based on a new camera perspective, and pictures generated by a diffusion model are used for supervision to optimize the fitting quality of the 3D object.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for generating a three-dimensional object from a two-dimensional image according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for generating a three-dimensional object from a two-dimensional image according to any one of claims 1 to 7 are implemented.

Images