Three-dimensional reconstruction method and device, terminal and storage medium

Abstract

The invention provides a three-dimensional reconstruction method and device, a terminal and a storage medium. The three-dimensional reconstruction method comprises the following steps: inputting a single-view image and a camera position thereof into a trained diffusion model; outputting a three-dimensional Gaussian splashing parameter corresponding to an object in the single-view image through a diffusion model; and performing three-dimensional object rendering by using the three-dimensional Gaussian splashing parameters to obtain a reconstructed three-dimensional object. According to the method, the three-dimensional Gaussian splashing parameters are directly generated through the single-view-angle image, the three-dimensional knowing capability is provided, and the problem of three-dimensional inconsistency is solved; in addition, the reconstruction pipeline is composed of the single diffusion model, no extra model assembly is needed, and the efficiency of reconstructing and generating the pipeline is improved. Besides, in the aspect of reconstruction of the three-dimensional representation, the rendering quality and speed are improved by adopting the multi-view three-dimensional Gaussian splash representation, and the reconstructed three-dimensional object can be quickly and accurately rendered.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to three-dimensional reconstruction methods, apparatus, terminals and storage media. Background Technology

[0002] Reconstructing 3D objects from single-view images is a typical ill-posed problem with wide applications in game development, filmmaking, industrial generation, and electronic design. The challenge lies in the fact that the reconstructed 3D object must not only match the appearance of the input image from the input viewpoint but also maintain logical geometry from any other viewpoint. Reconstruction methods, however, infer the overall appearance and geometric information of any object solely from the input image, posing significant challenges to accuracy and generalization. Furthermore, in practical applications, improving user experience and productivity demands high speed from reconstruction methods.

[0003] A commonly used single-view 3D object reconstruction technique is called CAT3D (Create Anything in 3D with Multi-View Diffusion Models). This method uses an image generation model to first use the single-view image as a constraint to generate object images from multiple specified new viewpoints. Then, it uses the generated multiple images with camera viewpoints to reconstruct the 3D object using a multi-view reconstruction method, namely Neural Radiance Field (NeRF).

[0004] Image generation models, lacking internal 3D constraints, are essentially probabilistic distribution prediction problems performed on a 2D plane. Therefore, the generated multi-view images suffer from 3D inconsistencies, and using these inconsistent images as input for the next step severely degrades the final reconstruction quality. Furthermore, multi-view image generation relies solely on a single image from the input viewpoint as a conditional guide, making it impossible to control ambiguous single-view reconstruction via text commands, significantly reducing user controllability during the reconstruction process. Moreover, existing reconstruction pipelines consist of two unrelated modules: the first step, multi-view image generation, typically takes less than 10 seconds, while the second step, reconstructing a single 3D neural radiation field from the multi-view images, often takes over ten minutes, resulting in extremely low reconstruction efficiency. Additionally, the rendering of the reconstructed neural radiation field (NeRF) employs ray tracing, requiring the sampling of hundreds of points along the ray propagation path for real-time computation to render a single pixel, leading to high computational costs after training. Therefore, further improvements in these areas are desired. Summary of the Invention

[0005] To address the existing problems, this disclosure provides a three-dimensional reconstruction method, apparatus, terminal, and storage medium.

[0006] The following technical solution is adopted in this disclosure.

[0007] Embodiments of this disclosure provide a three-dimensional reconstruction method, which includes: inputting a single-view image and its camera position into a trained diffusion model; outputting three-dimensional Gaussian splash parameters corresponding to objects in the single-view image through the diffusion model; and rendering the three-dimensional object using the three-dimensional Gaussian splash parameters to obtain a reconstructed three-dimensional object.

[0008] Another embodiment of this disclosure provides a three-dimensional reconstruction apparatus. The processing apparatus includes: an input module configured to input a single-view image and its camera position into a trained diffusion model; an output module configured to output three-dimensional Gaussian splash parameters corresponding to objects in the single-view image through the diffusion model; and a rendering module configured to render three-dimensional objects using the three-dimensional Gaussian splash parameters to obtain reconstructed three-dimensional objects.

[0009] In some embodiments, this disclosure provides a terminal, including: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the above-described three-dimensional reconstruction method.

[0010] In some embodiments, this disclosure provides a storage medium for storing program code for executing the above-described three-dimensional reconstruction method.

[0011] This disclosure directly generates 3D Gaussian splatter parameters from a single-view image, providing 3D discernibility and resolving the issue of 3D inconsistency. Furthermore, the reconstruction pipeline of this disclosure consists of a single diffusion model, eliminating the need for additional model components and improving the efficiency of the reconstruction generation pipeline. In addition, regarding the reconstruction of 3D representations, this disclosure employs multi-view 3D Gaussian splatting to enhance rendering quality and speed, enabling fast and accurate rendering of reconstructed 3D objects. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of a three-dimensional reconstruction method according to an embodiment of the present disclosure.

[0014] Figure 2 A diffusion model structure diagram according to some embodiments is shown.

[0015] Figure 3 The process of multi-view 3D Gaussian reconstruction according to some embodiments is illustrated.

[0016] Figure 4 The reconstruction effect of a single-view 3D object according to an embodiment of the present disclosure is shown.

[0017] Figure 5 The reconstruction effect of a single-view 3D object according to an embodiment of the present disclosure is shown.

[0018] Figure 6 The reconstruction effect of a single-view 3D object according to an embodiment of the present disclosure is shown.

[0019] Figure 7 The reconstruction results corresponding to the same input image and different text description information are shown in the embodiments of this disclosure.

[0020] Figure 8 The reconstruction results corresponding to the same input image and different text description information are shown in the embodiments of this disclosure.

[0021] Figure 9 This is a partial module of a three-dimensional reconstruction apparatus according to another embodiment of this disclosure.

[0022] Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should be understood that the various steps described in the method embodiments of this disclosure can be performed in sequence and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0025] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0026] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0027] It should be noted that the use of the word "a" in this disclosure is illustrative rather than restrictive, and those skilled in the art should understand that it should be understood as "one or more" unless otherwise expressly indicated in the context.

[0028] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0029] This disclosure represents objects using multi-view 3D Gaussian Splatting (3DGS) and uses diffusion models to directly generate various parameters of 3DGS, which significantly improves the rendering quality and generation speed of single-view 3D object reconstruction.

[0030] Figure 1 A flowchart of a three-dimensional reconstruction method according to embodiments of the present disclosure is provided. The three-dimensional reconstruction method of the present disclosure may include step S101, inputting a single-view image and its camera position into a trained diffusion model. In some embodiments, the single-view image is a typical two-dimensional image. In some embodiments, the camera position is the angle information of capturing the single-view image, for example, 0°, 60°, 180°, etc. In some embodiments, the trained diffusion model may include a trained convolutional neural network, etc., and is a generative model whose execution process includes a noise addition process (forward process) and a noise reduction process (backward process).

[0031] In some embodiments, the method of this disclosure may further include step S102, outputting three-dimensional Gaussian splash parameters corresponding to objects in a single-view image through a diffusion model. The diffusion model of this disclosure, after training, is capable of outputting three-dimensional Gaussian splash parameters corresponding to objects in a single-view image based on the single-view image and its camera position, thereby enabling its use for three-dimensional reconstruction of objects in a single-view image. In some embodiments, at this stage, the diffusion model receives conditional information such as noise sampled from a random distribution, the single-view image, and its fixed camera position, and obtains three-dimensional Gaussian splash parameters representing the three-dimensional object through an iterative denoising process.

[0032] In some embodiments, the method of this disclosure may further include step S103, rendering a 3D object using 3D Gaussian splashing parameters to obtain a reconstructed 3D object. Therefore, compared to existing 3D object reconstruction methods, this disclosure provides 3D predictability by outputting 3D Gaussian splashing parameters corresponding to the object in a single-view image through a diffusion model, and then using these parameters for 3D object reconstruction, thus solving the problem of 3D inconsistency. Furthermore, the reconstruction pipeline consists of a single diffusion model, eliminating the need for additional model components and improving the efficiency of the reconstruction pipeline. Additionally, regarding the reconstruction of 3D representations, the use of multi-view 3D Gaussian splashing representation enhances rendering quality and speed, enabling fast and accurate rendering of the reconstructed 3D object.

[0033] In some embodiments, inputting a single-view image and its camera position into a trained diffusion model includes: inputting the single-view image and its corresponding textual description information, and the camera position into the trained diffusion model. In addition to the input image, this disclosure incorporates control information from textual instructions (i.e., textual description information), thereby enabling the use of textual description information to assist in the reconstruction of single-view 3D objects, improving the flexibility and user controllability of 3D reconstruction.

[0034] In some embodiments, the 3D Gaussian splash parameters include information such as 3D coordinates, 3D dimensions, rotation angle, color, and opacity. In some embodiments, rendering a 3D object using the 3D Gaussian splash parameters includes rendering the 3D object using the 3D Gaussian splash parameters via a rasterization-based 3D Gaussian splashing method. This disclosure replaces the reconstructed 3D representation with a rasterization-based 3DGS, reducing the rendering cost after training.

[0035] In some embodiments, the 3D reconstruction method disclosed herein further includes: training a diffusion model to obtain a trained diffusion model before inputting a single-view image and its camera position into the trained diffusion model. Figure 2A diffusion model structure diagram according to some embodiments is shown. In some embodiments, training the diffusion model includes: acquiring multi-view images; acquiring three-dimensional Gaussian splash parameters corresponding to the multi-view images; adding randomly sampled noise to the three-dimensional Gaussian splash parameters corresponding to the multi-view images; inputting the noisy three-dimensional Gaussian splash parameters, one of the single-view images in the multi-view images and its corresponding text description information and camera position into the diffusion model; outputting the denoised three-dimensional Gaussian splash parameters through the diffusion model, and using the three-dimensional Gaussian splash parameters corresponding to the multi-view images before adding noise as the training target to perform supervised network optimization on the diffusion model to obtain the trained diffusion model. In some embodiments, the multi-view images may include any suitable number of single-view images, for example, the multi-view images may include four single-view images, such as single-view images with 0°, 90°, 180° and 270° perspectives.

[0036] In some embodiments, the object is represented in this disclosure as consisting of N three-dimensional Gaussian prototypes: in, From RGB colors Location size Rotation Quaternion Opacity It is parameterized. To simplify the parameterized representation and constraint of the generated 3D Gaussian distribution, the position x is determined by the depth. Camera intrinsic and extrinsic parameters (extrinsic parameter: R∈SO(3)) Internal Reference: ), pixel coordinates It can be deduced that:

[0037]

[0038] In some embodiments, the 3D Gaussian splash parameters corresponding to the multi-view image are 3D Gaussian splash parameters normalized to the range [0,1]. For the subsequent diffusion generation process, the 3D Gaussian splash parameters are normalized to the range [0,1]. Therefore, when the model predicts the 3D Gaussian parameters, except for the rotation quaternion r which undergoes L2 norm, all outputs are normalized to the range [0,1] using the sigmoid(·) activation function. RGB color and opacity naturally satisfy the range [0,1]. To handle the uncertain magnitude and depth of numerical ranges, this disclosure proposes the following method:

[0039] • The maximum value S of the three-dimensional Gaussian is predetermined. max and minimum value S min The final Gaussian size is determined by the initial value. It is obtained by interpolation between the maximum and minimum values:

[0040]

[0041] • Avoid directly modeling realistic depth values, since objects can be normalized to [-1, 1]. 3 Instead, it models the depth relative to the image projection plane.

[0042]

[0043] In some embodiments, obtaining the three-dimensional Gaussian splash parameters corresponding to the multi-view image includes: obtaining the three-dimensional Gaussian splash parameters corresponding to the multi-view image by reconstructing a model based on the multi-view image and the normal vector map and coordinate map corresponding to the multi-view image. For example... Figure 3 As shown, the multi-view 3D Gaussian representation is obtained from multi-view images within 0.1 seconds using a lightweight reconstruction model. Unlike previous methods that only use multi-view images as reconstruction information, this disclosure additionally uses normal maps and coordinate maps during the training phase, significantly improving the reconstruction quality of the 3D Gaussian representation. It is important to note that the additional normal maps and coordinate maps are only needed during the training phase and are not required during actual deployment and inference.

[0044] In some embodiments, the reconstruction model can employ any existing model for acquiring 3D Gaussian splash parameters, or it can be trained independently, for example, using a labeled dataset where the labels are known 3D Gaussian parameters. Typically, an encoder-decoder architecture is employed, where the encoder extracts features from multi-view images, and the decoder converts these features into the desired 3D Gaussian parameters. In this disclosure, by employing additional normal vector maps and coordinate maps, the normal vector map provides the normal direction of the surface at each pixel, which is very useful for understanding the local geometric features of the object's surface, and the coordinate map typically represents the position of each pixel in the object's global coordinate system, which helps the model understand the spatial structure of the object. Thus, the reconstruction quality of the 3D Gaussian representation is greatly improved.

[0045] like Figure 2 As shown, this disclosure uses a network structure based on an image diffusion model to directly process the three-dimensional Gaussian parameters. For the V to be generated... in The three-dimensional Gaussian parameters from each viewpoint are organized into an arrangement similar to image pixels. Where N = H × W, H and W correspond to height and width respectively, meaning that each Gaussian prototype is obtained from one pixel. The Plucker gaze vector, representing viewpoint information, is concatenated with the three-dimensional Gaussian parameters in the feature dimension. Then, the image diffusion model independently processes the Gaussian parameters under one viewpoint each time. In the attention layer, the tensor shape processed by the network is rearranged as follows: m is the dimension of the feature, which enables the network to process 3D Gaussian parameters from different perspectives.

[0046] To input a single-view image as conditional information, the features of this single-view image are concatenated with Gaussian parameters in the view dimension; that is, the true input shape of the network is... An additional dense binary mask (all 0s or all 1s) is concatenated along the feature dimension to distinguish between the image input as a condition and the Gaussian parameter input as the target of network processing.

[0047] Unlike previous single-view 3D object reconstruction methods, this disclosed method, in addition to the input image information, also supports text description information as control information to specify the generated 3D content. For example... Figure 2 As shown, the text description information of the object is processed by a text encoder (e.g., CLIP text encoder, or T5 text encoder) to obtain text feature embedding. This text feature interacts with the Gaussian parameter features in the network through a cross-attention mechanism, thereby guiding the generation process of Gaussian parameters.

[0048] In some embodiments, supervising network optimization of the diffusion model using the three-dimensional Gaussian splash parameters corresponding to the multi-view images before adding noise may include supervising network optimization of the diffusion model based on the principle of minimum Gaussian loss function using the three-dimensional Gaussian splash parameters corresponding to the multi-view images before adding noise.

[0049] Figures 4 to 6 The reconstruction effects of single-view 3D objects according to embodiments of this disclosure are shown respectively. Figure 7 and Figure 8 The reconstruction results of embodiments of this disclosure corresponding to the same input image and different textual description information are shown respectively. Thanks to the direct generation of high-quality 3D Gaussian parameters, the consistency and accuracy of the reconstructed 3D objects are greatly improved, as shown in the following figures. Figure 4 , Figure 5 and Figure 6 As shown. Furthermore, thanks to additional text control conditions, the reconstruction pipeline proposed in this disclosure offers significantly improved controllability for the user, with effects such as... Figure 7 and Figure 8As shown, different textual descriptions can be input into the same single-view image to obtain corresponding 3D reconstructed objects. Furthermore, thanks to the adoption of high-quality multi-view 3D Gaussian splashing technology, this disclosure not only improves the quality of the final rendered 3D objects but also increases rendering speed, enabling real-time rendering and interactive applications.

[0050] This disclosure utilizes a generalizable Gaussian reconstruction model and a carefully designed normalization method. During the training phase, it combines normal vector maps and coordinate maps to obtain high-quality, normalized 3D Gaussian splash parameters from multi-view images within 0.1 seconds. This constructs the dataset required for training the diffusion model, improving the efficiency of the reconstruction pipeline, providing 3D discernibility, and resolving the issue of 3D inconsistency. Furthermore, this disclosure uses textual description information as an additional control condition to guide single-view 3D reconstruction. Through different conditional guidance mechanisms, the diffusion model generating 3D Gaussian parameters simultaneously supports both image and textual description information as control conditions, significantly improving the flexibility and controllability of 3D reconstruction. Regarding 3D representation reconstruction, multi-view 3D Gaussian splash representation is employed to improve rendering quality and speed, enabling fast and accurate rendering of reconstructed 3D objects.

[0051] Embodiments of this disclosure also provide a three-dimensional reconstruction apparatus 400. Figure 9 A 3D reconstruction apparatus 400 according to some embodiments is shown. The 3D reconstruction apparatus 400 includes an input module 401, an output module 402, and a rendering module 403. In some embodiments, the input module 401 is configured to input a single-view image and its camera position into a trained diffusion model. In some embodiments, the output module 402 is configured to output 3D Gaussian splash parameters corresponding to objects in the single-view image through the diffusion model. In some embodiments, the rendering module 403 is configured to render the 3D object using the 3D Gaussian splash parameters to obtain a reconstructed 3D object.

[0052] It should be understood that the description of the three-dimensional reconstruction method also applies to the three-dimensional reconstruction device 400 described here, but for simplicity, it will not be described in detail here.

[0053] In some embodiments, inputting a single-view image and its camera position into a trained diffusion model includes: inputting the single-view image and its corresponding textual description information, and the camera position into the trained diffusion model. In some embodiments, the 3D Gaussian splash parameters include 3D coordinates, 3D dimensions, rotation angle, color, and opacity. In some embodiments, rendering a 3D object using the 3D Gaussian splash parameters includes: rendering a 3D object using the 3D Gaussian splash parameters via a rasterization-based 3D Gaussian splashing method. In some embodiments, the 3D reconstruction apparatus further includes: a training module configured to train a diffusion model before inputting a single-view image and its camera position into a trained diffusion model to obtain a trained diffusion model; wherein training the diffusion model includes: acquiring multi-view images; acquiring 3D Gaussian splash parameters corresponding to the multi-view images; adding randomly sampled noise to the 3D Gaussian splash parameters corresponding to the multi-view images; inputting the noisy 3D Gaussian splash parameters, one of the single-view images in the multi-view images and its corresponding text description information and camera position into the diffusion model; outputting the denoised 3D Gaussian splash parameters through the diffusion model, and performing supervised network optimization on the diffusion model using the 3D Gaussian splash parameters corresponding to the multi-view images before adding noise, to obtain a trained diffusion model. In some embodiments, the 3D Gaussian splash parameters corresponding to the multi-view images are 3D Gaussian splash parameters normalized to the range [0,1]. In some embodiments, acquiring the 3D Gaussian splash parameters corresponding to the multi-view images includes: acquiring the 3D Gaussian splash parameters corresponding to the multi-view images through a reconstruction model based on the multi-view images and the normal vector map and coordinate map corresponding to the multi-view images. In some embodiments, in the diffusion model, text description information is processed by a text encoder to obtain text features, and the text features interact with the three-dimensional Gaussian splashing parameter features through a cross-attention mechanism.

[0054] In addition, this disclosure also provides a terminal, including: at least one memory and at least one processor; wherein, the memory is used to store program code, and the processor is used to call the program code stored in the memory to execute the above-described three-dimensional reconstruction method.

[0055] In addition, this disclosure also provides a computer storage medium storing program code for executing the above-described three-dimensional reconstruction method.

[0056] The above description, based on embodiments and application examples, illustrates the three-dimensional reconstruction method and apparatus of this disclosure. Furthermore, this disclosure also provides a terminal and a storage medium, which are described below.

[0057] The following is for reference. Figure 10The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device or a server) 500 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0058] like Figure 10 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0059] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0060] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0061] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0062] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0063] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0064] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods of the present disclosure.

[0065] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0066] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0067] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0068] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0069] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0070] According to one or more embodiments of this disclosure, a three-dimensional reconstruction method is provided, the three-dimensional reconstruction method comprising: inputting a single-view image and its camera position into a trained diffusion model; outputting three-dimensional Gaussian splash parameters corresponding to objects in the single-view image through the diffusion model; and rendering three-dimensional objects using the three-dimensional Gaussian splash parameters to obtain reconstructed three-dimensional objects.

[0071] According to one or more embodiments of this disclosure, inputting a single-view image and its camera position into a trained diffusion model includes: inputting the single-view image and its corresponding textual description information, and the camera position into the trained diffusion model.

[0072] According to one or more embodiments of this disclosure, the three-dimensional Gaussian splash parameters include three-dimensional coordinates, three-dimensional dimensions, rotation angle, color, and opacity.

[0073] According to one or more embodiments of this disclosure, rendering a three-dimensional object using the three-dimensional Gaussian splash parameters includes: rendering a three-dimensional object using the three-dimensional Gaussian splash parameters through a rasterization-based 3D Gaussian splashing method.

[0074] According to one or more embodiments of this disclosure, the 3D reconstruction method further includes: training the diffusion model to obtain a trained diffusion model before inputting a single-view image and its camera position into the trained diffusion model; wherein training the diffusion model includes: acquiring multi-view images; acquiring 3D Gaussian splash parameters corresponding to the multi-view images; adding randomly sampled noise to the 3D Gaussian splash parameters corresponding to the multi-view images; inputting the 3D Gaussian splash parameters containing the noise, one of the single-view images in the multi-view images and its corresponding text description information and camera position into the diffusion model; outputting the denoised 3D Gaussian splash parameters through the diffusion model, and performing supervised network optimization on the diffusion model using the 3D Gaussian splash parameters corresponding to the multi-view images before adding noise, to obtain the trained diffusion model.

[0075] According to one or more embodiments of this disclosure, the three-dimensional Gaussian splash parameters corresponding to the multi-view image are three-dimensional Gaussian splash parameters normalized to the range of [0,1].

[0076] According to one or more embodiments of this disclosure, obtaining the three-dimensional Gaussian splash parameters corresponding to the multi-view image includes: obtaining the three-dimensional Gaussian splash parameters corresponding to the multi-view image by reconstructing a model based on the multi-view image and the normal vector map and coordinate map corresponding to the multi-view image.

[0077] According to one or more embodiments of this disclosure, in the diffusion model, the text description information is processed by a text encoder to obtain text features, and the text features interact with three-dimensional Gaussian splash parameter features through a cross-attention mechanism.

[0078] According to one or more embodiments of this disclosure, a three-dimensional reconstruction apparatus is provided, the three-dimensional reconstruction apparatus comprising: an input module configured to input a single-view image and its camera position into a trained diffusion model; an output module configured to output three-dimensional Gaussian splash parameters corresponding to an object in the single-view image through the diffusion model; and a rendering module configured to render the three-dimensional object using the three-dimensional Gaussian splash parameters to obtain a reconstructed three-dimensional object.

[0079] According to one or more embodiments of this disclosure, a terminal is provided, comprising: at least one memory and at least one processor; wherein the at least one memory is used to store program code, and the at least one processor is used to invoke the program code stored in the at least one memory to execute the method described in any one of the above descriptions.

[0080] According to one or more embodiments of the present disclosure, a storage medium is provided for storing program code for performing the methods described above.

[0081] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0082] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0083] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A three-dimensional reconstruction method, characterized in that, The three-dimensional reconstruction method includes: The single-view image and its camera position are input into the trained diffusion model; The diffusion model outputs three-dimensional Gaussian splash parameters corresponding to the objects in the single-view image. The three-dimensional object is rendered using the aforementioned three-dimensional Gaussian splash parameters to obtain the reconstructed three-dimensional object.

2. The three-dimensional reconstruction method according to claim 1, characterized in that, Inputting a single-view image and its camera position into a trained diffusion model includes: inputting a single-view image and its corresponding text description information, and the camera position into a trained diffusion model.

3. The three-dimensional reconstruction method according to claim 1, characterized in that, The three-dimensional Gaussian splash parameters include three-dimensional coordinates, three-dimensional dimensions, rotation angle, color, and opacity.

4. The three-dimensional reconstruction method according to claim 1, characterized in that, Rendering a 3D object using the aforementioned 3D Gaussian splash parameters includes: rendering a 3D object using the aforementioned 3D Gaussian splash parameters through a rasterization-based 3D Gaussian splash method.

5. The three-dimensional reconstruction method according to claim 1, characterized in that, Also includes: The diffusion model is trained before the single-view image and its camera position are input into the trained diffusion model to obtain the trained diffusion model. Training the diffusion model includes: Acquire multi-view images; Obtain the three-dimensional Gaussian splash parameters corresponding to the multi-view images; Randomly sampled noise is added to the three-dimensional Gaussian splash parameters corresponding to the multi-view images; The three-dimensional Gaussian splash parameters containing the noise, one of the single-view images in the multi-view images and its corresponding text description information, and the camera position are input into the diffusion model; The diffusion model outputs denoised 3D Gaussian splash parameters, and the diffusion model is optimized by a supervised network using the 3D Gaussian splash parameters corresponding to the multi-view image before adding noise, to obtain a trained diffusion model.

6. The three-dimensional reconstruction method according to claim 5, characterized in that, The three-dimensional Gaussian splash parameters corresponding to the multi-view image are three-dimensional Gaussian splash parameters normalized to the range of [0,1].

7. The three-dimensional reconstruction method according to claim 5, characterized in that, Obtaining the three-dimensional Gaussian splash parameters corresponding to the multi-view image includes: obtaining the three-dimensional Gaussian splash parameters corresponding to the multi-view image by reconstructing a model based on the multi-view image and the normal vector map and coordinate map corresponding to the multi-view image.

8. The three-dimensional reconstruction method according to claim 5, characterized in that, In the diffusion model, the text description information is processed by a text encoder to obtain text features, and the text features interact with the three-dimensional Gaussian splashing parameter features through a cross-attention mechanism.

9. A three-dimensional reconstruction device, characterized in that, The three-dimensional reconstruction device includes: The input module is configured to input a single-view image and its camera position into a trained diffusion model; The output module is configured to output three-dimensional Gaussian splash parameters corresponding to the objects in the single-view image through the diffusion model; The rendering module is configured to render a 3D object using the three-dimensional Gaussian splash parameters to obtain a reconstructed 3D object.

10. A terminal, comprising: At least one memory and at least one processor; The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the three-dimensional reconstruction method according to any one of claims 1 to 8.

11. A storage medium for storing program code for executing the three-dimensional reconstruction method according to any one of claims 1 to 8.

Images