Three-dimensional model reconstruction method and device

Abstract

The invention discloses a three-dimensional model reconstruction method and device, and the method comprises the steps: obtaining an input image, carrying out the spatial calculation processing of the input image, obtaining a three-dimensional model containing a structure and texture, and separating a structure reconstructed based on the spatial calculation; acquiring an input sampling pose, and extracting an RGB graph and a normal graph based on the sampling pose; two map channel sets independently representing the roughness attribute and the metal degree attribute respectively and a diffuse reflection map representing the diffuse reflection optical characteristic of the surface of the object in the PBR material are obtained; obtaining a text prompt word and a reference graph, and integrating two types of coding results to form a subject condition; and inputting the RGB graph, the normal graph, the map channel set, the diffuse reflection map and the theme condition into a pre-trained artificial neural network, and outputting a fused three-dimensional model which retains a high-precision structure and has a specific theme texture. The stylized textures of various target themes can be generated.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, specifically to a method and apparatus for three-dimensional model reconstruction. Background Technology

[0002] Generative 3D large models are usually generated by using a 2D-lifting-3D approach based on a StableDiffusion model that has been trained on 2D images. Because the model is essentially learned from 2D images, this method has weak spatial perception and the resulting 3D model has insufficient spatial representation. Another method is to directly learn the spatial features of the 3D model. This method generates a model with stronger spatial perception, but it is limited by insufficient training data for the 3D model and the problem of the large amount of target data that the 3D model needs to learn. Summary of the Invention

[0003] The main objective of this invention is to provide a three-dimensional model reconstruction method and apparatus to address the shortcomings of related technologies.

[0004] To achieve the above objectives, according to a first aspect of the present invention, a three-dimensional model reconstruction method is provided, comprising: acquiring an input image; performing spatial computation processing on the input image to obtain a three-dimensional model containing structure and texture, while separating the structure reconstructed based on spatial computation; acquiring an input sampling pose; extracting an RGB image and a normal image based on the sampling pose in the three-dimensional model containing structure and texture and the structure; acquiring two texture channel sets x_rm that independently represent roughness and metallicity attributes in a PBR material, and a diffuse reflection map x_d that represents the diffuse optical properties of an object surface; acquiring a text prompt and a reference image; inputting the text prompt into a text encoder for encoding, and inputting the reference image into a visual encoder for encoding, and integrating the two types of encoding results to form a theme condition c; inputting the RGB image, normal image, texture channel set x_rm, diffuse reflection map x_d, and theme condition c into a pre-trained artificial neural network, and outputting a fused three-dimensional model that retains high-precision structure and has a specific theme texture.

[0005] Optionally, extracting the RGB image and normal map based on the sampled pose includes: based on the camera sampled pose, and ensuring that the normal map and the RGB image viewpoint positions are completely corresponding, extracting the normal map x that corresponds one-to-one with the pose. n RGB image x_rgb.

[0006] Optionally, the input is fused and predicted by the pre-trained neural network to determine the PBR material texture data that matches the specific theme condition c; a texture coordinate mapping relationship is established based on the camera sampling pose, and the material texture data is accurately adapted and fused with the spatial calculation and reconstruction structure to generate a fused 3D model that retains the high-precision structure and has the specific theme texture.

[0007] Optionally, establishing a texture coordinate mapping relationship based on camera sampling pose includes: calculating the correspondence between pixel coordinates and three-dimensional structure coordinates in PBR material texture data through camera intrinsic parameter matrix and sampling pose, so as to achieve point-by-point fitting of PBR material texture data and spatially reconstructed structure.

[0008] Optionally, spatial computation processing of the input image includes: performing pose calculation on the input image to determine pose information; generating a sparse point cloud based on the pose information, and after completion and optimization by a dense reconstruction algorithm, outputting the spatially reconstructed 3D model and the structure based on the spatially reconstructed structure.

[0009] Optionally, when training the artificial neural network, the training method includes: acquiring multiple input image samples; performing spatial computation processing on each input image sample to obtain a spatially reconstructed 3D model containing high-precision structure and initial texture, and an independent spatially reconstructed structure; simultaneously collecting multiple sets of camera sampling pose (R,t) samples and subject condition samples, where R represents the camera rotation matrix, t represents the camera displacement, and the subject condition samples include text prompt word samples and reference image samples; for each set of spatially reconstructed 3D model and spatially reconstructed structure, based on the corresponding camera sampling pose (R,t), and ensuring that the normal map and RGB image viewpoint positions are completely corresponding, extracting a normal map training sample x corresponding one-to-one with that pose. n The training samples are RGB image x_rgb, and simultaneously, training samples x_rm representing roughness and metallicity properties of PBR materials and diffuse reflection map training samples x_d representing diffuse reflection optical properties are obtained. The text prompt word samples and reference image samples in the topic conditional samples are encoded to obtain topic conditional training samples c. A StableDiffusion artificial neural network is used as the basic network model, and four SDS-type loss functions are constructed corresponding to x_rgb and x_d respectively. n The input branches are x_rgb, x_rm, and x_d, where the overall loss function of the four SDS-type loss functions is: L=E[L_SDSn+L_SDS_rgb+L_SDS_rm+L_SDS_d], where E represents the expectation operation; the training samples are input into the StableDiffusion artificial neural network by concatenation, and the network model is iteratively trained based on the overall loss function, and the network parameters are optimized by minimizing the loss value.

[0010] Optionally, ,in, This indicates that the network model uses normal map training samples. The noise prediction results express Corresponding real noise; ,in, This represents the noise prediction result of the network model on the training samples of the RGB image. This represents the actual noise corresponding to the training samples of the RGB image; , where ε_θ(x_rm;c,p) represents the noise prediction result of the network model for the training sample x_rm of the texture channel, and ε_rm represents the real noise corresponding to x_rm; ,in This represents the noise prediction results of the network model on the training samples of the diffuse map. This represents the real noise corresponding to the training samples of the diffuse map, where c represents the topic condition, which includes two parts: text prompts and reference images, and p represents the sampling pose of the camera.

[0011] According to a second aspect of the present invention, a three-dimensional model reconstruction apparatus is provided, comprising: a spatial computing processing unit for acquiring an input image, performing spatial computing processing on the input image to obtain a three-dimensional model containing structure and texture, and simultaneously separating the structure reconstructed based on spatial computing; an information extraction unit for acquiring an input sampling pose, extracting an RGB image and a normal image based on the three-dimensional model containing structure and texture and the structure; and acquiring two texture channel sets x_rm that independently represent roughness and metallicity attributes in a PBR material, and a diffuse reflection map x_d that represents the diffuse reflection optical properties of an object surface; a theme condition generation unit for acquiring text prompts and reference images, inputting the text prompts into a text encoder for encoding, inputting the reference images into a visual encoder for encoding, and integrating the two types of encoding results to form a theme condition c; and a reconstruction unit for inputting the RGB image, normal image, texture channel set x_rm, diffuse reflection map x_d, and theme condition c into a pre-trained artificial neural network, and outputting a fused three-dimensional model that retains high-precision structure and possesses a specific theme texture.

[0012] According to a third aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing the computer to perform the method described in any one of the first aspects.

[0013] According to a fourth aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the method described in any implementation of the first aspect.

[0014] This embodiment presents a 3D model reconstruction method and apparatus. The method includes: acquiring an input image; performing spatial computation processing on the input image to obtain a 3D model containing structure and texture, while separating the structure reconstructed based on spatial computation; acquiring the input sampling pose; extracting an RGB image and a normal image based on the 3D model containing structure and texture, and the structure; acquiring two texture channel sets x_rm that independently represent roughness and metallicity attributes in a PBR material, and a diffuse reflection map x_d that represents the diffuse optical properties of the object surface; acquiring text prompts and a reference image; inputting the text prompts into a text encoder for encoding, and inputting the reference image into a visual encoder for encoding; integrating the two types of encoding results to form a theme condition c; and inputting the RGB image, normal image, texture channel set x_rm, diffuse reflection map x_d, and theme condition c into a pre-trained artificial neural network to output a fused 3D model that retains high-precision structure and possesses a specific theme texture. By integrating spatial computation with generative large models, we can achieve both the high precision of spatial computation in reconstructing model structures and fully leverage the imaginative capabilities of generative large models to generate stylized textures for various target themes. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the three-dimensional model reconstruction method according to an embodiment of the present invention; Figures 2-3 This is an application diagram of the three-dimensional model reconstruction method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] According to embodiments of the present invention, a three-dimensional model reconstruction method is provided, such as... Figure 1 As shown, steps 101 to 103 are included below: Step 101: Obtain the input image, perform spatial calculation processing on the input image to obtain a three-dimensional model containing structure and texture, and separate the structure reconstructed based on spatial calculation.

[0021] As an optional implementation of this embodiment, spatial computation processing of the input image includes: performing pose calculation on the input image to determine pose information; generating a sparse point cloud based on the pose information, and after completion and optimization by a dense reconstruction algorithm, outputting the spatially reconstructed 3D model and the structure based on the spatially reconstructed structure.

[0022] In this step, the input image is acquired, spatial computation processing is performed on the input image, high-precision image pose information is restored for the successfully registered image and an overall sparse point cloud is formed, and after dense reconstruction, a spatial computation reconstructed 3D model containing high-precision structure and initial texture is obtained, and the spatial computation reconstructed structure is separated.

[0023] Step 102: Obtain the input sampling pose. Based on the 3D model containing structure and texture, and the structure, extract the RGB image and normal map based on the sampling pose. Also, obtain the two texture channel sets x_rm that independently represent roughness and metallicity attributes in the PBR material, and the diffuse reflection map x_d that represents the diffuse reflection optical properties of the object surface.

[0024] As an optional implementation of this embodiment, extracting the RGB image and normal map based on the sampled pose includes: based on the camera sampled pose, while ensuring that the normal map and the RGB image viewpoint positions are completely corresponding, extracting the normal map x that corresponds one-to-one with the pose. n RGB image x_rgb.

[0025] In this step, based on the spatially reconstructed 3D model and its structure, the camera sampling pose (R,t) is determined according to the target viewpoint requirements, where R represents the camera's rotation matrix and t represents the camera's displacement. (Reference) Figure 2 For the input image, spatial calculation is first performed. For successfully registered images, high-precision image pose information is restored and an overall sparse point cloud is formed. Based on the output of this stage, through dense reconstruction and other processes, a spatially calculated 3D model (3D model structure and texture) is finally obtained. The 3D model structure obtained here has high precision and good 3D model structure characteristics.

[0026] refer to Figure 3 Based on the model and structure reconstructed by spatial computing, different camera sampling poses (R,t, where R represents the camera's rotation matrix and t represents the camera's displacement) are input. Through the RGB image and normal map extraction module, the normal map of a specific sampling camera pose is extracted while ensuring that the viewpoint positions of the normal map and the RGB image are completely corresponding. and RGB image Furthermore, obtain the channels representing the roughness and metallicity of the two material maps in the PBR material. ; indicates the diffuse map in the PRB material. .

[0027] Step 103: Obtain the text prompt and reference image. Input the text prompt into the text encoder for encoding, and input the reference image into the visual encoder for encoding. Integrate the two types of encoding results to form the topic condition c.

[0028] In this step, refer to Figure 3 The text prompts are input into the CLIP text encoder for encoding, and the reference image is input into the CLIP visual encoder and encoded in conjunction with the IP-Adaptor. The two types of encoding results are then integrated to form the topic condition c.

[0029] Step 104: Input the RGB image, normal map, texture channel set x_rm, diffuse texture x_d, and theme condition c into a pre-trained artificial neural network, and output a fused 3D model that retains high-precision structure and has specific theme texture.

[0030] In this step, refer to Figure 3 The above three types of information are used as input: first, x n The three types of information—x_rgb, x_rm, and x_d—are used to constrain the network model to learn feature mapping relationships. First, there are the material and image features composed of x_rgb, x_rm, and x_d. Second, there are the style-guided features corresponding to the theme condition c. Third, there are the spatial viewpoint features corresponding to the camera sampling pose sample p. These three types of information collaboratively constrain the network model to learn feature mapping relationships. n The material and image features, consisting of x_rgb, x_rm, and x_d, are used as inputs, corresponding to the four branches, and are concatenated together and input into the StableDiffusion artificial neural network.

[0031] For example, a pre-trained StableDiffusion artificial neural network is invoked to process the normal map x. n The RGB image (x_rgb), roughness and metallicity material texture channels (x_rm), and diffuse texture (x_d) are input into the neural network through a splicing method, along with the theme condition (c) and camera sampling pose (p). The pre-trained neural network performs feature fusion and noise prediction on the input information, directly outputting PBR material texture data that matches the specific theme. This material texture data is then adapted and fused with the spatially reconstructed structure to generate a fused 3D model that retains high-precision structure and possesses the texture of the specific theme.

[0032] As an optional implementation of this embodiment, the input is collaboratively fused and noise is predicted by the pre-trained neural network to determine the PBR material texture data that matches the specific theme condition c; a texture coordinate mapping relationship is established based on the camera sampling pose, and the material texture data is accurately adapted and fused with the spatial calculation and reconstruction structure to generate a fused 3D model that retains high-precision structure and has specific theme texture.

[0033] In this optional implementation, the input information is fused and noise is predicted by the pre-trained neural network, and PBR material texture data matching a specific theme is directly output. The material texture data is then adapted and fused with the structure reconstructed by spatial calculation to generate a fused 3D model that retains high-precision structure and has a specific theme texture.

[0034] As an optional implementation method in this embodiment, establishing a texture coordinate mapping relationship based on camera sampling pose includes: calculating the correspondence between pixel coordinates and three-dimensional structure coordinates in PBR material texture data through camera intrinsic parameter matrix and sampling pose, so as to achieve point-by-point fitting of PBR material texture data and spatially reconstructed structure.

[0035] In this optional implementation, the camera intrinsic parameter matrix (such as focal length, principal point coordinates, etc.) and the sampling pose (R,t) constitute a "pixel-3D mapping bridge": by projecting the 3D coordinates (X,Y,Z) of the spatially calculated reconstructed structure into the pixel coordinates (u,v) of the PBR material texture data, it is ensured that each pixel of the texture data can accurately correspond to a certain vertex / surface position of the 3D structure, and finally point-by-point bonding is achieved.

[0036] As an optional implementation of this embodiment, the training method for training the artificial neural network includes: Multiple input image samples are acquired, and spatial computation processing is performed on each input image sample to obtain a spatially reconstructed 3D model containing high-precision structure and initial texture, and an independent spatially reconstructed structure. At the same time, multiple sets of camera sampling pose (R,t) samples and subject condition samples are collected, where R represents the camera rotation matrix, t represents the camera displacement, and the subject condition samples include text prompt word samples and reference image samples. For each set of spatially reconstructed 3D models and spatially reconstructed structures, based on the corresponding camera sampling pose (R,t), while ensuring that the viewpoint positions of the normal map and the RGB map are completely corresponding, training samples of the normal map and RGB map corresponding one-to-one with the pose are extracted. At the same time, training samples of the texture channels representing roughness and metallicity properties and training samples of the diffuse reflection texture representing diffuse optical properties in the PBR material are obtained. The text prompt word samples and reference map samples in the theme condition samples are encoded to obtain theme condition training sample c. Using a StableDiffusion artificial neural network as the base network model, four SDS-type loss functions were constructed, each corresponding to x. n The input branches of x_rgb, x_rm, and x_d are given, wherein the overall loss function of the four SDS-type loss functions is L=E[L_SDSn+L_SDS_rgb+L_SDS_rm+L_SDS_d], where E represents the expectation operation; The training samples are input into the StableDiffusion artificial neural network by concatenation, and the network model is iteratively trained based on the overall loss function. The network parameters are optimized by minimizing the loss value.

[0037] As an optional implementation method in this embodiment, ,in, This indicates that the network model uses normal map training samples. The noise prediction results express Corresponding real noise; ,in, This represents the noise prediction result of the network model on the training samples of the RGB image. This represents the actual noise corresponding to the training samples of the RGB image; , where ε_θ(x_rm;c,p) represents the noise prediction result of the network model for the training sample x_rm of the texture channel, and ε_rm represents the real noise corresponding to x_rm; ,in This represents the noise prediction results of the network model on the training samples of the diffuse map. denoted as the real noise corresponding to the training samples of the diffuse map; where c represents the topic condition, which includes two parts: text prompts and reference images, and p represents the sampling pose of the camera.

[0038] In this optional implementation, during the training phase, the generative large model is enabled to construct specific theme PBR material textures based on spatial computation reconstruction models by learning to predict noise.

[0039] This embodiment incorporates theme-based constraints. When predicting noise, the model is input with text prompts and a theme condition c encoded from a reference image. This means that the model does not perform indiscriminate noise reduction, but rather denoises according to the requirements of theme c. For example, if the theme is cyberpunk, the model will learn to retain texture features related to cyberpunk while removing noise interference that does not conform to the style, ultimately generating textures that conform to the theme.

[0040] Incorporating spatial constraints (camera sampling pose p): When predicting noise, the camera sampling pose p (rotation matrix R + displacement t) is also input. This allows the model to know from which viewpoint the texture was generated, ensuring that the spatial dimensions (such as contour, surface orientation, and texture position) of the texture reconstructed after noise prediction accurately match the high-precision structure reconstructed by spatial computation. This avoids texture-structure misalignment (such as the convex position of the texture not matching the convex position of the 3D structure), solving the pain point of weak spatial awareness in existing 2D-lifting-3D generation models.

[0041] If the noise predicted by the model differs significantly from the actual noise (high loss value), it indicates that the model has not yet mastered the characteristics of this type of data (such as the spatial orientation characteristics of the normal map, the roughness properties of the PBR material). Through the backpropagation algorithm, the model will adjust the weight parameters based on the loss value until the noise prediction result is infinitely close to the actual noise (loss convergence). This process forces the model to accurately learn all four types of input data simultaneously (x... n Features of x_rgb, x_rm, x_d are used to ensure that the generated PBR material textures satisfy the following: the viewpoint of the normal map and the RGB map are completely corresponding (spatial consistency); the material properties of roughness, metallicity, and diffuse reflection map are matched (material consistency); and the geometric constraints of the texture and the spatial calculation and reconstruction structure are matched (structural consistency).

[0042] When the model learns to generate textures by predicting noise, it does not need to rely on massive amounts of 3D training data because its generation basis is a high-precision structure reconstructed by spatial computation (with clear 3D geometric constraints). At the same time, the spatial constraints of the camera pose p are incorporated into the noise prediction process, giving the generated textures spatial awareness (avoiding the pain point of "insufficient spatial representation of 2D-lifting-3D path").

[0043] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0044] According to an embodiment of the present invention, a three-dimensional model reconstruction apparatus is also provided, comprising: a spatial computing processing unit for acquiring an input image, performing spatial computing processing on the input image to obtain a three-dimensional model containing structure and texture, and simultaneously separating the structure reconstructed based on spatial computing; an information extraction unit for acquiring the input sampling pose, extracting an RGB image and a normal image based on the three-dimensional model containing structure and texture and the structure; and acquiring two texture channel sets x_rm that independently represent roughness and metallicity attributes in a PBR material, and a diffuse reflection map x_d that represents the diffuse reflection optical properties of the object surface; a theme condition generation unit for acquiring text prompts and reference images, inputting the text prompts into a text encoder for encoding, inputting the reference images into a visual encoder for encoding, and integrating the two types of encoding results to form a theme condition c; and a reconstruction unit for inputting the RGB image, normal image, texture channel set x_rm, diffuse reflection map x_d, and theme condition c into a pre-trained artificial neural network, and outputting a fused three-dimensional model that retains high-precision structure and has a specific theme texture.

[0045] According to embodiments of the present invention, the present invention also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the methods described in any of the above embodiments.

[0046] According to embodiments of the present invention, the present invention also provides a readable storage medium storing computer instructions that enable a computer to perform the methods described in any of the above embodiments when executed.

[0047] According to embodiments of the present invention, the present invention also provides a computer program product that, when executed by a processor, can implement the methods described in any of the above embodiments.

[0048] Figure 4 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0049] like Figure 4 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the electronic device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0050] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0051] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as the object matching method. For example, in some embodiments, the object matching method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the methods described above may be performed.

[0052] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0053] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0054] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

Claims

1. A method for reconstructing a three-dimensional model, characterized in that, include: The input image is acquired, and spatial computation processing is performed on the input image to obtain a three-dimensional model containing structure and texture, while the structure reconstructed based on spatial computation is separated. The input sampling pose is obtained, and based on the 3D model containing structure and texture, and the structure, the RGB image and normal map are extracted from the sampling pose. And obtain two texture channel sets x_rm in the PBR material that independently represent roughness and metallicity attributes, and diffuse reflection map x_d that represents the diffuse reflection optical properties of the object surface; Obtain text prompts and reference images. Input the text prompts into a text encoder for encoding. Input the reference images into a visual encoder for encoding. Integrate the two types of encoding results to form the topic condition c. The RGB image, normal map, texture channel set x_rm, diffuse texture x_d, and theme condition c are input into a pre-trained artificial neural network, and the output is a fused 3D model that retains high-precision structure and has specific theme texture.

2. The three-dimensional model reconstruction method according to claim 1, characterized in that, Extracting the RGB image and normal map based on the sampled pose includes: Based on the camera sampling pose, while ensuring that the normal map and the RGB map view positions are completely corresponding, the normal map x corresponding to the pose is extracted. n RGB image x_rgb.

3. The three-dimensional model reconstruction method according to claim 1, characterized in that, The pre-trained neural network is used to collaboratively fuse and predict the input to determine the PBR material texture data that matches the specific theme condition c. Based on the camera sampling pose, a texture coordinate mapping relationship is established, and the material texture data is accurately adapted and fused with the spatial calculation and reconstruction structure to generate a fused 3D model that retains high-precision structure and has specific theme texture.

4. The three-dimensional model reconstruction method according to claim 3, characterized in that, Establishing texture coordinate mapping relationships based on camera sampling pose includes: By calculating the correspondence between pixel coordinates and 3D structure coordinates in PBR material texture data using camera intrinsic parameter matrix and sampling pose, point-to-point fitting of PBR material texture data and spatially reconstructed structure is achieved.

5. The three-dimensional model reconstruction method according to claim 1, characterized in that, Spatial computation processing of the input image includes: Perform pose calculation on the input image to determine pose information; Based on the pose information, a sparse point cloud is generated. After being completed and optimized by a dense reconstruction algorithm, the spatially reconstructed 3D model and the structure based on the spatially reconstructed model are output.

6. The three-dimensional model reconstruction method according to claim 1, characterized in that, The training method for training the artificial neural network includes: Multiple input image samples are acquired, and spatial computation processing is performed on each input image sample to obtain a spatially reconstructed 3D model containing high-precision structure and initial texture, and an independent spatially reconstructed structure. At the same time, multiple sets of camera sampling pose (R,t) samples and subject condition samples are collected, where R represents the camera rotation matrix, t represents the camera displacement, and the subject condition samples include text prompt word samples and reference image samples. For each set of spatially reconstructed 3D models and structures, based on the corresponding camera sampling pose (R,t), while ensuring that the normal map and RGB map viewpoint positions are completely corresponding, training samples x of the normal map corresponding one-to-one with that pose are extracted. n The training samples are RGB image training samples x_rgb, and the training samples are also obtained from the PBR material texture channels x_rm representing roughness and metallicity properties, and diffuse reflection texture training samples x_d representing diffuse reflection optical properties. The text prompt word samples and reference image samples in the theme condition samples are encoded to obtain theme condition training samples c. Using a StableDiffusion artificial neural network as the base network model, four SDS-type loss functions were constructed, each corresponding to x. n The input branches of x_rgb, x_rm, and x_d, wherein the overall loss function of the four SDS type loss functions is: L = E[L_SDSn + L_SDS_rgb + L_SDS_rm + L_SDS_d], where E represents the expected operation; The training samples are input into the StableDiffusion artificial neural network by concatenation, and the network model is iteratively trained based on the overall loss function. The network parameters are optimized by minimizing the loss value.

7. The three-dimensional model reconstruction method according to claim 5, characterized in that, ,in, This indicates that the network model uses normal map training samples. The noise prediction results express Corresponding real noise; ,in, This represents the noise prediction result of the network model on the training samples of the RGB image. This represents the actual noise corresponding to the training samples of the RGB image; , where ε_θ(x_rm;c,p) represents the noise prediction result of the network model for the training sample x_rm of the texture channel, and ε_rm represents the real noise corresponding to x_rm; ,in This represents the noise prediction results of the network model on the training samples of the diffuse map. This represents the actual noise corresponding to the training samples of the diffuse texture; Where 'c' represents the topic conditions, which include text prompts and reference images, and 'p' represents the camera's sampling pose.

8. A three-dimensional model reconstruction device, characterized in that, include: The spatial computing processing unit is used to acquire the input image, perform spatial computing processing on the input image to obtain a three-dimensional model containing structure and texture, and separate the structure reconstructed based on spatial computing. The information extraction unit acquires the input sampling pose, and extracts RGB and normal maps based on the sampling pose in a 3D model containing structure and texture, and the structure; and acquires two texture channel sets x_rm that independently represent roughness and metallicity attributes in the PBR material, and a diffuse reflection map x_d that represents the diffuse reflection optical properties of the object surface. The topic condition generation unit is used to obtain text prompts and reference images. The text prompts are input into the text encoder for encoding, and the reference images are input into the visual encoder for encoding. The two types of encoding results are integrated to form topic condition c. The reconstruction unit is used to input RGB image, normal map, texture channel set x_rm, diffuse texture x_d, and theme condition c into a pre-trained artificial neural network, and output a fused 3D model that retains high-precision structure and has specific theme texture.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-7.

10. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the method according to any one of claims 1-7.

Images