A single image three-dimensional head reconstruction method and system based on guided diffusion model

Abstract

The application discloses a single-image three-dimensional head reconstruction method and system based on a guided diffusion model, and the method comprises the following steps: obtaining a head portrait from a single image to be reconstructed; reconstructing a three-dimensional head model according to the head portrait; weakly projecting and mapping the three-dimensional head model to the head portrait under a predicted camera matrix, and obtaining a three-dimensional head model with front texture by using pixel information of the head portrait; performing cylindrical UV unfolding on the three-dimensional head model with the front texture to obtain a two-dimensional UV texture map and record a UV mapping relationship; inputting the two-dimensional UV texture map into a guided diffusion network model to predict the texture of an invisible area and obtain a complete two-dimensional UV texture map; and pasting the complete two-dimensional UV texture map back to the three-dimensional head model. The application can reconstruct a high-fidelity, artifact-free and realistic three-dimensional head model with rich texture details from a single portrait of an individual of different ages, genders, races and facial expressions.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and computer graphics, and more specifically, relates to a method and system for single-image 3D head reconstruction based on a guided diffusion model. Background Technology

[0002] Single-image 3D head reconstruction refers to the automatic generation of a 3D head model represented by a mesh from a single portrait image using algorithms. With the rise of the metaverse, 3D head reconstruction has shown great application potential. In addition, 3D head reconstruction has broad application prospects in film and television production, game entertainment, and intelligent education. Currently, there are roughly two ways to obtain 3D head models: (1) manual modeling or reconstruction of 3D head models based on high-precision 3D scanners and stereo vision systems; (2) reconstruction of 3D head models directly from portrait images based on deep learning technology. The first method can obtain high-precision 3D head models, but the equipment used (such as laser ranging 3D scanners) is expensive, has low operability, and is difficult to popularize; and during the scanning process, the individual is prone to shaking, which generates noise and leads to incomplete 3D head models. The second method, based on deep learning, can directly reconstruct 3D head models from single portrait images, which can save costs to the greatest extent and greatly improve the convenience of operation.

[0003] In recent years, researchers in this field have proposed a series of deep learning-based methods to learn prior knowledge from data. However, most studies have focused on the reconstruction of geometric structures, with little attention paid to the reconstruction of the entire head texture. Although existing techniques have proposed methods for estimating the 3D shape and texture details of the head from unrestricted inputs (such as wild images), many problems remain. An ideal reconstruction should be high-fidelity and artifact-free; more specifically, it should faithfully convey the human head pose and texture details in the image. The generated 3D head model should be a complete head without gaps or missing parts, and it should not be a non-human head shape or other artifacts.

[0004] Existing texture reconstruction methods can be mainly divided into two categories: one is texture reconstruction based on 3D meshes, and the other is texture reconstruction based on novel perspective synthesis. Early researchers proposed reconstructing 3D head models by combining UV (two-dimensional texture coordinates corresponding to the vertex information of the geometry) texture mapping with 3DMM (3D Deformable Model). However, 3DMM itself has limitations; it can only perform relatively coarse geometric estimations and uses a uniform template. Even when complete UV textures can be recovered, its fixed template causes the generated results to differ significantly from real human heads, with substantial discrepancies in reconstruction results, and the hair portion is not adequately considered during reconstruction. In addition, there is a method that reconstructs meshes using input image features and uses a diffusion model to repair incomplete textures. Although current diffusion models have shown some effectiveness in repairing UV maps, existing research has not yet achieved ideal results, and it does not consider the invisible areas on the back side during UV repair. Another study mainly used image synthesis methods to generate heads from various perspectives, but only a small portion of them could generate 360-degree views, and there were also many problems such as low resolution, incomplete images, and the generated results not matching the real situation.

[0005] In summary, although single-image 3D head texture reconstruction methods have made some progress in recent years, the following problems still exist: (1) The reconstructed 3D model has coarse texture and low fidelity. Although the texture of the 3D head model reconstructed by some methods looks very realistic, it is difficult to match the identity of the person in the input image. (2) It is unable to reconstruct the texture of the invisible areas of the input image or has less detail and poor accuracy. Although some methods can recover the texture of facial expressions, they are weak in expressing the texture of the back part other than the facial area. (3) The algorithm has poor generalization ability and is difficult to adapt to changes in age, race, gender, etc. For example, when a model trained on an adult dataset reconstructs a child's portrait, the reconstructed result still looks like an adult. (4) The generated result is flat, distorted, and does not resemble the person. Although the new perspective synthesis method can reconstruct a 360-degree perspective of a real person, the resolution is low, and the generated result is very different from the person in the input image. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a single-image 3D head reconstruction method and system based on a guided diffusion model. This method can reconstruct a high-fidelity, artifact-free, and texture-rich 3D head model from a single portrait image of an individual with different ages, genders, ethnicities, and facial expressions.

[0007] To achieve the above objectives, according to one aspect of the present invention, a single-image 3D head reconstruction method based on a guided diffusion model is provided, comprising the steps of:

[0008] The single image to be reconstructed is preprocessed to obtain the head portrait;

[0009] A three-dimensional head model is reconstructed from the head image, and the reconstructed three-dimensional head model is optimized to restore smooth surface details;

[0010] The 3D head model with restored surface smooth details is weakly projected onto the head image under the predicted camera matrix, and the pixel information of the head image is used as the texture information of the corresponding 3D points to obtain a 3D head model with frontal texture.

[0011] The 3D head model with frontal texture is subjected to cylindrical UV unwrapping to obtain a 2D UV texture map and the UV mapping relationship is recorded.

[0012] The two-dimensional UV texture map is input into a trained guided diffusion network model to conditionally predict the texture of invisible areas in the two-dimensional UV texture map and repair it to obtain a complete two-dimensional UV texture map.

[0013] Using the UV mapping relationship, the complete two-dimensional UV texture map is back-painted back onto the three-dimensional head model with the frontal texture.

[0014] Preferably, before pasting the complete two-dimensional UV texture map back onto the three-dimensional head model with the frontal texture, the method further includes the step of inputting the complete UV texture map into the super-resolution network model to improve the resolution of the complete UV texture map.

[0015] Preferably, the step of weakly projecting the restored 3D head model onto the head image under the predicted camera matrix, and using the pixel information of the head image as the texture information of the corresponding 3D points to obtain a 3D head model with frontal texture includes the following steps:

[0016] Interpolation processing is performed on the 3D head model after restoring the surface smooth details. The coordinates of the 3D points of the interpolated 3D head model are denoted as V(X,Y,Z). At the same time, the 3D points of the 3D head model with a value greater than 0.5 on the z-axis are taken as the 3D points of the frontal texture to be determined, and the remaining 3D points are taken as the invisible area of ​​the texture to be predicted.

[0017] Under the predicted camera matrix, the three-dimensional points V(X,Y,Z) in the three-dimensional space are mapped onto the head image through weak projection mapping transformation, so as to obtain the two-dimensional pixel coordinate point v(x,y) corresponding to each three-dimensional point V(X,Y,Z).

[0018] The pixel information of the two-dimensional pixel coordinate point v(x,y) of the head image is used as the texture information of the corresponding three-dimensional point V(X,Y,Z) to obtain a three-dimensional head model with frontal texture.

[0019] Preferably, the calculation formula for weak projection mapping transformation is:

[0020]

[0021] Where f is the camera focal length.

[0022] Preferably, the step of using the pixel information of the two-dimensional pixel coordinate point v(x,y) of the head image as the texture information of the corresponding three-dimensional point V(X,Y,Z) includes the following steps:

[0023] Extract the neighborhood information of the two-dimensional pixel coordinate point v(x,y) of the head image, and determine the color of the two-dimensional pixel coordinate point v(x,y) based on the neighborhood information, denoted as C. x,y ;

[0024] The color C of the two-dimensional pixel coordinate point v(x,y) x,y This serves as the texture information for the corresponding 3D point V(X,Y,Z).

[0025] Preferably, the cylindrical UV unwrapping of the 3D head model with frontal texture includes the following steps:

[0026] The 3D head model with frontal texture is subjected to cylindrical UV unwrapping. The corresponding point in UV space after the 3D point V(X,Y,Z) is subjected to cylindrical UV unwrapping is denoted as T(u,v).

[0027] The colors of each triangular facet of the 3D head model with a frontal texture are rendered onto the corresponding UV color regions in the UV space using diffuse reflection. The calculation formula is as follows:

[0028]

[0029] Where R(·) represents diffuse rendering, Area(u 1...n ,v 1...n Area(V) represents a color region in UV space bounded by multiple uv coordinates. 1...n () represents the color area of ​​a triangular facet in a 3D head model with a frontal texture. This represents the two-dimensional UV texture map. This represents the frontal texture of a 3D head model.

[0030] Preferably, the UV mapping relationship is as follows:

[0031]

[0032] Where ρ is the distance from the 3D point V(X,Y,Z) to the z-axis, θ is the angle from the positive x-axis direction to the projection of the point (X,Y) onto the xy-plane, and z... min It is the minimum value of a 3D point in the 3D head model along the z-axis. max It is the maximum value of the three-dimensional points of the three-dimensional head model along the z-axis.

[0033] Preferably, the training of the guided diffusion network model includes the following steps:

[0034] For the 3D head model samples, a 2D UV texture map is obtained through UV mapping and exposed at different ratios to form a training image sample set. At the same time, the mask of the invisible area corresponding to each training image is obtained.

[0035] The diffusion network model is trained using a set of training image samples.

[0036] Preferably, the loss function of the diffusion network model is:

[0037]

[0038] Where x is the training image, P is the step size set during denoising, and γ represents the current noise level. This represents the added noise predicted by the diffusion network model based on x and γ, where ε refers to the noise added at each time step. Indicates a noisy image The loss Z obtained by fitting the original training image x after denoising at each time step ε,γ The noise at the current time step of the prediction is represented by ||.|| P This indicates that the noise reduction process continues from step 1 to step P. T This indicates the matrix transpose.

[0039] According to another aspect of the present invention, a single-image 3D head reconstruction system based on a guided diffusion model is provided, comprising:

[0040] The preprocessing module is used to preprocess the single image to be reconstructed to obtain the head portrait;

[0041] The reconstruction mesh module is used to reconstruct a three-dimensional head model from the head image and optimize the reconstructed three-dimensional head model to restore smooth surface details.

[0042] The weak projection shading module is used to weakly project the 3D head model after restoring the smooth surface details onto the head image under the predicted camera matrix, and use the pixel information of the head image as the texture information of the corresponding 3D points, thereby obtaining a 3D head model with frontal texture.

[0043] The cylindrical UV mapping module is used to perform cylindrical UV unwrapping on a 3D head model with frontal texture to obtain a 2D UV texture map and record the UV mapping relationship.

[0044] The UV texture map guided repair module is used to input the two-dimensional UV texture map into a trained guided diffusion network model to conditionally predict the texture of the invisible areas in the two-dimensional UV texture map, repair the complete two-dimensional UV texture map, and use the UV mapping relationship to back-paste the complete two-dimensional UV texture map back onto the three-dimensional head model with the front texture.

[0045] Overall, the technical solutions conceived in this invention have beneficial effects compared with the prior art:

[0046] (1) This invention uses a guided diffusion model to repair the invisible areas of incomplete UV texture maps, thereby reconstructing the full head texture. This invention not only preserves the facial texture details of the original input (such as wrinkles, moles, etc.) to the greatest extent, but also maintains the high-resolution output of the 3D head model. Therefore, it enriches the surface details of the reconstructed 3D head model and improves the accuracy of local detail textures. Compared with other methods for head reconstruction, this invention can reconstruct a high-resolution, high-fidelity, artifact-free, and texture-rich 3D head from a single image without considering the individual's age, gender, race, or facial expression.

[0047] (2) The guided diffusion model proposed in this invention is trained only on the invisible area and does not directly embed the mask. Instead, it directly adds noise prediction to the invisible area, so it can directly repair the invisible area of ​​the incomplete UV texture map and thus reconstruct the full head texture.

[0048] (3) Compared with manual modeling or reconstruction of three-dimensional head models based on three-dimensional scanners and stereo vision systems, this invention only requires inputting a single portrait image into the computer to complete the reconstruction, which can save costs to the greatest extent and greatly improve the convenience of operation, and has broad application prospects in the future.

[0049] (4) In the method of directly reconstructing a three-dimensional head model from multi-view images based on deep learning technology, the present invention only needs to input one image to reconstruct a high-fidelity three-dimensional head model, which minimizes the application conditions. Attached Figure Description

[0050] Figure 1 This is a flowchart of a single-image 3D head reconstruction method according to an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the network framework used in the single-image 3D head reconstruction method of this invention.

[0052] Figure 3 This is a schematic diagram of the projection relationship between weak projection mapping and UV mapping in an embodiment of the present invention;

[0053] Figure 4 This is an example of the effect of performing 3D head reconstruction on a standard public dataset according to an embodiment of the present invention, compared with other methods;

[0054] Figure 5 This is an example of the effect of performing three-dimensional head reconstruction on field images according to an embodiment of the present invention, compared with other methods;

[0055] Figure 6 This is a 3600-degree example of the effect of three-dimensional head reconstruction of field images according to an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0057] In the description of the embodiments of this application, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, apparatus, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product or device.

[0058] The naming or numbering of steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved.

[0059] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0060] This invention provides a method and system for single-image 3D head reconstruction based on a guided diffusion model, which will be described below.

[0061] like Figure 1 and Figure 2 As shown, the single-image 3D head reconstruction method based on the guided diffusion model of this invention includes the following steps:

[0062] S101, preprocess the single image to be reconstructed to obtain the head image.

[0063] Specifically, this could involve acquiring the portrait image to be reconstructed, extracting the portrait mask from it, removing background elements, and obtaining the head image.

[0064] The background of the input image P to be reconstructed is segmented to obtain a black and white image describing the human figure region, denoted as M. The black and white image is used to limit the computer's operation on the head image I of the image to be reconstructed.

[0065]

[0066] This indicates dot product.

[0067] In one embodiment, the background outside the human figure can be removed from the input image using the API interface provided by the existing platform, and then the image mask can be segmented using code.

[0068] S102, reconstruct a 3D head model from head portrait I, and optimize the reconstructed 3D head model to restore smooth surface details.

[0069] The reconstruction of the 3D head model in step S102 can be achieved using any existing technology, such as the SeIF method. SeIF is a method for generating an animated 3D head model from a single image. This method includes the following steps: acquiring a head image and generating a parametric model; passing semantic information from a normalized semantic template to the parametric model, generating a specific semantic model for the head image and sampling it, and assigning a semantic code to each sampling point; extracting the normal map and feature map of the head image; inputting each sampling point, feature vector, and semantic code into a deep implicit function to predict the occupancy value and precise semantic code for each sampling point; extracting equipotential surfaces from the occupancy field to generate a 3D head model, and performing structural unification using precise semantic codes; and dynamically driving the structurally unified 3D head model using an expression-driven algorithm.

[0070] Optimizing the reconstructed 3D head model can be achieved using any existing technique, such as a normal geometry optimization algorithm. This algorithm performs an affine transformation between the normal map rendered by the SeIF method and the head image to be reconstructed, thus offsetting the 3D points in the reconstructed 3D head model. By calibrating the normal map and the head image to be reconstructed, the positions of the 3D points in the SeIF-reconstructed 3D head model are moved, thereby optimizing the 3D head model.

[0071] S103, the 3D head model after restoring the smooth surface details in step S102 is weakly projected onto the head image under the predicted camera matrix, and then the pixel information of the head image I is used as the texture information of the corresponding 3D points to obtain a 3D head model with frontal texture.

[0072] When using weak projection mapping, the normal direction can be used to determine the positive and negative directions of the 3D model.

[0073] Furthermore, S103 includes the following sub-steps:

[0074] (1) The reconstructed three-dimensional head model is interpolated to preserve the details of the input image I to the maximum extent. The coordinates of the three-dimensional points of the interpolated three-dimensional head model are marked as V(X,Y,Z). At the same time, the three-dimensional points with a value greater than 0.5 on the z-axis of the three-dimensional model are selected as the three-dimensional points to be colored, that is, the three-dimensional points to be determined for the frontal texture. The remaining points are set to gray, that is, the invisible areas of the texture to be predicted.

[0075] Preferably, such as Figure 3 As shown, the coordinates of a 3D point in 3D space are denoted as V(X,Y,Z). Under the predicted camera matrix (camera focal length f), each 3D point is mapped onto the head image I through weak projection mapping transformation to obtain the corresponding 2D pixel coordinates v(x,y) of each 3D point. A floor function is then used to convert these coordinates to integer coordinates to index the corresponding pixel position. The projection relationship is as follows:

[0076]

[0077] When weakly projecting the 3D point V(X,Y,Z) onto the head image I, a rounding down operation may have caused errors when obtaining the corresponding 2D pixel coordinates v(x,y). To obtain the most accurate color information, the neighborhood information of the 2D pixel coordinates v(x,y) of the head image is extracted. The color of the 2D pixel coordinates v(x,y) is determined based on the neighborhood information to improve the robustness of the algorithm. The obtained color value of the corresponding pixel position v(x,y) is denoted as C. x,y ;

[0078] Furthermore, C x,y The calculation formula is:

[0079] C x,y =(C r (x,y),C g (x,y),C b (x,y))

[0080] C(I,V)=χ(C x,y )

[0081] Among them, C r C g C b χ(·) represents the values ​​of the red, green, and blue channels, respectively, and χ(·) represents the neighborhood function.

[0082] S104: Unfold the three-dimensional head model with frontal texture using cylindrical UV mapping to obtain a two-dimensional UV texture map and record the UV mapping relationship.

[0083] UV is a two-dimensional texture coordinate system that uses the letters U and V to indicate axes in two-dimensional space. It helps to place two-dimensional image textures on 3D surfaces. The UV mapping relationship defines how two-dimensional texture coordinates are mapped to the surface mesh of a three-dimensional model.

[0084] Preferably, such as Figure 3 As shown, for the 3D point V(X,Y,Z) of the 3D head model, it is represented by surface parametric equations:

[0085] X=χ(s,t),Y=η(s,t),Z=z(s,t)

[0086] Where χ(·), η(·), and z(·) are transformation functions of the three-dimensional surface, and (s,t) are parameters corresponding to a point on the three-dimensional surface.

[0087] For a coordinate point T(u,v) in UV space, it can be represented as:

[0088]

[0089] Where, φ(·), It is the transformation function of UV space based on the three-dimensional surface parameters (s,t).

[0090] Based on the above representation relationship, the three-dimensional point V(X,Y,Z) of the mesh is projected onto the UV space through cylindrical UV mapping. For the cylindrical coordinates of the three-dimensional point in the cylindrical space coordinate system... The correspondence between the three-dimensional coordinate point V(X,Y,Z) and its corresponding coordinates is as follows:

[0091] X = ρcosθ, Y = ρsinθ, Z = z

[0092] Where ρ is the distance from the point to the z-axis (i.e., θ is the angle projected onto the xy-plane from the point (X,Y) by rotating counterclockwise from the positive x-axis.

[0093] For a point T(u,v) in the UV space, it is defined as:

[0094]

[0095] Among them, z min It is the minimum value of the 3D head model in the z-axis direction, z max It is the maximum value of the 3D head model in the z-axis direction.

[0096] After UV unwrapping the mesh, the colors of each triangular facet of the 3D model are rendered onto the corresponding UV color regions in the UV space through diffuse reflection. The calculation formula is as follows:

[0097]

[0098] Where R(·) represents diffuse rendering, Area(u 1...n ,v 1...n Area(V) represents a color region in UV space bounded by multiple uv coordinates. 1...n ) represents the color area of ​​the triangular facet in the 3D head model.

[0099] The color value of a pixel T(u,v) in a UV texture image can be denoted as C(u,v), and its calculation formula is as follows:

[0100] C(u,v)=(r(u,v),g(u,v),b(u,v))

[0101] Where r, g, and b represent the values ​​of the red, green, and blue channels, respectively. This calculation yields the color value of the corresponding pixel T(u,v) in the UV space.

[0102] S105: Input the two-dimensional UV texture map into the trained guided diffusion network model to conditionally predict the texture of the invisible area and repair it to obtain a complete two-dimensional UV texture map.

[0103] The principle of the guided diffusion model will be explained in detail below.

[0104] The guided diffusion model adds noise to the texture regions to be learned in the real UV texture map through a given step until it becomes a pure noise image. Then, it denoises and restores the original image according to a given step size. In the reverse denoising process, it continuously compares with the noise added at each step in the forward process, calculates the loss difference between the two images before and after denoising, and iterates until the loss is minimized. In this way, it learns the image distribution pattern and image generation direction in the reverse denoising process, so as to achieve the purpose of repairing the invisible areas of the UV incomplete texture map.

[0105] Given a real UV texture image x, add noise to the regions that are set to gray after weak projection in step S103, given the noise image at a certain moment during the noise addition process. It can be expressed as:

[0106]

[0107] Where γ represents the current noise level, which is the cumulative noise added at each time step, and ε refers to the noise added at each time step.

[0108] Noise image The real UV texture image x and the current noise level γ are used as input parameters to parameterize the neural network model. The network model can be represented as:

[0109]

[0110] The training process is explained in detail below.

[0111] (1) Prepare the training sample set.

[0112] For existing 3D head model samples, a complete 2D UV texture map is obtained through UV mapping and exposed at a ratio of 0.7 to 1.3 to form a training image set sample. At the same time, the mask of the invisible area corresponding to the training image is obtained.

[0113] (2) The diffusion network model is trained using the training image sample set.

[0114] To predict the noise added at each time step, the loss function designed in this embodiment of the invention is as follows:

[0115]

[0116] Where P is the step size set during noise reduction. This indicates that the image contains noise. The loss obtained after denoising and fitting the original training image x at each time step is combined with the noise added at the current time step. During training, the loss obtained from each iteration and the noise at the current time step are used as weights and re-inputted into the neural network. This process is repeated until the output loss is minimized; the predicted noise at this point is the optimal solution. P Let T represent the denoising process from step 1 to step P, and let T represent the transpose matrix. When the result of ||.|| is a vector, the rows and columns must be consistent before multiplication with the weight coefficients. During training, P is set to 250, 500, and 1000 respectively to verify that the model's repair efficiency is improved as much as possible while ensuring training effectiveness, and time and memory consumption are reduced.

[0117] By iteratively comparing noisy images and real UV texture images using a loss function, and returning the parameters of the neural network, the accuracy of the predicted noise vector during training is evaluated. This allows the model to learn the rules in the UV texture image denoising process, that is, the rules that should be followed when a UV texture image becomes a pure noise image after being denoised, in order to denoise and restore it back to the original UV texture image.

[0118] Furthermore, the trained guided diffusion model is used for UV-damaged texture map restoration, including the following steps:

[0119] The UV incomplete texture map restoration process is achieved through the inverse denoising process learned during training. Given a UV incomplete texture map x, the incomplete part of x is denoised with a step size of T to obtain a pure noise image. Then, P iterations of denoising are performed. The corresponding real image x0 is obtained by restoring the predicted UV incomplete texture map x through the trained guided diffusion model. The calculation formula is as follows:

[0120]

[0121] γ p Let x represent the noise at the P-th time step. p This represents the image after P time steps of noise reduction.

[0122] S106: Input the complete UV texture map into the super-resolution network model and super-resolution it to 4096 resolution to ensure that the input head image can be preserved in the 3D model with high fidelity.

[0123] Step S106 is a preferred step, but not a necessary one.

[0124] S107: Using the recorded UV mapping relationship, the UV texture map is back-painted back into the 3D head model with frontal texture in S103.

[0125] Preferably, such as Figure 3As shown, for a 3D point V(X,Y,Z) on the surface of a 3D model, after obtaining T(u,v) through UV mapping, the color value of the UV texture image at point T(u,v) can be assigned to the 3D point V(X,Y,Z) through a predetermined relationship, thereby realizing the attachment of texture to the surface of the 3D model.

[0126] Thanks to the powerful expressive power and flexibility of the guided diffusion model in image restoration, this invention effectively solves the problems of low resolution, unrecognizable results, low fidelity, and poor generalization ability of the 3D head model reconstructed by existing methods. Figure 4 The image shown is an example of the effect of performing 3D head reconstruction on a standard public dataset according to an embodiment of the present invention, compared with other methods. Figure 5 The image shown is an example of the effect of three-dimensional head reconstruction of field images according to an embodiment of the present invention, compared with other methods. Figure 6 The image shown is a 360° view example of a 3D head model reconstructed from a single portrait image using this invention. It can be seen that this invention can reconstruct high-resolution, high-fidelity, and richly detailed 3D head models from portrait images of different ages, races, genders, and facial expressions.

[0127] It should be noted that steps S101 and S106 above are not mandatory steps, and users can adjust them flexibly according to their needs.

[0128] An embodiment of the present invention provides a single-image 3D head reconstruction system based on a guided diffusion model, comprising:

[0129] The preprocessing module is used to preprocess the single image to be reconstructed to obtain the head portrait;

[0130] The reconstruction mesh module is used to reconstruct a three-dimensional head model from the head image and optimize the reconstructed three-dimensional head model to restore smooth surface details.

[0131] The weak projection shading module is used to weakly project the 3D head model after restoring the smooth surface details onto the head image under the predicted camera matrix, and use the pixel information of the head image as the texture information of the corresponding 3D points, thereby obtaining a 3D head model with frontal texture.

[0132] The cylindrical UV mapping module is used to perform cylindrical UV unwrapping on a 3D head model with frontal texture to obtain a 2D UV texture map and record the UV mapping relationship.

[0133] The UV texture map guided repair module is used to input the two-dimensional UV texture map into a trained guided diffusion network model to conditionally predict the texture of the invisible areas in the two-dimensional UV texture map, repair the complete two-dimensional UV texture map, and use the UV mapping relationship to back-paste the complete two-dimensional UV texture map back onto the three-dimensional head model with the front texture.

[0134] The working principle and technical effects of the single-image 3D head reconstruction system are the same as those of the single-image 3D head reconstruction method described above, and will not be repeated here.

[0135] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A single-image 3D head reconstruction method based on a guided diffusion model, characterized in that, Including the following steps: The single image to be reconstructed is preprocessed to obtain the head portrait; A three-dimensional head model is reconstructed from the head image, and the reconstructed three-dimensional head model is optimized to restore smooth surface details; The 3D head model with restored surface smooth details is weakly projected onto the head image under the predicted camera matrix, and the pixel information of the head image is used as the texture information of the corresponding 3D points to obtain a 3D head model with frontal texture. The 3D head model with frontal texture is subjected to cylindrical UV unwrapping to obtain a 2D UV texture map and the UV mapping relationship is recorded. The two-dimensional UV texture map is input into a trained guided diffusion network model to conditionally predict the texture of invisible areas in the two-dimensional UV texture map and repair it to obtain a complete two-dimensional UV texture map. Using the UV mapping relationship, the complete two-dimensional UV texture map is back-painted back onto the three-dimensional head model with the frontal texture.

2. The single-image 3D head reconstruction method based on the guided diffusion model as described in claim 1, characterized in that, Before pasting the complete 2D UV texture map back onto the 3D head model with frontal texture, the method further includes the step of inputting the complete UV texture map into the super-resolution network model to improve the resolution of the complete UV texture map.

3. The single-image 3D head reconstruction method based on the guided diffusion model as described in claim 1, characterized in that, The step of weakly projecting the restored 3D head model onto the head image using the predicted camera matrix, and using the pixel information of the head image as the texture information of the corresponding 3D points to obtain a 3D head model with frontal texture includes the following steps: Interpolation processing is performed on the 3D head model after restoring the surface smooth details. The coordinates of the 3D points of the interpolated 3D head model are denoted as V(X,Y,Z). At the same time, the 3D points of the 3D head model with a value greater than 0.5 on the z-axis are taken as the 3D points of the frontal texture to be determined, and the remaining 3D points are taken as the invisible area of ​​the texture to be predicted. Under the predicted camera matrix, the three-dimensional points V(X,Y,Z) in the three-dimensional space are mapped onto the head image through weak projection mapping transformation, so as to obtain the two-dimensional pixel coordinate point v(x,y) corresponding to each three-dimensional point V(X,Y,Z). The pixel information of the two-dimensional pixel coordinate point v(x,y) of the head image is used as the texture information of the corresponding three-dimensional point V(X,Y,Z) to obtain a three-dimensional head model with frontal texture.

4. The single-image 3D head reconstruction method based on the guided diffusion model as described in claim 3, characterized in that, The formula for calculating weak projection mapping transformation is: Where f is the camera focal length.

5. The single-image 3D head reconstruction method based on the guided diffusion model as described in claim 3, characterized in that, The step of using the pixel information of the two-dimensional pixel coordinate point v(x,y) of the head image as the texture information of the corresponding three-dimensional point V(X,Y,Z) includes the following steps: Extract the neighborhood information of the two-dimensional pixel coordinate point v(x,y) of the head image, and determine the color of the two-dimensional pixel coordinate point v(x,y) based on the neighborhood information, denoted as C. x,y ; The color C of the two-dimensional pixel coordinate point v(x,y) x,y This serves as the texture information for the corresponding 3D point V(X,Y,Z).

6. The single-image 3D head reconstruction method based on the guided diffusion model as described in claim 1, characterized in that, The process of performing cylindrical UV unwrapping on a 3D head model with frontal texture includes the following steps: The 3D head model with frontal texture is subjected to cylindrical UV unwrapping. The corresponding point in UV space after the 3D point V(X,Y,Z) is subjected to cylindrical UV unwrapping is denoted as T(u,v). The colors of each triangular facet of the 3D head model with a frontal texture are rendered onto the corresponding UV color regions in the UV space using diffuse reflection. The calculation formula is as follows: Where R(·) represents diffuse rendering, Area(u 1...n ,v 1...n Area(V) represents a color region in UV space bounded by multiple uv coordinates. 1...n () represents the color area of ​​a triangular facet in a 3D head model with a frontal texture. This represents the two-dimensional UV texture map. This represents the frontal texture of a 3D head model.

7. The single-image 3D head reconstruction method based on the guided diffusion model as described in claim 6, characterized in that, The UV mapping relationship is as follows: Where ρ is the distance from the 3D point V(X,Y,Z) to the z-axis, θ is the angle from the positive x-axis direction to the projection of the point (X,Y) onto the xy-plane, and z... min It is the minimum value of a 3D point in the 3D head model along the z-axis. max It is the maximum value of the three-dimensional points of the three-dimensional head model along the z-axis.

8. The single-image 3D head reconstruction method based on the guided diffusion model as described in claim 1, characterized in that, The training of the guided diffusion network model includes the following steps: For the 3D head model samples, a 2D UV texture map is obtained through UV mapping and exposed at different ratios to form a training image sample set. At the same time, the mask of the invisible area corresponding to each training image is obtained. The diffusion network model is trained using a set of training image samples.

9. The single-image 3D head reconstruction method based on the guided diffusion model as described in claim 8, characterized in that, The loss function of the diffusion network model is: Where x is the training image, P is the step size set during denoising, and γ represents the current noise level. This represents the added noise predicted by the diffusion network model based on x and γ, where ε represents the noise added at each time step. Indicates a noisy image The loss Z obtained by fitting the original training image x after denoising at each time step ε,γ The noise at the current time step of the prediction is represented by ||.|| P This indicates that the noise reduction process continues from step 1 until it stops at step P. T This indicates the matrix transpose.

10. A single-image 3D head reconstruction system based on a guided diffusion model, characterized in that, include: The preprocessing module is used to preprocess the single image to be reconstructed to obtain the head portrait; The reconstruction mesh module is used to reconstruct a three-dimensional head model from the head image and optimize the reconstructed three-dimensional head model to restore smooth surface details. The weak projection shading module is used to weakly project the 3D head model after restoring the smooth surface details onto the head image under the predicted camera matrix, and use the pixel information of the head image as the texture information of the corresponding 3D points, thereby obtaining a 3D head model with frontal texture. The cylindrical UV mapping module is used to perform cylindrical UV unwrapping on a 3D head model with frontal texture to obtain a 2D UV texture map and record the UV mapping relationship. The UV texture map guided repair module is used to input the two-dimensional UV texture map into a trained guided diffusion network model to conditionally predict the texture of the invisible areas in the two-dimensional UV texture map, repair the complete two-dimensional UV texture map, and use the UV mapping relationship to back-paste the complete two-dimensional UV texture map back onto the three-dimensional head model with the front texture.

Images