A method for intelligently generating three-dimensional objects based on a single photo

Abstract

The present invention discloses a method for intelligently generating a three-dimensional target based on a single photo, comprising the following steps: S1, constructing a data set, including a single character model and a scene composed of multiple characters, obtaining textured renderings under multiple perspectives and original height maps under corresponding perspectives; S2, constructing a feature extraction network model, wherein the feature extraction network model includes a structural feature extraction module and a detail feature extraction module, inputting an original height map, obtaining a structural height map through the structural feature extraction module, and obtaining a detail height map through the detail feature extraction module; S3, obtaining a complete three-dimensional target image model through weighted fusion of the obtained structural height map and detail height map; S4, performing optional post-processing on the three-dimensional target image model to perform feature enhancement. This method reduces the dependence and requirements of the process on equipment and professional skills, lowers the threshold for each Internet user to produce virtual content, and makes it as convenient as taking a photo.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a method for intelligently generating three-dimensional objects based on a single photo. Background Art

[0002] In the digitalization of human figures, beyond constructing complete 3D models, creating 3D avatars with a certain sense of depth holds great artistic and practical value in enriching people's virtual experiences and increasing the diversity of online content. With the gradual development of the metaverse, the demand for diverse avatars will become even more urgent. However, unlike traditional methods of generating flat image data, the design and production of 3D avatars relies on specialized modeling software, hindering widespread adoption.

[0003] The 3D image of a target is different from a complete 3D model. It is limited to a certain height range, but it needs to present important features and a high sense of hierarchy, which is an artistic form between a 2D image and a 3D model. How to quickly convert a target photo into a digitized 3D image of the target has become an important foundation for virtual image-related applications. Existing methods for constructing 3D digital targets mainly rely on professional modeling software, 3D scanning reconstruction, or multi-viewing. Figure 3 dimensional reconstruction and target three-dimensional suppression algorithm conversion not only has high requirements on technology and equipment, but is also very time-consuming and labor-intensive. Summary of the Invention

[0004] The present invention proposes a method for intelligently generating three-dimensional objects based on a single photo, which reduces the process's dependence on and requirements for equipment and professional skills, lowers the threshold for each Internet user to produce virtual content, and makes it as convenient as taking a photo.

[0005] In order to solve the above technical problems, the technical solution of the present invention is:

[0006] A method for intelligently generating a three-dimensional object based on a single photo, comprising the following steps:

[0007] S1. Build a dataset, including a single character model and a scene composed of multiple characters, and obtain textured renderings from multiple perspectives and original height maps from the corresponding perspectives;

[0008] S2. Construct a feature extraction network model, wherein the feature extraction network model includes a structural feature extraction module and a detail feature extraction module. The original height map is input to the structural feature extraction module to obtain a structural height map, and the detail feature extraction module is used to obtain a detail height map.

[0009] The loss function of the structural feature extraction module is as follows:

[0010]

[0011] Among them, h i (u, v) represents the original height map of the 3D scene, φ1(u, v) represents the structural gradient adjustment function, Ω represents the entire image domain, and h s (u,v) represents the structural height map output by the model,

[0012] The loss function of the detail feature extraction module is as follows:

[0013]

[0014] Among them, h i (u,v) represents the original height map of the 3D scene, φ2(u,v) represents the detail gradient adjustment function, h d (u,v) represents the detail height map output by the model, Δ represents the Laplace operator, and div represents the divergence operator;

[0015] S3, obtaining a complete target three-dimensional image model by weighted fusion of the obtained structure height map and detail height map;

[0016] S4. The target three-dimensional image model is subjected to optional image post-processing, wherein the image post-processing includes image re-illumination, grayscale conversion of color photos, and non-realistic rendering of photos.

[0017] Preferably, in step S1, the original height map is collected by setting the lighting conditions to simulate the collection environment of real photos, and a large number of three-dimensional scenes are obtained by deformation and combination of the target three-dimensional model, and a data set suitable for deep network models of various scales is constructed by combining multiple views.

[0018] Preferably, the lighting conditions include no lighting environment, high dynamic range map ambient light and frontal light source lighting.

[0019] Preferably, in step S2, the structural height map acquisition method processes the gradient vector of the original height map by introducing a gradient adjustment function through the structural feature extraction module, thereby removing the larger height gap between the original height map of the three-dimensional scene and the background while maintaining the necessary height hierarchy. Then, the parameters in the specific gradient adjustment function are determined in combination with the gradient distribution of the height map in the data set to remove unnecessary gradient vectors with large modulus values, and a loss function defined on the gradient domain is introduced.

[0020] Preferably, the gradient adjustment function adopts the following gradient adjustment function, which is expressed as follows:

[0021]

[0022] The value of the parameter α can be determined according to the resolution of the height map. For example, when the resolution is 1024×1024, the value is 256. The characteristic of this function is that the gradient modulus values ​​greater than a certain value are uniformly transformed to 1, thereby implicitly eliminating the original larger gradient modulus values.

[0023] Preferably, in the detail height map acquisition method in step S2, the detail feature extraction model processes the gradient vector of the height map through a detail gradient adjustment function, and then calculates the local details of the height map in combination with the divergence operator div. The detail gradient adjustment function φ2(u,v) is the difference between the gradient adjustment functions S(x,α) using two different parameter α values, and its specific form is,

[0024] Φ2(x,α1,α2)=(S(||x||,α2)-S(||x||,α1))x

[0025] Preferably, the weighted fusion expression of step S3 is as follows:

[0026] h(u,v)=Γ(h s (u,v),h d (u,v))

[0027] Here, Γ(·,·) represents the specific fusion method.

[0028] The present invention has the following characteristics and beneficial effects:

[0029] The above-described technical solution eliminates the need for a 3D model to generate a 3D target image. Instead, a single portrait photograph can be used to generate a 3D target image that reflects both overall 3D effects and local geometric details. This overcomes the limitations of existing 3D target generation methods, which require manual design and production or time-consuming and labor-intensive 3D scanning and reconstruction, hindering their application. Furthermore, the generated 3D target height map provides geometric information, such as normals and curvature, for processing effects tailored to the original photo, such as relighting, anime stylization, and line drawing non-photorealistic rendering. Therefore, this invention not only presents a new application solution but also offers a new interdisciplinary field combining image processing and computer graphics. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1This is a flow chart of a method for intelligently generating a three-dimensional object based on a single photo according to an embodiment of the present invention.

[0032] Figure 2 The dataset of the present invention includes an albedo map, a mask map, a height map, and several lighting rendering maps.

[0033] Figure 3 Schematic diagram of the structural feature extraction module flow in an embodiment of the present invention.

[0034] Figure 4 Schematic diagram of the detail feature extraction module flow in an embodiment of the present invention.

[0035] Figure 5 This is an example diagram of the fusion of the outputs of the two modules in an embodiment of the present invention.

[0036] Figure 6 This is a multi-viewing effect diagram of an embodiment of the present invention.

[0037] Figure 7 、 8 More examples of the effects of the embodiments of the present invention are shown below. DETAILED DESCRIPTION

[0038] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0039] The present invention provides a method for intelligently generating a stereoscopic target based on a single photo. Figure 1 As shown, the following steps are included:

[0040] S1. Build a dataset, including a single character model and a scene consisting of multiple characters, and obtain textured renderings from multiple perspectives and original height maps from the corresponding perspectives.

[0041] It should be noted that the original height map is collected based on the fact that the same scene will produce different photos under different lighting conditions. To simulate this, several lighting conditions are set, including no lighting with only texture mapping, high dynamic range mapping with ambient lighting, and frontal lighting. These methods approximate the real-world photo collection environment, thereby improving the model's processing performance for actual photos. By deforming and combining the target 3D model, thousands of 3D scenes can be obtained. Combining multiple views can construct datasets suitable for deep network models of various sizes.

[0042] S2. Construct a feature extraction network model, wherein the feature extraction network model includes a structural feature extraction module and a detail feature extraction module. The original height map is input to the structural feature extraction module to obtain a structural height map, and the detail feature extraction module is used to obtain a detail height map.

[0043] The loss function of the structural feature extraction module is as follows:

[0044]

[0045] Among them, h i (u, v) represents the original height map of the 3D scene, φ1(u, v) represents the structural gradient adjustment function, Ω represents the entire image domain, and h s (u,v) represents the structural height map output by the model,

[0046] Specifically, the structural height map acquisition method introduces a gradient adjustment function to process the gradient vector of the original height map through the structural feature extraction module, thereby removing the large height gap between the original height map of the three-dimensional scene and the background while maintaining the necessary height hierarchy. Then, the parameters in the specific gradient adjustment function are determined in combination with the gradient distribution of the height map in the dataset to remove unnecessary gradient vectors with large modulus values, and a loss function defined on the gradient domain is introduced.

[0047] It should be noted that the gradient adjustment function is not limited to a specific function. Its main function is to reduce or eliminate the larger gradient modulus values ​​in the height map (such as the junction of the foreground and background) and increase the smaller gradient modulus values, thereby removing the larger height gap between the original height map of the three-dimensional scene and the background, while maintaining the necessary height layering.

[0048] Therefore, in this embodiment, the gradient adjustment function adopts the following gradient adjustment function, which is expressed as follows:

[0049]

[0050] The value of the parameter α can be determined according to the resolution of the height map. For example, when the resolution is 1024×1024, the value is 256. The characteristic of this function is that the gradient modulus values ​​greater than a certain value are uniformly transformed to 1, thereby implicitly eliminating the original larger gradient modulus values.

[0051] In the above technical solution, each input original height map in the data set does not have the target three-dimensional image of the person in the image, so the supervised learning method cannot be directly used in this embodiment.

[0052] It should be noted that in order to combine features at multiple resolutions, the neural network from the rendering image to the structural height map adopts an architecture based on the combination of DenseNet and Unet, achieving a good compromise between performance and scale.

[0053] It is understandable that in the above technical solution, only the input is known, and the output of the network (the target stereoscopic height map) is not given in advance. In order to solve this problem, this embodiment proposes a clever solution, which is to let the network learn the gradient of the height map instead of directly learning the height map representing the target stereo. The pre-learned gradient value is obtained by processing the gradient value of the original height map sampled from the three-dimensional scene through an adjustment function. Different adjustment functions can extract information of different frequencies. In order to be more reasonably compatible with input photos of different resolutions, this embodiment adopts a dual-branch network architecture, one of which is a structural module for extracting the overall structure and layering; the other is a detail module for extracting local details.

[0054] As you can imagine, the goal is to extract a height map that can reflect the overall height hierarchy from the photo. Since there is no known target stereoscopic height map in the dataset, the available information is mined from the original height map of the 3D scene corresponding to the photo. In order to remove the large height gaps between the foreground and background and between the parts within the foreground, while maintaining medium-sized gaps to present the necessary height hierarchy (e.g. Figure 3 In the right picture, the height of the character's right arm is greater than the height of other areas).

[0055] Furthermore, the loss function of the detail feature extraction module is as follows:

[0056]

[0057] Among them, h i (u,v) represents the original height map of the 3D scene, φ2(u,v) represents the structural gradient adjustment function, and h d (u,v) represents the detail height map output by the model, Δ represents the Laplace operator, and div represents the divergence operator;

[0058] Specifically, in the step S2, the detail height map acquisition method, the detail feature extraction model processes the gradient vector of the height map through the detail gradient adjustment function, and then calculates the local details of the height map in combination with the divergence operator div. The detail gradient adjustment function φ2(u,v) is the difference between the gradient adjustment functions S(x,α) using two different parameter α values, and its specific form is,

[0059] Φ2(x,α1,α2)=(S(||x||,α2)-S(||x||,α1))x.

[0060] It can be understood that the above technical solution is aimed at detail height maps. Different from the acquisition of structure height maps, it is not necessary to maintain the sense of hierarchy here, so the focus is on local details.

[0061] It should be noted that in order to combine features at multiple resolutions, the neural network from the rendering image to the structural height map adopts an architecture based on the combination of DenseNet and Unet, achieving a good compromise between performance and scale.

[0062] Understandably, Figure 4 The figure shows the detail module, which aims to extract local details from a photo. Because the structure height map primarily maintains gradient vectors with larger magnitudes, it ignores details corresponding to gradients with smaller magnitudes. Therefore, the detail height map ignores the sense of height hierarchy and focuses on local details with smaller gradient magnitudes.

[0063] The key technology of this embodiment is to deconstruct the target three-dimensional model at two levels: structure and detail, and design corresponding loss functions in the gradient domain.

[0064] As you can understand, the gradient adjustment function is used to perform a nonlinear transformation on the height map's gradient vectors. The key step is to introduce a threshold to detect and remove unnecessary gradients with large modulosity, and then perform nonlinear scaling on the remaining gradient vectors. Different thresholds are used for the structure module and the detail module.

[0065] S3. Obtain a complete target three-dimensional image model by weighted fusion of the obtained structure height map and detail height map

[0066] The specific weighted fusion expression is as follows:

[0067] h(u,v)=Γ(h s (u,v),h d (u,v))

[0068] Here, Γ(·,·) represents a specific fusion method, such as linear or nonlinear weighting.

[0069] It can be understood that in this embodiment, the character model rendering and the binary mask image representing the character foreground are input into the feature extraction network model to obtain the output of the structural feature extraction module and the detail feature extraction module, and the two modules of the model are trained using two loss functions. The outputs of the two modules themselves can also be used as the target three-dimensional image, but the height map output by the structural feature extraction module mainly presents a sense of height hierarchy and often lacks details; while the height map output by the detail feature extraction module only presents local details and lacks an overall sense of three-dimensionality. Therefore, a fusion function is used to weightedly fuse the two in a linear or nonlinear manner into a target three-dimensional height map. In the fusion step, each pixel is calculated separately, which is suitable for efficient parallel computing and is convenient for users to select effects in real time by selecting weighting parameters.

[0070] S4. The target three-dimensional image model is subjected to optional image post-processing, wherein the image post-processing includes image re-illumination, grayscale conversion of color photos, and non-realistic rendering of photos.

[0071] Specifically, optional post-processing is performed on any facial regions in a photo. A module is trained to convert 2D images into 3D facial models for these regions, and the resulting model is then weightedly fused with the target stereoscopic height map. This module can be integrated into the entire network, modifying the dataset so that the network model directly learns this effect. Alternatively, separating it as a separate, optional module allows users to choose from a variety of effects.

[0072] Further, Figure 6 A practical example is given. Combining multiple perspectives, it can be seen that the entire 3D image of the target maintains a high degree of layering of the various parts of the person in the photo, maintaining important features such as facial features, facial edges, clothing folds, etc., and the edges of the entire 3D image of the target are all suppressed to the background plane. More examples including multiple person photos, such as Figure 7 、 8 As shown. It's worth noting that the target 3D image is not a complete 3D model; rather, it's highly constrained within a certain range while still retaining key features. This invention avoids the intermediate step of 3D reconstruction and directly achieves the intelligent conversion process from a person's photo to a 3D image.

[0073] In summary, the technical features of the present invention are as follows: First, in addition to manual production, existing methods for generating stereoscopic targets mainly generate stereoscopic target image models based on existing three-dimensional target models, so it is necessary to first construct the three-dimensional geometric information of the target model, which is time-consuming and labor-intensive. The present invention is characterized by directly inputting a single photo, and the trained deep learning model can quickly generate a stereoscopic target image, saving time and effort. Second, in general, existing stereoscopic images lack texture, while photos lack geometric information. The present invention integrates the two, facilitating the combination of algorithms from the two fields of image processing and computer graphics to generate more content and effects.

[0074] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for intelligently generating a three-dimensional object based on a single photo, characterized in that: The steps include: S1. Build a dataset, including a single target model and a scene consisting of multiple targets, and obtain textured renderings from multiple perspectives and original height maps from the corresponding perspectives; S2. Construct a feature extraction network model, wherein the feature extraction network model includes a structural feature extraction module and a detail feature extraction module. The original height map is input to the structural feature extraction module to obtain a structural height map, and the detail feature extraction module is used to obtain a detail height map. The loss function of the structural feature extraction module is as follows: Among them, h i (u, v) represents the original height map of the 3D scene, φ1(u, v) represents the structural gradient adjustment function, Ω represents the entire image domain, and h s (u,v) represents the structural height map output by the model, The loss function of the detail feature extraction module is as follows: Among them, h i (u,v) represents the original height map of the 3D scene, φ2(u,v) represents the detail gradient adjustment function, h d (u,v) represents the detail height map output by the model, Δ represents the Laplace operator, and div represents the divergence operator; S3, obtaining a complete target three-dimensional image model by weighted fusion of the obtained structure height map and detail height map; S4. The target three-dimensional image model is subjected to image post-processing, wherein the image post-processing includes re-illumination of the image, grayscale conversion of the color photo, and non-realistic rendering of the photo.

2. The method for intelligently generating a three-dimensional object based on a single photo according to claim 1, characterized in that: In step S1, the original height map is collected by first setting the lighting conditions to simulate the real-life photo collection environment, obtaining a large number of three-dimensional scenes through deformation and combination of the target three-dimensional model, and combining multiple views to construct a data set suitable for deep network models of various scales.

3. The method for intelligently generating a three-dimensional object based on a single photo according to claim 2, characterized in that: The lighting conditions include no lighting environment, high dynamic range map ambient light, and frontal light source lighting.

4. The method for intelligently generating a three-dimensional object based on a single photo according to claim 1, characterized in that: In step S2, the method for obtaining the structural height map is as follows: using a structural feature extraction module, processing the gradient vector of the original height map by introducing a gradient adjustment function, thereby removing the large height gap between the original height map of the three-dimensional scene and the background, while maintaining the necessary height layering, and then combining the gradient distribution of the height map in the data set to determine the parameters in the specific gradient adjustment function for removing unnecessary gradient vectors with large moduli, and introducing a loss function defined on the gradient domain.

5. The method for intelligently generating a three-dimensional object based on a single photo according to claim 4, characterized in that: The structural gradient adjustment function is expressed as follows: The value of parameter α depends on the resolution of the height map. When the resolution is 1024 × When 1024, the value is 256. The characteristic of the structural gradient adjustment function is to uniformly transform the gradient modulus values ​​greater than a certain value to 1, thereby implicitly eliminating the original larger gradient modulus values.

6. The method for intelligently generating a three-dimensional object based on a single photo according to claim 5, characterized in that: In step S2, the method for obtaining the detail height map is as follows: the detail feature extraction module processes the gradient vector of the height map through the detail gradient adjustment function, and then calculates the local details of the height map in combination with the divergence operator div. The detail gradient adjustment function φ2(u,v) is the difference between the structure gradient adjustment functions S(x,α) using two different parameter α values, and its specific form is: Φ2(x,α1,α2)=(S(||x||,α2)-S(||x||,α1))x.

7. The method for intelligently generating a three-dimensional object based on a single photo according to claim 1, characterized in that: The weighted fusion expression of step S3 is as follows: h(u,v)=Γ(h s (u,v),h d (u,v)) Here, Γ(·,·) represents the specific fusion method.

Images