Texture-controllable 3D fine face reconstruction method and device based on sketch input
Through the two-stage reconstruction method based on sketch input, the problem of texture control and training difficulty in three-dimensional face reconstruction is solved, and the efficient three-dimensional face reconstruction effect is achieved, enhancing the style robustness.
Patent Information
- Application Number
- CN202411019660.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-29
AI Technical Summary
The prior art has limitations in controlling textures during the three-dimensional face reconstruction process, resulting in poor reconstruction effect and difficulty in training frames based on deep learning, due to the lack of matching data between sketched images and three-dimensional faces.
Using a two-stage reconstruction method based on sketch input, firstly, the two-dimensional face sketch image is processed through a rough reconstruction network to obtain the coefficients of the three-dimensional face deformable model, and then the fine reconstruction network is combined with sketch texture coding and position coding for fine reconstruction, and finally Carry out texture construction to achieve efficient reconstruction of three-dimensional faces.
Effectively control face texture, improve the effect of three-dimensional face reconstruction, reduce training difficulty, avoid texture interference during reconstruction, and enhance style robustness.
Smart Images

Figure CN118553001B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image data processing, and in particular to a texture-controllable three-dimensional fine face reconstruction method and device based on sketch input. Background Art
[0002] Reconstructing a textured 3D face through face sketching is widely used in the fields of customized 3D avatars, art design, film and television entertainment, criminal investigation, etc. However, the existing methods of reconstructing 3D faces have the following problems: On the one hand, texture is crucial in accurately representing facial appearance, and the existing technology has limitations in controlling texture in the 3D face reconstruction process, resulting in poor face reconstruction results. On the other hand, the lack of matching data between the sketch image and the 3D face makes it very difficult to train the data-driven deep learning-based framework. Summary of the invention
[0003] The present invention provides a texture-controllable three-dimensional fine face reconstruction method and device based on sketch input, so as to overcome the defects of the sketch-based three-dimensional face reconstruction method in the prior art, such as lack of texture control and difficulty in training the reconstruction model.
[0004] The present invention provides a texture-controllable three-dimensional fine face reconstruction method based on sketch input, comprising the following steps:
[0005] Acquire a two-dimensional face sketch image to be reconstructed, and perform prediction processing on the two-dimensional face sketch image to be reconstructed through a rough reconstruction network to obtain coefficients of a three-dimensional face deformable model;
[0006] Based on the coefficients of the three-dimensional deformable face model, a pre-reconstruction result of the two-dimensional face sketch image to be reconstructed is constructed;
[0007] Performing feature extraction on the pre-reconstruction result to obtain a three-dimensional position coding map of the pre-reconstruction result, and performing feature extraction on the two-dimensional face sketch image to be reconstructed to obtain a sketch texture coding map;
[0008] Predicting the sketch texture coding map and the three-dimensional position coding map through a fine reconstruction network to obtain a three-dimensional normal offset map;
[0009] Determining a fine position information coding map based on the three-dimensional normal offset map and the three-dimensional position coding map, and performing mapping processing on the fine position information coding map to obtain a three-dimensional face reconstruction result;
[0010] The three-dimensional face reconstruction result is textured to obtain a final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed.
[0011] In some embodiments, the performing of texture construction on the three-dimensional face reconstruction result to obtain the final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed includes:
[0012] Acquire description text for describing three-dimensional human face texture, and determine a target human face texture image matching the description text from a preset human face texture template library;
[0013] Performing texture estimation on the target face texture image to obtain a texture estimation map, a PCA texture map, a masked area of the texture estimation map, and a non-masked area of the PCA texture map;
[0014] By using the PCA texture map corresponding to the non-masked area, the texture estimation map of the masked area is completed to obtain a texture unfolded map of the target face texture image;
[0015] The texture unfolding image is texture-fitted with the three-dimensional face reconstruction result to obtain a final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed.
[0016] In some embodiments, determining a target face texture image matching the description text from a preset face texture template library includes:
[0017] Calling the CLIP model to encode the description text to obtain text encoding features;
[0018] For each face texture image in the preset face texture template library, the CLIP model is called to encode the face texture image to obtain image coding features;
[0019] The feature similarity between the text encoding feature and the image encoding feature is determined, and the facial texture image with the greatest feature similarity is determined as the target facial texture image that matches the description text.
[0020] In some embodiments, the coefficients of the three-dimensional deformable face model include a rotation angle, a translation coefficient, an identity shape coefficient, and an expression shape coefficient. The pre-reconstruction result of the two-dimensional face sketch image to be reconstructed based on the coefficients of the three-dimensional deformable face model includes:
[0021] Determining the average shape features of the three-dimensional human face, the identity shape vector basis, and the expression shape vector basis preset in the three-dimensional human face deformable model;
[0022] The identity shape vector basis is weighted by the identity shape coefficient to obtain an identity shape vector, and the expression shape vector basis is weighted by the expression shape coefficient to obtain an expression shape vector;
[0023] The three-dimensional average face shape feature, the identity shape vector and the expression shape vector are fused to obtain a fused feature, and the fused feature is rotated and adjusted by a rotation angle matrix to obtain an adjusted feature, wherein the rotation angle matrix is constructed according to the rotation angle;
[0024] The adjustment feature is fused with the translation coefficient to obtain a pre-reconstruction result of the two-dimensional face sketch image to be reconstructed.
[0025] In some embodiments, determining the fine position information encoding map based on the three-dimensional normal offset map and the three-dimensional position encoding map includes:
[0026] Determining an overall amplitude control coefficient corresponding to the three-dimensional normal offset map, and determining a normal code of the three-dimensional position code map;
[0027] Performing amplitude adjustment on the three-dimensional normal offset map by using the overall amplitude control coefficient to obtain an amplitude-adjusted offset map, and determining a Hadamard product of the amplitude-adjusted offset map and the normal code;
[0028] The Hadamard product is fused with the three-dimensional position coding map to obtain a fine position information coding map.
[0029] In some embodiments, the final reconstruction result of the 3D face is obtained by calling a 3D reconstruction model to perform 3D reconstruction on the 2D face sketch image to be reconstructed, the 3D reconstruction model includes a coarse reconstruction network, a fine reconstruction network, a differentiable renderer and a sketch differentiable module, and the training process of the 3D reconstruction model includes:
[0030] Obtaining a face sketch corresponding to the face image sample, wherein each face sketch has a corresponding sketch style;
[0031] Inputting the face sketch into the coarse reconstruction network and the fine reconstruction network for forward propagation, and predicting a predicted face reconstruction result of the face sketch;
[0032] Predicting the face sketch through the rough reconstruction network to obtain texture coefficients and spherical harmonic illumination coefficients, and determining rough texture features according to the texture coefficients, a preset three-dimensional average face texture, and a preset face texture vector basis;
[0033] Determine the illumination texture features of the face sketch based on the spherical harmonic illumination coefficient, and render the rough texture features and the illumination texture features by the differentiable renderer to obtain a plurality of rendered images;
[0034] Randomly select a sketch style from the sketch styles and input it into the sketch differentiable module, and perform prediction processing on the rendered image through the sketch differentiable module to obtain a plurality of predicted style sketch images;
[0035] A final loss function of the three-dimensional reconstruction model is constructed based on the predicted style sketch, the rendered image and the predicted face reconstruction result, and back-propagation is performed in the three-dimensional reconstruction model through the final loss function to update the parameters of the three-dimensional reconstruction model.
[0036] In some embodiments, the face image sample includes a training truth value, the training truth value includes face image key point information and face image segmentation information, and the final loss function of the three-dimensional reconstruction model constructed based on the predicted style sketch, the rendered image and the predicted face reconstruction result includes:
[0037] Constructing a three-dimensional sketch information supervision function according to the face sketch and the predicted style sketch, and constructing an image metric loss supervision function and a perceptual metric loss function according to the face image sample and the rendered image;
[0038] Determining an overall amplitude control coefficient used in the forward propagation process of the face sketch, and constructing a regularized loss function according to the overall amplitude control coefficient and the coefficient of the three-dimensional face deformable model;
[0039] Extract key points from the predicted face reconstruction result to obtain reconstruction key point information, and construct a key point supervision function based on the difference between the reconstruction key point information and the key point information of the face image;
[0040] Projecting the predicted face reconstruction result onto a two-dimensional image plane to obtain corresponding two-dimensional face image coordinates, dividing the two-dimensional face image coordinates into blocks to obtain two-dimensional face block information, and constructing a component segmentation distance supervision function according to a difference value between the two-dimensional face block information and the face image segmentation information;
[0041] The three-dimensional sketch information supervision function, the image metric loss supervision function, the perceptual metric loss function, the key point supervision function, the component segmentation distance supervision function and the regularization loss function are fused to obtain the final loss function of the three-dimensional reconstruction model.
[0042] In some embodiments, constructing a three-dimensional sketch information supervision function according to the face sketch and the predicted style sketch includes:
[0043] Determining a true sketch filtering mask map of the face sketch map, and determining a predicted sketch filtering mask map of the predicted style sketch map;
[0044] Determining an image metric loss value according to a difference between the predicted sketch filter mask image and the real sketch filter mask image, and weighting the image metric loss value by an image metric loss weight to obtain an image metric weighted result;
[0045] Calling a pre-trained face recognition network to perform feature extraction on the real sketch filter mask image to obtain a real mask feature vector, and performing feature extraction on the predicted sketch filter mask image through the pre-trained face recognition network to obtain a predicted mask feature vector;
[0046] Constructing a perceptual metric loss value according to the true mask feature vector and the predicted mask feature vector, and weighting the perceptual metric loss value by a perceptual metric loss weight to obtain a perceptual metric weighted result;
[0047] The weighted result of the image metric is summed with the weighted result of the perceptual metric to obtain a three-dimensional sketch information supervision function.
[0048] In some embodiments, constructing an image metric loss supervision function and a perceptual metric loss function according to the face image sample and the rendered image includes:
[0049] Determining a real image filtering mask map of the face image sample, and determining a rendered image filtering mask map of the rendered image;
[0050] Determining an image metric loss supervision function according to a difference between the rendered image filter mask map and the real image filter mask map;
[0051] Calling a pre-trained face recognition network to perform feature extraction on the real image filter mask image to obtain a real mask image feature vector, and performing feature extraction on the rendered image filter mask image through the pre-trained face recognition network to obtain a rendered mask image feature vector;
[0052] A perceptual metric loss function is constructed according to the real mask image feature vector and the rendered mask image feature vector.
[0053] The present invention also provides a texture-controllable 3D fine face reconstruction device based on sketch input, comprising the following modules:
[0054] An acquisition module is used to acquire a two-dimensional face sketch image to be reconstructed, and perform prediction processing on the two-dimensional face sketch image to be reconstructed through a rough reconstruction network to obtain coefficients of a three-dimensional face deformable model;
[0055] A determination module, configured to construct a pre-reconstruction result of the two-dimensional face sketch image to be reconstructed based on the coefficients of the three-dimensional face deformable model;
[0056] A feature extraction module is used to perform feature extraction on the pre-reconstruction result to obtain a three-dimensional position coding map of the pre-reconstruction result, and to perform feature extraction on the two-dimensional face sketch image to be reconstructed to obtain a sketch texture coding map;
[0057] A prediction module, used for predicting the sketch texture coding map and the three-dimensional position coding map through a fine reconstruction network to obtain a three-dimensional normal offset map;
[0058] A mapping module, used to determine a fine position information coding map based on the three-dimensional normal offset map and the three-dimensional position coding map, and perform mapping processing on the fine position information coding map to obtain a three-dimensional face reconstruction result;
[0059] The construction module performs texture construction on the three-dimensional face reconstruction result to obtain the final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed.
[0060] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a texture-controllable three-dimensional fine face reconstruction method based on sketch input as described above is implemented.
[0061] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described texture-controllable three-dimensional fine face reconstruction methods based on sketch input.
[0062] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned texture-controllable three-dimensional fine face reconstruction methods based on sketch input.
[0063] The texture-controllable three-dimensional fine face reconstruction method based on sketch input provided by the present invention divides the three-dimensional face reconstruction process into two stages: in the rough reconstruction stage, the two-dimensional face sketch image is processed by a rough reconstruction network and a three-dimensional face deformable model to obtain a pre-reconstruction result and enter the fine reconstruction stage; in the fine reconstruction stage, the three-dimensional normal offset is obtained by combining the sketch image features and the pre-reconstructed three-dimensional position features to perform fine reconstruction on the pre-reconstruction result to obtain a fine three-dimensional face reconstruction result. Finally, texture construction is performed to obtain the final reconstruction result. Therefore, the present invention only performs two-stage processing and reconstruction on the two-dimensional face sketch image, and does not use the three-dimensional image data matching the sketch image. Therefore, it can be seen that no three-dimensional image data training is required when training the rough reconstruction network and the fine reconstruction network, which reduces the difficulty of training. In addition, the present invention does not perform texture construction during the reconstruction process, but only performs texture construction on the three-dimensional face reconstruction result after the two-stage reconstruction process is completed, so as to perform the final texture improvement on the reconstruction result, which can reduce the interference caused by the reconstruction process to the texture, and thus can effectively control the face texture and improve the effect of three-dimensional face reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0065] Figure 1 It is a flow chart of a texture-controllable three-dimensional fine face reconstruction method based on sketch input provided by the present invention.
[0066] Figure 2 It is a schematic diagram of the framework of the texture-controllable three-dimensional fine face reconstruction method based on sketch input provided by the present invention.
[0067] Figure 3 It is a sample schematic diagram of the three-dimensional reconstruction model training data provided by the present invention.
[0068] Figure 4 It is a schematic diagram of the principle of the three-dimensional reconstruction model training process provided by the present invention.
[0069] Figure 5 It is a structural schematic diagram of a texture-controllable three-dimensional fine face reconstruction device based on sketch input provided by the present invention.
[0070] Figure 6 It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION
[0071] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0072] Combine the following Figure 1 The invention describes a texture-controllable three-dimensional fine face reconstruction method based on sketch input. Figure 1 FIG. 1 is a flow chart of a texture-controllable 3D fine face reconstruction method based on sketch input provided by the present invention. Figure 1 As shown, the method includes the following steps.
[0073] In step 101, a two-dimensional face sketch image to be reconstructed is obtained, and a prediction process is performed on the two-dimensional face sketch image to be reconstructed through a rough reconstruction network to obtain coefficients of a three-dimensional face deformable model.
[0074] First, obtain the 2D face sketch image to be reconstructed, denoted as S, as the 2D input image to be reconstructed. In addition, each 2D face sketch image to be reconstructed corresponds to a description text Text, which is used to describe the 3D face texture obtained after reconstruction for subsequent texture construction.
[0075] See also Figure 2 , Figure 2 : is a schematic diagram of the framework of the texture-controllable 3D fine face reconstruction method based on sketch input provided by the present invention. Figure 2 As shown in the figure, in the rough reconstruction stage of the reconstruction process, the 2D face sketch image to be reconstructed will be input as an input image into the rough reconstruction network, and the 2D face sketch image to be reconstructed will be predicted and processed by the rough reconstruction network to obtain the coefficients of the 3D Morphable Face Model (3DMM). Here, the rough reconstruction network can be a convolutional neural network, denoted as The coefficients of the 3D face deformable model specifically include the rotation angle , translation coefficient , identity shape coefficient and expression shape coefficient , these coefficients are the convolutional neural network In the process of predicting the reconstructed two-dimensional face sketch image, a convolutional neural network is used The result is obtained by encoding the two-dimensional face sketch image to be reconstructed.
[0076] Continue to see Figure 1 In step 102, based on the coefficients of the three-dimensional face deformable model, a pre-reconstruction result of the two-dimensional face sketch image to be reconstructed is constructed.
[0077] Here, the reconstruction process of the rough reconstruction stage is completed through the 3DMM model (i.e., three-dimensional deformable face model). Figure 2 As shown in the figure, after the coefficients of the 3DMM model are obtained through the rough reconstruction network prediction, it is necessary to first determine the average shape feature V of the three-dimensional face and the identity shape vector basis preset in the 3DMM model. , Expression Shape Vector Basis Next is the processing of the 3DMM model: first pass the identity shape coefficient Identity Shape Vector Basis The identity shape vector is obtained by weighting (i.e. ), and by the expression shape coefficient Vector basis for expression shapes The expression shape vector is obtained by weighting (i.e. ). Next, the average shape feature of the three-dimensional face, the identity shape vector and the expression shape vector are fused to obtain the fusion feature. The fusion process here is to directly add the feature vectors. Then, the rotation angle matrix The fused features are rotated and adjusted to obtain the adjusted features, where the rotation angle matrix According to the rotation angle The calculation process of the rotation adjustment here is to transform the rotation angle matrix Multiply it with the fusion feature to get the adjusted feature. Finally, the adjusted feature and the translation coefficient The fusion process is performed (the fusion process here is also an addition calculation) to obtain the pre-reconstruction result of the two-dimensional face sketch image to be reconstructed. This pre-reconstruction result is a rough three-dimensional face, recorded as , so the pre-reconstruction result The determination process can be expressed as shown in the following formula (1):
[0078] (1)
[0079] In the above formula (1), Indicates the rotation angle, represents the translation coefficient, represents the identity shape coefficient, represents the expression shape coefficient, V represents the average shape feature of the three-dimensional face, represents the identity shape vector basis, represents the expression shape vector basis, Represents the rotation angle matrix.
[0080] Continue to see Figure 1 In step 103, feature extraction is performed on the pre-reconstruction result to obtain a three-dimensional position coding map of the pre-reconstruction result, and feature extraction is performed on the two-dimensional face sketch image to be reconstructed to obtain a sketch texture coding map.
[0081] After the two-dimensional face sketch image to be reconstructed obtains the pre-reconstruction result through the rough reconstruction stage, it enters the fine reconstruction stage of the reconstruction process. In the fine reconstruction stage, the embodiment of the present invention adopts UV space expansion technology, such as Figure 2 As shown, UV expansion is first used to pre-reconstruct the results Perform feature extraction to obtain pre-reconstruction results 3D position coding map , and extract features from the reconstructed 2D face sketch image S to obtain the sketch texture coding map , this sketch texture encoding map Used to represent sketch texture information.
[0082] Continue to see Figure 1 In step 104, the sketch texture coding map and the three-dimensional position coding map are predicted and processed through a fine reconstruction network to obtain a three-dimensional normal offset map.
[0083] like Figure 2 As shown, determine the three-dimensional position coding map And sketch texture encoding map After that, the three-dimensional position coding map And sketch texture encoding map Input to the fine reconstruction network, through which the sketch texture encoding map is reconstructed And 3D position coding map Perform prediction processing to obtain a three-dimensional normal offset map, recorded as Here, the 3D normal offset map represents the detail displacement map in UV space, and the fine reconstruction network can be specifically an end-to-end deep neural network, denoted as .
[0084] In step 105, a fine position information coding map is determined based on the three-dimensional normal offset map and the three-dimensional position coding map, and mapping processing is performed on the fine position information coding map to obtain a three-dimensional face reconstruction result.
[0085] Since the three-dimensional position coding map It can only represent the rough three-dimensional information in UV space, so here we use the three-dimensional normal offset map Therefore, here we use the 3D normal offset map 3D Position Encoding Map Determine the detailed location information encoding map , and encode the fine position information Perform mapping processing to obtain the 3D face reconstruction result .
[0086] First determine the 3D normal offset map The corresponding overall amplitude control coefficient , and determine the three-dimensional position coding map Normal encoding , where the normal encoding is a fine reconstruction network Encoding the Sketch Texture And 3D position coding map During the prediction process, the network is reconstructed finely 3D Position Encoding Map The encoding process is used to characterize the three-dimensional position coding map The normal information of the corresponding 3D geometric surface.
[0087] Next, the overall amplitude control coefficient 3D Normal Offset Map Perform amplitude adjustment to obtain the amplitude adjustment offset diagram (denoted as ), where the amplitude adjustment process uses the overall amplitude control coefficient 3D Normal Offset Map At this point, it is necessary to further determine the amplitude adjustment offset diagram With normal encoding The Hadamard product of .
[0088] Finally, the Hadamard product combining the offset information and the amplitude information is integrated into the rough three-dimensional position coding map In the above, the Hadamard product is fused with the three-dimensional position coding map to obtain a fine position information coding map. , the fusion process here is also an addition operation. Therefore, the fine position information encoding map The determination process can be expressed as the following formula (2):
[0089] (2)
[0090] In the above formula (2), represents the overall amplitude control coefficient, represents the 3D normal offset map, Represents a three-dimensional position coding map Normal encoding, Represents the Hadamard product operation.
[0091] Determine the detailed location information encoding map Afterwards, the fine position information is encoded in the UV space through UV mapping. Mapping is performed to obtain a detailed 3D face reconstruction result. , thus the processing of the fine reconstruction stage is completed.
[0092] In step 106, texture construction is performed on the three-dimensional face reconstruction result to obtain a final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed.
[0093] pass Figure 1 In the above steps 101 to 105, after the two-dimensional face sketch image S to be reconstructed is subjected to the process of coarse reconstruction and fine reconstruction, a fine three-dimensional face reconstruction result is finally reconstructed. The last stage is the texture construction stage, where the 3D face reconstruction result is Texture construction is performed to obtain the final reconstruction result of the three-dimensional face of the two-dimensional face sketch image S to be reconstructed. The specific process of texture construction is described in detail below.
[0094] First, obtain the description text corresponding to the two-dimensional face sketch image to be reconstructed, that is, the description text Text used to describe the three-dimensional face texture, and determine the target face texture image that matches the description text Text from the preset face texture template library Lib The matching process here can be implemented by calling the Contrastive Language-Image Pre-Training (CLIP) model.
[0095] Specifically, if Figure 2 As shown, the CLIP model is called to encode the description text Text to obtain the text encoding feature, that is, the encoder for text in the CLIP model is used Encode the description text Text and obtain the text encoding feature, recorded as , the encoding process is expressed as follows formula (3):
[0096] (3)
[0097] For each face texture image in the preset face texture template library Lib , call the CLIP model for the face texture image Encode to obtain image encoding features, that is, use the encoder for the image in the CLIP model For face texture image Encode and obtain the image encoding features, recorded as , the encoding process is expressed as follows formula (4):
[0098] (4)
[0099] In the above formula (4), Represents the face texture image in the preset face texture template library Lib The total quantity.
[0100] Next, text-image similarity matching is performed to determine the feature similarity between the text encoding features and the image encoding features, and the face texture image with the largest feature similarity is determined as the target face texture image that matches the description text.
[0101] Here, due to the text encoding characteristics describing the text Text , while the face texture image Image encoding features have Therefore, the text encoding features are calculated here Separately encode features for each image The feature similarity is then sorted from high to low, and then the face texture image with the largest feature similarity (i.e., the first in the sorting) is selected. The target face texture image determined to match the description text Text , or you can select any one of the top k face texture images as the target face texture image ,in, .
[0102] Select the target face texture image After that, the target face texture image Texture estimation is performed to obtain a texture estimation map, a PCA texture map, a masked area of the texture estimation map, and a non-masked area of the PCA texture map.
[0103] Here, if Figure 2 As shown, the present invention adopts a 3D face reconstruction technology (specifically, it can be a 3D reconstruction algorithm or a pre-trained 3D reconstruction model, denoted as ) for the target face texture image The target face texture image Input to The texture is estimated in The incomplete texture expansion map A includes: texture estimation map , PCA texture map , Texture estimation map The mask area And PCA texture map The non-masked area , where the texture estimation map is the target face texture image use Unwrapping the texture in UV space, mask area The texture estimation map PCA texture map obtained by mask processing yes Using principal component analysis (PCA), that is The texture map is estimated using the texture vector basis of the 3DMM face model. is the same as the mask area The corresponding non-masked area, therefore, the relationship between the two satisfies " ”.
[0104] Next, the texture estimation map of the masked area is completed by using the PCA texture map corresponding to the non-masked area to obtain the texture expansion map of the target face texture image. Corresponding PCA texture map , for the mask area Texture estimation map Image completion is performed. The completion process may be to overlap and fuse the two images according to the mask area. In order to reduce the problem of unnatural transition of the fusion boundary edge, the embodiment of the present invention may use Poisson fusion or median filtering technology to process the texture image or the mask image, optimize the visual effect, and finally obtain a complete texture estimation image A. The process is shown in the following formula (5):
[0105] (5)
[0106] In the above formula (5), Represents the Hadamard product operation.
[0107] Finally, the texture unfolding image is matched with the 3D face reconstruction result to obtain the final 3D face reconstruction result of the 2D face sketch image to be reconstructed. Figure 2 As shown in FIG. 1 , after determining the complete texture expansion map A, the texture expansion map A is then fitted to the 3D face reconstruction result. On the surface, a fine three-dimensional face image with texture is obtained , as the final reconstruction result of the three-dimensional face of the two-dimensional face sketch image S to be reconstructed.
[0108] Through the embodiment of the present invention, the three-dimensional face reconstruction process is divided into two stages: in the rough reconstruction stage, the two-dimensional face sketch image is processed by the rough reconstruction network and the three-dimensional face deformable model to obtain the pre-reconstruction result and enter the fine reconstruction stage; in the fine reconstruction stage, the pre-reconstruction result is finely reconstructed in combination with the sketch image features, the three-dimensional position features and the three-dimensional normal offset features to obtain a fine three-dimensional face reconstruction result. Finally, texture construction is performed to obtain the final reconstruction result. Therefore, only the two-dimensional face sketch image is processed and reconstructed in two stages, and the three-dimensional image data matching the sketch image is not used. Therefore, it can be seen that no three-dimensional image data training is required when training the rough reconstruction network and the fine reconstruction network, which reduces the difficulty of training. In addition, texture construction is not performed in the face reconstruction process. Only after the two-stage reconstruction process is completed, the texture construction is performed on the three-dimensional face reconstruction result to perform the final texture improvement on the reconstruction result, which can reduce the interference of the reconstruction process on the texture, so that the face texture can be effectively controlled and the effect of three-dimensional face reconstruction can be improved.
[0109] Considering the diverse styles of face sketches, such as from realistic paintings with detailed shading to cartoons with simplified lines, this places high demands on the style robustness of 3D face reconstruction methods. However, existing 3D face reconstruction methods usually focus on processing sketches of frontal poses, or rely heavily on sketches with realistic shading as input, and thus have poor style robustness.
[0110] In order to improve the style robustness of the 3D face reconstruction method, the embodiment of the present invention also performs 3D reconstruction on the 2D face sketch image to be reconstructed by calling the 3D reconstruction model to obtain the final 3D face reconstruction result, wherein the 3D reconstruction model includes a coarse reconstruction network, a fine reconstruction network, a differentiable renderer, and a sketch differentiable module. Based on this, the embodiment of the present invention provides a training strategy for the 3D reconstruction model. Figure 4 , Figure 4 This is a schematic diagram of the principle of the 3D reconstruction model training process provided by the present invention. Figure 4 The following steps 201 to 206 are used to specifically introduce the training process of the 3D reconstruction model.
[0111] In step 201, a face sketch corresponding to a face image sample is obtained, wherein each face sketch has a corresponding sketch style.
[0112] First, we need to determine the training data. Here, we use the existing face dataset to construct the required training data. The training data includes face sketches of various sketch styles and the corresponding training truth values. Each face sketch has one or more corresponding sketch styles. Figure 3 , Figure 3is a schematic diagram of a sample of the 3D reconstruction model training data provided by the present invention, Figure 3 (a), (b), (c), (d), and (e) represent face sketches in five sketch styles. The training truth value consists of three parts: texture truth value, key point truth value, and two-dimensional set information truth value.
[0113] The method for obtaining a face sketch is as follows: for any two-dimensional face image sample I, use filtering technology, generative adversarial network and other technologies to generate two-dimensional face sketches of different styles, denoted as S. Here, this process can be differentiable, extracted and merged into a sketch differentiable module , and the obtained face sketches of different sketch styles are recorded as ,in, Indicates the corresponding sketch style, so the face sketch The method for obtaining is expressed as the following formula (6):
[0114] (6)
[0115] In the above formula (6), I represents the face image sample, It indicates the corresponding sketch style.
[0116] The training truth value includes texture truth value, key point truth value (i.e., face image key point information, denoted as lmk) and two-dimensional segmentation set information truth value (i.e., face image segmentation information, denoted as C), so the training truth value can be denoted as {I, lmk, C}. Among them, the texture truth value can be replaced by the face image sample I itself, because the original two-dimensional face image itself carries the real texture information of the face. The key point truth value lmk can use the key point detector to extract the 240 key points of the face in the face image sample I, while the two-dimensional segmentation set information truth value C uses the face segmentation algorithm to extract the face segmentation features in the face image sample I, such as Figure 3 As shown in (f) in .
[0117] In step 202, the face sketch is input into the coarse reconstruction network and the fine reconstruction network for forward propagation, and a predicted face reconstruction result of the face sketch is predicted.
[0118] After the training data is obtained in step 201, the face sketch in the training data and the corresponding training truth value are input into the 3D reconstruction model for training. First, the face sketch is input into the coarse reconstruction network and the fine reconstruction network for forward propagation to predict the face reconstruction result of the face sketch. Figure 4 As shown, here is the face sketch Rough reconstruction network input to 3D reconstruction model And fine reconstruction network The forward propagation is performed in the process to obtain the predicted face reconstruction result. The forward propagation process here is the reasoning process, and the reasoning order is also the execution order. Figure 1 The predicted face reconstruction result here includes the pre-reconstruction result obtained in step 102 (i.e., the rough three-dimensional face reconstruction result, denoted as ) and the three-dimensional face reconstruction result obtained in step 105 (i.e., the fine three-dimensional face reconstruction result, denoted as ).
[0119] In step 203, the face sketch is predicted by a rough reconstruction network to obtain texture coefficients and spherical harmonic illumination coefficients, and the rough texture features are determined according to the texture coefficients, a preset three-dimensional average face texture, and a preset face texture vector basis.
[0120] In the above forward propagation process (ie, the inference process), the embodiment of the present invention predicts the face sketch through a rough reconstruction network to obtain texture coefficients and spherical harmonic illumination coefficients, and determines the rough texture features based on the texture coefficients and the preset three-dimensional average texture of the face.
[0121] Specific as Figure 4 As shown, during the inference process, the coarse reconstruction network Sketching faces Perform prediction processing to obtain the correlation coefficient of the 3DMM model (i.e. the rotation angle , translation coefficient , identity shape coefficient , Expression shape coefficient ), and finely reconstruct the network The processing will also predict the 3D normal offset map And the corresponding overall amplitude control coefficient However, during the training process, when the rough reconstruction network predicts the face sketch, it also predicts the texture coefficient. Sketching of human face The texture coefficients in the 3DMM model will be further predicted Spherical Harmonic Illumination Coefficient , and further according to the texture coefficient The predicted rough texture features are obtained and recorded as . Rough texture features The prediction process can be expressed as the following formula (7):
[0122] (7)
[0123] In the above formula (7), T represents the average shape feature of the three-dimensional face preset in the 3DMM model. Represents the preset texture vector basis in the 3DMM model. Then it is the texture vector basis B abl The corresponding texture coefficient.
[0124] In step 204, the illumination texture features of the face sketch are determined based on the spherical harmonic illumination coefficients, and the rough texture features and the illumination texture features are rendered by a differentiable renderer to obtain a plurality of rendered images.
[0125] The spherical harmonic illumination coefficient is obtained by prediction in step 203 ,Next, the illumination texture features of the face sketch are determined based on the spherical harmonic illumination coefficients, and the rough texture features and the illumination texture features are rendered through a differentiable renderer, and a number of rendered images are obtained.
[0126] Here, the illumination texture feature is used to characterize the texture information with illumination. Spherical harmonics (SH) can be selected to model the illumination prediction. The calculation method of SH prediction can be found in the following formula (8):
[0127] (8)
[0128] In the above formula (8), B represents illumination information (such as rough texture information and grayscale information), N represents the surface normal vector of the three-dimensional object whose illumination is calculated (such as rough and fine three-dimensional face reconstruction results), represents the spherical harmonic vector basis, represents the spherical harmonic illumination coefficient, k represents the parameter number in the spherical harmonic function, and the maximum value is 9.
[0129] After calculating the illumination texture features, the rough texture features and illumination texture features are rendered by a differentiable renderer to obtain multiple rendered images. Figure 4 As shown, the differentiable renderer is an integrated module, denoted by Specifically, it can be a software renderer, which is equipped with a rendering algorithm and can use ray tracing to render. Here, the rough texture feature And lighting texture features are input to the differentiable renderer Rendering is performed and the correlation coefficient of the 3DMM model (i.e. the rotation angle) is added during the rendering process. , translation coefficient , identity shape coefficient , Expression shape coefficient ), 3D normal offset map , overall amplitude control coefficient , Spherical Harmonic Illumination Coefficient , texture coefficient By setting different lighting information B, multiple rendered images can be obtained ( Figure 4 The example shown in , , , Four rendered images), differentiable renderer The rendering process is as follows: Formula (9) to Formula (12):
[0130] (9)
[0131] (10)
[0132] (11)
[0133] (12)
[0134] In the above formulas (9) to (12), represents the pre-reconstruction result in the predicted face reconstruction result (i.e., the rough 3D face reconstruction result), and N represents the rough 3D face reconstruction result The normal vector of the surface, Represents the three-dimensional face reconstruction result in the predicted face reconstruction result (i.e., the refined three-dimensional face result ) the surface normal vector, represents the spherical harmonic illumination coefficient, represents the rough texture characteristics under illumination, Represents the grayscale feature vector under illumination.
[0135] In step 205, a sketch style is randomly selected from the sketch styles and input into the sketch differentiable module, and the rendered image is predicted by the sketch differentiable module to obtain a plurality of predicted style sketch images.
[0136] Here, considering various face sketches Various sketch styles are included , and the sketch style is an important factor affecting the style robustness of the 3D reconstruction model, so the 3D reconstruction model needs to learn these sketch styles in the process of predicting the style sketch. Here, Figure 4 As shown, before prediction, a sketch style is randomly selected from the sketch styles. Add sketch differentiable module , used to add sketch style to the rendered image, so that the sketch differentiable module can learn the sketch style. Then multiple rendered images ( , , , ) added to the Sketch Differentiable Module The prediction process is to further predict the sketch information of the sketch, thereby obtaining multiple predicted style sketches ( Figure 4 Four predicted style sketches are shown in FIG. , , , ), sketch differentiable module The prediction process is expressed as the following formula (13):
[0137] (13)
[0138] In the above formula (13), represents multiple rendered images, Represents the sketch style, j takes values of 1, 2, …, i, representing the jth sketch style, and i represents the number of sketch styles. Represents multiple predicted style sketches obtained by prediction, and n represents the types of four predicted style sketches, a, b, c, and d.
[0139] In step 206, a final loss function of the 3D reconstruction model is constructed based on the predicted style sketch, the rendered image, and the predicted face reconstruction result, and backpropagation is performed in the 3D reconstruction model through the final loss function to update the parameters of the 3D reconstruction model.
[0140] Through the above steps 201 to 205, the 3D reconstruction model finally obtains the predicted face reconstruction result and the predicted style sketch in the inference process. In the training process, the training target of the 3D reconstruction model (i.e., the final loss function) needs to be based on the predicted style sketch. , Render the image , predict the face reconstruction result (here we use the rough 3D face reconstruction result ) is constructed. In the embodiment of the present invention, the purpose of the final loss function is to train the rough reconstruction network in the 3D reconstruction model , fine reconstruction network And the overall amplitude control coefficient The parameters of these three parts are described below based on the predicted style sketch. , Render the image , Rough 3D face reconstruction results To construct the final loss function, the construction process is divided into steps 301 to 305 to introduce the construction process one by one.
[0141] In step 301, according to the face sketch Sketch diagram with forecast style , construct 3D sketch information supervision function , and according to the face image sample I and the rendered image , construct image metric loss supervision function And the perceptual metric loss function .
[0142] First, we introduce the construction of 3D sketch information supervision function Here, Figure 4 As shown in the figure, first determine the face sketch The real sketch filter mask M is used to determine the predicted style sketch Prediction sketch filter mask map However, in the determination process, it is necessary to obtain the mask obtained in the process of determining the two-dimensional segmentation set information truth value C included in the training truth value , and through a differentiable renderer The mask obtained during rendering , to calculate the filter mask map, the calculation formula of the determination process is as follows: Formula (14) and Formula (15):
[0143] (14)
[0144] (15)
[0145] In the above formulas (14) to (15), represents the mask obtained in the process of determining the true value C of the two-dimensional segmentation set information included in the training true value, Represented by a differentiable renderer The mask obtained during rendering, Represents the Hadamard product operation.
[0146] The mask image is then filtered based on the predicted sketch The difference between the real sketch filter mask M Determine the image metric loss value, here the Euclidean distance of the difference is calculated as the image metric loss value, and the image metric loss weight is The image metric loss value is weighted to obtain the image metric weighted result.
[0147] Next, call the pre-trained face recognition network Perform feature extraction on the real sketch filter mask M to obtain the real mask feature vector , and through the pre-trained face recognition network Filter the mask image for the predicted sketch Perform feature extraction to obtain the predicted mask feature vector The pre-trained face recognition network here is a face recognition network that has been trained in advance and does not need to update parameters. It is used as a tool for extracting features in the three-dimensional reconstruction model.
[0148] Finally, according to the true mask feature vector And the predicted mask feature vector Construct the perceptual metric loss value and pass the perceptual metric loss weight The perceptual metric loss value is weighted to obtain the perceptual metric weighted result. Then the image metric weighted result is summed with the perceptual metric weighted result to obtain the 3D sketch information supervision function , which can be specifically expressed as the following formula (16):
[0149] (16)
[0150] In the above formula (16), represents the image metric loss weight, represents the perceptual metric loss weight, The real mask feature vector representing the real sketch filtering mask map M, Represents the predicted sketch filter mask image The predicted mask feature vector of represents the pre-trained face recognition network, n represents the types of four predicted style sketches {a, b, c, d}, Represents the calculation of the predicted mask feature vector and the true mask feature vector The vector inner product of Represents the calculation of the Euclidean distance of the feature vector.
[0151] The following describes the face image sample I and the rendered image , construct image metric loss supervision function And the perceptual metric loss function specific process.
[0152] Metric loss supervision function for images First, determine the real image filtering mask map MI of the face image sample I, and determine the rendered image Rendered image filter mask map , here you only need to select One of the rendered images To calculate, , and then filter the mask map based on the rendered image The difference between the real image filtering mask MI -MI determines the image metric loss supervision function , here the Euclidean distance of the difference is calculated as the loss supervision function , the formula is expressed as follows formula (17):
[0153] (17)
[0154] Next, we construct the perceptual metric loss function , here we call the pre-trained face recognition network again Perform feature extraction on the real image filter mask image MI to obtain the real mask image feature vector , and through the pre-trained face recognition network Filter the rendered image with a mask Perform feature extraction to obtain the rendering mask image feature vector .
[0155] Finally, according to the true mask image feature vector And the rendering mask image feature vector Constructing the perceptual metric loss function , the specific calculation formula of the construction process is as follows (18):
[0156] (18)
[0157] In the above formula (18), The real mask image feature vector representing the real image filtering mask map MI, Represents the rendered image filter mask The rendered mask image feature vector, represents the pre-trained face recognition network, Represents the calculation of the rendering mask image feature vector and the true mask image feature vector The vector inner product of Represents the calculation of the Euclidean distance of the feature vector.
[0158] In step 302, the overall amplitude control coefficient used in the forward propagation process of the face sketch is determined, and a regularized loss function is constructed according to the overall amplitude control coefficient.
[0159] Here, in order to train the overall amplitude control coefficient in the 3D reconstruction model Therefore, during the training process, it is necessary to determine the face sketch The overall amplitude control coefficient used during forward propagation , and according to the overall amplitude control coefficient and the correlation coefficient of the 3DMM model (i.e. the rotation angle , translation coefficient , identity shape coefficient and the expression shape factor etc.) to construct the regularized loss function .
[0160] In step 303, key points are extracted from the predicted face reconstruction result to obtain reconstructed key point information, and a key point supervision function is constructed based on the difference between the reconstructed key point information and the key point information of the face image.
[0161] Next, the loss function is constructed based on the training truth values corresponding to the face sketches in the training data, and the training truth values {I, lmk, C} include the texture truth value (i.e., the face image sample I itself), the key point truth value (i.e., the face image key point information lmk) and the two-dimensional segmentation set information truth value (i.e., the face image segmentation information C). Among them, the face image sample I has been used to construct the image metric loss supervision function And the perceptual metric loss function Here, the key point supervision function is constructed based on the face image key point information lmk included in the training truth value .
[0162] The key point truth value lmk (i.e., the key point information of the face image) is the 240 key points extracted from the face image sample I using the key point detector. Here, we aim to predict the face reconstruction result (rough 3D face reconstruction result). ), the predicted key points can be directly extracted by using the topological consistency calibration of the key points provided in the 3D deformable face model, that is, the corresponding 240 key points are extracted as the reconstruction key point information, recorded as .
[0163] Then, based on the reconstructed key point information The difference between the key point information lmk of the face image , construct key point supervision function , here the key point supervision function is obtained by calculating the Euclidean distance of the difference , the formula is expressed as follows (19):
[0164] (19)
[0165] In the above formula (19), Indicates the calculation of Euclidean distance, and i represents the key point number.
[0166] In step 304, the predicted face reconstruction result is projected onto the two-dimensional image plane to obtain the corresponding two-dimensional face image coordinates, and the two-dimensional face image coordinates are divided into blocks to obtain the two-dimensional face image block information, and a component segmentation distance supervision function is constructed based on the difference between the two-dimensional face image block information and the face image segmentation information.
[0167] The training truth value {I, lmk, C} also includes the two-dimensional segmentation set information truth value (i.e., face image segmentation information C). Here, the component segmentation distance supervision function is constructed based on the two-dimensional segmentation set information truth value C included in the training truth value The true value of the two-dimensional segmentation set information is obtained by extracting the face segmentation features in the face image sample I using the face segmentation algorithm. This extraction process is performed in the two-dimensional image, so here we aim to predict the face reconstruction result (rough three-dimensional face reconstruction result). ) is also divided into blocks to obtain segmentation information.
[0168] First, considering the predicted face reconstruction result (rough 3D face reconstruction result ) is a three-dimensional stereogram, so here the predicted face reconstruction result is projected by the perspective projection camera Project it onto the two-dimensional image plane to get the two-dimensional face image coordinates , in this process the three-dimensional coordinates Mapping to two-dimensional face image coordinates .
[0169] Next, the coordinates of the two-dimensional face image The face is divided into blocks to obtain two-dimensional face block information. The block process (denoted as P) here is similar to the process of face segmentation algorithm. It uses the block topology consistency calibration provided in the three-dimensional deformable face model. The block standard is to segment the two-dimensional coordinate information according to the key facial features of the face, such as eyes, nose, skin and other facial features. In the block process, in order to reflect the emphasis of facial features, a weight is set for each block, denoted as .
[0170] Finally, according to the two-dimensional face segmentation information The difference between the face image segmentation information C and the component segmentation distance supervision function is constructed , where the difference value can be calculated by calculating the distance between each point set. The point set distance is used to describe the geometric information of the image block, denoted as , P is the above-mentioned block. Therefore, the two-dimensional face image block information The point set distance is denoted as , and the point set distance of face image segmentation information C is recorded as , so the difference is Finally, the component segmentation distance supervision function is constructed based on the difference value , the calculation formulas constructed are as follows: Formula (20) and Formula (21):
[0171] (20)
[0172] (twenty one)
[0173] In the above formulas (20) and (21), Indicates setting weights for each block, and P means setting weights for the left eye. , right eye , left eyebrow , right eyebrow , Upper lip , lower lip ,nose ,skin The process of dividing eight facial features into blocks, p represents the sequence number of the facial feature, Indicates calculating the Euclidean distance.
[0174] In step 305, the three-dimensional sketch information supervision function, the image measurement loss supervision function, the perceptual measurement loss function, the key point supervision function, the component segmentation distance supervision function and the regularization loss function are fused to obtain the final loss function of the three-dimensional reconstruction model.
[0175] After the above steps 301 to 305, a 3D sketch information supervision function is constructed. , image metric loss supervision function , Perceptual Metric Loss Function , key point supervision function , Part segmentation distance supervision function And the regularized loss function , then these loss functions are fused to obtain the final loss function L of the 3D reconstruction model. Before the fusion process, it is necessary to set corresponding weights for each loss function to distinguish the focus of the 3D reconstruction model learning objective. The corresponding loss functions are weighted by weights, and the weighted results are summed to obtain the final loss function L of the 3D reconstruction model. The calculation process is as follows:
[0176] (twenty two)
[0177] In the above formula (22), Represents the 3D sketch information supervision function The weight of Represents the image metric loss supervision function The weight of Represents the perceptual metric loss function The weight of Keypoint Supervision Function The weight of Part segmentation distance supervision function The weight of Represents the regularized loss function The weight of .
[0178] Thus, through the above steps 301 to 305, the final loss function L of the three-dimensional reconstruction model is finally constructed. Next, the final loss function is used to perform back propagation (i.e., the order opposite to the forward propagation reasoning order) in the three-dimensional reconstruction model to update the parameters of the three-dimensional reconstruction model.
[0179] Here, the final loss function L of the gradient optimization algorithm is back-propagated to the 3D reconstruction model for iteration. The back-propagation process is also the gradient propagation process, which can be seen here. Figure 4 , the final back propagation order of the loss function L is the sketch differentiable module , Differentiable Renderer , fine reconstruction network , Rough Reconstruction Network In the iterative back-propagation process, the final loss function L is optimized (updated) by the gradient optimization algorithm, thereby updating the weight parameters of the 3D reconstruction model. Here, the main purpose is to update the rough reconstruction network in the 3D reconstruction model. , fine reconstruction network And the overall amplitude control coefficient Parameters of these parts.
[0180] Each iteration is considered as a training round. Each iteration uses the gradient optimization algorithm to optimize (update) the final loss function L. When the final loss function L converges directly through continuous decreases through iterations or the training rounds reach the specified number of rounds, the training is stopped. The trained 3D reconstruction model can be directly used to reconstruct the 2D face sketch image S to be reconstructed into 3D to obtain the 3D face reconstruction result, that is, the fine 3D face reconstruction result. This concludes the introduction to the training process of the 3D reconstruction model.
[0181] The embodiment of the present invention designs a three-dimensional reconstruction model and provides a training strategy for the three-dimensional reconstruction model. Various sketch styles are added for rendering in the forward propagation reasoning process to construct the learning goal of the model. The model constructs an image metric loss supervision function and a perceptual metric loss function by rendering the image to learn the sketch style in the reconstruction process; learns the sketch feature information through the three-dimensional sketch information supervision function; learns the facial features of the image through the key point supervision function and the component segmentation distance supervision function, and accurately controls the overall amplitude of the reconstructed image through the regularization loss function. The three-dimensional reconstruction model trained in this way can overcome the problem of insufficient style robustness when constructing a three-dimensional face image from a two-dimensional sketch image, so that the reconstructed three-dimensional face image can retain the original sketch style in the sketch image, further improving the reconstruction effect. In addition, when training the three-dimensional reconstruction model, the two-dimensional face sketch corresponding to the face image sample is used, not the three-dimensional image data, which greatly reduces the difficulty of model training.
[0182] Moreover, the embodiment of the present invention does not perform texture construction during the face reconstruction process, but only performs texture construction on the three-dimensional face reconstruction result after the three-dimensional face image reconstruction process is completed, so as to perform final texture improvement on the reconstruction result, which can reduce the interference of the reconstruction process on the texture, thereby effectively controlling the face texture and improving the effect of three-dimensional face reconstruction.
[0183] The texture-controllable three-dimensional fine face reconstruction device based on sketch input provided by the present invention is described below. The texture-controllable three-dimensional fine face reconstruction device based on sketch input described below and the texture-controllable three-dimensional fine face reconstruction method based on sketch input described above can be referenced to each other.
[0184] Figure 5 is a structural schematic diagram of a texture-controllable three-dimensional fine face reconstruction device based on sketch input provided by the present invention, such as Figure 5As shown, the present invention provides a texture-controllable three-dimensional fine face reconstruction device based on sketch input, and the device specifically includes: an acquisition module 501, a determination module 502, a feature extraction module 503, a prediction module 504, a mapping module 505, and a construction module 506. Among them, the acquisition module 501 is used to obtain the two-dimensional face sketch image to be reconstructed, and predict the two-dimensional face sketch image to be reconstructed through a rough reconstruction network to obtain the coefficients of the three-dimensional face deformable model; the determination module 502 is used to construct a pre-reconstruction result of the two-dimensional face sketch image to be reconstructed based on the coefficients of the three-dimensional face deformable model; the feature extraction module 503 is used to extract features from the pre-reconstruction result to obtain a three-dimensional position coding map of the pre-reconstruction result, and extract features from the two-dimensional face sketch image to be reconstructed to obtain a sketch texture coding map; the prediction module 504 is used to predict the sketch texture coding map and the three-dimensional position coding map through a fine reconstruction network to obtain a three-dimensional normal offset map; the mapping module 505 is used to determine the fine position information coding map based on the three-dimensional normal offset map and the three-dimensional position coding map, and map the fine position information coding map to obtain the three-dimensional face reconstruction result; the construction module 506 performs texture construction on the three-dimensional face reconstruction result to obtain the final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed.
[0185] Specifically, the above-mentioned texture-controllable three-dimensional fine face reconstruction device based on sketch input provided by the present invention can implement all the method steps implemented by the above-mentioned texture-controllable three-dimensional fine face reconstruction method embodiment based on sketch input, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0186] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor (processor) 610 , a communication interface (Communications Interface) 620 , a memory (memory) 630 and a communication bus 640 , wherein the processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute a texture-controllable three-dimensional fine face reconstruction method based on sketch input, the method comprising: obtaining a two-dimensional face sketch image to be reconstructed, and predicting the two-dimensional face sketch image to be reconstructed through a rough reconstruction network to obtain coefficients of a three-dimensional face deformable model; constructing a pre-reconstruction result of the two-dimensional face sketch image to be reconstructed based on the coefficients of the three-dimensional face deformable model; performing feature extraction on the pre-reconstruction result to obtain a three-dimensional position coding map of the pre-reconstruction result, and performing feature extraction on the two-dimensional face sketch image to be reconstructed to obtain a sketch texture coding map; predicting the sketch texture coding map and the three-dimensional position coding map through a fine reconstruction network to obtain a three-dimensional normal offset map; determining a fine position information coding map based on the three-dimensional normal offset map and the three-dimensional position coding map, and mapping the fine position information coding map to obtain a three-dimensional face reconstruction result; performing texture construction on the three-dimensional face reconstruction result to obtain a three-dimensional face final reconstruction result of the two-dimensional face sketch image to be reconstructed.
[0187] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0188] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the texture-controllable three-dimensional fine face reconstruction method based on sketch input provided by the above methods. The method includes: obtaining a two-dimensional face sketch image to be reconstructed, and predicting the two-dimensional face sketch image to be reconstructed through a rough reconstruction network to obtain coefficients of a three-dimensional face deformable model; based on the coefficients of the three-dimensional face deformable model, constructing a pre-reconstruction result of the two-dimensional face sketch image to be reconstructed ; Perform feature extraction on the pre-reconstruction result to obtain a three-dimensional position coding map of the pre-reconstruction result, and perform feature extraction on the two-dimensional face sketch image to be reconstructed to obtain a sketch texture coding map; perform prediction processing on the sketch texture coding map and the three-dimensional position coding map through a fine reconstruction network to obtain a three-dimensional normal offset map; determine a fine position information coding map based on the three-dimensional normal offset map and the three-dimensional position coding map, and perform mapping processing on the fine position information coding map to obtain a three-dimensional face reconstruction result; perform texture construction on the three-dimensional face reconstruction result to obtain a final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed.
[0189] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the texture-controllable three-dimensional fine face reconstruction method based on sketch input provided by the above-mentioned methods, the method comprising: obtaining a two-dimensional face sketch image to be reconstructed, and predicting the two-dimensional face sketch image to be reconstructed through a rough reconstruction network to obtain coefficients of a three-dimensional face deformable model; constructing a pre-reconstruction result of the two-dimensional face sketch image to be reconstructed based on the coefficients of the three-dimensional face deformable model; and performing feature extraction on the pre-reconstruction result. , obtain a three-dimensional position coding map of the pre-reconstruction result, and perform feature extraction on the two-dimensional face sketch image to be reconstructed to obtain a sketch texture coding map; predict the sketch texture coding map and the three-dimensional position coding map through a fine reconstruction network to obtain a three-dimensional normal offset map; determine a fine position information coding map based on the three-dimensional normal offset map and the three-dimensional position coding map, and perform mapping processing on the fine position information coding map to obtain a three-dimensional face reconstruction result; perform texture construction on the three-dimensional face reconstruction result to obtain a final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed.
[0190] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0191] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A texture-controllable 3D fine face reconstruction method based on sketch input, characterized in that: The method comprises: Acquire a two-dimensional face sketch image to be reconstructed, and perform prediction processing on the two-dimensional face sketch image to be reconstructed through a rough reconstruction network to obtain coefficients of a three-dimensional face deformable model, wherein the coefficients of the three-dimensional face deformable model include a rotation angle, a translation coefficient, an identity shape coefficient, and an expression shape coefficient; Determining the average shape features of the three-dimensional human face, the identity shape vector basis, and the expression shape vector basis preset in the three-dimensional human face deformable model; The identity shape vector basis is weighted by the identity shape coefficient to obtain an identity shape vector, and the expression shape vector basis is weighted by the expression shape coefficient to obtain an expression shape vector; The three-dimensional average face shape feature, the identity shape vector and the expression shape vector are fused to obtain a fused feature, and the fused feature is rotated and adjusted by a rotation angle matrix to obtain an adjusted feature, wherein the rotation angle matrix is constructed according to the rotation angle; The adjustment feature is merged with the translation coefficient to obtain a pre-reconstruction result of the two-dimensional face sketch image to be reconstructed; Performing feature extraction on the pre-reconstruction result to obtain a three-dimensional position coding map of the pre-reconstruction result, and performing feature extraction on the two-dimensional face sketch image to be reconstructed to obtain a sketch texture coding map; Predicting the sketch texture coding map and the three-dimensional position coding map through a fine reconstruction network to obtain a three-dimensional normal offset map; Determining a fine position information coding map based on the three-dimensional normal offset map and the three-dimensional position coding map, and performing mapping processing on the fine position information coding map to obtain a three-dimensional face reconstruction result; The three-dimensional face reconstruction result is texture constructed to obtain a final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed.
2. The texture-controllable 3D fine face reconstruction method based on sketch input according to claim 1, characterized in that: The step of performing texture construction on the three-dimensional face reconstruction result to obtain a final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed includes: Acquire description text for describing three-dimensional facial texture, and determine a target facial texture image matching the description text from a preset facial texture template library; Performing texture estimation on the target face texture image to obtain a texture estimation map, a PCA texture map, a masked area of the texture estimation map, and a non-masked area of the PCA texture map; By using the PCA texture map corresponding to the non-masked area, the texture estimation map of the masked area is completed to obtain a texture unfolding map of the target face texture image; The texture unfolding image is texture-fitted with the three-dimensional face reconstruction result to obtain a final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed.
3. The texture-controllable 3D fine face reconstruction method based on sketch input according to claim 2, characterized in that: The step of determining a target face texture image matching the description text from a preset face texture template library includes: Calling the CLIP model to encode the description text to obtain text encoding features; For each face texture image in the preset face texture template library, the CLIP model is called to encode the face texture image to obtain image coding features; The feature similarity between the text encoding feature and the image encoding feature is determined, and the facial texture image with the greatest feature similarity is determined as the target facial texture image that matches the description text.
4. The texture-controllable 3D fine face reconstruction method based on sketch input according to claim 1, characterized in that: The determining of the fine position information coding map based on the three-dimensional normal offset map and the three-dimensional position coding map includes: Determining an overall amplitude control coefficient corresponding to the three-dimensional normal offset map, and determining a normal code of the three-dimensional position code map; Performing amplitude adjustment on the three-dimensional normal offset map by using the overall amplitude control coefficient to obtain an amplitude-adjusted offset map, and determining a Hadamard product of the amplitude-adjusted offset map and the normal code; The Hadamard product is fused with the three-dimensional position coding map to obtain a fine position information coding map.
5. The texture-controllable 3D fine face reconstruction method based on sketch input according to claim 1, characterized in that: The final reconstruction result of the 3D face is obtained by calling the 3D reconstruction model to perform 3D reconstruction on the 2D face sketch image to be reconstructed, the 3D reconstruction model includes a coarse reconstruction network, a fine reconstruction network, a differentiable renderer and a sketch differentiable module, and the training process of the 3D reconstruction model includes: Obtaining a face sketch corresponding to the face image sample, wherein each face sketch has a corresponding sketch style; Inputting the face sketch into the coarse reconstruction network and the fine reconstruction network for forward propagation, and predicting a predicted face reconstruction result of the face sketch; Predicting the face sketch through the rough reconstruction network to obtain texture coefficients and spherical harmonic illumination coefficients, and determining rough texture features according to the texture coefficients, a preset three-dimensional average face texture, and a preset face texture vector basis; Determine the illumination texture features of the face sketch based on the spherical harmonic illumination coefficient, and render the rough texture features and the illumination texture features by the differentiable renderer to obtain a plurality of rendered images; Randomly select a sketch style from the sketch styles and input it into the sketch differentiable module, and perform prediction processing on the rendered image through the sketch differentiable module to obtain a plurality of predicted style sketch images; A final loss function of the three-dimensional reconstruction model is constructed based on the predicted style sketch, the rendered image and the predicted face reconstruction result, and back-propagation is performed in the three-dimensional reconstruction model through the final loss function to update the parameters of the three-dimensional reconstruction model.
6. The texture-controllable 3D fine face reconstruction method based on sketch input according to claim 5, characterized in that: The face image sample includes a training truth value, the training truth value includes face image key point information and face image segmentation information, and the final loss function of the three-dimensional reconstruction model constructed based on the predicted style sketch, the rendered image and the predicted face reconstruction result includes: Constructing a three-dimensional sketch information supervision function according to the face sketch and the predicted style sketch, and constructing an image metric loss supervision function and a perceptual metric loss function according to the face image sample and the rendered image; Determine the overall amplitude control coefficient and the coefficient of the three-dimensional face deformable model used in the forward propagation process of the face sketch, and construct a regularized loss function according to the overall amplitude control coefficient and the coefficient of the three-dimensional face deformable model; Extract key points from the predicted face reconstruction result to obtain reconstruction key point information, and construct a key point supervision function based on the difference between the reconstruction key point information and the key point information of the face image; Projecting the predicted face reconstruction result onto a two-dimensional image plane to obtain corresponding two-dimensional face image coordinates, dividing the two-dimensional face image coordinates into blocks to obtain two-dimensional face block information, and constructing a component segmentation distance supervision function according to a difference value between the two-dimensional face block information and the face image segmentation information; The three-dimensional sketch information supervision function, the image metric loss supervision function, the perceptual metric loss function, the key point supervision function, the component segmentation distance supervision function and the regularization loss function are fused to obtain the final loss function of the three-dimensional reconstruction model.
7. The texture-controllable 3D fine face reconstruction method based on sketch input according to claim 6, characterized in that: The constructing of a three-dimensional sketch information supervision function according to the face sketch and the predicted style sketch includes: Determining a true sketch filtering mask map of the face sketch map, and determining a predicted sketch filtering mask map of the predicted style sketch map; Determining an image metric loss value according to a difference between the predicted sketch filter mask image and the real sketch filter mask image, and weighting the image metric loss value by an image metric loss weight to obtain an image metric weighted result; Calling a pre-trained face recognition network to perform feature extraction on the real sketch filter mask image to obtain a real mask feature vector, and performing feature extraction on the predicted sketch filter mask image through the pre-trained face recognition network to obtain a predicted mask feature vector; Constructing a perceptual metric loss value according to the true mask feature vector and the predicted mask feature vector, and weighting the perceptual metric loss value by a perceptual metric loss weight to obtain a perceptual metric weighted result; The weighted result of the image metric is summed with the weighted result of the perceptual metric to obtain a three-dimensional sketch information supervision function.
8. The texture-controllable 3D fine face reconstruction method based on sketch input according to claim 6, characterized in that: The step of constructing an image metric loss supervision function and a perceptual metric loss function according to the face image sample and the rendered image includes: Determining a real image filtering mask map of the face image sample, and determining a rendered image filtering mask map of the rendered image; Determining an image metric loss supervision function according to a difference between the rendered image filter mask map and the real image filter mask map; Calling a pre-trained face recognition network to perform feature extraction on the real image filter mask image to obtain a real mask image feature vector, and performing feature extraction on the rendered image filter mask image through the pre-trained face recognition network to obtain a rendered mask image feature vector; A perceptual metric loss function is constructed according to the real mask image feature vector and the rendered mask image feature vector.
9. A texture-controllable 3D fine face reconstruction device based on sketch input, characterized in that: The device comprises: An acquisition module is used to acquire a two-dimensional face sketch image to be reconstructed, and predict the two-dimensional face sketch image to be reconstructed through a rough reconstruction network to obtain coefficients of a three-dimensional face deformable model, wherein the coefficients of the three-dimensional face deformable model include a rotation angle, a translation coefficient, an identity shape coefficient, and an expression shape coefficient; A determination module, used to determine the average shape features of the three-dimensional face, the identity shape vector basis, and the expression shape vector basis preset in the three-dimensional face deformable model; The identity shape vector basis is weighted by the identity shape coefficient to obtain an identity shape vector, and the expression shape vector basis is weighted by the expression shape coefficient to obtain an expression shape vector; The three-dimensional average face shape feature, the identity shape vector and the expression shape vector are fused to obtain a fused feature, and the fused feature is rotated and adjusted by a rotation angle matrix to obtain an adjusted feature, wherein the rotation angle matrix is constructed according to the rotation angle; The adjustment feature is merged with the translation coefficient to obtain a pre-reconstruction result of the two-dimensional face sketch image to be reconstructed; A feature extraction module is used to perform feature extraction on the pre-reconstruction result to obtain a three-dimensional position coding map of the pre-reconstruction result, and to perform feature extraction on the two-dimensional face sketch image to be reconstructed to obtain a sketch texture coding map; A prediction module, used for predicting the sketch texture coding map and the three-dimensional position coding map through a fine reconstruction network to obtain a three-dimensional normal offset map; A mapping module, used to determine a fine position information coding map based on the three-dimensional normal offset map and the three-dimensional position coding map, and perform mapping processing on the fine position information coding map to obtain a three-dimensional face reconstruction result; The construction module performs texture construction on the three-dimensional face reconstruction result to obtain the final three-dimensional face reconstruction result of the two-dimensional face sketch image to be reconstructed.
