Weakly supervised 3d head reconstruction method based on style transfer constraint

Abstract

The application discloses a method for three-dimensional head reconstruction in a single picture. The application comprises: a style transfer model, which is used for providing high-dimensional supervision for a three-dimensional head reconstruction model; and the three-dimensional head reconstruction model, which is used for extracting facial features from an input face photo and estimating a group of FLAME model parameters. Finally, the FLAME model parameters are decoded into a three-dimensional head mesh model. The application can effectively estimate the shape and expression of a head from a single photo, and the modeling accuracy of facial expression and contour is improved. Finally, the head model reconstructed by decoding the FLAME parameters can be used for three-dimensional modeling, digital people, face recognition and other purposes.

Description

Technical Field

[0001] This invention relates to the field of pedestrian re-identification technology, and more specifically, to a weakly supervised 3D head reconstruction method based on style transfer constraints. Background Technology

[0002] Since Vetter and Blanz first proposed the algorithm for reconstructing 3D faces from a single image, 3D head / face reconstruction technology has developed rapidly. 3D head / face reconstruction technology is being applied in various fields, such as face recognition, VR, digital humans, expression transfer, and voice-driven facial animation. To make the reconstruction problem easier to solve, most existing methods combine prior knowledge of the shape and texture of the head model with deep learning models, freeing them from the dependence on a large number of real 3D scans when training neural networks. With further technological advancements, the first 3D Morphable Models (3DMMs), introduced in 2009, achieved a breakthrough in the application of face reconstruction technology. Using 3DMM models can provide a fast and robust solution for the application of human head reconstruction algorithms.

[0003] 3DMMs typically reduce the shape and texture of a 3D model to a set of parameters that satisfy a specific distribution. We can describe a 3D face mesh and its texture using a set of parameters and latent texture codes, and then combine projection and lighting to render a complete head model. More importantly, 3DMMs offer the possibility of solving the "inverse problem" of reconstructing models from photographs. Methods using 3DMM models simplify the process of directly reconstructing complex face meshes by regressing parameters controlling their shape, texture, and environment parameters using image feature information. The key to the effective application of this type of method is that the decisive factors in the final scene generation (lighting, model shape, camera, and pose) are separable. The process of solving this inverse problem is called "model fitting."

[0004] However, existing single-image face reconstruction algorithms still have much room for improvement in areas such as facial detail reconstruction, facial depth reconstruction, and how to apply head reconstruction to commercial scenarios. While some algorithms have discussed and explored how to accurately recover details such as facial wrinkles and natural depressions, these methods are limited by the parameter variation space controlling the face shape in 3DMMs during coarse reconstruction, failing to recover true facial depth variations. Furthermore, the training model may converge to local optima during the convergence process. Adding high-dimensional, global constraints can help the model converge faster and avoid getting trapped in local optima. Summary of the Invention

[0005] The problems addressed by this invention are how to prevent 3D head reconstruction networks from focusing too much on local features extracted from images, how to improve the overall accuracy of the model, and how to optimize the loss function, thereby further improving the robustness of the network.

[0006] To address the above problems, this invention provides a weakly supervised 3D head reconstruction method based on style transfer constraints, comprising the following steps:

[0007] S1: Obtain an unlabeled dataset of face images Where N represents the number of images in the dataset, and Xi represents the i-th face image in the dataset. Each image is preprocessed based on the landmark detection results of the face.

[0008] S2: Training the style transfer model. Face photos are obtained from an open-source dataset, and corresponding pencil-drawn style face photos are hand-drawn by artists, ultimately resulting in a dataset of paired face-pencil-drawn style photos. Where M represents the number of images in the dataset, Yi and Y′i represent the i-th face image and its corresponding pencil-drawing style image in the dataset, respectively, and the face image and the pencil-drawing style image are preprocessed.

[0009] S3: Construct a deep learning model for pencil drawing style transfer. Input the style transfer training data E obtained from the preprocessing in S2 into the network, and use supervised learning methods to enable the network model to learn to convert head photos into head images in pencil drawing style.

[0010] S4: Construct a 3D head reconstruction deep learning model. Input the preprocessed face image training data D obtained in S1 into the network to estimate the FLAME model parameters and texture latent encoding. The FLAME parameters are decoded into a head mesh using the FLAME model, and the texture latent encoding is decoded into a texture map through the learned decoding network.

[0011] S5: Using predefined UVs and a differentiable renderer, the head mesh and texture map decoded in S4 are combined and rendered onto the input image to obtain the rendered face image. 2D keypoint loss, pixel distance between the original and rendered images on the face, and the original and rendered images are respectively input into the style transfer network trained in S2 to calculate pencil drawing style feature loss, etc., supervising the learning of the head reconstruction network and texture reconstruction mesh.

[0012] S6: Use a trained network model to reconstruct a 3D human head. Input a photo of the face to be reconstructed, and output a 3D human head mesh model and texture.

[0013] In the methods described above, during the pencil-drawing style transfer process for face images, the feature extractor is typically able to effectively extract the contour information and local depth information of the face surface. Furthermore, pixel differences at the background, skin tone, and pore levels in the original image are filtered out during the pencil-drawing style transfer process. Therefore, pencil-drawing style transfer is suitable for providing high-dimensional supervision for the face in 3D head reconstruction in this method. Combining face landmark constraints and head re-rendering pixel loss can effectively reconstruct the depth variations and detailed textures of the face surface.

[0014] Further, in S1, the preprocessing steps involve using landmark estimation and a face segmentation model to process the image, obtaining the sample's landmark and face mask. The landmark is used to calculate the bounding box of the face portion of the input sample and to establish an affine transformation to the fixed-size image network input. This affine transformation is then applied to the original image and the face mask to obtain the fixed-size network input sample.

[0015] In S2, the face photos were obtained from an open-source dataset, and the corresponding pencil-drawing style photos were drawn by professional artists. During the preprocessing stage, the paired face-pencil-drawing style photos underwent data augmentation processes such as rotation, scaling, and horizontal flipping. To prevent background interference, the background, except for the head images, was removed using ModNet, retaining only the portrait portion.

[0016] In S3, this method employs a pix2pix network architecture as the main structure of the pencil drawing style transfer model. The input photo is used by a UNet network as a generator to predict and generate pencil drawing style images. The generated pencil drawing style images and real pencil drawings by artists are compared using a CGA discriminant network to calculate the game loss, and a regularization loss is added to improve image quality. After training, this network can effectively transform face images into pencil drawing style face images.

[0017] In S4, the steps include:

[0018] S41: Input the processed face image into a ResNet50 network to estimate the FLAME model parameters (including shape parameters, pose parameters), texture latent code, camera extrinsic parameters, and lighting parameters.

[0019] S42: Decode the parameters estimated in the previous step. The FLAME parameters are decoded into a head mesh model through the FLAME model, and the texture latent code is decoded into a fixed-size texture map with predefined UV mapping through a decoding network. Other estimated parameters can be directly input into the renderer without decoding.

[0020] In S5, the steps include:

[0021] S51: Using predefined UVs, the texture is combined with the head mesh model. The texture map, head mesh model, and other environmental parameters are input into the PyTorch3D renderer for rendering to obtain a head texture map. The texture map and the original image are multiplied pixel by pixel using a mash mask to extract the face, and the pixel-by-pixel L1 distance is calculated as the loss function.

[0022] S52: After reconstruction, the head mesh calculates the 3D key points of the head through a key point regression module, and obtains the 2D key points by projecting them onto the 2D camera plane through camera parameters. The landmark prediction results of the input image are used as ground truth to calculate the landmark loss.

[0023] S53: Extract the face from both the rendered head image and the original input image using a mask. Then, input these images into the pencil drawing style transfer model trained in S3, resulting in rendered pencil drawing head images and original image pencil drawing head images. After extracting the head portion again using a mask, calculate the pixel-wise product and the pixel-wise L1 distance as the style transfer loss.

[0024] As a preferred approach, another way to implement this loss is to input the rendered image and the original image into Unet respectively, and then extract the hidden code (also known as features) of the 8th layer respectively, and directly calculate the L1 distance of the hidden code of the layer as the feature loss.

[0025] S6: The face photo to be reconstructed is preprocessed according to the steps in S1 to extract a fixed-size head image, which is then input into the 3D head reconstruction model. The model obtains FLAME parameters and a decoded texture map through inference. The FLAME model is then used to decode the FLAME to obtain a head mesh model. Finally, the texture map and the head mesh model are combined using predefined UVs to form a textured 3D head model.

[0026] The beneficial effect of this invention is that, based on the original sparse constraints on the location of facial key points and the re-rendering pixel loss that mainly affects texture, a style transfer model is introduced as a feature extractor to extract high-dimensional features and serve as weak supervision for head reconstruction, thereby improving the convergence speed and reconstruction accuracy of the model.

[0027] Meanwhile, since the 3D head reconstruction model itself also has the ability to extract facial features from photos, the head reconstruction model and the style transfer model can actually form a set that provides mutual constraints. By alternately freezing the network weights of one model and optimizing the weights of the other model, it is possible to achieve a cyclical improvement in the accuracy of both the style transfer model and the head reconstruction model. Attached Figure Description

[0028] Figure 1This is a flowchart of the present invention;

[0029] Figure 2 This is a diagram of the three-dimensional head reconstruction network architecture of the present invention;

[0030] Figure 3 The process of calculating the loss during the training phase of a 3D head reconstruction model. Detailed Implementation

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0033] Example

[0034] This embodiment provides a weakly supervised 3D head reconstruction method based on style transfer constraints, such as... Figure 1 As shown, this method includes the following steps:

[0035] S1: Obtain an unlabeled dataset of face images Where N represents the number of images in the dataset, and Xi represents the i-th face image in the dataset. Each image is preprocessed based on the landmark detection results of the face.

[0036] S2: Obtain paired face photos and pencil-drawn style face photos Where M represents the number of images in the dataset, and Yi and Y′i represent the i-th face image and its corresponding pencil-drawing style image in the dataset, respectively. The face image and the pencil-drawing style image are preprocessed.

[0037] S3: Construct a deep learning model for pencil drawing style transfer, input preprocessed training data into the network, and use supervised learning methods to enable the network model to learn to convert head photos into head images in pencil drawing style.

[0038] S4: As Figure 2 As shown, a 3D head reconstruction deep learning model is constructed. Preprocessed training data is input into the network to estimate FLAME model parameters (shape parameter β, expression parameter ψ, pose parameter θ), texture latent encoding α, and other environmental parameters required for rendering (lighting parameter l, camera parameter c, etc.). The FLAME parameters are decoded into a head mesh using the FLAME model, and the texture latent encoding is decoded into a texture map through the learned decoding network.

[0039] S5: Using predefined UVs, the head mesh and texture map are combined and input along with other environmental parameters into a differentiable renderer. The head mesh is then rendered onto the input image to obtain the rendered face image. The learning of the head reconstruction network and texture reconstruction mesh is supervised using 2D keypoint loss, pixel distance between the original and rendered images on the face, and input of the original and rendered images into a style transfer network to calculate pencil drawing style feature loss.

[0040] S6: Use a trained network model to reconstruct a 3D human head. Input a photo of the face to be reconstructed, and output a 3D human head mesh model and texture.

[0041] In S1, the face landmark prediction model uses the open-source face-alignment algorithm to predict 68 key points of the face. To ensure the stability of the face segmentation algorithm, the input image is first scaled to 500*500 pixels, and after being input into the model, a 500*500 pixel face mask is output, which is then scaled back to the original image size. A bounding box that can enclose all key points is calculated using landmarks, and expanded by 15% in each direction (up, down, left, and right) to ensure that the cropped image contains the entire head. Finally, padding and scaling are applied to the cropped head to obtain a 224*224 pixel network model input.

[0042] In S2, paired face photos—pencil drawing style photos—are subjected to data augmentation processes such as rotation, scaling, and horizontal flipping. The augmented face images are then segmented using ModNet for foreground image segmentation. The image data is cropped and scaled to a size of 224*224.

[0043] In S3, the generator G in the pencil drawing style transfer network uses Unet as its main framework, with an input size of 224*224 pixels. The network outputs a pencil drawing style face image of the same size. The generated pencil drawing style face image is divided into 70*70 pixel image blocks, which are then sequentially fed to the discriminator D. The discriminator D calculates the conditional GAN ​​loss function by combining the feature vectors extracted from the original image and the generated image through the discriminator network. This loss function can be expressed as...

[0044]

[0045] Where x is the input image, y is the target image corresponding to x, and z is random noise.

[0046] To improve the quality of the generated image, an L1 loss constraint is directly applied to both the target style map and the generated style map, which can be expressed as follows:

[0047]

[0048] In summary, the total loss of the pencil drawing style transfer network can be expressed as:

[0049]

[0050] Where λ is a hyperparameter used to control the effect of L1 loss. In this embodiment, it is set to 0.5.

[0051] In S4, the fixed-size face image processed in S1 is input into a ResNet50 model to estimate the FLAME model parameters (100-dimensional shape parameter β, 50-dimensional expression parameter ψ, and 12-dimensional pose parameter θ), the 50-dimensional texture latent code, camera extrinsic parameters, and lighting parameters. A total of 236 intermediate parameters are estimated. The 50-dimensional texture latent code is decoded into a 256*256 texture map using a decoder. UVs are mappings from model vertex indices to the 2D texture map, manually specified by the engineer.

[0052] In S5, such as Figure 3 As shown, the steps for calculating network supervision loss include:

[0053] S51: The texture is combined with the head mesh model using predefined UVs. The texture map, head mesh model, and other environmental parameters are input into the renderer for rendering. Since the rasterization process in traditional renderers is usually non-differentiable, this example uses the differentiable rendering pipeline provided by PyTorch3d to ensure that gradients can be smoothly transferred from the rendered image to the reconstruction network. After the rendered texture map and the original image are multiplied pixel-by-pixel using a mask, the face is extracted, and the pixel-by-pixel L1 distance is calculated as the loss function. The specific calculation of the loss function is as follows:

[0054] L pho =||V I ⊙(II r )||

[0055] Among them, V I This refers to the face mask obtained during the preprocessing stage.

[0056] S52: After reconstruction, the head mesh calculates 3D keypoints of the head using a keypoint regression module, and then projects these keypoints onto a 2D camera plane using camera parameters to obtain 2D keypoints. The landmark prediction results of the input image are used as ground truth to calculate the landmark loss. The landmark loss is calculated as follows:

[0057]

[0058] Where, k i For realistic 2D landmarks of human faces, sΠ(M i The surface mesh M of the human head model generated by )-t calculation iThe predicted 2D landmark is obtained by projecting it onto the camera space.

[0059] S53: Extract the face from both the rendered head image and the original input image using a mask, then input them into the pencil drawing style transfer model p trained in S3 to obtain the rendered pencil drawing head image p(I). r The image is a pencil drawing of a human head, p(I). The head portion is extracted again using a mask, and then the pixel-wise product is calculated. The pixel-wise L1 distance is then calculated as the style transfer loss. The style transfer loss is calculated as follows:

[0060] L p =||V I ⊙(p(I)-p(I r ))||

[0061] Among them, V I This refers to the face mask obtained during the preprocessing stage.

[0062] Another way to implement this loss is to input the rendered image and the original image into the Unet generator network, respectively, and then extract the hidden code (also known as features) of layer 8. The L1 distance of the hidden code of this layer is then directly calculated as the feature loss. The loss is calculated as follows:

[0063]

[0064] Where p(I) r p(I) is the feature vector extracted by the style transfer network from the input image of the rendered head. p(I) is the feature vector extracted by the style transfer network from the input image of the original head.

[0065] In this example, the first method for calculating style transfer loss is used.

[0066] S6: Preprocess the face photo to be reconstructed according to the steps in S1, extracting a fixed-size head image and inputting it into the 3D head reconstruction model. The model infers the FLAME parameters and the decoded texture map, then uses the FLAME model to decode the FLAME to obtain the head mesh model. The FLAME model uses the β parameter to control the shape features of the head, the ψ parameter to control the facial expression features, and the θ parameter to control the rotation of the head relative to four joints. Its decoding formula is as follows:

[0067]

[0068] Where θ, β, and ψ are FLAME model parameters, and T is the statistical average template model generated when constructing 3D human head model data. For linear blendshape function, Head posture correction function, B E (ψ;ε) is the facial expression correction function. The correction function is used to compensate for defects in the facial mesh generated by blendshape.

[0069] The final texture map and the head mesh model are combined into a textured 3D head model using predefined UVs.

[0070] Following the steps of the pedestrian re-identification method described above, the learning rate was set to 0.01, and the entire training process terminated after approximately 2.5 epochs, with approximately 75,000 iterations per epoch and validation performed every 5,000 iterations. The training dataset was obtained from the VGGFACE2 dataset. The VGGface2 dataset contains approximately 2,000,000 face photos. During the preprocessing stage, photos that were too large or too small were filtered, and photos without detectable landmarks were removed. The test set used the NoW dataset. The NoW dataset includes photos of 352 objects and a real scanned 3D mesh. According to the validation results, the median vertex distance error was approximately 1.33, the mean vertex error was approximately 1.68, and the mean squared error was approximately 1.43.

Claims

1. A weakly supervised 3D head reconstruction method based on style transfer constraints, characterized in that, Includes the following steps: S1: Obtain a face picture dataset without labels wherein represents the number of pictures in the dataset, represents the i-th face picture in the dataset, Preprocess each picture according to the landmark detection result of the face. S2: Obtain face photos from open source datasets and manually draw corresponding pencil drawing style face photos by artists, finally obtain a pair of face-pencil drawing style photo dataset wherein represents the number of pictures in the dataset, , respectively represent the th face picture and its corresponding pencil drawing style picture in the dataset, and pre-process the face photo and the pencil drawing style photo; S3: Construct a pencil drawing style transfer deep learning model, and the style transfer training data obtained by preprocessing in S2 An input network is used to make the network model learn to convert a human face photo into a human head picture in a pencil drawing style using a supervised learning method. S4: Constructing a 3D human head reconstruction deep learning model, training the human face picture data obtained by preprocessing in S1 Inputting the 3D human head reconstruction network, estimating the FLAME model parameters and the texture latent code; the FLAME parameters are decoded into a human head mesh using the FLAME model, and the texture latent code is decoded into a texture map through a learned decoding network; S5: Use predefined UVs and differentiable renderers to combine the head mesh and texture map and render them onto the input image to obtain the rendered face image; use 2D key point loss, pixel distance between the original image and the rendered image on the face, and input the original image and the rendered image into the style transfer network to calculate the pencil drawing style feature loss to supervise the learning of the head reconstruction network and texture reconstruction mesh. S51: Use predefined UVs to combine textures with a human head mesh model. Input the texture map, human head mesh model and other environmental parameters into the PyTorch3D renderer for rendering to obtain a human head texture map. After performing pixel-by-pixel product operation on the texture map and the original image through a mash mask, the face is extracted separately, and the pixel-by-pixel L1 distance is calculated as the loss function. S52: After reconstruction, the head mesh calculates the 3D key points of the head through a key point regression module, and obtains the 2D key points by projecting them onto the 2D camera plane through camera parameters. The landmark prediction result of the input image is used as the ground true to calculate the landmark loss. S53: Extract the face from the rendered head image and the input original image using a mask, and then input them into the pencil drawing style transfer model trained in S3 to obtain the rendered-pencil drawing head image and the original-pencil drawing head image. After extracting the head portion again using the mask, the pixel-wise product is calculated, and the pixel-wise L1 distance is calculated as the style transfer loss. S6: Use the trained network model to reconstruct a 3D human head. Input the face photo to be reconstructed and output a 3D human head mesh model and texture.

2. The weakly supervised 3D head reconstruction method based on style transfer constraint of claim 1, wherein, The preprocessing in step S1 involves performing landmark detection and face mask segmentation on the input face image, determining a bounding box using the detected landmarks, and cropping the face image using the bounding box. The preprocessing in step S2 involves rotating, scaling, and horizontally flipping the paired face photos—pencil drawing style photos; to prevent background interference, a foreground segmentation algorithm is applied to the head images, retaining only the portrait portion.

3. The weakly supervised 3D head reconstruction method based on style transfer constraints according to claim 1, characterized in that, The head reconstruction deep learning model and texture latent code decoder in step S4 are composed of ResNet50 models.

4. The weakly supervised 3D head reconstruction method based on style transfer constraints according to claim 3, characterized in that, Step S4 includes the following sub-steps: S41: Input the processed face image into a ResNet50 network to estimate the FLAME model parameters, including shape parameters, expression parameters, pose parameters, texture latent code, camera extrinsic parameters, and lighting parameters; S42: Decode the parameters estimated in the previous step; the FLAME parameters are decoded into a head mesh model through the FLAME model, and the latent texture code is decoded into a fixed-size texture map through a decoding network, whose UV mapping relationship is predefined; other estimated parameters can be directly input into the renderer without decoding.

5. The weakly supervised 3D head reconstruction method based on style transfer constraints according to claim 1, characterized in that, In step S3, the pencil drawing style transfer deep learning model consists of the Unet model and the PatchGAN model.

Images