Single-image-based three-dimensional face reconstruction and editing method

Abstract

The present invention provides a single-image-based three-dimensional face reconstruction and editing method, comprising: extracting features by means of a pre-trained EG3D network, so as to generate a preliminary three-dimensional face; using an inversion module to map a monocular image to a latent code space, and optimizing a latent code to generate a realistic three-dimensional face image; making use of a depth estimation technology and multi-view projection to create a pseudo multi-view image; then, by means of semantic segmentation and expression feature extraction, fusing the features to generate a comprehensive representation; according to editing requirements, optimizing and adjusting the latent code, so as to implement customized editing; and finally, a generator outputting an edited face image. The present invention can enable more efficient, flexible and high-quality three-dimensional face reconstruction and editing, thereby effectively overcoming the defects of high cost, low universality and lack of customized editing in existing face reconstruction technologies.

Description

A single-image-based three-dimensional face reconstruction and editing method TECHNICAL FIELD

[0001] The present application relates to the field of computer vision processing, in particular to a single-image-based three-dimensional face reconstruction and editing method. BACKGROUND

[0002] Digital human is a virtual avatar of human, which can be used as an identity feature of human and as a medium for interaction between users. In the fields of entertainment and education, the demand for digital human is showing a huge growth. The application of digital human in the fields of live broadcast, game and other entertainment fields, and media fields such as news dissemination is becoming a popular trend. The creation, editing and application of digital human involve three-dimensional modeling, computer graphics and computer vision technology. There are generally the following methods to realize three-dimensional face reconstruction. One is to use a three-dimensional scanning device such as structured light to scan the head and then realize reconstruction, but this method is high in economic cost. Two is to ask professional designers to model the three-dimensional face through three-dimensional modeling software, but this method is also high in cost and low in universality. Three is the parameter fitting method based on 3DMM (3D Morphable Face Model, 3DMM) face parameterization model, but this method is difficult to render realistic images. Four is to use the neural radiance field three-dimensional reconstruction technology to reconstruct the multi-view face, which can render realistic images, but needs dense multi-view images and is difficult to edit. Therefore, a technology method which is strong in universality, low in cost and based on monocular or sparse view to realize realistic three-dimensional face reconstruction and customized editing has become an attractive innovation point.

[0003] Three-dimensional reconstruction from a single view to a three-dimensional face is a difficult problem in computer vision, because the recovery from single two-dimensional information to three-dimensional information is itself an ill-posed problem. Therefore, it is often necessary to introduce appropriate prior information to reasonably supplement the unknown information. In recent years, the application of generative adversarial networks (GAN) has made great progress in image generation in the field of computer vision. The generation of two-dimensional pictures has achieved high resolution, high quality, and high fidelity. Given that the rendering of high-resolution images of three-dimensional faces often requires the involvement of professional designers, and the three-dimensional face model based on the 3DMM method is difficult to render high resolution. Therefore, researchers try to use GAN to synthesize three-dimensional perceptual face multi-view images, and a hybrid explicit-implicit three-plane three-dimensional representation is introduced into GAN (EG3D), which achieves the best three-dimensional perceptual face image generation without sacrificing expressiveness. With this realistic three-dimensional perceptual face image generation model containing strong prior information, researchers apply the inverse algorithm of the generative adversarial network to this three-dimensional perceptual generative adversarial network, and realize the single-purpose three-dimensional face reconstruction by reversing the single image to the latent code space.

[0004] However, the current three-dimensional face reconstruction based on the inverse algorithm of the generative adversarial network cannot achieve perfect results, and mainly has the following shortcomings. First, the optimization target of the inverse is only on the image level loss, which is easy to cause over-fitting in the optimization process, resulting in a decline in the generation effect and low editability of the latent code. Second, the inverse needs to adjust the parameters of the entire model, so the parameters of the entire generative adversarial network generator model need to be saved for each reconstructed face, resulting in huge memory overhead. Finally, the reconstructed face lacks personalized customization editing operations, and due to the coupling of the generative adversarial network latent code space, the existing three-dimensional face editing method often affects other irrelevant features while editing certain facial features.

[0005] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The main purpose of the present application is to overcome the defects of the above background technology, provide a three-dimensional face reconstruction and editing method based on a single image, and provide a more efficient, flexible and high-quality three-dimensional face reconstruction and editing method to solve the defects of current face reconstruction technology, such as high cost, low universality and lack of customized editing. In order to achieve the above purpose, the present application adopts the following technical solutions:

[0007] A three-dimensional face reconstruction and editing method based on a single image, comprising:

[0008] For monocular face images, a pre-trained three-dimensional perception face generative adversarial EG3D network is used for feature extraction, the EG3D network generates three orthogonal plane features through a generator with multiple convolution modules, and combines camera parameters to perform three-dimensional rendering on the generated three-plane features through a neural body renderer to generate a preliminary three-dimensional face representation;

[0009] Through an inversion module, the input monocular face image is mapped to a latent code space to obtain a latent code, and the obtained latent code is used to generate a three-dimensional face image through the forward process of the EG3D network, and the difference between the generated three-dimensional face image and the original monocular face image is calculated, wherein the loss calculation is used for latent code optimization;

[0010] Based on the preliminary three-dimensional face representation and the latent code, depth estimation technology is used to generate depth information from the original monocular face image, and the monocular face image is projected to different camera perspectives using the depth information to generate pseudo multi-view images;

[0011] Based on the preliminary three-dimensional face representation and the latent code, a semantic segmentation network is used to extract semantic features of the face, an expression feature extraction network is used to obtain expression features, and the extracted semantic features and expression features are fused with the features of the pseudo multi-view images to generate a comprehensive feature representation;

[0012] According to the editing requirements, the latent code is optimized, adjusted and regularized through a latent code optimization network using a loss function;

[0013] Based on the comprehensive feature representation and the optimized, adjusted and regularized latent code, three-dimensional face model editing operations are performed;

[0014] Based on the edited feature representation and the optimized latent code, a final edited face image is generated by the generator.

[0015] A computer program product comprising a computer program, which, when executed by a processor, implements the three-dimensional face reconstruction and editing method based on a single image.

[0016] The present application has the following advantages:

[0017] The present application provides a three-dimensional face reconstruction and editing method based on a single image, which can provide more efficient, flexible and high-quality three-dimensional face reconstruction and editing, effectively overcoming the defects of current face reconstruction technology, such as high cost, low universality and lack of customized editing.

[0018] The main technical advantage of the present application is to provide a cost-effective, widely applicable single-image-based three-dimensional face reconstruction and editing method, which effectively overcomes many limitations in the prior art. Compared with traditional methods, the present application uses a pre-trained EG3D network and an advanced GAN inversion algorithm to achieve efficient and high-fidelity reconstruction from a single face image to a three-dimensional face model. The present application generates pseudo-multi-view images through depth estimation technology, enriches the perspective information of the face model, and enhances the model's ability to capture semantic and expression features through semantic segmentation and expression feature extraction networks. In addition, through innovative latent code optimization and regularization strategies, the present application not only improves the stability and controllability of the editing process, but also achieves high decoupling between features, allowing precise editing of specific facial features without affecting other irrelevant features, thereby enhancing the flexibility of personalized customization. In addition, the present application further reduces storage consumption through a local parameter optimization strategy, while improving the efficiency of feature decoupling during the editing process, maintaining the accuracy and editability of face reconstruction, and solving the problems of large memory overhead and lack of personalized editing capability in traditional methods.

[0019] Other benefits of the embodiments of the present application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0020] FIG. 1 is a schematic diagram of the three-plane feature generated by the three-dimensional perception generative adversarial network EG3D generator of the embodiments of the present application.

[0021] FIG. 2 is a schematic diagram of the EG3D using three-plane features for neural rendering according to the embodiments of the present application.

[0022] FIG. 3 is a schematic diagram of the inversion architecture based on the EG3D generator according to the embodiments of the present application.

[0023] FIG. 4 is a schematic diagram of the synthesized pseudo-multi-view according to the embodiments of the present application.

[0024] FIG. 5 is a schematic diagram of the inversion process supervision according to the embodiments of the present application.

[0025] FIG. 6 is a schematic diagram of the face semantic editing framework based on semantic segmentation mask reconstruction according to the embodiments of the present application.

[0026] FIG. 7 is a schematic diagram of the reconstructed face expression editing framework according to the embodiments of the present application. DETAILED DESCRIPTION

[0027] The embodiments of the present application are described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope of the present application and its applications.

[0028] Referring to FIGS. 1-7, the embodiments of the present application provide a single-image-based three-dimensional face reconstruction and editing method, comprising the following steps:

[0029] S1. For monocular face images, a pre-trained three-dimensional perception face generative adversarial EG3D network is used for feature extraction. The EG3D network generates three orthogonal plane features through a generator StyleGAN2 with multiple convolution modules, and combines camera parameters to perform three-dimensional rendering on the generated three-plane features through a neural body renderer to generate a preliminary three-dimensional face representation.

[0030] S2. Through an inverse module, the input monocular face image is mapped to a latent code space to obtain a latent code. The obtained latent code is used to generate a three-dimensional face image through the forward process of the EG3D network, and the difference between the generated three-dimensional face image and the original monocular face image is calculated, wherein the latent code is optimized through loss calculation.

[0031] S3. Based on the preliminary three-dimensional face representation and the latent code, depth information is generated from the original monocular face image using depth estimation technology, and the monocular face image is projected to different camera perspectives using the depth information to generate pseudo multi-view images.

[0032] S4. Based on the preliminary three-dimensional face representation and the latent code, semantic features of the face are extracted using a semantic segmentation network, expression features are obtained using an expression feature extraction network, and the extracted semantic features and expression features are fused with the features of the pseudo multi-view images to generate a comprehensive feature representation.

[0033] S5. According to the editing requirements, the latent code is optimized, adjusted and regularized using a loss function through a latent code optimization network.

[0034] S6. Based on the comprehensive feature representation and the optimized, adjusted and regularized latent code, editing operations are performed on the three-dimensional face model.

[0035] S7. Based on the edited feature representation and the optimized latent code, a final edited face image is generated by the generator.

[0036] In a preferred embodiment, in step S2, the EG3D network further optimizes the details of the generated image through a super-resolution module.

[0037] In a preferred embodiment, in step S2, the loss function used for loss calculation includes a pixel-level loss and a perception loss to calculate the difference between the generated image and the supervised image; a latent code regularization loss is also introduced to calculate the two-norm distance between the optimized latent code and the average latent code in the latent code space, limiting the variation range of the latent code and avoiding overfitting.

[0038] In a preferred embodiment, in step S2, local parameter optimization is implemented through a local parameter optimization strategy, and the network parameter optimization is set in the two synthesis modules closer to the output end in the StyleGAN2 backbone.

[0039] In a preferred embodiment, step S3 specifically comprises:

[0040] Using the input monocular face image as supervision, the generator and latent code are initially optimized;

[0041] Based on the optimized generator, the density of the image space is predicted, and the depth information of the monocular image is extracted;

[0042] Using the extracted depth information, the monocular image is projected under different camera perspectives to generate pseudo multi-view images;

[0043] The pseudo multi-view images obtained by projection are fused with the images generated by the generator under the same perspective to form pseudo supervised views;

[0044] Based on the pseudo multi-view projection, a mask is generated by Gaussian blur processing, and the mask is normalized to between 0 and 1;

[0045] The Gaussian blurred mask is rounded up and down to produce two masks for supervision, ensuring the accuracy of the supervision information;

[0046] When fusing the generated image and the projected image, the mask is used to discard inaccurate supervision information in the edge area;

[0047] Using the generated mask to supervise part of the projected image and the generated image, the pseudo view supervision loss is calculated.

[0048] In the pseudo multi-view generation stage, the monocular image is effectively used to generate depth information and create a pseudo multi-view image set for three-dimensional face reconstruction, while the mask processing optimizes the supervised image, improving the accuracy and reliability of the reconstruction process.

[0049] In a preferred embodiment, step S5 comprises: using a loss function including L1, L2 loss and perceptual loss to evaluate feature differences, and using latent code and three-plane feature regularization loss, identity consistency and non-changing area loss to enhance the decoupling between edited features and irrelevant features; limit the change of latent code through latent code regularization loss function to maintain stability during editing; execute optimization algorithm, update latent code according to the calculation result of loss function, and continuously adjust latent code through iteration process until the features of generated image meet the editing requirements.

[0050] In a preferred embodiment, step S6 comprises: analyzing the face image using a pre-trained semantic segmentation network to identify and segment different semantic regions; using an expression feature extraction network to identify the expression features of the face and convert them into editable parameters; generating a corresponding editing mask based on the results of semantic segmentation and expression feature extraction; applying the generated mask to the optimized latent code, editing specific semantic regions based on the editing mask, and adjusting the expression features of the face image according to the extracted expression parameters and the generated mask; and fusing the results of semantic editing and expression editing.

[0051] In a preferred embodiment, the method further comprises the following steps:

[0052] S9. Smoothly transition between latent codes of different expressions through latent code interpolation techniques to generate animations of face expression changes.

[0053] In a more preferred embodiment, step S9 comprises: performing linear interpolation on the latent codes in different editing states to perform smooth interpolation; using the intermediate latent code obtained by interpolation to generate a transition animation of the face expression or feature; outputting the generated final face image as the result of the three-dimensional face reconstruction and editing process; and if a transition animation is generated, outputting the animation sequence to show the process of face expression changes.

[0054] The main technical advantage of the present application is to provide a cost-effective, widely applicable three-dimensional face reconstruction and editing method based on a single image, which effectively overcomes many limitations in the prior art. Compared with traditional methods, the present application uses a pre-trained EG3D network and advanced GAN inversion algorithm to achieve efficient and high-fidelity reconstruction from a single face image to a three-dimensional face model. The present application generates pseudo-multi-view images through depth estimation technology, enriches the perspective information of the face model, and enhances the model's ability to capture semantic and expression features through semantic segmentation and expression feature extraction networks. In addition, through innovative latent code optimization and regularization strategies, the present application not only improves the stability and controllability of the editing process, but also achieves high decoupling between features, allowing precise editing of specific face features without affecting other irrelevant features, thereby enhancing the flexibility of personalized customization. Furthermore, the present application further reduces storage consumption through local parameter optimization strategies, while improving the efficiency of feature decoupling during editing, maintaining the accuracy and editability of face reconstruction, and solving the problems of large memory overhead and lack of personalized editing capability in traditional methods.

[0055] The feature decoupling is realized by independent feature extraction, optimization and editing operations in the three-dimensional face reconstruction and editing method of the present application, ensuring that other irrelevant features remain unchanged when adjusting specific facial features. In the feature extraction and three-dimensional rendering step, the three independent plane features generated by the EG3D network lay the foundation for decoupling. The editing feature extraction and fusion step further realizes the decoupling of features by independently extracting and integrating facial features through a semantic segmentation and expression feature extraction network. In the latent code optimization step, the network focuses on maintaining the independence between features and reducing the impact on other features by regularizing and adjusting the latent code. The editing operation implementation step allows precise editing of specific features without affecting other areas, directly reflecting the application of feature decoupling. Finally, in the result generation and output step, the edited face image should show modifications to specific features while maintaining the original state of other features, intuitively reflecting the effect of feature decoupling. In the animation effect implementation step, the latent code interpolation technique is used to generate animations of expression changes, allowing smooth transitions between different expression states without disturbing other features of the face, further demonstrating the advantages of feature decoupling. The inversion process of the present application discards the traditional mapping of noise to latent codes and directly uses latent code input generators to reduce the loss between generated images and supervised images by optimizing the latent code. The introduction of latent code regularization loss avoids overfitting and optimizes the parameters of the latent code and generator network.

[0056] Some embodiments of the present application are further described below.

[0057] The embodiments of the present application are based on a single-image-based three-dimensional face reconstruction and feature decoupling editing method, and an inversion architecture is constructed based on a pre-trained three-dimensional perception face generative adversarial network (EG3D). The inversion process mainly optimizes the parameters of the latent code and the generator to reduce the loss between the images generated by the EG3D generator and the supervised images. The monocular image is projected to pseudo multi-view of other views through depth as the supervised image of inversion, and a latent code regularization loss is designed in the inversion process, and the network parameter optimization is limited to part of the modules in the generator network. In addition, a general face editing architecture is also designed, the first half of which is the same as the architecture of the inversion process, and in the second half, a network with prior knowledge can be introduced as supervision, which calculates the loss by extracting the features of the generated image and the specified target features, and only optimizes the latent code, realizing the editing of the reconstructed face. In this editing process, the latent code and three-plane feature regularization loss, identity and non-changing area loss are used to improve the decoupling of edited features and irrelevant features. In the implementation stage, a semantic segmentation network and an expression feature extraction are specifically introduced to realize the semantic editing and expression editing of the reconstructed face based on the mask.

[0058] In some embodiments, the present application provides a single-image-based three-dimensional face reconstruction and its feature decoupling editing method, comprising the following steps: first step: based on the pre-trained three-dimensional perception face generative adversarial network (EG3D), an inversion architecture is constructed. The main process of the architecture is to discard the noise to latent code mapping network, and directly input the latent code into multiple serial convolution modules (StyleGAN2) to generate three plane features. Then based on the camera parameters and the three orthogonal plane features in space, the volume rendering is performed, and finally the output generated image is calculated with the supervised image to optimize the latent code and the parameters of the generator network; the second step is based on the constructed inversion architecture, taking the monocular face image and its pseudo multi-view image set projected by depth information to other viewpoints as the supervised image, designing the loss function to optimize the network parameters and the latent code, and completing the three-dimensional reconstruction of the single-view face. In this process, on the basis of image color and perception loss, the present embodiment designs a latent code regularization loss, and designs a local inversion strategy to limit the network parameter optimization to part of the StyleGAN2 network, so as to improve the editability of the latent code and the storage consumption of the reconstruction; the third step is based on the inversion of the optimized network and latent code parameters, and the given generator different camera parameters are used to perform the forward process of the generator to synthesize the new view of the face; the fourth step is to construct the face editing architecture designed by the present application, the first half of which is the same as the inversion process architecture, and the latent code is input into the generator to output the rendered face image. Then the output image is input into any pre-trained network module with prior knowledge, and the image features extracted by the supervised network are calculated with the set features to optimize the latent code. In addition to the commonly used one-norm and cross-entropy loss and other feature losses, the present application designs the latent code and three-plane feature regularization loss, identity and non-changing area loss in this editing process, to improve the decoupling of the edited features and irrelevant features. For this universal editing method, the present application respectively introduces a semantic segmentation network and an expression feature extraction to realize the semantic editing and expression editing of the reconstructed face based on the mask; the fifth step is to input the latent code optimized by the editing framework into the generator to generate the edited face image at any viewpoint. In addition, the interpolation operation between different edited latent codes of the face can realize the smooth transition of the face animation.

[0059] In some embodiments, the first step specifically includes: constructing an image-to-latent code inversion architecture based on a pre-trained three-dimensional perception face generative adversarial network (EG3D). The total inversion structure of the model is shown in FIG. 3. The inversion process mainly consists of several parts: input latent code, EG3D generator, generated image and supervised image, and loss function between them. The EG3D generator includes two main modules: a three-plane feature generator composed of multiple convolution modules, and a neural volume renderer based on three-plane features and camera parameters, as shown in FIGS. 1 and 2, respectively. In this inversion process, the mapping network that maps noise to the latent code space is discarded, and the loss between the supervised image and the generated image is reduced by directly optimizing the latent code with an input dimension of 512x14 and the parameters of the network, thereby realizing the mapping of monocular images to the latent code space.

[0060] In some embodiments, the second step specifically includes the monocular face image to be reconstructed. And the method uses the pseudo view generated based on the original monocular image at a new view as the supervised image. The specific steps are shown in FIG. 4: first, use the input monocular image as supervision to optimize the generator and latent code with fewer rounds, then extract the depth of the monocular image based on the density predicted by the generator, and project the monocular image to other camera views based on the depth to obtain the pseudo view, and then optimize the generator G es The images generated at the same view are fused as pseudo-supervised views.

[0061] The loss calculation in the second step specifically includes L1 and L2 loss and perceptual loss between the generated image and the supervised image in terms of pixel color. At the same time, in order to avoid overfitting during the process of mapping the image to the latent code space, resulting in defects such as distortion artifacts of the reconstructed face at other new views, and low editability of the latent code, the method designs a latent code regularization loss, i.e., adding the two-norm distance between the optimized latent code and the average latent code in the latent code space to the loss calculation. For the synthetic pseudo view supervision loss calculation method, as shown in FIG. 5, the method does not directly calculate the loss between the synthesized pseudo view and the generated image, but uses the projection mask obtained based on the pseudo multi-view projection to supervise the generated image part of G es , and discards the inaccurate supervision information in the edge area during the fusion of the two images. Pseudo view supervision and latent code regularization both effectively avoid overfitting of face reconstruction.

[0062] The parameter optimization part includes the latent code with a dimension of 512x14 and the parameters of the generator. The general method fine-tunes the overall parameters of the generator, but this method results in the need to save about 120 MB of generator parameters for each reconstructed face, causing large memory consumption. Therefore, the method proposes to limit the optimized parameters of the network to the partial convolution modules in the three-plane feature synthesis module during the inversion process, greatly reducing the storage consumption required for face reconstruction. In the specific implementation, the method sets the network parameter optimization in the two modules SynthBlock256 2 and SynthBlock 128 2 , successfully reducing the storage consumption from 120.27 MB to 12.22 MB.

[0063] In some embodiments, the third step specifically includes: based on the optimized latent code and generator parameters of the first and second steps, inputting the latent code into the generator with different camera parameters of the viewpoints, so as to generate any new view of the reconstructed face.

[0064] In some embodiments, the fourth step specifically includes: designing a general reconstructed face editing algorithm framework, the first half of the framework is the same as the inversion framework, and any feature extraction network with prior knowledge can be introduced as supervision in the second half. The network calculates the loss by extracting the features of the generated image and the specified target features, and only optimizes the latent code, thereby realizing the editing of the reconstructed face. In the specific implementation, the method introduces a semantic segmentation prior model and a face parameterized feature extraction model as prior supervision, realizes the semantic editing and expression editing of the reconstructed face based on the segmentation mask, and designs latent code regularization, three-plane feature regularization, identity consistency, and non-changing area four losses to improve the decoupling between the edited features and irrelevant features.

[0065] In the decoupling loss design, directly using the prior model to supervise the optimization of the latent code will often cause irrelevant features to follow the editing features. To this end, the method designs four kinds of losses to improve the decoupling between features. In the latent code regularization loss, the optimized latent code and the original latent code that has not been optimized are calculated to calculate the two norm loss, so as to limit the change of unnecessary latent code. In the three-plane feature regularization loss, the three-plane features generated by the original latent code that has not been optimized are calculated to calculate the two norm loss with the three-plane features generated by the latent code in the optimization process, so as to limit the change of unnecessary features. In the identity consistency loss, the identity features extracted from the image generated by the latent code in the optimization process are calculated to calculate the norm loss with the identity features of the original input monocular image, so as to limit the change of identity information. In the non-changing area loss, a semantic segmentation model is used to segment the image into several semantic regions. When editing features, for example, editing the mouth area, other areas are combined into a mask, and the color and perception loss of the original image and the image generated by the latent code in the optimization process are calculated based on the mask to limit the change of unnecessary areas.

[0066] In the reconstruction face-based segmentation mask semantic editing framework design, a face semantic segmentation model is introduced as a prior model applied in the general reconstruction face editing framework. Its structure is shown in Figure 6. The semantic segmentation model is used as a prior model to segment the image generated by the generator, and the cross-entropy loss is calculated with the set semantics to optimize the latent code. At the same time, the loss calculation also adds the several decoupling loss functions.

[0067] In the reconstruction face expression editing framework design: the structure of expression editing is shown in Figure 7, a face expression parameter extractor is introduced and applied in the general reconstruction face editing framework. It is used to extract the expression parameters of the image generated by the generator, and the norm loss is calculated with the expression parameters extracted from the set expression image to optimize the latent code. Since the human expression is closely related to the mouth, the method also introduces the mouth area of the face supervised by the semantic segmentation model. At the same time, the loss calculation also adds the several decoupling loss functions.

[0068] In some embodiments, the fifth step specifically includes: based on the latent code obtained by editing the face according to the editing framework, inputting the camera parameters of different viewpoints into the generator to generate edited face images under different viewpoints. Compared with other methods in the reconstruction face semantic editing, the method can basically keep other features of the face unchanged when editing a certain feature of the face, while the method of IDE3D will cause changes in other irrelevant features. In addition, for the same face, linear interpolation is performed on different edited latent codes, and a series of interpolation latent codes are input into the generator to generate smooth transition images of two editing effects.

[0069] The details of the specific embodiments of the present application are further described below.

[0070] The overall network structure of the reverse process is shown in FIG. 3. A three-dimensional perceptual face generation model EG3D is used as a three-dimensional prior model. An optimization-based method is used to map the target face image to w + Latent code space.

[0071] Regularization of latent code: To solve the problem of low editability of the latent code, a regularization loss function of the latent code is proposed in the method, which is added to the optimization-based reverse process, thereby limiting the deviation between the latent code obtained by reverse and the main distribution of the latent code. In addition, since the latent code used during training has the same value in 14 channels, the regularization term proposed in the method can also constrain the distance between each channel. The regularization loss function contains two terms. The first term is to limit the two-norm distance between the optimized latent code and the average latent code. The second term is to limit the two-norm distance between each channel of the optimized latent code w^+ and the first channel of the latent code. The mathematical expression of the regularization loss function is as follows:

[0072] where w + is the latent code obtained by reverse, w + is the average latent code, and w + is the first channel of the latent code copied 14 times. + is updated at the same time . λ1 and λ2 are the weight hyperparameters of the regularization term.

[0073] Supervision of pseudo multi-view: As shown in FIG. 5, first, the input monocular image is used as supervision to optimize the generator and the latent code in a small number of rounds. Then, the density of the image space is predicted based on the optimized generator and the latent code in a small number of rounds, the depth of the monocular image is extracted, and the monocular image is projected to other camera perspectives based on the depth and the optimized generator G es The images generated in the same perspective are fused as pseudo-supervised views. The method does not directly calculate the loss between the synthesized pseudo-view and the generated image, but uses the mask obtained by projecting the pseudo multi-view to perform Gaussian blur PM blur After normalization to 0 to 1, two masks are generated by rounding up and down, respectively. Then, the two masks are respectively applied to the projection image and the generated image of G es to supervise the partial region of the generated image, and the inaccurate supervision information in the edge region during fusion of the two images is discarded. The mathematical expression of the optimization equation is as follows:

[0074] wherein, ⊙ represents Hadamard product (also known as Element-wise product), P v represents the camera parameters in a certain perspective, and I vp ​Target map projection to view P v PM blur L LPIPS+L2 G p Optimized generator.

[0075] Local parameter optimization: This method proposes a reversed local parameter optimization method, which effectively reduces the consumption of storage space while ensuring the accuracy of face reconstruction and the editability of reconstructed faces. This method limits the reversed parameter optimization to the parameters of the three-plane feature generation convolution module that affects the generation of facial features. In the implementation, this method sets the network parameter optimization in the two modules SynthBlock256 2 and SynthBlock 128 2 closer to the output end of the StyleGAN2 backbone (see Figure 1), successfully reducing the storage consumption from 120.27MB to 12.22MB.

[0076] Segmentation mask-based semantic editing of reconstructed faces:

[0077] The structure is shown in Figure 6, using the same optimization method as in GAN inversion, fixing the model parameters and taking the latent code as the optimization object. By continuously optimizing the loss function, the latent code w + is adjusted to gradually approach the set target. The main difference between this editing method and the inversion method is the difference in the loss function setting. In the inversion, the latent code is initialized as the average latent code, while in the editing algorithm, the initial latent code is set as the latent code of the reconstructed face. To enable the model to perceive the semantic information of the generated image, this method introduces a face semantic segmentation model and concatenates it with the generator. Under the supervision of this model, the cross-entropy loss between the semantic segmentation mask of the generated image and the target mask is calculated. In addition, this method also designs other loss functions, including a face identity loss function and a non-changing area loss function. Most importantly, this method proposes a regularization loss function for three-plane features and latent codes, improving the decoupling of different features in three-dimensional face perception images.

[0078] Mask cross-entropy loss:

[0079] where Mask t [j,i] represents the probability of the jth pixel position of the target mask being classified as label i, Bs represents the semantic segmentation model, w + and G p are the optimized latent code after face reconstruction and the EG3D generator, respectively. Therefore, Bs(G p (w + , P v)[j, i] represents the probability that the semantic segmentation model infers the mask of the generated image to be label i at the jth pixel. N is the number of labels equal to 20, and HW is the number of pixels in the image.

[0080] Identity and non-varying region loss function: BMask oc = Binary(PMask o , CL) BMask pc = Binary(PMask p , CL) BMask uc = 1 - (BMask oc ∪ BMask pc )

[0081] where VF represents a face feature extraction model, and its input is a face image and its output is a corresponding face feature code. represents the latent code of the reconstructed face, w + The latent code is the optimization object, and · represents the dot product.

[0082] where PMask p and PMask o respectively refer to the parsing mask of the edited face and the reconstructed face, Binary(PMask, CL) represents the binaryzation of the parsing mask according to the CL label set, L uc is the non-varying region loss function, which includes the LPIPS loss and the two-norm loss.

[0083] Latent code and three-plane feature regularization loss function:

[0084] where G p .SB represents the three-plane feature generation module in the generator, and its output is the three-plane feature.

[0085] Total loss function:

[0086] Reconstructing face expression editing:

[0087] The structure is shown in FIG. 7. The method introduces an EMOCA face expression parameter extractor as a face expression supervision model, and simultaneously uses a DECA face parameterization model extractor to extract a rough face texture image renderer, and combines a face semantic segmentation model to supervise the mouth shape of the face. At the same time, the identity, non-varying region, latent code regularization, and three-plane regularization loss functions mentioned above are used to promote the decoupling between different features.

[0088] Expression loss function: L exp= Mean[(EM exp (G p (w + ,P o ))-EM exp (I ref )) 2 ]# L av = Mean[(EM av (G p (w + ,P o ))-EM av (I ref )) 2 ]# L em = λ exp L exp + λ av L av + λ lt L lt #

[0089] where EM exp denotes the monocular image expression coefficients extracted by the EMOCA model, EM av denotes the arousal and valence values of the extracted image, EM lt denotes the illumination and texture coefficients of the extracted image, I ref denotes the reference expression face image, and L em is the total expression loss function.

[0090] Mask loss: To achieve better mouth shape transfer, a mask loss function is introduced to supervise the change of mouth shape in expression editing. A coarse face texture image with the corresponding mouth shape is rendered using DECA. The reference expression face image and the reconstructed face image are input into DECA, and the head pose of the reference face is made consistent with the reconstructed face by replacing the head pose coefficients. Then, a coarse texture image of the reference face in this pose is rendered, and the coarse texture image is input into the semantic segmentation model to obtain the corresponding face semantic mask. Finally, the mouth region of the semantic mask is used to replace the mouth region of the reconstructed face parsing mask to obtain the target mask. The mask loss function is similar in form to the loss function in semantic editing.

[0091] Total loss function:

[0092] Latent code interpolation: Based on the good characteristics of the three-dimensional prior model EG3D latent code space, the method can realize the animation effect of face expression change on the basis of face editing. Specifically, the latent codes of the original face and the face after expression editing are interpolated, and then the generator is output in sequence to generate the transition animation of the face between the two expressions.

[0093] The method of the embodiment of the present application can be summarized as follows:

[0094] The first step is to construct an inversion architecture based on a pre-trained three-dimensional perception face generative adversarial network (EG3D). The main process of the architecture is to discard the noise-to-latent code mapping network, directly input the latent code into multiple serial convolution modules (StyleGAN2) to generate three plane features. Then, based on the camera parameters and the three orthogonal plane features in space, perform volume rendering, and finally output the generated image through the super-resolution module to calculate the loss with the supervised image, thereby optimizing the latent code and the parameters of the generator network. The second step is to construct the inversion architecture, use the monocular face image and the pseudo multi-view image set projected by the depth information to other viewpoints as the supervised image, design the loss function to optimize the network parameters and the latent code, and complete the three-dimensional reconstruction of the single-view face. In this process, based on the image color and perception loss, the present application designs a latent code regularization loss, and designs a local inversion strategy to limit the network parameter optimization to part of the StyleGAN2 network, so as to improve the editability of the latent code and the storage consumption of the reconstruction; the third step is to give the generator different camera parameters to perform the forward process of the generator to synthesize the new view of the face based on the optimized network and latent code parameters after inversion; the fourth step is to construct the face editing architecture designed by the present application, the first half of which is the same as the inversion process architecture, and the latent code is input into the generator to output the rendered face image. Then, the output image is input into any pre-trained network module with prior knowledge, and the image features extracted by the supervised network are calculated with the set features to optimize the latent code. In addition to the feature loss such as one-norm, two-norm and cross-entropy loss commonly used, the present application designs latent code and three-plane feature regularization loss, identity and non-changing area loss in this editing process to improve the decoupling of the edited features and irrelevant features. For this universal editing method, the present application respectively introduces a semantic segmentation network and an expression feature extraction to realize the semantic editing and expression editing of the reconstructed face based on the mask; the fifth step is to input the latent code optimized through the editing framework into the generator to generate the edited face image at any viewpoint. In addition, the interpolation operation between different edited latent codes of the face can realize the smooth transition of the animation between face edits.

[0095] Finally, the present application can generate high-quality edited face images, meet the needs of users for customized editing of three-dimensional face models, and can effectively promote the application and development of three-dimensional face technology in the fields of entertainment, education, news dissemination and the like.

[0096] The embodiment of the present application also provides a storage medium for storing a computer program, which is executed to perform at least the method described above.

[0097] The embodiment of the present application further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is used to execute the computer program and at least execute the method.

[0098] The embodiment of the present application further provides a processor, which executes a computer program and at least executes the method.

[0099] The storage medium can be realized by any type of nonvolatile storage device or combination thereof. The nonvolatile storage device can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc or a compact disc read-only memory (CD-ROM). The magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiment of the present application is intended to include but is not limited to these and any other suitable types of storage devices.

[0100] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other manners. The described device embodiments are merely schematic, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0101] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e. can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purposes of the embodiment solutions.

[0102] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be separately as a unit, or two or more units can be integrated in one unit; the integrated unit can be realized in the form of hardware or hardware plus software function unit.

[0103] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes mobile storage device, read-only memory (ROM), random access memory (RAM), magnetic disc or optical disc and various storage program codes.

[0104] Alternatively, the integrated unit of the present application can be stored in a computer readable storage medium if it is realized in the form of software function module and sold or used as an independent product. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software product, which is stored in a storage medium and includes a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage device, ROM, RAM, magnetic disc or optical disc and various storage program codes.

[0105] The methods disclosed in the several method embodiments of the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0106] The features disclosed in the several product embodiments of the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0107] The features disclosed in the several method or device embodiments of the present application can be combined arbitrarily without conflict to obtain new method or device embodiments.

[0108] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the same performance or use should be regarded as belonging to the protection scope of the present application.

Claims

1. A method for 3D face reconstruction and editing based on a single image, characterized in that, Includes the following steps: S1. For monocular face images, feature extraction is performed using a pre-trained 3D perception face generation adversarial EG3D network. The EG3D network generates three orthogonal planar features through a generator with multiple convolutional modules, and combines them with camera parameters. The generated three-plane features are then rendered in 3D by a neural body renderer to generate a preliminary 3D face representation. S2. The input monocular face image is mapped to the latent code space through the inversion module to obtain the latent code. The obtained latent code is used to generate a three-dimensional face image through the forward process of the EG3D network, and the difference between the generated three-dimensional face image and the original monocular face image is calculated. The latent code is optimized through loss calculation. S3. Based on the preliminary 3D face representation and latent code, depth information is generated from the original monocular face image using depth estimation technology, and the monocular face image is projected onto different camera viewpoints using the depth information to generate pseudo multi-view images. S4. Based on the preliminary 3D face representation and latent code, a semantic segmentation network is used to extract the semantic features of the face, an expression feature extraction network is used to obtain the expression features, and the extracted semantic features and expression features are fused with the features of the pseudo-multi-view image to generate a comprehensive feature representation. S5. Based on editing needs, optimize and regularize the latent code using a loss function through a latent code optimization network; S6. Based on comprehensive feature representation and optimized and regularized latent codes, perform editing operations on 3D face models; S7. Based on the edited feature representation and optimized latent code, the generator generates the final edited face image.

2. The method for three-dimensional face reconstruction and editing based on a single image as described in claim 1, characterized in that, In step S2, the EG3D network also optimizes the details of the generated image through a super-resolution module.

3. The method for three-dimensional face reconstruction and editing based on a single image as described in claim 1 or 2, characterized in that, In step S2, the loss function used for loss calculation includes pixel-level loss and perceptual loss to calculate the difference between the generated image and the supervised image; latent code regularization loss is also introduced to calculate the L2 distance between the optimized latent code and the average latent code in the latent code space, limiting the variation range of the latent code and avoiding overfitting.

4. The method for three-dimensional face reconstruction and editing based on a single image as described in any one of claims 1 to 3, characterized in that, In step S2, local parameter optimization is implemented through a local parameter optimization strategy, and the network parameters are optimized and set in the two synthesis modules closer to the output end in the StyleGAN2 backbone.

5. The method for three-dimensional face reconstruction and editing based on a single image as described in any one of claims 1 to 4, characterized in that, Step S3 specifically includes: The generator and latent code are initially optimized using the input monocular face image as supervision. Based on the optimized generator, the density of the image space is predicted, and the depth information of the monocular image is extracted. By using the extracted depth information, the monocular image is projected onto different camera viewpoints to generate pseudo-multi-view images; The pseudo-multi-view image obtained by projection is fused with the image generated by the generator under the same viewpoint to form a pseudo-supervised view; Based on pseudo-multi-view projection, a mask is generated by Gaussian blurring and normalized to between 0 and 1. The Gaussian blurred mask is rounded up and down to generate two masks for supervision, ensuring the accuracy of the supervision information. When fusing the generated image and the projected image, inaccurate supervisory information in edge regions is discarded using a mask; The generated mask is used to supervise a portion of the projected image and the generated image, and the pseudoview supervision loss is calculated.

6. The method for three-dimensional face reconstruction and editing based on a single image as described in any one of claims 1 to 5, characterized in that, Step S5 includes: evaluating feature differences using loss functions including L1, L2 loss, and perceptual loss; and using latent code and tri-plane feature regularization loss, identity consistency, and non-changing region loss to enhance the decoupling between edited features and irrelevant features; limiting the variation of latent code through the latent code regularization loss function to maintain stability during the editing process; executing an optimization algorithm to update the latent code based on the calculation results of the loss function, and continuously adjusting the latent code through an iterative process until the features of the generated image match the editing requirements.

7. The method for three-dimensional face reconstruction and editing based on a single image as described in any one of claims 1 to 6, characterized in that, Step S6 includes: analyzing the face image using a pre-trained semantic segmentation network to identify and segment different semantic regions; using an expression feature extraction network to identify facial expression features and convert them into editable parameters; generating a corresponding edit mask based on the results of semantic segmentation and expression feature extraction; applying the generated mask to the optimized latent code, editing specific semantic regions based on the edit mask, and adjusting the facial expression features of the face image according to the extracted expression parameters and the generated mask; and fusing the results of semantic editing and expression editing.

8. The method for three-dimensional face reconstruction and editing based on a single image as described in any one of claims 1 to 7, characterized in that, It also includes the following steps: S9. By using latent code interpolation technology to smoothly transition between the latent codes of different expressions, an animation of facial expression changes is generated.

9. The method for three-dimensional face reconstruction and editing based on a single image as described in claim 8, characterized in that, Step S9 includes: performing smooth interpolation on the latent codes under different editing states using linear interpolation; using the intermediate latent codes obtained by interpolation to generate transition animations of facial expressions or features; outputting the generated final facial image as the result of the 3D facial reconstruction and editing process; if transition animations are generated, outputting the animation sequence to show the process of facial expression changes.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the single-image-based three-dimensional face reconstruction and editing method as described in any one of claims 1-9.

Images