Facial expression transfer method, device, equipment and medium based on adversarial autoencoder
By combining an adversarial autoencoder with a deep facial semantic module and super-resolution optimization, the problems of identity feature preservation, detail restoration, and lighting consistency in existing facial expression transfer technologies are solved, achieving high-resolution and high-realism expression transfer effects.
Patent Information
- Application Number
- CN202510990267.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-18
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2045-07-18
AI Technical Summary
Existing facial expression transfer technology has significant defects in insufficient identity feature retention, weak detail restoration ability and poor lighting consistency, and cannot meet the needs of high-resolution and high-realism transfer.
A facial expression transfer method based on adversarial autoencoder is adopted. By combining the encoder and decoder with a deep facial semantic module and super-resolution optimization, expression feature vectors are extracted and image fusion and optimization are performed to generate high-quality expression transfer images.
It achieves high-quality facial image generation and expression transfer, improves the details and naturalness of the image, and ensures the training stability and generalization ability of the model through the designed loss function.
Smart Images
Figure CN120495073B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of expression transfer, and in particular to a facial expression transfer method, apparatus, device and medium based on an adversarial autoencoder. Background Art
[0002] Facial expression transfer technology has significant application value in a variety of fields, including short video special effects production, film and television post-production restoration, medical rehabilitation intervention, and remote educational communication. In the short video and entertainment industries, this technology enables users to mimic others' expressions, enhancing the interest and appeal of content. In professional film and television production, it facilitates the restoration of classic films, accurately recreating the subtle facial expressions of actors and bringing classic characters to life. These applications all rely on high-fidelity, high-resolution transfer results to achieve realistic visual effects and enhance the user experience.
[0003] Existing facial expression transfer technologies suffer from significant deficiencies in key metrics, primarily manifesting in insufficient identity preservation, weak detail restoration capabilities, and poor lighting consistency. In terms of identity preservation, methods like Goust and BlendSwap tend to overwrite key features of the source face (such as eye and eyebrow shape) during the transfer process, resulting in a lack of personalized recognition in the synthesized results. In terms of detail restoration, Inswapper and SimSwap often exhibit texture blurring or artifacts, particularly in complex lighting or high-contrast areas, and are unable to effectively restore high-frequency details like skin texture. Hiface also suffers from poor lighting matching and adaptability to dynamic scenes, resulting in unstable performance when working with cross-ethnic groups or extreme expression changes, causing facial structural deformation.
[0004] These limitations restrict the practical application of the technology and cannot meet the migration requirements of high resolution and high realism. Summary of the Invention
[0005] The present invention provides a facial expression transfer method, apparatus, device and medium based on an adversarial autoencoder to improve at least one of the above-mentioned technical problems.
[0006] In a first aspect, the present invention provides a facial expression transfer method based on an adversarial autoencoder, which comprises steps S1 to S7.
[0007] S1. Acquire a target image for providing appearance features and a source image for providing expression features;
[0008] S2, extracting an expression feature vector according to the source image;
[0009] S3, using an encoder to encode the target image and the source image to obtain depth features;
[0010] S4. Adjusting the depth feature in combination with the expression feature vector to apply the expression of the source image to the target image to obtain an expression fusion feature;
[0011] S5. Using a decoder to decode the expression fusion features, and reconstructing and generating preliminary images of different resolutions;
[0012] S6. Optimizing the preliminary images of different resolutions using a deep facial semantic module to obtain optimized images;
[0013] S7. Perform super-resolution reconstruction on the optimized image to generate an output image; wherein the output image retains the identity characteristics of the target image while migrating the expression dynamics of the source image.
[0014] As a further solution of the present invention, the preliminary images of different resolutions are optimized by a deep facial semantic module to obtain optimized images, specifically including:
[0015] The preliminary image Downsample to a quarter of the target size to obtain the face area mask ;
[0016] According to the face area mask , the expression fusion feature and the preliminary image , perform weighted fusion on low-level features to obtain weighted fusion features; the weighted fusion model is: Where, is the weighted fusion feature, For face area mask, For the initial image, is the residual operation, It is the expression fusion feature;
[0017] The weighted fusion features are processed by the AdaIN block and compared with the target image. and weighted fusion features Fusion, obtaining low-resolution auxiliary images ;
[0018] The weighted fusion features are processed by the AdaIN block and then an upsampling module is used. For further processing, upsampling module Gradually enlarge the low-resolution features to the target resolution to obtain the output image and fusion mask ;
[0019] Fusing the output image with the target image to obtain a semantically fused image; Where, For semantic fusion images, is the fusion mask, To output the image, is the target image.
[0020] As a further solution of the present invention, super-resolution reconstruction is performed on the optimized image to generate an output image, specifically comprising:
[0021] The multi-scale input low-quality face image and its corresponding parsed image are constructed into a multi-scale pyramid, and the eyes, lip contours, nose tip, and cheekbones are marked as regions of interest at different resolutions.
[0022] According to the multi-scale pyramid, features of different scales are gradually modulated from coarse to fine, and feature values of different scales are fused according to the classification of regions of interest and regions of no interest by means of semantically guided style conversion to obtain an output image.
[0023] As a further solution of the present invention, features of different scales are gradually modulated from coarse to fine according to the multi-scale pyramid, and feature values of different scales are fused according to regions of interest and regions of non-interest by means of semantically guided style conversion to obtain an output image, specifically including:
[0024] For ROI data with a scale of 64×64, set the mask according to the mask setting model to obtain the first mask; wherein, the mask setting model is 1+0.02*n; where n is the number of cycles, which is initially 0;
[0025] Multiply the original pixel by the first mask and then upsample to 128x128 resolution to obtain the first feature;
[0026] For ROI data with a scale of 64×64, the ROI pixels are initially masked to 1, and then the residual is calculated with the first feature to determine whether the residual is less than 10%. If the residual is less than 10%, the number of loops is increased by one and the iteration continues. If the residual is not less than 10%, the pixel is upsampled to 256x256 resolution to obtain the second feature.
[0027] For 128×128 scale ROI data, the initial mask value of the ROI pixel is set to 1, and then the residual is calculated with the second feature, and it is determined whether the residual is less than 10%; if the residual is less than 10%, the ROI mask after upsampling to 128x128 resolution is set to 1+0.02*n, and then the iteration is repeated; if the residual is not less than 10%, the three scale features are fused in combination with the second mask to obtain the output image; wherein, the second mask is obtained by setting the initial mask value of non-ROI pixels to 0 for the 128×128 scale ROI data.
[0028] As a further solution of the present invention, the encoder includes five two-dimensional convolution modules, two intermediate layers and an upsampling block connected in sequence; wherein, the output of the first two-dimensional convolution module is fused with the output of the second two-dimensional convolution module through a jump connection, and then input into the third two-dimensional convolution module; the two-dimensional convolution module includes a two-dimensional convolution layer and a LeakyReLU activation function layer connected in sequence.
[0029] As a further solution of the present invention, the expression feature vector is extracted according to the source image, specifically: the expression feature vector is extracted from the source image by adopting a pre-trained CLFace algorithm network.
[0030] As a further solution of the present invention, the decoder includes three upsampling blocks and a two-dimensional convolution module connected in sequence; wherein, the upsampling block includes a two-dimensional convolution layer, a normalization layer and a LeakyReLU activation function layer connected in sequence.
[0031] In the second aspect, the present invention provides a facial expression migration device based on an adversarial autoencoder, which includes an image acquisition module, an expression feature module, a depth feature module, an expression fusion module, a reconstruction module, an optimization module and a super-resolution module.
[0032] The image acquisition module is used to acquire a target image for providing appearance features and a source image for providing expression features;
[0033] The expression feature module is used to extract an expression feature vector according to the source image;
[0034] The depth feature module is used to use an encoder to encode the target image and the source image to obtain depth features;
[0035] The expression fusion module is used to adjust the depth feature in combination with the expression feature vector to apply the expression of the source image to the target image to obtain the expression fusion feature;
[0036] The reconstruction module is used to decode the expression fusion features using a decoder to reconstruct and generate preliminary images of different resolutions;
[0037] The optimization module is used to optimize the preliminary images of different resolutions respectively through the deep facial semantic module to obtain optimized images;
[0038] The super-resolution module is used to perform super-resolution reconstruction processing on the optimized image to generate an output image; wherein, the output image retains the identity characteristics of the target image while migrating the expression dynamics of the source image.
[0039] In the third aspect, the present invention provides a facial expression migration device based on an adversarial autoencoder, which includes a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement a facial expression migration method based on an adversarial autoencoder as described in any paragraph of the first aspect.
[0040] In a fourth aspect, the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a facial expression migration method based on an adversarial autoencoder as described in any paragraph of the first aspect.
[0041] By adopting the above technical solution, the present invention can achieve the following technical effects:
[0042] Through the synergistic effects of expression feature vector extraction, adversarial autoencoders, deep facial semantic fusion, and super-resolution optimization, this model can generate more realistic facial expressions and movements, and has important application value in the field of expression transfer technology.
[0043] This method optimizes the face encoder and decoder, introduces a super-resolution deep facial semantic module, and achieves high-quality facial image generation and expression transfer. The image generated by this algorithm has improved details and naturalness, and the designed loss function ensures the training stability and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the specific implementation methods of the present invention. It should be understood that the following drawings only show certain specific implementation methods of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.
[0045] Figure 1 This is the network structure diagram of the facial expression transfer method.
[0046] Figure 2 It is a network structure diagram of the codec structure.
[0047] Figure 3 It is the network structure diagram of the deep facial semantic module.
[0048] Figure 4 This is the network structure diagram of the super-resolution generative adversarial network.
[0049] Figure 5 This is a network structure diagram of progressive semantic-aware style transfer.
[0050] Figure 6 It is a flowchart of the facial expression transfer method.
[0051] The facial data in the attached figures are all from the CelebA-HQ dataset, which is a public dataset and does not infringe on portrait rights. DETAILED DESCRIPTION
[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0053] Example 1, please refer to Figures 1 to 6 A first embodiment of the present invention provides a method for transferring facial expressions based on an adversarial autoencoder, which can be performed by a facial expression transfer device based on an adversarial autoencoder (hereinafter referred to as the facial expression transfer device). Specifically, steps S1 to S7 are implemented by one or more processors in the facial expression transfer device.
[0054] S1. Obtaining a target image for providing appearance features and the source image used to provide the expression features .
[0055] S2, extracting expression feature vectors from the source image. Specifically, extracting expression feature vectors from the source image by using a pre-trained CLFace algorithm network ;
[0056] S3. Use an encoder to encode the target image and the source image to obtain depth features.
[0057] In this embodiment, if Figure 2 As shown, the encoder includes five sequentially connected two-dimensional convolution modules, two intermediate layers and an upsampling block; wherein the output of the first two-dimensional convolution module is fused with the output of the second two-dimensional convolution module through a skip connection and then input into the third two-dimensional convolution module; the two-dimensional convolution module includes a sequentially connected two-dimensional convolution layer and a LeakyReLU activation function layer;
[0058] In the process of optimizing and improving the encoder, this embodiment implements more in-depth improvement measures for the overall upsampling structure, with the aim of improving feature extraction capabilities and optimizing the fusion method of multi-scale features. Traditional autoencoder structures may have problems with information loss or computational redundancy when performing feature extraction. In processing tasks for high-resolution images, these problems may affect the convergence speed and reconstruction quality of the model.
[0059] During the optimization process, this embodiment designs a lightweight autoencoder structure that balances computational efficiency and feature extraction integrity. The core idea is to leverage an efficient convolutional module combined with an improved upsampling strategy to fully express the input image information at different scales and restore the original details with high quality during the decoding process.
[0060] like Figure 2 As shown, this embodiment first introduces multiple two-dimensional convolutional layers (Conv2DBlock) in the encoding stage for feature extraction, combined with the residual module (ResidualBlock) to reduce the gradient vanishing phenomenon and improve the model's ability to capture deep features. The convolution block uses a smaller convolution kernel and combines it with an appropriate step size for downsampling to maintain high feature expression capabilities while reducing the amount of computation. To improve the stability of the model, this embodiment adds a normalization layer after some convolutional layers to reduce the distribution deviation of input data in different batches and improve the stability of training.
[0061] Regarding the choice of activation function, this embodiment introduces the LeakyReLU activation function to replace the traditional ReLU activation function. While the traditional ReLU activation function causes its gradient to vanish when encountering negative values, the LeakyReLU activation function allows values less than zero to propagate the gradient with a smaller slope, improving the model's ability to learn complex data distributions. This improvement allows the model to converge faster during training, while reducing the risk of overfitting and improving generalization, resulting in better performance across different datasets.
[0062] S4, adjusting the depth feature in combination with the expression feature vector to apply the expression of the source image to the target image to obtain the expression fusion feature. Figure 2 As shown, the facial expression feature vector extracted in step S2 is Fuse it with the deep features output by the encoder to obtain expression fusion features.
[0063] S5. Using a decoder to decode the expression fusion features, and reconstructing and generating preliminary images of different resolutions;
[0064] In this embodiment, if Figure 2 As shown, the decoder includes three upsampling blocks and a two-dimensional convolution module connected in sequence; wherein the upsampling block includes a two-dimensional convolution layer, a normalization layer and a LeakyReLU activation function layer connected in sequence;
[0065] During the decoding phase, this embodiment utilizes an efficient upsampling block (UpscaleBlock) to perform feature reconstruction. Traditional upsampling methods, such as bilinear interpolation or nearest neighbor interpolation, can result in image blurring or loss of detail when restoring high-resolution images. In contrast, the method proposed in this embodiment incorporates deep convolution operations, allowing the upsampled feature maps to more accurately restore the structural information of the original image. This strategy improves the quality of the reconstructed image and reduces computational redundancy, making the decoding process more efficient.
[0066] This embodiment also optimizes the overall design of the network architecture to ensure that the entire model can run stably on input data of varying resolutions. By leveraging multi-scale feature fusion, the model can adapt to input images of varying sizes, better extracting key details when processing complex scenes. This improvement makes the autoencoder proposed in this embodiment suitable for image super-resolution reconstruction and can also be extended to other computer vision tasks, such as medical image processing and remote sensing image restoration.
[0067] S6. Optimizing the preliminary images of different resolutions using a deep facial semantic module to obtain optimized images;
[0068] The main purpose of the deep facial semantics module is to intelligently fuse the information of the encoder and decoder at the low-level feature level, so as to preserve the clean texture and background information in the target image while making full use of the rich facial details and identity features in the source image. Figure 3 As shown, it is mainly divided into feature-level fusion and image-level fusion. Specifically, the step S6 includes steps S61 to S65.
[0069] S61, the preliminary image Downsample to a quarter of the target size to obtain the face area mask ;
[0070] In feature-based fusion, when the decoder outputs the preliminary image When downsampling to 1 / 4 of the target image size (here It first enters a residual module of the residual block (ResBlock) for downsampling, which is the orange cube part in the figure). The module will predict a face area mask. .
[0071] Face region mask Function: For pixels in the face area, the mask value tends to 1, and the decoder features are retained. , in order to preserve the rich expression information and facial details in the source image. For non-face areas, the mask value tends to 0.
[0072] S62, according to the face area mask , the expression fusion feature and the preliminary image , perform weighted fusion on low-level features to obtain weighted fusion features;
[0073] The weighted fusion model is:
[0074] ;
[0075] Where, is the weighted fusion feature, For face area mask, For the initial image, is the residual operation, It is the expression fusion feature;
[0076] It contains identity details and texture information of the face in the source image, but may introduce inappropriate identity features in non-face areas. The image from the target contains purer spatial structures and background details. The pixel value of the face area is close to 1, and the pixel value of the non-face area is close to 0. Equal to the corresponding block .
[0077] Specifically, the expression fusion feature of deep features and expression feature vectors is introduced , which can ensure that background and texture information are not disturbed by unnecessary expression information. The pixel-level control retains the advantages of the decoder in the facial area, while relying on the clean features of the encoder in other areas, thus achieving complementary advantages.
[0078] S63, the weighted fusion feature is processed by the AdaIN block (ie: Adain-ResBlock) and then compared with the target image and weighted fusion features Fusion, obtaining low-resolution auxiliary images ;
[0079] Auxiliary images Function: Used to intuitively evaluate the fusion effect. It provides a basis for the subsequent calculation of auxiliary loss. During training, the auxiliary loss helps the model better decouple expression information from other attributes (such as posture and lighting), improving the overall realism of the generated image.
[0080] S64, after the weighted fusion features are processed by the AdaIN block (ie: Adain-ResBlock), an upsampling module is used (usually composed of multiple residual blocks) for further processing, upsampling module Gradually enlarge the low-resolution features to the target resolution (e.g. 512×512) to obtain the output image and fusion mask ;
[0081] Multi-scale information is fully integrated during the upscaling process to ensure that high-resolution images have both delicate facial details and retain overall texture and background structure.
[0082] S65, fusing the output image with the target image to obtain a semantically fused image;
[0083] ;
[0084] Where, For semantic fusion images, is the fusion mask, To output the image, is the target image;
[0085] Image-level fusion primarily addresses occlusion and background preservation issues. Previous methods have directly used a target face mask. However, since facial shape may change during processing, this approach often introduces artifacts, affecting the final result.
[0086] To address this problem, this embodiment introduces an image-level fusion strategy in the deep facial semantics module. By learning a slightly expanded mask, the model can adapt to subtle changes in facial shape.
[0087] The specific process is: predict output image and fusion mask: output image The predicted 3-channel image represents the high-quality output after processing. Fusion mask The predicted 1-channel mask is slightly expanded to better adapt to the changes in face shape. and fusion mask . Perform image-level fusion.
[0088] when When the value is close to 1, it means that the area is mainly the face area, and more To ensure that facial details are fully restored. When the value is close to 0, it means that the area is mainly background or non-face area, and the target image is retained. information, ensuring that background details and overall consistency are not compromised.
[0089] The multi-scale optimization of this embodiment includes predicting a face region mask and performing weighted fusion operations at the feature level and image level to eliminate artifacts and enhance facial structure coherence. The fusion operation is the innovation of the present invention, extending the mask and fusion: resolving background details and overall consistency, and eliminating artifacts.
[0090] S7, performing super-resolution reconstruction processing on the optimized image to generate an output image; wherein the output image retains the identity characteristics of the target image while migrating the expression dynamics of the source image;
[0091] When processing a video file, the target video file is split into a series of individual frames. Each frame serves as an independent input for the subsequent super-resolution process. For each frame, super-resolution processing is performed using the prediction code of an improved super-resolution generative adversarial network model. The improved super-resolution generative adversarial network is specifically designed for restoring low-quality facial images. Its core function is to gradually restore facial details and textures, ensuring that the generated images have high resolution and maintain a realistic feel.
[0092] In order to achieve feature value comparison, smoothing and reconstruction on time series of the same scale and at different resolutions at the same time, it is necessary to transform the deep facial semantic module and the super-resolution generative adversarial network modules, so that their output changes from a single scale to multiple scales. The transformation of these two modules is the innovation of the present invention.
[0093] The method proposed in this embodiment uses image super-resolution processing and model output to enhance the details and overall clarity of each frame of the video to achieve a more realistic visual effect. Figure 4 and Figure 5 As shown in FIG, the network structure of the improved super-resolution generative adversarial network proposed in the present invention mainly includes the following parts:
[0094] S71. Construct a multi-scale pyramid from the multi-scale input low-quality face image and its corresponding parsed image, and mark the eyes, lip contours, nose tip, and cheekbones as regions of interest at different resolutions.
[0095] The model first constructs a multi-scale pyramid (typically at three resolutions: 64×64, 128×128, and 256×256) from multi-scale input low-quality facial images and their corresponding parsed images to capture feature information at different scales. This operation fully utilizes the detailed information at different levels in the image. At the same time, the model also demarcates the eyes, lip contours, nose tip, and cheekbones as regions of interest (ROIs) at different resolutions to enhance feature extraction and define them as low semantic loss areas.
[0096] S72: gradually modulating features of different scales from coarse to fine according to the multi-scale pyramid, and classifying feature values of different scales into regions of interest and regions of non-interest by means of semantically guided style conversion to perform feature fusion, and obtain an output image;
[0097] The implementation steps of ROI semantic-aware style conversion are as follows: Figure 5 As shown in Figure 3, this transformation can ensure that the detail recovery of the region of interest will not be disturbed by the information of the non-region of interest area, and retain the key identity features.
[0098] ROI semantic-aware style conversion includes steps S721 to S726.
[0099] S721. For ROI data with a scale of 64×64, set a mask according to a mask setting model to obtain a first mask; wherein the mask setting model is 1+0.02*n; where n is the number of cycles, which is initially 0;
[0100] S722: multiply the original pixel by the first mask, and then upsample to a resolution of 128×128 to obtain a first feature;
[0101] S723. For ROI data with a scale of 64×64, set the initial mask of the ROI pixels to 1, then calculate the residual with the first feature, and determine whether the residual is less than 10%; if the residual is less than 10%, increase the number of loops by one and continue iterating; if the residual is not less than 10%, upsample to 256×256 resolution to obtain the second feature; in this embodiment, the residual is calculated by subtracting the scaled pixel value from the large-scale pixel value and then taking the absolute value;
[0102] S724. For the 128×128 scale ROI data, the initial mask value of the ROI pixel is set to 1, and then the residual is calculated with the second feature, and it is determined whether the residual is less than 10%; if the residual is less than 10%, the ROI mask after upsampling to 128x128 resolution is set to 1+0.02*n, and then the iteration is repeated; if the residual is not less than 10%, the three-scale feature fusion is performed in combination with the second mask; wherein the second mask is obtained by setting the initial mask value of the non-ROI pixels to 0 for the 128×128 scale ROI data.
[0103] S725. The features obtained by fusion of the three scale features are input into the generator to obtain the output image;
[0104] During the training process, this paper adopts a generative adversarial network architecture. By using the GAN framework, the generator and discriminator are continuously improved during adversarial training, ultimately achieving high-quality image super-resolution reconstruction. Furthermore, a semantic-aware style loss is used to enhance the restoration effect. The improved super-resolution generative adversarial network employs a semantic-aware style loss during training. This loss function constrains the features of each semantic region, improving the representation of facial texture details and making the output image more realistic in style and detail.
[0105] After frame-by-frame processing using the improved super-resolution generative adversarial network, this embodiment produces an image sequence with enhanced detail and overall clarity within each frame. These enhanced frames are then resynthesized at the original video's frame rate to restore continuous video data. This ensures that facial details in each shot are preserved while maintaining smooth and coherent video playback.
[0106] In a facial expression transfer method based on an adversarial autoencoder in this embodiment, the target image Used to provide facial appearance features, source image Used to extract expression information. This embodiment uses the pre-trained CLFace algorithm to extract expression feature vectors from the source image. , ensuring that the resulting image retains the source person's facial features. This embodiment uses a high-precision encoder to extract features from the target image, converting them into a deep feature representation. This deep feature representation is then adjusted based on the facial expression feature vector. The decoder then reconstructs the fused features back into an image.
[0107] Since the reconstructed images may have blurred or unnatural details, in order to further improve the image quality, the network designed a deep facial semantic module, which optimizes the generated images at multiple scales (64×64, 128×128, 256×256) to make the facial structure more coherent and the identity features more stable.
[0108] Based on the output of the Deep Facial Semantics module, an improved super-resolution generative adversarial network is used to super-reconstruct the different resolution images generated by the deep facial semantics module, further improving the clarity and realism of the synthesized faces. The improved super-resolution generative adversarial network focuses on low-quality real-world faces, combining super-resolution and prior facial information to perform restoration, resulting in the final output image.
[0109] Through the synergistic effects of expression feature vector extraction, adversarial autoencoders, deep facial semantic fusion, and super-resolution optimization, this model can generate more realistic facial expressions and movements, and has important application value in the field of expression transfer technology.
[0110] This method optimizes the face encoder and decoder, introduces a super-resolution deep facial semantic module, and achieves high-quality facial image generation and expression transfer. The image generated by this algorithm has improved details and naturalness, and the designed loss function ensures the training stability and generalization ability of the model.
[0111] During training, we employ progressive learning and a generative adversarial network, incorporating facial analysis information to create more natural and well-structured restorations. This avoids artifacts that can occur with traditional GANs. By incorporating a discriminator (D) and employing adversarial learning to distinguish generated images from real images, we improve the authenticity of the generated results.
[0112] It is understandable that the facial expression transfer device may be an electronic device with computing capabilities, such as a portable notebook computer, a desktop computer, a server, a smart phone, or a tablet computer.
[0113] In this embodiment, a facial expression transfer method based on an adversarial autoencoder can be used for short video and special effects production. Specifically, when ordinary people shoot videos and want to imitate a celebrity's classic expression or popular demeanor, they can easily do so through expression transfer, making the imitation show more realistic and interesting. It is even more of a magic weapon for professional film and television post-production. For example, when restoring old movies, the actor's subtle expressions at the time can be more accurately "pasted" back to the high-definition restored face, giving the classic character new vitality; or in the production of animation and virtual characters, the rich expressions of real actors are used to drive the faces of cartoons or virtual images in real time, which is efficient and natural, making filming and animation much less troublesome.
[0114] To achieve better expression transfer results for a facial expression transfer method based on an adversarial autoencoder, the inventors also designed a loss function. This loss function consists of segmentation loss (SegmentationLoss), reconstruction loss (ReconstructionLoss), cycle loss (CycleLoss), perceptual loss (PerceptualLoss), and adversarial loss (AdversarialLoss). It aims to improve the realism of the generated images and make them more visually natural.
[0115] SegmentationLoss: In the deep facial semantics module, the face region mask and is the fusion mask Guided by the advanced face segmentation network HRNet. Since the face shape may change during the processing, this embodiment dilates the mask of the target image to eliminate the restriction on facial shape change and obtain the dilated target mask. .
[0116] ;
[0117] Where, For segmentation loss, The resize operation is used to make the mask match the target size. This loss ensures the matching degree between the predicted mask and the target mask, thereby improving the fusion quality.
[0118] Reconstruction Loss: If the source image and the target image share the same expression, the generated image should be Therefore, this embodiment defines a reconstruction loss that allows the model to restore the original image as faithfully as possible given the same identity. This loss ensures that the generated image is as close to the real target as possible through pixel-wise supervision.
[0119] ;
[0120] Where, To rebuild the losses, For semantic fusion images, For the target image, For low-resolution auxiliary images, For resizing operations, target image ;
[0121] Cycle Loss: Cycle consistency can also be applied to faceswapping. This loss complements pixel supervision and helps generate high-fidelity synthesis results.
[0122] ;
[0123] Where, is the circulation loss, is the target image, G represents the entire generator of the algorithm, is the re-target image obtained after the initial conversion. As new input, we want to generate an image that is consistent with the original identity but with matching attributes.
[0124] Perceptual Loss and Adversarial Loss: To capture detailed information and further enhance the realism of generated images, this embodiment uses LPIPS (Learned Perceptual Image Patch Similarity) to measure perceptual similarity using deep features, improving the perceived quality of synthesized images. Furthermore, Adversarial Loss employs an adversarial learning strategy to make generated images closer to real faces.
[0125] Combining all loss items, the total loss (RealismLoss) is defined as follows:
[0126] ;
[0127] Where, For total loss, To combat losses, For perception loss, 、 、 and are the hyperparameters for different losses;
[0128] In this embodiment, the weights of the hyperparameters are , , , These hyperparameters are used to balance the contributions of different loss terms to the final optimization objective.
[0129] The facial expression transfer method of this embodiment processes facial images to 128x128 or 256x256 pixels. These operations unify the image size and remove irrelevant background information, making the training data more accurate and improving the quality of expression transfer. After these preprocessing steps, the quality of the training data is significantly improved, providing high-quality input data for subsequent model training.
[0130] The facial expression transfer method of this embodiment shows excellent performance in expression transfer, which is largely due to the deep facial semantic module proposed in this article. This module has the ability to intelligently fuse the source face and target face information at the low-level feature level, so that key facial features such as eyes, nose, mouth and other facial structures can be better preserved during the synthesis process, ensuring that the overall lighting and background of the target face remain consistent.
[0131] This embodiment, based on an adversarial autoencoder, fully integrates the source face's expression information during encoding and utilizes a deep facial semantics module to intelligently fuse features during decoding. This allows for more complete preservation of expression information while reducing interference from background and non-critical areas. Optimizations have also been made for high-resolution generation, enabling better restoration of facial texture details and ensuring the final synthesized expression transfer result.
[0132] The facial expression transfer method of this embodiment is based on an adversarial autoencoder, combining the generative capabilities of a GAN with the feature extraction capabilities of an autoencoder. This allows the model to more efficiently capture facial expression information and detailed features in a high-dimensional feature space. This embodiment introduces a super-resolution reconstruction module during training, enabling the model to perform high-quality face replacement and improve resolution during synthesis, generating clearer, high-resolution facial images. The method of this embodiment also utilizes a multi-scale loss function, including adversarial, perceptual, and reconstruction losses, to ensure that the synthesized images achieve optimal realism, identity matching, and local detail.
[0133] Embodiment 2: The present invention provides a facial expression migration device based on an adversarial autoencoder, which includes an image acquisition module, an expression feature module, a depth feature module, an expression fusion module, a reconstruction module, an optimization module and a super-resolution module.
[0134] The image acquisition module is used to acquire a target image for providing appearance features and a source image for providing expression features;
[0135] The expression feature module is used to extract an expression feature vector according to the source image;
[0136] The depth feature module is used to use an encoder to encode the target image and the source image to obtain depth features;
[0137] The expression fusion module is used to adjust the depth feature in combination with the expression feature vector to apply the expression of the source image to the target image to obtain the expression fusion feature;
[0138] The reconstruction module is used to decode the expression fusion features using a decoder to reconstruct and generate preliminary images of different resolutions;
[0139] The optimization module is used to optimize the preliminary images of different resolutions respectively through the deep facial semantic module to obtain optimized images;
[0140] The super-resolution module is used to perform super-resolution reconstruction processing on the optimized image to generate an output image; wherein, the output image retains the identity characteristics of the target image while migrating the expression dynamics of the source image.
[0141] Embodiment 3. The present invention provides a facial expression transfer device based on an adversarial autoencoder, which includes a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement a facial expression transfer method based on an adversarial autoencoder as described in any paragraph of Embodiment 1.
[0142] Embodiment 4. The present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a facial expression transfer method based on an adversarial autoencoder as described in any paragraph of Embodiment 1.
[0143] Obviously, the embodiments described above are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0144] In the several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.
[0145] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.
[0146] If the functions are implemented as software modules and sold or used as standalone products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for causing a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories, random access memories, magnetic disks, or optical disks. It should be noted that in this embodiment, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. Without further limitation, the phrase "comprises a..." does not preclude the presence of other identical elements in the process, method, article, or device comprising the elements.
[0147] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.
[0148] It should be understood that the term "and / or" used in this embodiment is merely a description of the association relationship between associated objects, indicating that three possible relationships exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this embodiment generally indicates that the associated objects are in an "or" relationship.
[0149] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.
[0150] The references to "first" and "second" in the embodiments merely distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or precedence of "first" and "second" can be interchanged where appropriate. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.
[0151] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.
Claims
1. A facial expression transfer method based on adversarial autoencoder, characterized in that: Include: Acquire a target image for providing appearance features and a source image for providing expression features; Extracting an expression feature vector according to the source image; Encode the target image and the source image using an encoder to obtain depth features; Adjusting the depth feature in combination with the expression feature vector to apply the expression of the source image to the target image to obtain an expression fusion feature; Decoding the expression fusion features using a decoder to reconstruct and generate preliminary images of different resolutions; The preliminary images of different resolutions are optimized by the deep facial semantic module to obtain optimized images; wherein the preliminary images are Downsample to a quarter of the target size to obtain the face area mask ; According to the face area mask , the expression fusion feature and the preliminary image , perform weighted fusion on the low-level features to obtain weighted fusion features; the weighted fusion features are processed by the AdaIN block and then processed by an upsampling module For further processing, upsampling module Gradually enlarge the low-resolution features to the target resolution to obtain the output image and fusion mask ; Fusing the output image with the target image to obtain a semantically fused image; Where, For semantic fusion images, is the fusion mask, To output the image, is the target image; The optimized image is subjected to super-resolution reconstruction processing to generate an output image; wherein the output image transfers the expression dynamics of the source image while retaining the identity characteristics of the target image.
2. A facial expression transfer method based on an adversarial autoencoder according to claim 1, characterized in that, Performing super-resolution reconstruction on the optimized image to generate an output image specifically includes: The multi-scale input low-quality face image and its corresponding parsed image are constructed into a multi-scale pyramid, and the eyes, lip contours, nose tip, and cheekbones are marked as regions of interest at different resolutions. According to the multi-scale pyramid, features of different scales are gradually modulated from coarse to fine, and feature values of different scales are fused according to the classification of regions of interest and regions of no interest by means of semantically guided style conversion to obtain an output image.
3. A facial expression transfer method based on an adversarial autoencoder according to claim 2, characterized in that, According to the multi-scale pyramid, features of different scales are gradually modulated from coarse to fine, and feature values of different scales are fused according to regions of interest and regions of non-interest using semantically guided style conversion to obtain an output image, specifically including: For ROI data with a scale of 64×64, set the mask according to the mask setting model to obtain the first mask; wherein, the mask setting model is 1+0.02*n; where n is the number of cycles, which is initially 0; Multiply the original pixel by the first mask and then upsample to 128x128 resolution to obtain the first feature; For ROI data with a scale of 64×64, the ROI pixels are initially masked to 1, and then the residual is calculated with the first feature to determine whether the residual is less than 10%. If the residual is less than 10%, the number of loops is increased by one and the iteration continues. If the residual is not less than 10%, the pixel is upsampled to 256x256 resolution to obtain the second feature. For 128×128 scale ROI data, the initial mask value of the ROI pixel is set to 1, and then the residual is calculated with the second feature, and it is determined whether the residual is less than 10%; if the residual is less than 10%, the ROI mask after upsampling to 128x128 resolution is set to 1+0.02*n, and then the iteration is repeated; if the residual is not less than 10%, the three scale features are fused in combination with the second mask to obtain the output image; wherein, the second mask is obtained by setting the initial mask value of non-ROI pixels to 0 for the 128×128 scale ROI data.
4. A facial expression transfer method based on an adversarial autoencoder according to any one of claims 1 to 3, characterized in that The encoder includes five two-dimensional convolution modules, two intermediate layers and an upsampling block connected in sequence; the output of the first two-dimensional convolution module is fused with the output of the second two-dimensional convolution module through a jump connection, and then input into the third two-dimensional convolution module; the two-dimensional convolution module includes a two-dimensional convolution layer and a LeakyReLU activation function layer connected in sequence.
5. A facial expression transfer method based on an adversarial autoencoder according to any one of claims 1 to 3, characterized in that Extracting the expression feature vector according to the source image specifically comprises: extracting the expression feature vector from the source image by adopting a pre-trained CLFace algorithm network.
6. A facial expression transfer method based on an adversarial autoencoder according to any one of claims 1 to 3, characterized in that The decoder includes three upsampling blocks and a two-dimensional convolution module connected in sequence; wherein the upsampling block includes a two-dimensional convolution layer, a normalization layer and a LeakyReLU activation function layer connected in sequence.
7. A facial expression transfer device based on an adversarial autoencoder, characterized in that: Include: An image acquisition module, configured to acquire a target image for providing appearance features and a source image for providing expression features; An expression feature module, configured to extract an expression feature vector based on the source image; A depth feature module, configured to encode the target image and the source image using an encoder to obtain depth features; An expression fusion module, configured to adjust the depth feature in combination with the expression feature vector to apply the expression of the source image to the target image and obtain an expression fusion feature; A reconstruction module, configured to decode the expression fusion features using a decoder to reconstruct and generate preliminary images of different resolutions; The optimization module is used to optimize the preliminary images of different resolutions respectively through the deep facial semantic module to obtain optimized images; wherein, the preliminary images Downsample to a quarter of the target size to obtain the face area mask ; According to the face area mask , the expression fusion feature and the preliminary image , perform weighted fusion on the low-level features to obtain weighted fusion features; the weighted fusion features are processed by the AdaIN block and then processed by an upsampling module For further processing, upsampling module Gradually enlarge the low-resolution features to the target resolution to obtain the output image and fusion mask ; Fusing the output image with the target image to obtain a semantically fused image; Where, For semantic fusion images, is the fusion mask, To output the image, is the target image; The super-resolution module is used to perform super-resolution reconstruction on the optimized image to generate an output image; wherein the output image retains the identity characteristics of the target image while migrating the expression dynamics of the source image.
8. A facial expression transfer device based on an adversarial autoencoder, characterized in that: The invention comprises a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement a facial expression transfer method based on an adversarial autoencoder as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the facial expression transfer method based on the adversarial autoencoder according to any one of claims 1 to 6.