Face changing method and device and face changing model training method

Abstract

The invention discloses a face changing method and device and a face changing model training method. The face changing method comprises the steps that firstly, a source face image and a target face image are obtained; inputting the source face image and the target face image into a trained face changing model to generate a face-changed image; the face changing model performs feature extraction on the source face image and the target face image, performs key point detection on the target face image, aligns the features of the source face image with the features of the target face image after key point detection, performs adaptive fusion on the features of the source face image after expression alignment and the features of the target face image, and performs face changing on the target face image; and processing and rendering the fused features to generate a face-changed image. According to the hidden space feature decoupling method, the face changing task can achieve the effect of accurately duplicating the target face expression, fusion of the identity information of the source face and the non-identity attribute of the target face can be achieved through adaptive learning, the process is clear, and robustness is high.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a face swapping method and device, and a face swapping model training method. Background Art

[0002] Face swapping is a key component of controllable face generation, primarily focusing on the dual control of facial image appearance and motion information. Face swapping technology has not only attracted considerable attention in the fields of computer vision and graphics, but also holds broad application prospects in various fields, including entertainment, finance, security, and the service industry. The specific task of face swapping is to, given a target face image and a source face image, fuse the target face's facial motion information, identity-independent appearance information, and scene information with the source face image's identity information to generate a result that combines the source face image's identity information with the target face image's motion, non-identity attributes, and scene information.

[0003] The key and difficulty of face-swapping lies in the perception and controllable generation of faces. Regarding perception, existing methods extract identity information (such as face shape and skin color) that is highly coupled with non-identity information (such as hairstyle and glasses), interfering with the generated results. Regarding controllable generation, existing methods extract motion information with insufficient granularity, and the transferred motion information is affected by image differences. This results in the generated expression and posture failing to accurately replicate the target face image.

[0004] Early face replacement work stipulated the similarity between the source and target faces in posture and lighting conditions, and only replaced the pixels in the facial area, making it difficult to deal with the face replacement problem under unconstrained conditions.

[0005] Subsequently, researchers proposed some methods based on 3D models, using 3D models to fit faces and address the situation where the poses of the source and target faces differ significantly. However, 3D face models constructed from a single input image struggle to maintain the identity information of the source face in the face-swapping result, resembling both the source and target faces, creating a third identity. Furthermore, some work has introduced generative adversarial networks (GANs) to preserve the attributes of the target face. Other research works combine high-level inverted latent codes of the source and target images and directly feed the fused code vector into the generator to produce the face-swapping result. However, because identity information and action information are highly entangled in the latent space, identity information is coupled with other non-identity information besides action, resulting in poor face-swapping results. Summary of the Invention

[0006] The present invention provides a face swapping method and device, and a face swapping model training method, which are used to solve the technical defects of poor face swapping effect caused by the high entanglement of identity information and action information in the latent space, and the mutual coupling of identity information with other non-identity information except action.

[0007] In order to solve the above technical defects, the first aspect of the present invention provides a face swapping method, comprising: Obtain source face image; Obtain target face image; Inputting the source face image and the target face image into a trained face-swapping model to generate a face-swapping image; Among them, the face-changing model extracts features from the source face image and the target face image, detects key points on the target face image, aligns the features of the source face image with the features of the target face image after key point detection, adaptively fuses the features of the source face image and the features of the target face image after expression alignment, processes and renders the fused features to generate the face-changed image.

[0008] In one embodiment, the face-swap model extracts features from a source face image and a target face image, detects key points on the target face image, aligns features of the source face image with features of the target face image after key point detection, adaptively fuses features of the source face image and features of the target face image after expression alignment, processes and renders the fused features to generate a face-swap image, including: The face-changing model extracts features of the source face image and the target face image through the pSp encoder to obtain features of the source face image and features of the target face image; Perform key point detection on the target face image through the key point detection module to obtain the key point features of the target face image as the target face image features after key point detection; The expression alignment module aligns the source face image features with the target face image features after key point detection to obtain the expression-aligned source face image features; The face-changing module adaptively fuses the features of the source and target facial images that have been aligned with the expression to obtain the fused features. The fused features are processed by StyleGAN2 to obtain three-plane features; The three-plane features output by StyleGAN2 are rendered through the rendering module to generate the face-changed image.

[0009] In one embodiment, aligning source facial image features with target facial image features after key point detection by an expression alignment module to obtain expression-aligned source facial image features includes: The target face image features after key point detection are used to guide the expression of the input source face image features to obtain the source face image features that have been aligned with the expression.

[0010] In one embodiment, the face-changing module adaptively fuses the features of the source facial image and the target facial image after expression alignment to obtain fused features, including: splicing a specified intermediate layer latent vector in the expression-aligned source facial image latent vector with a target facial image latent vector, wherein the expression-aligned source facial image latent vector is used to represent features of the expression-aligned source facial image, and the specified intermediate layer latent vector is determined based on a correlation between the layer latent vector and non-identity information, the non-identity information including expression, posture, and hairstyle decoration; The concatenated vector is reduced in dimension through a linear layer, and the Sigmoid function is used to obtain the weighted variable; The weighted variable is used as the weight to weight the source face image latent vector and the target face latent vector to obtain the fused latent vector to represent the fused features.

[0011] In one embodiment, the method further includes: performing attention enhancement on latent vectors of two adjacent frames that have been adaptively fused.

[0012] In one embodiment, the step of performing attention enhancement on latent vectors of two adjacent frames after adaptive fusion includes: Use the first multi-layer perceptron to concatenate the latent vectors of two adjacent frames of images and then perform mapping; Multiply the mapped vector by the latent vector of the next burst of images mapped by the second multi-layer perceptron, calculate the attention, and process the calculated attention through the Sigmoid function; The result of the attention processing is added to the latent vector of the next burst of frames mapped by the second multi-layer perceptron to obtain the enhanced latent vector.

[0013] In one embodiment, the rendering module renders the three-plane features output by StyleGAN2 to generate a face-swapped image, including: The EG3D framework is used to render the three-plane features output by StyleGAN2 to generate the face-swapped image.

[0014] Based on the same inventive concept, the second aspect of the present invention provides a face swapping device, comprising: A first acquisition module is used to obtain a source face image; The second acquisition module is used to obtain the target face image; An input module, configured to input the source face image and the target face image into a trained face-swapping model to generate a face-swapping image; Among them, the face-changing model extracts features from the source face image and the target face image, detects key points on the target face image, aligns the features of the source face image with the features of the target face image after key point detection, adaptively fuses the features of the source face image and the features of the target face image after expression alignment, processes and renders the fused features to generate the face-changed image.

[0015] Based on the same inventive concept, the third aspect of the present invention provides a training method for a face-changing model, wherein the face-changing model includes a pSp encoder, a key point detection module, an expression alignment module, a face-changing module, StyleGAN2, and a rendering module. The training method includes: pre-training the expression alignment module and real-time training of the face-changing module.

[0016] In one embodiment, pre-training the expression alignment module includes: Obtain source face image samples and target face image samples; Use the pre-trained face replay model to replay the source face image sample and obtain the replay result; Input the features of the replay result extracted by the pSp encoder and the target face image features after the key point detection of the target face image sample into the expression alignment module for prediction; Calculate the feature contrast loss based on the consistency between the features predicted by the expression alignment module and the features extracted by the pSp encoder of the replay result; Calculate the image contrast loss based on the consistency between the face-swapped image and the reenactment result corresponding to the features predicted by the expression alignment module; Based on the feature contrast loss and the image contrast loss, the parameters of the expression alignment module are adjusted.

[0017] In one embodiment, real-time training of the face-changing module includes: Input the expression-aligned source face image features and target face image features into the face swap module; The loss function is constructed based on the network adversarial loss, identity loss, reconstruction loss, perceptual loss, and corresponding weights. The identity loss is used to constrain the identity feature distance between the generated face and the source face, the reconstruction loss is used to determine whether the source face and the target face are from the same person, and the perceptual loss is used to evaluate the perceptual similarity between the generated result and the target face. The parameters of the face-changing module are adjusted according to the loss function.

[0018] Based on the same inventive concept, the fourth aspect of the present invention provides a training device for a face-changing model, wherein the face-changing model includes a pSp encoder, a key point detection module, an expression alignment module, a face-changing module, StyleGAN2, and a rendering module. The training device includes: Pre-training module, used to pre-train the expression alignment module; The real-time training module is used to perform real-time training on the face-changing module.

[0019] Based on the same inventive concept, the fifth aspect of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the face-swapping method described in the first aspect is implemented.

[0020] Based on the same inventive concept, the sixth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the face-swapping method described in the first aspect is implemented.

[0021] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows: The present invention provides a face swapping method, which is implemented by using a face swapping model. The face swapping model extracts features from a source face image and a target face image, detects key points on the target face image, aligns features of the source face image with features of the target face image after key point detection, adaptively fuses features of the source face image and features of the target face image after expression alignment, processes and renders the fused features, and then generates a face-swapped image.

[0022] By aligning the source face image features with the target face image features after key point detection, the source face can be driven in advance according to the facial movements of the target face. As a way of transmitting action information in the face-changing task, the face is changed while maintaining the consistency of facial movements, reducing the interference between action information and identity information, and adaptively fusing the source face image features aligned with the target face image features, which can solve the coupling problem of identity information and non-identity information (except action). The latent space feature decoupling method of this scheme (aligning the source face image features with the target face image features after key point detection, and adaptively fusing the source face image features aligned with the target face image features) enables the face-changing task to accurately replicate the target face expression, and can realize the fusion of the source face identity information and the target face non-identity attributes through adaptive learning. The process is clear and robust.

[0023] Furthermore, the present invention also performs attention enhancement on the latent vectors of two adjacent frames after adaptive fusion, thereby strengthening the continuity of adjacent frames and further improving the quality of face swapping in face videos.

[0024] Furthermore, the rendering module adopts the EG3D framework, which can reconstruct facial details more clearly than the face-changing task based on the 2D generative model, and reduce the artifacts and unreasonable perspective effects that may appear in the face-changing results.

[0025] In general, the key technical points of the present invention include: This application uses feature-dimensional expression alignment technology. A pre-trained expression-driven model is used to obtain the result of reenacting the source face based on the target face. The reenactment result and the latent vector of the source face are extracted using a pre-trained pSp model. A pre-trained facial key point detection model is used to obtain the target face key points as expression information. Using the source face latent vector and expression information as input, the expression alignment module is trained to predict the latent vector of the reenactment result, thereby aligning the expressions in the feature dimension.

[0026] This application uses a face-swapping technology based on adaptive feature fusion. Based on the output of the expression feature alignment module, this technology fuses the source and target face latent vectors, which have been aligned with the expression features, in the W+ space of StyleGAN2. It then selects latent vector slices from the middle layer and sets a linear layer network to learn the fusion parameters, thereby achieving an adaptive fusion face-swapping effect.

[0027] This application utilizes inter-frame attention technology. Based on a single-frame face swapping module, this technology enhances attention on the face swapping latent vectors of two adjacent frames, which are adaptively fused. An MLP network is used to map the latent vectors, and then an attention matrix is calculated to enhance the latent vector of the subsequent frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 This is a flow chart of a face swapping method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a single-frame face swapping in an embodiment of the present invention; Figure 3 This is a processing flow chart of the face-changing module (feature adaptive fusion module) in an embodiment of the present invention; Figure 4 Schematic diagram of multi-frame face swapping in an embodiment of the present invention; Figure 5This is a flow chart of the training of the expression feature alignment module in an embodiment of the present invention; Figure 6 This is a diagram of the training framework of the expression feature alignment module in an embodiment of the present invention; Figure 7 This is a flow chart of face-changing module training in an embodiment of the present invention; Figure 8 This is a structural diagram of a face swapping device in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0031] Example 1 This embodiment discloses a face swapping method. Figure 1 , the method comprising: S101: Obtain a source face image; S102: Obtain a target face image; S103: Inputting the source face image and the target face image into a trained face-changing model to generate a face-changed image; Among them, the face-changing model extracts features from the source face image and the target face image, detects key points on the target face image, aligns the features of the source face image with the features of the target face image after key point detection, adaptively fuses the features of the source face image and the features of the target face image after expression alignment, processes and renders the fused features to generate the face-changed image.

[0032] Specifically, the source face image and the target face image can be real face images or virtual face images.

[0033] After extracting features from the source and target facial images, the present invention aligns the extracted source facial image features with the target facial image features after key point detection. This allows the source face to be pre-driven based on the target face's facial movements, reducing interference between the target face's motion information and the source face image's identity information. Furthermore, adaptive fusion of the expression-aligned source facial image features with the target facial image features resolves the coupling between the source face image's identity information and the target face's non-identity information, thereby improving the quality of the resulting face-swapped image.

[0034] To further enhance the face-swapping effect, S103 includes: S1031: The face-changing model extracts features from the source face image and the target face image through the pSp encoder to obtain features of the source face image and features of the target face image; S1032: Perform key point detection on the target facial image using a key point detection module to obtain key point features of the target facial image as features of the target facial image after key point detection; S1033: Aligning the source facial image features with the target facial image features after key point detection through an expression alignment module to obtain expression-aligned source facial image features; S1034: adaptively fusing the features of the source facial image and the target facial image after expression alignment through the face swap module to obtain fused features; S1035: Process the fused features through StyleGAN2 to obtain three-plane features; S1036: Render the three-plane features output by StyleGAN2 through the rendering module to generate the face-swapped image.

[0035] Specifically, the face-changing model provided in this embodiment includes a pSp encoder, a key point detection module, an expression alignment module, a face-changing module, StyleGAN2, and a rendering module. After the source face image and the target face image are input into the trained face-changing model, the pSp encoder performs feature extraction on the source face image and the target face image to obtain the corresponding W+ space latent vector, i.e., the source face image features and the target face image features. The key point detection module performs key point detection on the target face image to obtain the key point features of the target face image. The source face image features and the key point features of the target face image are input into the expression alignment module to obtain the expression-aligned source face image features. The features and the target face image features are input into the face-changing model for adaptive fusion; the fused features are sent to StyleGAN2 to generate three-plane features, and then the three-plane features are rendered by the rendering module to generate the face-changing result.

[0036] The face swapping method proposed in this paper utilizes a face swapping technique based on decoupling of W+ latent space features. First, to address the issue of insufficient granularity of motion information, an expression alignment module is introduced. This module pre-drives the source face based on the target face's facial motion. This serves as a means of transferring motion information during the face swapping task. This allows the face swap to be performed while maintaining facial motion consistency, minimizing interference between motion and identity information. In the subsequent face swapping phase, to address the coupling of identity and non-identity information (besides motion information), a face swapping module is implemented that adaptively fuses the expression-aligned source and target facial image features. The generative model StyleGAN2 serves as its foundation. StyleGAN2 is a generative framework that transforms from noise to image and from coarse-grained to fine-grained. Compared to the first-generation StyleGAN, it possesses a more expressive W+ latent space. The W+ latent space provides style encoding for each generative layer of StyleGAN2, enabling progressive image generation.

[0037] Therefore, the present invention proposes a method for adaptively fusing the source face identity and the target face expression and posture based on the different attributes displayed by the corresponding control generated images of different generation layers of StyleGAN2, thereby realizing the selection and fusion of the latent vector corresponding to the W+ latent space. In addition, considering the high-quality performance of the neural rendering method in the face image generation task, the present invention selects the EG3D framework in other embodiments to maintain more generation details. For supervision, the present invention uses loss functions such as BlendFace identity information loss and feature contrast loss for face-changing tasks to constrain the identity information, non-identity information, and action and scene information of the generated results.

[0038] The overall framework of this method can be divided into two parts, namely the facial expression alignment part and the face swapping part. The process of single-frame face swapping is as follows: Figure 2 As shown, given the source face and target face , after the pSp encoder, the W+ space features of the two face images are obtained and Source facial features And the key point features extracted from the target face image using the FSFA key point detection method As the input of the facial expression alignment module, the expression-aligned W+ spatial features are obtained . and target face image features Perform adaptive fusion and send it to StyleGAN2 to generate three-plane features. Render the three-plane features according to the generation process of the EG3D model to generate the face-changing result. .

[0039] For the single-frame face swapping model, the expression alignment module needs to be pre-trained, the face swapping module (LatentSwap) needs to be trained in real time, and the remaining modules (pSp encoder, expression alignment module, StyleGAN2, and rendering module) are frozen.

[0040] In the application scenario of video face swapping, the face swapping method is as follows: Figure 4 As shown in Figure 2, an attention module is added to the inter-frame prediction to strengthen the continuity between the next frame prediction and the current frame.

[0041] Among them, the position of the expression feature alignment module Mot in the overall face-changing framework is as follows: Figure 2 This module is a Transformer structure, which is used to guide the expression of the input source face features based on the action information of the target face, that is, the key point features, so that the output enhanced features can be rendered to obtain a face image with the same expression and posture as the target face.

[0042] Furthermore, the reasoning process of the expression alignment module in S1033 includes: The target face image features after key point detection are used to guide the expression of the input source face image features to obtain the source face image features that have been aligned with the expression.

[0043] Furthermore, the reasoning process of the face-changing module in S1034 includes: splicing a specified intermediate layer latent vector in the expression-aligned source facial image latent vector with a target facial image latent vector, wherein the expression-aligned source facial image latent vector is used to represent features of the expression-aligned source facial image, and the specified intermediate layer latent vector is determined based on a correlation between the layer latent vector and non-identity information, the non-identity information including expression, posture, and hairstyle decoration; The concatenated vector is reduced in dimension through a linear layer, and the Sigmoid function is used to obtain the weighted variable; The weighted variable is used as the weight to weight the source face image latent vector and the target face latent vector to obtain the fused latent vector to represent the fused features.

[0044] Specifically, the face-changing module (i.e. Figure 2 The Latent Swap module in

[15] receives the expression-aligned W+ spatial latent vector and applies adaptive feature fusion and rendering to it. This face swapping framework is EG3D, which first uses StyleGAN2 to process the W+ spatial latent vector, outputting a three-plane feature, which is then rendered at a specified angle.

[0045] For StyleGAN2, the input W+ spatial latent vector can be divided into different granularity and semantic generation effects according to the dimension. This embodiment designs an adaptive latent vector fusion module (face-changing module) to explore the nonlinear fusion of the latent vector of the source face image that has been aligned with the expression (the source face image features that have been aligned with the expression) and the latent vector of the target face image (the target face image features). Each of the latent vectors The generation effect of the pre-trained StyleGAN2 model corresponding to the vector is found in the middle layer of the W+ space of StyleGAN2. The latent vector of the dimension is least correlated with the action information such as expression and posture, as well as non-identity information such as hairstyle and decoration. This embodiment uses the designed adaptive fusion layer to transfer the identity information of the intermediate layer latent vector of the target face image and the intermediate layer latent vector of the source face, such as Figure 3 shown.

[0046] In the specific implementation process, first specify the latent vector of the source face image aligned with the expression of the middle i layer and the target face image latent vector conduct Concatenate and then pass through a linear layer Perform dimensionality reduction and use the Sigmoid function to obtain weighted variables : (1) Use this as the weight to weight the source face image latent vector and the target face latent vector to obtain the fused latent vector: (2) in , is the value obtained from experimental observation, and the coding layer related to facial expressions and non-identity information is obtained by analyzing the StyleGAN2 network through the control variable method. In this embodiment, The value of is 5, The value is 12. For latent vector components that do not need to be fused, the latent vector of the target image remains unchanged. Subsequently, StyleGAN2 outputs the three-plane features corresponding to the fused latent vector and renders the face-swapped result.

[0047] Furthermore, the reasoning process of the rendering module in S1036 includes: The EG3D framework is used to render the three-plane features output by StyleGAN2 to generate the face-swapped image.

[0048] In order to enable the face-changing model to be used in scenarios with multiple frames of images, this embodiment further includes: Attention enhancement is performed on the latent vectors of two adjacent frames that have been adaptively fused.

[0049] Specifically, previous face swapping methods are mostly based on single images. When performing video face swapping, they often use a frame-by-frame approach, resulting in discontinuities in facial expression and movement between frames. Some existing methods first fine-tune the network using multi-frame face swapping tasks, then perform single-frame face swapping to improve inter-frame continuity. However, such methods fail to capture continuous motion information from a temporal perspective. This invention directly addresses the inter-frame continuity issue at the feature level.

[0050] In the application scenario of video face swapping, the face swapping method is as follows: Figure 4 As shown in the figure, an attention module is added to the inter-frame prediction to strengthen the continuity between the next frame prediction and the current frame. Based on image face swapping, the present invention adds an attention mechanism between video frames to strengthen the continuity of adjacent frames, further improving the quality of face video swapping.

[0051] In one embodiment, performing attention enhancement on latent vectors of two adjacent frames after adaptive fusion includes: Use the first multi-layer perceptron to concatenate the latent vectors of two adjacent frames of images and then perform mapping; Multiply the mapped vector by the latent vector of the next burst of images mapped by the second multi-layer perceptron, calculate the attention, and process the calculated attention through the Sigmoid function; The result of the attention processing is added to the latent vector of the next burst of frames mapped by the second multi-layer perceptron to obtain the enhanced latent vector.

[0052] In the specific implementation process, Figure 4 As shown in the figure (the face images in this figure are from the public dataset VFHQ), a temporal attention module is added between adjacent face-swapped frames to modify the fused latent vector. The temporal attention module acts on the latent vectors of two adjacent frames. and the latent vector of the t+1th frame For example, the temporal attention module first uses a multi-layer perceptron The two latent vectors are concatenated and mapped, and then combined with the multi-layer perceptron The hidden vector of the mapped t+1th frame image Perform element-wise multiplication to calculate attention. Attention is processed using the Sigmoid function and compared with the latent vector of the source t+1 frame image Add together and get the enhanced latent vector as : (3) Then, according to the single frame generation and rendering steps, the face-changing results of the video frame are output.

[0053] In general, the beneficial technical effects of the present invention include: (1) The latent space feature decoupling method is used to enable the face swapping task to accurately reproduce the target face expression. It can also achieve the fusion of the source face identity information and the target face non-identity attributes through adaptive learning. The process is clear and robust.

[0054] (2) The 3D generative model is used for face-changing tasks. Compared with face-changing tasks based on 2D generative models, the reconstruction of facial details is clearer and the artifacts and unreasonable perspective effects that may appear in the face-changing results are reduced.

[0055] (3) It has far-reaching application prospects. Due to the superior property of latent space feature decoupling, this scheme can be used to edit facial attributes on this basis and assist in achieving more facial tasks.

[0056] Example 2 Based on the same inventive concept, this embodiment discloses a face-swapping device, see Figure 8 ,include: A first obtaining module 401 is used to obtain a source face image; The second obtaining module 402 is used to obtain a target face image; An input module 403 is configured to input the source face image and the target face image into a trained face-changing model to generate a face-changed image; Among them, the face-changing model extracts features from the source face image and the target face image, detects key points on the target face image, aligns the features of the source face image with the features of the target face image after key point detection, adaptively fuses the features of the source face image and the features of the target face image after expression alignment, processes and renders the fused features to generate the face-changed image.

[0057] Since the device described in Example 2 of the present invention is used to implement the face-swapping method described in Example 1 of the present invention, the specific structure and variations of the device are readily apparent to those skilled in the art based on the method described in Example 1 of the present invention, and therefore will not be further described here. All devices used in the method described in Example 1 of the present invention fall within the scope of protection of the present invention.

[0058] Example 3 Based on the same inventive concept, this embodiment discloses a training method for a face-changing model, wherein the face-changing model includes a pSp encoder, a key point detection module, an expression alignment module, a face-changing module, StyleGAN2, and a rendering module. The training method includes: pre-training the expression alignment module and real-time training of the face-changing module.

[0059] Specifically, the expression alignment module is pre-trained, the face swapping module is trained in real time, and the remaining modules are frozen.

[0060] See Figure 5 ,The training process of the expression alignment module includes: S201: Acquire source face image samples and target face image samples; S202: replaying the source face image sample using a pre-trained face replay model to obtain a replay result; S203: Inputting the features of the replay result extracted by the pSp encoder and the features of the target face image sample extracted by the key point detection module into the expression alignment module for prediction; S204: Calculate the feature contrast loss based on the consistency between the features predicted and output by the expression alignment module and the features extracted by the pSp encoder of the replay result; S205: Calculating the image contrast loss based on the consistency between the face-swapped image corresponding to the features predicted and output by the expression alignment module and the reenactment result; S206: Adjusting parameters of the expression alignment module based on the feature contrast loss and the image contrast loss.

[0061] The training framework of the expression alignment module is as follows Figure 6 As shown, the face image in this figure comes from the public dataset VFHQ.

[0062] The source face image and target face image The W+ space latent vector is marked as (representing the extracted source face image features) and (represents the extracted target face image features), where, ,Right now and Each consists of 18 512-dimensional vectors, recorded as .in 、 Both are 512-dimensional vector slices. is a vector index.

[0063] according to The facial expressions and gestures of the user are simulated using the LivePortrait face reenactment model (Face Reenactment module). Repeat and record the result as (Replay result, i.e. the image obtained after replaying the source face image), and the W+ feature extracted by the pSp encoder is marked as The input of Transformer (expression alignment module) is extracted through the pre-trained pSp encoder of Dimensional W+ spatial facial features And using key point detection model extract 1 68-size facial key point features , the predicted output is also dimensional features, denoted as : (4) At the same time, the predicted As a latent vector for rendering and generating faces, the final predicted face driving result : (5) To monitor the performance of the expression alignment module, this implementation sets two loss functions during training: 1) Feature contrast loss: comparing the W+ features predicted by the Mot module Driving results with actual source faces Corresponding W+ space characteristics , set L2 loss constraint to ensure the consistency between the two; 2) Image contrast loss: contrast After subsequent rendering, the corresponding image is generated , and the driving result of the actual source face , set the L2 loss constraint to ensure the consistency between the two.

[0064] In the specific implementation process, the loss function of the expression alignment module is composed of the L2 loss of the latent vector of the predicted face re-enactment image and the LPIPS perceptual loss between the reconstructed image and the true value of the re-enactment image: (6) In the inference stage, the source face image and the target face image are input, and the W+ space latent vector corresponding to the image is extracted using the pSp model. The target face key points are obtained through the key point detection module, and then the latent vector and key points are spliced and sent to the expression alignment module to output the source face image features after expression alignment.

[0065] See Figure 7,The training process of the face-changing module includes: S301: Input the source face image features obtained by the expression alignment module and the target face image features after the pSp encoder into the face swap module; S302: Construct a loss function based on the generative adversarial loss, identity loss, reconstruction loss, perceptual loss, and corresponding weights. The goal of the generative adversarial loss is to enable the discriminator to correctly classify real samples and generated samples, while allowing the generator to generate samples that are as realistic as possible to deceive the discriminator. The identity loss is used to constrain the distance between the identity features of the generated face and the source face. The reconstruction loss is used to determine whether the source face and the target face are from the same person. The perceptual loss is used to evaluate the perceptual similarity between the generated result and the target face. S303: Adjust the parameters of the face-changing module according to the loss function.

[0066] In the specific implementation process, the loss function of the face-changing module consists of four components, including generating adversarial loss , BlendFace identity loss , reconstruction losses LPIPS perception loss The identity loss is used to constrain the identity feature distance between the generated face and the source face.

[0067] (1) Generate adversarial loss The goal of the generative adversarial loss is to enable the discriminator to correctly classify real samples and generated samples, while allowing the generator to generate samples that are as realistic as possible to deceive the discriminator. Its mathematical form draws on the idea of margin maximization from SVM and optimizes the performance of the discriminator and generator by introducing a boundary. The discriminator is denoted as , the generator (i.e. StyleGAN2 followed by EG3D rendering) is , .

[0068] Loss function of the discriminator Discriminator It is composed of convolutional neural networks, whose goal is to distinguish real samples and generate samples The discriminator loss function is defined as: (7) The goal of the discriminator is to minimize , so that it can correctly classify real samples and generated samples. It is to seek expectations.

[0069] Generator loss function For the generator , whose goal is to generate samples that are as realistic as possible so that the discriminator It is impossible to distinguish between real samples and generated samples. The generator loss function is defined as: (8) The goal of the generator is to make the discriminator output the generated samples As close to 1 as possible.

[0070] The total generative adversarial loss is the generator loss plus the discriminator loss: (9) (2) BlendFace identity loss This implementation uses the pre-trained BlendFace face recognition network to extract the identity features of the generated results. and identity features of the source face image , and using cosine similarity to calculate the distance, it can be written as: (10) (3) Reconstruction loss If the source face and the target face are from the same person, the generated result should look the same as the target face. This implementation uses the reconstruction loss as a regularization term, which can be written as: (11) If the source and target faces are from different identities, set this to 0.

[0071] In addition, this method also adds LPIPS perception loss It is used to evaluate the perceptual similarity between the generated result and the target face, and is supervised with weak feature matching loss.

[0072] (4) LPIPS loss The core idea of LPIPS is to use pre-trained CNN models (such as VGG or AlexNet) to extract feature representations of images and measure the perceptual similarity of images by calculating the distance between feature maps. Compared with traditional pixel-level losses (such as L1 and L2), LPIPS better reflects the perceptual characteristics of the human visual system.

[0073] Given a reference image and the image to be evaluated Input the image into a pre-trained CNN model (usually VGG or AlexNet) to extract multiple layers of feature maps. Assume that the extracted layers are , the feature maps of each layer are and ,in . Normalize the feature maps of each layer: and Through L2 normalization (or unit vectorization), the normalized feature vector is obtained and . Calculate the L2 distance between the normalized feature maps of each layer: (12) The feature distances of each layer are weighted and summed to obtain the final LPIPS value: (13) in, It is The weight coefficient of the layer is usually learned through training.

[0074] In summary, the overall loss function can be written as: (14) 、 、 、 are the weights of each loss respectively.

[0075] Example 4 Based on the same inventive concept, the present invention also provides a training device for a face-swapping model, wherein the face-swapping model includes a pSp encoder, a key point detection module, an expression alignment module, a face-swapping module, StyleGAN2, and a rendering module. The training device includes: Pre-training module, used to pre-train the expression alignment module; The real-time training module is used to perform real-time training on the face-changing module.

[0076] Since the apparatus described in Example 4 of the present invention is used to implement the face-swapping model training method described in Example 3 of the present invention, the specific structure and variations of the apparatus are readily apparent to those skilled in the art based on the method described in Example 1 of the present invention, and thus will not be further described here. All apparatuses used in the method described in Example 1 of the present invention fall within the scope of protection of the present invention.

[0077] Example 5 Based on the same inventive concept, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the first embodiment is implemented.

[0078] Since the computer-readable storage medium described in Example 5 of the present invention is the computer-readable storage medium used to implement the face-swapping method described in Example 1 of the present invention, the specific structure and variations of the computer-readable storage medium will be understood by those skilled in the art based on the method described in Example 1 of the present invention, and thus will not be described in detail here. All computer-readable storage media used in the method of Example 1 of the present invention are within the scope of protection of the present invention.

[0079] Example 6 The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first embodiment when executing the program.

[0080] Since the computer device described in Example 6 of the present invention is the computer device used to implement the face swapping method described in Example 1 of the present invention, the specific structure and variations of the computer device are readily understood by those skilled in the art based on the method described in Example 1 of the present invention, and therefore will not be further described here. All computer devices used in the method described in Example 1 of the present invention fall within the scope of protection of the present invention.

[0081] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, the present invention is intended to include such changes and modifications to the embodiments of the present invention if they fall within the scope of the claims and their equivalents.

Claims

1. A face swapping method, characterized in that: include: Obtain source face image; Obtain target face image; Inputting the source face image and the target face image into a trained face-swapping model to generate a face-swapping image; Among them, the face-changing model extracts features from the source face image and the target face image, detects key points on the target face image, aligns the features of the source face image with the features of the target face image after key point detection, adaptively fuses the features of the source face image and the features of the target face image after expression alignment, processes and renders the fused features to generate the face-changed image.

2. The face swapping method according to claim 1, wherein: The face-swapping model extracts features from a source face image and a target face image, detects key points on the target face image, aligns features of the source face image with features of the target face image after key point detection, adaptively fuses features of the source face image and features of the target face image after expression alignment, processes and renders the fused features, and generates a face-swapping image, including: The face-changing model extracts features of the source face image and the target face image through the pSp encoder to obtain features of the source face image and features of the target face image; Perform key point detection on the target face image through the key point detection module to obtain the key point features of the target face image as the target face image features after key point detection; The expression alignment module aligns the source face image features with the target face image features after key point detection to obtain the expression-aligned source face image features; The face-changing module adaptively fuses the features of the source and target facial images that have been aligned with the expression to obtain the fused features. The fused features are processed by StyleGAN2 to obtain three-plane features; The three-plane features output by StyleGAN2 are rendered through the rendering module to generate the face-changed image.

3. The face swapping method according to claim 1, wherein: The method further includes: performing attention enhancement on latent vectors of two adjacent frames that have been adaptively fused.

4. The face swapping method according to claim 3, wherein: The step of performing attention enhancement on the latent vectors of two adjacent frames after adaptive fusion includes: Use the first multi-layer perceptron to concatenate the latent vectors of two adjacent frames of images and then perform mapping; Multiply the mapped vector by the latent vector of the next burst of images mapped by the second multi-layer perceptron, calculate the attention, and process the calculated attention through the Sigmoid function; The result of the attention processing is added to the latent vector of the next burst of frames mapped by the second multi-layer perceptron to obtain the enhanced latent vector.

5. The face swapping method according to claim 2, wherein: The rendering module renders the three-plane features output by StyleGAN2 to generate the face-swapped image, including: The EG3D framework is used to render the three-plane features output by StyleGAN2 to generate the face-swapped image.

6. A face swapping device, characterized in that: include: A first acquisition module is used to obtain a source face image; The second acquisition module is used to obtain the target face image; An input module, configured to input the source face image and the target face image into a trained face-swapping model to generate a face-swapping image; Among them, the face-changing model extracts features from the source face image and the target face image, detects key points on the target face image, aligns the features of the source face image with the features of the target face image after key point detection, adaptively fuses the features of the source face image and the features of the target face image after expression alignment, processes and renders the fused features to generate the face-changed image.

7. A training method for a face-changing model, characterized in that: The face-changing model includes a pSp encoder, a key point detection module, an expression alignment module, a face-changing module, StyleGAN2 and a rendering module. The training method includes: pre-training the expression alignment module and real-time training of the face-changing module.

8. The face-changing model training method according to claim 7, characterized in that: Pre-train the expression alignment module, including: Obtain source face image samples and target face image samples; Use the pre-trained face replay model to replay the source face image sample and obtain the replay result; Input the features of the replay result extracted by the pSp encoder and the target face image features after the key point detection of the target face image sample into the expression alignment module for prediction; Calculate the feature contrast loss based on the consistency between the features predicted by the expression alignment module and the features extracted by the pSp encoder of the replay result; Calculate the image contrast loss based on the consistency between the face-swapped image and the reenactment result corresponding to the features predicted by the expression alignment module; Based on the feature contrast loss and the image contrast loss, the parameters of the expression alignment module are adjusted.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the face swapping method as claimed in any one of claims 1 to 5 is implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the face swapping method according to any one of claims 1 to 5 is implemented.

Images