Face changing method, model training method, electronic device, and storage medium

Abstract

Embodiments of the present application provide a face changing method, a model training method, an electronic device and a storage medium, a first image to be face changed and a second image to be face changed are obtained; key point detection is performed on the second image to obtain a face missing image corresponding to the second image and a face binary mask image; a face region image corresponding to the first image is input into a face reference model to obtain face feature information of a person in the first image; posture extraction is performed based on the second image and a person posture analysis model to obtain posture feature information of the person in the second image; the face binary mask image, the face missing image and the face feature information are taken as a main input side of a pre-trained diffusion model, and the posture feature information is taken as a conditional input side of the diffusion model, to obtain a target image in which the person in the second image is face changed. The present application improves the quality of the face changing effect.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and particularly relate to a face changing method, a model training method, an electronic device and a storage medium. BACKGROUND

[0002] With the development of science and technology, face changing technology no longer needs manual intervention and can use artificial intelligence technology to perform image face changing and video face changing. The face changing technology can be widely applied to film production, game development and creation of virtual characters, etc. The key point of the face changing technology is how to accurately and sufficiently transfer identity-related facial features, including structure and texture, to a target face.

[0003] In the related art, a face region of a source image is extracted through key points of the source image, and then the face region of the source image is fused into a target image according to key points of the target image, so as to complete image face changing. This face changing method only transfers the relevant facial features to the target face, and cannot guarantee that the attributes of the target face remain unchanged, such as failing to retain the background, clothing and posture of the target face, etc., so that the face changing effect of the face changed image is poor.

[0004] Therefore, how to generate a high-quality face changing effect becomes a problem to be solved. SUMMARY

[0005] In view of this, embodiments of the present application provide a face changing method, a model training method, an electronic device and a storage medium to at least partially solve the above problems.

[0006] According to a first aspect of embodiments of the present application, a face changing method is provided, comprising:

[0007] obtaining a first image to be face changed and a second image to be face changed; performing key point detection on the second image to obtain a face missing image and a face binary mask image corresponding to the second image; inputting a face region image corresponding to the first image into a face reference model to obtain face feature information of a person in the first image; performing posture extraction based on the second image and a person posture analysis model to obtain posture feature information of the person in the second image; taking the face binary mask image, the face missing image and the face feature information as a main input side of a pre-trained diffusion model, and taking the posture feature information as a conditional input side of the diffusion model, to obtain a target image after the person in the second image is face changed.

[0008] According to a second aspect of embodiments of the present application, a model training method is provided, comprising:

[0009] The first image sample to be face-changed and the second image sample to be face-changed are acquired; key point detection is performed on the second image sample to obtain a face missing image and a face binary mask image of the second image sample; a face region image sample corresponding to the first image sample is input into a face reference model to obtain face feature information; posture extraction is performed based on the second image sample and a character posture analysis model to obtain posture feature information of a character in the second image sample; the binary mask image, the face missing image, and the face feature information are taken as main input sides of a diffusion model, and the posture feature information is taken as a conditional input side of the diffusion model to determine noise loss, identity loss, reconstruction loss, and perception loss; and the diffusion model is trained in stages through the noise loss, the identity loss, the reconstruction loss, and the perception loss to obtain a trained diffusion model.

[0010] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface complete communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method according to the first aspect or the second aspect.

[0011] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, and the computer storage medium stores a computer program, and the program is executed by a processor to implement the method according to the first aspect or the second aspect.

[0012] According to the scheme provided by the embodiments of the present application, a first image to be face-changed and a second image to be face-changed are acquired; key point detection is performed on the second image to obtain a face missing image and a face binary mask image corresponding to the second image; a face region image corresponding to the first image is input into a face reference model to obtain face feature information of a character in the first image; posture extraction is performed based on the second image and a character posture analysis model to obtain posture feature information of the character in the second image; the face binary mask image, the face missing image, and the face feature information are taken as main input sides of a pre-trained diffusion model, and the posture feature information is taken as a conditional input side of the diffusion model to obtain a target image in which the character in the second image is face-changed. In this process, the face of the character in the first image is redrawn in a face missing region corresponding to the face missing image corresponding to the second image in combination with the face binary mask image and the face feature information of the character in the first image, which can ensure natural fusion with the face missing image and retain other parts of the second image except the face part, so that the face of the first image and the second image can be coordinated and fused. Furthermore, the posture feature information is taken as a conditional constraint input to the conditional input side of the diffusion model to control the posture of the character in the generated target image, and finally a high-fidelity and natural target image can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0014] Figure 1 is a total architecture diagram of a face changing method provided by the embodiments of the present application;

[0015] Figure 2 is a flowchart of a face changing method provided by the embodiments of the present application;

[0016] Figure 3 is a flowchart of a model training method provided by the embodiments of the present application;

[0017] Figure 4 is a total flowchart of a face changing method provided by the embodiments of the present application;

[0018] Figure 5 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0019] In order to make the personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those skilled in the art should belong to the scope of protection of the embodiments of the present application.

[0020] In order to adapt to the face changing method provided by the embodiments of the present application, the embodiments of the present application provide a total architecture diagram of a face changing method, that is, the face changing method of the embodiments of the present application can be applied to the face changing system provided by the embodiments of the present application. As shown in Figure 1As shown, the face changing system includes an image acquisition module, an image processing module, and a target image generation module. The image acquisition module is configured to acquire a small-angle video or multiple images, and divide the acquired video or multiple images into a first image (source image) and a second image (target image). In the training stage, the person in the second image and the person in the first image can be the same, and in the inference stage, the person in the second image and the person in the first image can be different. In the image processing module, the second image and the first image are processed respectively, and the face binary mask image, the face missing image, and the face key point pose image are acquired from the second image. The face region image is acquired from the first image, and the face region image is input into the face reference model to obtain the face feature information of the person in the first image. The face key point pose image is input into the person pose analysis model to obtain the pose feature information of the person in the second image. The face binary mask image, the face missing image, the pose feature information, and the face feature information are input into the diffusion model to obtain the target image of the person in the second image after face changing.

[0021] The following will be described in combination with Figure 1 and Figure 2 The face changing method provided by the embodiments of the present application is introduced. As Figure 2 shown, Figure 2 is a flowchart of the face changing method provided by the embodiments of the present application Figure 1 The face changing method provided by the embodiments of the present application can be executed by an electronic device, which can be a computer, a server, etc.

[0022] As Figure 2 shown, the face changing method comprises:

[0023] S101, acquiring a first image to be changed and a second image to be changed.

[0024] In the embodiments of the present application, the first image to be changed is a source image, and the second image to be changed is a target image, and the face region in the first image needs to be changed to the face region in the second image. The first image and the second image can be input by a user. The first image can be a single image or a single image frame, and the second image can be a single image or multiple images or multiple image frames constituting a video.

[0025] In inference, the person in the first image and the person in the second image can be the same, and in training, the person in the first image and the person in the second image can be different. And in training, it is trained in two stages, such as in the first stage of training, selecting two images of different light, expression, and background in the video or image set of the same person as the second image and the first image respectively; in the second stage of training, selecting the image frames corresponding to the continuous L frames of video clips in the video of the same person and a random 1 frame of image of different light, expression, and background as the second image and the first image respectively.

[0026] S102, key point detection is performed on the second image to obtain a face missing image corresponding to the second image and a face binary mask image.

[0027] In the embodiments of the present application, key point detection is used to identify specific key positions or feature points in an image or video, which are usually prominent features on an object, such as joints, corner points, edges, or other local parts with unique characteristics. The face binary mask image refers to a binary image specifically used to represent the face region. In such an image, the pixel values of the face part are usually set to 255 (white), indicating the foreground; while the pixel values of the non-face part are set to 0 (black), indicating the background. Such a mask image is mainly used to highlight and isolate the face region. The face missing image corresponding to the second image is an image in which the face region is covered in the second image.

[0028] In the embodiments of the present application, a face key point detection algorithm can be used to detect the face region from the first image and the second image, to obtain the face missing image and the face binary mask image.

[0029] S103, inputting the face region image corresponding to the first image into the face reference model to obtain the face feature information of the person in the first image.

[0030] Face feature information refers to various structures and details on the face, which can be divided into two categories: geometric features and texture features. Among them, geometric features mainly focus on the positions of key points (landmarks) in the face, such as the positions and shapes of eyes, nose, mouth, eyebrows, and face contours. Geometric features can be used to describe the basic structural layout of the face, including but not limited to eyes, nose, mouth, and eyebrows. Texture features involve the details of the face skin, such as wrinkles, spots, pores, and hair. Texture features include but are not limited to skin texture, hair details, color information, and surface details.

[0031] In this embodiment, the face region in the first image is obtained through operations such as face detection and face alignment. Then, the face is cropped out to obtain the corresponding face region image. The face reference model can provide a face reference and includes a pre-trained face encoder, a linear layer, and a normalization layer. The specific process of inputting the obtained face region image into the face reference model is as follows: first, the face region image is input into the face encoder, then through the linear layer, and then through the normalization layer to obtain the facial feature information of the person in the first image.

[0032] S104. Based on the second image and the human posture analysis model, perform posture extraction to obtain the posture feature information of the human in the second image.

[0033] Facial landmark pose mapping is a technique that identifies the location of specific facial features (such as eyes, nose, mouth, chin, etc.) in an image or video. These landmarks are commonly used in various applications such as facial recognition, expression analysis, head pose estimation, and facial feature tracking, and include the pose and facial features of a person in a second image.

[0034] In embodiments of this application, facial landmark pose images are obtained from the second image (or a sequence of facial landmark pose images if the second image is an image frame). For example, an open-source algorithm is used to detect the second image to obtain facial landmark pose images. The obtained facial landmark pose images are then input into a person pose analysis model to obtain the pose feature information of the person in the second image. This pose feature information is used to control the pose of the person in the final generated target image after face swapping in the second image.

[0035] In embodiments of this application, the person pose analysis model may include a pose cueing ControlNet model and a Variational Autoencoder (VAE) encoder. Facial keypoint pose images are obtained from a second image. These images are input into the VAE encoder to obtain a latent space feature map of the pose. This latent space feature map is then input into the pose cueing ControlNet model, specifically a copy of the coded blocks and intermediate blocks of the diffusion model. After passing through zero-convolutional layers, the coded blocks and intermediate blocks of the ControlNet model are added to the decoded blocks and intermediate blocks of the diffusion model via residual connections, thus inputting the pose feature information into the diffusion model. Specifically, the ControlNet model branch creates a trainable copy of the coded blocks and intermediate blocks of the diffusion model U-Net and adds additional zero-convolutional layers. The ControlNet's coded blocks (Downblocks) and intermediate blocks are added to the diffusion model's decoded blocks (Upblocks) and intermediate blocks. The output of the pose reference ControlNet is passed to the corresponding layer of the face-swapping generation diffusion model via residual connections.

[0036] S105. The face binary mask, the face missing image, and the face feature information are used as the main input side of the pre-trained diffusion model, and the pose feature information is used as the conditional input side of the diffusion model to obtain the target image of the person in the second image after face swapping.

[0037] Pre-trained diffusion models allow the model to generate data based on additional input information (conditions), which can be category labels, text descriptions, another image, or any form of guiding signal. The purpose is to guide the generation process so that the generated results conform to specific conditions or attributes.

[0038] Stable Diffusion (LDM) is a classic latent diffusion model. LDM employs a variational autoencoder. Given an input image, the encoder maps the image to the latent space, allowing diffusion to occur within this space. During the forward diffusion process, Gaussian noise is gradually added to the latent variables over T iterations. The backward diffusion process aims to progressively recover the image from the previous state, and finally, the image is decoded to obtain the final image.

[0039] In this embodiment, the pre-trained diffusion model is initialized using a pre-trained StableDiffusion redrawing model. Its function is to generate the target image of the person in the second image after face swapping. The pre-trained diffusion model of this application can be considered a redrawing diffusion model, specifically referring to the process of regenerating the target image of the person after face swapping in the missing facial region of the second image using facial feature information from the first image, while ensuring natural integration with the redrawing background. When using the redrawing diffusion model to generate face-swapped images, the portion of the target image other than the face is retained, i.e., the resulting image with missing face is used as the redrawing background. The face from the first image is redrawn on the missing facial region, so that the face in the first image and the second image are harmoniously integrated to obtain the target image.

[0040] Specifically, the binary face mask, the image with missing face, and facial feature information are used as the main input to the pre-trained diffusion model, while pose feature information is used as the conditional input to guide the pre-trained diffusion model, resulting in the target image of the person in the second image after face swapping. The target image is obtained by redrawing the face region image of the first image in the missing face region corresponding to the image with missing face.

[0041] Understandably, in the embodiments of this application, a first image to be face-swapped and a second image to be face-swapped are obtained; key point detection is performed on the second image to obtain the corresponding missing face image and face binary mask; the face region image corresponding to the first image is input into a face reference model to obtain the facial feature information of the person in the first image; pose extraction is performed based on the second image and the person's pose analysis model to obtain the pose feature information of the person in the second image; the face binary mask, the missing face image, and the facial feature information are used as the main input side of a pre-trained diffusion model, and the pose feature information is used as the conditional input side of the diffusion model to obtain the target image of the person in the second image after face swapping. In this process, by combining the face binary mask and the facial feature information of the person in the first image, the face of the first image can be redrawn in the missing face region corresponding to the missing face image of the second image, and it can be guaranteed that it can be naturally integrated with the missing face image, preserving the other parts of the second image except for the face part, so that the face of the first image and the second image can be harmoniously integrated. Furthermore, the pose feature information is used as a conditional constraint input to the conditional input side of the diffusion model to control the pose of the person in the generated target image, ultimately resulting in a high-fidelity, natural target image.

[0042] In some embodiments of this application, S105 can be implemented by S1051 to S1056, which will be described through the following steps.

[0043] S1051. Based on the face binary mask image, latent space noise, and face missing image, feature extraction is performed to obtain the first latent space feature map.

[0044] Feature extraction refers to the process of extracting information or patterns from raw data that are useful for solving a specific problem. These extracted features are often more representative of the data's essence than the original data, while reducing redundancy and complexity. Feature extraction mainly includes data preprocessing, feature selection, feature extraction methods, and feature transformation.

[0045] In some embodiments of this application, feature extraction can be performed on the face binary mask image, latent space noise, and face missing image through preprocessing, feature selection, feature extraction methods, and feature transformation to obtain a first latent space feature map. The first latent space feature map is a more comprehensive and representative feature map of the face binary mask image, latent space noise, and face missing image.

[0046] S1052. Input the first latent space feature map into the self-attention layer to obtain the first query matrix.

[0047] The query matrix is ​​the starting point for searching or querying in the attention layer. Each input position generates a corresponding query matrix, which is used to calculate similarity with the key matrices of all positions to determine which positions' information should be focused on (or "attention") at the current position. The key matrix represents the target being queried or retrieved in the attention layer. Each input position also generates a corresponding key matrix, used to calculate similarity with the query matrix, helping the model understand the relationships between different positions in the sequence. The value matrix is ​​the vector containing the actual information in the attention layer. Each input position also generates a corresponding value matrix. It contains specific information about each position in the sequence, which is weighted and summed based on the attention weights calculated from the query matrix and the key matrix.

[0048] In some embodiments of this application, the self-attention layer captures rich contextual information by calculating the interrelationships between elements in the sequence, thereby improving the model's ability to process sequence data. After the first latent space feature map is input into the self-attention layer, the self-attention layer first maps it to different vector spaces to obtain the first query matrix.

[0049] S1053. Determine the reweighted second latent space feature map based on the first query matrix.

[0050] In some embodiments of this application, after the first latent space feature map is input into the self-attention layer, the self-attention layer first maps it to different vector spaces. After obtaining the first query matrix, a first key matrix and a first value matrix are also obtained. Based on the first query matrix, the first value matrix, and the first key matrix, attention scores are calculated, weights are normalized, and weighted summation is performed to obtain a reweighted second latent space feature map.

[0051] S1054. Input the first latent space feature map and facial feature information into the image cross-attention layer to obtain the first query matrix, the second value matrix, and the second key matrix.

[0052] In some embodiments of this application, after inputting the first latent space feature map and facial feature information into the image cross-attention layer, a second value matrix, a second key matrix, and a first query matrix are obtained. The query matrix determined in the image cross-attention layer is the same as the query matrix of the self-attention layer in the diffusion model. That is, the first query matrix corresponding to the self-attention layer is the same as the first query matrix corresponding to the image cross-attention layer.

[0053] For example, Q = ZW q K′=c i W′ k V′=c i W′ v Z is the first latent space feature map, W qFor the first preset weight matrix corresponding to the self-attention layer, c i For facial feature information, W′ k and W′ v These are the second and third preset weight matrices corresponding to the image cross-attention layer. Where W... q , W′ k and W′ v The trainingable linear mapping layer consists of a first preset weight matrix, a second preset weight matrix, and a third preset weight matrix. These three preset weight matrices are automatically learned during the training process of the diffusion model. During training, the goal of the diffusion model is to minimize a certain loss function (such as cross-entropy loss), which typically involves adjusting the parameters of the diffusion model, including W. q , W′ k and W′ v Once training is complete, the first, second, and third preset weight matrices will be saved for use during model inference. t Z represents facial feature information, and Z is the first latent space feature vector.

[0054] S1055. Determine the reweighted third latent space feature map based on the first query matrix, the second value matrix, and the second key matrix.

[0055] S1056. Determine the target image based on the second latent space feature map, the third latent space feature map, and pose feature information.

[0056] For example, the third latent space feature map is calculated according to the following formula (1):

[0057]

[0058] In the above formula (1), Z″ is the third latent space feature map, d k Q is the dimension of the key matrix, V′ is the first query matrix, V′ is the second value matrix, and K′ is the second key matrix.

[0059] After determining the third latent space feature map according to the above formula (1), the target image is determined based on the second latent space feature map, the third latent space feature map and the pose feature information.

[0060] It is understood that in some embodiments of this application, feature extraction is performed based on the face binary mask image, latent space noise, and the image with missing faces to obtain a first latent space feature map, reducing data redundancy and complexity. The first latent space feature map is input into a self-attention layer to obtain a first query matrix. A reweighted second latent space feature map is determined based on the first query matrix. The first latent space feature map and face feature information are input into an image cross-attention layer to obtain a first query matrix, a second value matrix, and a second key matrix. By sharing the query matrix, computational resources can be shared between different attention layers, thereby reducing the overall computational load and achieving information sharing. Finally, a reweighted third latent space feature map is determined based on the first query matrix, the second value matrix, and the second key matrix. The target image is determined based on the second latent space feature map, the third latent space feature map, and pose feature information, improving the generation quality of the target image.

[0061] It is understood that in some embodiments of this application, S1056 can be implemented by S1056A to S1056C, as described in the following steps.

[0062] S1056A. Perform a summation operation on the second latent space feature map and the third latent space feature map to obtain the fourth latent space feature map.

[0063] In some embodiments of this application, the second and third latent space feature maps are summed to fuse feature information from different sources or levels, which can improve the expressive power and performance of the diffusion model. When summing the second and third latent space feature maps, the two sets of feature maps must be aligned in spatial dimensions (height, width) and channel number (depth), i.e., they must have the same size. During the summation operation, the second and third latent space feature maps can be added element-wise to obtain the fourth latent space feature map.

[0064] For example, as shown in the following formula (2):

[0065]

[0066] In the above formula (2), Z′ is the second latent space feature map, and Z″ is the third latent space feature map. This is the feature map of the fourth hidden space.

[0067] S1056B: Reshape the fourth latent space feature map to obtain the fifth latent space feature map.

[0068] In some embodiments of this application, data reshaping allows changing the dimensions of data without altering its underlying data, for example, converting image data into sequence data for use in time series analysis.

[0069] For example, when the fourth latent space feature map is That is, the representation is a five-dimensional tensor z with shape (b, L, h, W, c), and we need to reshape it into a new shape ((b×h×w), L, c), to obtain This refers to the fifth latent space feature map. Here, b represents the batch size, L represents the time step or sequence length, h and w represent the image height and width, respectively, and c represents the number of channels. This reshaping is typically used to convert image data into a format suitable for specific operations such as time series analysis.

[0070] S1056C determines the target image based on the fifth hidden space feature map, temporal attention layer, pose feature information, and a preset decoder.

[0071] The preset decoder can be a decoder for a variational autoencoder (VAE). The role of the decoder in a VAE is to transform the latent vector sampled from the latent space back into the original data space, thereby generating reconstructed data that is as close as possible to the original input data, while preserving the key features of the data.

[0072] In some embodiments of this application, the fifth latent space feature map can be input into a temporal attention layer, i.e., a series of self-attention layers along the t-axis, to obtain a sixth latent space feature map. The sixth latent space feature map and pose feature information can be added together to obtain a processed latent space feature map. The processed latent space feature map is further processed by subsequent layers of the diffusion model, and finally decoded by a preset decoder to obtain the target image.

[0073] It is understood that, in some embodiments of this application, the second latent space feature map and the third latent space feature map are summed to obtain the fourth latent space feature map. By fusing feature information from different sources or different levels, the expressive power and performance of the diffusion model can be improved. The fourth latent space feature map is then reshaped to obtain the fifth latent space feature map, enabling the image data to be converted into a format suitable for specific operations. Based on the fifth latent space feature map, the temporal attention layer, pose feature information, and a preset decoder, the target image is determined, improving the naturalness and fluency of the generated target image.

[0074] In some embodiments of this application, S105 1 can be implemented by S105 1A to S105 1C, as described in the following steps.

[0075] S105 1A. Input the image with missing face into the pre-trained encoder to obtain the latent space feature map.

[0076] In some embodiments of this application, a face-missing image can be input into the encoder of a pre-trained variational autoencoder to obtain a latent space feature map such as a 4-channel image.

[0077] S105 1B. The latent space feature map, latent space noise and face binary mask map are concatenated to obtain the concatenated latent space feature map.

[0078] S105 1C. Based on the spliced ​​latent space feature map, feature extraction is performed to obtain the first latent space feature map.

[0079] Channel concatenation allows feature vectors from different levels to be combined, thus utilizing both high-level abstract features and low-level detailed features simultaneously. By increasing the number of channels, the model can learn more feature representations, which helps improve the model's expressive power and discriminative ability. Pose feature information already has noise added; concatenating pose feature information with added latent space noise can enhance the model's generalization ability, robustness, and perform data augmentation.

[0080] For example, the latent space feature vector of the 4-channel redrawn background, the pose feature information with added latent space noise of 4 channels, and the binary mask image of 1 channel are concatenated along the channels to obtain the input of 9 channels, which is the concatenated latent space feature vector.

[0081] In some embodiments of this application, after obtaining the stitched latent space feature map, preprocessing, feature selection, feature extraction methods, and feature transformation can be performed to extract features and obtain the first latent space feature map.

[0082] Understandably, in some embodiments of this application, a face-missing image is input into a pre-trained encoder to obtain a latent space feature map. The latent space feature map, latent space noise, and a binary face mask image are then concatenated to obtain a concatenated latent space feature map. Feature extraction is performed based on this concatenated latent space feature map to obtain a first latent space feature map. This process utilizes both high-level abstract features and low-level detailed features. By increasing the number of channels, the model can learn more feature representations, which helps improve the model's expressive power and discriminative ability, further enhancing the generation effect of the target image.

[0083] The following is combined Figure 1 and Figure 3 This application introduces the model training method provided in its embodiments. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of a model training process provided in an embodiment of this application.

[0084] S201. Obtain the first image sample to be swapped and the second image sample to be swapped.

[0085] In the embodiments of this application, the training of the diffusion model can be divided into two stages. Therefore, when collecting samples, in the first stage of training, two images with different lighting, expressions, and backgrounds are selected from the video or image set of the same person as the second image sample and the first image sample, respectively. In the second stage of training, the image frames corresponding to the video segments of consecutive L frames in the video of the same person and a random 1 frame of images with different lighting, expressions, and backgrounds are selected as the second image sample and the first image sample, respectively.

[0086] S202. Perform key point detection on the second image sample to obtain the face missing image and face binary mask image of the second image sample.

[0087] S203. Input the face region image sample corresponding to the first image sample into the face reference model to obtain face feature information.

[0088] S204. Based on the second image sample and the human posture analysis model, perform posture extraction to obtain the posture feature information of the human in the second image sample.

[0089] In the embodiments of this application, the implementation methods of obtaining the missing face image and the face binary mask in S202 and S102 are similar, the implementation methods of obtaining facial feature information in S203 and S103 are similar, and the implementation methods of obtaining the posture feature information of the person in S204 and S104 are similar, which will not be described in detail here.

[0090] S205. The binary mask image, the image with missing face, and the facial feature information are used as the main input side of the diffusion model, and the pose feature information is used as the conditional input side of the diffusion model to determine the noise loss, identity loss, reconstruction loss, and perception loss. The diffusion model is trained in stages through the noise loss, identity loss, reconstruction loss, and perception loss to obtain the trained diffusion model.

[0091] A loss function is a function used to quantify the difference between a model's prediction and the actual target value. The goal of training a model is to minimize this loss function by adjusting the model's parameters, thereby making the model's predictions as close as possible to the true output.

[0092] In the embodiments of this application, pose feature information is used as the conditional input of the diffusion model to be trained, and a binary mask image, a face-missing image, and facial feature information are used as the main input of the diffusion model to be trained, resulting in predicted noise and a predicted image. Based on the predicted noise and the predicted image, noise loss, identity loss, reconstruction loss, and perceptual loss are determined. Finally, the diffusion model is trained using the noise loss, identity loss, reconstruction loss, and perceptual loss until the training conditions are met, resulting in the trained diffusion model.

[0093] When training the diffusion model in stages, the model is initialized using pre-trained weights from the redrawn StableDiffusion algorithm. Training can be divided into two stages. In the first stage, a single face-swapping image is generated, enabling the generation of the target image based on a first image sample and a second image sample. During this first stage, the temporal attention layer of the diffusion model is not involved in training. In the second stage, the temporal attention layer is trained. In this stage, the first image and the corresponding image frame sequence of a consecutive L-frame video segment are input to generate the face-swapping video. Therefore, in both training stages, the noise loss includes the first noise loss corresponding to the image sample and the second noise loss corresponding to the image frame sample; the identity loss includes the first identity loss corresponding to the image sample and the second identity loss corresponding to the image frame sample; the reconstruction loss includes the first reconstruction loss corresponding to the image sample and the second reconstruction loss corresponding to the image frame sample; and the perceptual loss includes the first perceptual loss corresponding to the image sample and the second perceptual loss corresponding to the image frame sample.

[0094] When training the diffusion model, if the person pose analysis model is a ControlNet model, the ControlNet model can be trained using noise loss, identity loss, reconstruction loss, and perception loss. The ControlNet initialization uses pre-trained weights from the pose-referenced ControlNet.

[0095] It is understood that, in the embodiments of this application, based on the obtained first image sample to be face-swapped and second image sample to be face-swapped, the missing face image, the face binary mask, the face feature information and the person's pose feature information are obtained respectively. Then, the pose feature information is used as a conditional constraint input to the conditional input side of the diffusion model. The binary mask, the missing face image and the face feature information are used as the main input side of the diffusion model to be trained to determine the noise loss, identity loss, reconstruction loss and perceptual loss. Using multiple loss values ​​for training can improve the performance and generalization ability of the model.

[0096] In some embodiments of this application, S205 can be implemented by S205 1 to S2054, which will be explained in detail through the following steps.

[0097] S205 1. Summing the first noise loss, the first identity loss, the first reconstruction loss, and the first perception loss to obtain the first total loss.

[0098] In some embodiments of this application, in the first stage, a single face-swapping image is generated, such that a target image can be generated based on a first image and a second image.

[0099] For example, the first noise loss can be calculated using the following formula (3):

[0100]

[0101] In the above formula (3), when the human pose analysis model is the ControlNet model, T θ ,∈ θ These represent the pose reference ControlNet model and the face-swapping generation diffusion model, respectively, and are the trainable parts of the models. Z bg For the latent space feature map, z p c is the latent space feature map corresponding to the character's posture. i Here, z represents facial feature information, t is the time step for denoising, and z represents the time step for denoising. t It is the latent space feature map at time t.

[0102] For example, the first identity loss can be calculated using the following formula (4):

[0103]

[0104] In formula (4) above, Arc is the pre-trained encoder of ArcFace. For the first identity loss, cos represents the cos similarity, I gen Predicted image, I source This is the first image.

[0105] For example, the first reconstruction loss can be calculated using the following formula (5):

[0106]

[0107] In the above formula (5), For the first reconstruction loss, I target This is the second image.

[0108] For example, the first perception loss can be calculated using the following formula (6):

[0109]

[0110] In the above formula (6), For the first perception loss, f vgg Let l be a pre-trained VGG-19 network, and l be the output of the l-th layer of the VGG-19 network.

[0111] For example, the first total loss can be calculated using the following formula (7):

[0112]

[0113] In the above formula (7), For the first total loss, λ i , λ r and λ p These are the corresponding hyperparameters.

[0114] S2052. The second noise loss, the second reconstruction loss, the second identity loss, and the second perception loss are summed to obtain the second total loss.

[0115] In some embodiments of this application, the second stage inputs a first image and an image frame sequence corresponding to a video segment of consecutive L frames to generate a face-swapping video. The second noise loss, the second reconstruction loss, the second identity loss, and the second perception loss can be calculated according to formulas (3) to (6), and the calculated second noise loss, second reconstruction loss, second identity loss, and second perception loss are summed to obtain the second total loss.

[0116] For example, taking the second identity loss as an example, using the first image and at least one image frame, at least one predicted image is generated by swapping the face in the first image onto at least one image frame. The average value of the at least one predicted image is calculated, and finally, the calculated average value and the first image are substituted into formula (4) to calculate the second identity loss. The second noise loss, the second reconstruction loss, and the second perception loss are similarly calculated based on the average value of at least one image frame.

[0117] S2053. The first preset layer of the diffusion model is trained using the first total loss to obtain the diffusion model after the first stage of training; the first preset layer does not include the time attention layer.

[0118] S2054. When the weights of the first preset layer are fixed, the temporal attention layer of the diffusion model after the first stage of training is trained using the second total loss to obtain the trained diffusion model.

[0119] In some embodiments of this application, during the first training phase, the temporal attention layer of the diffusion model is temporarily removed, while all other layers (the first preset layer) participate in the training. The first preset layer of the diffusion model is trained using the first loss to obtain the diffusion model after the first training phase. After obtaining the diffusion model after the first training phase, training is performed again. This time, only the temporal attention layer that generates the diffusion model is trained, while the weights of other parts remain fixed.

[0120] In the training of the diffusion model, when the human pose analysis model is a ControlNet model, during the first training phase, the temporal attention layer of the ControlNet model is temporarily removed, while all other layers (the first preset layer) participate in the training. The first preset layer of the ControlNet model is trained using the first loss, resulting in the diffusion model after the first training phase. After obtaining the ControlNet model after the first training phase, training is performed again, this time only training the temporal attention layer of the ControlNet model, while the weights of other parts remain fixed. This process continues until training is complete, resulting in the trained diffusion model and ControlNet model.

[0121] It is understood that in some embodiments of this application, the first noise loss, the first identity loss, the first reconstruction loss, and the first perception loss are summed to obtain the first total loss, and the second noise loss, the second reconstruction loss, the second identity loss, and the second perception loss are summed to obtain the second total loss. By summing the various losses, different loss functions are defined for each type of data, so that the model can effectively integrate multiple information sources and improve the model's performance and generalization ability. The first preset layer of the diffusion model is trained using the first total loss to obtain the diffusion model after the first stage of training. The first preset layer does not include the temporal attention layer. When the weights of the first preset layer are fixed, the temporal attention layer of the diffusion model after the first stage of training is trained using the second total loss to obtain the trained diffusion model. On the one hand, phased training can decompose complex training tasks into multiple relatively simple sub-tasks. Each stage focuses on solving a specific problem or improving a certain aspect of the model's ability, which helps the model gradually approach the final goal and improves training efficiency and effectiveness. On the other hand, starting training from simple tasks and then gradually introducing more complex tasks allows the model to gradually build up its understanding and processing capabilities for complex problems during the training process. Each training stage may use data from different sources, formats, or domains. This data diversity helps the model learn richer feature representations and improves the model's generalization ability.

[0122] In the embodiments of this application, the above steps are combined, such as Figure 4 As shown, Figure 4 This is a general flowchart of a face-swapping method provided in an embodiment of this application. Figure 4In this process, a second image and a first image are acquired from a small-angle video or multiple images. The second and first images are preprocessed to obtain a binary face mask image, a missing face image, a facial landmark pose image, and a face region cropped from the first image, respectively. The missing face image is input into a VAE encoder for processing to obtain a latent space feature map. The latent space feature map, the binary face mask image, and noise are concatenated to obtain a concatenated latent space feature map, which is then input into a diffusion model. The facial landmark pose image is input into the VAE encoder to obtain a latent space feature map, which is then input into a person pose analysis model to obtain the pose feature information of the person in the second image. This pose feature information is then input into the diffusion model. The facial region corresponding to the first image is processed by a facial reference model (which may include a face encoder and linear layers, and may also include a normalization layer) to obtain facial feature information. This facial feature information is then input into the image cross-attention layer of the diffusion model (the image cross-attention layer and the self-attention layer in the diffusion model are decoupled). The diffusion model processes the input data to obtain a processed latent space feature map, which is then input into the VAE decoder to obtain the target image. Here, number 1 represents the image cross-attention layer, and number 2 represents the self-attention layer.

[0123] Reference Figure 5 This document illustrates a schematic diagram of an electronic device according to an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0124] like Figure 5 As shown, the electronic device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0125] in:

[0126] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0127] Communication interface 504 is used to communicate with other electronic devices or servers.

[0128] The processor 502 is used to execute program 510, specifically the relevant steps in the above method embodiments.

[0129] Specifically, program 510 may include program code that includes computer operation instructions.

[0130] The processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0131] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0132] Specifically, program 510 can be used to cause processor 502 to perform the operations corresponding to the methods described in the above method embodiments.

[0133] The specific implementation of each step in program 510 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0134] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0135] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0136] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0137] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A face-swapping method, characterized in that, include: Obtain the first image to be face-swapped and the second image to be face-swapped; Key point detection is performed on the second image to obtain the face missing image and face binary mask image corresponding to the second image; The facial region image corresponding to the first image is input into the facial reference model to obtain the facial feature information of the person in the first image; Based on the second image and the human posture analysis model, posture extraction is performed to obtain the posture feature information of the person in the second image; Using the binary face mask, the missing face image, and the facial feature information as the main input of a pre-trained diffusion model, and the pose feature information as the conditional input of the diffusion model, the target image of the person in the second image after face swapping is obtained. This includes: extracting features based on the binary face mask, latent space noise, and the missing face image to obtain a first latent space feature map; inputting the first latent space feature map into a self-attention layer to obtain a first query matrix; determining a reweighted second latent space feature map based on the first query matrix; inputting the first latent space feature map and the facial feature information into an image cross-attention layer to obtain the first query matrix, a second value matrix, and a second key matrix; determining a reweighted third latent space feature map based on the first query matrix, the second value matrix, and the second key matrix; and determining the target image based on the second latent space feature map, the third latent space feature map, and the pose feature information. The diffusion model includes a self-attention layer and an image cross-attention layer.

2. The method according to claim 1, characterized in that, The second image is an image frame from a video; the diffusion model also includes a temporal attention layer; Determining the target image based on the second latent space feature map, the third latent space feature map, and the pose feature information includes: The second latent space feature map and the third latent space feature map are summed to obtain the fourth latent space feature map. The fourth latent space feature map is reshaped to obtain the fifth latent space feature map; The target image is determined based on the fifth latent space feature map, the temporal attention layer, the pose feature information, and the preset decoder.

3. The method according to claim 1, characterized in that, The step of extracting features based on the face binary mask image, latent space noise, and the face missing image to obtain a first latent space feature map includes: The image of the missing face is input into a pre-trained encoder to obtain a latent space feature map; The latent space feature map, latent space noise, and face binary mask map are concatenated to obtain the concatenated latent space feature map. Feature extraction is performed based on the stitched latent space feature map to obtain the first latent space feature map.

4. The method according to claim 1, characterized in that, The first query matrix is ​​determined based on the first latent space feature map and the first preset matrix; the second value matrix is ​​determined based on the facial feature information and the second preset weight matrix; the second key matrix is ​​determined based on the facial feature information and the third preset weight matrix.

5. A model training method, characterized in that, include: Obtain the first image sample to be face-swapped and the second image sample to be face-swapped; Key point detection is performed on the second image sample to obtain the face missing image and face binary mask image of the second image sample; The facial region image sample corresponding to the first image sample is input into the facial reference model to obtain facial feature information; Based on the second image sample and the human pose analysis model, pose extraction is performed to obtain the pose feature information of the human in the second image sample; The binary mask image, the missing face image, and the facial feature information are used as the main input to the diffusion model, and the pose feature information is used as the conditional input to determine noise loss, identity loss, reconstruction loss, and perceptual loss. The noise loss includes a first noise loss corresponding to the image sample and a second noise loss corresponding to the image frame sample; the identity loss includes a first identity loss corresponding to the image sample and a second identity loss corresponding to the image frame sample; the reconstruction loss includes a first reconstruction loss corresponding to the image sample and a second reconstruction loss corresponding to the image frame sample; and the perceptual loss includes a first perceptual loss corresponding to the image sample and a second perceptual loss corresponding to the image frame sample. The diffusion model is trained in stages using the noise loss, identity loss, reconstruction loss, and perception loss to obtain a trained diffusion model. This includes: summing the first noise loss, first identity loss, first reconstruction loss, and first perception loss to obtain a first total loss; summing the second noise loss, second reconstruction loss, second identity loss, and second perception loss to obtain a second total loss; training a first preset layer of the diffusion model using the first total loss to obtain a diffusion model trained in the first stage; the first preset layer is the diffusion model with the temporal attention layer removed; and with the weights of the first preset layer fixed, training the temporal attention layer of the diffusion model trained in the first stage using the second total loss to obtain the trained diffusion model.

6. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to the method described in any one of claims 1-4 or claim 5.

7. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-4 or claim 5.

Images