Face changing method, model training method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a face changing method, a model training method, electronic equipment and a storage medium. The method comprises the following steps: acquiring a first image of a to-be-changed face and a second image of a changed face; performing key point detection on the second image to obtain a face missing image and a face binary mask image corresponding to the second image; inputting a face region image corresponding to the first image into a face reference model to obtain face feature information of a person in the first image; performing posture extraction based on the second image and a figure posture analysis model to obtain posture feature information of a figure in the second image; and taking the face binary mask image, the face missing image and the face feature information as a main input side of a pre-trained diffusion model, and taking the posture feature information as a condition input side of the diffusion model to obtain a target image after the face of the person in the second image is changed. The quality of the face changing effect is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a face-changing method, a model training method, an electronic device, and a storage medium. Background Art

[0002] With the development of science and technology, face-changing technology no longer requires human participation. Artificial intelligence technology can be used to perform image and video face-changing. Face-changing technology can be widely used in film production, game development, and creation of virtual characters. The key point of face-changing technology is how to accurately and fully transfer identity-related facial features, including structure and texture, to the target face.

[0003] In the related art, the facial area of ​​the source image is extracted through the key points of the source image, and then the facial area of ​​the source image is fused into the target image according to the key points of the target image, thereby completing the image face swap. This face swap method only transfers the relevant facial features to the target face, and cannot guarantee that the attributes of the target face remain unchanged. For example, the background, clothing, posture and other attributes of the target face are not retained, resulting in a poor face swap effect.

[0004] Therefore, how to generate high-quality face-changing effects becomes a problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, embodiments of the present application provide a face-changing method, a model training method, an electronic device and a storage medium to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of an embodiment of the present application, a face-changing method is provided, comprising:

[0007] Acquire a first image to be replaced with a face and a second image to be replaced with a face; perform key point detection on the second image to obtain a face-missing image and a face binary mask image corresponding to the second image; input the face area image corresponding to the first image into a face reference model to obtain facial feature information of a person in the first image; perform posture extraction based on the second image and a person posture analysis model to obtain posture feature information of the person in the second image; use the face binary mask image, the face-missing image and the facial feature information as the main input side of a pre-trained diffusion model, and use the posture feature information as the conditional input side of the diffusion model to obtain a target image after the person in the second image is replaced with a face.

[0008] According to a second aspect of an embodiment of the present application, a model training method is provided, comprising:

[0009] Acquire a first image sample to be replaced and a second image sample to be replaced; perform key point detection on the second image sample to obtain a face-missing image and a face binary mask map of the second image sample; input the face area image sample corresponding to the first image sample into a face reference model to obtain facial feature information; perform posture extraction based on the second image sample and a character posture analysis model to obtain the posture feature information of the character in the second image sample; use the binary mask map, the face-missing image and the facial feature information as the main input side of a diffusion model, and use the posture feature information as the conditional input side of the diffusion model to determine noise loss, identity loss, reconstruction loss and perception loss; and train the diffusion model in stages through the noise loss, the identity loss, the reconstruction loss and the perception loss to obtain a trained diffusion model.

[0010] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect or the second aspect.

[0011] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect or the second aspect is implemented.

[0012] According to the scheme provided by the embodiment of the present application, the first image to be replaced and the second image to be replaced are obtained; the key point detection is performed on the second image to obtain the face missing image and the face binary mask map corresponding to the second image; the face area image corresponding to the first image is input into the face reference model to obtain the facial feature information of the person in the first image; the posture extraction is performed based on the second image and the character posture analysis model to obtain the posture feature information of the person in the second image; the face binary mask map, the face missing image and the face feature information are used as the main input side of the pre-trained diffusion model, and the posture feature information is used as the conditional input side of the diffusion model to obtain the target image after the face of the person in the second image is replaced. In this process, the face binary mask map and the facial feature information of the person in the first image can be combined to redraw the face of the first image in the face missing area corresponding to the face missing image corresponding to the second image, which can ensure that it can be naturally integrated with the face missing image, and retain the other parts of the second image except the face part, so that the face of the first image and the second image can be harmoniously integrated. The posture feature information is input as a conditional constraint into the conditional input side of the diffusion model to control the posture of the character in the generated target image, and finally a high-fidelity and natural target image can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0014] Figure 1 A general architecture diagram of a face-changing method provided in an embodiment of the present application;

[0015] Figure 2 is a flowchart of a face-changing method provided in an embodiment of the present application;

[0016] Figure 3 It is a flowchart of a model training method provided in an embodiment of the present application;

[0017] Figure 4 A general flow chart of a face-changing method provided in an embodiment of the present application;

[0018] Figure 5 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0020] In order to adapt the face-changing method provided in the embodiment of the present application, the embodiment of the present application provides a general architecture diagram of the face-changing method, that is, the face-changing method of the embodiment of the present application can be applied to the face-changing system provided in the embodiment of the present application. Figure 1As shown, the face-changing system includes an image acquisition module, an image processing module, and a target image generation module. The image acquisition module is used to acquire a video or multiple images at a small angle (a small angle represents an angle less than a preset angle value. When it is a small angle, it represents that the face of the person in the video is facing the audience more directly), and the acquired video or multiple images are divided into a first image (source image) and a second image (target image). In the training stage, the person in the second image can be the same as the person in the first image, and in the inference stage, the person in the second image can be different from the person in the first image. In the image processing module, the second image and the first image are processed respectively, and a facial binary mask image, a facial missing image, and a facial key point posture image are obtained from the second image. A facial area image is obtained from the first image, and the facial area image is input into the facial reference model to obtain facial feature information of the person in the first image. The facial key point posture image is input into the character posture analysis model to obtain the posture feature information of the character in the second image, and the facial binary mask image, the face missing image, the posture feature information and the facial feature information are input into the diffusion model to obtain the target image after the character in the second image is replaced.

[0021] Combine the following Figure 1 and Figure 2 The face-changing method provided by the embodiment of the present application is introduced. Figure 2 As shown, Figure 2 This is a schematic diagram of the face-changing method provided in the embodiment of the present application. Figure 1 The face-changing method provided in the embodiment of the present application can be executed by an electronic device, and the electronic device can be, for example, a computer, a server, etc.

[0022] like Figure 2 As shown, the face-changing method includes:

[0023] S101, obtaining a first image to be replaced with a face and a second image to be replaced with a face.

[0024] In the embodiment of the present application, the first image to be replaced with the face is the source image, and the second image to be replaced with the face is the target image. The face area in the first image needs to be replaced with the face area in the second image. The first image and the second image can be input by the user. The first image can be a single image or a single image frame, and the second image can be a single image, or multiple images, or multiple image frames constituting a video.

[0025] During inference, the person in the first image and the person in the second image can be the same, and during training, the person in the first image and the person in the second image can be different. The training is divided into two stages. For example, in the first stage of training, two images with different light, shadow, expression, and background from the same person’s video or image set are selected as the second image and the first image respectively; in the second stage of training, the image frames corresponding to the video clips of L consecutive frames from the same person’s video and a random frame of image with different light, shadow, expression, and background are selected as the second image and the first image respectively.

[0026] S102: Perform key point detection on the second image to obtain a face missing image and a face binary mask image corresponding to the second image.

[0027] In an embodiment of the present application, key point detection is used to identify specific key positions or feature points in an image or video. These key points are usually significant features on an object, such as joints, corners, edges or other unique parts. A face binary mask image refers to a binary image specifically used to represent a face area. In this image, the pixel value of the face part is usually set to 255 (white), indicating the foreground; and the pixel value of the non-face part is set to 0 (black), indicating the background. This mask image is mainly used to highlight and isolate the face area. The face missing image corresponding to the second image is an image that covers the face area in the second image.

[0028] In an embodiment of the present application, a facial key point detection algorithm may be used on the first image and the second image to detect the face area, obtain a face missing image, and simultaneously obtain a face binary mask image.

[0029] S103: Input the facial region image corresponding to the first image into a facial reference model to obtain facial feature information of the person in the first image.

[0030] Facial feature information refers to the various structures and details on the face, which can be divided into two categories: geometric features and texture features. Among them, geometric features mainly focus on the location of key points (landmarks) in the face, such as the location and shape of the eyes, nose, mouth, eyebrows, and facial contours. Geometric features can be used to describe the basic structural layout of the face, including but not limited to eyes, nose, mouth, and eyebrows. Texture features involve details of the skin on the face, such as wrinkles, spots, pores, hair, etc. Texture features include but are not limited to skin texture, hair details, color information, and surface details.

[0031] In an embodiment of the present application, the first image is subjected to face detection, face alignment and other operations to obtain the face area in the first image, and then the face part is cropped to obtain the corresponding face area image. The face reference model can provide a face reference, including a pre-trained face encoder, a linear layer, and a normalization layer. The specific process of inputting the obtained face area image into the face reference model is to first input the face area image into the face encoder, then pass through the linear layer, and then pass through the normalization layer to obtain the facial feature information of the person in the first image.

[0032] S104: Perform posture extraction based on the second image and the character posture analysis model to obtain posture feature information of the character in the second image.

[0033] Face keypoint pose imaging is a technique that identifies the location of specific parts of a person's face (such as eyes, nose, mouth, chin, etc.) in an image or video. These keypoints are often used in a variety of applications such as facial recognition, expression analysis, head pose estimation, and facial feature tracking, including the pose and facial features of the person in the second image.

[0034] In an embodiment of the present application, a facial key point posture image is obtained in the second image (when the second image is an image frame, a facial key point posture image sequence is obtained), and the second image is detected using an open source algorithm to obtain the facial key point posture image. The acquired facial key point posture image is input into a character posture analysis model to obtain the posture feature information of the character in the second image, and the posture feature information is used to control the character posture in the target image after the character in the second image is finally generated after the face is replaced.

[0035] In an embodiment of the present application, a character posture analysis model may include a posture prompt ControlNet model and an encoder of a variational autoencoder, obtain a facial key point posture image in a second image, input the facial key point posture image into a VAE encoder, obtain a latent space feature map of the posture, input the latent space feature map into a posture prompt ControlNet model, i.e., a copy of the coding block and the intermediate block of the diffusion model, and add the coding block and the intermediate block of the posture prompt ControlNet model to the decoding block and the intermediate block of the diffusion model through a zero convolution layer, and input the posture feature information into the diffusion model. Among them, the ControlNet model branch creates a copy of the coding block and the intermediate block of a trainable diffusion model U-Net, and adds an additional zero convolution layer. The coding block (Downblocks) and the intermediate block of ControlNet are added to the decoding block (Upblocks) and the intermediate block of the diffusion model. The output of the posture reference ControlNet is passed to the corresponding layer of the face-changing generation diffusion model through a residual connection.

[0036] S105, using the facial binary mask image, the face missing image and the facial feature information as the main input side of the pre-trained diffusion model, and using the posture feature information as the conditional input side of the diffusion model, to obtain a target image after the face of the person in the second image is replaced.

[0037] Pre-trained diffusion models allow the model to generate data based on additional input information (conditions), which can be a category label, a text description, another image, or any form of guiding signal, with the goal of guiding the generation process so that the generated results meet specific conditions or properties.

[0038] Stable Diffusion is a classic latent diffusion model (LDM). LDM uses a variational autoencoder. Given an input image, the encoder first maps the image to the latent space, so that diffusion is performed in the latent space. In the forward process of diffusion, Gaussian noise is gradually added to the latent variable in T iterations. The reverse diffusion process is to gradually recover from to, and finally pass through the decoder to get the final image.

[0039] In an embodiment of the present application, the pre-trained diffusion model is initialized using a pre-trained StableDiffusion redrawing model. Its function is to generate a target image after the face of the person in the second image is replaced. The pre-trained diffusion model of the present application can be regarded as a redrawing diffusion model, specifically referring to the face-missing area of ​​the second image, which regenerates the target image of the person after the face is replaced required by the user in the face-missing area of ​​the second image with the facial feature information in the first image, and can ensure that it can be naturally integrated with the redrawn background. When using the redrawing diffusion model to generate a face-changing image, the part of the target image except the face part is retained, and the face-missing picture to be obtained is used as the redrawing background. The face in the first image is redrawn on the face-missing part, so that the face of the first image is harmoniously integrated with the second image to obtain the target image.

[0040] Specifically, the face binary mask image, the face missing image and the face feature information are used as the main input side of the pre-trained diffusion model, and the posture feature information is used as the conditional input side of the pre-trained diffusion model, and as an additional input condition of the pre-trained diffusion model for guidance, to obtain a target image after the face of the person in the second image is replaced. The target image is an image obtained by redrawing the face area image of the first image in the face missing area corresponding to the face missing image.

[0041] It can be understood that in the embodiment of the present application, the first image to be replaced and the second image to be replaced are obtained; the key point detection is performed on the second image to obtain the face missing image and the face binary mask map corresponding to the second image; the face area image corresponding to the first image is input into the face reference model to obtain the facial feature information of the person in the first image; the posture extraction is performed based on the second image and the character posture analysis model to obtain the posture feature information of the person in the second image; the face binary mask map, the face missing image and the face feature information are used as the main input side of the pre-trained diffusion model, and the posture feature information is used as the conditional input side of the diffusion model to obtain the target image after the face of the person in the second image is replaced. In this process, combined with the face binary mask map and the facial feature information of the person in the first image, the face of the first image can be redrawn in the face missing area corresponding to the face missing image corresponding to the second image, and it can be ensured that it can be naturally integrated with the face missing image, and the other parts of the second image except the face part are retained, so that the face of the first image and the second image can be harmoniously integrated. The posture feature information is input as a conditional constraint into the conditional input side of the diffusion model to control the posture of the character in the generated target image, and finally a high-fidelity and natural target image can be obtained.

[0042] In some embodiments of the present application, S105 can be implemented through S1051 to S1056, which will be explained through the following steps.

[0043] S1051. Perform feature extraction based on the facial binary mask image, latent space noise, and the face missing image to obtain a first latent space feature map.

[0044] Feature extraction refers to the process of extracting information or patterns from raw data that are useful for solving specific problems. These extracted features usually represent the essence of the data better than the raw data, while reducing the redundancy and complexity of the data. When performing feature extraction, it mainly includes data preprocessing, feature selection, feature extraction methods, and feature conversion.

[0045] In some embodiments of the present application, the face binary mask map, latent space noise, face missing image can be subjected to preprocessing, feature selection, feature extraction method, feature conversion and other processes to extract features, and obtain a first latent space feature map. Among them, the first latent space feature map is a more comprehensive and more representative feature map of the face binary mask map, latent space noise, and face missing image.

[0046] S1052. Input the first latent space feature map into the self-attention layer to obtain a first query matrix.

[0047] The query matrix is ​​the starting point of the search or query in the attention layer. The input at each position will generate a corresponding query matrix, which is used to calculate the similarity with the key matrix of all positions to determine which information at the current position should focus on (or "pay attention to"). The key matrix is ​​the target to be queried or retrieved in the attention layer. The input at each position will also generate a corresponding key matrix, which is used to calculate the similarity with the query matrix to help the model understand the relationship between different positions in the sequence. The value matrix is ​​the vector that actually contains the information in the attention layer. The input at each position will also generate a corresponding value matrix. It contains specific information for each position in the sequence, which will be weighted and summed according to the attention weights calculated by the query matrix and the key matrix.

[0048] In some embodiments of the present application, the self-attention layer captures rich contextual information by calculating the correlation between elements in the sequence, thereby improving the model's ability to process sequence data. After the first latent space feature map is input into the self-attention layer, the self-attention layer first maps it to different vector spaces to obtain a first query matrix.

[0049] S1053. Determine a re-weighted second latent space feature map based on the first query matrix.

[0050] In some embodiments of the present application, after the first latent space feature map is input into the self-attention layer, the self-attention layer first maps it to a different vector space. After obtaining the first query matrix, the first key matrix and the first value matrix are also obtained. Based on the first query matrix, the first value matrix and the first key matrix, the attention score is calculated, the weight is normalized, and the weighted summation operation is performed to obtain the re-weighted second latent space feature map.

[0051] S1054. Input the first latent space feature map and the facial feature information into the image cross attention layer to obtain a first query matrix, a second value matrix and a second key matrix.

[0052] In some embodiments of the present application, the first latent space feature map and facial feature information are input into the image cross attention layer to obtain the second value matrix, the second key matrix and the first query matrix. Among them, the query matrix determined in the image cross attention layer uses the same query matrix as the self-attention layer of the diffusion model. That is, the first query matrix corresponding to the self-attention layer is the same as the first query matrix corresponding to the image cross attention layer.

[0053] Exemplarily, Q = ZW q , K′=c i W′ k , V′=c i W′ v , Z is the first latent space feature map, W qis the first preset weight matrix corresponding to the self-attention layer, c i is the facial feature information, W′ k and W′ v are the second preset weight matrix and the third preset weight matrix corresponding to the image cross attention layer. q , W′ k and W′ v is a trainable linear mapping layer, which is respectively a first preset weight matrix, a second preset weight matrix and a third preset weight matrix, that is, the first preset weight matrix, the second preset weight matrix and the third preset weight matrix are automatically learned through the training process of the diffusion model. During the training process, the goal of the diffusion model is to minimize a certain loss function (such as cross entropy loss), which usually involves adjusting the diffusion model parameters, including W q , W′ k and W′ v When the training is completed, the first preset weight matrix, the second preset weight matrix and the third preset weight matrix will be saved for use in model inference. t is the facial feature information, and Z is the first latent space feature vector.

[0054] S1055. Determine a re-weighted third latent space feature map based on the first query matrix, the second value matrix, and the second key matrix.

[0055] S1056. Determine a target image based on the second latent space feature map, the third latent space feature map, and the posture feature information.

[0056] Exemplarily, the third latent space feature map is calculated according to the following formula (1):

[0057]

[0058] In the above formula (1), Z″ is the feature map of the third latent space, d k is the dimension of the key matrix, Q is the first query matrix, V′ is the second value matrix, and K′ is the second key matrix.

[0059] After the third latent space feature map is determined according to the above formula (1), the target image is determined based on the second latent space feature map, the third latent space feature map and the posture feature information.

[0060] It can be understood that in some embodiments of the present application, feature extraction is performed based on a facial binary mask map, latent space noise, and a face missing image to obtain a first latent space feature map, thereby reducing the redundancy and complexity of the data. The first latent space feature map is input into the self-attention layer to obtain a first query matrix, and a re-weighted second latent space feature map is determined based on the first query matrix. The first latent space feature map and facial feature information are input into the image cross-attention layer to obtain a first query matrix, a second value matrix, and a second key matrix. By sharing the query matrix, computing resources can be shared between different attention layers, thereby reducing the overall amount of computation and realizing information sharing. Finally, the re-weighted third latent space feature map is determined based on the first query matrix, the second value matrix, and the second key matrix, and the target image is determined based on the second latent space feature map, the third latent space feature map, and the posture feature information, thereby improving the generation quality of the target image.

[0061] It can be understood that in some embodiments of the present application, S1056 can be implemented through S1056A to S1056C, which is explained by the following steps.

[0062] S1056A. Perform an AND operation on the second latent space feature map and the third latent space feature map to obtain a fourth latent space feature map.

[0063] In some embodiments of the present application, the second latent space feature map and the third latent space feature map are summed to fuse feature information from different sources or different levels, which can improve the expressiveness and performance of the diffusion model. When the second latent space feature map and the third latent space feature map are summed, the two sets of feature maps are required to be aligned in spatial dimensions (height, width) and number of channels (depth), that is, they must have the same size. When performing the sum operation, the second latent space feature map and the third latent space feature map can be added element by element to obtain a fourth latent space feature map.

[0064] For example, as shown in the following formula (2):

[0065]

[0066] In the above formula (2), Z′ is the second latent space feature map, Z″ is the third latent space feature map, It is the feature map of the fourth latent space.

[0067] S1056B, reshape the fourth latent space feature map to obtain a fifth latent space feature map.

[0068] In some embodiments of the present application, data reshaping allows changing the dimension of data without changing its underlying data, which is used to convert image data into sequence data for time series analysis.

[0069] Exemplarily, when the fourth latent space feature map is That is, the representation is a five-dimensional tensor z, whose shape is (b, L, h, W, c), and now it needs to be reshaped into a new shape ((b×h×w), L, c), and we get That is, the fifth latent space feature map. Among them, b represents the batch size, L represents the time step or sequence length, h and w represent the height and width of the image respectively, and c represents the number of channels. This reshaping is often used to convert image data into a format suitable for specific operations such as time series analysis.

[0070] S1056C, determine the target image based on the fifth latent space feature map, the temporal attention layer, the posture feature information and the preset decoder.

[0071] The preset decoder can be a decoder of a variational autoencoder (VAE). The role of the decoder in the VAE is to convert the latent vector sampled from the latent space back to the original data space, thereby generating reconstructed data that is as close as possible to the original input data while maintaining the key features of the data.

[0072] In some embodiments of the present application, the fifth latent space feature map can be input into the temporal attention layer, that is, a series of self-attention layers along the t dimension, to obtain the sixth latent space feature map. The sixth latent space feature map and the posture feature information can be added to obtain a processed latent space feature map, and the processed latent space feature map can be further processed by the subsequent layers of the diffusion model, and finally decoded by a preset decoder to obtain a target image.

[0073] It is understandable that in some embodiments of the present application, the second latent space feature map and the third latent space feature map are ANDed to obtain a fourth latent space feature map, and by fusing feature information from different sources or different levels, the expression ability and performance of the diffusion model can be improved. The fourth latent space feature map is data reshaped to obtain a fifth latent space feature map, so that the image data can be converted into a format suitable for a specific operation. The target image is determined based on the fifth latent space feature map, the temporal attention layer, the posture feature information, and the preset decoder, which improves the naturalness and fluency of the target image generation.

[0074] In some embodiments of the present application, S1051 can be implemented through S1051A to S1051C, which is explained by the following steps.

[0075] S105 1A. Input the face missing image into the pre-trained encoder to obtain a latent space feature map.

[0076] In some embodiments of the present application, the face-missing image can be input into the encoder of a pre-trained variational autoencoder to obtain a latent space feature map with, for example, 4 channels.

[0077] S105 1B, concatenating the latent space feature map, the latent space noise and the face binary mask map to obtain a concatenated latent space feature map.

[0078] S105 1C. Perform feature extraction based on the concatenated latent space feature map to obtain a first latent space feature map.

[0079] Channel concatenation allows feature vectors of different levels to be combined, so that both high-level abstract features and low-level detailed features can be utilized. By increasing the number of channels, the model can learn more feature representations, which helps improve the model's expressiveness and discrimination. The pose feature information has been added with noise, and by concatenating the pose feature information with latent space noise, the model's generalization ability, robustness, and data enhancement can be enhanced.

[0080] Exemplarily, the latent space feature vector of the 4-channel redrawn background, the posture feature information with the 4-channel latent space noise added, and the 1-channel binary mask map are spliced ​​along the channel to obtain a 9-channel input, that is, the spliced ​​latent space feature vector.

[0081] In some embodiments of the present application, after obtaining the spliced ​​latent space feature map, preprocessing, feature selection, feature extraction method and feature conversion processes can be performed to extract features to obtain a first latent space feature map.

[0082] It is understandable that in some embodiments of the present application, the face-missing image is input into a pre-trained encoder to obtain a latent space feature map, the latent space feature map, the latent space noise and the face binary mask map are spliced ​​to obtain a spliced ​​latent space feature map, and feature extraction is performed based on the spliced ​​latent space feature map to obtain a first latent space feature map. Through this process, high-level abstract features and low-level detail features are utilized. By increasing the number of channels, the model can learn more feature representations, which helps to improve the model's expressiveness and discrimination, and further improve the generation effect of the target image.

[0083] Combine the following Figure 1 and Figure 3 The model training method provided in the embodiment of the present application is introduced. Figure 3 As shown, Figure 3 A schematic diagram of a model training process provided in an embodiment of the present application.

[0084] S201, obtaining a first image sample to be replaced and a second image sample to be replaced.

[0085] In an embodiment of the present application, the training of the diffusion model can be divided into two stages, so when collecting samples, in the first stage of training, two images with different lighting, expressions, and backgrounds in the video or image set of the same person are selected as the second image samples and the first image samples respectively; in the second stage of training, image frames corresponding to video clips of L consecutive frames in the video of the same person and a random 1-frame image with different lighting, expressions, and backgrounds are selected as the second image samples and the first image samples respectively.

[0086] S202: Perform key point detection on the second image sample to obtain a face missing image and a face binary mask image of the second image sample.

[0087] S203: Input the facial region image sample corresponding to the first image sample into a facial reference model to obtain facial feature information.

[0088] S204: Perform posture extraction based on the second image sample and the character posture analysis model to obtain posture feature information of the character in the second image sample.

[0089] In the embodiment of the present application, S202 is similar to the implementation method of S102 regarding obtaining the face missing image and the face binary mask image, S203 is similar to the implementation method of S103 regarding obtaining the facial feature information, and S204 is similar to the implementation method of S104 regarding obtaining the posture feature information of the character, which will not be described in detail here.

[0090] S205. The binary mask image, the face missing image and the face feature information are used as the main input side of the diffusion model, and the posture feature information is used as the conditional input side of the diffusion model to determine the noise loss, identity loss, reconstruction loss and perception loss; and the diffusion model is trained in stages by the noise loss, identity loss, reconstruction loss and perception loss to obtain a trained diffusion model.

[0091] The loss function is a function used to quantify the difference between the model's prediction and the actual target value. The training goal of the model is to minimize this loss function by adjusting the model's parameters so that the model's prediction is as close to the actual output as possible.

[0092] In an embodiment of the present application, the posture feature information is used as the conditional input side of the diffusion model to be trained, and the binary mask map, the face missing image and the face feature information are used as the main input side of the diffusion model to be trained, and the predicted noise and the predicted image are obtained. The noise loss, identity loss, reconstruction loss and perception loss are determined based on the predicted noise and the predicted image. Finally, the diffusion model is trained by the noise loss, identity loss, reconstruction loss and perception loss until the training conditions are met, and the trained diffusion model is obtained.

[0093] When the diffusion model is trained in stages, the diffusion model is initialized by redrawing the pre-trained weights of StableDiffusion. The training can be divided into two stages. In the first training stage, the training generates a single face-changing image, so that the target image can be generated based on a first image sample and a second image sample. In the first stage of training, the time attention layer of the diffusion model does not participate in the training. In the second training stage, the time attention layer of the diffusion model is trained. In this stage, the image frame sequence corresponding to the first image and the video clip of L consecutive frames is input to generate the face-changing video. Therefore, in the two training stages, the noise loss includes the first noise loss corresponding to the image sample and the second noise loss corresponding to the image frame sample; the identity loss includes the first identity loss corresponding to the image sample and the second identity loss corresponding to the image frame sample; the reconstruction loss includes the first reconstruction loss corresponding to the image sample and the second reconstruction loss corresponding to the image frame sample; the perceptual loss includes the first perceptual loss corresponding to the image sample and the second perceptual loss corresponding to the image frame sample.

[0094] When training the diffusion model, when the human posture analysis model is a ControlNet model, the ControlNet model can be trained by noise loss, identity loss, reconstruction loss and perception loss. The ControlNet initialization uses the pre-trained weights of the ControlNet with posture reference.

[0095] It can be understood that in the embodiments of the present application, based on the face missing image, face binary mask map, facial feature information and character posture feature information obtained from the first image sample to be replaced and the second image sample to be replaced, the posture feature information is then input as a conditional constraint into the conditional input side of the diffusion model, and the binary mask map, face missing image and facial feature information are used as the main input side of the diffusion model to be trained to determine the noise loss, identity loss, reconstruction loss and perceptual loss, and training with multiple loss values ​​can improve the performance and generalization ability of the model.

[0096] In some embodiments of the present application, S205 can be implemented through S205 1 to S2054, which will be specifically described through the following steps.

[0097] S205 1. Perform a sum operation on the first noise loss, the first identity loss, the first reconstruction loss, and the first perceptual loss to obtain a first total loss.

[0098] In some embodiments of the present application, in the first stage, training is performed to generate a single face-swapped picture, so that a target image can be generated based on a first image and a second image.

[0099] Exemplarily, the first noise loss may be calculated by the following formula (3):

[0100]

[0101] In the above formula (3), when the character posture analysis model is the ControlNet model, T θ ,∈ θ They are the posture reference ControlNet model and the face-changing generation diffusion model, which are the trainable parts of the model. bg is the latent space feature map, z p is the latent space feature map corresponding to the character posture, c i is the facial feature information, t is the denoising time step, z t It is the latent space feature map at time t.

[0102] Exemplarily, the first identity loss may be calculated using the following formula (4):

[0103]

[0104] In the above formula (4), Arc is the pre-trained ArcFace encoder, is the first identity loss, cos is cos similarity, I gen Prediction image, I source is the first image.

[0105] Exemplarily, the first reconstruction loss may be calculated using the following formula (5):

[0106]

[0107] In the above formula (5), is the first reconstruction loss, I target For the second image.

[0108] Exemplarily, the first perceptual loss may be calculated using the following formula (6):

[0109]

[0110] In the above formula (6), is the first perceptual loss, f vgg is the pre-trained VGG-19 network, and l is the l-th layer output of the VGG-19 network.

[0111] Exemplarily, the first total loss can be calculated by the following formula (7):

[0112]

[0113] In the above formula (7), is the first total loss, λ i , r and λ p are the corresponding hyper parameters respectively.

[0114] S2052. Perform a sum operation on the second noise loss, the second reconstruction loss, the second identity loss, and the second perception loss to obtain a second total loss.

[0115] In some embodiments of the present application, the second stage inputs the first image and the image frame sequence corresponding to the video clip of L consecutive frames to generate a face-swapped video. The second noise loss, the second reconstruction loss, the second identity loss, and the second perception loss can be calculated according to formulas (3) to (6), respectively, and the calculated second noise loss, second reconstruction loss, second identity loss, and second perception loss are summed to obtain a second total loss.

[0116] For example, taking the second identity loss as an example, the first image and at least one image frame are used to generate a face in the first image and replace it with at least one image frame to obtain at least one predicted image, and the at least one predicted image is averaged. Finally, the average value and the first image are brought into formula (4) to calculate and obtain the second identity loss. The second noise loss, the second reconstruction loss and the second perceptual loss are similarly obtained, and are all obtained based on the average value of at least one image frame.

[0117] S2053. Train the first preset layer of the diffusion model through the first total loss to obtain the diffusion model after the first stage of training; the first preset layer does not include the temporal attention layer.

[0118] S2054. When the weight of the first preset layer is fixed, the temporal attention layer of the diffusion model trained in the first stage is trained by the second total loss to obtain the trained diffusion model.

[0119] In some embodiments of the present application, in the first training stage, the diffusion model temporarily removes the time attention layer, and the other layers (the first preset layer) are all involved in the training. The first preset layer of the diffusion model is trained using the first loss to obtain the diffusion model after the first stage of training. After obtaining the diffusion model after the first stage of training, training is performed again. This time, only the time attention layer of the diffusion model is trained, and the weights of other parts are fixed.

[0120] Among them, when training the diffusion model, when the character posture analysis model is a ControlNet model, at the same time, in the first training stage, the ControlNet model temporarily removes the time attention layer, and the other layers (the first preset layer) are all involved in the training. The first preset layer of the ControlNet model is trained using the first loss to obtain the diffusion model after the first stage of training. After obtaining the ControlNet model after the first stage of training, training is performed again. This time, only the time attention layer of the ControlNet model is trained, and the weights of other parts are fixed. Until the end of the training, the trained diffusion model and ControlNet model are obtained.

[0121] It can be understood that in some embodiments of the present application, the first noise loss, the first identity loss, the first reconstruction loss and the first perceptual loss are summed to obtain a first total loss, and the second noise loss, the second reconstruction loss, the second identity loss and the second perceptual loss are summed to obtain a second total loss. By adding the losses, different loss functions are defined for each type of data, so that the model can effectively integrate multiple information sources and improve the performance and generalization ability of the model. The first preset layer of the diffusion model is trained by the first total loss to obtain the diffusion model after the first stage of training; the first preset layer does not include the time attention layer. When the weight of the first preset layer is fixed, the time attention layer of the diffusion model after the first stage of training is trained by the second total loss to obtain the trained diffusion model. On the one hand, staged training can decompose complex training tasks into multiple relatively simple subtasks, and each stage focuses on solving specific problems or improving a certain aspect of the model. This helps the model gradually approach the final goal and improves the training efficiency and effect. On the other hand, starting training from simple tasks and then gradually introducing more complex tasks can enable the model to gradually establish an understanding and processing ability for complex problems during the training process. Each training stage may use data from different sources, different formats or different fields. This data diversity helps the model learn richer feature representations and improves the model's generalization ability.

[0122] In the embodiments of the present application, in combination with the above steps, Figure 4 As shown, Figure 4 This is a general flow chart of a face-changing method provided in an embodiment of the present application. Figure 4In the method, a second image and a first image are obtained from a small-angle video or multiple images, and the second image and the first image are preprocessed to obtain a face binary mask image, a face missing image, a face key point posture image, and a face region cropped from the first image. The face missing image is input into a VAE encoder for processing to obtain a latent space feature map, the latent space feature map, the face binary mask image, and the noise are spliced ​​to obtain a spliced ​​latent space feature map, and the spliced ​​latent space feature map is input into a diffusion model. The face key point posture map is input into a VAE encoder to obtain a latent space feature map, and the latent space feature map is input into a character posture analysis model to obtain the posture feature information of the character in the second image, and the posture feature information is input into the diffusion model. The facial region corresponding to the first image is processed by the facial reference model (which may include a face encoder and a linear layer, and may also include a normalization layer) to obtain facial feature information, and the facial feature information is input into the image cross attention layer in the diffusion model (the image cross attention layer and the self-attention layer in the diffusion model are decoupled), the diffusion model processes the input data to obtain a processed latent space feature map, and the processed latent space feature map is input into the VAE decoder to obtain the target image. Among them, serial number 1 represents the image cross attention layer, and serial number 2 represents the self-attention layer.

[0123] Reference Figure 5 , shows a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0124] like Figure 5 As shown, the electronic device may include: a processor (processor) 502, a communication interface (Communications Interface 504, a memory (memory) 506, and a communication bus 508.

[0125] in:

[0126] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0127] The communication interface 504 is used to communicate with other electronic devices or servers.

[0128] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above method embodiment.

[0129] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0130] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0131] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0132] The program 510 may be specifically used to enable the processor 502 to execute operations corresponding to the methods described in the above method embodiments.

[0133] The specific implementation of each step in program 510 can refer to the corresponding description of the corresponding steps and units in the above method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0134] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0135] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0136] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0137] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A face-changing method, characterized in that: include: Acquire a first image to be replaced with a face and a second image to be replaced with a face; Performing key point detection on the second image to obtain a face missing image and a face binary mask image corresponding to the second image; Inputting the facial region image corresponding to the first image into a facial reference model to obtain facial feature information of the person in the first image; Performing posture extraction based on the second image and the character posture analysis model to obtain posture feature information of the character in the second image; The facial binary mask image, the face missing image and the facial feature information are used as the main input side of the pre-trained diffusion model, and the posture feature information is used as the conditional input side of the diffusion model to obtain a target image after the face of the person in the second image is replaced.

2. The method according to claim 1, characterized in that The diffusion model includes a self-attention layer and an image cross-attention layer; The method uses the face binary mask image, the face missing image and the face feature information as the main input side of the pre-trained diffusion model, and uses the posture feature information as the conditional input side of the diffusion model to obtain a target image after the face of the person in the second image is replaced, including: Performing feature extraction based on the facial binary mask image, latent space noise, and the facial missing image to obtain a first latent space feature map; Inputting the first latent space feature map into the self-attention layer to obtain a first query matrix; Determine a re-weighted second latent space feature map based on the first query matrix; Inputting the first latent space feature map and the facial feature information into the image cross attention layer to obtain the first query matrix, the second value matrix and the second key matrix; Determine a reweighted third latent space feature map based on the first query matrix, the second value matrix, and the second key matrix; The target image is determined based on the second latent space feature map, the third latent space feature map and the posture feature information.

3. The method according to claim 2, characterized in that The second image is an image frame in a video; the diffusion model further includes a temporal attention layer; The determining the target image based on the second latent space feature map, the third latent space feature map and the posture feature information includes: Performing an AND operation on the second latent space feature map and the third latent space feature map to obtain a fourth latent space feature map; Reshaping the fourth latent space feature map to obtain a fifth latent space feature map; The target image is determined based on the fifth latent space feature map, the temporal attention layer, the posture feature information and a preset decoder.

4. The method according to claim 2, characterized in that: The step of extracting features based on the facial binary mask image, latent space noise, and the facial missing image to obtain a first latent space feature map includes: Inputting the face missing image into a pre-trained encoder to obtain a latent space feature map; splicing the latent space feature map, the latent space noise and the face binary mask map to obtain a spliced ​​latent space feature map; Feature extraction is performed based on the spliced ​​latent space feature map to obtain the first latent space feature map.

5. The method according to claim 2, characterized in that: The first query matrix is ​​determined according to the first latent space feature map and a first preset matrix; the second value matrix is ​​determined according to the facial feature information and the second preset weight matrix; and the second key matrix is ​​determined according to the facial feature information and a third preset weight matrix.

6. A model training method, characterized in that: include: Acquire a first image sample to be replaced with a face and a second image sample to be replaced with a face; Performing key point detection on the second image sample to obtain a face missing image and a face binary mask image of the second image sample; Inputting the facial region image sample corresponding to the first image sample into a facial reference model to obtain facial feature information; Performing posture extraction based on the second image sample and the character posture analysis model to obtain posture feature information of the character in the second image sample; The binary mask image, the face missing image and the face feature information are used as the main input side of the diffusion model, and the posture feature information is used as the conditional input side of the diffusion model to determine the noise loss, identity loss, reconstruction loss and perception loss; and the diffusion model is trained in stages by the noise loss, the identity loss, the reconstruction loss and the perception loss to obtain a trained diffusion model.

7. The method according to claim 6, characterized in that The noise loss includes a first noise loss corresponding to the image sample and a second noise loss corresponding to the image frame sample; the identity loss includes a first identity loss corresponding to the image sample and a second identity loss corresponding to the image frame sample; the reconstruction loss includes a first reconstruction loss corresponding to the image sample and a second reconstruction loss corresponding to the image frame sample; the perceptual loss includes a first perceptual loss corresponding to the image sample and a second perceptual loss corresponding to the image frame sample.

8. The method according to claim 7, characterized in that The step of training the diffusion model in stages by using the noise loss, the identity loss, the reconstruction loss and the perceptual loss to obtain the trained diffusion model comprises: Performing a sum operation on the first noise loss, the first identity loss, the first reconstruction loss, and the first perceptual loss to obtain a first total loss; Performing a sum operation on the second noise loss, the second reconstruction loss, the second identity loss, and the second perceptual loss to obtain a second total loss; The first preset layer of the diffusion model is trained by the first total loss to obtain a diffusion model after the first stage of training; the first preset layer does not include a temporal attention layer; When the weight of the first preset layer is fixed, the temporal attention layer of the diffusion model trained in the first stage is trained by the second total loss to obtain the trained diffusion model.

9. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method as described in any one of claims 1-5 or any one of claims 6-8.

10. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 5 or any one of claims 6 to 8 is implemented.

Images