Deepfake face swapping image generation method, system, device and storage medium

Abstract

The application belongs to the field of image processing, and discloses a deep fake face swapping image generation method, system, device and storage medium, which comprises the following steps: acquiring a source face image and a target face image; calling a preset face encoder to obtain a latent vector of the source face image in a style space, a latent vector of the target face image in the style space, a latent vector of the target face image in a pose space and a face region mask image of the target face image; calling a preset latent vector selector to obtain a preliminary exchange latent vector; calling a preset face attribute stabilizer to obtain an exchange latent vector; calling a preset face generator to obtain a preliminary face swapping image; and finally calling a preset face fusioner to fuse the preliminary face swapping image, the target face image and the face region mask image of the target face image to obtain a face swapping image. The image quality of the face swapping image is effectively improved, and the gap with a real image is narrowed.

Description

Technical Field

[0001] This invention belongs to the field of image processing and relates to a method, system, device and storage medium for generating deepfake face-swapping images. Background Technology

[0002] With the development of deep learning technology, the technical difficulty of generating deepfake face-swapping images is constantly decreasing, and the quality of the generated deepfake face-swapping images and videos is also continuously improving. Face swapping is the process of seamlessly transferring the identity information of a source face image to a target face image while maintaining the target face image's facial attributes, such as expression, pose, skin tone, and lighting, to produce a highly realistic result. Face swapping techniques can be divided into target-specific face swapping and non-target-specific face swapping. The former uses a large amount of image and video data to train a model for a specific pair of people, but its versatility is poor. The latter, after training, can be applied to face swapping between different people.

[0003] Currently, the common method for face swapping involves mapping the target and source face images into a latent space to obtain corresponding latent vectors, partially swapping them, and then inputting the swapped latent vectors into a pre-trained StyleGAN generator to achieve face swapping for non-target objects. While this technique can achieve face swapping, it doesn't consider the selection of latent vectors, leading to issues such as incorrect swapping of some facial attributes. Furthermore, the images generated by the StyleGAN generator often suffer from background and hair blurring, resulting in low-quality face-swapped images that differ significantly from realistic images. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing deepfake face-swapping image generation methods, which produce face-swapping images of low quality and significant differences from real images, and to provide a deepfake face-swapping image generation method, system, device, and storage medium.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] In a first aspect, the present invention provides a method for generating deepfake face-swapping images, comprising: acquiring a source face image and a target face image; invoking a preset face encoder based on the source face image to obtain latent vectors in the style space of the source face image; invoking a preset face encoder based on the target face image to obtain latent vectors in the style space, latent vectors in the pose space, and a face region mask image of the target face image; invoking a preset latent vector selector based on the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image to obtain preliminary swapped latent vectors; invoking a preset face attribute stabilizer based on the preliminary swapped latent vectors and the latent vectors in the style space of the target face image to obtain swapped latent vectors; invoking a preset face generator based on the swapped latent vectors and the latent vectors in the pose space of the target face image to obtain a preliminary face-swapping image; and invoking a preset face fusion device based on the preliminary face-swapping image, the target face image, and the face region mask image of the target face image to perform fusion, thereby obtaining the face-swapping image.

[0007] Optionally, the face encoder includes a face feature extractor, a latent space mapper, and a face region perceptron. When the input to the face feature extractor is a source face image, the multi-scale output of the face feature extractor is used as the input to the latent space mapper through a feature pyramid structure, and the latent space mapper obtains the latent vector in the style space of the source face image. When the input to the face feature extractor is a target face image, the multi-scale output of the face feature extractor is used as the input to the latent space mapper through a feature pyramid structure, and the latent space mapper obtains the latent vector in the style space and the latent vector in the pose space of the target face image. Additionally, the input to the face region perceptron is obtained through a U-Net structure using a concatenated feature map approach, and the face region perceptron obtains the face region mask image of the target face image.

[0008] Optionally, the face feature extractor is a residual neural network ResNet50.

[0009] Optionally, the latent vector selector includes a global pooling layer, a first fully connected layer, a ReLU layer, and a second fully connected layer connected in sequence. The step of calling a preset latent vector selector to obtain preliminary exchange latent vectors based on the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image includes: inputting the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image into the global pooling layer of the latent vector selector for feature compression, encoding the entire spatial features of each latent vector to obtain global descriptive features for each latent vector; inputting the global descriptive features of each latent vector into the first fully connected layer and then activating it with a ReLU layer, then inputting the output into the second fully connected layer to restore the original dimensions, and activating it with a sigmoid function to obtain the score of each latent vector; selecting the latent vector with the highest score in the style space of the source face image as the first latent vector, selecting the latent vector with the highest score in the style space of the target face image as the second latent vector, and combining the first latent vector and the second latent vector as the preliminary exchange latent vector.

[0010] Optionally, the face attribute stabilizer is based on the Transformer model architecture and is trained using preset face attribute stabilizer training samples. In this process, the latent vectors are initially swapped as the Q matrix of the Transformer model architecture, and the latent vectors in the style space of the target face are simultaneously used as the K matrix and V matrix of the Transformer model architecture.

[0011] Optionally, the face generator is based on the StyleGAN2 model architecture and is trained using preset face generator training samples; wherein, the latent vector in the pose space of the target face image is used as the basic input of the StyleGAN2 model architecture, the swapped latent vector is used as the style control input of the StyleGAN2 model architecture, and the noise input of the StyleGAN2 model architecture is removed.

[0012] Optionally, the step of calling a preset face fusion processor for fusion includes: blurring the face region mask image of the target face image, decreasing the pixel values ​​of the outer ring of the blurred face region mask image from the inside out according to a preset adjustment threshold to obtain a softened face region mask image; performing a dot product between the softened face region mask image and the initial face swap image and extracting the face image; and fusing the face image and the target face image based on the Boson fusion method.

[0013] In a second aspect, the present invention provides a deepfake face-swapping image generation system, comprising: an image acquisition module for acquiring a source face image and a target face image; a face encoding module for invoking a preset face encoder based on the source face image to obtain latent vectors in the style space of the source face image; and invoking a preset face encoder based on the target face image to obtain latent vectors in the style space, latent vectors in the pose space, and a face region mask image of the target face image; a latent vector exchange module for invoking a preset latent vector selector to obtain preliminary exchange latent vectors based on the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image; and invoking a preset face attribute stabilizer to obtain exchange latent vectors based on the preliminary exchange latent vectors and the latent vectors in the style space of the target face image; and a fusion module for invoking a preset face generator to obtain a preliminary face-swapping image based on the exchange latent vectors and the latent vectors in the pose space of the target face image, and for invoking a preset face fusion unit to perform fusion based on the preliminary face-swapping image, the target face image, and the face region mask image of the target face image to obtain the face-swapping image.

[0014] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described deepfake face-swapping image generation method.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described deepfake face-swapping image generation method.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] This invention discloses a deepfake face-swapping image generation method. It maps the source and target face images into different latent spaces using a face encoder, ensuring the target face image's pose is maintained during subsequent face-swapping image generation using a face generator. Simultaneously, a latent vector selector adaptively selects latent vectors to be swapped to control the swapped content. Furthermore, a face attribute stabilizer stabilizes certain face attributes that need to remain unchanged, optimizing the swapped latent vectors. Finally, a preset face fusion processor is used to fuse the initial face-swapping image, the target face image, and the target face image's face region mask image. This addresses issues such as background and hairstyle blurring present in the generated face-swapping image. The fusion of the target face image's face region mask image with the target face image ensures the stability of non-facial regions, resulting in a more realistic face-swapping image with a smaller gap from the real image and high image quality. Attached Figure Description

[0018] Figure 1 This is a flowchart of a deepfake face-swapping image generation method according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram illustrating the principle of the deepfake face-swapping image generation method according to an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the face encoder principle according to an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] The present invention will now be described in further detail with reference to the accompanying drawings:

[0024] See Figure 1 and 2 In one embodiment of the present invention, a method for generating deepfake face-swapping images is provided, comprising the following steps:

[0025] S1: Obtain the source face image and the target face image.

[0026] S2: Based on the source face image, call the preset face encoder to obtain the latent vector in the style space of the source face image; based on the target face image, call the preset face encoder to obtain the latent vector in the style space, the latent vector in the pose space, and the face region mask image of the target face image.

[0027] S3: Based on the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image, call the preset latent vector selector to obtain the initial swapped latent vectors; based on the initial swapped latent vectors and the latent vectors in the style space of the target face image, call the preset face attribute stabilizer to obtain the swapped latent vectors.

[0028] S4: Based on the latent vectors of the swapped latent vectors and the latent vectors in the pose space of the target face image, call the preset face generator to obtain the preliminary face-swapping image, and call the preset face fusion processor to fuse the preliminary face-swapping image, the target face image, and the face region mask image of the target face image to obtain the face-swapping image.

[0029] This invention discloses a deepfake face-swapping image generation method. It maps the source and target face images into different latent spaces using a face encoder, ensuring the target face image's pose is maintained during subsequent face-swapping image generation using a face generator. Simultaneously, a latent vector selector adaptively selects latent vectors to be swapped to control the swapped content. Furthermore, a face attribute stabilizer stabilizes certain face attributes that need to remain unchanged, optimizing the swapped latent vectors. Finally, a preset face fusion processor is used to fuse the initial face-swapping image, the target face image, and the target face image's face region mask image. This addresses issues such as background and hairstyle blurring present in the generated face-swapping image. The fusion of the target face image's face region mask image with the target face image ensures the stability of non-facial regions, resulting in a more realistic face-swapping image with a smaller gap from the real image and high image quality.

[0030] In one possible implementation, the face encoder includes a face feature extractor, a latent space mapper, and a face region perceptron.

[0031] When the input to the face feature extractor is a source face image, the multi-scale output of the face feature extractor is used as the input to the latent space mapper through a feature pyramid structure. The latent space mapper then obtains the latent vector in the style space of the source face image. When the input to the face feature extractor is a target face image, the multi-scale output of the face feature extractor is used as the input to the latent space mapper through a feature pyramid structure. The latent space mapper then obtains the latent vector in the style space and the latent vector in the pose space of the target face image. Additionally, the input to the face region perceptron is obtained through a U-Net structure using concatenated feature maps. The face region perceptron then obtains the face region mask image of the target face image.

[0032] Specifically, the face encoder includes a face feature extractor composed of a network structure with several convolutional layers, pooling layers, normalization layers, and ReLU layers; a latent space mapper composed of a network structure with several convolutional layers, pooling layers, normalization layers, upsampling layers, and fully connected layers; and a face region perceptron composed of a network structure with several convolutional layers, pooling layers, normalization layers, upsampling layers, and fully connected layers. The face encoder is a multi-task model that performs face-to-latent space mapping and face region perception. The face feature extractor uses ResNet50 to obtain face features. The face latent space mapping task uses an FPN structure for multi-scale output, with additional large-scale feature map output at the bottom layer. The top-level minimum-scale feature map is mapped to the pose space through a mapping layer network, and the remaining scale feature maps are mapped to the style space through a mapping layer network. The face region perceptron uses a U-Net structure to output a face region mask image.

[0033] Before inputting the source and target face images into the face encoder, the face encoder can be trained using pre-constructed training samples, including face image samples and face region mask image samples. During training, the face image samples are input into the face encoder to obtain latent vectors in pose space, latent vectors in style space, and face region mask images. The latent vectors in pose space are reconstructed as the basic input to the face generator, and the latent vectors in style space are used as the style input to the face generator, resulting in a reconstructed face image. The reconstructed face image and the original face image are compared to calculate identity information loss, pixel-level mean square error loss, pose loss, latent vector reconstruction loss, and facial landmark loss. The face region mask image and its labels are compared to calculate a binary classification cross-entropy loss. These losses are then weighted and summed as the loss function of the face encoder. This loss function is used to calculate the deviation between the two images. During model training, the learnable parameters of the face encoder are iteratively updated using a gradient descent algorithm to optimize the parameters. When the training reaches the maximum number of iterations, the preset face encoder is obtained.

[0034] See Figure 3In this embodiment, the face encoder includes a face feature extractor, based on a ResNet50 network and composed of five residual blocks. The face encoder includes a latent space mapper, consisting of multiple convolutional layers, multiple fully connected layers, and multiple batch normalization (BN) layers. A pyramid structure is used for multi-scale output, with the feature map scale increasing from top to bottom. The feature map from the previous layer is convolved with the feature map from the next layer to form the output feature map. The mapping layer network consists of multiple convolutional layers, multiple fully connected layers, and multiple BN layers. Multiple layers of output feature maps are input into the mapping layer network; each mapping layer produces a latent vector, and multiple mapping layers complete the mapping of the latent space. The face encoder includes a face region perceptron, consisting of multiple convolutional layers, multiple fully connected layers, and multiple BN layers. The small-scale feature map from the upper layer is passed through the upper convolutional layer network and the BN layer, and then concatenated with the feature map of the current layer in one channel, finally outputting a face region mask image with one channel.

[0035] In one possible implementation, the latent vector selector includes a global pooling layer, a first fully connected layer, a ReLU layer, and a second fully connected layer connected in sequence. The step of calling a preset latent vector selector to obtain preliminary exchange latent vectors based on the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image includes: inputting the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image into the global pooling layer of the latent vector selector for feature compression, encoding the entire spatial features of each latent vector to obtain global descriptive features for each latent vector; inputting the global descriptive features of each latent vector into the first fully connected layer and then activating it with a ReLU layer, then inputting the output into the second fully connected layer to restore the original dimensions, and activating it with a sigmoid function to obtain the score of each latent vector; selecting the latent vector with the highest score in the style space of the source face image as the first latent vector, selecting the latent vector with the highest score in the style space of the target face image as the second latent vector, and combining the first latent vector and the second latent vector as the preliminary exchange latent vector.

[0036] The latent vector selector consists of a global pooling layer, two fully connected layers, and a ReLU layer. These layers perform compression, feature extraction, dimension restoration, and activation of the input latent vectors, respectively, yielding a score for each latent vector. During forward propagation, the latent vector selector uses a max function to select latent vectors with high scores, and during gradient backpropagation, it uses softmax to approximate the gradient.

[0037] Specifically, before inputting the latent vectors of the source and target face images in style space into the latent vector selector and face attribute stabilizer, the latent vector selector and face attribute stabilizer can be trained using pre-constructed training samples. These training samples include source and target face image samples. During training, the source and target face image samples are input into a pre-trained face encoder to obtain the latent vectors of the target face image in pose space, the latent vectors of the source and target face images in style space, and the face region mask image of the target face image. The latent vectors of the source and target face images in style space are then input into the latent vector selector to obtain preliminary swapped latent vectors. These preliminary swapped latent vectors are then input into the face attribute stabilizer to obtain swapped latent vectors. The latent vectors of the target face image in pose space are reconstructed as the basic input to the face generator. The swapped latent vectors are used as the style input to the face generator to obtain a preliminary face-swapping image. This preliminary face-swapping image is then input into the face fusion unit to obtain the final face-swapping image. The loss calculations include identity information loss between the face-swapped image and the source face image, pixel-level mean squared error loss between the face-swapped image and the source face image, pixel-level mean squared error loss between the face-swapped image and the target face image, pose loss between the face-swapped image and the target face image, and facial landmark loss between the face-swapped image and the target face image. These losses are then weighted and summed as the loss function for the latent vector selector and the face attribute stabilizer. This loss function is used to calculate the deviation between the two. During model training, the learnable parameters of the face encoder and the latent vector selector are iteratively updated using a gradient descent algorithm to optimize them. When the training reaches the maximum number of iterations, the preset face encoder and latent vector selector are obtained.

[0038] In one possible implementation, the face attribute stabilizer is based on the Transformer model architecture and is trained using preset face attribute stabilizer training samples. In this process, the latent vectors are initially swapped as the Q matrix of the Transformer model architecture, and the latent vectors in the style space of the target face are simultaneously used as the K and V matrices of the Transformer model architecture.

[0039] In one possible implementation, the face generator is based on the StyleGAN2 model architecture and trained using preset face generator training samples. Specifically, the latent vectors in the pose space of the target face image are used as the basic input to the StyleGAN2 model architecture, the swapped latent vectors are used as the style control input, and the noise input to the StyleGAN2 model architecture is removed. In this implementation, the face generator generates a preliminary face-swapped image with a resolution of 1024x1024.

[0040] In one possible implementation, the step of calling a preset face fusion processor includes: blurring the face region mask image of the target face image, decreasing the pixel values ​​of the outer ring of the blurred face region mask image from the inside out according to a preset adjustment threshold to obtain a softened face region mask image; multiplying the softened face region mask image with the initial face-swapping image and extracting the face image; and fusing the face image with the target face image based on the Boson fusion method.

[0041] In summary, the deepfake face-swapping image generation method of this invention involves inputting a source face image into a pre-trained face encoder to obtain the latent vector of the source face image in style space, inputting a target face image into the pre-trained face encoder to obtain the latent vector of the target face image in pose space and style space, and a face region mask image of the target face image; inputting the latent vectors of the source face image and the target face image in style space into a latent vector selector to obtain a preliminary swapped latent vector; inputting the preliminary swapped latent vector and the latent vector of the target face image in style space into a face attribute stabilizer to obtain a swapped vector; inputting the swapped vector and the latent vector of the target face image in pose space into a face generator to obtain a preliminary face-swapping image; and inputting the preliminary face-swapping image, the target face image, and the face region mask image of the target face into a face fusion unit to obtain the face-swapping image. This face-swapping image includes the source face image, and the source face image possesses the attribute features of the target face image. Applying the deepfake face-swapping image generation method of this invention can significantly improve the quality of the generated face-swapping image.

[0042] Meanwhile, the face swapper proposed in this invention outputs larger-scale feature maps, enhancing the consideration of facial details and resulting in face-swapped images containing more detail. Multi-task training makes the facial feature extraction network in the face swapper more stable, avoiding overfitting. Furthermore, no additional auxiliary networks need to be trained during training, making the entire training method more elegant and stable.

[0043] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0044] In another embodiment of the present invention, a deepfake face-swapping image generation system is provided, which can be used to implement the above-mentioned deepfake face-swapping image generation method. Specifically, the deepfake face-swapping image generation system includes an image acquisition module, a face encoding module, a latent vector exchange module, and a fusion module. The system comprises the following modules: an image acquisition module for acquiring source and target face images; a face encoding module for using a preset face encoder to obtain latent vectors in style space of the source face image; and a preset face encoder for obtaining latent vectors in style space, pose space, and a face region mask image of the target face image; a latent vector swapping module for using a preset latent vector selector to obtain preliminary swapped latent vectors based on the latent vectors in style space of the source and target face images; and a preset face attribute stabilizer for obtaining swapped latent vectors based on the preliminary swapped latent vectors and the latent vectors in style space of the target face image; and a fusion module for using a preset face generator to obtain a preliminary face-swapped image based on the swapped latent vectors and the latent vectors in pose space of the target face image, and a preset face fusion module for fusing the preliminary face-swapped image, the target face image, and the face region mask image of the target face image to obtain the final face-swapped image.

[0045] In one possible implementation, the face encoder includes a face feature extractor, a latent space mapper, and a face region perceptron. When the input to the face feature extractor is a source face image, the multi-scale output of the face feature extractor is used as the input to the latent space mapper through a feature pyramid structure, and the latent space mapper obtains the latent vector in the style space of the source face image. When the input to the face feature extractor is a target face image, the multi-scale output of the face feature extractor is used as the input to the latent space mapper through a feature pyramid structure, and the latent space mapper obtains the latent vector in the style space and the latent vector in the pose space of the target face image. Furthermore, the input to the face region perceptron is obtained through a U-Net structure using a concatenated feature map approach, and the face region perceptron obtains the face region mask image of the target face image.

[0046] In one possible implementation, the face feature extractor is a residual neural network ResNet50.

[0047] In one possible implementation, the latent vector selector includes a global pooling layer, a first fully connected layer, a ReLU layer, and a second fully connected layer connected in sequence. The step of calling a preset latent vector selector to obtain preliminary exchange latent vectors based on the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image includes: inputting the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image into the global pooling layer of the latent vector selector for feature compression, encoding the entire spatial features of each latent vector to obtain global descriptive features for each latent vector; inputting the global descriptive features of each latent vector into the first fully connected layer and then activating it with a ReLU layer, then inputting the output into the second fully connected layer to restore the original dimensions, and activating it with a sigmoid function to obtain the score of each latent vector; selecting the latent vector with the highest score in the style space of the source face image as the first latent vector, selecting the latent vector with the highest score in the style space of the target face image as the second latent vector, and combining the first latent vector and the second latent vector as the preliminary exchange latent vector.

[0048] In one possible implementation, the face attribute stabilizer is based on the Transformer model architecture and is trained using preset face attribute stabilizer training samples. In this process, the latent vectors are initially swapped as the Q matrix of the Transformer model architecture, and the latent vectors in the style space of the target face are simultaneously used as the K and V matrices of the Transformer model architecture.

[0049] In one possible implementation, the face generator is based on the StyleGAN2 model architecture and is trained using preset face generator training samples; wherein the latent vector in the pose space of the target face image is used as the basic input of the StyleGAN2 model architecture, the swapped latent vector is used as the style control input of the StyleGAN2 model architecture, and the noise input of the StyleGAN2 model architecture is removed.

[0050] In one possible implementation, the step of calling a preset face fusion processor includes: blurring the face region mask image of the target face image, decreasing the pixel values ​​of the outer ring of the blurred face region mask image from the inside out according to a preset adjustment threshold to obtain a softened face region mask image; multiplying the softened face region mask image with the initial face-swapping image and extracting the face image; and fusing the face image with the target face image based on the Boson fusion method.

[0051] All relevant content of each step involved in the aforementioned embodiments of the deepfake face-swapping image generation method can be referenced to the functional description of the corresponding functional module of the deepfake face-swapping image generation system in the embodiments of the present invention, and will not be repeated here.

[0052] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0053] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a deepfake face-swapping image generation method.

[0054] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the deepfake face-swapping image generation method in the above embodiments.

[0055] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for generating deepfake face-swapping images, characterized in that, include: Obtain the source face image and the target face image; The preset face encoder is invoked based on the source face image to obtain the latent vector in the style space of the source face image; the preset face encoder is invoked based on the target face image to obtain the latent vector in the style space, the latent vector in the pose space, and the face region mask image of the target face image. Based on the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image, a preset latent vector selector is invoked to obtain the initial swapped latent vectors; based on the initial swapped latent vectors and the latent vectors in the style space of the target face image, a preset face attribute stabilizer is invoked to obtain the swapped latent vectors. Based on the latent vectors of the swapped latent vectors and the latent vectors in the pose space of the target face image, a preset face generator is called to obtain a preliminary face-swapping image. Based on the preliminary face-swapping image, the target face image, and the face region mask image of the target face image, a preset face fusion device is called to fuse them to obtain the face-swapping image. The latent vector selector includes a global pooling layer, a first fully connected layer, a ReLU layer, and a second fully connected layer connected in sequence; the step of obtaining preliminary exchange latent vectors by calling a preset latent vector selector based on the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image includes: The latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image are input into the global pooling layer of the latent vector selector for feature compression. The entire spatial features of each latent vector are encoded to obtain the global descriptive features of each latent vector. The global descriptive features of each latent vector are input into the first fully connected layer and then activated by a ReLU layer. The output is then input into the second fully connected layer to restore the original dimensions, and the sigmoid function is used to activate and obtain the quantitative score of each latent vector. The latent vector with the highest score in the style space of the source face image is selected as the first latent vector, and the latent vector with the highest score in the style space of the target face image is selected as the second latent vector. The first latent vector and the second latent vector are combined as the initial exchange latent vector. The process of calling a preset facial fusion processor for fusion includes: The face region mask image of the target face image is blurred. The pixel values ​​of the outer ring of the blurred face region mask image are decreased from the inside to the outside according to the preset adjustment threshold to obtain the softened face region mask image. The softened face region mask image is multiplied by the initial face-swapping image to extract the face image, and the face image and the target face image are fused based on the Boson fusion method.

2. The method for generating deepfake face-swapping images according to claim 1, characterized in that, The face encoder includes a face feature extractor, a latent space mapper, and a face region perceptron. When the input to the face feature extractor is the source face image, the multi-scale output of the face feature extractor is used as the input to the latent space mapper through the feature pyramid structure. The latent space mapper then obtains the latent vector in the style space of the source face image. When the input to the face feature extractor is the target face image, the multi-scale output of the face feature extractor is used as the input to the latent space mapper through the feature pyramid structure. The latent space mapper obtains the latent vectors in the style space and pose space of the target face image. The input to the face region perceptron is obtained through the U-Net structure and the concatenation of feature maps. The face region perceptron obtains the face region mask image of the target face image.

3. The method for generating deepfake face-swapping images according to claim 2, characterized in that, The facial feature extractor is a residual neural network ResNet50.

4. The method for generating deepfake face-swapping images according to claim 1, characterized in that, The face attribute stabilizer is based on the Transformer model architecture and is trained using preset face attribute stabilizer training samples. In this process, the latent vectors are initially swapped as the Q matrix of the Transformer model architecture, and the latent vectors in the style space of the target face are simultaneously used as the K and V matrices of the Transformer model architecture.

5. The method for generating deepfake face-swapping images according to claim 1, characterized in that, The face generator is based on the StyleGAN2 model architecture and is trained using preset face generator training samples. The latent vector in the pose space of the target face image is used as the basic input of the StyleGAN2 model architecture, the swapped latent vector is used as the style control input of the StyleGAN2 model architecture, and the noise input of the StyleGAN2 model architecture is removed.

6. A deepfake face-swapping image generation system, characterized in that, include: The image acquisition module is used to acquire source face images and target face images; The face encoding module is used to call a preset face encoder based on the source face image to obtain the latent vector in the style space of the source face image; and to call a preset face encoder based on the target face image to obtain the latent vector in the style space, the latent vector in the pose space, and the face region mask image of the target face image. The latent vector exchange module is used to obtain a preliminary exchange latent vector by calling a preset latent vector selector based on the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image; and to obtain a final exchange latent vector by calling a preset face attribute stabilizer based on the preliminary exchange latent vectors and the latent vectors in the style space of the target face image. The fusion module is used to call a preset face generator to obtain a preliminary face-swapped image based on the latent vectors of the swapped latent vectors and the latent vectors in the pose space of the target face image, and to call a preset face fusion module to fuse the preliminary face-swapped image, the target face image and the face region mask image of the target face image to obtain the face-swapped image. The latent vector selector includes a global pooling layer, a first fully connected layer, a ReLU layer, and a second fully connected layer connected in sequence; the step of obtaining preliminary exchange latent vectors by calling a preset latent vector selector based on the latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image includes: The latent vectors in the style space of the source face image and the latent vectors in the style space of the target face image are input into the global pooling layer of the latent vector selector for feature compression. The entire spatial features of each latent vector are encoded to obtain the global descriptive features of each latent vector. The global descriptive features of each latent vector are input into the first fully connected layer and then activated by a ReLU layer. The output is then input into the second fully connected layer to restore the original dimensions, and the sigmoid function is used to activate and obtain the quantitative score of each latent vector. The latent vector with the highest score in the style space of the source face image is selected as the first latent vector, and the latent vector with the highest score in the style space of the target face image is selected as the second latent vector. The first latent vector and the second latent vector are combined as the initial exchange latent vector. The process of calling a preset facial fusion processor for fusion includes: The face region mask image of the target face image is blurred. The pixel values ​​of the outer ring of the blurred face region mask image are decreased from the inside to the outside according to the preset adjustment threshold to obtain the softened face region mask image. The softened face region mask image is multiplied by the initial face-swapping image to extract the face image, and the face image and the target face image are fused based on the Boson fusion method.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the deepfake face-swapping image generation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the deepfake face-swapping image generation method as described in any one of claims 1 to 5.

Images