Face-swapping model training method, image face-swapping method, apparatus, electronic device, computer-readable storage medium, and computer program product

Abstract

Provided in the present application are a face-swapping model training method, an image face-swapping method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The face-swapping model training method comprises: acquiring a source image and a template image; by means of a face-swapping model to be trained, performing face-swapping processing on the template image on the basis of the source image, so as to obtain a synthetic facial image; performing loss calculation on the basis of the template image and the synthetic facial image, so as to obtain an angle-of-view loss value of the synthetic facial image relative to the template image; and updating model parameters of the face-swapping model on the basis of the angle-of-view loss value, so as to obtain a trained face-swapping model.

Description

Face-swapping model training methods, image face-swapping methods, devices, electronic devices, computer-readable storage media, and computer program products

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202410672158.7, filed on May 27, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of artificial intelligence, and in particular to a face-swapping model training method, an image face-swapping method, a device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0004] Image face-swapping is widely used in various applications, such as film and television portrait production, game character design, virtual avatars, and privacy protection. In film and television portrait production, when actors cannot perform professional actions, professionals can perform them first, and then face-swapping algorithms can automatically replace the professional's face with the actor's; alternatively, it can be used to replace certain artists, avoiding reshoots and saving significant costs. In virtual avatars, users can use face-swapping algorithms to swap faces with virtual characters, enhancing the entertainment value of live streams while protecting personal privacy. Furthermore, the results of image face-swapping can provide adversarial attack training material for services such as facial recognition.

[0005] In film and television production, music video production, and humorous video production, ensuring that the perspective and expression of the synthesized facial image generated by face-swapping algorithms are consistent with the template image is crucial. However, face-swapping algorithms in related technologies typically only optimize the algorithm based on the overall face or facial features, neglecting the eye perspective, a small but very important part of the overall face. Summary of the Invention

[0006] This application provides a face-swapping model training method, an image face-swapping method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the model performance of face-swapping models.

[0007] The technical solution of this application embodiment is implemented as follows:

[0008] This application provides a face-swapping model training method, executed by an electronic device. The method includes: acquiring a source image and a template image; the source image includes a first face of a first object, and the template image includes a second face of a second object; performing face-swapping processing on the template image based on the source image using a face-swapping model to be trained, to obtain a synthesized facial image; performing loss calculation based on the template image and the synthesized facial image to obtain a viewpoint loss value of the synthesized facial image relative to the template image; wherein, the viewpoint loss value is used to characterize the degree of difference between the eye viewpoint of the third object in the synthesized facial image and the eye viewpoint of the second object in the template image; updating the model parameters in the face-swapping model based on the viewpoint loss value to obtain a trained face-swapping model.

[0009] This application provides an image face-swapping method, which includes: in response to a face-swapping request, obtaining a face-swapping dataset, the face-swapping dataset including a source image and a template image; inputting the source image and the template image into a trained face-swapping model, and performing image synthesis processing on the source image and the template image through the trained face-swapping model to obtain a face-swapping image; wherein, the trained face-swapping model is trained using the face-swapping model training method provided in this application.

[0010] This application provides a face-swapping model training device, comprising: an acquisition module configured to acquire a source image and a template image; the source image includes a first face of a first object, and the template image includes a second face of a second object; a face-swapping processing module configured to perform face-swapping processing on the template image based on the source image using a face-swapping model to be trained, to obtain a synthesized face image; a loss calculation module configured to perform loss calculation based on the template image and the synthesized face image, to obtain a viewpoint loss value of the synthesized face image relative to the template image; wherein the viewpoint loss value is used to characterize the degree of difference between the eye viewpoint of the third object in the synthesized face image and the eye viewpoint of the second object in the template image; and a model parameter update module configured to update the model parameters in the face-swapping model based on the viewpoint loss value, to obtain a trained face-swapping model.

[0011] This application provides an image face-swapping device, comprising: a dataset acquisition module configured to acquire a face-swapping dataset in response to a face-swapping request, the face-swapping dataset including a source image and a template image; and a face-swapping module configured to input the source image and the template image into a trained face-swapping model, and to perform image synthesis processing on the source image and the template image through the trained face-swapping model to obtain a face-swapping image; wherein the trained face-swapping model is trained using the face-swapping model training method provided in this application.

[0012] This application provides an electronic device, including: a memory for storing computer-executable instructions; and a processor for executing the computer-executable instructions stored in the memory to implement the face-swapping model training method provided in this application, or to implement the image face-swapping method provided in this application.

[0013] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the face-swapping model training method provided in this application, or implement the image face-swapping method provided in this application.

[0014] This application provides a computer program product, which includes computer-executable instructions stored in a computer-readable storage medium. When a processor of an electronic device reads and executes the computer-executable instructions from the computer-readable storage medium, it implements the face-swapping model training method or the image face-swapping method provided in this application.

[0015] The embodiments of this application have the following beneficial effects:

[0016] In training a face-swapping model, the source image and template image are first input into the model. The model then performs face-swapping on the template image based on the source image, resulting in a synthesized facial image. Next, loss calculation is performed based on the template image and the synthesized facial image to obtain the viewpoint loss value of the synthesized facial image relative to the template image. This viewpoint loss value is then used to update the model parameters. Thus, during the training process, the viewpoint loss value obtained from the loss calculation updates the model's parameters, increasing the model's attention to the viewpoint within the facial image. This improves the model's performance and ensures that the viewpoint in the synthesized facial image remains consistent with the viewpoint in the template image during face-swapping, enhancing the effectiveness of the face-swapping. Attached Figure Description

[0017] Figure 1 is a schematic diagram of a face-swapping result provided in an embodiment of this application;

[0018] Figure 2 is a schematic diagram of the structure of the face-swapping model training system provided in an embodiment of this application;

[0019] Figure 3 is a schematic diagram of the face-swapping model training device provided in an embodiment of this application;

[0020] Figure 4 is a schematic diagram of an optional process for training a face-swapping model provided in an embodiment of this application;

[0021] Figure 5 is another optional flowchart of the face-swapping model training method provided in the embodiments of this application;

[0022] Figure 6 is a schematic diagram of another result of face swapping provided by an embodiment of this application;

[0023] Figure 7 is a schematic diagram of the result obtained by the view estimation process through a pre-trained view estimation network provided in an embodiment of this application;

[0024] Figure 8 is a schematic diagram of the results obtained by the face registration process through a pre-trained 256-point registration network provided in an embodiment of this application;

[0025] Figure 9 is a schematic diagram of the training framework of the face-swapping model provided in the embodiments of this application;

[0026] Figure 10 is a schematic diagram of key eye points provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments, but it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0030] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0031] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0032] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0033] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0034] 1) Generative Adversarial Networks (GANs): This is an unsupervised learning method. A GAN consists of a generator network and a discriminator network, learning by having the two neural networks compete against each other. The generator network randomly samples data from the latent space as input, and its output should closely mimic real samples from the training set. The discriminator network takes either real samples or the generator network's output as input. Its goal is to distinguish the generator network's output from real samples as much as possible, while the generator network aims to confuse the discriminator network. The two neural networks compete against each other, continuously adjusting their parameters, until they eventually generate images that the discriminator network cannot distinguish between real and fake images.

[0035] 2) Image face swapping: Image face swapping is defined as converting the face in the input source image to the face in the template image, while maintaining the viewpoint, expression, and background information of the face in the output synthetic face image. As shown in Figure 1, the input source image 10 includes person A, and the template image 20 includes person B. After face swapping 101, the output synthetic face image 102 is a face image that combines the identity 106 of person A in the source image with the expression 103, viewpoint 104, and background 105 of person B in the template image. The viewpoint of this face image is the same as that of person B in the template image 20.

[0036] 3) Ground Truth (GT) Image: This is the expected output of the face-swapping result. GT images are often used as a standard for comparing and evaluating the performance of face-swapping algorithms. Evaluating the face-swapping performance of a face-swapping model using GT images can be achieved by comparing the face-swapped image generated by the model with the GT image. Various evaluation metrics can be used to assess the face-swapping performance of a model, such as face recognition accuracy, fusion quality, and consistency of facial expressions and poses. Furthermore, keypoint annotation methods can be used to mark key points in the face-swapping region to more accurately evaluate the performance of the face-swapping model.

[0037] In film and television production, music video production, and humorous video production, ensuring that the perspective and expression of the synthesized facial image generated by face-swapping algorithms are consistent with the template image is crucial. This is especially true in close-up scenes, where special attention needs to be paid to eye perspective. For example, when a person in a video requiring face-swapping is looking at the camera, it's desirable for the synthesized facial image generated by the face-swapping algorithm to also be looking at the camera accurately. However, related technologies are based on conventional face-swapping model training processes and often neglect perspective alignment. Typically, they construct image triplets of source, template, and ground truth images and design the generative network structure of the face-swapping model, training the model at the overall facial level. Therefore, face-swapping algorithms in related technologies usually only optimize the algorithm based on the overall face or facial features, ignoring the eye perspective, a small but very important component of the overall face.

[0038] Based on the technical problems existing in related technologies, this application proposes a viewpoint-aligned image face-swapping method, which involves a face-swapping model. The input to this face-swapping model is a source image and a template image. The goal of the face-swapping model is to ensure that the face-swapping result (i.e., the face in the synthesized facial image) maintains the facial expression, angle, and background of the template image, and is as similar as possible to the face in the source image. To achieve accurate eye viewpoint alignment while performing face-swapping, it is first necessary to determine whether the eyes are closed in both the template image and the synthesized facial image. For scenarios where the person in the template image has closed eyes, the person in the generated synthesized facial image must also maintain a closed eye state to ensure consistency in the eye closure state before and after face-swapping. For scenarios where the person in the template image has open eyes, by estimating the eye viewpoint and estimating 256 face registration points, and optimizing the eye viewpoint, the eye viewpoint of the person in the generated synthesized facial image is kept consistent with the eye viewpoint of the person in the template image, thereby better meeting the face-swapping needs of practical applications.

[0039] Here, 256-point face registration estimation is a face registration point estimation method. Its purpose is to align and normalize faces by locating key points (such as eyes, nose, and mouth). Face registration point estimation automatically detects the locations of key points in a face image using algorithms. These key points can include multiple points from features such as eyes, eyebrows, nose, and mouth, and are used for face alignment and normalization. In some applications, the number of face registration points can reach 256, allowing for a more detailed description of the face's shape and pose.

[0040] The following describes exemplary applications of the face-swapping model training device (i.e., electronic device) provided in the embodiments of this application. The device provided in the embodiments of this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, or in-vehicle terminals, or it can be implemented as a server. The following will describe exemplary applications when the face-swapping model training device is implemented as a server.

[0041] Referring to Figure 2, which is a schematic diagram of the architecture of the face-swapping model training system 100 provided in the embodiment of this application, in order to support a face-swapping model training application, the face-swapping model training application runs on the terminal 400. The terminal 400 is connected to the server 200 through the network 300, which can be a wide area network or a local area network, or a combination of both.

[0042] Terminal 400 sends a face-swapping model training request for the current virtual scene to server 200. Server 200 constitutes the face-swapping model training device in this embodiment of the application. In response to the face-swapping model training request, server 200 acquires a source image and a template image. Then, it performs face-swapping processing on the template image based on the source image using the face-swapping model to be trained, obtaining a synthesized facial image. Next, it performs loss calculation based on the template image and the synthesized facial image to obtain the viewpoint loss value of the synthesized facial image relative to the template image. Finally, it updates the model parameters in the face-swapping model based on the viewpoint loss value, obtaining the trained face-swapping model. After obtaining the trained face-swapping model, server 200 returns the model parameters of the trained face-swapping model to terminal 400, so that terminal 400 can output the model parameters of the trained face-swapping model, or terminal 400 can call the trained face-swapping model based on the model parameters to continue the next business processing step.

[0043] In this embodiment, server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0044] Referring to Figure 3, which is a schematic diagram of the structure of an electronic device 40 provided in an embodiment of this application, the electronic device 40 shown in Figure 3 can be a face-swapping model training device. The face-swapping model training device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the face-swapping model training device are coupled together through a bus system 440. It is understood that the bus system 440 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 440 in Figure 3.

[0045] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc. User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls. Memory 450 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 450 may optionally include one or more storage devices physically located away from processor 410. Memory 450 includes volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), and volatile memory can be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory. In this application embodiment, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0046] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; network communication module 452 for reaching other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB); presentation module 453 for enabling the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., display screen, speaker, etc.); input processing module 454 for detecting and translating one or more user inputs or interactions from one or more input devices 432.

[0047] The apparatus provided in this application embodiment can be implemented in software. Figure 3 shows a face-swapping model training apparatus 455 stored in memory 450. This apparatus can be software in the form of programs and plugins, including the following software modules: an acquisition module 4551, a face-swapping processing module 4552, a loss calculation module 4553, and a model parameter update module 4554. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0048] In other embodiments, the apparatus provided in this application can be implemented in hardware. For example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the face-swapping model training method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0049] In this embodiment, the terminal or server can implement the face-swapping model training method provided in this embodiment by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system, native applications (APPs), i.e., programs that need to be installed in the operating system to run, or small programs that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In short, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0050] The face-swapping model training method provided in this application is used to train a face-swapping model, and the trained face-swapping model is used to implement the image face-swapping method. The image face-swapping method provided in this application can be applied to various business scenarios, such as film and television production, music video production, and humorous video production, as illustrated below.

[0051] In film and television production, this technology can restore actors' appearances and recreate historical figures. For example, during film production, the image face-swapping method provided in this application can be used to restore the image of actor A at age 20. This technology can also be used to restore facial details of actors in old films, giving classic works a fresh new look. Alternatively, it can create entirely new characters and digitally age actors. For example, during film production, the image face-swapping method provided in this application can be used to digitally age actors B and C, allowing them to appear younger. This technology can create entirely new characters for films or make actors' appearances at different ages more natural. It can also reduce filming risks and costs. For example, in the filming of high-risk or special scenes, the image face-swapping method provided in this application can be used to replace an actor's face with a stand-in or virtual character, reducing filming risks. Furthermore, for filming interruptions due to actors' schedules, health, or other reasons, the image face-swapping method provided in this application can also complete the remaining shots by replacing faces, saving on reshoot costs.

[0052] In music video (MV) production, creative visual effects can be achieved. For example, the image face-swapping method provided in this application can be used to replace a singer's facial expressions with those of different characters or scenes, creating unique visual effects. For instance, a singer can instantly transform into a movie character, an anime character, or another style of image in the MV, enhancing its fun and appeal. Alternatively, it can be used to achieve cross-temporal collaboration. The image face-swapping method provided in this application can "merge" singers from different periods or locations to collaborate on the same MV. For example, replacing the face of a deceased singer with that of a modern singer can create a cross-temporal duet, adding emotion and topicality to the MV. Furthermore, it can be used to quickly generate multiple versions. To adapt to different marketing needs, production teams can use the image face-swapping method provided in this application to quickly generate multiple MV versions of the same song, each with different facial expressions or images of the singer, thus satisfying the preferences of different audience groups.

[0053] In the context of creating humorous videos, users can create funny emojis and videos. They can replace their own facial expressions with various funny images or video clips to generate personalized emojis or videos. For example, they can transform themselves into a comedic character in a movie or swap faces with friends to create hilarious effects for sharing on social media. Alternatively, it can be used for virtual character experiences. Using the image face-swapping method provided in this application, users can replace their faces with virtual characters to experience different styles of virtual worlds. For example, in anime, games, or science fiction scenes, users can attribute their facial features to virtual characters and experience the fun of becoming a virtual persona. Furthermore, it can be used for creative video challenges. In some creative video challenges, such as the "face-swapping challenge" or "face-changing challenge," participants can use face-swapping technology to replace their faces with various celebrities, animals, or other objects to create highly creative and entertaining video content.

[0054] The face-swapping model training method provided in the embodiments of this application will be described below. The face-swapping model training method provided in the embodiments of this application can be executed by an electronic device, which can be a server or a terminal. That is, the face-swapping model training method in the embodiments of this application can be executed by a server, by a terminal, or through interaction between a server and a terminal.

[0055] Referring to Figure 4, which is an optional flowchart of the face-swapping model training method provided in this application embodiment, the method will be described in conjunction with the steps shown in Figure 4. Taking the server as the execution subject of the face-swapping model training method as an example, the method includes the following steps S101 to S104:

[0056] Step S101: Obtain the source image and the template image.

[0057] In this embodiment, the source image refers to the original image that provides the face (first face) to be replaced during the face-swapping process. In other words, the source image is the original image to be processed by face-swapping. The source image contains facial information of a specific object (i.e., the first object), which will be used to replace the face in the target image. Specifically, the source image includes the first face of the first object, which can be any object requiring face-swapping. The first face is the face replaced during the face-swapping process; thus, the source image is the image that provides the identity information of the first face during the face-swapping process.

[0058] In image face-swapping, the source image serves to provide facial features. Facial features (such as eyes, nose, mouth, and skin texture) in the source image are extracted and used to replace the face in the template image. The face in the source image contains the identity information of the original object, which is preserved during the face-swapping process, making the face in the synthesized face image look like the original object in the source image. The facial style of the source image (such as makeup, expression, and lighting conditions) also affects the effect of the synthesized face image after face swapping.

[0059] Here, we take face-swapping in film and television production as an example. Suppose we are producing a historical drama and need to place the face of a modern actor onto the body of an ancient character. The source image can be a photograph or video frame of the modern actor, containing the actor's face (the first face). This photograph provides the modern actor's facial features and identity information. The template image can be an image of the ancient character's body, showing the facial portion that needs to be replaced. Accordingly, the face-swapping process can involve using a face-swapping model to extract the facial features of the modern actor from the source image and replace them with the facial features of the ancient character in the target image.

[0060] To ensure a natural and realistic face-swapping effect, the source image needs to be high-quality, with clear facial features and good lighting conditions. Facial expressions and postures in the source image will affect the quality of the synthesized face image after face swapping. If the expression or posture of the object in the template image differs too much from the source image, the source image can be preprocessed to match the template image. In this embodiment, the facial identity information in the source image is explicit so that the identity features can be accurately preserved during the face-swapping process.

[0061] The template image is the image to be face-swapping processed. It includes the second face of a second object, which can be any object to be face-swapped. The second face is the face that will be replaced during the face-swapping process. In other words, the template image provides information such as the expression, angle, and background of the second face during face-swapping. The facial expression and posture in the template image determine the final face-swapping effect, while the facial angle and lighting conditions affect the naturalness of the result. The background and other elements of the template image are also retained after face-swapping, therefore they need to match the facial features of the source image.

[0062] This example uses character replacement in film production. Suppose a movie is being made, and the face of a deceased actor needs to be replaced on the body of a modern actor. The template image could be a shot of the modern actor in the movie, containing the modern actor's face (second face). The template image provides the expression, angle, and background information of the face to be replaced. The source image is a photo or video frame of the deceased actor, containing the deceased actor's face (first face). Accordingly, the face-swapping process can be achieved by using a face-swapping model to extract the facial features of the deceased actor from the source image and replace them with the facial features of the modern actor in the template image, thus "resurrecting" the deceased actor in the movie.

[0063] To ensure a natural and realistic face-swapping effect, the facial expressions and poses in the template image need to match the facial features in the source image to guarantee a natural look. The lighting conditions of the template image also affect the face-swapping effect and should be kept as consistent as possible. The background and other elements of the template image will be retained after the face swap, so they need to be consistent with the facial features of the source image.

[0064] In the face-swapping process, the identity information of the first face in the source image is fused with the expression, angle, background, and other information of the second face in the template image to obtain the final face-swapping result. The source image and the template image can be used as model inputs to the face-swapping model to be trained, participating in the model training process.

[0065] For example, suppose the source image is a photo containing a man, and the template image is a photo containing a woman. The user wants to swap the face of the woman in the template image with the face of the man in the source image. In this example, the ground truth image is the ideal face replacement image obtained after face swapping the source and template images. The training objective of the face-swapping model is to obtain an effect as close as possible to the ground truth image by processing the source and template images.

[0066] Here, by obtaining the source image and the template image, it is easier to use the source image and the template image as input for model training of the face-swapping model.

[0067] Step S102: The face-swapping model to be trained performs face-swapping processing on the template image based on the source image to obtain a synthesized facial image.

[0068] In this embodiment of the application, the face-swapping model to be trained refers to the face-swapping model that needs to be trained. The face-swapping model to be trained can be any network structure, such as Generative Adversarial Network (GAN), Encoder-Decoder network, Autoencoder, Variational Autoencoder (VAE), Masked Autoencoder (MAE), Transformer-based network, Diffusion Model, Multi-stage network, and Region-aware face-swapping network, etc.

[0069] Encoder-decoder networks are a common structure in deepfake generation networks. An encoder-decoder network contains at least one encoder and one decoder, with narrow intermediate layers, forcing the network to learn high-level semantic concepts from the input data. In face-swapping tasks, the encoder extracts features from the source and template images, while the decoder fuses these features to generate a synthetic facial image. This structure is simple and efficient, suitable for various face-swapping scenarios. Autoencoders are unsupervised learning neural network models consisting of an encoder and a decoder. During face-swapping, autoencoders learn key features of the input data by mapping it to a latent representation and reconstructing the input data. For example, the DeepFakes framework consists of two autoencoders whose encoders share parameters, achieving face-swapping effects by learning to reconstruct faces in the source and template images. Variational autoencoders are a variant of autoencoders, performing better in content generation. Variational autoencoders generate new samples in the latent space by learning the probability distribution of the input data. In face-swapping, variational autoencoders can be used to generate face-swapping images with different identities and attributes while maintaining good generalization ability. Mask autoencoders (MAEs) are an improved autoencoder structure with high generalization ability. For example, FlowFace uses a pre-trained MAE to extract facial features from source and template images, and adaptively fuses the identity features of the first object and the attribute features of the second object through a cross-attention fusion module to achieve high-quality face swapping. Some face swapping models also incorporate Transformer architectures to better handle semantic information and details of faces. For example, by learning the semantic-aware correspondence between faces in source and template images, the identity features of the first object are mapped to the corresponding regions of the second object. Diffusion models are probabilistic generative models that iteratively transform random noise into samples that follow a target distribution. In face swapping, the diffusion model uses the identity information of the first object as a guiding condition to diffuse the template image, achieving realistic replacement of facial expressions, poses, and other details. Some face swapping models also employ multi-stage network structures; for example, FaceShifter consists of two networks: AEI-Net and HEAR-Net. AEI-Net generates the initial face swapping result, and HEAR-Net refines the output to achieve high-fidelity and occlusion-aware face swapping. Region-Aware Face Swapping (RAFSwap) networks achieve consistent high-resolution face generation through a local-global approach. RAFSwap introduces a Transformer to simulate misaligned cross-scale semantic interactions and combines it with a face mask predictor module to generate harmonious high-resolution face-swapping effects.These network structures each have their own advantages and are suitable for different face-swapping tasks and scenarios. Choosing the appropriate network structure according to specific needs can improve the naturalness and realism of the face-swapping effect.

[0070] By training the face-swapping model on the input image, the model can learn the relationships and patterns in the input. During training, the face-swapping model continuously optimizes its parameters to minimize the loss function, thereby enabling it to generate the optimal face-swapping result based on a given source image and template image.

[0071] Synthetic facial images are the result of face-swapping processing using a face-swapping model. The synthesized facial images output by the trained face-swapping model can achieve the same realistic effect as the real images.

[0072] Face-swapping processing achieved through a face-swapping model to be trained involves using a source image and a template image as model inputs to the face-swapping model to be trained. Based on the first face of the first object in the source image, the second face of the second object in the template image is replaced. Furthermore, the identity information of the first face in the source image is fused with the expression, angle, and background information of the second face in the template image, so that the synthesized face image obtained after face-swapping processing has the identity information of the first face, while maintaining the expression, angle, and background information of the second face in the template image.

[0073] Here, the source image and template image are input into the face-swapping model to be trained. At this point, since the model parameters of the face-swapping model have not been optimized, the resulting synthetic face image is a preliminary face-swapping result, not the optimal one. The synthetic face image is obtained through face-swapping processing using the face-swapping model to facilitate subsequent parameter optimization of the face-swapping model, thereby improving its performance and generalization ability.

[0074] Step S103: Based on the template image and the synthesized face image, perform loss calculation to obtain the viewpoint loss value of the synthesized face image relative to the template image.

[0075] In this embodiment, the viewpoint loss value is used to characterize the degree of difference between the eye viewpoints in the synthesized facial image and the template image; that is, the viewpoint loss value is used to quantify the difference in eye viewpoints between the synthesized facial image and the template image. Here, the eye viewpoint refers to the viewpoint of the target object (e.g., a third object in the synthesized facial image or a second object in the template image), where the eye viewpoint is the angle formed by observing the two ends of the object from the optical center (or nodal point) of the eye. In image face-swapping technology, the eye viewpoint can be understood as the direction and angle of the eye in the image, usually determined by key points of the eye (such as the pupil, corner of the eye, etc.). For a person, the "viewpoint" can be calculated from the head elevation angle and the viewing direction of the eyes, and is a spatial angle concept.

[0076] This application provides a method for calculating the eye's visual angle. This method is a geometry-based calculation method: key points of the eye (such as the pupil, corner of the eye, etc.) can be detected first, and the angular relationship between these key points can be calculated. Specifically, geometric formulas can be used to calculate the eye's visual angle. For example, the visual angle θ of an object can be calculated using formula (1):

[0077] The height or width of the object can be the size of the eye in the image (such as a template image or a synthetic face image).

[0078] Specifically, the viewpoint loss value in this embodiment measures the difference between the eye viewpoint of the third object in the synthesized facial image and the viewpoint of the second object in the template image. By calculating this difference, the face-swapping model can optimize the generated face-swapping image (i.e., the synthesized facial image) so that the face-swapping image is closer to the template image in terms of eye viewpoint.

[0079] In this embodiment, the calculation of the viewpoint loss value can be achieved as follows: First, keypoint detection is performed, that is, the coordinates of the eyes in the template image and the synthesized face image are detected using face detection and keypoint detection algorithms (such as Dlib or MTCNN); then, the viewpoint difference is calculated, which can be based on the coordinates of the eyes and the intrinsic parameters of the image (such as focal length and sensor size). The eye viewpoint can be calculated using the following formula (2):

[0080] The eye height can be the vertical distance between the upper and lower eyelid points in the image.

[0081] Finally, the difference between the eye viewpoints in the template image and the synthesized face image is calculated. For example, the viewpoint loss value can be defined as the sum of squares of the differences between the eye viewpoints in the synthesized face image and the template image, as shown in the following formula (3): Viewpoint loss value = ∑ i (Perspective)合成，i -Perspective 模板，i ) 2 (3);

[0082] Where 'i' represents the eye number (e.g., left eye and right eye). Viewpoint 合成，i This represents the eye view of the i-th eye in the synthesized facial image. 模板，i This represents the eye perspective of the i-th eye in the template image.

[0083] The application of viewpoint loss in face-swapping model training mainly includes: optimizing the generator by minimizing the viewpoint loss to optimize the model parameters, making the face-swapping images generated by the optimized model more natural and realistic in terms of eye viewpoint; improving the naturalness of the face-swapping effect by ensuring that the eye viewpoint in the synthesized facial image is consistent with the template image, thereby improving the overall naturalness of the face-swapping effect; and enhancing the generalization ability of the face-swapping model by introducing viewpoint loss during training, enabling the face-swapping model to better handle changes in eye viewpoint in different scenarios and enhancing the generalization ability of the face-swapping model.

[0084] In this embodiment, the eye perspective can be represented in two ways depending on the open / closed state of the eyes in the template image and the synthesized facial image (i.e., whether the eyes are open or closed). When both the template image and the synthesized facial image have open eyes, the eye perspective can be the direction of the eye's gaze or the position of the pupil within the eye. The viewing direction refers to the direction of the target object's (e.g., a third object in the synthesized facial image or a second object in the template image) line of sight, i.e., the direction from the target object's eye or the camera's optical center towards the observed object. When both the template image and the synthesized facial image have closed eyes, the eye perspective is the degree of eye closure. Accordingly, when the eyes in both the template image and the synthesized facial image are open, the degree of difference in eye perspective refers to the difference between the gaze direction of the left eye in the template image and the gaze direction of the left eye in the synthesized facial image, and the difference between the gaze direction of the right eye in the template image and the gaze direction of the right eye in the synthesized facial image; or the difference between the position of the pupil in the left eye of the template image and the position of the pupil in the left eye of the synthesized facial image, and the difference between the position of the pupil in the right eye of the template image and the position of the pupil in the right eye of the synthesized facial image. When the eyes in both the template image and the synthesized facial image are closed, the degree of difference in eye perspective refers to the difference between the degree of closure of the left eye in the template image and the degree of closure of the left eye in the synthesized facial image, and the difference between the degree of closure of the right eye in the template image and the degree of closure of the right eye in the synthesized facial image.

[0085] Loss calculation based on template image and synthesized face image can be achieved by obtaining the viewpoint loss value of synthesized face image relative to template image. This can be done as follows: First, feature extraction is performed on template image to obtain image features; feature extraction is performed on synthesized face image to obtain image features of synthesized face image; then, based on the image features of template image and synthesized face image, loss calculation is performed on synthesized face image to obtain the viewpoint loss value of synthesized face image relative to template image.

[0086] Here, the process of extracting features from both the template image and the synthesized face image can be the same type of feature extraction workflow. For example, feature extraction could involve face registration processing on both the template image and the synthesized face image. The image features of the template image obtained through face registration could be the set of facial key points of the second face, and the image features of the synthesized face image could be the set of facial key points of the third face of the third object in the synthesized face image. Alternatively, feature extraction could involve viewpoint estimation on both the template image and the synthesized face image. The image features of the template image obtained through viewpoint estimation could be the viewpoint features of the second face, and the image features of the synthesized face image could be the viewpoint features of the third face of the third object in the synthesized face image. The specific feature extraction process will be explained in detail below.

[0087] In this embodiment, the image features of the template image and the image features of the synthesized face image are obtained through the feature extraction process. This allows the face-swapping model to better optimize its parameters during training by constructing a viewpoint loss based on the image features of the template image and the image features of the synthesized face image. This enables the face-swapping model to achieve better model performance while focusing on the viewpoint consistency between the template image and the synthesized face image.

[0088] To ensure that the viewpoint of the face in the synthesized facial image remains consistent with that of the second face in the template image, this embodiment of the application considers a viewpoint loss value when calculating the loss of the synthesized facial image based on the image features of the template image and the image features of the synthesized facial image. The viewpoint loss value characterizes the error between the viewpoints of the third face and the second face in the synthesized facial image. By calculating the viewpoint loss of the synthesized facial image, the viewpoint loss value of the synthesized facial image relative to the template image can be obtained. The viewpoint loss value can serve as a feedback signal during the training process of the face-swapping model, used to subsequently adjust the model parameters of the face-swapping model, thereby gradually optimizing the face-swapping effect.

[0089] Step S104: Update the model parameters in the face-swapping model based on the viewpoint loss value to obtain the trained face-swapping model.

[0090] In this embodiment, the viewpoint loss value is used to evaluate the difference in viewpoint between the synthetic facial image generated by the face-swapping model and the template image, reflecting the performance of the face-swapping model in terms of viewpoint features. Based on the viewpoint loss value, the model parameters can be updated using optimization algorithms such as gradient descent to minimize the viewpoint loss value, thereby training the optimal face-swapping model.

[0091] During the process of updating the model parameters in the face-swapping model based on the viewpoint loss value, the gradient of the model parameters can be calculated based on the viewpoint loss value. For example, the gradient of the viewpoint loss value with respect to the model parameters can be calculated using the backpropagation algorithm. That is, the model parameters are updated along the direction that reduces the viewpoint loss value, so that the viewpoint loss value gradually decreases and the model performance is gradually optimized. Finally, the face-swapping model is trained iteratively. The above steps can be repeated to continuously update the model parameters, calculate the total loss value, and optimize the model performance until the model converges or reaches the preset iteration condition. At this point, the update of the model parameters in the face-swapping model is stopped, and the trained face-swapping model is saved. The preset iteration condition can be a preset iteration duration or a preset number of iterations.

[0092] Here, model convergence refers to the gradual stabilization of the face-swapping model's performance (such as the loss function value) during training, indicating that the model has learned the main features of the data. Model convergence can be judged by indicators such as stable loss function value or no longer improving validation set performance. Stable loss function value means that after training to a certain extent, the loss function value no longer decreases significantly, and may even show slight fluctuations, which usually means the face-swapping model has converged. No longer improving validation set performance means that if the face-swapping model's performance (such as accuracy or loss value) no longer improves on the validation set, or even begins to decline, this may be an indication of overfitting, and can also be considered as the face-swapping model having converged.

[0093] The preset iteration conditions can be manually set training stopping conditions, which are used to control the termination of the training process. In this embodiment, the preset iteration conditions include: a fixed number of iterations, a loss threshold, an early stopping strategy, and manual stopping. A fixed number of iterations means setting a fixed number of iterations; when this number is reached, training stops regardless of whether the face-swapping model has converged. For example, the number of iterations can be set to 10,000. The loss threshold means stopping training when the value of the loss function drops to a preset threshold. For example, training stops when the loss value drops below 0.1. The early stopping strategy involves detecting performance metrics on the validation set, such as loss value or accuracy. If these metrics do not improve significantly or begin to deteriorate after a certain number of iterations, training is stopped early. Manual stopping means manually judging whether the face-swapping model has been trained to a satisfactory effect based on the quality of the preview images during training, and then stopping training.

[0094] The face-swapping model training method provided in this application involves inputting a source image and a template image into the face-swapping model to be trained. The model then performs face-swapping processing on the template image based on the source image to obtain a synthesized facial image. Loss calculation is performed based on the template image and the synthesized facial image to obtain the viewpoint loss value of the synthesized facial image relative to the template image. The model parameters are then updated based on this viewpoint loss value to obtain the trained face-swapping model. Thus, during the model training process, the viewpoint loss value obtained through viewpoint loss calculation updates the model's parameters, increasing the model's attention to the viewpoint in the facial image and improving its performance. This ensures that the viewpoint in the synthesized facial image obtained after face-swapping is consistent with the viewpoint in the template image, thereby enhancing the realism of the synthesized facial image.

[0095] The following will describe the face-swapping model training method in this application embodiment, focusing on the interaction between the terminal and the server in the face-swapping model training system. It should be noted that the face-swapping model training method here is implemented through interaction between the terminal and the server, and is essentially the same as the face-swapping model training method executed by the server in the above embodiments. The only difference is that this application embodiment also describes the actions performed by the terminal during the execution of the face-swapping model training method. Furthermore, some steps can be executed by either the terminal or the server. Therefore, for steps in this embodiment that are the same as those in the above embodiments but have different execution subjects, this embodiment is merely illustrative. In the implementation process, any execution subject can perform these steps, and this application embodiment does not limit this.

[0096] Based on the face-swapping model training method provided in the above embodiments, this application further provides a face-swapping model training method, which can be combined with the face-swapping model training methods in the above embodiments. Figure 5 is another optional flowchart of the face-swapping model training method provided in this application. As shown in Figure 5, the method includes the following steps S201 to S210:

[0097] Step S201: The terminal receives the face-swapping model training operation input by the user.

[0098] In this embodiment, the user can be either a developer or a user of the face-swapping model training application. The user can input face-swapping model training operations on the client side of the application. The application provides a face-swapping model training function, on which the user can input face-swapping model training operations to trigger a face-swapping model training request. When inputting the face-swapping model training operations, the user can also simultaneously input the model input content during the training process, namely the source image, template image, and ground truth image. When the terminal receives the source image, template image, and ground truth image, a confirmation button will pop up on the face-swapping model training function page to confirm the face-swapping model training. After the terminal detects that the user has clicked the confirmation button, further face-swapping model training is performed based on the source image, template image, and ground truth image. Alternatively, in some embodiments, the user can directly input the source image, template image, and ground truth image on the face-swapping model training function page. Upon receiving the source image, template image, and ground truth image, the terminal can directly trigger the face-swapping model training function and then perform further face-swapping model training based on the source image, template image, and ground truth image. Alternatively, in some embodiments, when the terminal receives the user's input for face-swapping model training, the source image, template image, and ground truth image are directly obtained from the background of the face-swapping model training application, and then further face-swapping model training is performed based on the source image, template image, and ground truth image.

[0099] In step S202, the terminal generates a face-swapping model training request in response to the face-swapping model training operation.

[0100] In this embodiment, the user-inputted model input can be encapsulated into the face-swapping model training request. For example, the display interface of the face-swapping model training application shows the source image, template image, and ground truth image. The user can select or input data according to actual needs, and then the user-inputted or selected source image, template image, and ground truth image are encapsulated into the face-swapping model training request.

[0101] In step S203, the terminal sends the face-swapping model training request to the server.

[0102] In this embodiment, after generating a face-swapping model training request, the terminal can send the request to the server. The server responds to the request by training the face-swapping model based on the source image, template image, and ground truth image carried in the request. The terminal can send the training request using protocols such as HTTP or WebSocket.

[0103] In step S204, the server responds to the face-swapping model training request by obtaining the source image and the template image.

[0104] In this embodiment of the application, the source image includes the first face of a first object, and the template image includes the second face of a second object.

[0105] In response to a face-swapping model training request, if the request includes a source image and a template image, these can be directly parsed and obtained. For an explanation of the specific meaning and implementation of the step of obtaining the source image and template image in response to the face-swapping model training request, please refer to the description of step S101 above; it will not be repeated here.

[0106] In step S205, the server uses the face-swapping model to be trained to perform face-swapping processing on the template image based on the source image to obtain a synthesized facial image.

[0107] In this embodiment of the application, the specific meaning and implementation of the step of performing face swapping on the template image based on the source image using the face swapping model to be trained to obtain a synthesized facial image can be found in the description of step S102 above, and will not be repeated here.

[0108] In this embodiment, the face-swapping model includes at least a generator network, which includes an encoder and a decoder. The face-swapping model, used to train, performs face-swapping processing on a template image based on a source image to obtain a synthetic facial image. This can be achieved in the following way: First, the source image and the template image are encoded using a convolutional module in the encoder to obtain encoded features. Next, identity features are extracted from the source image to obtain the identity features of the first face. Then, the encoded features and the identity features of the first face are fused using a feature fusion layer in the encoder to obtain fused features. Finally, the fused features are decoded using a decoder to obtain the synthetic facial image.

[0109] Here, the encoder's convolutional module includes multiple convolutional layers, with the output dimension decreasing layer by layer. The decoder includes multiple deconvolutional layers, with the output dimension increasing layer by layer. The output dimension of the last deconvolutional layer is the same as the input dimension of the first convolutional layer in the encoder. The encoder's input data can be the result of concatenating the source and template images. The encoder continuously halves the output dimension of the input data through convolution calculations, while gradually increasing the number of channels. For example, if both the source and template images have 3 RGB channels, the encoder's input data dimension can be 512*512*6. After passing through multiple convolutional layers in the convolutional module, the input data is progressively encoded into 256*256*32, 128*128*64, 64*64*128, and finally 32*32*256. During the encoder's convolution calculation process, continuously halving the output dimension helps to abstract and compress the spatial information of the input data. Meanwhile, gradually increasing the number of channels helps to capture the characteristics of the input data in different aspects. This processing method helps to improve the generalization ability and expressive ability of the face-swapping model, thereby enabling the face-swapping model to better understand and process the input data.

[0110] In this embodiment, identity feature extraction from the source image can be achieved using a pre-trained face recognition network. The pre-trained face recognition network can be a Convolutional Neural Network (CNN) model, such as VGGFace or FaceNet. The source image is input into the pre-trained face recognition network, and the output of the last layer of the face recognition network is obtained through forward propagation. This output is then used as the identity feature of the first face in the source image.

[0111] After obtaining the encoded features output by the encoder's convolutional module and the identity features of the first face, these features are input into the encoder's feature fusion layer. For example, the feature fusion layer could be an adaptive instance normalization layer. The feature fusion layer fuses the encoded features and the identity features of the first face together to obtain fused features. Decoding these fused features yields a synthesized face image. This results in a synthesized face image that retains both the high-level features extracted by the encoder's convolutional module and the identity information of the first face in the source image, thereby improving the model performance and face-swapping effect of the face-swapping model.

[0112] The decoder primarily performs deconvolution calculations, which are the reverse of the convolutional calculations in the encoder. The decoder progressively increases the output dimension of the fused features while gradually decreasing the number of channels, thus mapping the abstract representation (i.e., the fused features) back to the original data space and reconstructing the complete structure of the input data. This ensures that the resolution of the synthesized facial image remains consistent with the resolutions of the source and template images. For example, the decoder progressively decodes the fused feature dimensions to 32*32*256, 64*64*128, 128*128*64, 256*256*32, and finally 512*512*3, obtaining the final output data of the face-swapping model, i.e., the synthesized facial image.

[0113] In step S206, the server calculates the eye-closing loss of the synthesized face image based on the template image and the synthesized face image, and obtains the eye-closing loss value of the synthesized face image relative to the template image.

[0114] In this embodiment, the opening and closing state of the eyes in the second face includes a closed eye state and an open eye state. The closed eye loss value is the loss value when the opening and closing state of the eyes in the second face is the closed eye state. This closed eye loss value is used to characterize the degree of difference between the eye view of the third object and the eye view of the second object when the opening and closing state of the eyes in the second face is the closed eye state.

[0115] The opening and closing state of the eyes in the second face can be determined as follows: Based on the template image, determine the set of key points for the eyes of the second face; from the set of key points for the eyes of the second face, determine two corner points and two eyelid points for each eye in the second face; for the first eye in the second face, determine the first key point distance between the two eyelid points and the second key point distance between the two corner points; based on the distance ratio between the first key point distance and the second key point distance, determine the opening and closing state of the eyes in the second face.

[0116] Here, the set of key facial points for the second face is obtained by performing facial registration processing on the template image. Facial registration processing refers to detecting the facial regions of the second face in the template image and determining multiple facial key points of the second face, i.e., the set of facial key points for the second face. In other words, facial registration processing refers to adjusting the template image containing the second face using a preset algorithm to align the same facial feature points in different images. In face-swapping models, the main purpose of facial registration processing is to detect facial regions in the template image and determine multiple facial key points to form the set of facial key points for the second face.

[0117] In this embodiment, facial registration can be achieved as follows: First, facial region detection is performed, which can be done using face detection algorithms, including: a Haar feature classifier (a cascaded classifier based on Haar features, suitable for fast face detection); Histogram of Oriented Gradients (HOG) (detecting faces by calculating the histogram of gradient directions in the image); and deep learning methods (such as Multi-Task Cascaded Convolutional Networks (MTCNN), which achieves high-precision face detection and key point localization through cascaded convolutional neural networks). After facial region detection, facial key point localization is performed. These key points typically include the positions of the eyes, eyebrows, nose, mouth, and facial contours. Methods for facial key point localization include feature-based methods and deep learning methods. Feature-based methods, such as Scale-Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF), locate facial keypoints by extracting feature points from an image. Deep learning methods, on the other hand, use convolutional neural networks to directly predict the locations of keypoints from the image. After locating facial keypoints, a set of facial keypoints is generated. At this point, multiple facial keypoints of the second face in the template image can be obtained, forming a set of facial keypoints.

[0118] In other embodiments, facial registration processing of the template image can also be achieved through a pre-trained facial registration network, which is a deep learning model for facial registration processing. The facial registration network can locate and label multiple facial key points of the second face in the template image, and align the second face in the template image using these multiple facial key points. These facial key points cover the main feature points of the second face, such as the eyes, nose, and mouth, to achieve more refined alignment and matching. For example, the facial registration network can be a 256-point registration network, and the network structure of the 256-point registration network can be based on a convolutional neural network or an encoder-decoder architecture. Here, the input of the facial registration network is the template image, and the output is the coordinates of multiple key points of the second face in the template image. That is, the set of facial key points of the second face is obtained through the pre-trained facial registration network, which includes the coordinates of key points in key positions such as the eyes, nose, and mouth of the second face. The specific network structure of the facial registration network is not limited here.

[0119] The set of key points for the eyes in the second face includes two corner points and two eyelid points for each eye. The two eyelid points refer to the key points located at the top and bottom of the eye in the second face, respectively. The two corner points refer to the key points located at the leftmost and rightmost points of the eye in the second face, respectively. The first key point distance is the vertical distance between the top and bottom key points of the eye in the second face, and the second key point distance is the horizontal distance between the leftmost and rightmost key points of the eye in the second face. The distance ratio is the ratio between the first key point distance and the second key point distance.

[0120] In this embodiment, the opening / closing state of the eyes in the second face is determined based on the distance ratio between the first key point distance and the second key point distance. This can be achieved in the following way: when the distance ratio between the first key point distance and the second key point distance is less than a preset eye-closing threshold, the opening / closing state of the eyes in the second face is determined to be closed; when the distance ratio between the first key point distance and the second key point distance is greater than or equal to the preset eye-closing threshold, the opening / closing state of the eyes in the second face is determined to be open. That is, the preset eye-closing threshold is a pre-set threshold used to measure the opening / closing state of the eyes in the second face. If the distance ratio is less than the preset eye-closing threshold, the eyes in the second face are in a closed state; if the distance ratio is greater than or equal to the preset eye-closing threshold, the eyes in the second face are in an open state. Therefore, the set of eye key points for the second face can be directly extracted from the set of facial key points. Based on the set of eye key points for the second face, the opening and closing state of the eyes in the second face can be determined. This makes it easier to calculate different loss values ​​based on the different opening and closing states of the eyes in the second face. This allows the face-swapping model to better capture the eye features of the template image during the model training process, improve the eye matching degree between the template image and the synthesized facial image, and make the face-swapping effect more natural and realistic.

[0121] In this embodiment, when the eyes in the second face of the template image are in a closed state, the closed-eye loss value is used to measure the closed-eye error of the eyes in the second face and the eyes in the third face. The closed-eye loss value of the synthesized facial image can be determined as follows: determine the set of eye key points of the third face of the third object included in the synthesized facial image; then, from the set of eye key points of the third face, determine two eyelid points of each eye in the third face; for each eye in the third face, the third key point distance between the two eyelid points can be determined; finally, based on the first key point distance and the third key point distance, determine the closed-eye loss value of the synthesized facial image relative to the template image.

[0122] Here, the set of key points for the eyes of the third face is obtained by performing facial registration processing on the synthetic facial image. Facial registration processing refers to determining multiple key points of the third face, i.e., the set of key points for the third face, by detecting the facial regions of the third face in the synthetic facial image.

[0123] In this embodiment, facial registration processing of the synthesized facial image can be achieved through a pre-trained facial registration network. A facial registration network is a deep learning model for facial registration processing. It can locate and label multiple facial key points of the third face in the synthesized facial image, and align the third face in the synthesized facial image using these key points. These key points cover the main feature points of the third face, such as the eyes, nose, and mouth, to achieve finer alignment and matching. For example, the facial registration network can be a 256-point registration network, whose network structure can be based on a convolutional neural network or an encoder-decoder architecture. Here, the input to the facial registration network is the synthesized facial image, and the output is the coordinates of multiple key points of the third face in the synthesized facial image. That is, the set of facial key points of the third face is obtained through the pre-trained facial registration network, which includes the coordinates of key points in key positions such as the eyes, nose, and mouth of the third face. The specific network structure of the facial registration network is not limited here.

[0124] In this embodiment, the set of key points for the eyes of the third face includes two corner points and two eyelid points for each eye in the third face. The two eyelid points refer to the key points located at the uppermost and lowermost points of the eyes in the third face, respectively. The two corner points refer to the key points located at the leftmost and leftmost points of the eyes in the third face, respectively. The first key point distance is the vertical distance between the uppermost and lowermost key points of the eyes in the second face, and the third key point distance is the vertical distance between the uppermost and lowermost key points of the eyes in the third face. The eye-closing loss value determined based on the first and third key point distances can be the absolute difference between the first and third key point distances.

[0125] Here, the eye-closure loss value can be used to determine whether the degree of eye closure in the third face of the synthesized facial image is consistent with the degree of eye closure in the second face of the template image. The process of determining the eye-closure loss value facilitates the continuous updating of the face-swapping model parameters, making the degree of eye closure in the synthesized facial image and the template image increasingly similar, thereby improving the face-swapping effect.

[0126] In step S207, the server calculates the eye-opening loss of the synthesized facial image based on the template image and the synthesized facial image, and obtains the eye-opening loss value of the synthesized facial image relative to the template image.

[0127] In this embodiment of the application, the eye-opening loss value is used to characterize the degree of difference between the eye view of the third object and the eye view of the second object when the eye opening state of the second face is the open state.

[0128] The eye-opening loss value includes a gaze loss value used to characterize the degree of difference in eye gaze direction between the synthesized facial image and the template image. In this embodiment, based on the template image and the synthesized facial image, the eye-opening loss is calculated on the synthesized facial image to obtain the eye-opening loss value of the synthesized facial image relative to the template image. This can be determined in the following way: First, gaze direction features are extracted from both the template image and the synthesized facial image to obtain the gaze direction features of the second face and the third face, respectively. That is, gaze direction features are extracted from the template image to obtain the gaze direction features of the second face, and gaze direction features are extracted from the synthesized facial image to obtain the gaze direction features of the third face. Then, the gaze loss value is determined based on the distance between the gaze direction features of the second face and the gaze direction features of the third face.

[0129] Here, the gaze direction feature of the second face includes the feature value corresponding to the gaze direction of the eyes in the second face of the template image. This gaze direction feature includes the gaze direction of the left eye and the gaze direction of the right eye in the second face. The synthesized face image includes the third face of the third object. The third object is the synthesized object after face-swapping processing, and the third face is the result of face-swapping. The third face integrates the facial features (such as identity information) of the first face in the source image and the background, expression, and pose information in the template image. The synthesized object refers to the complete object obtained after face-swapping processing, and the synthesized object contains the third face. The synthesized object is the final result of face-swapping processing. The synthesized object visually looks like a real person, but it is actually generated by the face-swapping model. The synthesized object retains the background and other non-facial areas of the template image, while the facial areas are replaced with the first face of the first object.

[0130] The gaze direction features of the third face include feature values ​​corresponding to the gaze directions of the eyes in the synthesized facial image's third face. These gaze direction features include the gaze directions of the left and right eyes in the third face. The gaze direction features of the second and third faces are obtained by extracting gaze direction features from the template image and the synthesized facial image respectively. Gaze direction feature extraction refers to estimating the gaze directions of the left and right eyes in the template image's second face, and also estimating the gaze directions of the left and right eyes in the synthesized facial image's third face. The gaze direction feature extraction from the template image and the synthesized facial image can be achieved using a pre-trained gaze direction estimation network; that is, the gaze direction features of the second face are obtained through the pre-trained gaze direction estimation network. The gaze direction estimation network is a computer vision model whose main function is to identify and predict the gaze direction or points of interest of the eyes in the template image's second face and the synthesized facial image's third face. It can accurately distinguish the eye region from different angles, lighting conditions, and facial expressions, extract effective features, and then predict the gaze direction. For example, the view estimation network can be U-Net or a fully convolutional network. No specific restrictions are placed on the network structure of the view estimation network here.

[0131] In this embodiment, the distance between the gaze direction features of the second face and the gaze direction features of the third face refers to the difference between the left eye gaze direction of the second face and the left eye gaze direction of the third face, and the difference between the right eye gaze direction of the second face and the right eye gaze direction of the third face. The gaze loss value is used to characterize the degree of difference between the gaze directions of the eyes in the template image and the synthesized face image, that is, the difference between the gaze directions of the second face and the third face. The gaze loss value can be determined by calculating the Euclidean distance between the gaze direction features of the second face and the third face, that is, calculating the Euclidean distance between the left eye gaze direction of the second face and the left eye gaze direction of the third face, and the Euclidean distance between the right eye gaze direction of the second face and the right eye gaze direction of the third face, and summing these two Euclidean distances as the gaze loss value.

[0132] Here, the gaze direction loss value can be used to determine whether the gaze direction of the eyes in the third face of the synthesized facial image is consistent with the gaze direction of the eyes in the second face of the template image. This process of determining the gaze direction loss value facilitates the continuous updating of the face-swapping model parameters, ensuring that the gaze directions of the third face in the synthesized facial image and the second face in the template image remain consistent, thereby improving the face-swapping effect.

[0133] In this embodiment, when the eyes in the second face are in an open state, the open-eye loss value further includes a pupil position loss value to characterize the degree of difference between the pupil positions in the synthesized facial image and the template image. This pupil position loss value characterizes the degree of difference between the pupil positions of the third object in the synthesized facial image and the pupil positions of the second object in the template image. In other words, the pupil position loss value measures the positional error between the pupil position in the third face of the synthesized facial image and the pupil position in the second face of the template image. The pupil position loss value of the synthesized facial image relative to the template image can be determined as follows: First, determine the first pupil point and the first contour point of each eye in the second face from the set of eye key points of the second face, and determine the second pupil point and the second contour point of each eye in the third face from the set of eye key points of the third face; then, determine the first pupil distance between the first contour point and the first pupil point, and the second pupil distance between the second pupil point and the second contour point; finally, determine the pupil position loss value of the synthesized facial image relative to the template image based on the first pupil distance and the second pupil distance.

[0134] The first pupil point refers to the key point located within the pupil of the eye in the set of key points for the second face. The first pupil point can be the left or right pupil point of the second face, and there is one first pupil point for each eye in the second face. The first contour point refers to the key point located within the eye contour in the set of key points for the second face. The first contour point can be the left or right contour point of the second face, and there are multiple first contour points for each eye in the second face. The second pupil point refers to the key point located within the pupil of the eye in the set of key points for the third face. The second pupil point can be the left or right pupil point of the third face, and there is one second pupil point for each eye in the third face. Similarly, the second contour point refers to the key point located within the eye contour in the set of key points for the third face. The second contour point can be the left or right contour point of the third face, and there are multiple second contour points for each eye in the third face.

[0135] The first pupil distance characterizes the relative position of the pupil in the second face relative to the contour within the eye area. The second pupil distance characterizes the relative position of the pupil in the third face relative to the contour within the eye area. The first pupil distance can be determined by: based on the first contour point and the first pupil point, determining the squared difference between each of the multiple first contour key points and the first pupil point, and then calculating the average of these squared differences. The second pupil distance can be determined by: based on the second contour point and the second pupil point, determining the squared difference between each of the multiple second contour points and the second pupil point, and then calculating the average of these squared differences. The pupil position loss value is the loss value determined based on the difference between the first pupil distance and the second pupil distance when the eyes in the second face are open. The pupil position loss value can be determined by: using the absolute difference between the first pupil distance and the second pupil distance as the pupil position loss value.

[0136] Here, the pupil position loss value determines whether the position of the pupil point in the synthesized facial image matches the position of the pupil point in the template image. This process of determining the pupil position loss value facilitates continuous updating of the face-swapping model parameters, ensuring that the position of the pupil point in the synthesized facial image becomes increasingly consistent with that in the template image, thus improving the face-swapping effect.

[0137] In the process of updating the model parameters in the face-swapping model based on the viewpoint loss value, other types of loss values ​​can be fused on the basis of the viewpoint loss value. That is, other types of loss values ​​are weighted and summed with the viewpoint loss value to obtain the total loss value of the face-swapping model, so as to achieve a comprehensive update of the model parameters in the face-swapping model based on the total loss value.

[0138] In this embodiment, after the face-swapping model to be trained performs face-swapping processing on the template image based on the source image to obtain a synthesized facial image, in addition to extracting features from the template image and the synthesized facial image respectively to obtain image features of the template image and image features of the synthesized facial image, identity features can also be extracted from the source image and the synthesized facial image respectively to obtain the identity features of the first face and the identity features of the third face in the synthesized facial image. Then, the cosine similarity between the identity features of the first face and the identity features of the third face can be determined, thereby determining the identity feature loss value of the synthesized facial image relative to the template image based on the cosine similarity. Finally, based on the viewpoint loss value and the identity feature loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model.

[0139] Here, the synthesized facial image includes a third face of a third object; the image features of the synthesized facial image include the identity features of the third face, which are obtained by extracting identity features from the synthesized facial image. The image features of the source image include the identity features of a first face, which are obtained by extracting identity features from the source image.

[0140] In this embodiment, the extraction of identity features from the source image and the synthesized facial image can be achieved using a pre-trained facial recognition network. The pre-trained facial recognition network obtains the identity features of the first face and the third face. A facial recognition network is a system that uses deep learning technology for face recognition. It can be used to identify and verify the identity of a first object in the source image and a third object in the synthesized facial image. The facial recognition network achieves accurate facial recognition by learning the features and representations of the first face of the first object in the source image and the third face of the third object in the synthesized facial image. For example, the facial recognition network can be VGGFace, FaceNet, DeepFace, OpenFace, and SphereFace, etc. The specific network structure of the facial recognition network is not limited here.

[0141] The identity features of the first face can be the eye features, nose features, and mouth features, etc., and the identity features of the third face can be the eye features, nose features, and mouth features, etc. The identity feature loss value can be determined by calculating the cosine similarity between the identity features of the first face and the third face, and using this cosine similarity as the identity feature loss value. The identity feature loss value is used to measure the closeness between the identity features of the third face in the synthesized face image and the identity features of the first face in the source image. Here, the identity feature loss value can determine whether the identity features of the third face in the synthesized face image are close to the identity features of the first face in the source image. This process of determining the identity feature loss value facilitates the continuous updating of the face-swapping model parameters, ensuring that the identity features of the first object in the source image are incorporated into the synthesized face image, and that the faces of the third object in the synthesized face image and the first object in the source image become increasingly similar.

[0142] Updating the model parameters of the face-swapping model based on viewpoint loss and identity feature loss to obtain the trained face-swapping model involves fusing the identity feature loss into the viewpoint loss. This is achieved by weighting and summing the viewpoint loss and identity feature loss using preset loss weights. The weighted summed loss can be used to continuously update the model parameters of the face-swapping model, ensuring that the viewpoint of the eyes in the third face in the synthesized face image remains consistent with the viewpoint of the second face in the template image, while simultaneously maintaining consistency in the identity features between the third face in the synthesized face image and the first face in the source image. This improves the face-swapping effect.

[0143] In this embodiment, after the face-swapping model to be trained performs face-swapping processing on the template image based on the source image to obtain a synthesized facial image, in addition to extracting features from the template image and the synthesized facial image respectively to obtain image features of the template image and image features of the synthesized facial image, features can also be extracted from the synthesized facial image and the ground truth image at different resolutions to obtain synthesized features of the synthesized facial image at multiple resolutions and ground truth features of the ground truth image at multiple resolutions. Then, the absolute difference between the synthesized features and the ground truth features at each resolution is determined. Thus, based on this absolute difference, the feature-level loss value of the synthesized facial image relative to the template image can be determined. Finally, based on the viewpoint loss value and the feature-level loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model.

[0144] Here, the ground truth image is the expected output of the face-swapping model. It can be an accurate face-swapping result or fusion effect, either manually or automatically annotated. In other words, the ground truth image includes the ground truth face after face-swapping processing of the first face in the source image and the second face in the template image. The ground truth image can serve as a standard for comparing and evaluating the performance of the face-swapping model. The synthetic features of the synthesized face image at multiple resolutions are obtained by extracting features from the synthesized face image at different resolutions; similarly, the ground truth features of the ground truth image at multiple resolutions are obtained by extracting features from the ground truth image at different resolutions. During the feature extraction process at different resolutions for both the synthesized face image and the ground truth image, the number of feature extractions is the same, meaning each feature extraction corresponds to the same resolution. Each synthesized feature at the same resolution corresponds to a ground truth feature.

[0145] In this embodiment, feature extraction at different resolutions for the ground truth image and the synthesized facial image can be achieved using a pre-trained AlexNet network. The pre-trained AlexNet network yields synthesized features and ground truth features at multiple resolutions. For example, the pre-trained AlexNet network may include five convolutional layers and three fully connected layers, with each layer's output corresponding to a feature at a specific resolution. The ground truth features at multiple resolutions can be the features output by the first, third, fifth, and last fully connected layers of the AlexNet network after the ground truth image is input into the pre-trained AlexNet network. Similarly, the synthesized features at multiple resolutions can be the features output by the first, third, fifth, and last fully connected layers of the AlexNet network after the synthesized facial image is input into the pre-trained AlexNet network. The specific network structure of the AlexNet network and the number of resolutions for the synthesized and ground truth features at multiple resolutions are not specifically limited here.

[0146] Here, the feature-level loss value is used to measure how close the third face in the synthesized face image is to the ground face in the ground image, based on the features of the ground image and the synthesized face image at different resolutions. The process of determining the feature-level loss value facilitates the continuous updating of the face-swapping model parameters, ensuring that the features of the third face in the synthesized face image become increasingly similar to the ground face in the ground image at different resolutions.

[0147] Updating the model parameters of a face-swapping model based on viewpoint loss and feature-level loss to obtain the trained face-swapping model involves fusing the feature-level loss into the viewpoint loss. This is achieved by weighting and summing the viewpoint loss and feature-level loss using preset loss weights. The weighted summed loss can be used to continuously update the model parameters, ensuring that the viewpoint of the eyes in the third face in the synthesized face image remains consistent with the second face in the template image. Simultaneously, it ensures that the features of the third face in the synthesized face image remain consistent with the features of the ground truth face in the ground truth image at different resolutions. This improves the face-swapping effect, making the synthesized face images increasingly realistic and natural.

[0148] In this embodiment, after the face-swapping model to be trained performs face-swapping processing on the template image based on the source image to obtain a synthesized facial image, in addition to extracting features from the template image and the synthesized facial image respectively to obtain image features of the template image and image features of the synthesized facial image, pixel features can also be extracted from each pixel of the synthesized facial image and each pixel of the ground truth image to obtain multiple synthesized pixel values ​​and multiple ground truth pixel values. Then, the absolute difference between the synthesized pixel value and the ground truth pixel value of each pixel is determined, thereby determining the reconstruction loss value of the synthesized facial image relative to the template image based on the absolute difference between the synthesized pixel value and the ground truth pixel value. Finally, based on the viewpoint loss value and the reconstruction loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model.

[0149] Here, multiple synthesized pixel values ​​are obtained by extracting pixel features from each pixel of the synthesized facial image; multiple ground truth pixel values ​​are obtained by extracting pixel features from each pixel of the ground truth image. During the pixel feature extraction process for different pixels in the synthesized facial image and the ground truth image, the number of pixel feature extractions is the same, meaning that each pixel feature extraction corresponds to the same pixel, and the synthesized pixel value for each identical pixel corresponds to the ground truth pixel value. The number of ground truth pixel values ​​and synthesized pixel values ​​is consistent with the number of pixels in the ground truth image and the synthesized facial image.

[0150] In this embodiment, pixel feature extraction is performed on the ground truth image and the synthesized facial image, respectively. This can be achieved using a pre-trained pixel recognition network, which yields multiple ground truth pixel values ​​and multiple synthesized pixel values. A pixel recognition network is a neural network capable of classifying or segmenting each pixel in the ground truth image and the synthesized facial image. For example, a pixel recognition network could be U-Net, SegNet, FCN, or PSPNet, etc. The specific network structure of the pixel recognition network is not limited here.

[0151] Here, the reconstruction loss value is used to measure how close the third face in the synthesized face image is to the ground face in the ground image, based on the pixel value of each pixel in the ground image and the ground face image. This process of determining the reconstruction loss value facilitates the continuous updating of the face-swapping model parameters, ensuring that the pixel values ​​of the third face in the synthesized face image become increasingly similar to those of the ground face in the ground image at different pixel points.

[0152] Updating the model parameters of the face-swapping model based on viewpoint loss and reconstruction loss to obtain the trained face-swapping model involves fusing the reconstruction loss value into the viewpoint loss value. This is achieved by weighting and summing the viewpoint loss and reconstruction loss values ​​using preset loss weights. The weighted summed loss value can be used to continuously update the model parameters of the face-swapping model. This ensures that the viewpoint of the eyes in the third face in the synthesized face image remains consistent with the viewpoint of the second face in the template image. Simultaneously, it ensures that the pixel values ​​of the third face in the synthesized face image are consistent with the pixel values ​​of the ground truth face in the ground truth image at different pixel points. This makes the synthesized face image obtained through the face-swapping model increasingly closer to the realism of the ground truth image.

[0153] In this embodiment, after the face-swapping model to be trained performs face-swapping processing on the template image based on the source image to obtain a synthesized facial image, in addition to extracting features from both the template image and the synthesized facial image to obtain image features of the template image and the synthesized facial image respectively, feature mapping can also be performed on the synthesized facial image to obtain a discrimination probability that characterizes the synthesized facial image as a natural image. Then, based on the difference between the discrimination probability of the synthesized facial image and a preset natural image threshold, the generation loss value of the synthesized facial image relative to the template image is determined. Finally, based on the viewpoint loss value and the generation loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model.

[0154] Here, the natural image threshold is a preset value for assuming the synthesized face image is a natural image. The discrimination probability of the synthesized face image is obtained by feature mapping of the synthesized face image. Feature mapping of the synthesized face image can be achieved through a discriminant network, which obtains the discrimination probability corresponding to the synthesized face image. A discriminant network is a neural network used for classification or recognition tasks, which can distinguish whether the synthesized face image is a natural image or a fake image obtained through face swapping. For example, the discriminant network can be a convolutional neural network, a recurrent neural network, a long short-term memory network, or a recurrent neural network. The specific network structure of the discriminant network is not limited here. The discrimination probability of the synthesized face image is a probability value used to determine the authenticity of the synthesized face image. When the discrimination probability of the synthesized face image is greater than the discrimination threshold, the discriminant network determines that the synthesized face image is a fake image obtained through face swapping; otherwise, the discriminant network determines that the synthesized face image is a natural image. The preset natural image threshold is a preset value used to determine that the image is an absolutely natural image, and this natural image threshold can be set to the value 1.

[0155] The generation loss value is used to measure the realism of the synthesized facial image. By determining the generation loss value, the model parameters of the face-swapping model can be continuously updated, making the synthesized facial images obtained by the face-swapping model increasingly realistic and natural.

[0156] Updating the model parameters of the face-swapping model based on viewpoint loss and generation loss to obtain the trained face-swapping model involves fusing the generation loss into the viewpoint loss. This is achieved by weighting and summing the viewpoint loss and generation loss using preset loss weights. The weighted summed loss can be used to continuously update the model parameters of the face-swapping model, ensuring that the viewpoint of the eyes in the synthesized third face image and the second face image in the template image remain consistent, while also maintaining the same level of realism between the synthesized third face image and the natural image, ultimately achieving an effect identical to the natural image.

[0157] In this embodiment, after the face-swapping model to be trained performs face-swapping processing on the template image based on the source image to obtain a synthesized facial image, in addition to extracting features from the template image and the synthesized facial image respectively to obtain image features of the template image and image features of the synthesized facial image, feature mapping can also be performed on the synthesized facial image and the ground truth image to obtain the discrimination probability used to characterize the synthesized facial image as a natural image and the discrimination probability used to characterize the ground truth image as a natural image. Then, a first logarithmic value of the discrimination probability of the ground truth image is determined, and a second logarithmic value of the difference between the discrimination probability of the synthesized facial image and a preset natural image threshold is determined. Based on the first logarithmic value and the second logarithmic value, the discrimination loss value of the synthesized facial image relative to the template image is determined. Finally, based on the viewpoint loss value and the discrimination loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model.

[0158] Here, the discrimination probability of the ground truth image is obtained by performing feature mapping on the ground truth image. Feature mapping of the ground truth image can be implemented using a discriminant network. The discriminant network then yields the discrimination probability corresponding to the ground truth image, which is the probability value used to determine whether the ground truth image is genuine or not. The specific network structure of the discriminant network is not limited here. The discriminant loss value is used to measure the performance of the discriminant network in determining the authenticity of images.

[0159] The process of determining the discriminant loss value facilitates the continuous updating of the face-swapping model's parameters, thereby enhancing its ability to distinguish between real and fake images. Updating the model parameters based on the viewpoint loss value and the discriminant loss value to obtain the trained face-swapping model involves fusing the discriminant loss value into the viewpoint loss value. This is achieved by weighting and summing the viewpoint loss value and the discriminant loss value using preset loss weights. This weighted summed loss value can be used to continuously update the face-swapping model's parameters, ensuring that the viewpoint direction of the eyes in the third face in the synthesized face image and the second face in the template image remains consistent, while simultaneously improving the face-swapping model's ability to distinguish between real and fake images. This results in increasingly realistic and natural synthesized face images obtained through the face-swapping model.

[0160] In step S208, the server updates the model parameters in the face-swapping model based on the viewpoint loss value to obtain the trained face-swapping model.

[0161] In this embodiment of the application, the specific meaning and implementation of the step of updating the model parameters in the face-swapping model based on the viewpoint loss value to obtain the trained face-swapping model can be found in the description of step S104 above, and will not be repeated here.

[0162] In this embodiment, the model parameters of the face-swapping model can be updated using loss values ​​for closed eyes, gaze direction, pupil position, identity features, feature level, reconstruction, generation, and discrimination. For the two cases where the eyes in the second face are in a closed state and an open state, two different loss values ​​need to be determined to update the model parameters of the face-swapping model. These two different loss values ​​are the first target loss value in the closed-eye state and the second target loss value in the open-eye state.

[0163] In this embodiment, when the eyes in the second face are in a closed state, a first target loss value can be determined in the closed-eye state. Based on this first target loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model. Here, the model parameters can be updated in the following way: First, when the eyes in the second face are in a closed-eye state, the closed-eye loss value, identity feature loss value, feature-level loss value, reconstruction loss value, and generation loss value are weighted and summed to obtain a first enhancement loss value. Then, based on the first enhancement loss value, the model parameters in the generation network are updated to obtain the trained generation network. Furthermore, based on the discriminant loss value, the model parameters in the discriminant network are updated to obtain the trained discriminant network. That is, the face-swapping model includes a generation network and a discriminant network, and the model training process of the face-swapping model includes the model training process of the generation network and the model training process of the discriminant network. The first target loss value includes the first enhancement loss value and the discriminant loss value. The first enhancement loss value is obtained by weighted summing of the closed-eye loss value, identity feature loss value, feature-level loss value, reconstruction loss value, and generation loss value. This first enhancement loss value is used to update the model parameters in the generative network during model training. The discriminative loss value is used to update the model parameters in the discriminative network during model training. When the eyes in the second face are in a closed-eye state, the update of the model parameters in the face-swapping model based on the first target loss value does not need to consider the pupil position loss value and gaze loss value corresponding to the open-eye state. When the first target loss value tends to converge or reaches the preset iteration conditions (e.g., number of iterations and iteration duration), the update of the model parameters in the generative and discriminative networks of the face-swapping model stops, thus obtaining the trained face-swapping model.

[0164] In this embodiment, when the eyes in the second face are in an open state, a second target loss value can be determined in the open-eye state. Based on this second target loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model. The model parameter update can be achieved as follows: First, when the eyes in the second face are in an open-eye state, the gaze loss value, pupil position loss value, identity feature loss value, feature-level loss value, reconstruction loss value, and generation loss value are weighted and summed to obtain a second enhancement loss value. Then, the model parameters in the generation network are updated based on the second enhancement loss value to obtain the trained generation network. Furthermore, the model parameters in the discriminant network are updated based on the discriminant loss value to obtain the trained discriminant network. In other words, the second target loss value includes both the second enhancement loss value and the discriminant loss value. The second enhancement loss value is obtained by weighted summing of the gaze loss value, pupil position loss value, identity feature loss value, feature-level loss value, reconstruction loss value, and generation loss value. This second enhancement loss value is used to update the model parameters in the generation network during the model training process. The discriminative loss value is used to update the model parameters in the discriminative network during model training. When the eyes in the second face are in an open state, the closed-eye loss value is not considered during the update of the model parameters in the face-swapping model based on the second target loss value. When the second target loss value converges or reaches the preset iteration conditions (e.g., number of iterations and iteration duration), the update of the model parameters of the generator and discriminator networks in the face-swapping model stops, thus obtaining the trained face-swapping model.

[0165] In step S209, the server sends the trained face-swapping model to the terminal.

[0166] Here, the server sends the trained face-swapping model to the terminal, which may include the model file, dependency libraries and configuration files, as well as pre-trained model and asset files. The model file refers to the trained face-swapping model file, which is the core content and can be the weight file of one or more trained neural network models. Dependency libraries and configuration files, such as onnxruntime-gpu and torch, are also required to run the face-swapping model on the terminal. Pre-trained model and asset files are typically found in the .assets / models folder. Some face-swapping tools require additional pre-trained model and asset files, which also need to be sent from the server to the terminal.

[0167] In this embodiment, the server can send the trained face-swapping model to the terminal via cloud storage services. The trained model file can be uploaded to a cloud storage service, and a download link can be sent to the end user. The user can then download the model file to their local terminal via this link. Alternatively, the server can send the trained face-swapping model to the terminal via command-line tools. If both the server and terminal support command-line operations, the Secure Copy Protocol (SCP) command can be used to directly copy the model file from the server to the terminal. Alternatively, a file transfer protocol can be used to transfer the model file from the server to the terminal device; for example, the file transfer protocol could be File Transfer Protocol (FTP) or Secure File Transfer Protocol (SFTP). The server can also send the trained face-swapping model to the terminal via a web interface. A download link can be provided. A web service can be set up on the server, the model file can be hosted on the web server, and a download link can be provided to the end user. The user can access the download link through a browser and download the model file. Example: A web UI interface allows users to upload images and videos for face-swapping processing, and also allows them to download the trained model via the web interface. The server can also send the trained face-swapping model to the terminal through an Application Programming Interface (API). The API can be customized. If the API is deployed on the server, the terminal can download the model file from the server via an HTTP request.

[0168] Step S210: The terminal outputs the trained face-swapping model.

[0169] In this embodiment, firstly, the source image and template image are input into the face-swapping model to be trained. The face-swapping model then performs face-swapping processing on the template image based on the source image to obtain a synthesized facial image. Next, an eye-closing loss is calculated based on the template image and the synthesized facial image to obtain the eye-closing loss value of the synthesized facial image relative to the template image. Then, an eye-opening loss is calculated based on the template image and the synthesized facial image to obtain the eye-opening loss value of the synthesized facial image relative to the template image. The eye-closing loss value and the eye-opening loss value together constitute the viewpoint loss value. Finally, the model parameters in the face-swapping model are updated based on the viewpoint loss value obtained through loss calculation to obtain the trained face-swapping model. Thus, during the model training process of the face-swapping model, the parameters of the face-swapping model are updated based on the viewpoint loss value obtained through loss calculation. This increases the face-swapping model's attention to the viewpoint in the face image, thereby improving the model performance. This ensures that the viewpoint in the synthesized facial image obtained by face-swapping through the trained face-swapping model is consistent with the viewpoint in the template image, improving the realism of the synthesized facial image.

[0170] Furthermore, during the training process of the face-swapping model, in addition to considering the viewpoint loss value, the loss values ​​for closed eyes, pupil position, identity features, feature level, reconstruction, generation, and discrimination can also be considered simultaneously. This allows the model parameters of the face-swapping model to be updated by integrating the influence of multiple loss values ​​on the model parameters, thereby improving the model performance. As a result, the synthesized facial image obtained by the trained face-swapping model can maintain consistency with the background, expression, angle, and eye state (i.e., open / closed state, gaze direction, and pupil position) of the template image while retaining the identity features of the original image, thus further enhancing the realism of the synthesized facial image.

[0171] In this application embodiment, after obtaining the trained face-swapping model through any of the above embodiments, an image face-swapping method can also be provided. The image face-swapping method provided in this application embodiment can be implemented using the face-swapping model trained by the face-swapping model training method provided in any of the above embodiments. That is, the image face-swapping method provided in this application embodiment can be combined with the above-mentioned face-swapping model training method to obtain a trained face-swapping model, and then use the face-swapping model to implement the image face-swapping method. The image face-swapping method can be executed by an image face-swapping device. This image face-swapping device can be the same electronic device as the electronic device used to implement the face-swapping model training method, or it can be a different electronic device. That is, the image face-swapping device used to implement the image face-swapping method and the face-swapping model training device used to implement the face-swapping model training method can be located in the same electronic device or in different electronic devices. The trained face-swapping model can be used to perform image face-swapping on the face-swapping dataset under the current face-swapping service to obtain the face-swapping image under the current face-swapping service.

[0172] In the implementation of the image face-swapping method, in response to face-swapping requests under the current face-swapping service, a face-swapping dataset can be obtained, which includes source images and template images. Then, the face-swapping dataset is input into a trained face-swapping model, which performs image synthesis processing on the source and template images to obtain the face-swapped image. This image face-swapping method can produce sufficiently realistic face-swapped images to meet the actual face-swapping requirements of the current face-swapping service.

[0173] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0174] This application provides a face-swapping model training method, which can be combined with the face-swapping model training methods and image face-swapping methods in any of the above embodiments. This method is a face-swapping model training method based on viewpoint alignment. To achieve accurate viewpoint alignment while performing face-swapping, it is first necessary to determine whether the template image and the synthesized facial image generated by the face-swapping model are closed. For scenarios where the person in the template image has closed eyes, the person in the generated synthetic facial image must also maintain closed eyes to ensure consistency in the eye closure state before and after face-swapping. For scenarios where the person in the template image has open eyes, viewpoint estimation and 256 face registration point estimation are performed on the eyes, followed by viewpoint optimization, thereby ensuring that the viewpoint of the person in the generated synthetic facial image is consistent with the viewpoint of the person in the original image, better meeting the face-swapping needs of practical applications. As shown in Figure 6, after face-swapping processing of the template image 601 and the source image 602, this method obtains a synthesized facial image 603. This synthesized facial image 603 has good facial feature details and can be used in various application scenarios such as film and television portrait production, game character design, virtual avatars, and privacy protection.

[0175] Before training a face-swapping model, the input image needs to be preprocessed to obtain cropped source and template images. Image preprocessing can be achieved through the following steps: First, since faces often occupy a small area in the input image, face detection is performed to identify the face region. Then, face registration is performed within the face region to obtain key facial points, focusing on the eyes and corners of the mouth. Finally, face alignment is performed based on the key facial points to obtain the cropped source and template images, which are then used as input data for the face-swapping model.

[0176] Simultaneously, this method requires four additional pre-trained network models to assist the face-swapping model in learning. The first network model is a pre-trained face recognition network, which is used to extract the identity features of the face. The dimension of these identity features can be 1024. Since this embodiment of the application needs to ensure that the face in the synthetic face image generated by the face-swapping model is as close as possible to the identity of the face in the source image, this can be constrained by extracting the identity features of the faces in the synthetic face image and the source image. The second network model is a pre-trained AlexNet network, which is used to calculate the perceptual loss (Learned Perceptual Image Patch Similarity, LPIPS) by extracting features from different layers of the image. In deep network models, low-level features can be used to represent low-level features such as lines and colors. For example, convolutional layer 1 outputs features such as edges and spots; high-level features can represent high-level features such as parts. For example, convolutional layer 5 outputs features such as object parts and fully connected layer 8 outputs features such as object type. The similarity of two images can be measured by comparing the features extracted from them by a pre-trained AlexNet network. For example, by extracting features from a synthetic face image and a ground truth image using a pre-trained AlexNet network, features from multiple different layers of the synthetic and ground truth images can be obtained. The similarity between the synthetic and ground truth images can then be measured by comparing these features. The third network model is a pre-trained viewpoint estimation network, as shown in Figure 7. This network can also be used to estimate the gaze direction of the eyes in the image, with the gaze direction divided into left eye gaze direction 701 and right eye gaze direction 702. The fourth network model is a pre-trained 256-point registration network. As shown in Figure 8, this network can accurately locate the contour coordinates and pupil coordinates of the eyes.

[0177] Referring to Figure 9, which is a schematic diagram of the training framework of the face-swapping model provided in the embodiment of this application, when training the face-swapping model, face-swapping data can be prepared first, that is, the input data of the face-swapping model can be prepared. The input data of the face-swapping model includes source image 10, template image 11 and ground truth image triplet pairs.

[0178] The generator network 13 in the face-swapping model can be divided into two parts: encoder 131 and decoder 132. Encoder 131 continuously halves the data dimension of the input data and gradually increases the number of channels through convolutional calculations such as convolutional layers 1310 and 1311. The input data of the face-swapping model can be 512*512*6 (the source image 10 and template image 11 are concatenated as input data, and both the source image 10 and template image 11 have 3 RGB channels). After passing through multiple convolutional layers, the input data is gradually encoded into 256*256*32, 128*128*64, 64*64*128, and finally 32*32*256, ultimately obtaining an intermediate feature (denoted as swap_features) in the latent space.

[0179] Furthermore, the identity features of the source image 10 can be extracted using the pre-trained face recognition network 12 (i.e., the face recognition network mentioned above), to obtain the identity features (denoted as src_id_features) corresponding to the source image (i.e., the identity features of the first face mentioned above). Then, through the adaptive instance normalization layer 1312 (AdaIN), the identity features src_id_features and the intermediate features swap_features corresponding to the source image are fused using formula (4) to obtain the fused features (AdaIN(swap_features,src_id_features)):

[0180] Where σ(src_id_features) represents the standard deviation of src_id_features; μ(src_id_features) represents the mean of src_id_features; σ(swap_features) represents the standard deviation of swap_features; and μ(swap_features) represents the mean of swap_features.

[0181] After the fused features are processed, they are fed into the decoder 132. The decoder mainly performs deconvolution operations through deconvolution layers 1313, 1314, etc., gradually doubling the resolution of the fused features, decoding them step by step to 32*32*256, 64*64*128, 128*128*64, 256*256*32, and so on, until 512*512*3, finally obtaining the face-swapping result, namely the synthesized facial image 14 (denoted as fake).

[0182] After obtaining the synthetic face image 14, the identity features of the synthetic face image 14 are extracted through the pre-trained face recognition network 12 to obtain the identity features (denoted as fake_id_features) corresponding to the synthetic face image 14 (i.e. the identity features of the third face mentioned above).

[0183] At the same time, the pre-trained 256-point registration network 19 will be used to extract 256 face registration points from the template image 11 to obtain the key point set (template_pts256) corresponding to the template image.

[0184] Please refer to Figure 9. You can also perform a closed-eye determination 16 on the eyes in template image 11. Obtain the coordinates of the eye keypoints in the template image from the keypoint set corresponding to the template image (i.e., the eye keypoint set of the second face mentioned above). The logic for closed-eye determination is based on these eye keypoint coordinates, using L2 distance (Euclidean distance) to calculate the distance between the top and bottom points of the eye (denoted as `template_eye_top_bottom`) (i.e., the first keypoint distance mentioned above), and the distance between the left and right corner points of the eye (denoted as `template_eye_len`) (i.e., the second keypoint distance mentioned above). If the ratio of `template_eye_top_bottom` to `template_eye_len` is less than a preset closed-eye threshold, then the eyes in template image 11 are recorded as closed. For example, the closed-eye threshold can be 0.06. When (`template_eye_top_bottom / template_eye_len`) < 0.06, the eyes in template image 11 are recorded as closed. The degree of eye closure (`eye_close`) is obtained through the closed-eye determination. The value of eye_close can be 1 or 0, where 1 means the eyes are closed and 0 means the eyes are open.

[0185] Please refer to Figure 9. Alternatively, a pre-trained 256-point registration network 19 can be used to extract 256 face registration points from the synthesized face image 14 to obtain the key point set corresponding to the synthesized face image 14 (denoted as fake_pts256). From the key point set corresponding to the synthesized face image 14, the coordinates of the eye key points of the synthesized face image 14 (i.e., the eye key point set of the third face mentioned above) can be obtained. Based on the eye key point coordinates, the distance between the top and bottom points of the eyes (denoted as fake_eye_top_bottom) (i.e., the distance of the third key point mentioned above) can be calculated.

[0186] Please refer to Figure 9 for further calculation of the eye-closed loss value 17. When eye_close is 1, it means that the eyes in the template image 11 are closed. The eye-closed loss value can be calculated using formula (5). It is desirable for the degree of eye closure in the synthesized face image 14 and the template image 11 to be as close as possible. eye_close_loss=abs(template_eye_top_bottom-fake_eye_top_bottom) (5)

[0187] Where eye_close_loss represents the loss value when the eyes are closed, and abs represents the absolute value function.

[0188] When eye_close is 0, it means that the eyes in the template image 11 are open. It is desirable that the degree of eye closure in the synthesized face image 14 and the template image 11 is as consistent as possible. This degree of eye closure can be constrained by performing viewpoint prediction and eye point prediction on the synthesized face image 14 and the template image 11.

[0189] Please refer to Figure 9. Viewpoint prediction of template image 11 can also be performed. The viewpoint estimation network 18 can be used to predict the viewpoint of template image 11 to obtain the template viewpoint (denoted as template_gaze) (i.e., the viewpoint features of the second face mentioned above).

[0190] Please refer to Figure 9. Viewpoint prediction can also be performed on the synthetic face image 14. The viewpoint estimation network 18 can be used to predict the viewpoint of the synthetic face image 14 to obtain the synthetic viewpoint (denoted as fake_gaze) (i.e., the viewpoint features of the third face mentioned above).

[0191] After obtaining the template view and the synthesized view, the view loss value can be calculated. Based on template_gaze and fake_gaze, the view loss value can be calculated using formula (6): Gaze_loss=L2(template_gaze-fake_gaze) (6)

[0192] Where Gaze_loss represents the view loss value, and L2() represents the Euclidean distance function.

[0193] After obtaining the keypoint set template_pts256 corresponding to the template image, the coordinates of the eye keypoints of template image 11 can be extracted from template_pts256. That is, in template_pts256, the coordinates of the eye contour (denoted as template_eye_contour) (i.e., the first contour point mentioned above) and the coordinates of the pupil point (denoted as template_pupil) (i.e., the first pupil point mentioned above) in template image 11 are extracted respectively. As shown in Figure 10, keypoints 32 to 56 are the eye contour points, and keypoint 222 is the pupil point.

[0194] After obtaining the key point set fake_pts256 corresponding to the synthesized facial image 14, the coordinates of the eye key points of the synthesized facial image 14 can be extracted from fake_pts256. Using the same steps as the above extraction of the eye key point coordinates of the template image 11, the coordinates of the eye contour (denoted as fake_eye_contour) (i.e. the second contour point mentioned above) and the coordinates of the pupil point (denoted as fake_pupil) (i.e. the second pupil point mentioned above) in the synthesized facial image 14 are extracted.

[0195] In this embodiment, the pupil position loss value can also be calculated. To maintain the consistency of the viewing angle, it is desirable for the position of the pupil point in the eye of the synthesized facial image 14 to be as consistent as possible with the position of the pupil point in the eye of the template image 11. Based on template_eye_contour, template_pupil, fake_eye_contour, and fake_pupil, the pupil position loss value can be calculated using formula (7). eye_pupil_loss= L1(L2(fake_eye_contour-fake_pupil)–L2(template_eye_contour–template_pupil))(7)

[0196] Where eye_pupil_loss represents the pupil position loss value, and L1() represents the calculation of the Manhattan distance function.

[0197] In this embodiment, feature-level loss values ​​can also be calculated, aiming for the synthesized face image 14 to be as close as possible to the ground truth image. Using a pre-trained AlexNet network, features are extracted from the synthesized face image 14 and the ground truth image at different resolutions, resulting in multiple features at different resolutions corresponding to the synthesized face image (denoted as fake_fea1, fake_fea2, fake_fea3, and fake_fea4) (i.e., the synthesized features of the synthesized face image at multiple resolutions), and multiple features at different resolutions corresponding to the ground truth image (denoted as GT_fea1, GT_fea2, GT_fea3, and GT_fea4) (i.e., the ground truth features of the ground truth image at multiple resolutions). Specifically, the expressions for feature extraction from the synthesized face image 14 and the ground truth image at different resolutions are as follows: (8) and (9): fake_fea1, fake_fea2, fake_fea3, fake_fea4 = alexnet_feature(fake) (8)

[0198] And, GT_fea1,GT_fea2,GT_fea3,GT_fea4=alexnet_feature(GT) (9)

[0199] Based on the synthesized features and ground truth features at multiple different resolutions, the feature-level loss value is calculated using formula (10): LPIPS_loss=|fake_fea1-GT_fea1|+|fake_fea2-GT_fea2|+|fake_fea3-GT_fea3|+|fake_fea4-GT_fea4| (10)

[0200] Wherein, LPIPS_loss represents the feature-level loss value.

[0201] In this embodiment, cosine similarity can also be used to calculate the identity feature loss value using formula (11), aiming to make the generated synthetic face image 14 and the source image 10 as similar as possible. ID_loss=1–cosine_similarity(fake_id_features,src_id_features) (11)

[0202] Where ID_loss represents the identity feature loss value, cosine_similarity() represents the cosine similarity function, and the cosine similarity calculation formula between fake_id_features and src_id_features is shown in formula (12).

[0203] Where fake_id_features(i) represents the i-th component of fake_id_features, and src_id_features(i) represents the i-th component of src_id_features.

[0204] In this embodiment, the adversarial loss of the synthesized facial image 14, i.e., the discriminant loss value of the discriminant network 15, can also be calculated. The discriminant network D is used to determine whether the input image is a real image. That is, the synthesized facial image 14 generated by the generator network can be input into the discriminant network 15, and the output discrimination probability can be used to determine whether the synthesized facial image 14 is a real image, thereby determining whether the synthesized facial image 14 output by the face-swapping model is realistic. The discriminant loss value of the overall discriminant network 18 can be calculated based on formula (13), and the parameters of the discriminant network D can be optimized based on the discriminant loss value: D_loss=-logD(GT)-log(1–D(fake)) (13)

[0205] Where D_loss represents the discrimination loss value, log() represents the logarithmic function, D(GT) represents the discrimination probability corresponding to the ground truth image (i.e., the discrimination probability of the ground truth image mentioned above), and D(fake) represents the discrimination probability corresponding to the synthetic face image 14 (i.e., the discrimination probability of the synthetic face image mentioned above).

[0206] In this embodiment of the application, the adversarial loss of the synthesized facial image 14, i.e. the generation loss value G_loss of the generator network 13, can also be calculated based on formula (14). G_loss=log(1–D(fake)) (14)

[0207] In this embodiment, the reconstruction loss value L1_loss of the synthesized face image 14 can also be calculated based on formula (15), with the aim of making the synthesized face image 14 as close as possible to the ground truth image. L1_loss=abs(fake-GT) (15)

[0208] Based on all the loss values ​​calculated above, the loss values ​​related to the generator network 13 and the loss values ​​related to the viewpoint are added together to obtain the total loss value. The network parameters of the generator network 13 are optimized based on the total loss value. The total loss value is different depending on the degree of eye closure in the synthesized face image 14 and the template image 11. When the eyes in the template image 11 are closed, the formula for calculating the total loss value Total_loss1 (i.e., the first target loss value mentioned above) is formula (16); when the eyes in the template image 11 are open, the formula for calculating the total loss value Total_loss2 (i.e., the second target loss value mentioned above) is formula (17). Total_loss1=ω1*L1_loss+ω2*LPIPS_loss+ω3*ID_loss+G_loss+ω4* eye_close_loss (16) Total_loss2=ω5*L1_loss+ω6*LPIPS_loss+ω7*ID_loss+G_loss+ω8* gaze_loss+ω9*eye_pupil_loss (17)

[0209] In formulas (16) and (17), ω1 to ω9 are the set loss weights.

[0210] Furthermore, the overall video face-swapping process can be as follows: video acquisition; image input; face detection; face region cropping; face swapping with viewpoint alignment; and face swapping result display. Among these, face swapping with viewpoint alignment is a method module in this application embodiment. In actual use, the method module of this application embodiment needs to cooperate and interact with other modules. This application embodiment offers the following benefits: when the eyes in the template image are closed, the generated synthetic face image can maintain a closed-eye state. When the eyes in the template image are open, the eye viewpoint in the generated synthetic face image can be kept consistent with the eye viewpoint in the template image, thereby better meeting the face-swapping needs of practical applications.

[0211] It is understood that in the embodiments of this application, if the content involving user information, such as source images, template images, synthetic facial images, ground truth images, and face-swapping models, involves data related to user information or enterprise information, when the embodiments of this application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to obfuscate this information to eliminate the correspondence between this information and the user; and the collection and processing of related data should strictly comply with the requirements of relevant national laws and regulations when applied in practice, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0212] The following description further illustrates the exemplary structure of the face-swapping model training device 455 provided in this application embodiment as a software module. As shown in Figure 3, the software module stored in the face-swapping model training device 455 in the memory 450 may include: an acquisition module 4551, configured to acquire a source image and a template image; the source image includes a first face of a first object, and the template image includes a second face of a second object; a face-swapping processing module 4552, configured to perform face-swapping processing on the template image based on the source image using a face-swapping model to be trained, to obtain a synthesized face image; a loss calculation module 4553, configured to perform loss calculation based on the template image and the synthesized face image, to obtain a viewpoint loss value of the synthesized face image relative to the template image; wherein, the viewpoint loss value is used to characterize the degree of difference between the eye viewpoint of the third object in the synthesized face image and the eye viewpoint of the second object in the template image; and a model parameter update module 4554, configured to update the model parameters in the face-swapping model based on the viewpoint loss value, to obtain a trained face-swapping model.

[0213] In this embodiment of the application, the loss calculation module 4553 is further configured to: perform eye-closing loss calculation on the synthetic face image based on the template image and the synthetic face image to obtain the eye-closing loss value of the synthetic face image relative to the template image; the eye-closing loss value is used to characterize the degree of difference between the eye view of the third object and the eye view of the second object when the eye opening and closing state in the second face is the closed state; and determine the eye-closing loss value as the viewpoint loss value of the synthetic face image.

[0214] In this embodiment of the application, the face-swapping model training device 455 further includes an open / closed state determination module, which is configured to: determine a set of key points for the eyes of the second face based on the template image; determine two corner points and two eyelid points for each eye in the second face from the set of key points for the eyes of the second face; for the first eye in the second face, determine a first key point distance between the two eyelid points and a second key point distance between the two corner points; and determine the open / closed state of the eyes in the second face based on the distance ratio between the first key point distance and the second key point distance.

[0215] In this embodiment of the application, the loss calculation module 4553 is further configured to: determine a set of eye key points of the third face of the third object included in the synthesized facial image; determine two eyelid points of each eye in the third face from the set of eye key points of the third face; determine a third key point distance between the two eyelid points for each eye in the third face; and determine the closed-eye loss value of the synthesized facial image relative to the template image based on the first key point distance and the third key point distance.

[0216] In this embodiment of the application, the loss calculation module 4553 is further configured to: perform eye-opening loss calculation on the synthetic face image based on the template image and the synthetic face image to obtain an eye-opening loss value of the synthetic face image relative to the template image; the eye-opening loss value is used to characterize the degree of difference between the eye view of the third object and the eye view of the second object when the eye opening and closing state in the second face is an open eye state; and determine the eye-opening loss value as the viewpoint loss value of the synthetic face image.

[0217] In this embodiment, the loss calculation module 4553 is further configured to: extract gaze features from the template image to obtain gaze features of the second face; extract gaze features from the synthesized face image to obtain gaze features of the third face; determine the distance between the gaze features of the second face and the gaze features of the third face; determine the gaze loss value of the synthesized face image relative to the template image based on the distance; the gaze loss value is used to characterize the degree of difference between the gaze direction of the third object and the gaze direction of the second object; and determine the gaze loss value as the eye-opening loss value.

[0218] In this embodiment, the loss calculation module 4553 is further configured to: determine a first pupil point and a first contour point for each eye in the second face from the set of eye key points of the second face; determine a second pupil point and a second contour point for each eye in the third face from the set of eye key points of the third face; determine a first pupil distance between the first contour point and the first pupil point, and determine a second pupil distance between the second pupil point and the second contour point; determine a pupil position loss value of the synthesized facial image relative to the template image based on the first pupil distance and the second pupil distance; the pupil position loss value is used to characterize the degree of difference between the pupil position of the third object in the synthesized facial image and the pupil position of the second object in the template image; and determine the pupil position loss as the eye-opening loss value.

[0219] In this embodiment of the application, the model parameter update module 4554 is further configured to: determine the identity feature loss value of the synthesized facial image relative to the template image; and update the model parameters in the face-swapping model based on the viewpoint loss value and the identity feature loss value to obtain the trained face-swapping model.

[0220] In this embodiment of the application, the model parameter update module 4554 is further configured to: extract identity features from the source image to obtain the identity features of the first face; extract identity features from the synthesized face image to obtain the identity features of the third face in the synthesized face image; determine the cosine similarity between the identity features of the first face and the identity features of the third face; and determine the identity feature loss value of the synthesized face image relative to the template image based on the cosine similarity.

[0221] In this embodiment of the application, the model parameter update module 4554 is further configured to: determine the feature level loss value of the synthesized facial image relative to the template image; and update the model parameters in the face-swapping model based on the viewpoint loss value and the feature level loss value to obtain the trained face-swapping model.

[0222] In this embodiment, the model parameter update module 4554 is further configured to: extract features from the synthesized facial image at different resolutions to obtain synthesized features of the synthesized facial image at multiple resolutions; extract features from the ground truth image at different resolutions to obtain ground truth features of the ground truth image at multiple resolutions; the ground truth image includes a ground truth face after face swapping of the first face in the source image and the second face in the template image; determine the absolute difference between the synthesized features and the ground truth features at each resolution; and determine the feature level loss value of the synthesized facial image relative to the template image based on the absolute difference.

[0223] In this embodiment of the application, the model parameter update module 4554 is further configured to: determine the reconstruction loss value of the synthesized facial image relative to the template image; and update the model parameters in the face-swapping model based on the viewpoint loss value and the reconstruction loss value to obtain the trained face-swapping model.

[0224] In this embodiment of the application, the model parameter update module 4554 is further configured to: extract pixel features for each pixel of the synthesized facial image to obtain multiple synthesized pixel values; extract pixel features for each pixel of the ground truth image to obtain multiple ground truth pixel values; determine the absolute difference between the synthesized pixel value and the ground truth pixel value for each pixel; and determine the reconstruction loss value of the synthesized facial image relative to the template image based on the absolute difference.

[0225] In this embodiment of the application, the model parameter update module 4554 is further configured to: determine the generation loss value of the synthesized facial image relative to the template image; and update the model parameters in the face-swapping model based on the viewpoint loss value and the generation loss value to obtain the trained face-swapping model.

[0226] In this embodiment of the application, the model parameter update module 4554 is further configured to: perform feature mapping on the synthesized facial image to obtain a discrimination probability that characterizes the synthesized facial image as a natural image; determine the difference between the discrimination probability and a preset natural image threshold; the natural image threshold is a preset value when the synthesized facial image is a natural image; and determine the generation loss value of the synthesized facial image relative to the template image based on the difference.

[0227] In this embodiment of the application, the model parameter update module 4554 is further configured to: determine the discrimination loss value of the synthesized facial image relative to the template image; and update the model parameters in the face-swapping model based on the viewpoint loss value and the discrimination loss value to obtain the trained face-swapping model.

[0228] In this embodiment, the model parameter update module 4554 is further configured to: perform feature mapping on the synthesized facial image to obtain a discrimination probability characterizing the synthesized facial image as a natural image; perform feature mapping on the ground truth image to obtain a discrimination probability characterizing the ground truth image as a natural image; determine a first logarithmic value of the discrimination probability of the ground truth image, and determine a second logarithmic value of the difference between the discrimination probability of the synthesized facial image and a preset natural image threshold; and determine a discrimination loss value of the synthesized facial image relative to the template image based on the first logarithmic value and the second logarithmic value.

[0229] In this embodiment, the face-swapping model includes at least a generator network, which includes an encoder and a decoder. The face-swapping processing module 4552 is further configured to: encode the source image and the template image using a convolutional module in the encoder to obtain encoded features; extract identity features from the source image to obtain identity features of a first face; fuse the encoded features and the identity features of the first face using a feature fusion layer in the encoder to obtain fused features; and decode the fused features using the decoder to obtain the synthesized face image.

[0230] This application provides an image face-swapping device, comprising: a dataset acquisition module configured to acquire a face-swapping dataset in response to a face-swapping request, the face-swapping dataset including a source image and a template image; and a face-swapping module configured to input the source image and the template image into a trained face-swapping model, and to perform image synthesis processing on the source image and the template image through the trained face-swapping model to obtain a face-swapping image; wherein the trained face-swapping model is trained using the face-swapping model training method provided in this application.

[0231] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0232] This application provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor will execute the face-swapping model training method provided in this application, such as the face-swapping model training method shown in FIG4, or execute the image face-swapping method provided in this application.

[0233] This application provides a computer program product including computer-executable instructions stored in a computer-readable storage medium. An electronic device's processor reads the computer-executable instructions from the computer-readable storage medium and executes the instructions, causing the electronic device to perform the face-swapping model training method described above in this application, or to perform the image face-swapping method provided in this application.

[0234] In the embodiments of this application, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0235] In the embodiments of this application, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0236] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0237] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0238] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A face-swapping model training method, the method being executed by an electronic device, the method comprising: Obtain a source image and a template image; the source image includes a first face of a first object, and the template image includes a second face of a second object; The template image is processed by the face-swapping model to be trained based on the source image to obtain a synthesized facial image. Loss calculation is performed based on the template image and the synthesized facial image to obtain the viewpoint loss value of the synthesized facial image relative to the template image; wherein, the viewpoint loss value is used to characterize the degree of difference between the eye viewpoint of the third object in the synthesized facial image and the eye viewpoint of the second object in the template image; The model parameters in the face-swapping model are updated based on the viewpoint loss value to obtain the trained face-swapping model.

2. The method according to claim 1, wherein, The step of calculating the loss based on the template image and the synthesized facial image to obtain the viewpoint loss value of the synthesized facial image relative to the template image includes: Based on the template image and the synthesized facial image, an eye-closing loss calculation is performed on the synthesized facial image to obtain an eye-closing loss value of the synthesized facial image relative to the template image; the eye-closing loss value is used to characterize the degree of difference between the eye view of the third object and the eye view of the second object when the eye opening and closing state in the second face is the closed state. The eye-closing loss value is determined as the viewpoint loss value of the synthesized facial image.

3. The method according to claim 1 or 2, wherein, Before calculating the eye-closing loss on the synthesized facial image based on the template image and the synthesized facial image, the method further includes: Based on the template image, determine the set of key eye points for the second face; From the set of key eye points of the second face, determine two corner points and two eyelid points for each eye in the second face; For the first eye in the second face, determine the first key point distance between the two eyelid points and the second key point distance between the two corner points; The opening and closing state of the eyes in the second face is determined based on the distance ratio between the first key point distance and the second key point distance.

4. The method according to any one of claims 1 to 3, wherein, The step of calculating the eye-closing loss of the synthesized facial image based on the template image and the synthesized facial image to obtain the eye-closing loss value of the synthesized facial image relative to the template image includes: Determine the set of key eye points of the third face of the third object included in the synthesized facial image. From the set of key eye points of the third face, determine two eyelid points for each eye in the third face; For each eye in the third face, determine the distance between the two eyelid points to the third key point; Based on the distance between the first key point and the distance between the third key point, the eye-closing loss value of the synthesized facial image relative to the template image is determined.

5. The method according to any one of claims 1 to 4, wherein, The step of calculating the loss based on the template image and the synthesized facial image to obtain the viewpoint loss value of the synthesized facial image relative to the template image includes: Based on the template image and the synthesized facial image, an eye-opening loss calculation is performed on the synthesized facial image to obtain an eye-opening loss value of the synthesized facial image relative to the template image; the eye-opening loss value is used to characterize the degree of difference between the eye view of the third object and the eye view of the second object when the eye opening and closing state in the second face is the open eye state. The eye-opening loss value is determined as the viewpoint loss value of the synthesized facial image.

6. The method according to any one of claims 1 to 5, wherein, The step of calculating the eye-opening loss of the synthesized facial image based on the template image and the synthesized facial image to obtain the eye-opening loss value of the synthesized facial image relative to the template image includes: The gaze features of the second face are obtained by extracting gaze features from the template image. The gaze direction features of the synthesized facial image are extracted to obtain the gaze direction features of the third face; Determine the distance between the gaze features of the second face and the gaze features of the third face; Based on the distance, the gaze loss value of the synthesized facial image relative to the template image is determined; the gaze loss value is used to characterize the degree of difference between the gaze direction of the third object and the gaze direction of the second object. The visual loss value is determined as the eye-opening loss value.

7. The method according to any one of claims 1 to 6, wherein, The step of calculating the eye-opening loss of the synthesized facial image based on the template image and the synthesized facial image to obtain the eye-opening loss value of the synthesized facial image relative to the template image includes: Determine the first pupil point and the first contour point of each eye in the second face from the set of key eye points of the second face; The second pupil point and the second contour point of each eye in the third face are determined from the set of key eye points of the third face. Determine the first pupil distance between the first contour point and the first pupil point, and determine the second pupil distance between the second pupil point and the second contour point; Based on the first pupil distance and the second pupil distance, a pupil position loss value is determined for the synthesized facial image relative to the template image; the pupil position loss value is used to characterize the degree of difference between the pupil position of the third object and the pupil position of the second object. The pupil position loss is defined as the eye-opening loss value.

8. The method according to any one of claims 1 to 7, wherein, The step of updating the model parameters in the face-swapping model based on the viewpoint loss value to obtain the trained face-swapping model includes: Determine the identity feature loss value of the synthesized facial image relative to the template image; Based on the viewpoint loss value and the identity feature loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model.

9. The method according to any one of claims 1 to 8, wherein, Determining the identity feature loss value of the synthesized facial image relative to the template image includes: The identity features of the first face are obtained by extracting identity features from the source image; The identity features of the synthesized facial image are extracted to obtain the identity features of the third face in the synthesized facial image; Determine the cosine similarity between the identity features of the first face and the identity features of the third face; Based on the cosine similarity, the identity feature loss value of the synthesized facial image relative to the template image is determined.

10. The method according to any one of claims 1 to 9, wherein, The step of updating the model parameters in the face-swapping model based on the viewpoint loss value to obtain the trained face-swapping model includes: Determine the feature-level loss value of the synthesized facial image relative to the template image; Based on the viewpoint loss value and the feature-level loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model.

11. The method according to any one of claims 1 to 10, wherein, Determining the feature-level loss value of the synthesized facial image relative to the template image includes: Feature extraction is performed on the synthesized facial image at different resolutions to obtain the synthesized features of the synthesized facial image at multiple resolutions; Feature extraction is performed on the ground truth image at different resolutions to obtain ground truth features of the ground truth image at multiple resolutions; the ground truth image includes a ground truth face after face swapping of the first face in the source image and the second face in the template image. Determine the absolute difference between the synthesized features and the ground truth features at each resolution; Based on the absolute difference, the feature-level loss value of the synthesized facial image relative to the template image is determined.

12. The method according to any one of claims 1 to 11, wherein, The step of updating the model parameters in the face-swapping model based on the viewpoint loss value to obtain the trained face-swapping model includes: Determine the reconstruction loss value of the synthesized facial image relative to the template image; Based on the viewpoint loss value and the reconstruction loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model.

13. The method according to any one of claims 1 to 12, wherein, Determining the reconstruction loss value of the synthesized facial image relative to the template image includes: Pixel features are extracted from each pixel of the synthesized facial image to obtain multiple synthesized pixel values; Pixel features are extracted from each pixel of the ground truth image to obtain multiple ground truth pixel values; Determine the absolute difference between the synthesized pixel value and the true pixel value for each pixel; Based on the absolute difference, the reconstruction loss value of the synthesized facial image relative to the template image is determined.

14. The method according to any one of claims 1 to 13, wherein, The step of updating the model parameters in the face-swapping model based on the viewpoint loss value to obtain the trained face-swapping model includes: Determine the generation loss value of the synthesized facial image relative to the template image; Based on the viewpoint loss value and the generation loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model.

15. The method according to any one of claims 1 to 14, wherein, Determining the generation loss value of the synthesized facial image relative to the template image includes: The synthesized facial image is subjected to feature mapping to obtain a discrimination probability that characterizes the synthesized facial image as a natural image; Determine the difference between the discrimination probability and a preset natural image threshold; the natural image threshold is a preset value when the synthesized facial image is a natural image; Based on the difference, the generation loss value of the synthesized facial image relative to the template image is determined.

16. The method according to any one of claims 1 to 15, wherein, The step of updating the model parameters in the face-swapping model based on the viewpoint loss value to obtain the trained face-swapping model includes: Determine the discrimination loss value of the synthesized facial image relative to the template image; Based on the viewpoint loss value and the discrimination loss value, the model parameters in the face-swapping model are updated to obtain the trained face-swapping model.

17. The method according to any one of claims 1 to 16, wherein, Determining the discriminative loss value of the synthesized facial image relative to the template image includes: The synthesized facial image is subjected to feature mapping to obtain a discrimination probability that characterizes the synthesized facial image as a natural image; By performing feature mapping on the ground truth image, a discrimination probability is obtained to characterize the ground truth image as a natural image; A first logarithmic value is used to determine the discrimination probability of the ground truth image, and a second logarithmic value is used to determine the difference between the discrimination probability of the synthesized face image and a preset natural image threshold. Based on the first logarithmic value and the second logarithmic value, the discrimination loss value of the synthesized facial image relative to the template image is determined.

18. The method according to any one of claims 1 to 17, wherein, The face-swapping model includes at least a generative network, which includes an encoder and a decoder; The process of performing face-swapping processing on the template image based on the source image using a face-swapping model to obtain a synthesized facial image includes: The source image and the template image are encoded using the convolution module in the encoder to obtain encoded features; The identity features of the first face are extracted from the source image. The encoder's feature fusion layer fuses the encoded features and the identity features of the first face to obtain fused features. The decoder is used to decode the fused features to obtain the synthesized facial image.

19. An image face-swapping method, the image face-swapping method comprising: In response to a face-swapping request, a face-swapping dataset is obtained, which includes a source image and a template image; The source image and the template image are input into the trained face-swapping model, and the trained face-swapping model is used to perform image synthesis processing on the source image and the template image to obtain a face-swapping image; wherein, the trained face-swapping model is trained using the face-swapping model training method provided in any one of claims 1 to 18.

20. A face-swapping model training device, the face-swapping model training device comprising: The acquisition module is configured to acquire a source image and a template image; the source image includes a first face of a first object, and the template image includes a second face of a second object; The face-swapping processing module is configured to perform face-swapping processing on the template image based on the source image using a face-swapping model to be trained, thereby obtaining a synthesized facial image; The loss calculation module is configured to perform loss calculation based on the template image and the synthesized facial image to obtain the viewpoint loss value of the synthesized facial image relative to the template image; wherein, the viewpoint loss value is used to characterize the degree of difference between the eye viewpoint of the third object in the synthesized facial image and the eye viewpoint of the second object in the template image; The model parameter update module is configured to update the model parameters in the face-swapping model based on the viewpoint loss value, so as to obtain the trained face-swapping model.

21. An image face-swapping device, the image face-swapping device comprising: The dataset acquisition module is configured to acquire a face-swapping dataset in response to a face-swapping request, wherein the face-swapping dataset includes a source image and a template image; A face-swapping module is configured to input the source image and the template image into a trained face-swapping model, and to perform image synthesis processing on the source image and the template image through the trained face-swapping model to obtain a face-swapping image; wherein the trained face-swapping model is trained using the face-swapping model training method provided in any one of claims 1 to 18.

22. An electronic device, comprising: Memory is used to store executable instructions for a computer; The processor, when executing computer-executable instructions stored in the memory, implements the face-swapping model training method according to any one of claims 1 to 18, or implements the image face-swapping method according to claim 19.

23. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the face-swapping model training method according to any one of claims 1 to 18, or implement the image face-swapping method according to claim 19.

24. A computer program product comprising computer-executable instructions stored in a computer-readable storage medium; in, When the processor of the electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, it implements the face-swapping model training method according to any one of claims 1 to 18, or implements the image face-swapping method according to claim 19.

Images