Face changing method, model training method, program product, device, and storage medium

Abstract

Embodiments of the present application provide a face changing method, a model training method, a program product, equipment and a storage medium. In the embodiments of the present application, the method comprises: obtaining a to-be-face-changed image and a target face image; replacing a face region in the to-be-face-changed image with a skeleton image of the face region to obtain a background image; and generating a face-changed image by using a trained image generation model according to the background image and the target face image. The technical scheme provided in the embodiments of the present application can ensure that the expression and identity information of the generated face in the face-changed image are highly similar to the target face. Furthermore, the pose information of the generated face is constrained according to the skeleton image in the background image, and the fusion of the generated face and the background is constrained according to the background information in the background image that is not replaced, which helps to improve the face changing effect and improve the fidelity of the face-changed image.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a face-swapping method, a model training method, a program product, a device, and a storage medium. Background Technology

[0002] With the rapid development of computer technology and artificial intelligence, image face-swapping technology has emerged. Existing image face-swapping technology involves replacing the identity information of the face to be replaced in the image to be swapped with the identity information of the target face in the target image, while preserving the facial expressions (including open eyes, closed eyes, and smiling lips), angles, and other information of the face to be swapped in the original image.

[0003] Taking the image face-swapping model based on GAN (Generative Adversarial Nets) as an example, the approach is to extract the expression features of the face to be replaced in the image to be face-swapping and the identity features of the target face in the target face image based on GAN, and then generate a face-swapping image based on the expression features of the face to be replaced and the identity features of the target face in the target face image.

[0004] However, in some scenarios, users want the identity and expression of the face in the final face-swapped image to be consistent with the target face in the original image. This is impossible to achieve using existing face-swapping technology. Summary of the Invention

[0005] This application provides a face-swapping method, a model training method, a program product, a device, and a storage medium to solve the aforementioned technical problems.

[0006] Firstly, an image face-swapping method is provided, including:

[0007] Obtain the image to be swapped and the target face image;

[0008] The face region in the image to be face-swapped is replaced with the skeletal image of the face region to obtain a background image; the size of the background image is larger than the size of the face region in the image to be face-swapped.

[0009] Based on the background image and the target face image, a face-swapped image is generated using a trained image generation model.

[0010] In other words, in this embodiment, the face region in the background image input to the image generation model is replaced with the skeletal image of that face region. The skeletal image only shows the bone points and the connections between them. That is, the skeletal image can only represent the pose information (i.e., position and posture) of the face to be replaced, and cannot represent the facial expression information or more detailed information of the face to be replaced. In this way, when generating face-swapped images, the image generation model mainly refers to the facial information of the target face image to generate facial content (e.g., expression and identity). This ensures that the expression and identity information of the generated face are highly similar to the target face. Furthermore, constraining the pose information of the generated face based on the skeletal image in the background image and constraining the fusion between the generated face and the background based on the background information that has not been replaced in the background image helps to improve the face-swapping effect and the realism of the face-swapped image.

[0011] Optionally, the size of the background image is larger than the size of the face region in the image to be face-swapped, or in other words, the size of the image to be face-swapped is larger than the size of the face region in the image to be face-swapped. Thus, the image to be face-swapped includes not only the face region but also a background region, which may include: shoulders, hair, the environment in which the person is situated, etc.

[0012] Optionally, the image generation model can be a GAN-based model.

[0013] Optionally, the image generation model can be a diffusion-based model, such as a diffusion model. Specifically, the image generation model can be a stable diffusion-based model. Stable diffusion-based models offer significant advantages in terms of image quality, speed, and cost.

[0014] In one embodiment provided in the first aspect, the stable diffusion model includes: a background image input network, a target face image input network, and a stable diffusion network;

[0015] Based on the background image and the target face image, a face-swapped image is generated using a trained image generation model, including:

[0016] The target face image is input into a network to perform feature encoding on the target face image to obtain face features;

[0017] The background image is input into a network to perform feature encoding on the background image, thereby obtaining background features;

[0018] The facial features and the background features are used as input information for the stable diffusion network.

[0019] Among them, stable diffusion networks are networks based on stable diffusion.

[0020] The stable diffusion network in the stable diffusion model references both the background features of the background image and the facial features of the target face image when generating face-swapped images, which helps to improve the face-swapping effect and provide the realism of the face-swapped images.

[0021] In one embodiment provided in the first aspect, the background image is feature-encoded using the background image input network to obtain background features, including:

[0022] The background image is input to the network to extract features from the background image, thereby obtaining initial background features;

[0023] The background features are obtained by using the cross-attention mechanism of the background image input network to fuse the initial background features and the face features.

[0024] In other words, by pre-fusing the facial features into the background features before inputting them into the stable diffusion network, the realism of the final face-swapped image can be improved.

[0025] In one embodiment provided in the first aspect, generating a face-swapped image using a trained image generation model based on the background image and the target face image further includes:

[0026] The background features and the facial features are fused using the cross-attention mechanism of the stable diffusion network.

[0027] In other words, the cross-attention mechanism in a stable diffusion network is used to fuse background features and facial features.

[0028] The stable diffusion network generates face-swapped images based on the fusion results.

[0029] In one embodiment provided in the first aspect, the background image is feature-encoded using the background image input network to obtain background features, including:

[0030] The background image is input to the network to encode its features, resulting in Q background features at different scales. The scale of the i-th background feature is larger than that of the (i+1)-th background feature. The value of i ranges from 1 to Q-1, and Q is an integer greater than 1.

[0031] The stable diffusion network includes a U-net; the U-net includes a first feature extraction module and Q downsampling stages;

[0032] The background features and facial features are fused using the cross-attention mechanism of the stable diffusion network, including:

[0033] The first feature extraction module is used to extract features from random noise to obtain noise features;

[0034] The input information of the face features and the relevant background features in the i-th downsampling stage is fused using the cross-attention layer in the i-th downsampling stage.

[0035] The input information for the background features in the first downsampling stage is the sum of the noise features and the first background features; the input information for the background features in the (i+1)th downsampling stage is the sum of the output information of the ith downsampling stage and the (i+1)th background features.

[0036] Background features at different scales contain varying levels of detail and semantic information. For example, smaller background feature scales contain less detail but stronger semantic information. Integrating background features at different scales into a stable diffusion network not only provides the network with stronger semantic information but also more detailed background information. This helps improve the image generation performance of the stable diffusion network, thus enhancing the realism of face-swapping images.

[0037] In one embodiment provided in the first aspect, the background image is feature-encoded using the background image input network to obtain Q background features at different scales, including:

[0038] The background image is input to the network to encode its features, resulting in Q background features at different scales and a target background feature; wherein the size of the target background feature is smaller than the size of the Q background features at different scales.

[0039] The U-net also includes a second feature extraction module;

[0040] Based on the background image and the target face image, a face-swapped image is generated using a trained image generation model, including:

[0041] The second feature extraction module is used to extract features from the sum of the output information of the Qth downsampling stage and the target background features.

[0042] In one embodiment provided in the first aspect, the method further includes:

[0043] Obtain the initial image to be swapped;

[0044] Face detection is performed on the initial face-swap image to obtain the face detection box to be replaced;

[0045] Based on the face detection box to be replaced, the initial face-swapping image is cropped to obtain a cropped image; the size of the cropped image is larger than the size of the face detection box to be replaced.

[0046] Based on the cropped image, determine the image to be swapped;

[0047] Based on the position information of the cropped image in the initial face-swappable image, the face-swappable image is fused into the initial face-swappable image to obtain the target face-swappable image.

[0048] In other words, when an image needs face swapping, it is first cropped to extract the image to be swapped, containing the face and surrounding background. Then, subsequent processing is performed based on this image. This process removes most of the background information irrelevant to the face-swapping effect. This not only avoids interference from irrelevant background information with the image generation model but also allows the model to focus more on useful background information, thereby improving the quality of the generated face-swapping image.

[0049] In one embodiment provided in the first aspect, the face-swapped image is fused into the initial face-swapped image based on the position information of the cropped image in the initial face-swapped image to obtain a target face-swapped image, including:

[0050] Based on the position information of the cropped image in the initial face-swappable image, the face-swappable image is fused into the initial face-swappable image using the Poisson fusion algorithm to obtain the target face-swappable image.

[0051] Using the Poisson fusion algorithm to fuse face-swapped images into the initial face-swappable image can improve the fusion effect and reduce color cast.

[0052] In one embodiment provided in the first aspect, the initial face-swap image is cropped according to the face detection box to be replaced to obtain a cropped image, including:

[0053] Adjust the width or height of the face detection box to be replaced to obtain a first square box;

[0054] The first square frame is enlarged proportionally by a preset factor to obtain a second square frame; the preset factor is greater than 1; the first square frame, the second square frame, and the face detection frame to be replaced have the same center point coordinates;

[0055] From the initial image to be swapped, the area enclosed by the second square frame is cropped to obtain the cropped image.

[0056] The preset multiplier can be set according to actual needs, and this application does not impose specific limitations on it.

[0057] Optionally, the preset multiplier is greater than 1 and less than 2. For example, the preset multiplier is 1.6.

[0058] It should be noted that adjusting the width or height of the detection box to obtain a square is to meet the input requirements of the image generation model. Scaling the first square by a preset factor to obtain the second square ensures that the final face-swapped image contains not only the face but also the surrounding background content relevant to the generated face-swapped image.

[0059] In one embodiment provided in the first aspect, the method further includes:

[0060] The skeletal key points of the image to be swapped are detected using a skeletal detection algorithm to obtain the initial skeletal image of the image to be swapped.

[0061] Based on the location information of the face region in the image to be face-swapped, the skeletal image of the face region is cropped from the initial skeletal image.

[0062] Optionally, the skeleton detection algorithm can specifically be the Openpose algorithm.

[0063] In one embodiment provided in the first aspect, the method further includes:

[0064] Acquire a sample image; the sample image contains a sample human face;

[0065] In the sample image, determine the sample face region of the sample face;

[0066] The sample face region is cropped from the sample image to obtain the sample target face image;

[0067] The sample face region in the sample image is replaced with the skeletal image of the sample face region to obtain the sample background image;

[0068] With the goal of minimizing image generation error, the image generation model is trained based on the sample background image, the sample target face image, and the sample image as the ground truth.

[0069] Optionally, the size of the sample image can be larger than the size of the sample face region within the sample image. The sample image can also be obtained based on the initial sample image. The process of obtaining a sample image based on the initial sample image can refer to the process described above for obtaining a face-swappable image based on the initial face-swappable image.

[0070] This solution provides a model training method that trains or optimizes the image generation model using training samples, thereby obtaining an image generation model capable of performing face swapping.

[0071] Secondly, a model training method is provided, including:

[0072] Acquire a sample image; the sample image contains a sample human face;

[0073] In the sample image, determine the sample face region of the sample face;

[0074] The sample face region is cropped from the sample image to obtain the sample target face image;

[0075] The sample face region in the sample image is replaced with the skeletal image of the sample face region to obtain the sample background image;

[0076] With the goal of minimizing image generation error, the image generation model is trained based on the sample background image, the sample target face image, and the sample image as the ground truth.

[0077] This solution provides a model training method that trains or optimizes the image generation model using training samples, thereby obtaining an image generation model capable of performing face swapping.

[0078] In one embodiment provided in the second aspect, the image generation model includes: a background image input network, a target face image input network, and a stable diffusion network;

[0079] With minimizing image generation error as the training objective, the image generation model is trained using the sample background image, the sample target face image, and the sample image used as the ground truth. The training includes:

[0080] The image generation model is trained with the training objective of minimizing the difference between predicted noise and random noise, based on the sample background image, the sample target face image, and the sample image used as the real value.

[0081] The predicted noise is predicted by the stable diffusion network when the image generation model is trained based on the training samples; the random noise is noise randomly added by the stable diffusion network when the image generation model is trained based on the training samples.

[0082] In one embodiment provided in the second aspect, the image generation model is trained with the training objective of minimizing the difference between predicted noise and random noise, based on the sample background image, the sample target face image, and the sample image as the ground truth, including:

[0083] The target face image is input into a network to encode features of the sample target face image, thereby obtaining sample face features;

[0084] The background image is input into the network to encode the features of the sample background image, thereby obtaining the sample background features;

[0085] The sample facial features and the sample background features are used as input information for the stable diffusion network.

[0086] In one embodiment provided in the second aspect, the sample background image is feature-encoded using the background image input network to obtain sample background features, including:

[0087] The background image input network is used to extract features from the sample background image to obtain initial sample background features;

[0088] The cross-attention mechanism of the background image input network is used to fuse the initial sample background features and the sample face features to obtain the sample background features.

[0089] In one embodiment provided in the second aspect, the image generation model is trained based on the sample background image, the sample target face image, and the sample image used as the ground truth, with the training objective being to minimize the difference between predicted noise and random noise. The training further includes:

[0090] The cross-attention mechanism of the stable diffusion network is used to fuse the background features and facial features of the sample.

[0091] In one embodiment provided in the first aspect, the sample background image is feature-encoded using the background image input network to obtain sample background features, including:

[0092] The background image is input to the network to encode the sample background image to obtain Q sample background features at different scales; the scale of the i-th sample background feature is larger than the scale of the (i+1)-th sample background feature; the value of i is from 1 to Q-1; Q is an integer greater than 1.

[0093] The stable diffusion network includes a U-net; the U-net includes a first feature extraction module and Q downsampling stages;

[0094] The cross-attention mechanism of the stable diffusion network is used to fuse the sample background features and the sample face features, including:

[0095] The first feature extraction module is used to extract features from the sum of the sample image features and random noise of the sample image to obtain the sample noise features;

[0096] The input information of the sample face features and the relevant background features of the i-th downsampling stage is fused using the cross-attention layer of the i-th downsampling stage.

[0097] The input information for the background features in the first downsampling stage is the sum of the sample noise features and the background features of the first sample; the input information for the background features in the (i+1)th downsampling stage is the sum of the output information of the ith downsampling stage and the background features of the (i+1)th sample.

[0098] In one embodiment provided in the first aspect, the background image input network is used to perform feature encoding on the sample background image to obtain Q sample background features at different scales, including:

[0099] The background image is input to the network to encode the sample background image to obtain Q sample background features at different scales and target sample background features; wherein the size of the target sample background features is smaller than the size of the Q sample background features at different scales.

[0100] The U-net also includes a second feature extraction module;

[0101] The image generation model is trained with the training objective of minimizing the difference between predicted noise and random noise, based on the sample background image, the sample target face image, and the sample image used as the ground truth. The training also includes:

[0102] The second feature extraction module is used to extract features from the sum of the output information of the Qth downsampling stage and the background features of the target sample.

[0103] Thirdly, an image face-swapping method is provided, including:

[0104] Acquire a first group photo image of multiple people and a second group photo image of the same multiple people;

[0105] The first face image of the target person is cropped from the first group photo image;

[0106] Based on the first face image, a face-swap image is determined; the face-swap image contains the face of the target person.

[0107] From the second group photo image, a second face image of the target person is cropped out as the target face image; the target face image contains the face of the target person.

[0108] The face region of the target person in the image to be face-swapped is replaced with the skeletal image of the face region to obtain the background image;

[0109] Based on the background image and the target face image, a face-swapped image is generated using a trained image generation model;

[0110] Based on the position information of the first face image in the first group photo image, the face-swapped image is merged into the first group photo image.

[0111] In other words, if in a group photo A, person a's facial expression is unpleasant (e.g., closed eyes, unnatural expression), but the facial expressions of the other people in photo A are relatively good, we can find a group photo B where that person's expression is better. Using the face-swapping technology provided in this solution, we can replace the face of that person in photo A with the face of that person in photo B. In this way, we can obtain a more realistic face-swapped group photo.

[0112] Fourthly, a computer program product is provided, comprising a computer program that, when executed by an electronic device, implements the method described in any of the preceding claims.

[0113] Fifthly, an electronic device is provided, comprising: a memory, a processor, and a display screen, wherein,

[0114] The memory is used to store programs;

[0115] The processor, coupled to the memory and the display screen, is used to execute the program stored in the memory to implement the method described above.

[0116] In a sixth aspect, a computer-readable storage medium is provided storing a computer program that, when executed by a computer, enables the implementation of the method described in any of the preceding claims. Attached Figure Description

[0117] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0118] Figure 1 A schematic diagram of the internal framework of an electronic device provided in an exemplary embodiment of this application;

[0119] Figure 2 A schematic diagram of a software layered architecture provided for an exemplary embodiment of this application;

[0120] Figure 3 A schematic flowchart of an image face-swapping method provided as an exemplary embodiment of this application;

[0121] Figure 4 A first set of interface schematic diagrams provided for an exemplary embodiment of this application;

[0122] Figure 5 A second set of interface schematic diagrams provided for an exemplary embodiment of this application;

[0123] Figure 6 Interface illustration provided for an exemplary embodiment of this application Figure 1 ;

[0124] Figure 7 Interface illustration provided for an exemplary embodiment of this application Figure 2 ;

[0125] Figure 8 An image processing flow provided for an exemplary embodiment of this application;

[0126] Figure 9 A first scaled image of a face region filled with black pixels is provided for an exemplary embodiment of this application;

[0127] Figure 10 The face region provided in an exemplary embodiment of this application is replaced with a background image of a skeletal image;

[0128] Figure 11 A schematic diagram illustrating the fusion process between a partial face-swapping image and an initial face-swapping image provided in an exemplary embodiment of this application;

[0129] Figure 12 A schematic diagram of the internal structure and internal processing logic of a stable diffusion model provided for an exemplary embodiment of this application;

[0130] Figure 13 A schematic flowchart of a model training method provided for an exemplary embodiment of this application;

[0131] Figure 14 An example diagram of a face-swapping method for group photos provided as an exemplary embodiment of this application. Detailed Implementation

[0132] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0133] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0134] Depending on the context, the words “if” or “suppose” used here can be interpreted as “when”, “when”, or “in response to”.

[0135] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0136] This application provides a face-swapping method that can be applied to electronic devices. These electronic devices can be mobile phones, personal computers (PCs), tablets, AR (Augmented Reality) devices, VR (Virtual Reality) devices, in-vehicle computers, wearable devices, smart home devices, etc.

[0137] Figure 1 A schematic diagram of the structure of the electronic device 100 is shown.

[0138] Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a user identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0139] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0140] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0141] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0142] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0143] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0144] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini LED, MicroLED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0145] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0146] NPU stands for Neural Network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0147] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses a layered Android system as an example to illustrate the software structure of electronic device 100.

[0148] Figure 2 This is a software structure block diagram of the electronic device 100 according to an embodiment of this application.

[0149] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into three layers: the application layer, the application framework layer, and the kernel layer, from top to bottom. It should be noted that the Android system may also include more or fewer layers; this application does not specifically limit this.

[0150] The application layer can include a series of application packages.

[0151] like Figure 2 As shown, the application package can include applications such as gallery apps, voice assistants, and face-swapping apps.

[0152] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0153] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0154] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0155] The following section will use a mobile phone as an example to introduce the technical solutions provided in the embodiments of this application.

[0156] Figure 3 A schematic flowchart of the face-swapping method provided in this application embodiment is shown. The execution subject of this face-swapping method is a mobile phone. Figure 3 As shown, the method includes the following steps:

[0157] 301. Receive the initial face-swap image input by the user.

[0158] 302. Receive the initial target face image input by the user.

[0159] 303. Determine whether there is a face region in the initial image to be swapped and the initial target face image.

[0160] For example, a face detection algorithm can be used to perform face detection on the initial image to be swapped and the initial target face image to determine whether there is a face region in the initial image to be swapped and the initial target face image.

[0161] For example, an image classification algorithm can be used to classify the initial image to be swapped and the initial target face image to determine whether they belong to the face type image. When the initial image to be swapped belongs to the face type image, it is determined that a face region exists in the initial image to be swapped. When the initial image to be swapped does not belong to the face type image, it is determined that no face region exists in the initial image to be swapped. When the initial target face image belongs to the face type image, it is determined that a face region exists in the initial target face image. When the initial target face image does not belong to the face type image, it is determined that no face region exists in the initial target face image.

[0162] If there is no face region in the initial face-swap image or the initial target face image, proceed to step 304; if there is a face region in both the initial face-swap image and the initial target face image, proceed to steps 305-318 below.

[0163] 304. Prompt the user to input an image with a human face.

[0164] In one feasible approach, when it is determined that there is no face region in the initial face-swap image, the user is prompted to input an initial face-swap image with a face; when it is determined that there is no face region in the initial target face image, the user is prompted to input an initial target face image with a face.

[0165] In another feasible approach, if either the initial image to be swapped or the initial target face image does not contain a face region, the user is prompted to input an initial image to be swapped and an initial target face image containing a face.

[0166] For example, such as Figure 4 As shown, when the phone displays Figure 4 When the interface 40 shown in (a) is in the unlocked state, in response to the user's long press operation on the power button 41 of the phone, the phone displays the voice assistant's response interface 42; the response interface 42 displays the voice pickup animation 420 and the AI ​​control 421; in response to the user's click operation on the AI ​​control 421, the phone displays the voice assistant's recommendation interface 43; the recommendation interface 43 displays multiple function cards; different function cards correspond to different functions; the function cards display function descriptions; among the multiple function cards is a face-swapping function card 431.

[0167] like Figure 5As shown, in response to the user's click on the face-swapping function card 431 on interface 43, the mobile phone displays the voice assistant's dialogue interface 51; the dialogue interface 51 displays the instant message "Please enter the initial face-swapping image", the gallery icon 510, and the camera icon 511; in response to the user's click on the gallery icon 510 on the dialogue interface 51, the mobile phone displays the image selection interface 52; in response to the user's selection and sending operation on the image selection interface 52, the mobile phone displays the dialogue interface 53; the dialogue interface 53 displays the image selected by the user (i.e., the initial face-swapping image) and the instant message "Please enter the initial target face image". The input process of the initial target face image can refer to the input process of the initial face-swapping image, and will not be described in detail here.

[0168] For example, when the initial face-swap image input by the user contains a facial region, the aforementioned dialog interface 53 is displayed; when the initial face-swap image input by the user does not contain a facial region, the following can be displayed: Figure 6 The dialog interface 61 is shown. Dialog interface 61 displays the initial face-swapping image selected by the user and the instant message "Please enter the initial face-swapping image with a human face." The user's subsequent image input process can refer to the above-described input process for the initial face-swapping image, and will not be detailed here.

[0169] In practical applications, the initial image to be swapped and the initial target face image can be images pre-stored in a photo library application, or images taken by the user. For example: Figure 5 As shown in (a), the user can enter the camera shooting interface by clicking the camera icon 511 on the dialog interface 51, and obtain the initial face-swappable image and / or the initial target face image through the camera shooting interface.

[0170] In another feasible solution, such as Figure 7 As shown, in response to the user opening the creation interface of the gallery application, the mobile phone displays the creation interface 71; the creation interface 71 displays multiple functional controls; different functional controls correspond to different functions; among the multiple functional controls is a face-swapping function control 710. The user can enter the face-swapping operation interface by clicking the face-swapping function control 710. The specific style of the face-swapping operation interface can be designed according to actual needs, and this application embodiment does not specifically limit it.

[0171] It should be noted that the rectangular boxes in the interface diagram represent images.

[0172] 305. Perform face detection on the initial image to be swapped to obtain the detection bounding box of the face to be replaced.

[0173] Face detection algorithms can be used to detect faces in the initial image to be swapped, resulting in bounding boxes for the faces to be replaced. These face detection algorithms can be deep learning-based object detection algorithms.

[0174] Typically, the face detection bounding box is rectangular.

[0175] For example, such as Figure 8 As shown, face detection is performed on the initial face-swap image 800 to obtain the face detection box 801 to be replaced. The size of the face detection box 801 to be replaced is 100 (width) * 200 (height).

[0176] 306. Expand the detection box of the face to be replaced along the width direction or along the height direction to obtain the first square box.

[0177] The coordinates of the center point of the first square frame are the same as the coordinates of the center point of the face detection frame. Figure 8 The intersection of two intersecting lines.

[0178] Continuing with the previous example, such as Figure 8 As shown, the width of the face detection box 801 to be replaced is less than its height. Therefore, the face detection box 801 to be replaced needs to be expanded along the width direction to obtain a first square box 802 with a size of 200*200.

[0179] It should be noted that the square bounding box of the face to be replaced is expanded to meet the shape requirements of the background image for the face-swapping model based on the stable diffusion algorithm. Generally speaking, face-swapping models require the input background image to be square.

[0180] Furthermore, if the width of the face detection box to be swapped is equal to its height, step 306 is unnecessary. If the width of the face detection box to be swapped is greater than its height, then the face detection box to be replaced needs to be expanded along the height direction to obtain the first square box.

[0181] 307. Enlarge the first square frame by a factor of 1.6 to obtain the second square frame.

[0182] The coordinates of the center point of the second square frame are the same as the coordinates of the center point of the face detection frame. Figure 8 The intersection of two intersecting lines.

[0183] It should be noted that the first square frame was enlarged proportionally by 1.6 times in order to include useful background information such as the hair, neck, and part of the shoulders around the face within the frame.

[0184] Continuing with the previous example, such as Figure 8As shown, the first square frame 802 is enlarged by 1.6 times to obtain the second square frame 803, wherein the size of the second square frame 803 is 320*320.

[0185] The aforementioned 1.6 times is merely a specific example of a preset multiple. The size of the preset multiple can be set according to actual needs, and this application does not impose any specific limitations on it.

[0186] Optional, the preset multiplier is greater than 1.

[0187] Optional, the preset multiplier is greater than 1.5.

[0188] Optional, the preset multiplier is less than 2.

[0189] 308. From the initial image to be swapped, crop out the area enclosed by the second square frame to obtain the first cropped image.

[0190] Continuing with the previous example, such as Figure 8 As shown, the size of the first cropped image 804 is also the size of the second square frame 803: 320*320.

[0191] 309. Scale the size of the first cropped image proportionally to 512*512 to obtain the image to be swapped.

[0192] Generally, face-swapping models have requirements regarding the size of the input background image. This required size can be referred to as the preset size. The size of the preset size can be set according to actual needs, and this application embodiment does not impose any specific limitations on it.

[0193] Suppose the face-swapping model requires the input background image to be 512*512 pixels. Therefore, as shown... Figure 8 As shown, the size of the first cropped image 804 can be proportionally enlarged to 512*512 to obtain the face-swappable image 805, where the proportional enlargement factor is 512 / 320.

[0194] It should be noted that when the size of the first cropped image is larger than the input size of the background image required by the face-swapping model, the first cropped image needs to be proportionally reduced to the specified size. When the size of the first cropped image is equal to the input size of the background image required by the face-swapping model, no scaling is required.

[0195] 310. Obtain the skeletal image of the face region in the image to be face-swapped.

[0196] Optionally, face detection can be performed on the image to be face-swapped to obtain face detection bounding boxes. The area enclosed by the face detection bounding boxes is the face region.

[0197] Optionally, the face detection box in the scaled face-swapped image can be derived based on the coordinate information of the face detection box to be replaced in the face-swapped image, the coordinate information of the second square box, and the aforementioned proportional scaling factor, thereby determining the face region within it.

[0198] In one embodiment, a skeletal detection algorithm can be used to detect skeletal key points in the image to be swapped, thereby obtaining a skeletal image. For example, the skeletal detection algorithm could be the OpenPose algorithm.

[0199] Generally, the size of the skeletal image is the same as the scaled image to be swapped. The skeletal image of the face region in the image to be swapped can be cropped from the skeletal image according to the face detection box in the image to be swapped.

[0200] 311. Replace the pixels in the face area of ​​the image to be face-swapped with black pixels.

[0201] The face region in the image to be swapped can be the area enclosed by the face detection box in the image to be swapped.

[0202] In other words, the pixels of the face area in the image to be swapped are replaced with the same color, which can be black, white or other colors.

[0203] like Figure 9 As shown, the pixels in the face region of the image 805 to be swapped are replaced with black pixels.

[0204] 312. Fill the black area of ​​the image to be swapped with the skeletal image to obtain the background image.

[0205] The skeletal image of the face region in the image to be face-swapped is filled into the black area of ​​the image to be face-swapped, thus obtaining the background image.

[0206] like Figure 10 As shown, the black area in background image 806 is filled with the skeletal image of a face region.

[0207] The black area is the face area filled with the same color.

[0208] In practical applications, step 311 above can be skipped and the skeletal image can be directly filled into the face area of ​​the image to be swapped by pixel coverage to obtain the background image (that is, the face area in the image to be swapped is directly replaced with the skeletal image of the face area).

[0209] In an alternative implementation, steps 310, 311, and 312 may be performed before step 308. That is, pixel replacement is performed first, followed by size scaling.

[0210] 313. Perform face detection on the initial target face image to obtain the target face detection box.

[0211] The specific implementation of face detection can be found in the relevant content above, and will not be repeated here.

[0212] 314. Expand the target face detection box along the width or height direction to obtain a third square box.

[0213] For details on the specific implementation of the extension, please refer to the relevant content above, which will not be repeated here.

[0214] The purpose of the extension is to meet the shape requirements of the target face image input by the face-swapping model.

[0215] The center coordinates of the third square frame are the same as the center coordinates of the target face detection frame.

[0216] 315. From the initial target face image, crop out the area enclosed by the third square frame to obtain the second cropped image.

[0217] The size of the second cropped image is the same as the size of the third square frame.

[0218] 316. Scale the size of the second cropped image to 224*224 to obtain the target face image.

[0219] The purpose of scaling is to meet the size requirements of the target face image input by the face-swapping model.

[0220] 317. Input the background image and the target face image into the stable augmentation model to obtain the local face-swapped image.

[0221] The size of the face-swapped image is the same as the size of the background image.

[0222] Continuing with the previous example, such as Figure 11 As shown, the size of the partial face-swapped image 1101 is 512*512.

[0223] The specific structure and processing flow of the face-swapping model will be described in detail in the following embodiments.

[0224] 318. Based on the scaling factor used in step 309 and the coordinates of the second square frame, the Poisson fusion algorithm is used to paste the local face-swapped image back into the initial face-swapped image.

[0225] Continuing with the previous example, such as Figure 11 As shown, the local face-swapping image 1101 is scaled down proportionally according to the reciprocal of the aforementioned proportional magnification factor; the scaled-down local face-swapping image, the coordinates of the second square, and the initial face-swapping image 800 are input into the Poisson fusion algorithm to obtain the target face-swapping image 1103 output by the algorithm.

[0226] Using the Poisson fusion algorithm can reduce color cast in the fused image and improve the fusion effect.

[0227] Optionally, the initial image to be face-swapped can be used directly, that is, the initial image to be replaced is not preprocessed.

[0228] Optionally, the initial target face image can be used directly as the target face image, that is, the initial target face image is not preprocessed.

[0229] The following will combine Figure 12 The internal structure and processing logic of the stable diffusion model are described in detail.

[0230] like Figure 12 As shown, the stable diffusion model includes a stable diffusion network 126 and two input branches: a background feature input branch (located in...). Figure 12 On the left side of the middle, the background image is input to the network and the face feature input branch is located in the middle. Figure 12 The right side of the image represents the target face image input network. The background feature input branch processes the background image, while the face feature input branch processes the target face image.

[0231] The facial feature input branch includes an image encoder module 124 and an image projection module 125. The target facial image is processed by image encoding in the image encoder module 124 and image projection in the image projection module 125 to obtain facial features. These facial features are then directly input into the stable diffusion network 126.

[0232] The image encoding module 124 can be the image encoder in the CLIP (Contrastive Language-Image Pre-Training) model.

[0233] For example, the target face image is an RGB image, that is, it has 3 color channels (red, green and blue) and its size is 3*224*224; the image encoding module 124 performs feature encoding (or feature extraction) on the target face image (3*224*224) to obtain a face feature of 257*1280; then the face feature is mapped to a face feature of 16*768 by the image mapping module 1212.

[0234] The background feature input branch includes: an inverse pixel recombination module 121, a convolution module 122, and N sequentially connected (which can be regarded as a series) feature fusion modules 123.

[0235] In this process, the background image is sequentially processed by the inverse pixel recombination module 121 and the convolution module 122 to extract features and obtain initial background features.

[0236] For example, N is set to 4. The four sequentially connected feature fusion modules 123, along the direction from model input to model output, are: the first feature fusion module, the second feature fusion module, the third feature fusion module, and the fourth feature fusion module. The size of the convolution kernel used by the convolution module 122 can be set according to actual needs; this embodiment does not impose a specific limitation on this. For example, the convolution kernel size is 3*3.

[0237] Each feature fusion module involves two inputs: background feature input and face feature input. The feature fusion module is used to fuse these two inputs. Optionally, the feature fusion module can fuse the background feature input and face feature input based on a cross-attention mechanism. By effectively combining background features and face features through the feature fusion module, the fidelity of face generation in the stable diffusion network can be improved. In this paper, background feature input refers to input information about background features, and face feature input refers to input information about face features.

[0238] The facial feature input for each feature fusion module is the same, which is the facial feature obtained through the facial feature input branch. The background feature input for each feature fusion module is different. For example, the output of the i-th feature fusion module is used as the background feature input for the (i+1)-th feature fusion module. Here, i is an integer greater than 0 and less than N. In the above example, N is 4. It should be noted that the number of feature fusion modules can be set according to actual needs, and this embodiment does not specifically limit this. The initial background features output by the convolution module 122 are used as the background feature input for the first feature fusion module.

[0239] like Figure 12As shown, each feature fusion module 123 includes: an adapter block 1231, a reshaping layer 1232, a linear processing layer 1233, M sequentially connected (which can be considered as cascaded) cross-attention networks 1234, a linear processing layer 1235, a normalization layer 1236, and a reshaping layer 1237. For example, M is set to 4. The four sequentially connected cross-attention networks are, along the model input to output direction, the first cross-attention network, the second cross-attention network, the third cross-attention network, and the fourth cross-attention network. Except for the adapter in the first feature fusion module, which does not downsample the background feature input, the adapters in the other feature fusion modules do downsample the background feature input. That is, each feature fusion module has a different output size.

[0240] Each cross-attention network involves two inputs: background feature input and face feature input. The face feature input is the same for each cross-attention network, consisting of the face features processed by linear processing layer 1233. The background feature input is different for each cross-attention network; specifically, the sum of the output of the j-th cross-attention network and its background feature input is used as the background feature input for the (j+1)-th cross-attention network. Here, j is an integer greater than 0 and less than M, where M is the number of cross-attention networks. In the example above, M is 4. It should be noted that the number of cross-attention networks can be set according to actual needs, and this embodiment does not impose a specific limitation on this. The background feature input of the first cross-attention sub-network is the background feature processed by adapter 1231 and reshaping layer 1232.

[0241] Each cross-attention network 1234 includes a cross-attention layer c1 and a feedforward layer c2.

[0242] The processing logic of the cross-attention layer c1 includes: taking the background feature input of its cross-attention network 1234 as Q, taking the face feature input as K and V, and then fusing Q, K, and V using the following formula (1) to obtain the fusion result:

[0243]

[0244] Where, d k Where K is the dimension size, Softmax() is the normalization exponential function.

[0245] Taking the first feature fusion module as an example: the size of the background feature input of the first feature fusion module is 320*64*64; the size of the background feature input after the adapter processing is still 320*64*64, and the size of the background feature input after the reshaping layer processing is 4096*320. The size of the face feature obtained after the face feature input branch is 16*768, and the size of the face feature after passing through the linear processing layer 1233 is still 16*768. That is: the size of Q is 4096*320, the size of K is 16*768, and the size of V is 16*768. It should be noted that before processing Q, K, and V through the above formula (1), Q, K, and V can be processed through their respective linear mapping layer and reshaping layer to obtain 8*4096*40 Q, 8*16*40 K, and 8*16*40 V, and then through formula (1), the size of the fusion result is 4096*320.

[0246] The aforementioned feedforward layer c2 is used to normalize, linearize, and activate the output of the cross-attention layer c1.

[0247] For example, the image to be swapped is an RGB image, that is, it has 3 color channels (red, green and blue) and its size is 3*512*512; the inverse pixel recombination module 1201 performs an inverse pixel recombination operation on the image to be swapped (3*512*512) to obtain a recombined image of 192*64*64; then, the recombined image of 192*64*64 is convolved by the convolution module 1202 to obtain a background feature of 320*64*64.

[0248] In the first feature fusion module, the adapter module 1203 does not downsample the 320*64*64 background feature. Therefore, the size of the background feature output by the first feature fusion network is still 320*64*64.

[0249] The background feature output by the second feature fusion module is 640*32*32; the background feature output by the third feature fusion module is 1280*16*16; and the background feature output by the fourth feature fusion module is 1280*8*8.

[0250] That is, the size of the i-th background feature output by the i-th feature fusion module is greater than the size of the (i+1)-th background feature output by the (i+1)-th feature fusion module. Here, i takes a value greater than 0 and less than N-1. It should be noted that in this embodiment, "size" is the same as the scale mentioned above. It should also be noted that the N-th background feature is the same as the target background feature mentioned above. The 1st to (N-1)th background features are the same as the Q background features mentioned above.

[0251] The stable diffusion network 126 includes the Unet network. The Unet network is the backbone structure of the stable diffusion network. Facial features obtained from the facial feature input branch are added to the face generation process of the Unet network to achieve fidelity.

[0252] The Unet network comprises an encoding network and a decoding network. The encoding network includes a first feature extraction module, N-1 downsampling stages, and a second feature extraction module. For example, N is set to 4, meaning the encoding network includes three downsampling stages: the first downsampling stage, the second downsampling stage, and the third downsampling stage. The first feature extraction module can be a convolutional module.

[0253] The first feature extraction module is used to extract features from the random noise to obtain noise features. It should be noted that the first feature extraction module does not change the width and height of the input information, but it does change the number of channels in the input information. For example, the size of the random noise is 4*64*64, and the noise feature is 320*64*64. The random noise is generated by a random function.

[0254] The noise features output by the first feature extraction module are summed with the first background features output by the first feature fusion module to obtain the first summed feature.

[0255] The first downsampling stage: The first summed feature and the facial feature are fused through a cross-attention mechanism to obtain the first fused feature. The first fused feature is then downsampled to obtain the first downsampled feature.

[0256] For example, the size of the feature after the first downsampling is 640*32*32.

[0257] The first downsampled feature is summed with the second background feature output by the second feature fusion module to obtain the second summed feature.

[0258] The second downsampling stage fuses the second summed feature with the facial feature through a cross-attention mechanism to obtain the second fused feature. This second fused feature is then downsampled to obtain the second downsampled feature.

[0259] For example, the size of the feature after the second downsampling is 1280*16*16.

[0260] The second downsampled feature is summed with the third background feature output by the third feature fusion module to obtain the third summed feature.

[0261] The third downsampling stage fuses the third summed feature with the facial feature through a cross-attention mechanism to obtain the third fused feature. This third fused feature is then downsampled to obtain the third downsampled feature.

[0262] For example, the size of the feature after the third downsampling is 1280*8*8.

[0263] The features obtained after the third downsampling are summed with the fourth background feature output by the fourth feature fusion module to obtain the fourth summed feature. This fourth summed feature serves as input to subsequent network parts in the Unet network. Specifically, it is used as input to the second feature extraction module in the Unet network, which extracts features from the fourth summed feature.

[0264] It should be noted that the noise feature output by the first feature extraction module has the same size as the first background feature output by the first feature fusion module; the feature after the first downsampling has the same size as the second background feature output by the second feature fusion module; the feature after the second downsampling has the same size as the third background feature output by the third feature fusion module; and the feature after the third downsampling has the same size as the fourth background feature output by the fourth feature fusion module.

[0265] The size of the local face-swapped image output by the stable diffusion network is the same as the size of the image to be swapped.

[0266] It should be noted that the number of downsampling stages of Unet in a stable diffusion network can be set according to actual needs, and this application embodiment does not impose specific limitations on this.

[0267] The above embodiments only describe one specific method for incorporating background features into a stable diffusion network. In practical applications, other incorporation methods can be configured according to actual needs. For example, only the background features output by the last feature fusion module are incorporated into the stable diffusion network.

[0268] The following section will introduce a model training method. For example... Figure 13 As shown, the method includes the following steps:

[0269] 1301. Obtain the initial sample image.

[0270] The initial sample image contains sample faces.

[0271] 1302. Perform face detection on the initial sample image to obtain sample face detection boxes.

[0272] 1303. Expand the sample face detection box along the width or height direction to obtain the fourth square box.

[0273] The coordinates of the center point of the fourth square frame are the same as the coordinates of the center point of the sample face detection frame.

[0274] 1304. Enlarge the fourth square by a factor of 1.6 to obtain the fifth square.

[0275] The coordinates of the center point of the fifth square frame are the same as the coordinates of the center point of the sample face detection frame.

[0276] 1305. From the initial image to be swapped, crop out the area enclosed by the fifth square frame to obtain the sample image.

[0277] 1306. Crop the sample face region from the sample image to obtain the sample target face image.

[0278] 1307. Replace the sample face region in the sample image with the skeletal image of the sample face region to obtain the sample background image.

[0279] 1308. With minimizing image generation error as the training objective, the image generation model is trained based on the sample background image, the sample target face image, and the sample image used as the real value.

[0280] For example, the sample background image and the sample target face image are input into the image generation model to obtain the predicted face-swapped image output by the image generation model; the difference between the predicted face-swapped image and the sample image as the ground truth (i.e., the image generation error) is calculated using a loss function, and the parameters to be optimized in the image generation model are optimized with minimizing the difference as the training objective.

[0281] For example, when the image generation model includes a stable diffusion model, the sample background image and the sample target face image are input into the image generation model to obtain the predicted noise generated by the stable diffusion model. The difference between the predicted noise and random noise is calculated using a loss function, and the parameters to be optimized in the stable diffusion model are optimized with minimizing the difference as the training objective. Specifically, only the parameters to be optimized in the stable diffusion model can be optimized. Figure 12 The parameters in the background feature input branch and the parameters in the image mapping module of the face feature input branch are used. The parameters of other parts of the stable diffusion model (e.g., image encoding module 124 and stable diffusion network) can be pre-trained. In this embodiment, the loss function can be mean squared error (MSE).

[0282] The predicted noise is predicted by the stable diffusion network when the image generation model is trained based on the training samples; the random noise is noise randomly added by the stable diffusion network when the image generation model is trained based on the training samples.

[0283] The specific processing flow of the stable diffusion model for the sample background image and the sample target face image can be referred to the processing flow of the stable diffusion model for the background image and the target face image in the above embodiment. The only difference is that the first feature extraction module in the Unet network extracts features from the noisy sample image to obtain sample noise features. The noisy sample image features are obtained by superimposing random noise on the sample image features. The sample image features are obtained by encoding the sample image. For example, the sample image features can be obtained by encoding the sample image using a variational autoencoder (VAE). Note: For distinction, during the model training phase, the obtained face features are called sample face features, the obtained background features are called sample background features, the obtained summed features are called sample summed features, and the obtained fusion features are called sample fusion features.

[0284] Furthermore, if the dimensions of the sample background image, the sample target face image, and the sample image used as the ground truth do not meet the input size requirements of the image generation model, scaling can be used to satisfy the input size requirements of the image generation model.

[0285] For details not fully described in steps 1301 to 1308 above, please refer to the corresponding content in the above embodiments.

[0286] The following will combine Figure 14 This application details the use of the face-swapping method provided in this paper in group photo scenarios.

[0287] like Figure 14 The first group photo image 141 and the second group photo image 142 show multiple people. Only three people are shown in the images. In actual applications, the number of people included in the group photo may be more or less. This application embodiment does not make a specific limitation on this.

[0288] In the first group photo image 141, person A's expression is not good (eyes closed, slight mouth movements are unattractive), while the expressions of the other people are better. In the second group photo image 142, person A's expression is better.

[0289] Therefore, the image generation models provided in the above embodiments can be used to replace the face of person A in the first group photo image 141 with the face of person A in the second group photo image 142, to obtain... Figure 14 The target face-swapped image is 143.

[0290] Specifically, the following steps can be taken to achieve this:

[0291] 1401. Obtain a first group photo image of multiple people and a second group photo image of the multiple people.

[0292] The first and second group photos can be manually entered by the user. The input process can refer to the input process of the initial face-swap image and the initial target face image described above, and will not be described in detail here.

[0293] 1402. From the first group photo image, crop out the first face image of the target person.

[0294] In one feasible approach, the target person can be specified by the user within the first group photo image. For example, in response to a user's click on the pixel region containing the first person in the first group photo image, that person is identified as the target person.

[0295] In another feasible approach, multiple individuals in a first group photo image are identified using a facial recognition algorithm; images of these individuals are provided to the user for selection; and the individual represented by the selected image is identified as the target individual.

[0296] The specific implementation process of cropping the first face image of the target person from the first group photo image can refer to the specific implementation process of cropping the first cropped image from the initial face-swap image. This application embodiment does not specifically limit this process.

[0297] 1403. Based on the first face image, determine the image to be swapped.

[0298] The image to be face-swapped contains the face of the target person.

[0299] For a detailed implementation of step 1403, please refer to the detailed implementation process of obtaining the face-swap image based on the first cropped image.

[0300] 1404. From the second group photo image, crop out the second face image of the target person to serve as the target face image.

[0301] The target face image contains the face of the target person.

[0302] In one feasible approach, the target person can be located in the second group photo image by pixel matching based on the first image of the target person; then, step 1404 is executed.

[0303] In another possible implementation, the target person can be identified in response to a user's click on the pixel area containing the first person in the second group photo image. It should be noted that when the target person in both the first and second group photos is manually specified by the user, it can be determined whether the two specified persons are the same person; if they are the same person, a cropping operation is performed; if they are not the same person, the user is prompted to specify again.

[0304] 1405. Replace the face region of the target person in the image to be face-swapped with the skeletal image of the face region to obtain the background image.

[0305] 1406. Based on the background image and the target face image, generate a face-swapped image using a trained image generation model.

[0306] 1407. Based on the position information of the first face image in the first group photo image, merge the face-swapping image into the first group photo image.

[0307] The specific implementation of steps 1405, 1406 and 1407 above can be found in the corresponding contents of the above embodiments, and will not be repeated here.

[0308] The face-swapping method provided in this application is not only applicable to group photo face-swapping scenarios, but also to scenarios such as ID photo generation and artistic photo generation.

[0309] It should be noted that some of the processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 301, 302, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0310] This application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store a program; the processor is coupled to the memory and is used to execute the program stored in the memory to implement the methods in the above embodiments.

[0311] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer, can implement one or more steps of the method in any of the above embodiments.

[0312] Computer-readable storage media can be non-transitory computer-readable storage media, such as ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices.

[0313] Another embodiment of this application provides a computer program product comprising a computer program. When the computer program is executed by a computer, it can perform one or more steps in any of the above method embodiments.

[0314] In this embodiment, the electronic device, computer-readable storage medium, and computer program product are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0315] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0316] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0317] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0318] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0319] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A face-swapping method for images, characterized in that, include: Obtain the image to be swapped and the target face image; The face region in the image to be face-swapped is replaced with the skeletal image of the face region to obtain a background image. The skeletal image contains the pose information of the face in the image to be face-swapped, but does not contain the expression information or identity information of the face in the image to be face-swapped. Based on the background image and the target face image, a face-swapped image is generated using a trained image generation model. The facial expression and identity information of the face in the face-swapped image are similar to those of the face in the target face image, and the pose information of the face in the face-swapped image is constrained by the skeletal image.

2. The method according to claim 1, characterized in that, The image generation model includes a stable diffusion model.

3. The method according to claim 2, characterized in that, The stable diffusion model includes: a background image input network, a target face image input network, and a stable diffusion network; Based on the background image and the target face image, a face-swapped image is generated using a trained image generation model, including: The target face image is input into a network to perform feature encoding on the target face image to obtain face features; The background image is input into a network to perform feature encoding on the background image, thereby obtaining background features; The facial features and the background features are used as input information for the stable diffusion network.

4. The method according to claim 3, characterized in that, The background image is input to the network to perform feature encoding on the background image to obtain background features, including: The background image is input to the network to extract features from the background image, thereby obtaining initial background features; The background features are obtained by using the cross-attention mechanism of the background image input network to fuse the initial background features and the face features.

5. The method according to claim 3, characterized in that, Based on the background image and the target face image, a face-swapped image is generated using a trained image generation model, further comprising: The background features and the facial features are fused using the cross-attention mechanism of the stable diffusion network.

6. The method according to claim 5, characterized in that, The background image is input to the network to perform feature encoding on the background image to obtain background features, including: The background image is input to the network to encode its features, resulting in Q background features at different scales. The scale of the i-th background feature is larger than that of the (i+1)-th background feature. The value of i ranges from 1 to Q-1, and Q is an integer greater than 1. The stable diffusion network includes a U-net; the U-net includes a first feature extraction module and Q downsampling stages; The background features and facial features are fused using the cross-attention mechanism of the stable diffusion network, including: The first feature extraction module is used to extract features from random noise to obtain noise features; The input information of the face features and the relevant background features in the i-th downsampling stage is fused using the cross-attention layer in the i-th downsampling stage. The input information for the background features in the first downsampling stage is the sum of the noise features and the first background features; the input information for the background features in the (i+1)th downsampling stage is the sum of the output information of the ith downsampling stage and the (i+1)th background features.

7. The method according to claim 1, characterized in that, Also includes: Obtain the initial image to be swapped; Face detection is performed on the initial face-swap image to obtain the face detection box to be replaced; Based on the face detection box to be replaced, the initial face-swap image is cropped to obtain the cropped image; The size of the cropped image is larger than the size of the face detection box to be replaced; Based on the cropped image, determine the image to be swapped; Based on the position information of the cropped image in the initial face-swappable image, the face-swappable image is fused into the initial face-swappable image to obtain the target face-swappable image.

8. The method according to claim 7, characterized in that, Based on the position information of the cropped image in the initial face-swappable image, the face-swappable image is fused into the initial face-swappable image to obtain the target face-swappable image, including: Based on the position information of the cropped image in the initial face-swappable image, the face-swappable image is fused into the initial face-swappable image using the Poisson fusion algorithm to obtain the target face-swappable image.

9. The method according to claim 7, characterized in that, Based on the detected face to be replaced, the initial face-swap image is cropped to obtain a cropped image, including: Adjust the width or height of the face detection box to be replaced to obtain a first square box; The first square frame is enlarged proportionally by a preset factor to obtain a second square frame; the preset factor is greater than 1; the first square frame, the second square frame, and the face detection frame to be replaced have the same center point coordinates; From the initial image to be swapped, the area enclosed by the second square frame is cropped to obtain the cropped image.

10. The method according to claim 1, characterized in that, Also includes: The skeletal key points of the image to be swapped are detected using a skeletal detection algorithm to obtain the initial skeletal image of the image to be swapped. Based on the location information of the face region in the image to be face-swapped, the skeletal image of the face region is cropped from the initial skeletal image.

11. The method according to claim 1, characterized in that, Also includes: Acquire a sample image; the sample image contains a sample human face; In the sample image, determine the sample face region of the sample face; The sample face region is cropped from the sample image to obtain the sample target face image; The sample face region in the sample image is replaced with the skeletal image of the sample face region to obtain the sample background image; With the goal of minimizing image generation error, the image generation model is trained based on the sample background image, the sample target face image, and the sample image as the ground truth.

12. A model training method, characterized in that, include: Acquire a sample image; the sample image contains a sample human face; In the sample image, determine the sample face region of the sample face; The sample face region is cropped from the sample image to obtain the sample target face image; The sample face region in the sample image is replaced with the skeletal image of the sample face region to obtain the sample background image. The skeletal image contains the pose information of the face in the sample image, but does not contain the expression information or identity information of the face in the sample image. With minimizing image generation error as the training objective, the image generation model is trained based on the sample background image, the sample target face image, and the sample image as the ground truth. This includes: generating a face-swapped image using the image generation model based on the sample background image and the sample target face image. The facial expression and identity information of the face in the face-swapped image are similar to those of the face in the target face image, and the pose information of the face in the face-swapped image is constrained by the skeletal image.

13. The method according to claim 12, characterized in that, The image generation model includes a stable diffusion model.

14. The method according to claim 13, characterized in that, The stable diffusion model includes: a background image input network, a target face image input network, and a stable diffusion network; With minimizing image generation error as the training objective, the image generation model is trained using the sample background image, the sample target face image, and the sample image used as the ground truth. The training includes: The image generation model is trained with the training objective of minimizing the difference between predicted noise and random noise, based on the sample background image, the sample target face image, and the sample image used as the real value. The predicted noise is predicted by the stable diffusion network during the training of the image generation model; the random noise is noise randomly added by the stable diffusion network during the training of the image generation model.

15. The method according to claim 14, characterized in that, The image generation model is trained with the training objective of minimizing the difference between predicted noise and random noise, using the sample background image, the sample target face image, and the sample image used as the ground truth. The training includes: The target face image is input into a network to encode features of the sample target face image, thereby obtaining sample face features; The background image is input into the network to encode the features of the sample background image, thereby obtaining the sample background features; The sample facial features and the sample background features are used as input information for the stable diffusion network.

16. The method according to claim 15, characterized in that, The background image is input to the network to encode the sample background image to obtain sample background features, including: The background image input network is used to extract features from the sample background image to obtain initial sample background features; The cross-attention mechanism of the background image input network is used to fuse the initial sample background features and the sample face features to obtain the sample background features.

17. The method according to claim 15, characterized in that, The image generation model is trained with the training objective of minimizing the difference between predicted noise and random noise, based on the sample background image, the sample target face image, and the sample image used as the ground truth. The training also includes: The cross-attention mechanism of the stable diffusion network is used to fuse the background features and facial features of the sample.

18. The method according to claim 17, characterized in that, The background image is input to the network to encode the sample background image to obtain sample background features, including: The background image is input to the network to encode the sample background image to obtain Q sample background features at different scales; the scale of the i-th sample background feature is larger than the scale of the (i+1)-th sample background feature; the value of i is from 1 to Q-1; Q is an integer greater than 1. The stable diffusion network includes a U-net; the U-net includes a first feature extraction module and Q downsampling stages; The cross-attention mechanism of the stable diffusion network is used to fuse the sample background features and the sample face features, including: The first feature extraction module is used to extract features from the sum of the sample image features and random noise of the sample image to obtain the sample noise features; The input information of the sample face features and the relevant background features of the i-th downsampling stage is fused using the cross-attention layer of the i-th downsampling stage. The input information for the background features in the first downsampling stage is the sum of the sample noise features and the background features of the first sample; the input information for the background features in the (i+1)th downsampling stage is the sum of the output information of the ith downsampling stage and the background features of the (i+1)th sample.

19. A face-swapping method for images, characterized in that, include: Acquire a first group photo image of multiple people and a second group photo image of the same multiple people; The first face image of the target person is cropped from the first group photo image; Based on the first face image, a face-swap image is determined; the face-swap image contains the face of the target person. From the second group photo image, a second face image of the target person is cropped out as the target face image; the target face image contains the face of the target person. The face region of the target person in the face-swapped image is replaced with the skeletal image of the face region to obtain the background image. The skeletal image contains the pose information of the target face in the face-swapped image, but does not contain the expression information or identity information of the target face in the face-swapped image. Based on the background image and the target face image, a face-swapping image is generated using a trained image generation model. The facial expression and identity information of the face in the face-swapping image are similar to those of the face in the target face image, and the pose information of the face in the face-swapping image is constrained by the skeletal image. Based on the position information of the first face image in the first group photo image, the face-swapped image is merged into the first group photo image.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by an electronic device, it implements the method described in any one of claims 1 to 19.

21. An electronic device, characterized in that, include: Memory, processor, and display screen, among which, The memory is used to store programs; The processor, coupled to the memory and the display screen, is used to execute the program stored in the memory to implement the method of any one of claims 1 to 19.

22. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a computer, it can implement the method of any one of claims 1 to 19.

Images