Face Swapping Method Based on Segmentation and Phased Diffusion Model

Abstract

The present application discloses a head and face swapping method based on a segmentation and phased diffusion model, including: recognizing a to-be-replaced image through a human body parsing model and a face parsing model to obtain the face region of the to-be-replaced image and the mask image of the face region, cropping the face region of the to-be-replaced image to obtain a face image containing the face part; according to the mask image of the face region, redrawing the face image multiple times through the diffusion model to obtain a target image containing the target face part, wherein a preset redrawing amplitude is adjusted during each redrawing, and the diffusion model includes the lora weight corresponding to the target face part; fusing the target face part of the target image with the face region of the to-be-replaced image to obtain a replaced image. The present application can store the face information of the target face part in the form of lora weights. When it is necessary to replace a specific target face, the diffusion model can directly load the corresponding lora weights, thereby improving the face swapping efficiency.

Description

Technical Field

[0001] This application relates to the technical field of image redrawing, and particularly to a head and face swapping method based on segmentation and a staged diffusion model. Background Art

[0002] Existing head and face swapping technologies mainly use generative adversarial networks (GANs) to generate global images. The generator outputs fake images, and the discriminator discriminates the real probability of the fake images and the input images. Based on adversarial training, the generator and the discriminator are trained, and finally the generator is used to generate face-swapped images.

[0003] However, the face-swapped images generated by generative adversarial networks often have problems such as blurred textures and distorted structures. At the same time, generating high-definition images (such as 1024x1024) requires extremely deep networks and a large amount of video memory, and the training time is long. Summary of the Invention

[0004] The purpose of this application is to provide a head and face swapping method based on segmentation and a staged diffusion model to solve the technical problems of low face swapping efficiency and accuracy in existing face swapping methods. The many technical effects that can be produced by the preferred technical solutions provided in this application are described in detail below.

[0005] To achieve the above purpose, this application provides the following technical solutions:

[0006] In a first aspect, a head and face swapping method based on segmentation and a staged diffusion model provided by this application includes: identifying a to-be-replaced image through a human body parsing model and a face parsing model to obtain the face area of the to-be-replaced image and the mask image of the face area, cropping the face area of the to-be-replaced image to obtain a face image containing the face part; according to the mask image of the face area, redrawing the face image multiple times through a diffusion model to obtain a target image containing the target face part, where a preset redrawing amplitude is adjusted each time of redrawing, and the diffusion model includes the lora (Low-Rank Adaptation) weight corresponding to the target face part; fusing the target face part of the target image with the face area of the to-be-replaced image to obtain a replaced image.

[0007] In some embodiments, the head and face swapping method based on segmentation and a staged diffusion model further includes: obtaining any target model image, and training the lora adapter of the diffusion model with the target model image to obtain the lora weight corresponding to the target face part.

[0008] In some embodiments, training the LoRA adapter of the diffusion model using the target model image includes: fine-tuning the linear layer in the attention layer of the diffusion model, decomposing the original large matrix parameters of the diffusion model into two low-rank matrices, and updating the two low-rank matrices based on the loss value of the target model image to obtain the LoRA adapter of the diffusion model.

[0009] In some embodiments, redrawing the face image multiple times through the diffusion model according to the mask map of the face region includes: adding random Gaussian noise and original image noise to the face image to obtain a noise image, and then denoising the noise image in combination with the LoRA weights to obtain an initial denoised image; resetting the noise of the initial denoised image and then performing denoising in combination with the LoRA weights, repeating the noise reset operation and the denoising operation multiple times until the target image is obtained.

[0010] In some embodiments, the face swapping method based on the segmentation and phased diffusion model further includes: cropping the mask map of the face region to obtain a face swapping frame, and using the face swapping frame as the face region of the image to be replaced, where the face swapping frame is the largest square frame in the mask map of the face region.

[0011] In some embodiments, fusing the target face part of the target image with the face region of the image to be replaced includes: obtaining the weight matrix of the face swapping frame, and performing weighted mixing of the weight matrix with the target face part of the target image and the face region of the image to be replaced respectively, where the weights of the weight matrix decrease gradually from the center point to the edge of the face swapping frame.

[0012] In some embodiments, identifying the image to be replaced through the human body parsing model and the face parsing model includes: identifying the head region of the person in the image to be replaced through the human body parsing model, and identifying the head region through the face parsing model to remove the hair part and the accessory part of the head region.

[0013] In a second aspect, a computer-readable storage medium provided by the present application stores a computer program thereon, and when the computer program is executed, it implements the face swapping method based on the segmentation and phased diffusion model as described above.

[0014] In a third aspect, a processing device provided by the present application includes: one or more processors; a memory for storing one or more computer programs, and one or more of the processors are configured to execute the one or more computer programs stored in the memory, so that the one or more processors execute the face swapping method based on the segmentation and phased diffusion model as described above.

[0015] In a fourth aspect, a computer program product provided by the present application is stored on a data carrier and is designed to execute the face swapping method based on the segmentation and phased diffusion model as described above.

[0016] Implementing one of the technical solutions in the above technical solutions of the present application has the following advantages or beneficial effects: In the present application, the face area of the image to be replaced is recognized and cropped, and the face image is redrawn multiple times through the diffusion model. The preset redrawing amplitude is adjusted each time of redrawing. At the same time, the diffusion model has lora weights corresponding to the target face part. Finally, an image with the target face replaced is obtained. In this case, the face information of the target face part can be stored in the form of lora weights. When it is necessary to replace a specific target face, the diffusion model can directly and timely load the corresponding lora weights, thereby improving the face swapping efficiency; at the same time, in the present application, by redrawing multiple times and adjusting the preset redrawing amplitude, the control of face swapping details can be strengthened, and the loss rate of key areas of the face part can be reduced; in addition, in the present application, local processing is performed through cropping, which can improve the processing speed of high-precision images; in addition, through multiple redrawings and background fusion, the transition of the face boundary can be made more natural. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0018] Figure 1 is a flowchart of a face swapping method based on the segmentation and phased diffusion model according to an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a mask map of the face area according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of the target face part according to an embodiment of the present application;

[0021] Figure 4A is a schematic diagram of an image to be replaced according to an embodiment of the present application, Figure 4BIt is a schematic diagram of the replaced image in the embodiment of the present application;

[0022] Figure 5A It is another schematic diagram of the image to be replaced in the embodiment of the present application, Figure 5B It is another schematic diagram of the replaced image in the embodiment of the present application;

[0023] Figure 6 It is a structural block diagram of the processing device in the embodiment of the present application.

[0024] In the figure: 1. Processing device; 10. Memory; 11. Processor. Detailed implementation manners

[0025] In order to make the purpose, technical solutions and advantages of the present application clearer, various exemplary embodiments to be described below will refer to the corresponding drawings, which form a part of the exemplary embodiments and describe various exemplary embodiments that may be adopted to implement the present application. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. It should be understood that they are only examples of processes, methods, devices, etc. consistent with some aspects of the present application disclosed in detail in the appended claims, and other embodiments may also be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and essence of the present application.

[0026] In the description of the present application, it should be understood that terms such as "center", "longitudinal", "transverse", etc. indicate the orientation or positional relationship based on the orientation or position shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the elements referred to must have a specific orientation, be constructed and operated in a specific orientation. Terms such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. The meaning of the term "plurality" is two or more. The terms "connected" and "connected" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and may be the communication inside two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more of the related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0027] In order to illustrate the technical solutions described in the present application, the following will be described by specific examples, and only the parts related to the embodiments of the present application are shown.

[0028] Such as Figure 1As shown, the present application provides a head and face swapping method based on a segmentation and phased diffusion model, including the following steps (step S1 to step S3):

[0029] S1. Identify the face region of the image to be replaced and the mask image of the face region through a human body parsing model and a face parsing model, and crop the face region of the image to be replaced to obtain a face image containing the face part.

[0030] In some embodiments, identifying the image to be replaced through a human body parsing model and a face parsing model may include: identifying the head region of the person in the image to be replaced through the human body parsing model, and identifying the head region through the face parsing model to remove the hair part and ornament part of the head region.

[0031] In some embodiments, the face parsing model and the human body parsing model can be trained using a public dataset. For example, for the face parsing model, the public dataset can be the CelebAMask-HQ dataset or the Helen dataset, and for the human body parsing model, the public dataset can be the LIP dataset or the PASCAL-Person-Part dataset. The face parsing model and the human body parsing model can also be trained using some publicly uploaded images on the Internet.

[0032] In some embodiments, the human body parsing model can be used to segment a human body image into different semantic regions, such as the head, upper limbs, lower limbs, clothing, etc.; the face parsing model can be used to segment a face image into multiple semantic regions, such as skin, eyes, lips, hair, etc. Further, through the cooperation between the face parsing model and the human body parsing model, the face part of the image to be replaced can be identified and located, and the occluders on the face can be identified. Figure 2 The figure shows the mask image of the face region in the embodiment of the present application.

[0033] In some embodiments, the face parsing model and the human body parsing model can be built based on the mask2former framework or the U-Net framework. The Mask2Former framework is a general image segmentation framework based on the Transformer (global attention) architecture, which supports panoramic, instance, and semantic segmentation tasks, and can uniformly process various segmentation tasks, especially object cutting in complex scenes; the U-Net framework is also an image segmentation framework, which includes an encoder and a decoder. The encoder can extract multi-scale features through convolution and downsampling, and the decoder can restore the spatial resolution through upsampling and convolution, and combine skip connections to fuse the high-resolution details of the encoder and the semantic information of the decoder.

[0034] In some embodiments, the face swapping method based on the segmentation and phased diffusion model may further include: cropping the mask map of the face region to obtain a face swapping frame, and using the face swapping frame as the face region of the image to be replaced, where the face swapping frame is the largest square frame in the mask map of the face region. By further cropping to obtain the face swapping frame, the actual scope of face swapping can be reduced, the computing power required for face swapping can be saved, and the face swapping efficiency can be improved.

[0035] S2. According to the mask map of the face region, the face image is redrawn multiple times through the diffusion model to obtain a target image containing the target face part, where the preset redrawing amplitude is adjusted each time of redrawing, and the diffusion model includes the lora weight corresponding to the target face part.

[0036] In some embodiments, the face swapping method based on the segmentation and phased diffusion model may further include: obtaining any target model image, and training the lora adapter of the diffusion model by using the target model image to obtain the lora weight corresponding to the target face part. Figure 3 What is shown is the target face part of the present application.

[0037] Specifically, the diffusion model can be used for image generation, that is, it can be used to generate the image of the target face part. Lora can be used to fine-tune the parameters of the diffusion model, that is, the diffusion model can generate an image conforming to the target face part according to the lora weight.

[0038] In some embodiments, training the lora adapter of the diffusion model by using the target model image may include: fine-tuning the linear layer in the attention layer of the diffusion model, decomposing the original large matrix parameters of the diffusion model into two low-rank matrices, updating the two low-rank matrices based on the loss value of the target model image, and obtaining the lora adapter of the diffusion model. Specifically, the diffusion model can insert the above two low-rank matrices in the target layer, fix the original parameters of the diffusion model, and fine-tune the two low-rank matrices to obtain the lora adapter. The lora adapter can be independent, that is, the target model image can correspond to the lora adapter, and the lora adapter can be used to store the face information of the target model image, that is, the target face part. The diffusion model can generate different target face parts by loading different lora adapters.

[0039] In some embodiments, repeatedly redrawing a face image through a diffusion model based on a mask image of a face region may include: adding random Gaussian noise and original image noise to the face image to obtain a noise image, and then denoising the noise image in combination with LoRA weights to obtain an initial denoised image; resetting the noise of the initial denoised image and then performing denoising in combination with LoRA weights, repeating the noise reset operation and the denoising operation multiple times until a target image is obtained. Specifically, the redrawing amplitude can be used to control the intensity of image restoration. For example, when the redrawing amplitude is equal to 1, the face image can be completely replaced, and when the redrawing amplitude is equal to 0.5, some information of the face image can be retained.

[0040] Specifically, the random Gaussian noise and the original image noise can be composed by weighting, and the original image noise can be obtained by calculating the face image. The process of adding noise can refer to the forward diffusion process of the diffusion model, which can generate a noise image corresponding to the time step t. The denoising process can refer to the reverse sampling process of the diffusion model, which can restore the noise image to a face image. By combining LoRA weights, it can be restored to an image including the target face part until a target image meeting the requirements is obtained.

[0041] S3. Fuse the target face part of the target image with the face region of the image to be replaced to obtain a replaced image. The replaced image can refer to the final image in which the target face part has been replaced to the face region of the image to be replaced.

[0042] In some embodiments, fusing the target face part of the target image with the face region of the image to be replaced may include: obtaining a weight matrix of a face replacement frame, and performing weighted mixing of the weight matrix with the target face part of the target image and the face region of the image to be replaced respectively, where the weights of the weight matrix decrease gradually from the center point to the edge of the face replacement frame.

[0043] Specifically, the weight of the center point of the face replacement frame can be 1, and the weight of the edge of the face replacement frame can be 0. The distance from each point of the face replacement frame to the edge can be calculated, the distance can be normalized, and finally the normalized distance can be converted into a weight value by using a linear function or a cosine function. This can enable the target face part to gradually transition to the background part and make the replaced image more natural.

[0044] Figure 4A 、 Figure 4B The following shows a set of images to be replaced and replaced images of the present application. Figure 5A 、 Figure 5B The following shows another set of images to be replaced and replaced images of the present application. Both sets of images use Figure 3 as the target face part to be replaced. It should be noted that the Figure 3 、 Figure 4A 、Figure 4B , Figure 5A and Figure 5B the characters and human faces in are computer-simulated and not real people.

[0045] In this application, the face region of the image to be replaced is recognized and cropped, and the face image is redrawn multiple times through a diffusion model. The preset redrawing amplitude is adjusted each time during redrawing. At the same time, the diffusion model has lora weights corresponding to the target face part. Finally, an image with the target face replaced is obtained. In this case, the face information of the target face part can be stored in the form of lora weights. When it is necessary to replace a specific target face, the diffusion model can directly and timely load the corresponding lora weights, thereby improving the face replacement efficiency; at the same time, through multiple redraws and adjustment of the preset redrawing amplitude in this application, the control of face replacement details can be strengthened, and the loss rate of key regions in the face part can be reduced; in addition, through local processing by cropping in this application, the processing speed of high-precision images can be improved; in addition, through multiple redraws and background fusion, the transition of the face boundary can be made more natural.

[0046] Those of ordinary skill in the art can understand that all or part of the features / steps of implementing the above method embodiments can be realized by a method, a data processing system, or a computer program. These features can be implemented without using hardware, entirely using software, or using a combination of hardware and software. The aforementioned computer program can be stored in one or more computer-readable storage media. When the computer program stored on the storage medium is executed (such as by a processor), it executes the steps of the above method embodiments of the face swapping method based on segmentation and a staged diffusion model.

[0047] The aforementioned storage media that can store program codes include: a static hard disk, a solid-state drive, a random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), an optical storage device, a magnetic storage device, a flash memory, a magnetic disk, or an optical disc and / or a combination of the above devices, that is, it can be implemented by any type of volatile or non-volatile storage device or a combination thereof.

[0048] As Figure 6 shown, this application also provides an embodiment of a processing device 1, including one or more processors 11 and a memory 10; wherein, the memory 10 is used to store one or more computer programs, and one or more processors 11 are used to execute the one or more computer programs stored in the memory 10, so that the processor 11 executes the features / steps of the above method embodiments of the face swapping method based on segmentation and a staged diffusion model.

[0049] The present application also provides a computer program product. The computer program product is stored on a data carrier and is designed to execute the face swapping method based on the segmentation and phased diffusion model as described above. Therefore, the computer program product according to the present application has the same advantages as those described in detail with reference to the device according to the present application. The computer program product can be executed as computer-readable instruction codes in each appropriate programming language such as JAVA, C++. In addition, the computer program product can be provided on a network, such as the Internet, or a network user can download the computer program product from a network, such as the Internet, when needed. The computer program product can be implemented either by means of a computer program, i.e., software, or by means of one or more dedicated electronic circuits, i.e., hardware, or in any mixed form, i.e., by means of software components and hardware components, or in a form of software, hardware, or a mixture of software and hardware.

[0050] The above are only the preferred embodiments of the present application. Those skilled in the art will know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present application. Additionally, under the teachings of the present application, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application belong to the protection scope of the present application.

Claims

1. A head swapping and face swapping method based on a segmentation and phased diffusion model, characterized in that, Including: Identifying the to-be-replaced image through a human body parsing model and a face parsing model to obtain the face region of the to-be-replaced image and the mask image of the face region, cropping the face region of the to-be-replaced image to obtain a face image containing the face part; According to the mask image of the face region, redrawing the face image multiple times through a diffusion model to obtain a target image containing the target face part, wherein a preset redrawing amplitude is adjusted during each redrawing, and the diffusion model includes the lora weight corresponding to the target face part; Fusing the target face part of the target image with the face region of the to-be-replaced image to obtain a replaced image; The face swapping method based on segmentation and phased diffusion model further includes: cropping the mask image of the face region to obtain a face swapping frame, and using the face swapping frame as the face region of the to-be-replaced image, wherein the face swapping frame is the largest square frame in the mask image of the face region; The fusing the target face part of the target image with the face region of the to-be-replaced image includes: obtaining the weight matrix of the face swapping frame, and performing weighted mixing of the weight matrix with the target face part of the target image and the face region of the to-be-replaced image respectively, wherein the weight of the weight matrix decreases gradually from the center point to the edge of the face swapping frame; The redrawing the face image multiple times through a diffusion model according to the mask image of the face region includes: adding random Gaussian noise and original image noise to the face image to obtain a noise image, and then denoising the noise image in combination with the lora weight to obtain an initial denoised image; performing noise reset on the initial denoised image and then denoising it in combination with the lora weight, repeating the noise reset operation and the denoising operation multiple times until the target image is obtained.

2. The face swapping method based on the segmentation and phased diffusion model according to claim 1, wherein The face swapping method based on segmentation and phased diffusion model further includes: obtaining any target model image, and training the lora adapter of the diffusion model by using the target model image to obtain the lora weight corresponding to the target face part.

3. The face swapping method based on the segmentation and phased diffusion model according to claim 2, wherein, The training the lora adapter of the diffusion model by using the target model image includes: fine-tuning the linear layer in the attention layer of the diffusion model, decomposing the original large matrix parameters of the diffusion model into two low-rank matrices, and updating the two low-rank matrices based on the loss value of the target model image to obtain the lora adapter of the diffusion model.

4. The face-swapping method based on the segmentation and phased diffusion model according to claim 1, wherein The identifying the to-be-replaced image through a human body parsing model and a face parsing model includes: identifying the head region of the person in the to-be-replaced image through the human body parsing model, and identifying the head region through the face parsing model to remove the hair part and the accessory part of the head region.

5. A computer-readable storage medium, characterized in that, A computer program is stored on the storage medium, and when the computer program is executed, the face swapping method based on segmentation and phased diffusion model according to any one of claims 1-4 is implemented.

6. A processing device, characterized in that, Including: One or more processors; A memory for storing one or more computer programs, and one or more of the processors are configured to execute the one or more computer programs stored in the memory, so that one or more of the processors execute the face swapping method based on a segmentation and phased diffusion model according to any one of claims 1-4.

7. A computer program product, characterized in that, The computer program product is stored on a data carrier and is designed to execute the face swapping method based on a segmentation and phased diffusion model according to any one of claims 1-4.

Images