Face-changing portrait generation method and device and electronic equipment

Through portrait training and face mask image processing, face-changing portrait photos that take into account the similarity and style of portrait faces are generated, which solves the shortcomings that cannot be taken into account in the existing technology and realizes high-quality face-changing portrait photos generation.

CN119027301AActive Publication Date: 2024-11-26深圳墨世科技有限公司

Patent Information

Application Number
CN202410941370.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-26
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

The existing face-changing portrait photo generation solution cannot take into account the similarity and style of portrait face similarity and style.

Method used

By obtaining portrait images and style template images, a portrait model is obtained based on portrait training, a target portrait template image is generated, and a face mask image corresponding to the style template image is constructed, and a target face change is generated based on the target portrait template image and the face mask image is generated. Portrait photography achieves the consistency of portrait face similarity and style.

Benefits of technology

The target face-changing portrait photo generation is achieved, which takes into account both the similarity and style of portrait faces, improving the generation effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027301A_ABST
    Figure CN119027301A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a face-changing portrait generation method and device and electronic equipment. The method comprises the following steps: in response to a face-changing portrait photo generation instruction, obtaining a portrait image and a style template image, the face-changing portrait photo generation instruction being an instruction for indicating to change a portrait face area in the portrait image to a style face area in the style template image to generate a target face-changing portrait photo; carrying out portrait training based on the portrait image to obtain a portrait model, and generating a target portrait template image based on the portrait model and the style template image; and constructing a face mask image corresponding to the style template image, and generating a target face-changing portrait based on the target portrait template image and the face mask image. According to the technical scheme provided by the embodiment of the invention, the target face-changing portrait picture considering the similarity and style consistency of the face of the portrait can be generated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image visual generation, and in particular to a method, device and electronic device for generating face-swapped portrait photos. Background Art

[0002] At present, artificial intelligence (AI) portraits can generate highly imaginative images according to specified characters, bringing great convenience to content creation, art design, and image creation. Among them, AI portraits with anime style (i.e. anime face-changing portraits) are widely used.

[0003] In the process of realizing the present invention, the inventors found that the following technical problems exist in the prior art: the currently used face-changing portrait photo generation solution cannot take into account both the facial similarity and the style consistency of the portrait. Summary of the invention

[0004] The embodiments of the present invention provide a method, device and electronic device for generating a face-swapped portrait photo, which solves the problem of not being able to take into account both the facial similarity and the style consistency of the portrait.

[0005] According to one aspect of the present invention, a method for generating a face-changing portrait is provided, which may include:

[0006] In response to a face-swapped portrait photo generation instruction, a portrait image and a style template image are acquired, wherein the face-swapped portrait photo generation instruction is an instruction for instructing to replace a portrait face region in the portrait image with a style face region in the style template image to generate a target face-swapped portrait photo;

[0007] Performing portrait training based on the portrait image to obtain a portrait model, and generating a target portrait template image based on the portrait model and the style template image;

[0008] A facial mask image corresponding to the style template image is constructed, and a target face-swapped portrait photo is generated based on the target portrait template image and the facial mask image.

[0009] According to another aspect of the present invention, a device for generating a face-changing portrait is provided, which may include:

[0010] An image acquisition module, which can be used to acquire a portrait image and a style template image in response to a face-swapped portrait photo generation instruction, wherein the face-swapped portrait photo generation instruction is an instruction for instructing to replace a portrait face region in the portrait image with a style face region in the style template image to generate a target face-swapped portrait photo;

[0011] A target portrait template image generation module is used to perform portrait training based on the portrait image to obtain a portrait model, and to generate a target portrait template image based on the portrait model and the style template image;

[0012] The target face-swapped portrait generation module is used to construct a facial mask image corresponding to the style template image, and generate a target face-swapped portrait based on the target portrait template image and the facial mask image.

[0013] According to another aspect of the present invention, there is provided an electronic device, which may include:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor implements the face-changing portrait generation method provided by any embodiment of the present invention when executing the computer program.

[0017] The technical solution of the embodiment of the present invention is directed to a face-changing portrait photo generation instruction for instructing to replace a portrait facial region in a portrait image with a style facial region in a style template image to generate a target face-changing portrait photo. By responding to the face-changing portrait photo generation instruction, a portrait image and a style template image are obtained; then, portrait training is performed based on the portrait image to obtain a portrait model, and then a target portrait template image is generated based on the portrait model and the style template image, wherein the facial region in the target portrait template image is highly similar to the portrait facial region, but the remaining regions in the target portrait template image are changed compared to the style template image, that is, the style of the target portrait template image is no longer consistent with the style of the style template image; further, a face mask image corresponding to the style template image is constructed, wherein the face mask image can characterize the region related to the style template image that needs to be replaced (i.e., the facial region to ensure the similarity of the portrait face) and the region that does not need to be replaced (i.e., the remaining regions to ensure the consistency of style), and then a target face-changing portrait photo is generated based on the target portrait template image and the face mask image. The above technical solution replaces the facial area in the target portrait template image with the style template image through the facial mask image, thereby achieving the effective generation of the target face-changing portrait photo that takes into account both the portrait facial similarity and the style consistency.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 It is a schematic diagram of two anime style template images;

[0021] Figure 2a is a schematic diagram of a generation example in a currently used face-changing portrait generation method;

[0022] Figure 2b is a schematic diagram of a generation example in another currently used face-changing portrait generation method;

[0023] Figure 3 is a flow chart of a method for generating a face-swapped portrait photo according to an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of a mask example of a processed image and a facial mask image in a face-swap portrait generation method provided according to an embodiment of the present invention;

[0025] Figure 5 In a face-changing portrait generation method provided by an embodiment of the present invention, Figure 2a and Figure 2b Schematic diagram of anime face-changing portraits generated based on ;

[0026] Figure 6 is a flowchart of another method for generating a face-swapped portrait photo according to an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of a style template image and a corresponding face template image in another face-changing portrait generation method provided according to an embodiment of the present invention;

[0028] Figure 8 is a flowchart of another method for generating a face-swapped portrait photo provided according to an embodiment of the present invention;

[0029] Fig. 9 is a schematic diagram of a process of generating a target portrait template image in another method for generating a face-changing portrait photo provided according to an embodiment of the present invention;

[0030] Fig.10 is a flowchart of another method for generating a face-swapped portrait photo provided according to an embodiment of the present invention;

[0031] Fig.11is a schematic diagram of a process of generating a face-swapped portrait photo before alignment in another method for generating a face-swapped portrait photo provided according to an embodiment of the present invention;

[0032] Fig.12 is a flowchart of an optional example of another method for generating a face-swapped portrait provided by an embodiment of the present invention;

[0033] Fig.13 is a structural block diagram of a face-swapped portrait generation device provided according to an embodiment of the present invention;

[0034] Fig.14 It is a structural schematic diagram of an electronic device for implementing the face-changing portrait photo generation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The situations of "target", "original", etc. are similar and will not be repeated here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0038] Before introducing the embodiments of the present invention, an exemplary description is first given of application scenarios that may be involved in the embodiments of the present invention, so as to better understand the face-changing portrait photo generation process described in the following embodiments.

[0039] For example, here, the process of generating anime face-changing portraits is taken as an example, a number of portrait images input by a user are obtained, and an anime style template image selected by the user from a plurality of preset anime style template images is obtained, wherein: Figure 1 Two anime style template images are shown as examples. On this basis, an anime face-swapped portrait photo can be generated based on the portrait image and the selected anime style template image. The anime face-swapped portrait photo can be understood as a portrait photo obtained by replacing the portrait face area in the portrait image with the style face area in the anime style template image.

[0040] However, the currently used anime face-changing portrait generation scheme cannot take into account both the facial similarity and style consistency of the portrait. Figure 2a The examples given (from left to right are anime style template images, portrait images, and anime face-swapped portraits) perform well in terms of style consistency, but poorly in terms of portrait face similarity; Figure 2b The examples given (from left to right, the images are Figure 2a The performance of the proposed method is good in terms of facial similarity, but poor in terms of style consistency, which needs to be improved urgently.

[0041] In addition, before introducing the embodiments of the present invention, some technical terms that may be involved in the embodiments of the present invention are explained to better understand the face-changing portrait photo generation process described below.

[0042] 1.AIGC: The abbreviation of AI-Generated Content, also known as Generative AI, is a new content creation method after professional-generated content (PGC) and user-generated content (UGC). It can create new forms of digital content generation and interaction in dialogue, stories, images, videos and music production.

[0043] 2.Stable Diffusion: A text-to-image generation model based on Latent Diffusion Models that can generate high-quality, high-resolution, and highly realistic images based on arbitrary text input.

[0044] 3. Magic Avatar: A technology for generating images based on AIGC text. In this scenario, users upload multiple single-person portraits, and Magic Avatar generates multiple imaginative portraits based on the portraits, using the StableDiffusion text image technology.

[0045] Figure 3 This is a flow chart of a face-swapped portrait generation method provided by an embodiment of the present invention. This embodiment is applicable to the generation of face-swapped portraits, and is particularly applicable to the generation of face-swapped portraits that take into account both facial similarity and style consistency. The method can be executed by a face-swapped portrait generation device provided by an embodiment of the present invention, which can be implemented by software and / or hardware, and can be integrated in an electronic device, which can be various user terminals or servers.

[0046] See also Figure 3 The method of the embodiment of the present invention specifically includes the following steps:

[0047] S110. In response to a face-swapped portrait photo generation instruction, a portrait image and a style template image are obtained, wherein the face-swapped portrait photo generation instruction is an instruction for instructing to replace the portrait facial area in the portrait image with the style facial area in the style template image to generate a target face-swapped portrait photo.

[0048] The portrait image may be understood as an image that can represent a person's image, and may further be understood as an image that includes a portrait face region, where the portrait face region is the region where the face in the portrait image is located.

[0049] The style template image can be understood as an image that can represent a certain style and is used as a template. The style template image at least includes a style face area, which is the area where the face in the style template image is located. Exemplarily, the style template image can be, for example, an anime style template image, a professional ID photo style template image, a wedding photo style template image, or a lazy coffee style template image, which depends on the actual situation and is not specifically limited here.

[0050] The face-swapped portrait generation instruction can be understood as an instruction for instructing to replace the portrait face area with the style face area to generate a target face-swapped portrait, that is, an instruction for instructing to replace the style face area in the style template image with the portrait face area to generate a target face-swapped portrait. Exemplarily, the instruction can be an instruction triggered by the user clicking a corresponding control after uploading a portrait image and selecting a style template image. In response to the face-swapped portrait generation instruction, a portrait image and a style template image are obtained.

[0051] S120. Perform portrait training based on the portrait image to obtain a portrait model, and generate a target portrait template image based on the portrait model and the style template image.

[0052] Among them, portrait training is performed based on the portrait image to obtain a portrait model, and the portrait model can reflect the character image represented by the portrait image, and especially can reflect the portrait face area in the portrait image. Exemplarily, portrait training is performed based on the portrait image, and the character image in the portrait image is trained to a portrait label (such as the word sequence ABC) to obtain a portrait model, so that the portrait label can be obtained from the portrait model later, and then the character image represented by the portrait image, especially the portrait face area in the portrait image, can be obtained based on the portrait label. In practical applications, optionally, portrait training can be performed based on Text Inversion, DreamBooth or Lora Training and other schemes, which are not limited here.

[0053] Based on the portrait model and the style template image, a target portrait template image is generated. Compared with the style template image, the target portrait template image is more similar to the character image represented by the portrait model, especially the portrait face area in the portrait image, which also makes the above-mentioned target portrait template image also called a real person template image. In addition, it should be noted that the remaining areas in the target portrait template image may change compared with the style template image, which makes the style of the target portrait template image no longer consistent with the style of the style template image, which is processed in subsequent steps. In practical applications, optionally, the above-mentioned remaining areas are areas other than the face area, for example, it can be at least one of the background area, clothing area and hair area, etc. This is related to the actual situation and is not specifically limited here.

[0054] S130. Construct a facial mask image corresponding to the style template image, and generate a target face-swapped portrait photo based on the target portrait template image and the facial mask image.

[0055] The facial mask image may be an image corresponding to the style template image and used to distinguish the facial area from the remaining areas. The facial mask image may be constructed, for example, directly based on the style template image, that is, the facial mask image may be used to distinguish the facial area from the remaining areas in the style template image; or the style template image may be processed first, and then the facial mask image may be constructed based on the processed image, that is, the facial mask image may be used to distinguish the facial area from the remaining areas in the processed image; and so on, which are not specifically limited here. Exemplary, Figure 4 The left side shows an example of a mask for a processed image and Figure 4The face mask image is shown on the right. It can be understood that the face area is the area related to the style template image that needs to be replaced to ensure the similarity of the portrait face, while the remaining area is the area related to the style template image that does not need to be replaced to ensure the consistency of style.

[0056] As explained above, the facial region in the target portrait template image is similar to the facial region of the portrait, but the style of the target portrait template image is inconsistent with the style template image. On this basis, in order to generate a target face-swapped portrait photo that can take into account both the similarity of the portrait face and the consistency of the style, the face mask image can represent the areas that need to be replaced and the areas that do not need to be replaced related to the style template image, and the target face-swapped portrait photo can be generated based on the target portrait template image and the face mask image. Further, the target face-swapped portrait photo can be generated based on the target portrait template image, the face mask image, and the constructed image (such as the style template image or the processed image) used to construct the face mask image. Exemplarily, area 1 that needs to be replaced in the constructed image can be determined based on the facial mask image, and then area 2 corresponding to area 1 in the target portrait template image can be replaced with area 1. Area 1 is the facial area in the constructed image, and area 2 is the facial area in the target portrait template image that is relatively similar to the portrait facial area. This ensures that only the facial area in the replaced constructed image changes and the remaining areas remain unchanged, thereby ensuring that the final generated target face-swapped portrait can take into account both the portrait facial similarity and style consistency.

[0057] For example, here Figure 2a and Figure 2b Taking the anime style template image and portrait image shown in as an example, after processing the anime style template image and portrait image using the above face-changing portrait generation process, we can get Figure 5 The anime face-changing portrait shown. Figure 5 It can be seen that the style of the anime face-swapped portrait is consistent with the style of the anime style template image, and the facial area in the anime face-swapped portrait is highly similar to the facial area in the portrait image, achieving a balance between facial similarity and style consistency. In addition, the above-mentioned style consistency is mainly for the remaining areas, and the style of the facial area in the anime face-swapped portrait is also consistent with the style of the facial area in the anime style template image, thereby further ensuring style consistency; on this basis, the facial area in the anime face-swapped portrait has integrity and normality, thereby achieving high-quality anime face-swapped.

[0058] The technical solution of the embodiment of the present invention is directed to a face-changing portrait photo generation instruction for instructing to replace a portrait facial region in a portrait image with a style facial region in a style template image to generate a target face-changing portrait photo. By responding to the face-changing portrait photo generation instruction, a portrait image and a style template image are obtained; then, portrait training is performed based on the portrait image to obtain a portrait model, and then a target portrait template image is generated based on the portrait model and the style template image, wherein the facial region in the target portrait template image is highly similar to the portrait facial region, but the remaining regions in the target portrait template image are changed compared to the style template image, that is, the style of the target portrait template image is no longer consistent with the style of the style template image; further, a face mask image corresponding to the style template image is constructed, wherein the face mask image can characterize the region related to the style template image that needs to be replaced (i.e., the facial region to ensure the similarity of the portrait face) and the region that does not need to be replaced (i.e., the remaining regions to ensure the consistency of style), and then a target face-changing portrait photo is generated based on the target portrait template image and the face mask image. The above technical solution replaces the facial area in the target portrait template image with the style template image through the facial mask image, thereby achieving the effective generation of the target face-changing portrait photo that takes into account both the portrait facial similarity and the style consistency.

[0059] Figure 6 It is a flowchart of another face-changing portrait generation method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, based on the portrait model and the style template image, generating a target portrait template image may include: based on the style face area, processing the style template image to obtain a face template image, and obtaining alignment data for aligning the face template image to the style template image; based on the portrait model and the face template image, generating the target portrait template image; the face mask image corresponding to the style template image includes the face mask image of the face template image, then constructing a face mask image corresponding to the style template image, and based on the target portrait template image and the face mask image, generating a target face-changing portrait, including: constructing a face mask image of the face template image, generating a face-changing portrait before alignment based on the target portrait template image, the face template image and the face mask image; based on the alignment data, aligning the face-changing portrait before alignment to the style template image to obtain the target face-changing portrait. Among them, the explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.

[0060] See also Figure 6 The method of this embodiment may specifically include the following steps:

[0061] S210. In response to a face-swapped portrait photo generation instruction, a portrait image and a style template image are obtained, wherein the face-swapped portrait photo generation instruction is an instruction for instructing to replace the portrait facial area in the portrait image with the style facial area in the style template image to generate a target face-swapped portrait photo.

[0062] S220. Perform portrait training based on the portrait image to obtain a portrait model.

[0063] S230. Based on the style face region, the style template image is processed to obtain a face template image, and alignment data for aligning the face template image to the style template image is obtained.

[0064] The style template image is processed based on the style face area to obtain a face template image that can highlight the style face area in the style template image. In practical applications, the style template image can be cropped based on retaining the style face area, and the face template image can be obtained on this basis. Exemplarily, for the cropped template image obtained after the cropping process, the cropped template image can be directly used as the face template image; the cropped template image can also be processed again to obtain the face template image, for example, the style face area is positioned in the middle area as the target, and the cropped template image is filled to obtain the face template image, such as Figure 7 The style template image shown on the left and the face template image shown on the right, wherein the face template image can be a square image with a resolution of 512x512; etc., which are not specifically limited here.

[0065] The alignment data can be understood as data used to align the face template image to the style template image. In practical applications, the alignment data can be optionally represented by an affine transformation matrix. The alignment data is obtained and applied in the following steps to align the generated face-swapped portrait photo to the face template image to obtain a target face-swapped portrait photo that matches the style template image.

[0066] In combination with possible application scenarios of the embodiments of the present invention, the cropping and alignment of the style template image may utilize Modelscope; of course, it may also be implemented based on other solutions, which are not specifically limited here.

[0067] S240. Generate a target portrait template image based on the portrait model and the face template image.

[0068] Among them, compared with directly generating the target portrait template image based on the style template image, this step generates the target portrait template image based on the face template image, which can pay more attention to the face area during the generation process. Combined with the portrait model, it helps to improve the similarity between the face area in the target portrait template image and the face area of ​​the portrait.

[0069] S250. Construct a face mask image of the face template image, and generate a face-changing portrait photo before alignment based on the target portrait template image, the face template image and the face mask image.

[0070] Among them, a face mask image is constructed that can be used to distinguish the face area and the remaining areas in the face template image, and then the face-changing portrait before alignment can be generated based on the target portrait template image, the face template image and the face mask image. Exemplarily, the area 1 that needs to be replaced in the face template image can be determined based on the face mask image, and then the area 2 corresponding to the area 1 in the target portrait template image is replaced on the area 1, the area 1 is the face area in the face template image, and the area 2 is the face area in the target portrait template image that is more similar to the portrait face area, so as to ensure that only the face area in the replaced face template image changes and the remaining areas remain unchanged, so as to ensure that the face-changing portrait before alignment generated subsequently can take into account both the portrait face similarity and the style consistency.

[0071] S260. Based on the alignment data, align the face-swapped portrait photo before alignment to the style template image to obtain the target face-swapped portrait photo.

[0072] Among them, since the face-swapped portrait photo before alignment is generated based on the facial template image, that is, it matches the facial template image but does not match the style template image, the alignment data can be used to align (i.e. reply) the face-swapped portrait photo before alignment to the style template image, thereby obtaining the target face-swapped portrait photo that matches the style template image.

[0073] The technical solution of the embodiment of the present invention processes the style template image based on the style facial area to obtain a facial template image that can highlight the style facial area, and then uses the facial template image and the portrait model to generate a target portrait template image, which helps to improve the similarity between the facial area in the target portrait template image and the portrait facial area, and further helps to improve the portrait facial similarity of the target face-swapped portrait photo finally generated.

[0074] Figure 8It is a flowchart of another face-changing portrait generation method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, based on the portrait model and the face template image, generating the target portrait template image may include: generating an intermediate portrait template image based on the portrait model and the face template image; judging the similarity between the intermediate portrait template image and the target portrait for the target portrait represented by the portrait model, whether the preset first iteration end condition is met; if not met, regenerating the intermediate portrait template image based on the portrait model, the face template image and the intermediate portrait template image, and returning to the step of judging whether the similarity between the intermediate portrait template image and the target portrait meets the preset iteration end condition; if met, using the intermediate portrait template image as the target portrait template image. The explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.

[0075] See also Figure 8 The method of this embodiment may specifically include the following steps:

[0076] S310. In response to a face-swapped portrait photo generation instruction, a portrait image and a style template image are obtained, wherein the face-swapped portrait photo generation instruction is an instruction for instructing to replace the portrait facial area in the portrait image with the style facial area in the style template image to generate a target face-swapped portrait photo.

[0077] S320. Perform portrait training based on the portrait image to obtain a portrait model.

[0078] S330. Based on the style face region, the style template image is processed to obtain a face template image, and alignment data for aligning the face template image to the style template image is obtained.

[0079] S340. Generate an intermediate portrait template image based on the portrait model and the face template image.

[0080] In order to ensure that the facial region in the finally generated target portrait template image is highly similar to the facial region of the portrait, the target portrait template image may be generated in an iterative manner.

[0081] Specifically, in the first round of iteration, an intermediate portrait template image may be generated based on the portrait model and the face template image. The intermediate portrait template image may be understood as an intermediate result in the iteration process.

[0082] It can be understood that this step is the first round of iterative process.

[0083] S350. For the target portrait represented by the portrait model, determine whether the similarity between the intermediate portrait template image and the target portrait satisfies a preset first iteration end condition.

[0084] The target portrait can be understood as the image of a person represented by the portrait model. The first iteration end condition can be understood as a pre-set condition for ending the iteration process of generating the target portrait template image, for example, the similarity between the intermediate portrait template image and the target portrait is greater than a preset first similarity threshold. It is determined whether the similarity meets the first iteration end condition.

[0085] S360. If not satisfied, regenerate the intermediate portrait template image based on the portrait model, the face template image and the intermediate portrait template image, and return to execute S350.

[0086] Among them, if it is not satisfied, this means that the similarity between the newly generated intermediate portrait template image and the target portrait is not high enough, specifically, the similarity between the facial area in the intermediate portrait template image and the facial area of ​​the portrait is not high enough, and it is necessary to regenerate the intermediate portrait template image based on the portrait model, the facial template image and the intermediate portrait template image, and return to execute S350 to re-judge, that is, continue to iterate.

[0087] S370. If satisfied, use the intermediate portrait template image as the target portrait template image.

[0088] Among them, if it is satisfied, it means that the similarity between the newly generated intermediate portrait template image and the target portrait is high enough, specifically, the similarity between the facial area in the intermediate portrait template image and the facial area of ​​the portrait is high enough, then the intermediate portrait template image can be used as the target portrait template image.

[0089] At this point, the iterative process of generating the target portrait template image is completed.

[0090] S380. Construct a face mask image of the face template image, and generate a face-changing portrait photo before alignment based on the target portrait template image, the face template image and the face mask image.

[0091] S390. Based on the alignment data, align the face-swapped portrait photo before alignment to the style template image to obtain the target face-swapped portrait photo.

[0092] The technical solution of the embodiment of the present invention generates a target face-swapped portrait photo in an iterative manner, and in each round of iteration, determines whether the iteration is completed based on the similarity between the generated intermediate face-swapped portrait photo and the target portrait, thereby improving the similarity between the finally generated target face-swapped portrait photo and the facial area of ​​the portrait.

[0093] An optional technical solution, based on a portrait model and a face template image, generates an intermediate portrait template image, including:

[0094] In the image-generated-image mode, using the portrait model and the face template image, and, in the control network, using the face template image, generating an intermediate portrait template image;

[0095] Based on the portrait model, the face template image and the intermediate portrait template image, the intermediate portrait template image is regenerated, including:

[0096] In the image-generated-image mode, the intermediate portrait template image is regenerated using the portrait model and the intermediate portrait template image, and, in the control network, the intermediate portrait template image and the face template image are used.

[0097] Among them, Stable Diffusion not only supports images generated by the image-to-image (Img2Img) mode, but also supports various control methods such as control network (ControlNet) to customize the generated images. Therefore, in Stable Diffusion, the Img2Img mode and ControlNet can be used to generate intermediate portrait template images.

[0098] Exemplarily, in the Img2Img mode, the portrait model and the first initialization image are used, and in ControlNet, the first initialization image and the face template image are used, and the two cooperate to generate an intermediate portrait template image. Wherein, in the first round of iteration, the first initialization image uses the face template image, and in subsequent rounds of iteration, the first initialization image uses the intermediate portrait template image generated by the previous round of iteration. On this basis, optionally, parameters such as denoising strength can be set in the Img2Img mode; parameters such as lineart_anime, Depth and IPAdpaterPlus can be set in ControlNet, wherein the input images of lineart_anime and Depth are the same as the first initialization image, and the input image of IPAdpaterPlus is the same as the face template image. Exemplarily, Fig. 9 The three intermediate portrait template images generated sequentially during the iteration process are shown.

[0099] Fig.10It is a flowchart of another face-changing portrait generation method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, based on the target portrait template image, the face template image and the face mask image, the face-changing portrait before alignment is generated, which may include: based on the target portrait template image, the face template image and the face mask image, an intermediate face-changing portrait is generated; for the target portrait represented by the portrait model, the similarity between the intermediate face-changing portrait and the target portrait is judged to see whether the preset second iteration end condition is met; if not met, the intermediate face-changing portrait is regenerated based on the intermediate face-changing portrait, the target portrait template image, the face template image and the face mask image, and the step of returning to execute the judgment of the similarity between the intermediate face-changing portrait and the target portrait to see whether the preset second iteration end condition is met; if met, the intermediate face-changing portrait is used as the face-changing portrait before alignment. Among them, the explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here.

[0100] See also Fig.10 The method of this embodiment may specifically include the following steps:

[0101] S410. In response to a face-swapped portrait photo generation instruction, a portrait image and a style template image are obtained, wherein the face-swapped portrait photo generation instruction is an instruction for instructing to replace the portrait facial area in the portrait image with the style facial area in the style template image to generate a target face-swapped portrait photo.

[0102] S420. Perform portrait training based on the portrait image to obtain a portrait model.

[0103] S430. Based on the style face region, the style template image is processed to obtain a face template image, and alignment data for aligning the face template image to the style template image is obtained.

[0104] S440. Generate a target portrait template image based on the portrait model and the face template image.

[0105] S450. Construct a face mask image of the face template image, and generate an intermediate face-changing portrait photo based on the target portrait template image, the face template image and the face mask image.

[0106] In order to ensure that the subsequently generated pre-alignment face-swapped portrait photo is highly similar to the target portrait, the pre-alignment face-swapped portrait photo may be generated in an iterative manner.

[0107] Specifically, in the first round of iteration, an intermediate face-swapped portrait photo can be generated based on the target portrait template image, the face template image and the face mask image, and the intermediate face-swapped portrait photo can be understood as an intermediate result in the iteration process. It can be understood that this step is the first round of iteration.

[0108] In practical applications, in addition to using the above three images, a portrait model can also be used to jointly generate an intermediate face-swapped portrait photo, so as to improve the similarity between the intermediate face-swapped portrait photo and the target portrait.

[0109] S460. For the target portrait represented by the portrait model, determine whether the similarity between the intermediate face-swapped portrait and the target portrait meets the preset second iteration end condition.

[0110] The second iteration end condition can be understood as a preset condition that indicates that the iteration process of generating the face-swapped portrait before alignment can be ended, for example, the similarity between the intermediate face-swapped portrait and the target portrait is greater than a preset second similarity threshold. It is determined whether the similarity satisfies the second iteration end condition.

[0111] S470. If not satisfied, regenerate the intermediate face-changing portrait photo based on the intermediate face-changing portrait photo, the target portrait template image, the facial template image and the facial mask image, and return to execute S460.

[0112] Among them, if it is not satisfied, this means that the similarity between the newly generated intermediate face-changing portrait and the target portrait is not high enough, and it is necessary to regenerate the intermediate face-changing portrait based on the intermediate face-changing portrait, the target portrait template image, the face template image and the face mask image, and return to execute S460 to continue iteration.

[0113] In practical applications, in addition to using the above four images, the portrait model can also be combined to regenerate the intermediate face-changing portrait photo, so as to improve the similarity between the intermediate face-changing portrait photo and the target portrait.

[0114] S480. If the conditions are met, the middle face-changing portrait photo is used as the face-changing portrait photo before alignment.

[0115] If the condition is satisfied, it means that the similarity between the newly generated intermediate face-changing portrait and the target portrait is high enough, and the intermediate face-changing portrait can be used as the face-changing portrait before alignment.

[0116] At this point, the iterative process of generating the face-changing portrait before alignment is completed.

[0117] S490. Based on the alignment data, align the face-swapped portrait photo before alignment to the style template image to obtain the target face-swapped portrait photo.

[0118] The technical solution of the embodiment of the present invention generates the face-swapped portrait photo before alignment in an iterative manner, and in each round of iteration, it is judged whether the iteration is completed based on the similarity between the generated intermediate face-swapped portrait photo and the target portrait, which helps to improve the similarity between the face-swapped portrait photo before alignment generated at the end of the iteration and the target portrait.

[0119] An optional technical solution generates an intermediate face-swap portrait photo based on a target portrait template image, a face template image, and a face mask image, including:

[0120] In the image generation mode, the face template image and the face mask image are used, and in the control network, the target portrait template image and the face template image are used to generate an intermediate face-swapped portrait photo;

[0121] Accordingly, based on the intermediate face-changing portrait photo, the target portrait template image, the face template image and the face mask image, the intermediate face-changing portrait photo is regenerated, including:

[0122] In the image-generated-image mode, the face mask image and the intermediate face-swapped portrait photo are used, and in the control network, the target portrait template image and the face template image are used to regenerate the intermediate face-swapped portrait photo.

[0123] Among them, in Stable Diffusion, the Img2Img mode and ControlNet can be used to generate an intermediate face-changing portrait photo. Exemplarily, in the Img2Img mode, the portrait model, the face mask image and the second initialization image can be used, and in ControlNet, the target portrait template image and the face template image can be used, and the two are coordinated to generate an intermediate face-changing portrait photo. Among them, in the first round of iteration, the second initialization image uses the face template image, and in the subsequent rounds of iteration, the second initialization image uses the intermediate face-changing portrait photo generated by the previous round of iteration. On this basis, optionally, parameters such as denoising strength can be set in the Img2Img mode; parameters such as lineart_anime, SoftEdge, IPAdpaterPlusFaceV2 and IPAdpaterPlus can be set in ControlNet, wherein the input images of lineart_anime, SoftEdge and IPAdpaterPlusFaceV2 are the same as the target portrait template image, and the input image of IPAdpaterPlus is the same as the face template image. Exemplarily, Fig.11 Two intermediate face-swapped portraits generated sequentially during the iterative process are shown.

[0124] In order to better understand the above-mentioned technical solutions as a whole, the following is an exemplary description of them in combination with specific examples. Fig.12 The process of generating anime face-changing portraits is shown below:

[0125] Step 1: Perform portrait training based on multiple portrait images uploaded by users to obtain a portrait model.

[0126] Step 2: Obtain an anime style template image, and perform face cropping on the anime style template image to obtain a face template image. The face template image may be a square image containing a front face (i.e., a template front face square image), for example Figure 7 The anime-style template image and the face template image are displayed; then, an affine transformation matrix that can align the face template image to the anime-style template image is obtained.

[0127] Step 3, generate a high-similarity target portrait template image (i.e., a real-person template image). Specifically, the StableDiffusion Img2Img mode is used, combined with the portrait model trained in step 1, the denoising strength is 0.4-0.6, and the cycle is repeated multiple times until the similarity > 0.6. Among them, the first initialization image first uses the template front face square image output by step 2, and then uses the real-person template image output by the previous iteration. ControlNet configures lineart_anime (weight 0.4, the input image is the same as the first initialization image), Depth (weight 0.5, the input image is the same as the first initialization image), IPAdpaterPlus (weight 0.3, the input image is the template front face square image output by step 2). Fig. 9 The real-person template image generated by three iterations is shown.

[0128] Step 4: Construct a face mask image of the template frontal face image output in step 2, see Figure 4 , where white areas represent areas that need to be replaced and black areas represent areas that do not need to be replaced.

[0129] Step 5. Generate the face-changing portrait before alignment (i.e., the face-changing result image). Use the Stable Diffusion Img2Img mode, combined with the portrait model output in step 1, with the denoising strength between 0.4 and 0.6. The mask uses the face mask image output in step 4, and the cycle continues until the similarity is greater than 0.5. Among them, the second initialization image first uses the template frontal face square image output in step 2, and then uses the face-changing result image output in the previous iteration. ControlNet configures lineart_anime (weight 0.75, the input image is the real-person template image output in step 3), SoftEdge (weight 0.75, the input image is the real-person template image output in step 3), IPAdpaterPlusFaceV2 (weight 1.0, the input image is the real-person template image output in step 3) and IPAdpaterPlus (weight 0.5, the input image is the template frontal face square image output in step 2). Fig.11 The face-changing results generated by two iterative processes are shown.

[0130] Step 6: Use the affine transformation matrix output from step 2 to paste the face-changing result image back into the anime-style template image to obtain an anime face-changing portrait (i.e., the final result image), as shown in Figure 5 shown.

[0131] The above example provides a process for generating anime face-changing portraits based on StableDiffusion, which can take into account both the facial similarity of the portrait and the consistency of the anime style.

[0132] Fig.13 This is a structural block diagram of a face-changing portrait generation device provided in an embodiment of the present invention, and the device is used to execute the face-changing portrait generation method provided in any of the above embodiments. The device and the face-changing portrait generation method of the above embodiments belong to the same inventive concept, and the details not described in detail in the embodiment of the face-changing portrait generation device can refer to the embodiment of the face-changing portrait generation method. Fig.13 The device may specifically include: an image acquisition module 510, a target portrait template image generation module 520, and a target face-changing portrait photo generation module 530.

[0133] The image acquisition module 510 is used to acquire a portrait image and a style template image in response to a face-swapped portrait photo generation instruction, wherein the face-swapped portrait photo generation instruction is an instruction for instructing to replace a portrait face region in the portrait image with a style face region in the style template image to generate a target face-swapped portrait photo;

[0134] A target portrait template image generation module 520 is used to perform portrait training based on the portrait image to obtain a portrait model, and to generate a target portrait template image based on the portrait model and the style template image;

[0135] The target face-swapped portrait generating module 530 is used to construct a facial mask image corresponding to the style template image, and generate a target face-swapped portrait based on the target portrait template image and the facial mask image.

[0136] Optionally, the target portrait template image generating module 520 may include:

[0137] An alignment data obtaining submodule is used to process the style template image based on the style face region to obtain the face template image and obtain alignment data for aligning the face template image to the style template image;

[0138] A target portrait template image generation submodule is used to generate a target portrait template image based on a portrait model and a face template image;

[0139] The facial mask image corresponding to the style template image includes the facial mask image of the facial template image, and the target face-swapped portrait generation module 530 may include:

[0140] A pre-alignment face-swapped portrait photo generation submodule is used to construct a face mask image of the face template image, and generate a pre-alignment face-swapped portrait photo based on the target portrait template image, the face template image and the face mask image;

[0141] The target face-swapped portrait photo obtaining submodule is used to align the pre-alignment face-swapped portrait photo to the style template image based on the alignment data to obtain the target face-swapped portrait photo.

[0142] On this basis, an optional target portrait template image generation submodule may include:

[0143] An intermediate portrait template image generating unit, used to generate an intermediate portrait template image based on the portrait model and the face template image;

[0144] A first similarity judgment unit is used to judge, for the target portrait represented by the portrait model, whether the similarity between the intermediate portrait template image and the target portrait satisfies a preset first iteration end condition;

[0145] A first return execution unit is used for, if the condition is not satisfied, regenerating the intermediate portrait template image based on the portrait model, the face template image and the intermediate portrait template image, and returning to the step of determining whether the similarity between the intermediate portrait template image and the target portrait satisfies a preset iteration end condition;

[0146] The target portrait template image obtaining unit is used to use the intermediate portrait template image as the target portrait template image if the conditions are met.

[0147] On this basis, optionally, the intermediate portrait template image generating unit may include:

[0148] An intermediate portrait template image generating subunit, used to generate an intermediate portrait template image by using the portrait model and the face template image in the image-generated-image mode, and by using the face template image in the control network;

[0149] The first return execution unit may include:

[0150] The intermediate portrait template image regeneration subunit is used to regenerate the intermediate portrait template image using the portrait model and the intermediate portrait template image in the image-generated-image mode, and using the intermediate portrait template image and the face template image in the control network.

[0151] Alternatively, the pre-alignment face-swap portrait generation submodule may include:

[0152] An intermediate face-swapped portrait photo generating unit, used to generate an intermediate face-swapped portrait photo based on a target portrait template image, a facial template image, and a facial mask image;

[0153] A second similarity judgment unit is used to judge, for the target portrait represented by the portrait model, whether the similarity between the intermediate face-swapped portrait and the target portrait satisfies a preset second iteration end condition;

[0154] A second return execution unit is used for, if not satisfied, regenerating the intermediate face-swapped portrait photo based on the intermediate face-swapped portrait photo, the target portrait template image, the facial template image and the facial mask image, and returning to execute the step of determining whether the similarity between the intermediate face-swapped portrait photo and the target portrait satisfies a preset second iteration end condition;

[0155] The unit for obtaining the face-swapped portrait photo before alignment is used to use the intermediate face-swapped portrait photo as the face-swapped portrait photo before alignment if the conditions are met.

[0156] On this basis, optionally, the intermediate face-changing portrait photo generation unit may include:

[0157] The intermediate face-swapped portrait photo generation subunit is used to generate the intermediate face-swapped portrait photo by using the face template image and the face mask image in the image generation mode, and by using the target portrait template image and the face template image in the control network;

[0158] The second return execution unit may include:

[0159] The intermediate face-swapped portrait photo regeneration subunit can be used to regenerate the intermediate face-swapped portrait photo in the image-generated image mode using the face mask image and the intermediate face-swapped portrait photo, and in the control network using the target portrait template image and the face template image.

[0160] Alternatively, aligning the data to obtain a submodule may include:

[0161] The facial template image obtaining unit can be used to perform cropping processing on the style template image based on retaining the style facial region to obtain the facial template image.

[0162] Based on any of the above devices, optionally, the style template image includes an animation style template image.

[0163] The face-swapped portrait generation device provided in the embodiment of the present invention obtains a portrait image and a style template image in response to a face-swapped portrait generation instruction through an image acquisition module; then, through a target portrait template image generation module, portrait training is performed based on the portrait image to obtain a portrait model, and then a target portrait template image is generated based on the portrait model and the style template image; further, through a target face-swapped portrait generation module, a facial mask image corresponding to the style template image is constructed, and then a target face-swapped portrait is generated based on the target portrait template image and the facial mask image. The above device can replace the facial area in the target portrait template image with the style template image through the facial mask image, thereby realizing the effective generation of a target face-swapped portrait that takes into account both the facial similarity and the consistency of the style.

[0164] The face-swapped portrait photo generation device provided in the embodiment of the present invention can execute the face-swapped portrait photo generation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0165] It is worth noting that in the embodiment of the above-mentioned face-changing portrait generation device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the various functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0166] Fig.14 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0167] like Fig.14As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0168] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0169] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the face-changing portrait generation method.

[0170] In some embodiments, the face-swapped portrait generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the face-swapped portrait generation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the face-swapped portrait generation method in any other appropriate manner (e.g., by means of firmware).

[0171] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0173] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0175] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0176] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0177] The various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document is not limited here.

[0178] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a face-changing portrait, characterized in that: include: In response to a face-swapped portrait photo generation instruction, a portrait image and a style template image are acquired, wherein the face-swapped portrait photo generation instruction is an instruction for instructing to replace a portrait face region in the portrait image with a style face region in the style template image to generate a target face-swapped portrait photo; Performing portrait training based on the portrait image to obtain a portrait model, and generating a target portrait template image based on the portrait model and the style template image; A facial mask image corresponding to the style template image is constructed, and the target face-swapped portrait photo is generated based on the target portrait template image and the facial mask image.

2. The method according to claim 1, characterized in that The step of generating a target portrait template image based on the portrait model and the style template image includes: Based on the style face region, the style template image is processed to obtain a face template image, and alignment data for aligning the face template image to the style template image is obtained; Based on the portrait model and the face template image, generating a target portrait template image; The facial mask image corresponding to the style template image includes the facial mask image of the facial template image, and the facial mask image corresponding to the style template image is constructed, and the target face-swapped portrait is generated based on the target portrait template image and the facial mask image, including: Constructing a face mask image of the face template image, and generating a face-swapped portrait photo before alignment based on the target portrait template image, the face template image and the face mask image; Based on the alignment data, the pre-alignment face-swapped portrait photo is aligned to the style template image to obtain the target face-swapped portrait photo.

3. The method according to claim 2, characterized in that The step of generating a target portrait template image based on the portrait model and the face template image comprises: Based on the portrait model and the face template image, generating an intermediate portrait template image; For the target portrait represented by the portrait model, determining whether the similarity between the intermediate portrait template image and the target portrait satisfies a preset first iteration end condition; If not, regenerate the intermediate portrait template image based on the portrait model, the face template image and the intermediate portrait template image, and return to the step of determining whether the similarity between the intermediate portrait template image and the target portrait satisfies the preset iteration end condition; If the conditions are met, the intermediate portrait template image is used as the target portrait template image.

4. The method according to claim 3, characterized in that The step of generating an intermediate portrait template image based on the portrait model and the face template image comprises: In the image-to-image mode, using the portrait model and the face template image, and, in the control network, using the face template image, generating an intermediate portrait template image; The regenerating the intermediate portrait template image based on the portrait model, the face template image and the intermediate portrait template image comprises: In the image-to-image mode, the intermediate portrait template image is regenerated using the portrait model and the intermediate portrait template image, and in the control network, the intermediate portrait template image and the face template image are used.

5. The method according to claim 2, characterized in that: The step of generating the face-swapped portrait photo before alignment based on the target portrait template image, the facial template image and the facial mask image comprises: Generate an intermediate face-swapped portrait photo based on the target portrait template image, the facial template image and the facial mask image; For the target portrait represented by the portrait model, determining whether the similarity between the intermediate face-swapped portrait and the target portrait satisfies a preset second iteration end condition; If not, regenerate the intermediate face-swapped portrait based on the intermediate face-swapped portrait, the target portrait template image, the facial template image and the facial mask image, and return to the step of determining whether the similarity between the intermediate face-swapped portrait and the target portrait satisfies the preset second iteration end condition; If the conditions are met, the intermediate face-changing portrait photo is used as the face-changing portrait photo before alignment.

6. The method according to claim 5, characterized in that The step of generating an intermediate face-swapped portrait photo based on the target portrait template image, the facial template image, and the facial mask image comprises: In the image-generated-image mode, using the facial template image and the facial mask image, and, in the control network, using the target portrait template image and the facial template image, generating an intermediate face-swapped portrait photo; Accordingly, the regenerating the intermediate face-swapped portrait photo based on the intermediate face-swapped portrait photo, the target portrait template image, the facial template image and the facial mask image includes: In the image-generating mode, the intermediate face-changing portrait photo is regenerated using the facial mask image and the intermediate face-changing portrait photo, and in the control network, the target portrait template image and the facial template image are used.

7. The method according to claim 2, characterized in that The step of processing the style template image based on the style face region to obtain the face template image includes: Based on retaining the style face region, the style template image is cropped to obtain a face template image.

8. The method according to any one of claims 1 to 7, characterized in that: The style template image includes an anime style template image.

9. A face-changing portrait generation device, characterized in that: include: an image acquisition module, configured to acquire a portrait image and a style template image in response to a face-swapped portrait photo generation instruction, wherein the face-swapped portrait photo generation instruction is an instruction for instructing to replace a portrait face region in the portrait image with a style face region in the style template image to generate a target face-swapped portrait photo; A target portrait template image generation module, used to perform portrait training based on the portrait image to obtain a portrait model, and to generate a target portrait template image based on the portrait model and the style template image; The target face-swapped portrait generating module is used to construct a facial mask image corresponding to the style template image, and to generate the target face-swapped portrait based on the target portrait template image and the facial mask image.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the face-changing portrait generation method as described in any one of claims 1-8.

Citation Information

Patent Citations

  • Face changing method and device based on artificial intelligence, equipment and storage medium

    CN114387656A

  • Image processing method and device, storage medium and electronic equipment

    CN114913061A

  • Deep fake face changing image generation method, system and device and storage medium

    CN116092153A

  • Image generation method and device, electronic equipment and medium

    CN117764812A

  • System and method for producing caricatures

    KR1020020069595A

Cited By

  • Figure group photo generation method based on depth generation model

    CN121120410A