Stylized face image editing method and device, product, medium and equipment

Abstract

The invention discloses a stylized face image editing method and device, a product, a medium and equipment, and relates to the technical field of computer vision, and the method comprises the steps: inputting a picture content target editing direction and a to-be-edited face image into a StyleCLIP model and a GAN Inversion model respectively, outputting the W feature space variable quantity corresponding to the picture content target editing direction and the W feature space code corresponding to the to-be-edited face image; performing fine tuning on model parameters of a generator in the StyleGAN model based on the to-be-edited face image and the W feature space code corresponding to the to-be-edited face image to obtain a fine-tuned generator; and editing the to-be-edited face image according to the W feature space variation, the W feature space code and the fine-tuned generator to obtain an edited face image. According to the method, the face image with uniform style and rich content can be quickly generated.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, product, medium and equipment for stylized face image editing. Background Art

[0002] Stylized face image editing has a wide range of applications in character design, visual creativity, and entertainment. For example, stylized face image editing can be used to design more personalized character images or to enrich the gameplay of entertainment content. With the continuous development of the digital content creation industry, the rapid generation of personalized face images with unified style and rich content has become an important demand of content creators. However, traditional face image editing methods usually rely on a large amount of annotated data and complex model training processes. Due to the limited data volume and computing power, these methods have certain limitations when dealing with the editing needs of specific stylized face images. Image editing requires a large amount of data and computing power. In order to meet this challenge, researchers have begun to focus on how to perform efficient editing based on a small number of stylized samples, so as to achieve flexible editing of image content while maintaining the consistency of image style. Summary of the invention

[0003] The purpose of this application is to provide a stylized facial image editing method, device, product, medium and equipment, which can quickly generate facial images with unified style and rich content.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a stylized face image editing method, comprising:

[0006] Input the target editing direction of the image content into the StyleCLIP model, and output the W feature space variation corresponding to the target editing direction of the image content;

[0007] Input the face image to be edited into the GAN Inversion model, and output the W feature space encoding corresponding to the face image to be edited;

[0008] Fine-tuning the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space encoding corresponding to the face image to be edited, to obtain a fine-tuned generator; the model parameters of the generator in the StyleGAN model include weights and biases of convolutional layers and fully connected layers;

[0009] According to the W feature space variation corresponding to the target editing direction of the image content, the W feature space encoding corresponding to the face image to be edited, and the fine-tuned generator, the face image to be edited is edited to obtain the edited face image.

[0010] Optionally, the StyleCLIP model, the GAN Inversion model, and the StyleGAN model are all models fully trained on a real style dataset FFHQ.

[0011] Optionally, fine-tuning the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space encoding corresponding to the face image to be edited to obtain the fine-tuned generator specifically includes:

[0012] Perform data enhancement processing on the W feature space codes corresponding to the face image to be edited, obtain multiple feature space code enhancement samples and form a feature space code enhancement set S;

[0013] Based on the face image to be edited and the feature space encoding enhancement set S, the model parameters of the generator in the StyleGAN model are fine-tuned to obtain the fine-tuned generator.

[0014] Optionally, fine-tuning the model parameters of the generator in the StyleGAN model based on the face image to be edited and the feature space coding enhancement set S to obtain the fine-tuned generator specifically includes:

[0015] Calculate the LPIPS loss based on 256 resolution between the image generated by the StyleGAN model for each feature space coding enhancement sample in the feature space coding enhancement set S and the face image to be edited, and optimize the model parameters of the generator in the StyleGAN model through gradient back propagation based on the LPIPS loss to obtain the fine-tuned generator Generator'.

[0016] Optionally, the step of editing the face image to be edited according to the W feature space variation corresponding to the target editing direction of the picture content, the W feature space encoding corresponding to the face image to be edited, and the fine-tuned generator to obtain the edited face image specifically includes:

[0017] According to the W feature space variation Δlatent code corresponding to the target editing direction of the image content and the W feature space encoding latent code corresponding to the face image to be edited, the modified feature space encoding latent code' is obtained;

[0018] The modified feature space encoding latent code' is input into the fine-tuned generator, and the face image to be edited is edited to obtain the edited face image.

[0019] Optionally, the step of obtaining a modified feature space code latentcode' according to the W feature space variation Δlatentcode corresponding to the target editing direction of the image content and the W feature space code latentcode corresponding to the face image to be edited specifically includes:

[0020] According to the W feature space variation Δlatent code corresponding to the target editing direction of the image content and the W feature space encoding latent code corresponding to the face image to be edited, the modified feature space encoding latent code' is obtained using the formula latent code'=latent code+β×Δlatent code; wherein β is the content editing strength.

[0021] In a second aspect, the present application provides a stylized face image editing device, comprising:

[0022] The W feature space variation acquisition module is used to input the target editing direction of the image content into the StyleCLIP model and output the W feature space variation corresponding to the target editing direction of the image content;

[0023] The W feature space encoding acquisition module is used to input the face image to be edited into the GAN Inversion model and output the W feature space encoding corresponding to the face image to be edited;

[0024] A fine-tuning module, used to fine-tune the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space encoding corresponding to the face image to be edited, so as to obtain a fine-tuned generator; the model parameters of the generator in the StyleGAN model include weights and biases of convolutional layers and fully connected layers;

[0025] The editing module is used to edit the face image to be edited according to the W feature space variation corresponding to the target editing direction of the image content, the W feature space encoding corresponding to the face image to be edited, and the fine-tuned generator to obtain the edited face image.

[0026] In a third aspect, the present application provides a computer program product, including a computer program, which implements the stylized facial image editing method when executed by a processor.

[0027] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the stylized facial image editing method when executed by a processor.

[0028] In a fifth aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the stylized facial image editing method.

[0029] According to the specific embodiments provided in this application, this application has the following technical effects:

[0030] The present application provides a stylized facial image editing method, apparatus, product, medium and equipment, which mainly relates to a stylized facial image editing method based on StyleGAN, which characterizes the content of the facial image to be edited by acquiring and using the W feature space encoding of the facial image to be edited, and completes content editing based on the W feature space encoding and the W feature space variation. At the same time, the model parameters of the generator in the StyleGAN model are fine-tuned based on the facial image to be edited and the W feature space encoding corresponding to the facial image to be edited, so as to ensure that the style of the generated facial image remains consistent while the content is edited. The stylized facial image editing method of the present application effectively improves the flexibility and efficiency of stylized facial image editing and meets the needs of digital content creators. Therefore, the stylized facial image editing method, apparatus, product, medium and equipment of the present application can quickly generate facial images with unified style and rich content. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A schematic diagram of the process of stylized face image editing method provided in this application;

[0033] Figure 2 Schematic diagram of fine-tuning the StyleGAN model provided in this application;

[0034] Figure 3 Schematic diagram of the edited face image output based on the fine-tuned generator provided for this application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] In traditional face image editing methods, due to the obvious limitations of retraining stylized sample data sets, researchers have begun to try to use prior information learned on real style data sets to guide the editing of stylized face images. The difficulty of this process is: how to ensure the decoupling between image content and style, that is, how to maintain the consistency of style while editing the content. Achieving this balance is the key to stylized face image editing. Therefore, this application provides a stylized face image editing method, device, product, medium and equipment that can quickly generate face images with unified style and rich content.

[0037] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0038] like Figure 1 As shown, the present application discloses a stylized face image editing method, comprising:

[0039] First of all, the prerequisite for implementing the stylized face image editing method in this application is to obtain in advance the StyleCLIP model, GAN Inversion model and StyleGAN model that have been fully trained on the real style dataset FFHQ, and perform subsequent data processing through the above models. Among them, there are multiple versions of the GAN Inversion model and the StyleGAN model, and the specific versions used in this application are: encoder4editing and StyleGAN 2.

[0040] Step 1: Input the target editing direction of the image content into the StyleCLIP model and output the W feature space variation corresponding to the target editing direction of the image content.

[0041] Specifically, the target editing direction of the image content is input into the StyleCLIP model, and the W feature space variation Δlatent code corresponding to the target editing direction of the image content (such as “face with curlyhair”) is calculated by the StyleCLIP model.

[0042] Step 2: Input the face image to be edited into the GAN Inversion model and output the W feature space encoding corresponding to the face image to be edited.

[0043] The extraction of feature space coding is achieved through the GAN Inversion model. Specifically, the face image to be edited is input into the GAN Inversion model which is fully trained on the real style dataset FFHQ, and the W feature space coding corresponding to the input face image to be edited is extracted, which is recorded as latent code.

[0044] Step 3: Fine-tune the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space encoding corresponding to the face image to be edited to obtain a fine-tuned generator.

[0045] First, if Figure 2 As shown in FIG. 1 , after obtaining the W feature space encoding latent code corresponding to the target editing direction of the image content through step 2, the W feature space encoding latent code corresponding to the target editing direction of the image content should be firstly subjected to data enhancement processing, so that the style fine-tuning can focus more on maintaining the consistency of the facial attributes between the input and output images, and improve the generalization ability of information such as hue. Multiple feature space encoding enhancement samples are generated through data enhancement operation, and the enhanced W feature space encoding, that is, multiple feature space encoding enhancement samples constitute a feature space encoding enhancement set, denoted as S = {latent code 0 , latent code 1 , latent code 2 ,…}.

[0046] Specifically, the data enhancement processing described in the present application is to weight the W feature space encoding latent code corresponding to the target editing direction of the image content with a certain weight and the average vector on several dimensions (usually 7-17 dimensions) that are less relevant to the editing content. Before each complete fine-tuning iteration of the StyleGAN model, the W feature space encoding latent code corresponding to the target editing direction of the image content is weighted with the average vector on several dimensions that are less relevant to the editing content, and multiple groups of feature space encoding enhancement samples are generated, where the i-th feature space encoding enhancement sample latent code i The enhancement process is:

[0047] First, determine the enhancement dimension k:

[0048]

[0049] Where |channel_swap|=k, channel_swap is the k random enhancement dimensions in the latent code encoded in the W feature space that are weakly correlated with the edited content; n latent Refers to the number of latent codes encoded in the W feature space in the StyleGAN model. In this application, n latent =18.

[0050] Then, the W feature space encoding latent code of the input face image to be edited is used to enhance the latent code of the i-th feature space encoding sample i To initialize:

[0051] latent code i =latent code (2)

[0052] Finally, the i-th feature space encoding after initialization enhances the sample latent code i Perform augmentation along a specified dimension:

[0053] latent code i [channel_swap]=

[0054] α×latent code[channel_swap]+(1-α)×mean_w[channel_swap](3)

[0055] Among them, α is the degree of sample enhancement; mean_w is the average code of the latent code encoded in the W feature space.

[0056] Based on the face image to be edited and the feature space coding enhancement set S, the model parameters of the generator in the StyleGAN model are fine-tuned to obtain a fine-tuned generator Generator' whose style information is consistent with the input face image to be edited. The model parameters of the generator in the StyleGAN model include weights and biases of all convolutional layers and fully connected layers. The perceptual loss (Learned Perceptual Image Patch Similarity Loss, LPIPS loss) based on 256 resolution between the image generated by the input StyleGAN model of each feature space coding enhancement sample in the feature space coding enhancement set S and the face image to be edited is calculated, and the model parameters of the generator in the StyleGAN model are optimized by gradient back propagation based on LPIPS loss to obtain the style fine-tuned generator Generator'.

[0057] Specifically, in each fine-tuning iterative process epoch, the fine-tuning loss function is LPIPS loss based on a resolution of 256. The model parameters of the generator in the StyleGAN model are optimized by gradient back propagation, so that the output result of each feature space encoding enhancement sample in the feature space encoding enhancement set S input into the StyleGAN model is close to the input stylized face image.

[0058] Step 4: Edit the face image to be edited according to the W feature space variation corresponding to the target editing direction of the image content, the W feature space encoding corresponding to the face image to be edited, and the fine-tuned generator to obtain the edited face image.

[0059] First, according to the W feature space variation Δlatent code corresponding to the target editing direction of the image content and the W feature space encoding latent code corresponding to the face image to be edited, the W feature space variation Δlatent code corresponding to the target editing direction of the image content is applied to the W feature space encoding latent code corresponding to the input face image to be edited, and the modified feature space encoding is obtained, which is recorded as latent code':

[0060] latent code'=latent code+β×Δlatent code (4)

[0061] Where β is the content editing strength. The content is edited according to the modified feature space encoding latent code'.

[0062] like Figure 3 As shown, the modified feature space encoding latent code' is input into the generator Generator' after style fine-tuning, and the face image to be edited is edited, so as to obtain an output face image with the same style as the face image to be edited and the edited content.

[0063] In summary, the stylized face image editing method of the present application first extracts the W feature space encoding latent code corresponding to the input stylized face picture (face image to be edited); secondly, one-shot style fine-tuning is performed on the input stylized face picture; next, the content of the input stylized face picture is edited based on the fine-tuned generator and the semantic information of the W feature space encoding latent code. The input of the stylized face image editing method of the present application is a single stylized face picture to be edited, and there is no need to retrain based on a large number of annotated pictures, and the output is a face picture of the same style after content editing. Therefore, the stylized face image editing method of the present application uses the prior information learned from the real style data set to guide the content editing of the input image, which improves the traditional method's dependence on a large amount of annotated data and high computational complexity, meets the needs of digital content creators for fast and efficient image generation, and can be widely used in face-related content creation scenarios. For example, character design, visual creativity, and entertainment applications.

[0064] Based on the stylized face image editing method, the present application also provides a stylized face image editing device, comprising:

[0065] The W feature space variation acquisition module is used to input the target editing direction of the image content into the StyleCLIP model and output the W feature space variation corresponding to the target editing direction of the image content.

[0066] The W feature space encoding acquisition module is used to input the face image to be edited into the GAN Inversion model and output the W feature space encoding corresponding to the face image to be edited.

[0067] A fine-tuning module is used to fine-tune the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space encoding corresponding to the face image to be edited, so as to obtain the fine-tuned generator; the model parameters of the generator in the StyleGAN model include the weights and biases of the convolutional layer and the fully connected layer.

[0068] The editing module is used to edit the face image to be edited according to the W feature space variation corresponding to the target editing direction of the image content, the W feature space encoding corresponding to the face image to be edited, and the fine-tuned generator to obtain the edited face image.

[0069] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0070] In some embodiments, the present application also provides a computer program product, including a computer program, which implements the stylized facial image editing method when executed by a processor.

[0071] In some embodiments, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the stylized facial image editing method when executed by a processor.

[0072] In some embodiments, the present application also provides a computer device, including a processor, a memory, an input / output interface (I / O for short), a communication interface, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the stylized facial image editing method.

[0073] The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the stylized face image editing method is implemented.

[0074] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0075] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ReadOnlyMemory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0076] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A stylized face image editing method, characterized in that: include: Input the target editing direction of the image content into the StyleCLIP model, and output the W feature space variation corresponding to the target editing direction of the image content; Input the face image to be edited into the GAN Inversion model, and output the W feature space encoding corresponding to the face image to be edited; Based on the face image to be edited and the W feature space encoding corresponding to the face image to be edited, the model parameters of the generator in the StyleGAN model are fine-tuned to obtain a fine-tuned generator; The model parameters of the generator in the StyleGAN model include weights and biases of the convolutional layer and the fully connected layer; According to the W feature space variation corresponding to the target editing direction of the image content, the W feature space encoding corresponding to the face image to be edited, and the fine-tuned generator, the face image to be edited is edited to obtain the edited face image.

2. The stylized face image editing method according to claim 1, characterized in that: The StyleCLIP model, the GAN Inversion model, and the StyleGAN model are all models that have been fully trained on the real style dataset FFHQ.

3. The stylized face image editing method according to claim 1, characterized in that: The fine-tuning of the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space encoding corresponding to the face image to be edited to obtain the fine-tuned generator specifically includes: Perform data enhancement processing on the W feature space codes corresponding to the face image to be edited, obtain multiple feature space code enhancement samples and form a feature space code enhancement set S; Based on the face image to be edited and the feature space encoding enhancement set S, the model parameters of the generator in the StyleGAN model are fine-tuned to obtain the fine-tuned generator.

4. The stylized face image editing method according to claim 3, characterized in that: The method of fine-tuning the model parameters of the generator in the StyleGAN model based on the face image to be edited and the feature space coding enhancement set S to obtain the fine-tuned generator specifically includes: Calculate the LPIPS loss based on 256 resolution between the image generated by the StyleGAN model for each feature space coding enhancement sample in the feature space coding enhancement set S and the face image to be edited, and optimize the model parameters of the generator in the StyleGAN model through gradient back propagation based on the LPIPS loss to obtain the fine-tuned generator Generator'.

5. The stylized face image editing method according to claim 4, characterized in that: The step of editing the face image to be edited according to the W feature space variation corresponding to the target editing direction of the image content, the W feature space encoding corresponding to the face image to be edited, and the fine-tuned generator to obtain the edited face image specifically includes: According to the W feature space variation Δlatent code corresponding to the target editing direction of the image content and the W feature space encoding latent code corresponding to the face image to be edited, the modified feature space encoding latent code' is obtained; The modified feature space encoding latent code' is input into the fine-tuned generator, and the face image to be edited is edited to obtain the edited face image.

6. The stylized face image editing method according to claim 5, characterized in that: The method of obtaining a modified feature space code latent code' according to the W feature space variation Δlatent code corresponding to the target editing direction of the image content and the W feature space code latent code corresponding to the face image to be edited specifically includes: According to the W feature space variation Δlatent code corresponding to the target editing direction of the image content and the W feature space encoding latent code corresponding to the face image to be edited, the modified feature space encoding latent code' is obtained using the formula latent code'=latent code+β×Δlatent code; wherein β is the content editing strength.

7. A stylized face image editing device, characterized in that: include: The W feature space variation acquisition module is used to input the target editing direction of the image content into the StyleCLIP model and output the W feature space variation corresponding to the target editing direction of the image content; The W feature space encoding acquisition module is used to input the face image to be edited into the GAN Inversion model and output the W feature space encoding corresponding to the face image to be edited; A fine-tuning module, used to fine-tune the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space encoding corresponding to the face image to be edited, so as to obtain a fine-tuned generator; The model parameters of the generator in the StyleGAN model include weights and biases of the convolutional layer and the fully connected layer; The editing module is used to edit the face image to be edited according to the W feature space variation corresponding to the target editing direction of the image content, the W feature space encoding corresponding to the face image to be edited, and the fine-tuned generator to obtain the edited face image.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the stylized facial image editing method described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the stylized facial image editing method described in any one of claims 1 to 6 is implemented.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the stylized facial image editing method according to any one of claims 1 to 6.

Images