A stylized face image editing method, device, product, medium and equipment

Abstract

The application discloses a style face image editing method, device, product, medium and equipment, and relates to the technical field of computer vision. The method comprises the following steps: inputting a picture content target editing direction and a face image to be edited into a StyleCLIP model and a GAN Inversion model respectively, and outputting a W feature space change amount corresponding to the picture content target editing direction and a W feature space code corresponding to the face image to be edited; fine-tuning model parameters of a generator in a StyleGAN model based on the face image to be edited and the W feature space code corresponding to the face image to be edited, so as to obtain a fine-tuned generator; and editing the face image to be edited according to the W feature space change amount, the W feature space code and the fine-tuned generator, so as to obtain an edited face image. The application can quickly generate a face image with unified style and rich content.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a stylized face image editing method, device, product, medium and equipment. BACKGROUND

[0002] Stylized face image editing has a wide range of applications in character design, visual creativity and entertainment fields, for example, designing more personalized character images through stylized face image editing or enriching the play of entertainment content through stylized face image editing. With the continuous development of the digital content creation industry, quickly generating stylized and content-rich personalized face images has become an important demand of content creators. However, traditional face image editing methods usually rely on a large amount of labeled data and complex model training processes, which are limited by data volume and computing power. These methods have certain limitations in processing specific stylized face image editing needs, and image editing requires a large amount of data and computation. To address this challenge, researchers have begun to focus on how to efficiently edit based on a small number of stylized samples to achieve flexible image content editing while maintaining image style consistency. SUMMARY

[0003] The purpose of the present application is to provide a stylized face image editing method, device, product, medium and equipment, which can quickly generate stylized and content-rich face images.

[0004] To achieve the above purpose, the present application provides the following solutions:

[0005] In a first aspect, the present application provides a stylized face image editing method, comprising:

[0006] inputting the picture content target editing direction into the StyleCLIP model to output the W feature space change amount corresponding to the picture content target editing direction;

[0007] inputting the face image to be edited into the GAN Inversion model to output the W feature space encoding corresponding to the face image to be edited;

[0008] fine-tuning the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space encoding corresponding to the face image to be edited, to obtain a fine-tuned generator; the model parameters of the generator in the StyleGAN model include the weights and biases of the convolutional layer and the fully connected layer;

[0009] editing the face image to be edited according to the W feature space change amount corresponding to the picture content target editing direction, the W feature space encoding corresponding to the face image to be edited, and the fine-tuned generator, to obtain an edited face image.

[0010] Optionally, the StyleCLIP model, the GAN Inversion model, and the StyleGAN model are all fully trained models on a real style dataset FFHQ.

[0011] Optionally, the model parameters of the generator in the StyleGAN model are fine-tuned based on the to-be-edited face image and the W feature space code corresponding to the to-be-edited face image to obtain a fine-tuned generator, and the fine-tuning specifically includes:

[0012] The W feature space code corresponding to the to-be-edited face image is subjected to data enhancement processing to obtain a plurality of feature space code enhancement samples and form a feature space code enhancement set S;

[0013] The model parameters of the generator in the StyleGAN model are fine-tuned based on the to-be-edited face image and the feature space code enhancement set S to obtain a fine-tuned generator.

[0014] Optionally, the model parameters of the generator in the StyleGAN model are fine-tuned based on the to-be-edited face image and the feature space code enhancement set S to obtain a fine-tuned generator, and the fine-tuning specifically includes:

[0015] The LPIPS loss based on 256 resolution between the image generated by each feature space code enhancement sample in the feature space code enhancement set S through the StyleGAN model and the to-be-edited face image is calculated, and the model parameters of the generator in the StyleGAN model are optimized through gradient back propagation based on the LPIPS loss to obtain a fine-tuned generator Generator'.

[0016] Optionally, the to-be-edited face image is edited based on the W feature space change corresponding to the target editing direction of the picture content, the W feature space code corresponding to the to-be-edited face image, and the fine-tuned generator to obtain an edited face image, and the editing specifically includes:

[0017] The modified feature space code latent code' is obtained according to the W feature space change corresponding to the target editing direction of the picture content and the W feature space code latent code corresponding to the to-be-edited face image;

[0018] The to-be-edited face image is edited by inputting the modified feature space code latent code' into the fine-tuned generator to obtain an edited face image.

[0019] Optionally, the W feature space change amount corresponding to the picture content target editing direction and the W feature space code corresponding to the face image to be edited are used to obtain a modified feature space code latent code' according to the formula latent code' = latent code + β × Δlatent code; wherein, β is the content editing intensity.

[0020] According to the W feature space change amount corresponding to the picture content target editing direction and the W feature space code corresponding to the face image to be edited, a modified feature space code latent code' is obtained by using the formula latent code' = latent code + β × Δlatent code; wherein, β is the content editing intensity.

[0021] In a second aspect, the present application provides a style face image editing device, comprising:

[0022] The W feature space change amount obtaining module is configured to input the picture content target editing direction into the StyleCLIP model, and output the W feature space change amount corresponding to the picture content target editing direction.

[0023] The W feature space code obtaining module is configured to input the face image to be edited into the GAN Inversion model, and output the W feature space code corresponding to the face image to be edited.

[0024] The fine-tuning module is configured to fine-tune the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space code corresponding to the face image to be edited, and obtain a fine-tuned generator; the model parameters of the generator in the StyleGAN model include the weights and biases of the convolutional layer and the fully connected layer.

[0025] The editing module is configured to edit the face image to be edited according to the W feature space change amount corresponding to the picture content target editing direction, the W feature space code corresponding to the face image to be edited, and the fine-tuned generator, and obtain an edited face image.

[0026] In a third aspect, the present application provides a computer program product, comprising a computer program which is executed by a processor to implement the style face image editing method.

[0027] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the style face image editing method.

[0028] In a fifth aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the style face image editing method.

[0029] According to the specific embodiments provided in the present application, the present application has the following technical effects:

[0030] The present application provides a style face image editing method, device, product, medium and equipment, mainly relates to a style face image editing method based on StyleGAN, which represents the content of the face image to be edited by obtaining and using the W feature space code of the face image to be edited, and completes the content editing based on the W feature space code and the W feature space change amount. At the same time, based on the W feature space code corresponding to the face image to be edited and the face image to be edited, the model parameters of the generator in the StyleGAN model are fine-tuned to ensure that the generated face image is consistent in style while the content is edited. The style face image editing method of the present application effectively improves the flexibility and efficiency of the style face image editing, and meets the needs of digital content creators. Therefore, the style face image editing method, device, product, medium and equipment of the present application can quickly generate face images with unified style and rich content. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0032] Figure 1 The flowchart of the style face image editing method provided by the present application is shown in the figure.

[0033] Figure 2 The principle diagram of the fine-tuning of the StyleGAN model provided by the present application is shown in the figure.

[0034] Figure 3 The principle diagram of the fine-tuning of the StyleGAN model provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0035] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0036] In traditional face image editing methods, there are obvious limitations in retraining for a stylized sample dataset. Therefore, researchers have begun to try to use prior information learned on a real style dataset to guide the editing of stylized face images. The difficulty of this process is how to ensure the decoupling between image content and style, that is, how to maintain the consistency of the style while editing the content. The realization of this balance is the key to editing stylized face images. Therefore, the present application provides a stylized face image editing method, device, product, medium and equipment, which can quickly generate face images with unified style and rich content.

[0037] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0038] As shown in Figure 1 The stylized face image editing method disclosed by the present application comprises:

[0039] Firstly, the stylized face image editing method of the present application requires the prior conditions to be obtained in advance, that is, the StyleCLIP model, the GAN Inversion model and the StyleGAN model which have been fully trained on the real style dataset FFHQ. The subsequent data processing is carried out through the above-mentioned models. Among them, the GAN Inversion model and the StyleGAN model have multiple versions, and the specific versions used in the present application are encoder4editing and StyleGAN 2 respectively.

[0040] Step 1: input the picture content target editing direction into the StyleCLIP model, and output the W feature space change corresponding to the picture content target editing direction.

[0041] Specifically, the picture content target editing direction is input into the StyleCLIP model, and the W feature space change Δlatent code corresponding to the picture content target editing direction (such as "face with curly hair") is calculated by the StyleCLIP model.

[0042] Step 2: input the face image to be edited into the GAN Inversion model, and output the W feature space code corresponding to the face image to be edited.

[0043] The feature space code is extracted by the GAN Inversion model. Specifically, by inputting the face image to be edited into the GAN Inversion model trained on the real style dataset FFHQ, the W feature space code corresponding to the input face image to be edited is extracted, denoted as latent code.

[0044] Step 3: fine-tune the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space code corresponding to the face image to be edited, and obtain the fine-tuned generator.

[0045] First, as shown in Figure 2 After obtaining the W feature space code latent code corresponding to the target editing direction of the picture content by step 2, the W feature space code latent code corresponding to the target editing direction of the picture content should be processed by data augmentation to make the style fine-tuning pay more attention to keeping the consistency of face attributes between the input and output images, and improve the generalization ability to information such as color tone. Through data augmentation operation, multiple feature space code enhanced samples are generated, and the enhanced W feature space code, i.e. multiple feature space code enhanced samples, constitute a feature space code enhanced set, denoted as S = {latent code0, latent code1, latent code2, …}.

[0046] Specifically, the data enhancement processing described in the present application is specifically to weight the W feature space code latent code corresponding to the target editing direction of the picture content with an average vector in several dimensions (usually 7-17 dimensions) with weak relevance to the editing content. Before each complete fine-tuning iteration of the StyleGAN model, the W feature space code latent code corresponding to the target editing direction of the picture content is weighted with an average vector in several dimensions with weak relevance to the editing content, to generate multiple groups of feature space code enhanced samples, wherein the i-th feature space code enhanced sample latent code i The enhancement process of the i-th feature space code enhanced sample latent code

[0047] First, determine the enhancement dimension k:

[0048]

[0049] wherein, |channel_swap| = k, channel_swap is k random augmented dimensions in the W feature space encoding latent code which are weakly related to the editing content; n latent denotes the number of W feature space encoding latent code in StyleGAN model, in the present application, n latent = 18.

[0050] Subsequently, the i-th feature space encoding augmented sample latent code i is initialized by the input W feature space encoding latent code of the face image to be edited:

[0051] latent code i = latent code (2)

[0052] Finally, the initialized i-th feature space encoding augmented sample latent code i is augmented in the specified dimensions:

[0053] latent code i [channel_swap] = latent code[channel_swap] (3)

[0054] α × latent code[channel_swap] + (1 - α) × mean_w[channel_swap] (3)

[0055] wherein, α is the sample augmentation degree; mean_w is the average encoding of the W feature space encoding latent code.

[0056] Based on the face image to be edited and the feature space encoding augmentation set S, the model parameters of the generator (Generator) in the StyleGAN model are fine-tuned to obtain a fine-tuned generator Generator'. The model parameters of the generator in the StyleGAN model include the weights and biases of all convolutional layers and fully connected layers. By calculating the perceptual loss (Learned Perceptual Image Patch Similarity Loss, LPIPS loss) between the image generated by inputting each feature space encoding augmented sample in the feature space encoding augmentation set S into the StyleGAN model and the face image to be edited based on 256 resolution, and optimizing the model parameters of the generator in the StyleGAN model based on the LPIPS loss through gradient backpropagation, a style fine-tuned generator Generator' is obtained.

[0057] Specifically, in each fine-tuning iteration epoch, the loss function is LPIPSloss based on 256 resolution. The model parameters of the generator in the StyleGAN model are optimized through gradient backpropagation, so that the output of the StyleGAN model for each feature space encoded augmented sample in the feature space encoded augmentation set S is close to the input stylized face image.

[0058] Step 4: Based on the W feature space change corresponding to the target editing direction of the image content, the W feature space encoding corresponding to the face image to be edited, and the fine-tuned generator, edit the face image to be edited to obtain the edited face image.

[0059] First, based on the W feature space change amount Δlatent code corresponding to the image content's target editing direction and the W feature space encoding latent code corresponding to the face image to be edited, the W feature space change amount Δlatent code corresponding to the image content's target editing direction is applied to the W feature space encoding latent code corresponding to the input face image to be edited, resulting in the modified feature space encoding, denoted as latent code'.

[0060] latent code'=latent code+β×Δlatent code (4)

[0061] Where β represents the content editing intensity. Content editing is performed based on the obtained modified feature space encoding, the latent code'.

[0062] like Figure 3 As shown, the modified latent code is input into the style-fine-tuned generator to edit the face image to be edited, and the output face image with the same style as the face image to be edited and the content is edited is obtained.

[0063] In summary, the stylized face image editing method of this application first extracts the latent code corresponding to the W feature space encoding of the input stylized face image (the face image to be edited); secondly, it performs one-shot style fine-tuning on the input stylized face image; and thirdly, it edits the content of the input stylized face image based on the semantic information of the fine-tuned generator and the W feature space encoding latent code. The input of this stylized face image editing method is a single stylized face image to be edited, eliminating the need for retraining based on a large number of labeled images, and the output is a face image with the same style after content editing. Therefore, this stylized face image editing method utilizes prior information learned from real-style datasets to guide the content editing of the input image, improving the dependence of traditional methods on large amounts of labeled data and high computational costs. It meets the needs of digital content creators for fast and efficient image generation and can be widely applied to face-related content creation scenarios, such as character design, visual creativity, and entertainment applications.

[0064] Based on the stylized face image editing method described above, this application also provides a stylized face image editing apparatus, comprising:

[0065] The W feature space change acquisition module is used to input the image content target editing direction into the StyleCLIP model and output the W feature space change corresponding to the image content target editing direction.

[0066] The W feature space encoding acquisition module is used to input the face image to be edited into the GAN Inversion model and output the W feature space encoding corresponding to the face image to be edited.

[0067] The fine-tuning module is used to fine-tune the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space encoding corresponding to the face image to be edited, so as to obtain the fine-tuned generator; the model parameters of the generator in the StyleGAN model include the weights and biases of the convolutional layer and the fully connected layer.

[0068] The editing module is used to edit the face image based on the W feature space change corresponding to the target editing direction of the image content, the W feature space encoding corresponding to the face image to be edited, and the fine-tuned generator, so as to obtain the edited face image.

[0069] In this application, all actions to acquire signals, information, or data are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.

[0070] In some embodiments, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the stylized face image editing method described above.

[0071] In some embodiments, this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the stylized face image editing method described above.

[0072] In some embodiments, this application also provides a computer device, including a processor, a memory, an input / output interface (I / O), a communication interface, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the stylized face image editing method described above.

[0073] The processor, memory, and input / output interface are connected via a system bus, and the communication interface is also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores pending transactions. The input / output interface allows the processor to exchange information with external devices. The communication interface allows communication with external terminals via a network connection. When executed by the processor, the computer program implements the stylized face image editing method.

[0074] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0075] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0076] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A style-based face image editing method, characterized by, The application comprises the following steps: inputting the picture content target editing direction into the StyleCLIP model to output the W feature space change corresponding to the picture content target editing direction; inputting the face image to be edited into the GAN Inversion model to output the W feature space code corresponding to the face image to be edited; based on the face image to be edited and the W feature space code corresponding to the face image to be edited, fine-tuning the model parameters of the generator in the StyleGAN model to obtain a fine-tuned generator. The model parameters of the generator in the StyleGAN model include the weights and biases of the convolutional layer and the fully connected layer. Based on the face image to be edited and the W feature space code corresponding to the face image to be edited, the model parameters of the generator in the StyleGAN model are fine-tuned to obtain a fine-tuned generator, which specifically includes: performing data enhancement processing on the W feature space code corresponding to the face image to be edited to obtain a plurality of feature space code enhancement samples and form a feature space code enhancement set S; based on the face image to be edited and the feature space code enhancement set S, the model parameters of the generator in the StyleGAN model are fine-tuned to obtain a fine-tuned generator; the data enhancement processing specifically includes weighting the W feature space code latent code corresponding to the picture content target editing direction by a certain weight and the average vector in a plurality of dimensions with weak relevance to the editing content; before each complete fine-tuning iteration process of the StyleGAN model, the W feature space code latent code corresponding to the picture content target editing direction is weighted by a certain weight and the average vector in a plurality of dimensions with weak relevance to the editing content to generate a plurality of feature space code enhancement samples; According to the W feature space change corresponding to the picture content target editing direction, the W feature space code corresponding to the face image to be edited, and the fine-tuned generator, the face image to be edited is edited to obtain an edited face image.

2. The stylized face image editing method of claim 1, wherein, The StyleCLIP model, the GAN Inversion model, and the StyleGAN model are all fully trained models on the real style dataset FFHQ.

3. The stylized face image editing method of claim 1, wherein, The fine-tuned generator is obtained by fine-tuning the model parameters of the generator in the StyleGAN model based on the face image to be edited and the feature space code enhancement set S, which specifically includes: calculating the LPIPS loss based on 256 resolution between the image generated by each feature space code enhancement sample in the feature space code enhancement set S through the StyleGAN model and the face image to be edited, and optimizing the model parameters of the generator in the StyleGAN model through gradient back propagation based on the LPIPS loss to obtain a fine-tuned generator Generator'.

4. The stylized face image editing method of claim 3, wherein, The edited face image is obtained by editing the face image to be edited according to the W feature space change corresponding to the picture content target editing direction, the W feature space code corresponding to the face image to be edited, and the fine-tuned generator, which specifically includes: According to the W feature space change amount Δlatent code corresponding to the picture content target editing direction and the W feature space code latent code corresponding to the face image to be edited, a modified feature space code latent code' is obtained. The modified feature space code latent code' is input into the fine-tuned generator to edit the face image to be edited, and an edited face image is obtained.

5. The stylized face image editing method of claim 4, wherein, According to the W feature space change amount Δlatent code corresponding to the picture content target editing direction and the W feature space code latent code corresponding to the face image to be edited, a modified feature space code latent code' is obtained. According to the W feature space change amount Δlatent code corresponding to the picture content target editing direction and the W feature space code latent code corresponding to the face image to be edited, a modified feature space code latent code' is obtained.

6. A style-oriented face image editing apparatus, characterized by comprising: It comprises: The W feature space change amount acquisition module is used for inputting the picture content target editing direction into the StyleCLIP model, and outputting the W feature space change amount corresponding to the picture content target editing direction; The W feature space code acquisition module is used for inputting the face image to be edited into the GAN Inversion model, and outputting the W feature space code corresponding to the face image to be edited; The fine-tuning module is used for fine-tuning the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space code corresponding to the face image to be edited, and obtaining a fine-tuned generator; The model parameters of the generator in the StyleGAN model include the weights and biases of the convolution layer and the full connection layer; the fine-tuning of the model parameters of the generator in the StyleGAN model based on the face image to be edited and the W feature space code corresponding to the face image to be edited to obtain the fine-tuned generator, specifically includes: data enhancement processing is performed on the W feature space code corresponding to the face image to be edited to obtain a plurality of feature space code enhancement samples and form a feature space code enhancement set S; the model parameters of the generator in the StyleGAN model are fine-tuned based on the face image to be edited and the feature space code enhancement set S to obtain the fine-tuned generator; the data enhancement processing is specifically that, in a plurality of dimensions with weak correlation with the editing content, the W feature space code latent code corresponding to the picture content target editing direction is weighted with an average vector according to a certain weight; before each complete fine-tuning iteration process of the StyleGAN model, the W feature space code latent code corresponding to the picture content target editing direction is weighted with an average vector according to a certain weight in a plurality of dimensions with weak correlation with the editing content, and a plurality of groups of feature space code enhancement samples are generated; The editing module is configured to edit the to-be-edited face image according to the W feature space variation corresponding to the target editing direction of the picture content, the W feature space code corresponding to the to-be-edited face image, and the fine-tuned generator, to obtain an edited face image.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the style face image editing method in any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the style face image editing method in any one of claims 1-5.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the style face image editing method in any one of claims 1-5.

Images