Aesthetic-guided Facial Intelligent Beautification Method and System

Abstract

The present invention relates to a method for intelligent beautification of human faces guided by aesthetics, aiming to achieve an adaptive intelligent beautification effect of human faces through the guidance of a human face beauty evaluation model. The present invention mainly includes the following steps: 1) Generating a latent code: The encoder encodes the image to generate a low-dimensional latent code representing the image content information; 2) Image reconstruction: The generator maps the low-dimensional latent code into a reconstructed image; 3) Inversion of the generator: The human face beauty evaluation model calculates the distance loss between the beauty degree of the reconstructed image and the target beauty degree, and the generator is guided to invert by this loss, thereby updating the latent code. Among them, the second step and the third step are continuously iterated until the algorithm converges. Through the guidance of the aesthetic information of the human face beauty evaluation model and the inversion technology of StyleGAN, the present invention modifies and beautifies the human face in the latent code space, so that the reconstructed image obtains an intelligent beautification effect.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and particularly to a method and system for intelligent face beautification guided by aesthetics. Background Art

[0002] Currently, existing face beautification algorithms are usually based on skin texture or facial feature contour beautification algorithms respectively. Among them, skin texture beautification aims to reduce facial blemishes or increase skin luster, generally by designing a smoothing operator to filter the image or transferring its makeup to the target face according to a reference face. And the beautification algorithm for facial feature contour aims to obtain a more attractive face structure through geometric deformation, generally by matching the facial key points of the target face with the key point template of an average face, or using a deep model to learn a more attractive facial key point template. However, existing face beautification algorithms rarely consider whether these beautifications conform to people's aesthetic cognition. For example, the attractiveness of some highly attractive faces decreases after beautification; on the other hand, most algorithms do not consider the personal preferences of users. For example, some users need a more obvious face beautification effect, while some users hope to retain more personal features, and most technologies are difficult to achieve the customized intelligent face beautification effect according to user needs. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method and system for intelligent face beautification guided by aesthetics, which solves the problems that existing face beautification technologies lack the guidance of aesthetic information and cannot consider the personal preferences of users.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions:

[0005] A method for intelligent face beautification guided by aesthetics, comprising the following steps:

[0006] Step S1: Encode the image to generate a low-dimensional latent code representing the image content information;

[0007] Step S2: Map the low-dimensional latent code into a reconstructed image;

[0008] Step S3: Calculate the distance loss between the beauty degree of the reconstructed image and the target beauty degree based on the face beauty evaluation model, and guide the generator to invert by this loss, so as to update the latent code

[0009] Step S4: Loop steps S2 and S3 until the preset requirements are met to obtain the beautified image.

[0010] Further, step S1 is specifically: encoding each input image x into a low-dimensional latent code ω through an encoder E with pre-trained weight θ e c∈W:

[0011] ω c = E(x|θ e )

[0012] where ω c (ω c ∈R 512 ) is a content latent code that encodes rough information of facial content.

[0013] Furthermore, the specific steps of step S2 are as follows:

[0014] Step S2.1: Input the low-dimensional latent code ω c into the pre-trained generator G with weight θ g for facial reconstruction to generate a coarse-grained image x ω :

[0015] x ω = G(ω c |θ g )

[0016] Step S2.2: Use the pre-trained encoder T with weight θ t to jointly encode the image x and x ω into an appearance latent code ω a :

[0017] ω a = T(x, x w |θ t )

[0018] Step S2.3: Combine the content latent code ω c and the appearance latent code ω a together. By adding ω a as the dynamic residual weight to the weight θ g of the generator, the generator G generates a reconstructed image

[0019]

[0020] Furthermore, the specific steps of step S3 are as follows:

[0021] Step S3.1: Calculate the loss between the finally generated reconstructed image and the source face x as follows:

[0022]

[0023] where η fbp , η rec , η per , η verrespectively represent the weights of each part of the loss; represents the beautification loss, which measures the loss of the gap between the predicted score of the face beauty evaluation model for the beautified image and the target score input by the user;

[0024]

[0025] represents the image reconstruction loss, which measures the pixel-level similarity between the input image x and the beautified image :

[0026]

[0027] represents the face perception loss, which measures the feature-level similarity between the input image x and the beautified image and extracts features through a pre-trained VGG model as the perceptual feature extractor h(.|θ h );

[0028]

[0029] represents the face verification loss, which measures the cosine similarity of the face identity features between the input image x and the beautified image and extracts the face identity features through a pre-trained ArcFace face recognition model v(.|θ v );

[0030]

[0031] Step S3.2: Use the loss function to guide the generator inversion to optimize the content latent code towards the target score input by the user, calculate the gradient of the content latent code and update it:

[0032]

[0033] where γ is the learning rate.

[0034] A face intelligent beautification system based on aesthetic guidance includes an encoder E, a generator G, an encoder T, a face beauty evaluation module, a face verification module, and a visual perception module. The encoder E is used to map an image into a content latent code in the hidden layer, and its weights are inherited from HyperInverter. The generator G is used in the image reconstruction process to map the content latent code into an image, and its weights are inherited from the StyleGAN-v2 model. The encoder T is used to map the reconstructed image into an apparent encoding in the hidden layer, and its weights are inherited from HyperInverter. The face beauty evaluation module is pre-trained on a face beauty dataset and is used to predict the face beauty degree of the reconstructed image. The face verification module is pre-trained on a face recognition dataset and is used to verify the identity of the reconstructed face. The visual perception module is pre-trained on an image classification dataset and is used to calculate the perceptual loss.

[0035] The present invention has the following beneficial effects compared with the prior art:

[0036] The present invention can adaptively adjust the beautification degree of the algorithm according to the beautification degree preset by the user, so as to achieve the user-adaptive face intelligent beautification effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic flow chart of the method of the present invention;

[0038] Figure 2 is a schematic diagram of the face beautification model of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0040] Please refer to Figure 2, the present invention provides a face intelligent beautification system based on aesthetic guidance, including an encoder E, a generator G, an encoder T, a face beauty evaluation module, a face verification module, and a visual perception module; the encoder E is used to map an image into a content latent code in the hidden layer, and its weights are inherited from HyperInverter; the generator G is used in the image reconstruction process to map the content latent code into an image, and its weights are inherited from the StyleGAN-v2 model; the encoder T is used to map the reconstructed image into a hidden layer appearance code, and its weights are inherited from HyperInverter; the face beauty evaluation module is a face beauty prediction model pre-trained on the face beauty dataset SCUT-FBP5500, which is used to predict the face beauty degree of the reconstructed image; the face verification module is an ArcFace model pre-trained through a face recognition dataset, which is used to verify the identity of the reconstructed face; the visual perception module is a VGG model pre-trained through an image classification dataset, which is used to calculate the perceptual loss.

[0041] In this embodiment, refer to Figure 1 , and also provides a face intelligent beautification method based on aesthetic guidance, including the following steps:

[0042] Step S1: Encode the image to generate a low-dimensional latent code representing the image content information;

[0043] In this embodiment, step S1 encodes each input image x into a low-dimensional latent code ω through the pre-trained encoder E with weights θ e : c ∈W:

[0044] ω c = E(x|θ e )

[0045] where ω c (ω c ∈R 512 ) is a content latent code, which encodes the rough information of the facial content

[0046] Step S2: Map the low-dimensional latent code into a reconstructed image;

[0047] In this embodiment, step S2 is specifically:

[0048] Step S2.1: Input the low-dimensional latent code ω c into the pre-trained generator G with weights θ g for facial reconstruction to generate a coarse-grained image x ω :

[0049] x ω = G(ω c |θg )

[0050] Step S2.2: Use the pre-trained encoder T with weights θ t to jointly encode the images x and x ω into the appearance latent code ω a :

[0051] ω a = T(x, x w | θ t )

[0052] Step S2.3: Combine the content latent code ω c and the appearance latent code ω a by adding ω a as the dynamic residual weights to the weights θ g of the generator G, so that the generator G generates the reconstructed image

[0053]

[0054] Step S3: Calculate the distance loss between the beauty score of the reconstructed image and the target beauty score based on the face beauty evaluation model, and guide the generator inversion by this loss to update the latent code;

[0055] In this embodiment, Step S3 is specifically:

[0056] Step S3.1: Calculate the loss for the finally generated reconstructed image and the source face x as follows:

[0057]

[0058] where η fbp , η rec , η per , η ver respectively represent the weights of each part of the loss; represents the beautification loss, which measures the loss of the gap between the predicted score of the face beauty evaluation model for the beautified image and the target score input by the user;

[0059]

[0060] represents the image reconstruction loss, which measures the pixel-level similarity between the input image x and the beautified image :

[0061]

[0062] Denotes the face perception loss, which measures the feature-level similarity between the input image x and the beautified image and extracts features through a pre-trained VGG model as the perceptual feature extractor h(.|θ h ):

[0063]

[0064] Denotes the face verification loss, which measures the cosine similarity of the face identity features between the input image x and the beautified image and extracts the face identity features through a pre-trained ArcFace face recognition model v(.|θ v ):

[0065]

[0066] Step S3.2: Use the loss function to guide the generator inversion to optimize the content latent code towards the target score input by the user, calculate the gradient of the content latent code and update it:

[0067]

[0068] where γ is the learning rate.

[0069] The updated content latent code follows the order (refer to Step S2.1, Step 2.2, Step 2.3) to generate the reconstructed image from coarse-grained to fine-grained.

[0070] Step S4: Loop Steps S2 and S3 until the beautification loss reaches a certain threshold to obtain the beautified image.

[0071] The above are only the preferred embodiments of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the scope covered by the present invention.

Claims

1. A face intelligent beautification method based on aesthetic guidance, characterized in that, it includes the following steps: Step S1: Encode the image to generate a low-dimensional latent code representing the image content information; Step S2: Map the low-dimensional latent code into a reconstructed image; Step S3: Calculate the distance loss between the beauty degree of the reconstructed image and the target beauty degree based on the face beauty evaluation model, and guide the generator to invert by this loss, so as to update the latent code; Step S4: Loop steps S2 and S3 until the preset requirements are met to obtain the beautified image; The specific content of step S3 is: Step S3.1: Calculate the loss for the finally generated reconstructed image and the source face x as follows: where η fbp , η rec , η per , η ver represent the weights of the respective losses; represents the beautification loss, which measures the loss of the gap between the predicted score of the face beauty evaluation model for the beautified image and the target score input by the user; Represents the image reconstruction loss, which measures the pixel-level similarity between the input image x and the beautified image : Denotes the face perception loss, which measures the feature-level similarity between the input image x and the beautified image , and extracts features through a pre-trained VGG model as the perception feature extractor h(.|θ h ): Denotes the face verification loss, which measures the cosine similarity of the face identity features between the input image x and the beautified image , and extracts the face identity features through a pre-trained ArcFace face recognition model v(.|θ v ): Step S3.2: Use the loss function to guide the generator to invert, optimize the content latent code towards the target score input by the user, calculate the gradient of the content latent code and update it: where γ is the learning rate.

2. The face intelligent beautification method based on aesthetic guidance according to claim 1, characterized in that, The specific content of step S1 is as follows: Each input image x is encoded into a low-dimensional latent code ω e by an encoder E with pre-trained weight θ c ∈ W: ω c = E(x|θ e ) where ω c ∈R 512 is a content latent code that encodes rough information about the facial content.

3. The face intelligent beautification method based on aesthetic guidance according to claim 1, characterized in that, The specific content of step S2 is: Step S2.1: Input the low-dimensional latent code ω c into the pre-trained generator G with weights θ g for face reconstruction to generate a coarse-grained image x ω : x ω = G(ω c |θ g ) Step S2.2: Use the encoder T with pre-trained weights θ t to jointly encode the images x and x ω into the appearance latent code ω a : ω a = T(x, x ω |θ t ) Step S2.3: Combine the content latent code ω c and the appearance latent code ω a together by adding ω a as the dynamic residual weight to the weight θ g of the generator, so that the generator G generates the reconstructed image 4. A face intelligent beautification system based on aesthetic guidance, characterized in that, the system is implemented by using the face intelligent beautification method according to any one of claims 1-3, and includes an encoder E, a generator G, an encoder T, a face beauty evaluation module, a face verification module and a visual perception module; the encoder E is used to map the picture into the content latent code of the hidden layer, and its weight is inherited from HyperInverter; the generator G is used for the picture reconstruction process, maps the content latent code into a picture, and its weight is inherited from the StyleGAN-v2 model; the encoder T is used to map the reconstructed picture into the hidden layer appearance encoding, and its weight is inherited from HyperInverter; the face beauty evaluation module is a face beauty prediction model pre-trained on the face beauty dataset SCUT-FBP5500, and is used to predict the face beauty degree of the reconstructed picture; the face verification module is an ArcFace model pre-trained through the face recognition dataset, and is used to verify the identity of the reconstructed face; the visual perception module is a VGG model pre-trained through the image classification dataset, and is used to calculate the perceptual loss.

Images