Face image generation method and device with face attribute

Abstract

The application relates to the field of IT applications and provides a face image generation method and device with face attributes and application thereof. The method comprises the following steps: obtaining a target face attribute image and a face image to be edited; inputting the target face attribute image and the face image to be edited into a face image generation network to obtain a target face image generated by the face image generation network; the face image generation network is used for generating the target face image based on the target face attribute image and the face image to be edited; wherein the face image generation network is trained based on a face sample image set, a feature face sample image set and a sample face attribute image extracted from the feature face sample image set. The method provided in the application can solve the technical problem that the appearance form of the face attribute on the generated face image is uncontrollable, and the generation efficiency of the face image with the face attribute is improved.

Description

Technical Field

[0001] This application relates to the field of IT application technology, specifically to a method and apparatus for generating faces with facial attributes. Background Technology

[0002] Currently, Cycle GANs are the first generative adversarial networks applied to image style transfer, achieving style transfer between unpaired images and demonstrating promising results. They are widely used to generate face images with facial attributes, such as adding freckles, adding glasses, or changing hairstyles. However, currently available datasets contain relatively few images of faces wearing glasses, making it difficult to meet the large image data requirements for training face attribute algorithms. Therefore, Cycle GANs have become a common technique for solving the problem of generating face image data with facial attributes.

[0003] Existing Recurrent Generative Adversarial Networks (RGANs) typically employ two generators, one for generating faces with facial attributes and the other for generating faces without facial attributes. Simultaneously, to counteract the generators, two discriminators are included, one for generating faces with facial attributes and the other for generating faces without facial attributes. These discriminators determine whether the input image is a real image or a generated image. This, combined with a recurrent consistency loss function and a discriminant loss function, completes the training of the RGAN, enabling it to generate faces with facial attributes.

[0004] The aforementioned prior art has the following disadvantages:

[0005] Since this scheme does not input image information about facial attributes, the appearance of facial attributes in the generated facial images is uncontrollable, affecting the stability and diversity of facial images and reducing the generation efficiency of facial images with facial attributes. Summary of the Invention

[0006] This application provides a method for generating face images with facial attributes to solve the technical problem of uncontrollable appearance of facial attributes in the generated face images.

[0007] In a first aspect, embodiments of this application provide a method for generating a face image with facial attributes, including:

[0008] Obtain the target face attribute image and the face image to be edited;

[0009] Input the target face attribute image and the face image to be edited into the face image generation network to obtain the target face image generated by the face image generation network;

[0010] The face image generation network is trained based on a set of face sample images, a set of feature face sample images, and sample face attribute images extracted from the set of feature face sample images.

[0011] In one embodiment, the face image generation network is trained as follows:

[0012] Obtain a set of face sample images and a set of feature face sample images;

[0013] Each face sample image in the face sample image set is paired with each feature face sample image in the feature face sample image set to obtain multiple sets of paired images.

[0014] Extract the facial attribute appearance features from the feature facial sample images of each pair of images to obtain the sample facial attribute images corresponding to each pair of images;

[0015] Construct a face image generation network;

[0016] Each pair of paired images and the corresponding sample face attribute images are sequentially input into the face image generation network to train the face image generation network.

[0017] In one embodiment, each pair of paired images and the corresponding sample face attribute images are sequentially input into a face image generation network to train the face image generation network. The process of inputting a pair of paired images and the corresponding sample face attribute images into the face image generation network for training includes:

[0018] The face sample image and the sample face attribute image in the paired image are input into the face image generation network to obtain the first one-way loss function value of the face image generation network;

[0019] The feature face sample image and the sample face attribute image in the paired image are input into the face image generation network to obtain the second one-way loss function value of the face image generation network;

[0020] The total value of the loss function is determined based on the first one-way loss function value and the second one-way loss function value;

[0021] The model parameters of the face image generation network are updated based on the total value of the loss function.

[0022] In one embodiment, the face image generation network includes a recurrent generative adversarial network and a similarity calculation network, wherein the similarity calculation network is used to determine the degree of similarity between the facial attribute appearance features in the face attribute image and the face generated image.

[0023] The face sample images and sample face attribute images from the paired images are input into the face image generation network to obtain the first one-way loss function value of the face image generation network, including:

[0024] Input the face sample image and the sample face attribute image in the current paired image into the first generator of the recurrent generative adversarial network to obtain the first face training generated image output by the first generator;

[0025] The first face training image is input into the first discriminator of the recurrent generative adversarial network to obtain the first training probability value output by the first discriminator.

[0026] The first face training image is input into the similarity calculation network to obtain the first training similarity value output by the similarity calculation network;

[0027] The first face training image is input into the second generator of the recurrent generative adversarial network to obtain the first face training reconstructed image output by the second generator.

[0028] The first one-way loss function value is determined based on the first face training generated image, the first training probability value, the first training similarity value, and the first face training reconstructed image.

[0029] In one embodiment, the feature face sample image and the sample face attribute image from the paired image are input into the face image generation network to obtain the second one-way loss function value of the face image generation network, including:

[0030] The feature face sample image in the current paired image is input into the second generator of the recurrent generative adversarial network to obtain the second face training generated image output by the second generator;

[0031] The training image of the second face is input into the second discriminator of the recurrent generative adversarial network to obtain the second training probability value output by the second discriminator.

[0032] The second face training image and the sample face attribute image are input into the first generator to obtain the second face training reconstructed image output by the first generator.

[0033] The reconstructed image of the second face is input into the similarity calculation network to obtain the second training similarity value output by the similarity calculation network.

[0034] The second one-way loss function value is determined based on the second face training generated image, the second training probability value, the second training similarity value, and the second face training reconstructed image.

[0035] In one embodiment, determining the first unidirectional loss function value based on the first face training generated image, the first training probability value, the first training similarity value, and the first face training reconstructed image includes:

[0036] The first face training generated image, the first training probability value, the first training similarity value, and the first face training reconstructed image are input into the first one-way loss function to obtain the first one-way loss function value;

[0037] The first one-way loss function is:

[0038] ;

[0039] in, The output image of the first generator. The output image of the second generator. A set of facial sample images, For feature face sample images, A collection of sample facial attribute images. For sample face attribute images, The output value of the first discriminator. The network output value is calculated for similarity.

[0040] The method for constructing the first one-way loss function specifically includes:

[0041] A first discriminant function is constructed based on the sample facial attribute images. The first discriminant function is:

[0042] ;

[0043] in, For facial sample images, The first discriminator's discrimination value for the feature face sample image is denoted as . This is the first training probability value;

[0044] A second discriminant function is constructed based on the similarity calculation network. The second discriminant function is as follows:

[0045] ;

[0046] in, This is the first training similarity value;

[0047] Summing the first discriminant function with the second discriminant function yields the discriminant loss function;

[0048] A cycle consistency loss function is constructed based on the sample face attribute images. The cycle consistency loss function is as follows:

[0049] ;

[0050] in, Train and reconstruct images for the first face;

[0051] The first one-way loss function is obtained by summing the discriminative loss function and the cycle consistency loss function.

[0052] In one embodiment, determining the second unidirectional loss function value based on the second face training generated image, the second training probability value, the second training similarity value, and the second face training reconstructed image includes:

[0053] The second face training generated image, the second training probability value, the second training similarity value, and the second face training reconstructed image are input into the second one-way loss function to obtain the value of the second one-way loss function.

[0054] The second one-way loss function is:

[0055] ;

[0056] in, This is the output value of the second discriminator.

[0057] Secondly, embodiments of this application provide a face image generation apparatus with facial attributes, comprising:

[0058] The image acquisition module is used to acquire the target face attribute image and the face image to be edited;

[0059] The face image generation module is used to input the target face attribute image and the face image to be edited into the face image generation network to obtain the target face image generated by the face image generation network;

[0060] A face image generation network is used to generate a target face image based on a target face attribute image and a face image to be edited;

[0061] The face image generation network is trained based on a set of face sample images, a set of feature face sample images, and sample face attribute images extracted from the set of feature face sample images.

[0062] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the face image generation method with facial attributes described in the first aspect.

[0063] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the face image generation method with facial attributes described in the first aspect.

[0064] The method and apparatus for generating face images with facial attributes provided in this application acquire a target face attribute image and a face image to be edited, input the target face attribute image and the face image to be edited into a face image generation network, and obtain a target face image generated by the face image generation network. The face image generation network is used to generate a target face image based on the target face attribute image and the face image to be edited. The face image generation network is trained based on a set of face sample images, a set of feature face sample images, and sample face attribute images extracted from the set of feature face sample images. This enables the trained face image generation network to controllably generate the required target face image under the guidance of the sample face attribute images. Thus, the face image generation network can be guided by the target face attribute image to generate a target face image with facial attributes, and the facial attribute appearance features on the target face image are controllable, effectively improving the stability of the target face image and improving the generation efficiency of face images with facial attributes. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is one of the flowcharts illustrating the method for generating face images with facial attributes provided in the embodiments of this application;

[0067] Figure 2 This is a second schematic flowchart of the face image generation method with facial attributes provided in the embodiments of this application;

[0068] Figure 3 This is the third flowchart illustrating the method for generating face images with facial attributes provided in this application embodiment;

[0069] Figure 4 This is the fourth flowchart illustrating the method for generating face images with facial attributes provided in this application embodiment;

[0070] Figure 5 This is the fifth flowchart illustrating the method for generating face images with facial attributes provided in this application embodiment;

[0071] Figure 6 This is a schematic diagram of the structure of the face image generation device with face attributes provided in the embodiments of this application;

[0072] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0074] Figure 1 This is one of the flowcharts illustrating a method for generating a face image with facial attributes provided in an embodiment of this application. (Refer to...) Figure 1 This application provides a method for generating face images with facial attributes, which may include:

[0075] Step 101: Obtain the target face attribute image and the face image to be edited.

[0076] Specifically, the target face attribute image is the image corresponding to the appearance of the target or specified face attribute, while the face image to be edited is the original face image to which face attributes need to be added. Face attributes include, but are not limited to, glasses, freckles, hairstyles, or even mouth or nose shapes. There are various types of face attributes. In practical applications, it is necessary to select appropriate face attributes according to the actual application and the requirements of the target face image to be generated, and obtain the corresponding target face attribute image accordingly. The type of face attribute is not uniquely limited here.

[0077] Step 102: Input the target face attribute image and the face image to be edited into the face image generation network to obtain the target face image generated by the face image generation network.

[0078] In this embodiment, the face image generation network is used to generate a target face image based on a target face attribute image and a face image to be edited. The face image generation network is trained on a set of face sample images, a set of feature face sample images, and sample face attribute images extracted from the feature face sample image set. The face sample image set refers to a dataset of face training images without face attributes, the feature face sample image set refers to a dataset of face training images with face attributes, and the sample face attribute images can be understood as images corresponding to the facial attribute appearance features extracted from each feature face sample image. For example, assuming the face attribute is glasses, the face sample image set is the set of face images without glasses, the feature face sample image set is the set of face images with glasses, and the sample face attribute images are the glasses images extracted from the set of face images with glasses. Because sample facial attribute images are used in training, the trained face image generation network has the ability to controllably generate the required face image, guided by the sample facial attribute images. This avoids excessive differences between the facial attribute appearance features in the generated face image and the required face attributes, enabling the face image generation network to controllably generate target face images based on the target facial attribute image and the face image to be edited in practical applications.

[0079] The following beneficial effects can be seen from the above embodiments:

[0080] The face image generation method with facial attributes provided in this application embodiment obtains a target face attribute image and a face image to be edited, inputs the target face attribute image and the face image to be edited into a face image generation network, and obtains a target face image generated by the face image generation network. The face image generation network is used to generate a target face image based on the target face attribute image and the face image to be edited. The face image generation network is trained based on a face sample image set, a feature face sample image set, and sample face attribute images extracted from the feature face sample image set. This enables the trained face image generation network to controllably generate the required target face image under the guidance of the sample face attribute image. Thus, the face image generation network can be guided by the target face attribute image to generate a target face image with facial attributes, and the facial attribute appearance features on the target face image are controllable, effectively improving the stability of the target face image and improving the generation efficiency of face images with facial attributes.

[0081] To facilitate understanding, the following embodiment of a method for generating face images with facial attributes is provided for illustration. Figure 2 This is a second schematic flowchart illustrating the method for generating face images with facial attributes provided in this application. (Refer to...) Figure 2This application provides a method for generating face images with facial attributes, wherein the training method of the face image generation network may include:

[0082] Step 201: Obtain the set of face sample images and the set of feature face sample images.

[0083] Specifically, the face sample image set is a dataset of face training images without face attributes. For example, if the face attribute is glasses, then all the images in the face sample image set are face images of people not wearing glasses; if the face attribute is hairstyle, then all the images in the face sample image set are face images of people without hairstyles. The feature face sample image set is a dataset of face training images with face attributes. For example, if the face attribute is glasses, then all the images in the feature face sample image set are face images of people wearing glasses; if the face attribute is hairstyle, then all the images in the feature face sample image set are face images of people with hairstyles.

[0084] Step 202: Pair each face sample image in the face sample image set with each feature face sample image in the feature face sample image set to obtain multiple sets of paired images.

[0085] Each face sample image in the face sample image set is paired one by one with each feature face sample image in the feature face sample image set to obtain multiple sets of paired images. It is understood that there are various pairing methods. In practical applications, random pairing can be selected, or rules can be defined for pairing. The pairing method needs to be determined according to the actual application situation, and no single limitation is made here.

[0086] Step 203: Extract the facial appearance features from the feature face sample images of each pair of images to obtain the sample face attribute images corresponding to each pair of images.

[0087] It is understood that any pair of paired images contains a face sample image and a feature face sample image, thus each pair of images can yield a corresponding sample face attribute image, and therefore multiple sample face attribute images can be obtained. In this embodiment, the extraction of face attribute appearance features can be achieved by using Photoshop software to extract the face attribute appearance features from the feature face sample image in the paired images, thereby obtaining the sample face attribute image. For example, if the face attribute is glasses, the glasses appearance can be extracted from the feature face sample image using Photoshop software, thus obtaining the glasses image. It is understood that there are various ways to extract face attribute appearance features, and the method of extraction using Photoshop software is only an example. The appropriate extraction method should be selected according to the actual application situation, and no single limitation is made here.

[0088] Step 204: Construct a face image generation network.

[0089] Specifically, the face image generation network includes a recurrent generative adversarial network and a similarity calculation network, wherein the recurrent generative adversarial network includes a first generator, a second generator, a first discriminator, and a second discriminator. Recurrent Generative Adversarial Networks (CycleGANs) typically consist of two generators and two discriminators. Taking glasses as an example, the two generators can convert between images of a face without glasses and images of a face with glasses. Specifically, the input image is a face without glasses; after passing through the first generator, it produces a fake face image with glasses. This fake face image with glasses then passes through the second generator to obtain the reconstructed image of the input image. Simultaneously, to compete with the generators, the two discriminators need to evaluate the images input to them. For example, if the first discriminator evaluates the image output by the first generator, it needs to determine whether the fake face image with glasses is a real image or a generated image. Similarly, the second discriminator evaluates the image output by the second generator, it needs to determine whether the reconstructed image of the input image is a real image or a generated image.

[0090] It is understood that the above exemplary description of the roles of the generator and discriminator is only for better understanding of the technical solution. In practical applications, the required face attributes need to be determined according to the actual application situation, and no unique limitation is made here.

[0091] The generator is constructed, specifically consisting of three parts: an encoder, a converter, and a decoder.

[0092] The encoder consists of two sub-encoders, each composed of three convolutional layers. The first autoencoder converts a face input image of size [256, 256, 3] without facial attributes (i.e., a face sample image or a face image to be edited) into a first feature vector corresponding to a face input image of size [64, 64, 256]. The second autoencoder converts a face attribute image of size [256, 256, 3] (i.e., a sample face attribute image or a target face attribute image) into a second feature vector corresponding to a face attribute image of size [64, 64, 256]. The sum of the first and second feature vectors can be defined as the input feature vector.

[0093] Additionally, the converter combines different similar features of an image and then, based on these features, transforms the input feature vector from one lacking facial attributes to one possessing them. For example, if the facial attribute is "glasses," the converter transforms a face without glasses into a feature vector representing a face with glasses. To achieve this, a 6-layer ResNet module is used. A ResNet module is a residual network module, a neural network consisting of two convolutional layers where some input data is directly added to the output. This ensures that the input data from previous layers directly affects subsequent layers, minimizing the deviation between the output and the original input. The transformed feature vector output by the converter has a size of [64, 64, 256], which can be viewed as a feature vector of a generated face image with facial attributes.

[0094] In addition, the steps performed by the decoder are the reverse of those performed by the encoder. Low-level features are reconstructed from the feature vectors. This can usually be achieved using deconvolution layers. After passing through the decoder, the transformed feature vectors output by the converter are transformed into a face image with facial attributes of size [256,256,3].

[0095] In the embodiments of this application, the construction steps of the first generator and the second generator are identical. It is understood that the above generator construction methods are merely exemplary. In practical applications, there are various ways to construct generators, and the construction method of the generator needs to be determined according to the actual application situation. No single limitation is made here.

[0096] A discriminator is constructed, and a similarity calculation network is added on the basis of the discriminator. Specifically, the discriminator includes four convolutional layers for extracting features from the image. Finally, a convolutional layer that produces a one-dimensional output is used to map the image features to a decimal between 0 and 1 to determine whether the image is the original image or the generated image.

[0097] In this embodiment of the application, the construction steps of the first discriminator and the second discriminator are the same. The first discriminator is connected to the first generator and is used to discriminate the image output by the first generator, while the second discriminator is connected to the second generator and is used to discriminate the image output by the second generator.

[0098] It is understood that the above discriminator construction methods are merely illustrative. In practical applications, there are various ways to construct discriminators, and the construction method should be determined according to the actual application situation. No single limitation is made here.

[0099] In addition, the similarity calculation network is used to determine the degree of similarity between the facial attribute image and the facial appearance features in the generated face image. Taking glasses as an example, the similarity calculation network compares the appearance of glasses in the glasses image with the appearance of glasses in the generated face image of a person wearing glasses to determine the degree of similarity between the two. The output of the similarity calculation network is a decimal between 0 and 1; the larger the output value, the higher the degree of similarity between the facial attribute image and the facial appearance features in the generated face image.

[0100] It is understood that the above exemplary description of the role of the similarity calculation network is only for better understanding of the technical solution. In practical applications, the configuration of the similarity calculation network needs to be determined according to the actual application situation, and no single limitation is made here. In addition, the similarity calculation network can be trained simultaneously when constructing the face image generation network, or it can be pre-trained, and no single limitation is made here.

[0101] A target loss function is constructed based on the generator, discriminator, and similarity calculation network. Specifically, a discriminative loss function and a cycle consistency loss function are constructed, and the target loss function is the sum of the discriminative loss function and the cycle consistency loss function. In a recurrent generative adversarial network (RBAN), a unidirectional RBAN is defined as an image input from the first generator that is reconstructed as an image output from the second generator, and vice versa. Each unidirectional RBAN requires two loss functions for training: a discriminative loss function and a cycle consistency loss function. Therefore, when constructing a face image generation network, it is also necessary to construct both a discriminative loss function and a cycle consistency loss function.

[0102] Step 205: Input each pair of images and the corresponding sample face attribute images into the face image generation network in sequence to train the face image generation network.

[0103] The following beneficial effects can be seen from the above embodiments:

[0104] Each face sample image in the face sample image set is paired one-to-one with each feature face sample image in the feature face sample image set, and the facial attribute appearance features in the feature face sample images are extracted. The resulting paired images and the resulting sample face attribute images are then sequentially input into the constructed face image generation network for training. This ensures that the sample face attribute images are used as prior knowledge to train the face image generation network in each training process, enabling the face image generation network to controllably generate the required target face images guided by the sample face attribute images, thereby improving the stability and output accuracy of the face image generation network.

[0105] To facilitate understanding, an embodiment of a face image generation method with facial attributes is provided below. When a set of paired images and sample face attribute images corresponding to the current paired image are input into the face image generation network to train the face image generation network, the total value of the loss function is determined based on the first one-way loss function value and the second one-way loss function value, and the model parameters of the face image generation network are updated based on the total value of the loss function.

[0106] Reference Figures 3 to 5 This application provides a method for generating face images with facial attributes, wherein the training method for the face image generation network may include:

[0107] Step 301: Input the face sample image and the sample face attribute image from the paired image into the face image generation network to obtain the first one-way loss function value of the face image generation network.

[0108] like Figure 4 As shown, the process of determining the value of the first one-way loss function may include the following steps:

[0109] S11. Input the face sample image and the sample face attribute image in the current paired image into the first generator of the recurrent generative adversarial network to obtain the first face training generated image output by the first generator.

[0110] Based on the face sample image as input, a sample face attribute image is added as a prior to generate the first face training image. The first face training image can be understood as a generated fake face image with the sample face attribute. For example, if the sample face attribute is glasses, then the first face training image is a fake face image of a person wearing glasses.

[0111] In this embodiment, the first generator receives a face attribute image and an original face image, forms a generated face image based on the face attribute image and the original face image, and determines whether to output the generated face image based on the degree of similarity. It is understood that during the training phase, the face attribute image can be a sample face attribute image, and the original face image can be a set of sample face images. In practical applications, the face attribute image can be a target face attribute image, and the original face image can be the face image to be edited.

[0112] S12. Input the first face training image into the first discriminator of the recurrent generative adversarial network to obtain the first training probability value output by the first discriminator.

[0113] The first training probability value refers to the probability that the image generated from the first face training is the original face image, that is, the probability of the face sample image.

[0114] S13. Input the first face training image into the similarity calculation network to obtain the first training similarity value output by the similarity calculation network.

[0115] The first training similarity value refers to the degree of similarity between the facial attribute appearance features generated in the first face training image and the sample face attribute image.

[0116] S14. Input the first face training image into the second generator of the recurrent generative adversarial network to obtain the first face training reconstructed image output by the second generator.

[0117] The first face training and reconstruction image can be understood as the face image reconstructed by the second generator from the first face training image. For example, if the sample face attribute is glasses, then the first face training and reconstruction image is a fake face image without glasses.

[0118] S15. Determine the first one-way loss function value based on the first face training generated image, the first training probability value, the first training similarity value, and the first face training reconstructed image.

[0119] The first face training generated image, the first training probability value, the first training similarity value, and the first face training reconstructed image are input into the first one-way loss function to obtain the first one-way loss function value. For example, the first one-way loss function can be:

[0120] Formula (1)

[0121] in, The output image of the first generator. The output image of the second generator. A set of facial sample images, For feature face sample images, A collection of sample facial attribute images. For sample face attribute images, The output value of the first discriminator. The network output value is calculated to determine the similarity. The function notation for the cycle consistency loss function. To determine the function sign of the loss function.

[0122] In this embodiment of the application, the method for constructing the first one-way loss function can be specifically as follows:

[0123] A first discriminant function is constructed based on the sample facial attribute images. The first discriminant function is exemplarily represented as follows:

[0124] Formula (2)

[0125] in, For facial sample images, The first discriminator's discrimination value for the feature face sample image is denoted as . This is the first training probability value. This is a function that calculates the expected value of a set of eigenface sample images. This is a function that calculates the expected value of a set of face sample images. This indicates that the sample face attribute images follow the distribution of the set of sample face attribute images.

[0126] In addition, a second discriminant function needs to be constructed based on the similarity calculation network. The second discriminant function is exemplarily represented as follows:

[0127] Formula (3)

[0128] in, This is the first training similarity value.

[0129] Summing the first discriminant function with the second discriminant function yields the discriminant loss function. It can be understood that the discriminant loss function increases the measurement of the similarity between the facial attribute features generated on the sample facial attribute image and the facial attribute appearance features generated on the first facial training image. That is, it takes into account the output of the similarity calculation network. Through training, the facial attribute appearance features generated on the first facial training image are made infinitely close to the sample facial attribute image, thereby achieving the goal of controllable facial attribute generation.

[0130] Furthermore, a cycle consistency loss function is constructed based on the sample face attribute images. The cycle consistency loss function is exemplarily expressed as follows:

[0131] Formula (4)

[0132] in, The image is reconstructed for training the first face.

[0133] The first one-way loss function is obtained by summing the discriminative loss function and the cycle consistency loss function.

[0134] Step 302: Input the feature face sample image and the sample face attribute image from the paired image into the face image generation network to obtain the second one-way loss function value of the face image generation network.

[0135] like Figure 5 As shown, the process of determining the value of the second one-way loss function may include the following steps:

[0136] S21. Input the feature face sample image from the current paired image into the second generator of the recurrent generative adversarial network to obtain the second face training generated image output by the second generator.

[0137] The second face training generated image can be understood as a fake face image that does not have the face attribute. For example, if the sample face attribute is glasses, then the second face training generated image is a fake face image without glasses.

[0138] S22. Input the second face training image into the second discriminator of the recurrent generative adversarial network to obtain the second training probability value output by the second discriminator.

[0139] The second training probability value refers to the probability that the second face training generated image is a feature face sample image. In the embodiments of this application, when the outputs of the first discriminator and the second discriminator are stable around 0.5, the first generator and the second generator achieve the best effect. At this time, it can be considered that the first generator can meet the requirements for generating the target face image.

[0140] S23. Input the second face training generated image and the sample face attribute image into the first generator to obtain the second face training reconstructed image output by the first generator.

[0141] The second face training and reconstruction image can be understood as a face image reconstructed by the first generator from the second face training image. For example, if the sample face attribute is glasses, then the second face training and reconstruction image is a fake face image of a person wearing glasses.

[0142] S24. Input the reconstructed image of the second face training into the similarity calculation network to obtain the second training similarity value output by the similarity calculation network.

[0143] The second training similarity value refers to the degree of similarity between the facial attribute appearance features in the second face training reconstructed image and the sample face attribute image.

[0144] S25. Determine the value of the second one-way loss function based on the second face training generated image, the second training probability value, the second training similarity value, and the second face training reconstructed image.

[0145] The second face training generated image, the second training probability value, the second training similarity value, and the second face training reconstructed image are input into the second one-way loss function to obtain the value of the second one-way loss function. The second one-way loss function is exemplarily represented as follows:

[0146] Formula (5)

[0147] in, This is the output value of the second discriminator.

[0148] It is understandable that the construction steps of the second one-way loss function are similar to those of the first one-way loss function. Similarly, it is necessary to construct a discriminative loss function and a cycle consistency loss function. It is also understandable that the above construction methods of the first and second one-way loss functions are exemplary. In practical applications, there are various ways to construct loss functions. The construction method needs to be determined according to the actual application situation. There is no single limitation here.

[0149] Step 303: Determine the total value of the loss function based on the first one-way loss function value and the second one-way loss function value.

[0150] In the embodiments of this application, the total value of the loss function can be obtained by summing the first one-way loss function value and the second one-way loss function value. In practical applications, there are various ways to determine the total value of the loss function, and an appropriate determination method should be selected according to the actual application situation. No single limitation is made here.

[0151] Step 304: Update the model parameters of the face image generation network based on the total value of the loss function.

[0152] Repeat the following steps until the preset conditions are met:

[0153] The gradient descent algorithm minimizes the total value of the loss function to obtain the minimum value of the loss function. Gradient descent is an iterative method that can be used to solve least squares problems. The calculation process of the gradient descent algorithm is to find the minimum value along the direction of gradient descent.

[0154] If the minimum value of the loss function is less than the stored value of the loss function, the model parameters are updated to the updated model parameters corresponding to the minimum value of the loss function. The stored value of the loss function is determined based on the model parameters. In other words, if the output loss function value (i.e., the stored value of the loss function) under the original model parameters of the face image generation network is greater than the minimum value of the loss function output under the currently updated model parameters, it means that the currently updated model parameters (i.e., the model parameters corresponding to the minimum value of the loss function) are superior to the original model parameters, resulting in a lower loss value when generating face images. Therefore, the current updated model parameters replace the original model parameters, thus achieving the purpose of updating the face image generation network.

[0155] If the minimum value of the loss function is greater than or equal to the stored value of the loss function, the model parameters are kept unchanged, that is, the face image generation network is not updated at this time.

[0156] Specifically, the preset conditions include: after updating the model parameters, the output values ​​of both the first discriminator and the second discriminator of the face image generation network are within a preset range. In this embodiment, the preset range can be set to be greater than 0.4 and less than 0.6. It is understood that in practical applications, the setting method of the preset range is diverse and needs to be set according to the actual application situation; no single limitation is made here. It is also understood that during the continuous iterative update of the model parameters, if the output values ​​of both the first discriminator and the second discriminator are within the preset range—for example, if the output values ​​of both the first discriminator and the second discriminator are 0.5—it indicates that the face image generation effect of the first generator has reached the optimal effect and can meet the generation requirements of the target face image. Training can then be terminated, and the first generator can be applied to the scenario where the required face image needs to be generated.

[0157] The following beneficial effects can be seen from the above embodiments:

[0158] Based on sample facial attribute images and a similarity calculation network, a first one-way loss function and a second one-way loss function are constructed to improve the controllability of the facial attribute appearance on the generated facial images. This ensures that the facial attribute appearance features in the target facial image are consistent with the target facial attribute image during the process of generating the target facial image by the facial image generation network, thereby improving the stability of image generation, providing a guarantee for the diversity of generated images, effectively increasing the amount of training image data for the facial recognition algorithm, and effectively improving the recognition accuracy and robustness of the facial recognition algorithm.

[0159] The following describes the face image generation apparatus with facial attributes provided in the embodiments of this application. The face image generation apparatus with facial attributes described below can be referred to in correspondence with the face image generation method with facial attributes described above.

[0160] Figure 6 This is a schematic diagram of the structure of a face image generation device with facial attributes provided in an embodiment of this application. (Refer to...) Figure 6 This application provides a face image generation apparatus with facial attributes, which may include:

[0161] The image acquisition module is used to acquire the target face attribute image and the face image to be edited;

[0162] The face image generation module is used to input the target face attribute image and the face image to be edited into the face image generation network to obtain the target face image generated by the face image generation network;

[0163] A face image generation network is used to generate a target face image based on a target face attribute image and a face image to be edited;

[0164] The face image generation network is trained based on a set of face sample images, a set of feature face sample images, and sample face attribute images extracted from the set of feature face sample images.

[0165] The face image generation device with facial attributes provided in this application embodiment acquires a target face attribute image and a face image to be edited, inputs the target face attribute image and the face image to be edited into a face image generation network, and obtains a target face image generated by the face image generation network. The face image generation network is used to generate a target face image based on the target face attribute image and the face image to be edited. The face image generation network is trained based on a face sample image set, a feature face sample image set, and sample face attribute images extracted from the feature face sample image set. This enables the trained face image generation network to controllably generate the required target face image under the guidance of the sample face attribute image. Thus, the face image generation network can be guided by the target face attribute image to generate a target face image with facial attributes, and the facial attribute appearance features on the target face image are controllable, effectively improving the stability of the target face image and increasing the generation efficiency of face images with facial attributes.

[0166] In one embodiment, the face image generation network is trained as follows:

[0167] Obtain a set of face sample images and a set of feature face sample images;

[0168] Each face sample image in the face sample image set is paired with each feature face sample image in the feature face sample image set to obtain multiple sets of paired images.

[0169] Extract the facial attribute appearance features from the feature facial sample images of each pair of images to obtain the sample facial attribute images corresponding to each pair of images;

[0170] Construct a face image generation network;

[0171] Each pair of paired images and the corresponding sample face attribute images are sequentially input into the face image generation network to train the face image generation network.

[0172] In one embodiment, the face image generation apparatus provided in this application may further include a training module (not shown in the figure), used for:

[0173] The face sample image and the sample face attribute image in the paired image are input into the face image generation network to obtain the first one-way loss function value of the face image generation network;

[0174] The feature face sample image and the sample face attribute image in the paired image are input into the face image generation network to obtain the second one-way loss function value of the face image generation network;

[0175] The total value of the loss function is determined based on the first one-way loss function value and the second one-way loss function value;

[0176] The model parameters of the face image generation network are updated based on the total value of the loss function.

[0177] In one embodiment, the face image generation network includes a recurrent generative adversarial network and a similarity calculation network, wherein the similarity calculation network is used to determine the degree of similarity between the facial attribute appearance features in the face attribute image and the face generated image.

[0178] The training module is specifically used for:

[0179] Input the face sample image and the sample face attribute image in the current paired image into the first generator of the recurrent generative adversarial network to obtain the first face training generated image output by the first generator;

[0180] The first face training image is input into the first discriminator of the recurrent generative adversarial network to obtain the first training probability value output by the first discriminator.

[0181] The first face training image is input into the similarity calculation network to obtain the first training similarity value output by the similarity calculation network;

[0182] The first face training image is input into the second generator of the recurrent generative adversarial network to obtain the first face training reconstructed image output by the second generator.

[0183] The first one-way loss function value is determined based on the first face training generated image, the first training probability value, the first training similarity value, and the first face training reconstructed image.

[0184] In one embodiment, the training module is specifically used for:

[0185] The feature face sample image in the current paired image is input into the second generator of the recurrent generative adversarial network to obtain the second face training generated image output by the second generator;

[0186] The training image of the second face is input into the second discriminator of the recurrent generative adversarial network to obtain the second training probability value output by the second discriminator.

[0187] The second face training image and the sample face attribute image are input into the first generator to obtain the second face training reconstructed image output by the first generator.

[0188] The reconstructed image of the second face is input into the similarity calculation network to obtain the second training similarity value output by the similarity calculation network.

[0189] The second one-way loss function value is determined based on the second face training generated image, the second training probability value, the second training similarity value, and the second face training reconstructed image.

[0190] In one embodiment, the training module is specifically used for:

[0191] The first face training generated image, the first training probability value, the first training similarity value, and the first face training reconstructed image are input into the first one-way loss function to obtain the first one-way loss function value;

[0192] The first one-way loss function is:

[0193] ;

[0194] in, The output image of the first generator. The output image of the second generator. A set of facial sample images, For feature face sample images, A collection of sample facial attribute images. For sample face attribute images, The output value of the first discriminator. The network output value is calculated for similarity.

[0195] In one embodiment, the training module is specifically used for:

[0196] The second face training generated image, the second training probability value, the second training similarity value, and the second face training reconstructed image are input into the second one-way loss function to obtain the value of the second one-way loss function.

[0197] The second one-way loss function is:

[0198] ;

[0199] in, This is the output value of the second discriminator.

[0200] In one embodiment, the method for constructing the first one-way loss function specifically includes:

[0201] A first discriminant function is constructed based on the sample facial attribute images. The first discriminant function is:

[0202] ;

[0203] in, For facial sample images, The first discriminator's discrimination value for the feature face sample image is denoted as . This is the first training probability value;

[0204] A second discriminant function is constructed based on the similarity calculation network. The second discriminant function is as follows:

[0205] ;

[0206] in, This is the first training similarity value;

[0207] Summing the first discriminant function with the second discriminant function yields the discriminant loss function;

[0208] A cycle consistency loss function is constructed based on the sample face attribute images. The cycle consistency loss function is as follows:

[0209] ;

[0210] in, Train and reconstruct images for the first face;

[0211] The first one-way loss function is obtained by summing the discriminative loss function and the cycle consistency loss function.

[0212] In one embodiment, the training module is specifically used for:

[0213] Repeat the following steps until the preset conditions are met:

[0214] The minimum value of the loss function is obtained by minimizing the total value of the loss function using the gradient descent algorithm.

[0215] If the minimum value of the loss function is less than the stored value of the loss function, then the model parameters are updated to the updated model parameters corresponding to the minimum value of the loss function.

[0216] If the minimum value of the loss function is greater than or equal to the stored value of the loss function, then the model parameters remain unchanged;

[0217] The stored value of the loss function is the loss function value determined based on the model parameters;

[0218] The preset conditions include:

[0219] After updating the model parameters, the output values ​​of the first and second discriminators of the face image generation network are both within the preset range.

[0220] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call a computer program in the memory 730 to execute steps of a face image generation method with facial attributes, such as:

[0221] Obtain the target face attribute image and the face image to be edited;

[0222] Input the target face attribute image and the face image to be edited into the face image generation network to obtain the target face image generated by the face image generation network;

[0223] A face image generation network is used to generate a target face image based on a target face attribute image and a face image to be edited;

[0224] The face image generation network is trained based on a set of face sample images, a set of feature face sample images, and sample face attribute images extracted from the set of feature face sample images.

[0225] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0226] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the face image generation method with facial attributes provided in the above embodiments, such as including:

[0227] Obtain the target face attribute image and the face image to be edited;

[0228] Input the target face attribute image and the face image to be edited into the face image generation network to obtain the target face image generated by the face image generation network;

[0229] A face image generation network is used to generate a target face image based on a target face attribute image and a face image to be edited;

[0230] The face image generation network is trained based on a set of face sample images, a set of feature face sample images, and sample face attribute images extracted from the set of feature face sample images.

[0231] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the methods provided in the above embodiments, such as including:

[0232] Obtain the target face attribute image and the face image to be edited;

[0233] Input the target face attribute image and the face image to be edited into the face image generation network to obtain the target face image generated by the face image generation network;

[0234] A face image generation network is used to generate a target face image based on a target face attribute image and a face image to be edited;

[0235] The face image generation network is trained based on a set of face sample images, a set of feature face sample images, and sample face attribute images extracted from the set of feature face sample images.

[0236] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0237] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0238] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating a face image with facial attributes, characterized in that, include: Obtain the target face attribute image and the face image to be edited; Input the target face attribute image and the face image to be edited into the face image generation network to obtain the target face image generated by the face image generation network; The face image generation network is trained in the following manner: Each face sample image in the face sample image set is paired with each feature face sample image in the feature face sample image set to obtain multiple sets of paired images. Each pair of paired images and the corresponding sample facial attribute image are sequentially input into the face image generation network to train the face image generation network. The process of inputting a pair of paired images and the corresponding sample facial attribute image into the face image generation network for training includes: The face sample image and the sample face attribute image in the paired image are input into the face image generation network to obtain the first one-way loss function value of the face image generation network; The feature face sample image and the sample face attribute image in the paired image are input into the face image generation network to obtain the second one-way loss function value of the face image generation network; The total value of the loss function is determined based on the first one-way loss function value and the second one-way loss function value; The model parameters of the face image generation network are updated based on the total value of the loss function.

2. The method for generating a face image with facial attributes according to claim 1, characterized in that, The face image generation network includes a recurrent generative adversarial network and a similarity calculation network. The similarity calculation network is used to determine the degree of similarity between the facial attribute appearance features in the face attribute image and the face generation image. The step of inputting the face sample image and the sample face attribute image from the paired image into the face image generation network to obtain the first one-way loss function value of the face image generation network includes: The face sample image and the sample face attribute image in the current paired image are input into the first generator of the recurrent generative adversarial network to obtain the first face training generated image output by the first generator; The first face training generated image is input into the first discriminator of the recurrent generative adversarial network to obtain the first training probability value output by the first discriminator; The first face training image is input into the similarity calculation network to obtain the first training similarity value output by the similarity calculation network; The first face training image is input into the second generator of the recurrent generative adversarial network to obtain the first face training reconstructed image output by the second generator; The first one-way loss function value is determined based on the first face training generated image, the first training probability value, the first training similarity value, and the first face training reconstructed image.

3. The method for generating a face image with facial attributes according to claim 2, characterized in that, The feature face sample image and the sample face attribute image from the paired image are input into the face image generation network to obtain the second one-way loss function value of the face image generation network, including: The feature face sample image in the current paired image is input into the second generator of the recurrent generative adversarial network to obtain the second face training generated image output by the second generator; The second face training image is input into the second discriminator of the recurrent generative adversarial network to obtain the second training probability value output by the second discriminator. The second face training image and the sample face attribute image are input into the first generator to obtain the second face training reconstruction image output by the first generator; The second face training and reconstruction image is input into the similarity calculation network to obtain the second training similarity value output by the similarity calculation network; The second one-way loss function value is determined based on the second face training generated image, the second training probability value, the second training similarity value, and the second face training reconstructed image.

4. The method for generating a face image with facial attributes according to claim 2, characterized in that, The step of determining the first one-way loss function value based on the first face training generated image, the first training probability value, the first training similarity value, and the first face training reconstructed image includes: The first face training generated image, the first training probability value, the first training similarity value, and the first face training reconstructed image are input into the first one-way loss function to obtain the first one-way loss function value. The first one-way loss function is: ； in, The output image of the first generator. The output image of the second generator. The set of face sample images, The set of feature face sample images, The set of sample facial attribute images, The sample face attribute image, The output value of the first discriminator. The output value of the similarity calculation network; The method for constructing the first one-way loss function specifically includes: A first discriminant function is constructed based on the sample facial attribute image, and the first discriminant function is: ； in, The face sample image, The discrimination value of the first discriminator for the feature face sample image is [value]. This is the first training probability value; A function to calculate the expected value of a set of feature face sample images; This is a function that calculates the expected value of a set of face sample images. This indicates that the sample face attribute images follow the distribution of the set of sample face attribute images; A second discriminant function is constructed based on the similarity calculation network, and the second discriminant function is: ； in, The first training similarity value; Summing the first discriminant function with the second discriminant function yields the discriminant loss function; A cycle consistency loss function is constructed based on the sample face attribute images. The cycle consistency loss function is as follows: ； in, Train and reconstruct images for the first face; The first one-way loss function is obtained by summing the discriminative loss function and the cycle consistency loss function.

5. The method for generating a face image with facial attributes according to claim 3, characterized in that, The step of determining the second one-way loss function value based on the second face training generated image, the second training probability value, the second training similarity value, and the second face training reconstructed image includes: The second face training generated image, the second training probability value, the second training similarity value, and the second face training reconstructed image are input into the second one-way loss function to obtain the value of the second one-way loss function. The second one-way loss function is: ； in, This is the output value of the second discriminator.

6. A face image generation device with facial attributes, characterized in that, include: The image acquisition module is used to acquire the target face attribute image and the face image to be edited; A face image generation module is used to input the target face attribute image and the face image to be edited into the face image generation network to obtain the target face image generated by the face image generation network; The face image generation network is used to generate the target face image based on the target face attribute image and the face image to be edited; The face image generation network is trained in the following manner: Each face sample image in the face sample image set is paired with each feature face sample image in the feature face sample image set to obtain multiple sets of paired images. Each pair of paired images and the corresponding sample facial attribute image are sequentially input into the face image generation network to train the face image generation network. The process of inputting a pair of paired images and the corresponding sample facial attribute image into the face image generation network for training includes: The face sample image and the sample face attribute image in the paired image are input into the face image generation network to obtain the first one-way loss function value of the face image generation network; The feature face sample image and the sample face attribute image in the paired image are input into the face image generation network to obtain the second one-way loss function value of the face image generation network; The total value of the loss function is determined based on the first one-way loss function value and the second one-way loss function value; The model parameters of the face image generation network are updated based on the total value of the loss function.

7. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the face image generation method with facial attributes as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the face image generation method with facial attributes as described in any one of claims 1 to 5.

Images