Training of a face attribute editing model and face attribute editing method

Abstract

The application discloses a kind of face attribute editing model training and face attribute editing method.The training method includes: according to the reconstruction parameter of face training image, initial face attribute editing model is constructed, target adversarial loss function and similarity loss function are preset in the face attribute editing model;The face training image is input into the face attribute editing model, the face attribute editing model is trained using the target adversarial loss function and the similarity loss function, and the face attribute editing model after training is obtained.The technical scheme of the application realizes the common constraint between target attribute and non-target attribute when face attribute editing model edits target attribute in edited face image, ensures that non-target attribute remains unchanged while editing target attribute, improves the editing accuracy of face attribute editing model for target attribute, and ensures the real naturalness of current face image after target attribute editing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a training method for a face attribute editing model and a face attribute editing method. Background Technology

[0002] Facial attribute editing is an important technology in the field of computer vision, widely used in content production, filmmaking, and entertainment videos. Examples include changing heads, hairstyles, appearances as children, and even celebrity faces. Facial attribute editing involves taking an input image containing a face and the target attributes to be edited, then transforming the input image into a target domain face image with the target attributes, while preserving other original attribute features of the face image.

[0003] Currently, a Generative Adversarial Network (GAN) is typically pre-trained to edit the target attributes of face images. In this case, a global loss function is uniformly set for the GAN, taking into account the differences between the output and input images. This global loss function is then used to train the GAN, thereby guiding the output face image to possess specific target attributes.

[0004] However, because the global loss function creates a strong constraint relationship between the input and output images of the generative adversarial network, although the target attributes in the face image output by the trained generative adversarial network can be edited, compared with the input image, editing the target attributes may also cause changes in the background region and non-target attributes. This makes the updates of the background region and non-target attribute region in the output image less natural and greatly reduces the accuracy of face target attribute editing. Summary of the Invention

[0005] This invention provides a training method for a face attribute editing model and a face attribute editing method. The face attribute editing model achieves common constraints between target attributes and non-target attributes when editing target attributes in a face image, thereby improving the accuracy of face target attribute editing and ensuring the realism and naturalness of the edited face image.

[0006] In a first aspect, embodiments of the present invention provide a method for training a facial attribute editing model, the method comprising:

[0007] An initial face attribute editing model is constructed based on the reconstruction parameters of the face training image. The face attribute editing model is pre-set with a target adversarial loss function and a similarity loss function. The target adversarial loss function is used to constrain the realism of the edited target attributes in the face training image, and the similarity loss function is used to constrain the similarity between the non-target attributes of the face training image when processed by the face attribute editing model and the non-target attributes when the face training image is reconstructed.

[0008] The face training image is input into the face attribute editing model, and the face attribute editing model is trained using the target adversarial loss function and the similarity loss function to obtain the trained face attribute editing model.

[0009] Secondly, embodiments of the present invention provide a method for editing facial attributes, the method comprising:

[0010] The current face image to be edited is input into the face attribute editing model trained by the training method of the face attribute editing model provided in the first aspect above, and the corresponding face edited image is obtained.

[0011] Perform target segmentation on the current face image to obtain the corresponding target domain mask image;

[0012] The face image is repaired using the target domain mask image to obtain a face image with the target attributes edited.

[0013] Thirdly, embodiments of the present invention provide a training apparatus for a facial attribute editing model, the apparatus comprising:

[0014] The model building module is used to construct an initial face attribute editing model based on the reconstruction parameters of the face training image. The face attribute editing model is pre-set with a target adversarial loss function and a similarity loss function. The target adversarial loss function is used to constrain the realism of the edited target attributes in the face training image, and the similarity loss function is used to constrain the similarity between the non-target attributes of the face training image when processed by the face attribute editing model and the non-target attributes when the face training image is reconstructed.

[0015] The model training module is used to input the face training image into the face attribute editing model, and train the face attribute editing model using the target adversarial loss function and the similarity loss function to obtain the trained face attribute editing model.

[0016] Fourthly, embodiments of the present invention provide a face attribute editing device, the device comprising:

[0017] The preliminary editing module is used to input the current face image to be edited into the face attribute editing model trained by the training method of the face attribute editing model provided in the first aspect above, so as to obtain the corresponding face edited image;

[0018] The target segmentation module is used to segment the current face image to obtain the corresponding target domain mask image;

[0019] The editing and repair module is used to perform image repair on the face editing image using the target domain mask image to obtain a face image with completed target attribute editing.

[0020] Fifthly, embodiments of the present invention provide an electronic device, the electronic device comprising:

[0021] One or more processors;

[0022] Storage device for storing one or more programs;

[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the training method for the face attribute editing model provided in the first aspect above, or implement the face attribute editing method provided in the second aspect above.

[0024] In a sixth aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for the face attribute editing model provided in the first aspect above, or implements the face attribute editing method provided in the second aspect above.

[0025] This invention provides a training method for a facial attribute editing model and a facial attribute editing method. An initial facial attribute editing model is constructed based on the reconstruction parameters of a facial training image. A target adversarial loss function and a similarity loss function are preset within the facial attribute editing model. The target adversarial loss function constrains the realism of the edited target attributes within the facial training image, while the similarity loss function constrains the similarity between the non-target attributes of the facial training image after processing by the facial attribute editing model and the non-target attributes during facial reconstruction. Then, the facial attribute editing model is trained using both the target adversarial loss function and the similarity loss function. This achieves a joint constraint between the target and non-target attributes when editing target attributes within a facial image, preventing errors in the editing of target attributes. This addresses the issue of changes in background regions and non-target attributes caused by sexual activity. It ensures that while editing target attributes, the invariance of non-target attributes is maintained, thereby improving the accuracy of the face attribute editing model in editing target attributes. Simultaneously, inputting the current face image to be edited into the face attribute editing model trained using the above method yields the corresponding face edit image. Target segmentation is then performed on the current face image to obtain the corresponding target domain mask map. This target domain mask map is then used for image inpainting, resulting in a face image with completed target attribute editing. Further mask inpainting processing on the face edit image output by the face attribute editing model further improves the accuracy of face target attribute editing, ensuring the naturalness and realism of the current face image after target attribute editing. Attached Figure Description

[0026] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0027] Figure 1A A flowchart illustrating a training method for a face attribute editing model provided in Embodiment 1 of the present invention;

[0028] Figure 1B This is a schematic diagram illustrating the principle of the training process of the face attribute editing model provided in Embodiment 1 of the present invention;

[0029] Figure 2A This is a flowchart illustrating a training method for a face attribute editing model provided in Embodiment 2 of the present invention.

[0030] Figure 2B This is a schematic diagram illustrating the principle of the training process of the face attribute editing model provided in Embodiment 2 of the present invention;

[0031] Figure 2C This is a schematic diagram of the structure of the face attribute editing model provided in Embodiment 2 of the present invention;

[0032] Figure 3A This is a flowchart of a face attribute editing method provided in Embodiment 3 of the present invention;

[0033] Figure 3B This is a schematic diagram illustrating the principle of the face attribute editing process provided in Embodiment 3 of the present invention;

[0034] Figure 4 This is a schematic diagram of the structure of a training device for a face attribute editing model provided in Embodiment 4 of the present invention;

[0035] Figure 5 This is a schematic diagram of the structure of a face attribute editing device provided in Embodiment 5 of the present invention;

[0036] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment Six of the present invention. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the drawings, not all structures. Moreover, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0038] Example 1

[0039] Figure 1A This is a flowchart illustrating a training method for a facial attribute editing model according to Embodiment 1 of the present invention. This embodiment is applicable to situations where various attributes in any facial image are re-edited to change the facial style. The training method for the facial attribute editing model provided in this embodiment can be executed by a training device for the facial attribute editing model provided in this embodiment. This device can be implemented through software and / or hardware and integrated into the electronic device executing this method.

[0040] For details, please refer to Figure 1A The method may include the following steps:

[0041] S110, Construct an initial face attribute editing model based on the reconstruction parameters of the face training image. The face attribute editing model has a pre-set target adversarial loss function and a similarity loss function.

[0042] Specifically, when training network models for facial attribute editing, a global loss function is typically used to analyze the differences between the output and input images to train the network model to accurately edit target attributes within facial images. However, because the global loss function creates strong constraints between the input and output images, editing target attributes may also alter background areas and non-target attributes, resulting in an unnatural and unrealistic output image.

[0043] Therefore, in order to accurately re-edit various attributes within facial images to meet users' diverse needs for facial attribute editing, this embodiment will specifically construct a facial attribute editing model for naturally and realistically editing target attributes within various types of facial images. The target attribute can be a specific key feature within the facial image that needs to be edited according to the user's editing requirements. For example, if a user needs to re-edit a face image to show a bald head, then the target attribute would be the hair region feature within the facial image.

[0044] It should be noted that, to ensure the training efficiency of the facial attribute editing model, and considering the widespread availability of facial image reconstruction methods, and the fact that similar reconstruction steps are typically performed before further target attribute editing when editing facial images, this embodiment can initially construct an initial facial attribute editing model based on the facial image reconstruction information. This initial model can then achieve the corresponding facial image reconstruction. Subsequently, using transfer learning, the reconstruction capabilities of the facial attribute editing model can be continuously modified, enabling it to accurately edit target attributes.

[0045] Specifically, when constructing the initial facial attribute editing model, a large number of facial training images are first acquired as training samples for the model. Then, existing facial image reconstruction methods are used to reconstruct each training image to determine the reconstruction parameters. Based on these parameters, the initial facial attribute editing model is constructed, enabling it to reconstruct facial images and facilitating subsequent transfer learning for target attribute editing.

[0046] Furthermore, to avoid the problem caused by the strong constraint relationship between the input and output images created by the global loss function, this embodiment pre-sets two loss functions within the face attribute editing model: a target adversarial loss function and a similarity loss function. The target adversarial loss function constrains the realism of the edited target attributes within the face training image, while the similarity loss function constrains the similarity between the non-target attributes of the face training image after processing by the face attribute editing model and the non-target attributes during face reconstruction.

[0047] In other words, this embodiment sets different loss functions within the face attribute editing model for the target attributes to be edited and the non-target attributes that do not need to be edited within the face image. This allows for differentiated guidance during the training process of the face attribute editing model, distinguishing between the editing of target and non-target attributes within the face training image. A target adversarial loss function is used to constrain the realism of the face attribute editing model's editing of target attributes within the face training image, ensuring accurate editing of target attributes. A similarity loss function is used to constrain the similarity between the non-target attributes of the face training image after processing by the face attribute editing model and the non-target attributes during face reconstruction. This ensures that editing target attributes in the face training image does not affect non-target attributes, thus maximizing the invariance of non-target attributes before and after editing.

[0048] For example, the face attribute editing model in this embodiment can employ a generative adversarial model (such as StyleGAN). In this case, the target adversarial loss function (GAN Loss) can be a WGAN-GP function with gradient penalty, used to guide the face attribute editing model towards the target attribute. Specifically, the WGAN-GP function replaces the metric of the two probability distributions in the existing GAN network with the Wasserstein distance, making the training process of the face attribute editing model more stable. Furthermore, the WGAN-GP function optimizes the existing loss function by adding a regularization term GP (gradient penalty), which constrains the L2 norm of the gradient to the discriminator input to be around 1. The target adversarial loss function in this embodiment can be:

[0049]

[0050] Where D is the discriminator when the face attribute editing model uses a generative adversarial network, G is the generator when the face attribute editing model uses a generative adversarial network, x is the face training image input to the generator in the face attribute editing model, and y is the output image after the generator edits the target attributes in the face training image, which is used to input the discriminator in the face attribute editing model for authenticity judgment.

[0051] Furthermore, as an optional solution in this embodiment, considering that the face training images will produce some intermediate products, namely intermediate low-resolution images, at different resolutions due to the residual bridging used on RGB images at different resolutions during target attribute editing by the face attribute editing model or during image reconstruction, in order to ensure the accuracy of the face attribute editing model in editing non-target attributes within the face training images, this embodiment will divide the similarity loss function into two types: a low-resolution perceptual loss function and a mask perceptual loss function.

[0052] Among them, the low-resolution perceptual loss function is used to constrain the similarity between the first intermediate low-resolution image output when the face training image is edited for target attributes by the face attribute editing model and the second intermediate low-resolution image generated when the face training image is reconstructed; the mask perceptual loss function is used to constrain the similarity between the first keypoint mask feature extracted after the face training image is edited for target attributes by the face attribute editing model and the second keypoint mask feature extracted after the face training image is reconstructed.

[0053] Specifically, this embodiment uses a low-resolution perceptual loss function to constrain the similarity of intermediate low-resolution images during target attribute editing and reconstruction of face training images. This allows the face attribute editing model to constrain the similarity between the input and output images without creating an overly strong constraint that would prevent the target attribute from being editable. The low-resolution perceptual loss function in this embodiment can be: low-resolutionperceptual loss = (VGG(G)) / (G) rec_l (x))-VGG(G transfer_l (x))) 2 Among them, G transfer_l (x) is the first intermediate low-resolution image output when the face training image is processed by the face attribute editing model to perform target attribute editing, G rec_l (x) is the second intermediate low-resolution image generated during the reconstruction of the face training image. VGG is a pre-trained feature extractor used to extract features from the first intermediate low-resolution image and the second intermediate low-resolution image.

[0054] Furthermore, if the final output of the face attribute editing model for non-target attributes is not constrained, the non-target attributes in the output image will not maintain the pose of the input image, resulting in a large deviation from the input image. Therefore, this embodiment uses a masked perceptual loss function to extract keypoint features from the face keypoints (e.g., eyes, nose, mouth) in the two output images after target attribute editing and reconstruction of the face training image using a face keypoint network. This yields the first keypoint mask features extracted after target attribute editing of the face training image and the second keypoint mask features extracted after reconstruction of the face training image. Then, the masked perceptual loss function is used to analyze the mask similarity between the same keypoints, thereby constraining the similarity between the non-target attributes represented by the keypoint mask features extracted after target attribute editing and reconstruction of the face training image. The masked perceptual loss function in this embodiment can be:

[0055] masked resolution perceptual loss=(VGG(m·G rec (x))-VGG(m·G transfer (x))) 2

[0056] Among them, VGG(m·G rec (x) represents the first keypoint mask feature extracted from the face training image after target attribute editing by the face attribute editing model, G transfer (x) is the second keypoint mask feature extracted after reconstructing the face training image, and m is the face keypoint.

[0057] S120: Input the face training image into the face attribute editing model, and train the face attribute editing model using the target adversarial loss function and the similarity loss function to obtain the trained face attribute editing model.

[0058] After constructing the initial facial attribute editing model, various training facial images are continuously input into the model. The model's parameters are used to perform feature processing and reconstruction on these images, yielding the parameters required for the target adversarial loss function and similarity loss function. The resulting loss is then backpropagated back into the model to correct its parameters. The next training facial image is then input into the corrected model, and the same process is repeated until the model accurately edits the target attributes while preserving non-target attributes. This process is repeated until the model is trained to accurately edit the target attributes while maintaining the non-target attributes.

[0059] The technical solution provided in this embodiment constructs an initial facial attribute editing model based on the reconstruction parameters of the facial training image. A target adversarial loss function and a similarity loss function are preset within the facial attribute editing model. The target adversarial loss function constrains the realism of the edited target attributes within the facial training image, while the similarity loss function constrains the similarity between the non-target attributes of the facial training image after passing through the facial attribute editing model and the non-target attributes during the reconstruction of the facial training image. Then, the facial attribute editing model is trained using both the target adversarial loss function and the similarity loss function. This achieves a joint constraint between the target and non-target attributes when editing the target attributes within the facial image, avoiding the problem of changes to the background area and non-target attributes occurring during target attribute editing. It ensures that the editing invariance of non-target attributes is maintained while editing the target attributes, thereby improving the accuracy of the facial attribute editing model in editing target attributes.

[0060] Example 2

[0061] Figure 2A This is a flowchart of a training method for a facial attribute editing model provided in Embodiment 2 of the present invention. Figure 2B This is a schematic diagram illustrating the training process of the face attribute editing model provided in Embodiment 2 of the present invention. Figure 2C This is a schematic diagram of the facial attribute editing model provided in Embodiment 2 of the present invention. This embodiment is an optimization based on the above embodiment. Specifically, as shown... Figure 2BAs shown, considering that the StyleGAN network has a powerful generative capability to generate realistic and natural images, and that the StyleGAN network is a model that maps random noise to an image and cannot directly accept real images as input, the face attribute editing model in this embodiment can include three parts: an image encoding network, a transfer decoding network, and the latent space of the transfer decoding network, so as to use the StyleGAN network to accurately achieve the target attribute editing of face images. Figure 2C As shown, the pSp Encoder in the face attribute editing model of this embodiment can adopt a feature pyramid network structure to input various face training images, thereby outputting the latent variable features of the face training images to the latent space of the transfer decoding network. At this time, the pSp Encoder based on the feature pyramid structure can map feature maps with different semantic information to the latent space, transforming features of different granularities into multiple latent variable features. These latent variable features with different semantic information are then input into different layers of the StyleGAN decoder, allowing the transfer decoding network to input various latent variable features in the latent space, edit the target attributes within the face training images, and output the corresponding face-edited image, thus enhancing image reconstruction capabilities. Furthermore, the basic module of the pSp Encoder uses a residual module, which applies the input of the pSp Encoder to its output in a shortcut manner, allowing the gradient to be directly propagated back to the shallow parameters during backpropagation, effectively suppressing problems such as gradient vanishing.

[0062] Optional, such as Figure 2A As shown, this embodiment may include the following steps:

[0063] S210, construct the initial transfer decoding network according to the network parameters of the trained reconstruction decoding network.

[0064] To ensure the efficiency of the face attribute editing model, this embodiment first uses a large number of face images with the highest possible diversity as training samples before constructing the initial face attribute editing model based on the reconstruction parameters of the face training images. This allows the corresponding reconstruction decoding network to be pre-trained, enabling it to reconstruct the face training images. Then, when constructing the initial face attribute editing model, the network parameters of the pre-trained reconstruction decoding network can be directly used to construct the initial transfer decoding network. This allows the transfer decoding network to continuously train based on the reconstruction decoding network, using transfer learning to obtain the final face attribute editing model with the target attribute editing capability.

[0065] S220 constructs an initial face attribute editing model by combining a pre-trained image encoding network, an initial transfer decoding network, and the latent space of the transfer decoding network.

[0066] Optionally, after constructing the initial transfer decoding network, a large number of face images can be used to train the corresponding image coding network, enabling the image coding network to accurately analyze the latent variable features in the face training images. Then, as... Figure 2B As shown, the pre-trained image encoding network, the initial transfer decoding network, and the latent space of the transfer decoding network are used to construct the initial face attribute editing model. Since the image encoding network within the face attribute editing model is pre-trained, the training of the face attribute editing model in this embodiment mainly focuses on training the transfer decoding network. After training the transfer decoding network, the trained face attribute editing model is obtained.

[0067] S230 inputs the face training image into the image coding network and outputs the latent variable features of the face training image into the latent space.

[0068] After constructing the initial facial attribute editing model, it will be trained. At this point, as... Figure 2B As shown, in this embodiment, each face training image is input into the image coding network within the face attribute editing model. The image coding network analyzes the features of the face training image and outputs the latent variable features of the face training image. Then, each latent variable feature is output into the latent space of the transfer decoding network, thereby realizing the transformation of the face training image from a real image to latent variables. This allows the subsequent transfer decoding network to input the corresponding latent variable features from the latent space, thus achieving the input accuracy of the transfer decoding network.

[0069] S240, input the latent variable features into the transfer decoding network to obtain the first face image associated with the target attribute editing; input the latent variable features into the trained reconstruction decoding network to obtain the second face image associated with reconstruction.

[0070] Considering the constraints of the adversarial loss function and similarity loss function set within the face attribute editing model, this embodiment inputs the latent variable features in the latent space into the transfer decoding network and the reconstruction decoding network, respectively. The transfer decoding network then performs the target attribute editing operation according to these latent variable features, thereby outputting a first face image associated with the target attribute editing. The reconstruction decoding network performs the corresponding image reconstruction operation according to these latent variable features, thereby outputting a second face image associated with the reconstruction. Then, the first and second face images serve as the loss analysis objects for the adversarial loss function and similarity loss function set within the face attribute editing model, allowing for the calculation of the model loss of the transfer decoding network, thus facilitating its training.

[0071] S250, the first face image is substituted into the target adversarial loss function, and the first face image and the second face image are substituted into the similarity loss function to train the transfer decoding network, thus obtaining the trained face attribute editing model.

[0072] Specifically, after obtaining the first face image associated with the target attribute editing output by the transfer decoding network and the second face image associated with reconstruction output by the reconstruction decoding network, the first face image can be directly substituted into the target adversarial loss function to analyze the editing loss of the target attribute within the face training image. Simultaneously, substituting the first and second face images into the similarity loss function allows analysis of the similarity loss between non-target attributes during target attribute editing and non-target attributes during reconstruction. Furthermore, the editing loss and similarity loss are used together for backpropagation of the transfer decoding network to correct its parameters, thus training the network. At this point, each face training image is continuously input into the image encoding network, and the above steps are repeated to continuously train the transfer decoding network until its loss function converges, resulting in a trained transfer decoding network and the trained face attribute editing model.

[0073] It should be noted that, since the similarity loss function in this embodiment includes both low-resolution perceptual loss function and mask-perceptual loss function, in order to ensure the training accuracy of the transfer decoding network, this embodiment will distinguish the loss objects of the low-resolution perceptual loss function and the mask-perceptual loss function. At this time, the first face image output by the transfer decoding network associated with the target attribute editing may include the first intermediate low-resolution image output when the face training image is processed by the transfer decoding network to perform target attribute editing and the face edited image output after the target attribute editing is performed. The second face image output by the reconstruction decoding network associated with reconstruction may include the second intermediate low-resolution image generated when the face training image is reconstructed by the reconstruction decoding network and the face reconstructed image output after the reconstruction is performed.

[0074] Therefore, in this embodiment, the first face image and the second face image are substituted into the similarity loss function to train the transfer decoding network. Specifically, this may include: substituting the first intermediate low-resolution image and the second intermediate low-resolution image into the low-resolution perception loss function, and substituting the first key point mask features extracted from the face editing image and the second key point mask features extracted from the face reconstruction image into the mask perception loss function to train the transfer decoding network.

[0075] In other words, since the low-resolution perceptual loss function is used to constrain the similarity between the first intermediate low-resolution image output by the face attribute editing model when the face training image is edited for the target attribute and the second intermediate low-resolution image generated when the face training image is reconstructed, the first intermediate low-resolution image output by the transfer decoder when performing the target editing operation and the second intermediate low-resolution image output by the reconstruction decoder when performing the image reconstruction operation can be substituted into the low-resolution perceptual loss function to analyze the corresponding low-resolution image editing loss. Simultaneously, for the face-edited image output by the transfer decoder after performing the target editing operation and the face-reconstructed image output by the reconstruction decoder when performing the image reconstruction operation, existing feature extractors are used to extract the first keypoint mask features in the face-edited image and the second keypoint mask features in the face-reconstructed image, respectively. Then, the first and second keypoint mask features are substituted into the mask perceptual loss function to analyze the mask loss at each keypoint. Furthermore, the transfer decoder is trained by combining the loss result of the target adversarial loss function, the low-resolution image editing loss of the low-resolution perceptual loss function, and the mask loss of the mask perceptual loss function at each key point, to obtain the trained face attribute editing model.

[0076] The technical solution provided in this embodiment constructs an initial facial attribute editing model based on the reconstruction parameters of the face training image. A target adversarial loss function and a similarity loss function are preset within the facial attribute editing model. The target adversarial loss function constrains the realism of the edited target attributes within the face training image, while the similarity loss function constrains the similarity between the non-target attributes of the face training image after processing by the facial attribute editing model and the non-target attributes during face reconstruction. Then, the facial attribute editing model is trained using both the target adversarial loss function and the similarity loss function. This achieves a joint constraint between the target and non-target attributes when editing target attributes within a face image, avoiding the problem of background and non-target attributes changing during target attribute editing. It ensures that the editing invariance of non-target attributes is maintained while editing target attributes, thereby improving the accuracy of the facial attribute editing model in editing target attributes.

[0077] Example 3

[0078] Figure 3A This is a flowchart of a face attribute editing method provided in Embodiment 3 of the present invention. This embodiment is applicable to situations where various attributes in any face image are re-edited to change the face style. The face attribute editing method provided in this embodiment can be executed by the face attribute editing device provided in this embodiment of the present invention. This device can be implemented by software and / or hardware and integrated into the electronic device executing this method.

[0079] For details, please refer to Figure 3A The method may include the following steps:

[0080] S310, the current face image to be edited is input into the face attribute editing model trained by the training method of the face attribute editing model provided in the above embodiment to obtain the corresponding face editing image.

[0081] Optionally, a corresponding face attribute editing model is trained using the training method of the face attribute editing model provided in the above embodiments. This face attribute editing model has the ability to accurately edit target attributes. Therefore, in this embodiment, the current face image to be edited is input into the trained face attribute editing model, which then performs target attribute editing on the current face image, thereby outputting a corresponding face-edited image. Compared with the current face image, the target attributes in the face-edited image have been re-edited, while non-target attributes remain unchanged.

[0082] S320: Perform target segmentation on the current face image to obtain the corresponding target domain mask image.

[0083] Since the ability of a face attribute editing model to edit target attributes is related to the training results of the face attribute editing model, in order to further ensure the realism and naturalness of the target attribute editing of the current face image, this embodiment will also perform post-processing optimization operations on the face edited image output by the face attribute editing model.

[0084] Specifically, image segmentation techniques are first used to segment the mask region containing the target attribute from the current face image, obtaining the corresponding target domain mask map. Taking the editing of target attributes by changing the bald head as an example, hair segmentation can be performed on the current face image to obtain the corresponding hair mask map (denoted as m). hair ), which serves as the target domain mask map in this embodiment.

[0085] S330 uses the target domain mask map to perform image restoration on the face editing image, and obtains a face image with completed target attribute editing.

[0086] Since the features of non-target attribute regions such as the background area of ​​the face in the face-edited image are usually inconsistent with the current face image, and the target domain mask map can completely represent the target attribute information in the current face image, this embodiment can use the target domain mask map to analyze the non-target attribute information such as the background area in the face-edited image. Then, the fusion result of the current face image and the target domain mask map for non-target attributes is used to perform image inpainting on the face-edited image, so as to obtain a face image with completed target attribute editing. This makes the non-target attributes in the final face image unchanged from those in the current face image, thereby maintaining the invariance of non-target attributes while editing the target attributes.

[0087] For example, in this embodiment, when performing image restoration on a face-edited image, the specific steps are as follows: Perform portrait segmentation on the face-edited image to obtain a corresponding target portrait mask; fuse the target portrait mask and the current face image to obtain a corresponding face fusion image; use the difference between the target domain mask and the target portrait mask to perform image restoration on the face fusion image, resulting in a face image with the target attribute edited. In other words, by performing portrait segmentation on the face-edited image to obtain the corresponding target portrait mask, the background area within the face-edited image can be excluded, preventing changes in the background area after target attribute editing. Taking bald head shaving for target attribute editing as an example, portrait segmentation can be performed on the face-edited image to obtain a corresponding bald head portrait mask (denoted as m). bald ), which serves as the target human image mask in this embodiment.

[0088] Then, as Figure 3B As shown, to ensure the invariance of non-target attributes before and after target attribute editing, this embodiment fuses the target portrait mask and the current face image, thereby preserving non-human regions (e.g., background regions) within the current face image and fusing them with the target portrait mask within the face-edited image to obtain the corresponding face fusion image. Finally, since some regions within the target domain mask will become background regions after target attribute editing, this embodiment calculates the difference between the target domain mask and the target portrait mask (i.e., m). hair For m bald The difference set is then input together with the face fusion image into a pre-trained image inpainting model. The image inpainting model performs image inpainting on the face fusion image according to the difference set of the target domain mask image, thereby obtaining a face image with completed target attribute editing.

[0089] The technical solution provided in this embodiment inputs the current face image to be edited into the face attribute editing model trained in the above manner to obtain the corresponding face editing image. Then, the current face image is segmented to obtain the corresponding target domain mask map. The target domain mask map is then used to perform image inpainting on the face editing image to obtain the face image with the target attribute edited. At this time, by performing further mask inpainting processing on the face editing image output by the face attribute editing model, the accuracy of face target attribute editing can be further improved, ensuring the realism and naturalness of the current face image after target attribute editing.

[0090] Example 4

[0091] Figure 4 This is a schematic diagram of the structure of a training device for a face attribute editing model provided in Embodiment 4 of the present invention. Specifically, as shown... Figure 4 As shown, the device may include:

[0092] The model building module 410 is used to build an initial face attribute editing model based on the reconstruction parameters of the face training image. The face attribute editing model is preset with a target adversarial loss function and a similarity loss function. The target adversarial loss function is used to constrain the editing authenticity of the target attributes in the face training image, and the similarity loss function is used to constrain the similarity between the non-target attributes of the face training image when it passes through the face attribute editing model and the non-target attributes when the face training image is reconstructed.

[0093] The model training module 420 is used to input the face training image into the face attribute editing model, and train the face attribute editing model using the target adversarial loss function and the similarity loss function to obtain the trained face attribute editing model.

[0094] The technical solution provided in this embodiment constructs an initial facial attribute editing model based on the reconstruction parameters of the face training image. A target adversarial loss function and a similarity loss function are preset within the facial attribute editing model. The target adversarial loss function constrains the realism of the edited target attributes within the face training image, while the similarity loss function constrains the similarity between the non-target attributes of the face training image after processing by the facial attribute editing model and the non-target attributes during face reconstruction. Then, the facial attribute editing model is trained using both the target adversarial loss function and the similarity loss function. This achieves a joint constraint between the target and non-target attributes when editing target attributes within a face image, avoiding the problem of background and non-target attributes changing during target attribute editing. It ensures that the editing invariance of non-target attributes is maintained while editing target attributes, thereby improving the accuracy of the facial attribute editing model in editing target attributes.

[0095] The training device for the face attribute editing model provided in this embodiment can be applied to the training method for the face attribute editing model provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0096] Example 5

[0097] Figure 5 This is a schematic diagram of the structure of a face attribute editing device provided in Embodiment 5 of the present invention. Specifically, as shown... Figure 5 As shown, the device may include:

[0098] The preliminary editing module 510 is used to input the current face image to be edited into the face attribute editing model trained by the training method of the face attribute editing model provided in the above embodiment, so as to obtain the corresponding face editing image;

[0099] The target segmentation module 520 is used to segment the current face image to obtain the corresponding target domain mask image;

[0100] The editing and repair module 530 is used to perform image repair on the face editing image using the target domain mask image to obtain a face image with completed target attribute editing.

[0101] The technical solution provided in this embodiment inputs the current face image to be edited into the face attribute editing model trained in the above manner to obtain the corresponding face editing image. Then, the current face image is segmented to obtain the corresponding target domain mask map. The target domain mask map is then used to perform image inpainting on the face editing image to obtain the face image with the target attribute edited. At this time, by performing further mask inpainting processing on the face editing image output by the face attribute editing model, the accuracy of face target attribute editing can be further improved, ensuring the realism and naturalness of the current face image after target attribute editing.

[0102] The face attribute editing device provided in this embodiment can be applied to the face attribute editing method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0103] Example 6

[0104] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment Six of the present invention, as shown below. Figure 6 As shown, the electronic device includes a processor 60, a storage device 61, and a communication device 62; the number of processors 60 in the electronic device can be one or more. Figure 6 Taking a processor 60 as an example; the processor 60, storage device 61, and communication device 62 in an electronic device can be connected via a bus or other means. Figure 6Taking the example of a connection between China and Israel via a bus.

[0105] The electronic device provided in this embodiment can be used to execute the training method of the face attribute editing model or the face attribute editing method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0106] Example 7

[0107] Embodiment 7 of the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, it can implement the training method of the face attribute editing model or the face attribute editing method in any of the above embodiments.

[0108] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the method operations described above, but can also execute the training method of the face attribute editing model provided in any embodiment of the present invention, or related operations in the face attribute editing method.

[0109] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0110] It is worth noting that the various units and modules included in the above-mentioned training device for the face attribute editing model, or in the embodiments of the face attribute editing device, are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0111] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can be modified and varied in various ways. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. A face attribute editing method, characterized by, The method comprises the following steps: An initial face attribute editing model is constructed according to a reconstruction parameter of a face training image, and a target adversarial loss function and a similarity loss function are preset in the face attribute editing model; wherein the target adversarial loss function is used to constrain the editing authenticity of a target attribute in the face training image, and the similarity loss function is used to constrain the similarity between a non-target attribute of the face training image after passing through the face attribute editing model and a non-target attribute of the face training image after reconstruction; the similarity loss function comprises a mask perception loss function, which is used to constrain the similarity between a first key point mask feature extracted from an output image after target attribute editing of the face training image by a face key point network and a second key point mask feature extracted from an output image after reconstruction of the face training image; the first key point mask feature is a discrete coordinate mask obtained by binarizing a key point heat map obtained by key point feature extraction of face key points in the output image after target attribute editing of the face training image by the face key point network, and the second key point mask feature is a discrete coordinate mask obtained by binarizing a key point heat map obtained by key point feature extraction of face key points in the output image after reconstruction of the face training image; The face training image is input into the face attribute editing model, and the face attribute editing model is trained by using the target adversarial loss function and the similarity loss function to obtain a trained face attribute editing model; A current face image to be edited is input into the trained face attribute editing model to obtain a corresponding face edited image; Target segmentation is performed on the current face image to obtain a corresponding target domain mask image; Portrait segmentation is performed on the face edited image to obtain a corresponding target portrait mask image; The target portrait mask image and the current face image are fused to obtain a corresponding face fusion image; A difference set of the target domain mask image with respect to the target portrait mask image is calculated, and the difference set and the face fusion image are jointly input into a pre-trained image inpainting model, so that the image inpainting model inpaints the face fusion image according to the difference set to repair a background region in the face fusion image due to a part of the region in the target domain mask image changing into a background region after target attribute editing, thereby obtaining a face image after target attribute editing.

2. The method of claim 1, wherein, The similarity loss function comprises a low resolution perception loss function; The low resolution perception loss function is used to constrain the similarity between a first intermediate low resolution image output after target attribute editing of the face training image by the face attribute editing model and a second intermediate low resolution image generated after reconstruction of the face training image.

3. The method of claim 1, wherein, The face attribute editing model comprises an image encoding network, a transfer decoding network and a hidden space of the transfer decoding network; The image encoding network is used to input the face training image and output hidden variable features of the face training image to the hidden space. The migration decoding network is used for inputting the implicit variable feature, editing a target attribute in the face training image, and outputting a corresponding face edited image.

4. The method of claim 3, wherein, The initial face attribute editing model is constructed according to the reconstruction parameters of the face training image, and includes: The initial migration decoding network is constructed according to the network parameters of the trained reconstruction decoding network. The pre-trained image encoding network, the initial migration decoding network and the hidden space of the migration decoding network are constructed into the initial face attribute editing model.

5. The method of claim 4, wherein, The face training image is input into the face attribute editing model, and the face attribute editing model is trained by using the target adversarial loss function and the similarity loss function, to obtain a trained face attribute editing model, and includes: The face training image is input into the image encoding network, and the implicit variable feature of the face training image is output to the hidden space; The implicit variable feature is input into the migration decoding network to obtain a first face image associated with target attribute editing; The implicit variable feature is input into the trained reconstruction decoding network to obtain a second face image associated with reconstruction; The first face image is substituted into the target adversarial loss function, and the first face image and the second face image are substituted into the similarity loss function, and the migration decoding network is trained to obtain a trained face attribute editing model.

6. The method of claim 5, wherein, The similarity loss function includes a low resolution perception loss function, the first face image includes a first intermediate low resolution image output when the face training image is subjected to target attribute editing by the migration decoding network and a face edited image output after target attribute editing is performed, and the second face image includes a second intermediate low resolution image generated when the face training image is subjected to reconstruction by the reconstruction decoding network and a face reconstructed image output after reconstruction is performed; Correspondingly, the first face image and the second face image are substituted into the similarity loss function to train the migration decoding network, and includes: The first intermediate low resolution image and the second intermediate low resolution image are substituted into the low resolution perception loss function, and first key point mask features extracted from the face edited image and second key point mask features extracted from the face reconstructed image are substituted into the mask perception loss function to train the migration decoding network.

7. A face attribute editing apparatus, comprising: includes: The model construction module is configured to construct an initial face attribute editing model according to reconstruction parameters of the face training image, and the face attribute editing model is preconfigured with a target adversarial loss function and a similarity loss function; the target adversarial loss function is configured to constrain the editing authenticity of the target attribute in the face training image, and the similarity loss function is configured to constrain the similarity between the non-target attribute of the face training image after passing through the face attribute editing model and the non-target attribute of the face training image after reconstruction; the similarity loss function includes a mask perception loss function, which is configured to constrain the similarity between a first key point mask feature extracted from an output image after target attribute editing of the face training image by a face key point network and a second key point mask feature extracted from an output image after reconstruction of the face training image by the face key point network; the first key point mask feature is a discrete coordinate mask obtained by binarizing a key point heat map obtained by key point feature extraction of face key points in the output image after target attribute editing of the face training image by the face key point network, and the second key point mask feature is a discrete coordinate mask obtained by binarizing a key point heat map obtained by key point feature extraction of face key points in the output image after reconstruction of the face training image by the face key point network; The model training module is configured to input the face training image into the face attribute editing model, train the face attribute editing model by using the target adversarial loss function and the similarity loss function, and obtain a trained face attribute editing model; The preliminary editing module is configured to input a current face image to be edited into the trained face attribute editing model, and obtain a corresponding face edited image; The target segmentation module is configured to perform target segmentation on the current face image, and obtain a corresponding target domain mask image; The editing and repairing module is configured to perform portrait segmentation on the face edited image, obtain a corresponding target portrait mask image, fuse the target portrait mask image and the current face image to obtain a corresponding face fusion image, calculate a difference set of the target domain mask image with respect to the target portrait mask image, and input the difference set and the face fusion image into a pre-trained image repairing model, so that the image repairing model repairs background regions in the face fusion image due to the existence of partial regions in the target domain mask image which become background regions after target attribute editing, thereby obtaining a face image after completion of target attribute editing.

8. An electronic device, comprising: The electronic device includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the face attribute editing method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the face attribute editing method of any one of claims 1-6.

Images