Facial image enhancement method, device and equipment

Abstract

The embodiment of the invention discloses a facial image enhancement method, device and equipment. Comprising the following steps: acquiring a source face image and at least one reference face image; extracting a reference attribute mask corresponding to the at least one reference face image and a non-reference attribute mask corresponding to the source face image; fusing the at least one reference attribute mask with the non-reference attribute mask to obtain an attribute fusion mask; and performing image reconstruction based on the source face image and the attribute fusion mask to obtain an enhanced image. And the first independent right is achieved. According to the face image enhancement method provided by the embodiment of the invention, the diversity and robustness of the face image can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, in particular to a face image enhancement method, device and equipment. BACKGROUND

[0002] In the financial field, face recognition technology as an important identity verification method is widely used in payment systems, bank counters, ATMs, mobile banking payments and other scenarios. In order to ensure system security and accuracy, data enhancement has become one of the important technologies to improve the performance of face recognition models. However, despite the significant advantages of face image data enhancement, there are still some challenges and problems.

[0003] First, the data quality and diversity problem is the core challenge of face image data enhancement. Face recognition in financial applications needs to handle diverse face images with different facial features and different age groups, while existing enhancement methods are often limited to traditional geometric transformations (such as rotation, cropping, flipping, etc.) and color transformations, which are difficult to fully simulate complex real-life situations. In particular, in the financial field, users' facial features are often protected by privacy, which makes it difficult to obtain a large amount of high-quality training data with high diversity. Second, the data bias problem is still serious. In the financial field, the face image data of user groups often shows an imbalance. For example, the face data of the elderly and ethnic minorities may be relatively less, which leads to the recognition bias of specific groups when training the model, and thus affects the fairness of the system.

[0004] Third, the security problem is also a major challenge faced by face recognition in the financial field. With the continuous upgrading of attack means, fake face images (such as photo and video replacement) have become more and more common. Existing data enhancement methods are often unable to effectively simulate various complex attack scenarios, especially in high-risk financial transaction environments, how to improve the system's ability to resist fake attacks through enhancement technology is still a problem to be solved. SUMMARY

[0005] The present application provides a face image enhancement method, device and equipment, which can improve the diversity of face images and alleviate the data imbalance problem.

[0006] In a first aspect, the embodiments of the present application provide a face image enhancement method, comprising:

[0007] obtaining a source face image and at least one reference face image;

[0008] extracting reference attribute masks corresponding to the at least one reference face image and non-reference attribute masks corresponding to the source face image;

[0009] fuse at least one of the reference attribute masks with the non-reference attribute mask to obtain an attribute fused mask;

[0010] perform image reconstruction based on the source face image and the attribute fused mask to obtain an enhanced image.

[0011] In a second aspect, an embodiment of the present application further provides a device for enhancing a face image, comprising:

[0012] a face image acquisition module, configured to acquire a source face image and at least one reference face image;

[0013] an attribute mask extraction module, configured to extract a reference attribute mask corresponding to each of the at least one reference face image and a non-reference attribute mask corresponding to the source face image;

[0014] an attribute mask fusion module, configured to fuse at least one of the reference attribute masks with the non-reference attribute mask to obtain an attribute fused mask;

[0015] a reconstruction module, configured to perform image reconstruction based on the source face image and the attribute fused mask to obtain an enhanced image.

[0016] In a third aspect, an embodiment of the present application further provides an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory connected with the at least one processor in communication; wherein,

[0019] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method for enhancing a face image according to the embodiments of the present application.

[0020] An embodiment of the present application discloses a method, device and equipment for enhancing a face image. The method comprises the following steps: acquiring a source face image and at least one reference face image; extracting a reference attribute mask corresponding to each of the at least one reference face image and a non-reference attribute mask corresponding to the source face image; fusing at least one of the reference attribute masks with the non-reference attribute mask to obtain an attribute fused mask; and performing image reconstruction based on the source face image and the attribute fused mask to obtain an enhanced image. The method for enhancing a face image provided by the embodiment of the present application can improve the diversity and robustness of a face image. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a flowchart of a method for enhancing a face image according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of a way of filling a semantic hole region in embodiment one of the present application;

[0023] Figure 3 is a schematic diagram of a structure reconstruction process in embodiment one of the present application;

[0024] Figure 4 is a schematic diagram of a texture reconstruction process in embodiment one of the present application;

[0025] Figure 5 is a schematic diagram of a color reconstruction process in embodiment one of the present application;

[0026] Figure 6 is a structural schematic diagram of an image enhancement device for a face in embodiment two of the present application;

[0027] Figure 7 is a structural schematic diagram of an electronic device in embodiment three of the present application. DETAILED DESCRIPTION

[0028] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, and not to limit the present application. In addition, it should be noted that, for the sake of description, only the parts related to the present application are shown in the drawings, not all the structures.

[0029] Face image data enhancement in the financial field is of great significance. By improving the diversity and robustness of data, data enhancement technology can effectively improve the accuracy and fraud prevention ability of face recognition models, reduce the cost of data collection, and enhance the adaptability to different devices, environments and user groups. In addition, data enhancement can improve the generalization ability of the model, alleviate the problem of data imbalance, and thus improve the fairness and security of the system, providing stronger identity verification and anti-fraud support for the financial field.

[0030] In this embodiment, the generator needs to be trained based on the generative adversarial network technology in advance, and the inverse mapping encoder needs to be trained based on the generative adversarial network inversion technology. The generative adversarial network (GAN) is a deep learning model composed of two neural networks: the generator and the discriminator. The task of the generator is to generate fake data similar to real data, while the task of the discriminator is to distinguish between real data and generated data. The training process of GAN is a game, in which the generator tries to create more and more realistic samples. The discriminator then tries to identify which samples are real and which are generated. This adversarial process allows the generator to continuously improve, eventually generating samples that are very close to real data. GAN inversion is a technique in deep learning that aims to map real images back to the latent space of a GAN. For image generation tasks, the forward process of GAN can be understood as mapping latent encodings to RGB images, which is a process of lifting low-dimensional vectors to high-dimensional images; GAN inversion means finding a latent encoding that allows the generator to reconstruct a sample similar to the real image, which is a process of reducing high-dimensional images to low-dimensional vectors. GAN inversion is very useful in image editing, reconstruction, and attribute manipulation, because it allows users to operate in the latent space to achieve customized adjustments to generated images. In this way, users can better utilize the content generated by GAN while maintaining the authenticity of the images.

[0031] Specifically, the generator and the inverse mapping encoder are trained based on two-stage trainers. The first-stage trainer is responsible for training the optimal face image generator and the discriminator. In this stage, the role of the trainer is to establish a zero-sum game between the generator and the discriminator. The generator is trained to generate fake but realistic face images, while the discriminator is trained to distinguish between real and fake face images. In such adversarial training, the generator's ability to generate high-quality images becomes stronger, and the discriminator's ability to identify fake images becomes stronger. The second-stage trainer is responsible for training the optimal inverse mapping encoder. In this stage, the inverse mapping encoder maps the input image to the latent encoding, and the trainer sends the latent encoding to the generator to obtain the reconstructed image, and continuously optimizes the inverse mapping encoder parameter weight to improve the similarity between the reconstructed image and the input image. This process enables the inverse mapping encoder to accurately map images to latent encodings.

[0032] Embodiment one

[0033] Figure 1A flowchart of a face image enhancement method provided for Embodiment One of the present application. The present embodiment can be applied to the case of enhancing a face image. The method can be executed by a face image enhancement device, which can be implemented in the form of software and / or hardware, and can be implemented by an electronic device, which can be a mobile terminal, a PC terminal, or a server, etc. As shown in FIG. 1, the method specifically includes the following steps: Figure 1

[0034] S110, obtaining a source face image and at least one reference face image.

[0035] In the source face image, the face is different from the faces in each reference face image, and both include a plurality of face attributes. Face attributes can be understood as semantic categories of each region in a face image, including 16 categories such as hair, left and right eyes, nose, jewelry, and background. The classification of face attributes depends on semantic segmentation technology.

[0036] S120, extracting a reference attribute mask corresponding to each of the at least one reference face image and a non-reference attribute mask corresponding to the source face image.

[0037] The attribute mask refers to the classification result set of each pixel of the input image by the semantic segmentation model. The length and width of the attribute mask correspond to the input image, and the channel number of the attribute mask corresponds to the number of predefined categories of the semantic segmentation model. The attribute mask is initialized to all 0, and when an input image pixel is classified as a certain attribute, the corresponding spatial position on the corresponding channel of the attribute mask will be set to 1.

[0038] The reference attribute can be set by the user or randomly selected. The reference attributes corresponding to different reference face images are different. The reference face image and the reference attribute can be in a one-to-one or one-to-many relationship, that is, one reference attribute mask can be extracted from one reference face image, or two reference attribute masks (containing two reference attributes) can be extracted. For example: assuming that there are two reference face images, the reference attribute corresponding to one reference face image is the eye, the reference attribute corresponding to the other reference face image is the nose and the mouth, and the non-reference attribute is other attributes except the eye and the nose.

[0039] Specifically, the way to extract the reference attribute mask corresponding to each of the at least one reference face image and the non-reference attribute mask corresponding to the source face image can be: performing face alignment processing on the source face image and the at least one reference face image respectively; based on the pre-set reference attribute and the semantic segmentation model, extracting the reference attribute mask corresponding to each of the at least one reference face image and the non-reference attribute mask corresponding to the source face image.

[0040] ​The preset reference attribute is set for the user or randomly selected. In this embodiment, the process of performing face alignment processing on the source face image and the at least one reference face image can be: for each face image, first, key points (boundary points) in the face image are detected, and then matrix transformation is performed on the face image based on the key points, so that the face size, center position and rotation angle of each face image are unified for subsequent data enhancement processes. Then, the image is cropped and standardized, so that the length and width, pixel value range and channel order of all face images are consistent. Figure 5

[0041] Specifically, the process of extracting the reference attribute mask corresponding to each reference face image and the non-reference attribute mask corresponding to the source face image based on the preset reference attribute and the semantic segmentation model can be: first, determining the reference attribute corresponding to each reference face image, and then extracting the corresponding reference attribute mask from each reference face image based on semantic segmentation technology; and finally, extracting a plurality of non-reference attribute corresponding non-reference attribute masks from the original source face image based on semantic segmentation technology.

[0042] S130, fusing the at least one reference attribute mask and the non-reference attribute mask to obtain an attribute fusion mask.

[0043] The fusing of the at least one reference attribute mask and the non-reference attribute mask can be understood as fusing the at least one reference attribute mask and the non-reference attribute mask into one attribute mask. So that the finally generated enhanced image is composed of reference attributes and non-reference attributes. In this embodiment, the process of fusing the at least one reference attribute mask and the non-reference attribute mask can be: first, splicing the at least one reference attribute mask and the non-reference attribute mask, and then performing semantic hole filling or repairing on the spliced mask, thereby obtaining the attribute fusion mask.

[0044] Optionally, the way of fusing the at least one reference attribute mask and the non-reference attribute mask to obtain the attribute fusion mask can be: splicing the at least one reference attribute mask and the non-reference attribute mask according to a preset occlusion priority to obtain an initial fusion mask; extracting a semantic hole region in the initial fusion mask; filling the semantic hole region to obtain the attribute fusion mask.

[0045] ​The occlusion priority can be understood as a rule or strategy for determining which pixel of which attribute is preferentially retained in the fusion result when multiple masks exist in an overlapping area and have an occlusion relationship. The core goal is to solve the "visual conflict" of the overlapping area, ensure that the fusion image is natural and consistent with human eye perception logic, and at the same time retain key information. For example, the occlusion priority of the five organs is higher than that of the skin. The semantic hollow area can be understood as an area in which, due to insufficient understanding of the semantic information (such as object category, structural relationship, spatial logic, etc.) of the object and scene in the image, key semantic information is lost or semantic logic is contradictory in the fusion result.

[0046] In this embodiment, the way to extract the semantic hollow area in the initial fusion mask can be to judge whether each pixel point in the initial fusion mask meets semantic consistency (including object integrity, structure rationality, spatial logic and common sense rules), and to extract the pixel points that do not meet semantic consistency to obtain the semantic hollow area. Correspondingly, the way to fill the semantic hollow area can be to generate an image in which each pixel point meets semantic consistency, and to replace the pixel points in the semantic hollow area with the corresponding pixel points in the attribute mask of the image.

[0047] Optionally, Figure 2 is a flowchart of a way to fill a semantic hollow area in an embodiment of the present application, as Figure 2 shown, the way includes the following steps:

[0048] S210, initialize the first latent code corresponding to the hollow filling.

[0049] The initialization of the first latent code corresponding to the hollow filling can be understood as a process of setting an initial value for the latent encoding input to the generator.

[0050] S220, input the first latent code to the generator to obtain a first generated image.

[0051] The generator is a generator based on the optimal face image obtained by training the first-stage trainer. After the first latent code is input to the generator, the first generated image corresponding to the first latent code is output.

[0052] S230, determine the cross-entropy loss between the attribute mask of the first generated image and the initial fusion mask.

[0053] The attribute mask of the first generated image can be obtained by performing attribute segmentation on the first generated image using a semantic segmentation technique.

[0054] S240, determine whether the cross-entropy loss meets the first iteration end condition. If not, perform S250, and if so, perform S260.

[0055] The first iteration end condition comprises that the cross-entropy loss is less than a first set threshold and / or the number of iterations reaches a preset upper limit. In this embodiment, if the cross-entropy loss is less than the first set threshold and / or the number of iterations reaches the preset upper limit, the first iteration end condition is met; if the cross-entropy loss is greater than or equal to the first set threshold and the number of iterations does not reach the preset upper limit, the iteration end condition is not met.

[0056] In this embodiment, if the cross-entropy loss does not meet the first iteration end condition, it means that the attribute mask of the first generated image does not satisfy the condition of filling the semantic hollow region of the initial fusion mask, and the first latent code needs to be iteratively optimized to obtain a first generated image that meets the condition.

[0057] S250, adjusting the first latent code based on the cross-entropy loss, and returning to perform S220.

[0058] In this embodiment, the process of adjusting the first latent code based on the cross-entropy loss can be to back-propagate the cross-entropy loss to optimize the first latent code. After optimizing the first latent code, the first latent code is input into the generator based on the optimized first latent code to obtain a first generated image, the cross-entropy loss between the attribute mask of the first generated image and the initial fusion mask is determined, and the operation of judging whether the cross-entropy loss meets the first iteration end condition is performed until the cross-entropy loss meets the first iteration end condition.

[0059] S260, filling the semantic hollow region in the initial fusion mask based on the attribute mask of the first generated image that meets the first iteration end condition to obtain an attribute fusion mask.

[0060] In this embodiment, when the cross-entropy loss between the attribute mask of the first generated image and the initial fusion mask meets the first iteration end condition, the first generated image is attribute segmented based on the semantic segmentation technology to obtain an attribute mask, and then the pixel points corresponding to the semantic hollow region are extracted from the attribute mask, and the extracted pixel points are replaced with the corresponding pixel points in the semantic hollow region in the initial fusion mask, thereby obtaining an attribute fusion mask. In this embodiment, by filling the semantic hollow region in the initial fusion mask, the visual coherence and authenticity of the attribute fusion mask obtained after filling are improved.

[0061] S140, performing image reconstruction based on the source face image and the attribute fusion mask to obtain an enhanced image.

[0062] The image reconstruction comprises at least one of structure reconstruction, texture reconstruction, and color reconstruction, so that the generated enhanced image is more natural. In this embodiment, the process of performing image reconstruction based on the source face image and the attribute fusion mask can be: reconstructing the structure, texture, and color of the latent code corresponding to the source face image based on the attribute fusion mask, and then generating an enhanced image based on the reconstructed latent code.

[0063] Specifically, the manner of reconstructing the image based on the source face image and the attribute fusion mask to obtain the enhanced image can be: inputting the source face image into the inverse mapping encoder to output a second latent code; sequentially performing structure reconstruction, texture reconstruction and color reconstruction on the second latent code based on the attribute fusion mask to obtain a reconstructed latent code; inputting the reconstructed latent code into the generator to output the enhanced image; and obtaining labels of each reference attribute and each non-reference attribute, adding the labels to the enhanced image to obtain enhanced data.

[0064] The inverse mapping encoder is trained by the second stage trainer in the above embodiment. The attribute label is used to describe the characteristics of a certain attribute, for example: the hair color label of an A image is that the probability of 99% of the hair color is black, the probability of 0.5% is brown, the probability of 0.5% is red, and the probability of 0% is other colors. Similarly, for example, the pupil color label, the mouth opening degree label, etc.

[0065] The labels of each reference attribute and each non-reference attribute can be directly obtained from the data corresponding to each reference image and the source image. In this embodiment, the structure reconstruction, the texture reconstruction and the color reconstruction on the second latent code are all performed by iteratively optimizing the second latent code to reconstruct the second latent code.

[0066] Specifically, Figure 3 is a schematic diagram of a structure reconstruction process provided by the present application, as Figure 3 shown, the process includes the following steps:

[0067] S310, inputting the second latent code into the generator to output a second generated image.

[0068] The generator is the generator based on the optimal face image obtained by training the first stage trainer. After inputting the second latent code into the generator, a second generated image corresponding to the second latent code is output S320, and the structure loss between the attribute mask of the second generated image and the attribute fusion mask is determined.

[0069] The attribute mask of the second generated image can be obtained by attribute segmentation of the second generated image using a semantic segmentation technique. The structure loss is composed of a cross-entropy loss.

[0070] S330, determining whether the structure loss satisfies a second iteration end condition, if not, performing S340, if yes, performing S350.

[0071] The second iteration end condition includes that the structure loss is less than a second set threshold and / or the number of iterations reaches a preset upper limit. If the structure loss is less than the second set threshold and / or the number of iterations reaches the preset upper limit, the structure loss satisfies the second iteration end condition; if the structure loss is greater than or equal to the second set threshold and the number of iterations does not reach the preset upper limit, the structure loss does not satisfy the second iteration end condition.

[0072] In this embodiment, if the result loss does not satisfy the second iteration end condition, it indicates that the deviation between the structure information of the attribute mask of the second generated image and the structure information of the attribute fusion mask is large, and the second latent code needs to be iteratively optimized to obtain the second generated image that satisfies the condition.

[0073] S340, updating the second latent code based on the structure loss, and returning to perform S310 based on the updated second latent code.

[0074] In this embodiment, the process of updating the second latent code based on the structure loss can be to back-propagate the structure loss to optimize the second latent code. After optimizing the second latent code, the operation of inputting the second latent code into the generator to output the second generated image, determining the structure loss between the attribute mask of the second generated image and the attribute fusion mask, and judging whether the structure loss satisfies the second iteration end condition is returned to be performed based on the optimized second latent code until the structure loss satisfies the second iteration condition.

[0075] S350, obtaining the second latent code after structure reconstruction.

[0076] Specifically, Figure 4 is a schematic diagram of a texture reconstruction process provided by the present application, as Figure 4 shown, the process includes the following steps:

[0077] S410, inputting the second latent code after structure reconstruction into the generator to output a third generated image.

[0078] S420, determining the texture loss between the attributes in the third generated image and the reference attributes and non-reference attributes, respectively.

[0079] The texture loss can be composed of the L2 distance sum of the Gram matrix of the first four layers of feature maps extracted by the auxiliary feature extraction model (VGG).

[0080] S430, judging whether the texture loss satisfies a third iteration end condition, if not, performing S440, and if so, performing S450.

[0081] The third iteration end condition includes that the texture loss is less than a third set threshold and / or the number of iterations reaches a preset upper limit.

[0082] S440 updates the second latent code based on the texture loss, and returns to execute S410 based on the updated second latent code.

[0083] One method for updating the second latent code based on texture loss is to backpropagate the texture loss to optimize the second latent code after structural reconstruction.

[0084] S440, obtain the second latent code after texture reconstruction.

[0085] Specifically, Figure 5 This invention provides a schematic diagram of a color reconstruction process, as shown below. Figure 5 As shown, the process includes the following steps:

[0086] S510 inputs the second latent code after texture reconstruction into the generator and outputs the fourth generated image;

[0087] S520, determine the color loss between the attributes in the fourth generated image and the reference attributes and non-reference attributes, respectively.

[0088] The color loss can be composed of the sum of the LPIPS losses of the first two layers of feature maps extracted by the auxiliary feature extraction model (VGG).

[0089] S530: Determine if the color loss meets the termination condition of the fourth iteration. If not, execute S540; if so, execute S550.

[0090] The fourth iteration termination conditions include color loss being less than the fourth set threshold and / or the number of iterations reaching the preset upper limit.

[0091] S540, update the second latent code based on the color loss, and return to execute S510 based on the updated second latent code.

[0092] One method for updating the second latent code based on color loss is to backpropagate the color loss to optimize the second latent code after texture reconstruction.

[0093] S550, obtain the second latent code after color reconstruction.

[0094] In this embodiment, after performing structural reconstruction, texture reconstruction, and color reconstruction on the second latent code corresponding to the source facial image in sequence, the reconstructed latent code is input into the generator, and the enhanced image is output.

[0095] The embodiment has the following advantages: 1. The application can select to mix with the attributes of other images, can expand the attribute label combination, thereby relieving the data bias problem and enhancing the system anti-forgery attack capability; 2. The application does not directly splice the reference attributes in the image dimension, but indirectly enables the enhanced image to obtain the features of each reference attribute by optimizing the latent code, sufficiently utilizes the generation capability of the generation model, and the generated enhanced image is more natural; 3. The attribute mask extraction unit of the application realizes automatic acquisition of the enhanced image label according to the participating mixed attribute, and compared with the acquisition of the enhanced image label according to the splicing area proportion, the method is more accurate and reasonable.

[0096] The technical scheme of the embodiment comprises the following steps: obtaining a source face image and at least one reference face image; extracting reference attribute masks corresponding to the at least one reference face image and a non-reference attribute mask corresponding to the source face image; fusing at least one reference attribute mask and the non-reference attribute mask to obtain an attribute fusion mask; and performing image reconstruction based on the source face image and the attribute fusion mask to obtain an enhanced image. The face image enhancement method provided by the embodiment can improve the diversity and robustness of the face image.

[0097] Embodiment two

[0098] Figure 6 is a structural schematic diagram of a face image enhancement device provided by the second embodiment of the application, as Figure 6 shown, the device comprises:

[0099] The face image acquisition module 610 is configured to obtain a source face image and at least one reference face image.

[0100] The attribute mask extraction module 620 is configured to extract reference attribute masks corresponding to the at least one reference face image and a non-reference attribute mask corresponding to the source face image.

[0101] The attribute mask fusion module 630 is configured to fuse at least one reference attribute mask and the non-reference attribute mask to obtain an attribute fusion mask.

[0102] The reconstruction module 640 is configured to perform image reconstruction based on the source face image and the attribute fusion mask to obtain an enhanced image.

[0103] Optionally, the attribute mask extraction module 620 is further configured to:

[0104] perform face alignment processing on the source face image and the at least one reference face image, respectively.

[0105] extract a reference attribute mask corresponding to each of the at least one reference face image and a non-reference attribute mask corresponding to the source face image based on a preset reference attribute and a semantic segmentation model, wherein the preset reference attribute is set by a user or randomly selected.

[0106] Optionally, the attribute mask fusion module 630 is further configured to:

[0107] splicing the at least one reference attribute mask and the non-reference attribute mask according to a preset occlusion priority to obtain an initial fusion mask;

[0108] extracting a semantic hollow region in the initial fusion mask;

[0109] filling the semantic hollow region to obtain an attribute fusion mask.

[0110] Optionally, the attribute mask fusion module 630 is further configured to:

[0111] initializing a first latent code corresponding to the hollow filling;

[0112] inputting the first latent code into the generator to obtain a first generated image;

[0113] determining a cross-entropy loss between an attribute mask of the first generated image and the initial fusion mask;

[0114] if the cross-entropy loss does not satisfy a first iteration end condition, adjusting the first latent code based on the cross-entropy loss, wherein the first iteration end condition includes that the cross-entropy loss is less than a first set threshold and / or the number of iterations reaches a preset upper limit;

[0115] based on the adjusted first latent code, returning to perform the operation of inputting the first latent code into the generator until the cross-entropy loss satisfies the first iteration end condition;

[0116] filling the semantic hollow region in the initial fusion mask based on the attribute mask of the first generated image that satisfies the first iteration end condition to obtain the attribute fusion mask.

[0117] Optionally, the reconstruction module 640 is further configured to:

[0118] inputting the source face image into the inverse mapping encoder to output a second latent code;

[0119] sequentially performing structure reconstruction, texture reconstruction, and color reconstruction on the second latent code based on the attribute fusion mask to obtain a reconstructed latent code;

[0120] inputting the reconstructed latent code into the generator to output an enhanced image; and obtaining labels of each reference attribute and each non-reference attribute, adding the labels to the enhanced image to obtain enhanced data.

[0121] Optionally, the reconstruction module 640 is further configured to:

[0122] inputting the second latent code into the generator to output a second generated image;

[0123] determining a structure loss between an attribute mask of the second generated image and the attribute fusion mask;

[0124] if the structure loss does not satisfy a second iteration end condition, updating the second latent code based on the structure loss; wherein the second iteration end condition comprises that the structure loss is less than a second set threshold and / or the number of iterations reaches a preset upper limit;

[0125] based on the updated second latent code, returning to perform the operation of inputting the second latent code into the generator until the structure loss satisfies the second iteration end condition, to obtain a structure-reconstructed second latent code.

[0126] Optionally, the reconstruction module 640 is further configured to:

[0127] inputting the structure-reconstructed second latent code into the generator to output a third generated image;

[0128] determining a texture loss between the attributes in the third generated image and the reference attributes and the non-reference attributes respectively;

[0129] if the texture loss does not satisfy a third iteration end condition, updating the second latent code based on the texture loss; wherein the third iteration end condition comprises that the texture loss is less than a third set threshold and / or the number of iterations reaches a preset upper limit;

[0130] based on the updated second latent code, returning to perform the operation of inputting the second latent code into the generator until the texture loss satisfies the third iteration end condition, to obtain a texture-reconstructed second latent code.

[0131] Optionally, the reconstruction module 640 is further configured to:

[0132] inputting the texture-reconstructed second latent code into the generator to output a fourth generated image;

[0133] determining a color loss between the attributes in the fourth generated image and the reference attributes and the non-reference attributes respectively;

[0134] if the color loss does not satisfy a fourth iteration end condition, updating the second latent code based on the color loss; wherein the fourth iteration end condition comprises that the color loss is less than a fourth set threshold and / or the number of iterations reaches a preset upper limit;

[0135] based on the updated second latent code, returning to perform the operation of inputting the second latent code into the generator until the color loss satisfies the third iteration end condition, to obtain a color-reconstructed second latent code.

[0136] The apparatus can perform the method provided by all the preceding embodiments of the present application, and has the corresponding function modules and advantages of performing the method. Technical details not described in the present embodiment can be referred to the method provided by all the preceding embodiments of the present application.

[0137] Embodiment three

[0138] Figure 7 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0139] As shown in Figure 7 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., connected to the at least one processor 11 in communication, where the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0140] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0141] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The processor 11 performs various methods and processes described above, such as the enhancement method of a facial image.

[0142] In some embodiments, the enhancement method of a facial image can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the enhancement method of a facial image described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the enhancement method of a facial image by any other suitable means, such as by means of firmware.

[0143] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0144] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a machine or a remote machine or a server.

[0145] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0147] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0148] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system to solve the defects of great management difficulty and weak business scalability in traditional physical hosts and VPS services.

[0149] The embodiment of the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the face image enhancement method provided in any embodiment of the present application.

[0150] The computer program product can be written in one or more programming languages or combinations of languages including object-oriented languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0151] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the spirit and scope of the present application. For example, the steps recited in the present application can be performed in parallel, in series, or in a different order, and the present application is not limited in this regard.

[0152] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for enhancing facial images, characterized in that, include: Obtain one source facial image and at least one reference facial image; Extract the reference attribute mask corresponding to the at least one reference facial image and the non-reference attribute mask corresponding to the source facial image; At least one of the reference attribute masks is fused with the non-reference attribute mask to obtain an attribute fusion mask; Image reconstruction is performed based on the source facial image and the attribute fusion mask to obtain an enhanced image.

2. The method according to claim 1, characterized in that, Extracting the reference attribute mask corresponding to each of the at least one reference facial image and the non-reference attribute mask corresponding to the source facial image, including: Facial alignment processing is performed on the source facial image and at least one reference facial image respectively; Based on preset reference attributes and a semantic segmentation model, the reference attribute mask corresponding to the at least one reference facial image and the non-reference attribute mask corresponding to the source facial image are extracted respectively; wherein, the preset reference attributes are set by the user or randomly selected.

3. The method according to claim 1, characterized in that, Fusing at least one of the reference attribute masks with the non-reference attribute mask to obtain an attribute fusion mask includes: At least one of the reference attribute masks and the non-reference attribute masks are stitched together according to a preset occlusion priority to obtain an initial fusion mask; Extract semantic hole regions from the initial fusion mask; The semantic void regions are filled to obtain an attribute fusion mask.

4. The method according to claim 3, characterized in that, Filling the semantic hole regions to obtain an attribute fusion mask includes: Initialize the first latent code corresponding to the hole filling; The first latent code is input into the generator to obtain the first generated image; Determine the cross-entropy loss between the attribute mask of the first generated image and the initial fusion mask; If the cross-entropy loss does not meet the first iteration termination condition, the first latent code is adjusted based on the cross-entropy loss; wherein, the first iteration termination condition includes the cross-entropy loss being less than a first set threshold and / or the number of iterations reaching a preset upper limit; Based on the adjusted first latent code, the operation of inputting the first latent code into the generator is performed until the cross-entropy loss satisfies the first iteration termination condition. The semantic hole regions in the initial fusion mask are filled based on the attribute mask of the first generated image that satisfies the first iteration termination condition, to obtain the attribute fusion mask.

5. The method according to claim 1, characterized in that, Image reconstruction is performed based on the source facial image and the attribute fusion mask to obtain an enhanced image, including: The source facial image is input into an inverse mapping encoder, which outputs a second latent code; Based on the attribute fusion mask, the second latent code is sequentially reconstructed in terms of structure, texture and color to obtain the reconstructed latent code; The reconstructed latent code is input into the generator, which outputs an enhanced image; and the labels of each of the reference attributes and each of the non-reference attributes are obtained, and the labels are added to the enhanced image to obtain enhanced data.

6. The method according to claim 5, characterized in that, Structural reconstruction of the second latent code based on the attribute fusion mask includes: The second latent code is input into the generator, and the second generated image is output. Determine the structural loss between the attribute mask of the second generated image and the attribute fusion mask; If the structural loss does not meet the second iteration termination condition, the second latent code is updated based on the structural loss; wherein, the second iteration termination condition includes the structural loss being less than a second set threshold and / or the number of iterations reaching a preset upper limit; Based on the updated second latent code, the operation of inputting the second latent code into the generator is performed until the structural loss satisfies the second iteration termination condition, thereby obtaining the second latent code after structural reconstruction.

7. The method according to claim 6, characterized in that, Structural reconstruction of the second latent code based on the attribute fusion mask includes: The second latent code after structural reconstruction is input into the generator, and the third generated image is output. Determine the texture loss between the attributes in the third generated image and the reference attributes and non-reference attributes, respectively; If the texture loss does not meet the third iteration termination condition, the second latent code is updated based on the texture loss; wherein, the third iteration termination condition includes the texture loss being less than a third set threshold and / or the number of iterations reaching a preset upper limit; Based on the updated second latent code, the operation of inputting the second latent code into the generator is performed until the texture loss satisfies the third iteration termination condition, and the second latent code after texture reconstruction is obtained.

8. The method according to claim 7, characterized in that, Based on the attribute fusion mask, the second latent code is reconstructed sequentially, including: The second latent code after texture reconstruction is input into the generator, and the fourth generated image is output. Determine the color loss between the attributes in the fourth generated image and the reference attributes and non-reference attributes, respectively; If the color loss does not meet the fourth iteration termination condition, the second latent code is updated based on the color loss; wherein, the fourth iteration termination condition includes the color loss being less than a fourth set threshold and / or the number of iterations reaching a preset upper limit; Based on the updated second latent code, the operation of inputting the second latent code into the generator is performed until the color loss satisfies the third iteration termination condition, and the second latent code after color reconstruction is obtained.

9. A facial image enhancement device, characterized in that, include: The facial image acquisition module is used to acquire a source facial image and at least one reference facial image; An attribute mask extraction module is used to extract the reference attribute mask corresponding to the at least one reference facial image and the non-reference attribute mask corresponding to the source facial image. An attribute mask fusion module is used to fuse at least one of the reference attribute masks with the non-reference attribute masks to obtain an attribute fusion mask; The reconstruction module is used to reconstruct the image based on the source facial image and the attribute fusion mask to obtain an enhanced image.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which enables the at least one processor to perform the facial image enhancement method according to any one of claims 1-8.

Images