Face attribute editing method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a face attribute editing method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, the regional image corresponding to the preset face attribute is extracted from the to-be-processed image, and the generated image corresponding to the preset face attribute editing effect and the face attribute mask are generated according to the regional image; taking the face attribute mask as a fusion weight to perform fusion processing on the regional image and the generated image to obtain a regional fusion image, and replacing a corresponding region in the to-be-processed image with the regional fusion image to obtain a target image, so that the noise of the generated image can be effectively reduced, the face attribute editing effect of the image is more real, and the user experience is improved. And the face attribute editing effect is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method, device, equipment, storage medium and product for editing facial attributes. Background Art

[0002] With the development of Internet technology and image processing technology, people are using digital entertainment products more and more widely, and digital entertainment products are providing more and more functions and gameplay.

[0003] For example, in video applications, facial attribute editing functions are often provided, such as converting facial attributes in videos (such as hair color, expression, beard, age, etc. of the characters in the image). Traditional facial attribute editing solutions generally input the entire face image into a generative adversarial network, which then outputs the converted face image. Since generative adversarial networks usually predict images by learning the mapping relationship between pixels and pixels, it is easy for the generated images to be unrealistic and noisy, resulting in poor facial attribute editing effects. Summary of the invention

[0004] The embodiments of the present application provide a facial attribute editing method, device, equipment, storage medium and product to solve the technical problem of poor facial attribute editing effect in the related art, reduce the noise of the generated image, make the facial attribute editing effect of the image more realistic, and effectively improve the facial attribute editing effect.

[0005] In a first aspect, an embodiment of the present application provides a method for editing face attributes, comprising:

[0006] Acquire an image to be processed, and extract a region image corresponding to a preset facial attribute from the image to be processed;

[0007] Generating a generated image and a facial attribute mask corresponding to a preset facial attribute editing effect according to the regional image;

[0008] Using the facial attribute mask as a fusion weight to fuse the regional image and the generated image to obtain a regional fused image;

[0009] The region fusion image is used to replace the corresponding region in the image to be processed to obtain a target image.

[0010] In a second aspect, an embodiment of the present application provides a facial attribute editing device, including an image extraction module, an image processing module, an image fusion module and an image generation module, wherein:

[0011] The image extraction module is configured to obtain an image to be processed and extract a regional image from the image to be processed;

[0012] The image processing module is configured to generate a generated image corresponding to a preset facial attribute editing effect and a facial attribute mask according to the regional image;

[0013] The image fusion module is configured to use the facial attribute mask as a fusion weight to perform fusion processing on the regional image and the generated image to obtain a regional fusion image;

[0014] The image generation module is configured to replace the corresponding area in the image to be processed with the regional fusion image to obtain a target image.

[0015] In a third aspect, an embodiment of the present application provides a face attribute editing device, including: a memory and one or more processors;

[0016] The memory is used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the facial attribute editing method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the facial attribute editing method as described in the first aspect.

[0019] In the fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, which is stored in a computer-readable storage medium, and at least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the facial attribute editing method described in the first aspect.

[0020] The embodiment of the present application extracts a regional image corresponding to preset facial attributes from an image to be processed, generates a generated image corresponding to a preset facial attribute editing effect and a facial attribute mask based on the regional image, fuses the regional image and the generated image using the facial attribute mask as a fusion weight to obtain a regional fused image, and replaces the corresponding region in the image to be processed with the regional fused image to obtain a target image. This can effectively reduce the noise of the generated image, make the facial attribute editing effect of the image more realistic, and improve the facial attribute editing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a method for editing face attributes provided by an embodiment of the present application;

[0022] Figure 2 is a flowchart of another method for editing face attributes provided by an embodiment of the present application;

[0023] Figure 3 It is a structural schematic diagram of a face attribute editing device provided in an embodiment of the present application;

[0024] Figure 4 It is a structural schematic diagram of a face attribute editing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0026] The face attribute editing method provided in this application can be applied to face editing of images. For example, in scenarios such as mobile live broadcast, video editing, and image editing, face attributes can be edited based on the face attribute editing method, such as modifying the hair color, expression, beard, age, etc. of the person in the image. The method aims to extract a regional image corresponding to a preset face attribute from the image to be processed, generate a generated image corresponding to the preset face attribute editing effect and a face attribute mask according to the regional image, fuse the face attribute mask, the regional image, and the generated image to obtain a regional fused image, and replace the corresponding region in the image to be processed with the regional fused image to obtain a target image.

[0027] In the existing face attribute editing scheme, it is generally performed through the face mesh method or the generative method. In the face mesh method, several corresponding feature points are marked on the image according to the face attributes, and the entire image is divided into several triangular mesh areas according to the feature points, so as to make changes in the mesh area of ​​the specified face attribute. In the generative method, multiple face attributes are generally predefined for the face, and then the generative adversarial network is trained to extract the potential code of the face, and the face attribute editing is realized by changing the potential code. However, this method has a large amount of calculation and needs to be run in real time when used in mobile scenarios. Therefore, the pixel-to-pixel generative adversarial network generation method is usually adopted. However, the pixel-to-pixel generative adversarial network generation method requires constraints on each pixel in the image to learn the mapping relationship from the source image to the target image. The generative adversarial network needs to learn the mapping relationship from pixel to pixel. The face attribute generation results are prone to unreliability (such as unrealistic, noisy, etc.), and in continuous videos, there is slight movement of face attributes, resulting in a large difference in the generation results of two consecutive frames of images, which is prone to jitter between video frames, and the face attribute editing effect is poor. Based on this, a facial attribute editing method is provided in an embodiment of the present application to solve the technical problem of poor existing facial attribute editing effects, provide reliable and stable facial attribute editing effects, and at the same time ensure good attribute editing image and video time domain stability.

[0028] Figure 1 A flowchart of a facial attribute editing method provided in an embodiment of the present application is given. The facial attribute editing method provided in an embodiment of the present application can be executed by a facial attribute editing device, which can be implemented by hardware and / or software and integrated in the facial attribute editing device.

[0029] The following description is made by taking the face attribute editing device executing the face attribute editing method as an example. Figure 1 , the face attribute editing method includes:

[0030] S110: Acquire an image to be processed, and extract a region image corresponding to preset facial attributes from the image to be processed.

[0031] The image to be processed provided by the present solution may be a video frame or multiple video frames in the video to be processed that needs to be edited for facial attributes. Exemplarily, the image to be processed that needs to be edited for facial attributes is obtained, and a region image corresponding to a preset facial attribute is extracted from the image to be processed.

[0032] For example, facial key points detection can be performed on the image to be processed to determine multiple facial key points in the image to be processed. Among them, facial key points are important feature points of various parts of the face in the image, usually contour points and corner points, including eyebrows, eyes, nose, mouth and other points. Facial key points can use 68, 106 or 240 points to represent the current facial features. For example, in order to balance the accuracy and time-consuming efficiency of key point detection, 106 points can be used to mark facial key points. Optionally, a pre-trained face detection model can be used to detect facial key points of the image to be processed.

[0033] After obtaining the key points of the face, the key points corresponding to the preset face attributes can be rotated, translated, and cropped based on the affine transformation method to obtain the face attribute image. Taking the preset face attribute as the beard attribute as an example, the key points corresponding to the beard attribute include multiple key points such as the nose, mouth, and contour. The center point of the beard attribute and the preset size to be cropped can be calculated based on the multiple key points of the nose, mouth, and contour. The beard image is obtained from the image to be processed through affine transformation alignment. The beard image is the regional image corresponding to the beard attribute.

[0034] S120: Generate a generated image and a facial attribute mask corresponding to a preset facial attribute editing effect according to the regional image.

[0035] Exemplarily, after obtaining the regional image, the regional image may be subjected to facial attribute editing processing based on the preset facial attribute editing effect to obtain a generated image and a facial attribute mask corresponding to the regional image. The size of the generated image and the facial attribute mask is consistent with the size of the regional image, and the facial attribute mask may reflect the distribution range of the pixel points corresponding to the preset facial attribute in the regional image. The value range of each pixel point of the facial attribute mask is [0, 1]. The higher the value of the pixel point corresponding to the facial attribute mask, the higher the probability that the pixel point is the pixel point corresponding to the preset facial attribute.

[0036] The generated image provided by this scheme has facial attributes that achieve changes corresponding to the preset facial attribute editing effect relative to the regional image. For example, when the preset facial attribute editing effect is to eliminate facial beards, beards exist in the regional image, while the beards in the generated image are eliminated. Since there may be a large color deviation between the facial skin color in the generated image and the image to be processed, if the generated image is directly used to replace the corresponding area in the image to be processed, the facial beard disappearance effect may be unreliable (such as obvious color deviation, obvious seams, unreality, high noise, etc.). In addition to performing facial attribute editing on the regional image to obtain the generated image, this scheme also generates a facial attribute mask, and fuses the facial attribute mask, the regional image, and the generated image, so that the fused image is closer to the facial skin color, and the transition between the preset facial attribute editing effect and the unedited area of ​​the face is more natural.

[0037] S130: Using the facial attribute mask as a fusion weight to fuse the regional image and the generated image to obtain a regional fused image.

[0038] Exemplarily, after obtaining the generated image and the facial attribute mask, the facial attribute mask can be used as a fusion weight to perform fusion processing on the regional image and the generated image, and the fusion processing result is used as the regional fusion image. For example, the values ​​of each pixel point of the facial attribute mask are used as fusion weights, and the pixel point values ​​of the corresponding pixel positions on the regional image and the generated image are weighted summed to complete the fusion processing of each pixel position on the regional image and the generated image, and obtain the regional fusion image. Optionally, the regional fusion image provided by this solution can be determined by the following formula:

[0039] I out =M pred ×I pred +(aM pred )×I in

[0040] Among them, M pred is the face attribute mask, I pred To generate an image, a is the preset maximum pixel value, for example, a=255, I in For a regional image, the regional image and the generated image are fused by taking the facial attribute mask as the fusion weight. The generated regional fusion image is closer to the facial skin color in the image to be processed, reducing the color cast in the facial attribute editing effect, and the transition between the preset facial attributes and the positions outside the preset facial attributes is more natural.

[0041] S140: replacing the corresponding region in the image to be processed with the regional fusion image to obtain a target image.

[0042] Exemplarily, after obtaining the regional fusion image, the regional fusion image is used to replace the area in the image to be processed that corresponds to the regional image to obtain the target image. At this time, the area in the target image that corresponds to the regional image realizes the preset facial attribute editing effect.

[0043] In the above, by extracting the regional image corresponding to the preset facial attributes from the image to be processed, generating the generated image corresponding to the preset facial attribute editing effect and the facial attribute mask according to the regional image, using the facial attribute mask as the fusion weight to fuse the regional image and the generated image to obtain the regional fused image, and replacing the corresponding area in the image to be processed with the regional fused image to obtain the target image, the noise of the generated image can be effectively reduced, the facial attribute editing effect of the image is more realistic, and the facial attribute editing effect is improved.

[0044] Based on the above embodiments, Figure 2 A flowchart of another method for editing face attributes provided by an embodiment of the present application is given, and the method for editing face attributes is a specific embodiment of the method for editing face attributes described above. Figure 2 , the face attribute editing method includes:

[0045] S210: Acquire an image to be processed, and extract a region image corresponding to preset facial attributes from the image to be processed.

[0046] S220: Input the regional image into the trained image processing model, and generate a generated image and a facial attribute mask corresponding to a preset facial attribute editing effect according to the regional image through the image processing model. The objective function of the trained image processing model is determined according to the generative adversarial network loss function, the absolute value error loss function, the perceptual loss function and the L1 regularization loss function.

[0047] This solution obtains the generated image and the facial attribute mask corresponding to the regional image through the trained image processing model. After the training of the image processing model is completed, the trained image processing model can be configured in the facial attribute editing device. Among them, the output of the image processing model includes not only the generated image corresponding to the preset facial attribute editing effect, but also the facial attribute mask corresponding to the regional image. The training of the image processing model can be based on the sample regional image and the corresponding label regional image (control sample), and the objective function of the training image processing model is determined according to the generative adversarial network loss function (GAN Loss), the absolute value error loss function (L1 Loss), the perceptual loss function (Perceptual loss) and the L1 regularization loss function.

[0048] Exemplarily, after acquiring the regional image, the regional image is input into the image processing model, the regional image is processed by the image processing model and a generated image and a facial attribute mask corresponding to a preset facial attribute editing effect are generated.

[0049] Optionally, the objective function of the training image processing model can be obtained by adding the generative adversarial network loss function, the absolute value error loss function, the perceptual loss function and the L1 regularization loss function, or can be obtained by weighted summation of the generative adversarial network loss function, the absolute value error loss function, the perceptual loss function and the L1 regularization loss function based on preset loss weights. For example, the objective function can be expressed by the following formula:

[0050] L=λ GAN L GAN (x,y)+λ pixel L pixel (x,y)+λ perc L perc (x,y)+λmask L mask (x)

[0051] Among them, x is the sample area image, y is the label area image, L GAN (x,y) is the loss function of the generative adversarial network, λ GAN To generate the loss weight corresponding to the adversarial network loss function, L pixel (x,y) is the absolute value error loss function, λ pixel is the loss weight corresponding to the absolute value error loss function, L perc (x,y) is the perceptual loss function, λ perc is the loss weight corresponding to the perceptual loss function, L mask (x) is the L1 regularization loss function, λ mask is the loss weight corresponding to the L1 regularized loss function.

[0052] This scheme accurately determines the generated image and facial attribute mask corresponding to the regional image through the image processing model, and trains the image processing model through the objective function determined by the generative adversarial network loss function, the absolute value error loss function, the perceptual loss function and the L1 regularization loss function. The adversarial network loss function and the absolute value error loss function can make the output of the image processing model closer to the control sample, and the perceptual loss function calculates the loss of the convolution output of the original image and the convolution output of the generated image on the basis of calculating the low-level feature loss (pixel color, edge, etc.), that is, the ability of the convolution layer to abstract high-level features is used to perceive the image from a global high-dimensional level. The use of perceptual loss can retain the overall information of the image, effectively ensure the authenticity of the generated image, and improve the quality of facial attribute editing. The L1 regularization loss function can be used to constrain the facial attribute mask generated by the image processing model and improve the accuracy of the facial attribute mask generated by the image processing model.

[0053] In one embodiment, the purpose of face attribute editing is to use the input source face attribute image (i.e., region image) to infer and generate the edited face attribute image (i.e., generated image) through the GAN model. This solution takes the construction of an image processing model based on a generative adversarial network (GAN) as an example, provides [sample region image x, label region image y (Ground Truth, GT)] as a training data pair, and trains the generator G and discriminator D in the image processing model. Optionally, the loss function of the generative adversarial network can be determined by the following formula:

[0054] L GAN (x,y)=E x,y [logD(x,y)]+E x,z [log(1-D(x,G(x)))]

[0055] Among them, D(·)∈[0,1] is the output of the discriminator D, which represents the probability that the predicted image is true, G(x) is the generated predicted image, and E x,y [log D(x,y)] means that when the discriminator D inputs the [x,y] data pair, the discriminator D should output 1, that is, make D(x,y) as large as possible, E x,z [log(1-D(x,G(x)))] means that when the discriminator D inputs the [x,G(x)] data pair, the discriminator D should input 0, that is, make D(x,G(x)) as small as possible.

[0056] In learning pixel-to-pixel mapping, the generated image is usually constrained pixel by pixel to make it closer to the label area image, that is, the loss between it and the label area image is calculated. The loss can be represented by different absolute value error loss functions such as L1 loss, L2 loss, smooth L1 loss, etc. In one embodiment, taking L1 loss as the absolute value error loss function as an example, the absolute value error loss function can be expressed as:

[0057] L pixel (x,y)=||yG(x)||1

[0058] Among them, the perceptual loss function perceives the image from the global high-dimensional level on the basis of calculating the feature loss of the low layer (such as pixel color, edge, etc.), which can effectively retain the global information of the image. Taking the VGG-19 network as the perceptual loss benchmark as an example, the L1 distance of multi-layer convolution is used as the perceptual loss. The perceptual loss function can be determined by the following formula:

[0059]

[0060] Among them, F i (·) represents the convolutional feature map of the i-th layer, λ i is the preset loss weight of the i-th layer convolutional feature map.

[0061] By using the above baseline network, we can generate an edited face attribute image that basically meets expectations by learning the local pixel information and global overall information of the image. However, if we only calculate the pixel-by-pixel loss and perceptual loss on the image, it is usually difficult to balance the global and local information differences of the image, which may cause color deviation that is inconsistent with the original image. At this time, we can further constrain the face attributes and only edit the attribute area, which can effectively ensure the consistency of the non-attribute area in the image with the source image.

[0062] In order to edit only the facial attributes, the model needs to be able to accurately identify the pixel locations of the specified facial attributes, that is, the facial attribute mask. However, if the facial attribute mask is supervised, each image needs to be manually labeled, and there are ambiguities in the regional details of different attributes, and the workload is huge. Therefore, this scheme can learn the facial attribute mask in an unsupervised manner. In the traditional point-to-point learning image mapping method, the input image is directly obtained through the model to generate the generated image, that is, I pred =G(x). This solution can generate a facial attribute mask based on the image generated by the image processing model output, that is, I pred ,M pred =G(x), this step does not require the direct participation of the label area image, and only requires adding a regularization penalty to the input sample area image to constrain it. Correspondingly, the L1 regularization loss function provided by this solution can be determined by the following formula:

[0063] L mask (x)=||x||1

[0064] In one possible embodiment, the absolute value error loss function provided by the present solution can be determined based on the pixel-by-pixel loss of the sample generated image and the label area image generated by the image processing model according to the sample area image, and the pixel-by-pixel loss of the sample area fusion image and the label area image, wherein the sample area fusion image is obtained by fusing the sample facial attribute mask, the sample generated image and the sample area image generated by the image processing model (for example, the sample facial attribute mask is used as the fusion weight to fuse the sample area image and the sample generated image to obtain the sample area fusion image). For example, the pixel-by-pixel loss of the sample generated image and the label area image generated by the image processing model according to the sample area image, and the sum of the pixel-by-pixel losses of the sample area fusion image and the label area image, or the weighted sum of the two, can be used as the absolute value error loss function. In one possible embodiment, the absolute value error loss function provided by the present solution is determined by the following formula:

[0065] L′ pixel (x,y)=||yI pred ||1+||yI blend ||1

[0066] Among them, x is the sample area image, y is the label area image, I pref Generate an image for the sample, I blend is the sample area fusion image, where ||yI pred ||1 is the pixel-by-pixel loss of the sample generated image and the label area image generated by the image processing model based on the sample area image, ||yI blend||1 is the pixel-by-pixel loss of the sample area fusion image and the label area image. Based on this, the objective function can be expressed as:

[0067] L'=λ GAN L GAN (x,y)+λ pixel L′ pixel (x,y)+λ perc L perc (x,y)+λ mask L mask (x)

[0068] This scheme determines the absolute value error loss function based on the pixel-by-pixel loss of the sample generated image and the label area image, as well as the pixel-by-pixel loss of the sample area fusion image and the label area image, effectively constraining the generation of facial attribute masks and ensuring that the fused regional fusion image has better consistency with the image to be processed, thereby improving the facial attribute editing effect.

[0069] In one possible embodiment, the image processing model provided by the present solution can also be fine-tuned based on the inter-frame motion constraint regularization term. It needs to be explained that the generative adversarial network can effectively learn point-to-point image mapping, however, this will be very sensitive to small changes in the input. The general generative adversarial network has a good performance in generating images, but it will produce temporal instability when applied to video sequences, resulting in flickering, artifacts and other inconsistencies in the image over time. In convolutional neural networks (CNNs), it is usually necessary to estimate dense frame-to-frame motion information (optical flow) during the training or inference phase, or to explore cyclic learning structures to make the model suitable for video sequences. Such methods are applied to the generative adversarial network for face attribute editing, which will bring additional and unnecessary computational overhead. The present solution solves this problem by using temporal stability as a regularization of the loss function. The regularization is configured to take into account different types of motion that may occur between frames, so that a temporally stable generative adversarial network can be trained without the need for video material or expensive motion estimation.

[0070] In order to ensure the temporal consistency of the video, CNNs usually consider calculating the loss between two consecutive frames and

[0071] E=||y t -W(y t-1 )||2

[0072] Where W describes the warping operation from frame to frame using the optical flow field between two frames. If there are inter-frame changes that cannot be explained by the flow field motion, these changes will be recorded as inconsistencies. In order to regularize this metric without the need for video data or optical flow information, this scheme introduces intra-frame warping and geometric transformation W(x)=T(x). The perturbation function T(·) in all introduced regularization terms depends on the transformation of the input image, which can effectively simulate the motion that may occur between frames in the video sequence. This scheme uses simple geometric transformations to achieve the simulation of the motion that may occur between frames in the video sequence. These transformations include translation, rotation, scaling, and shearing, which are applied to the input image as a transformation matrix. The matrix is ​​randomly specified for each image, and its transformation parameters are selected from the range in the table below through a uniform distribution.

[0073] Transformation Parameters Minimum Maximum Translation -2px 2px Rotation -1° 1° Zoom 0.97× 1.03× Shearing -1° 1°

[0074] By simulating the possible motion between frames in a video sequence through the above transformation operation, x and T(x) can be used to simulate two consecutive frames, and the output results G(x) and G(T(x)) of the two frames can be further inferred through the generator G of the image processing model. If G(x) and G(T(x)) are consistent in timing, the two frames should produce the same results. Based on this, the inter-frame motion constraint regularization term provided by this solution can be determined by the following formula:

[0075] L stability (x)=||G(T(x))-T(G(x))||2

[0076] Where x is the sample area image, G(·) is the image output by the generator in the image processing model, and T(·) is the result of inter-array motion prediction for the image.

[0077] Based on this, fine-tuning of the inter-frame motion constraint regularization term can be performed based on the following objective function:

[0078] L”=λ GAN L GAN (x,y)+λ pixel L pixel (x,y)+λ perc L perc (x,y)+λ mask L mask (x)+λ stability L stability (x)

[0079] or

[0080] L”=λ GAN L GAN (x,y)+λ pixel L′ plxel(x,y)+λ perc L perc (x,y)+λ mask L mask (x)+λ stability L stability (x)

[0081] Among them, λ stability is the inter-frame motion constraint regularization term weight corresponding to the inter-frame motion constraint regularization term.

[0082] Exemplarily, a training set including sample pairs [sample area image x, label area image y] can be used to train the image processing model based on the objective function L or L', and then the image processing model can be fine-tuned based on the objective function L". When the image processing model is trained based on the objective function L or L', the weights corresponding to the adversarial network loss function, the absolute error loss function, the perceptual loss function and the L1 regularization loss function can be set to a combination of a larger first preset value, and when the image processing model is fine-tuned based on the objective function L", the weights corresponding to the adversarial network loss function, the absolute error loss function, the perceptual loss function and the L1 regularization loss function can be set to a combination of a smaller second preset value, and the weight of the inter-frame motion constraint regularization term is set to a larger value (greater than the second preset value). This solution fine-tunes the image processing model based on the inter-frame motion constraint regularization term, so that the image processing model can ensure that the video sequence has good temporal stability and improve the face attribute editing effect. When constraining the inter-frame motion, fine-tuning is only performed during the training phase of the image processing model. There is no additional computational overhead for reasoning about the image processing model, and it is completely computationally friendly to the mobile terminal, reducing the difficulty of adapting the image processing model on the mobile terminal.

[0083] S230: Using the facial attribute mask as a fusion weight, a fusion process is performed on the regional image and the generated image to obtain a regional fused image.

[0084] S240: replacing the corresponding region in the image to be processed with the regional fusion image to obtain a target image.

[0085] In a possible embodiment, the face attribute editing method provided by the present solution may transform the face attribute mask according to a preset transformation coefficient before using the face attribute mask as a fusion weight to fuse the regional image and the generated image to obtain the regional fused image. Based on this, the face attribute editing method provided by the present solution may use the face attribute mask as a fusion weight to fuse the regional image and the generated image to obtain the regional fused image, which may be to use the transformed face attribute mask as a fusion weight to fuse the regional image and the generated image to obtain the regional fused image.

[0086] Exemplarily, after obtaining the facial attribute mask corresponding to the regional image, the facial attribute mask can be transformed according to the preset transformation coefficient to achieve the transformation of the fusion weight. After completing the transformation of the facial attribute mask, the transformed facial attribute mask can be used as the fusion weight to fuse the regional image and the generated image to obtain the regional fusion image. Optionally, the transformation of the facial attribute mask can be a power law transformation (gamma transformation) of the facial attribute mask. For example, the transformation of the facial attribute mask can be expressed by the following formula:

[0087]

[0088] Among them, M′ pred is the face attribute mask after transformation, M pred is the face attribute mask before transformation, a is the preset maximum pixel value, for example, a=255, and γ is the preset transformation coefficient (gamma transformation coefficient). This solution uses the face attribute mask after transformation to fuse the regional image and the generated image, and controls the intensity of face attribute editing through the preset transformation coefficient, thereby improving the flexibility of face attribute editing.

[0089] In the above, by extracting the regional image corresponding to the preset facial attribute from the image to be processed, generating the generated image and the facial attribute mask corresponding to the preset facial attribute editing effect according to the regional image, using the facial attribute mask as the fusion weight to fuse the regional image and the generated image to obtain the regional fusion image, and replacing the corresponding region in the image to be processed with the regional fusion image to obtain the target image, the noise of the generated image can be effectively reduced, the facial attribute editing effect of the image is more realistic, and the facial attribute editing effect is improved. At the same time, the generated image and the facial attribute mask corresponding to the regional image are accurately determined by the image processing model, and the image processing model is trained by the objective function determined according to the generative adversarial network loss function, the absolute value error loss function, the perceptual loss function and the L1 regularization loss function, which can effectively improve the accuracy of the image processing model in generating facial attribute masks and generated images.

[0090] Figure 3 Schematic diagram of the structure of a face attribute editing device provided by an embodiment of the present application. Figure 3 The face attribute editing device includes an image extraction module 31, an image processing module 32, an image fusion module 33 and an image generation module 34.

[0091] Among them, the image extraction module 31 is configured to obtain the image to be processed and extract the regional image from the image to be processed; the image processing module 32 is configured to generate a generated image and a facial attribute mask corresponding to a preset facial attribute editing effect based on the regional image; the image fusion module 33 is configured to use the facial attribute mask as a fusion weight to fuse the regional image and the generated image to obtain a regional fused image; the image generation module 34 is configured to replace the corresponding area in the image to be processed with the regional fused image to obtain the target image.

[0092] In the above, by extracting the regional image corresponding to the preset facial attributes from the image to be processed, generating the generated image corresponding to the preset facial attribute editing effect and the facial attribute mask according to the regional image, using the facial attribute mask as the fusion weight to fuse the regional image and the generated image to obtain the regional fused image, and replacing the corresponding area in the image to be processed with the regional fused image to obtain the target image, the noise of the generated image can be effectively reduced, the facial attribute editing effect of the image is more realistic, and the facial attribute editing effect is improved.

[0093] In one possible embodiment, the image processing module 32 generates a generated image and a facial attribute mask corresponding to a preset facial attribute editing effect based on the regional image, and is configured as follows: the regional image is input into a trained image processing model, and a generated image and a facial attribute mask corresponding to a preset facial attribute editing effect are generated based on the regional image through the image processing model, and the objective function of the trained image processing model is determined based on the generative adversarial network loss function, the absolute value error loss function, the perceptual loss function and the L1 regularization loss function.

[0094] In a possible embodiment, the absolute value error loss function is determined based on the pixel-by-pixel loss of a sample generated image and a label area image generated by an image processing model according to the sample area image, as well as the pixel-by-pixel loss of a sample area fusion image and a label area image. The sample area fusion image is obtained by fusing a sample facial attribute mask, a sample generated image and a sample area image generated by the image processing model.

[0095] In one possible embodiment, the absolute value error loss function is determined by the following formula:

[0096] L′ pixel (x,y)=||yI pred ||1+||yI blend ||1

[0097] Among them, x is the sample area image, y is the label area image, I pred Generate an image for the sample, I blend Fuse the images for the sample region.

[0098] In a possible embodiment, the image processing model is fine-tuned based on the inter-frame motion constraint regularization term, and the inter-frame motion constraint regularization term is determined by the following formula:

[0099] L stability (x)=||G(T(x))-T(G(x))||2

[0100] Where x is the sample area image, G(·) is the image output by the generator in the image processing model, and T(·) is the result of inter-array motion prediction for the image.

[0101] In a possible embodiment, the regional fusion image is determined by the following formula:

[0102] I out =M pred ×I pred +(255-M pred )×I in

[0103] Among them, M pred is the face attribute mask, I pred To generate an image, a is the preset maximum pixel value, I in For regional images.

[0104] In a possible embodiment, the face attribute editing device further includes a mask transformation module, and the mask transformation module is configured to transform the face attribute mask according to a preset transformation coefficient;

[0105] The image fusion module 33 uses the facial attribute mask as the fusion weight to fuse the regional image and the generated image to obtain a regional fused image, and is configured as: using the transformed facial attribute mask as the fusion weight to fuse the regional image and the generated image to obtain a regional fused image.

[0106] It is worth noting that in the above-mentioned embodiment of the face attribute editing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0107] The embodiment of the present application also provides a facial attribute editing device, which can integrate the facial attribute editing apparatus provided in the embodiment of the present application. Figure 4 Schematic diagram of the structure of a face attribute editing device provided by an embodiment of the present application. Figure 4The face attribute editing device includes: an input device 43, an output device 44, a memory 42 and one or more processors 41; the memory 42 is used to store one or more programs; when one or more programs are executed by one or more processors 41, the one or more processors 41 implement the face attribute editing method provided in the above embodiment. The face attribute editing device, equipment and computer provided above can be used to execute the face attribute editing method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0108] The embodiments of the present application also provide a non-volatile storage medium storing computer executable instructions, which are used to execute the facial attribute editing method provided in the above embodiments when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiments of the present application, whose computer executable instructions are not limited to the facial attribute editing method provided above, can also execute the related operations in the facial attribute editing method provided in any embodiment of the present application. The facial attribute editing apparatus, device and storage medium provided in the above embodiments can execute the facial attribute editing method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the facial attribute editing method provided in any embodiment of the present application.

[0109] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, including a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the facial attribute editing method provided in each embodiment of the present application.

Claims

1. A method for editing face attributes, characterized in that: include: Acquire an image to be processed, and extract a region image corresponding to a preset facial attribute from the image to be processed; Generate a generated image and a facial attribute mask corresponding to a preset facial attribute editing effect according to the regional image; Using the facial attribute mask as a fusion weight to fuse the regional image and the generated image to obtain a regional fused image; The target image is obtained by replacing the corresponding region in the image to be processed with the regional fusion image.

2. The method for editing face attributes according to claim 1, characterized in that: The step of generating a generated image and a facial attribute mask corresponding to a preset facial attribute editing effect according to the regional image includes: The regional image is input into a trained image processing model, and a generated image and a facial attribute mask corresponding to a preset facial attribute editing effect are generated according to the regional image by the image processing model. The objective function of training the image processing model is determined according to the generative adversarial network loss function, the absolute value error loss function, the perceptual loss function and the L1 regularization loss function.

3. The method for editing face attributes according to claim 2, characterized in that: The absolute value error loss function is determined based on the pixel-by-pixel loss of the sample generated image and the label area image generated by the image processing model according to the sample area image, and the pixel-by-pixel loss of the sample area fusion image and the label area image. The sample area fusion image is obtained by fusing the sample facial attribute mask generated by the image processing model, the sample generated image and the sample area image.

4. The method for editing face attributes according to claim 3, characterized in that: The absolute value error loss function is determined by the following formula: L′ pixel (x,y)=||y-I pred ||1+||y-I blend ||1 Among them, x is the sample area image, y is the label area image, I pred Generate an image for the sample, I blend Fuse the images for the sample region.

5. The method for editing face attributes according to claim 2, characterized in that: The image processing model is fine-tuned based on the inter-frame motion constraint regularization term, and the inter-frame motion constraint regularization term is determined by the following formula: L stability (x)=∥G(T(x))-T(G(x))∥2 Wherein, x is the sample area image, G(·) is the image output by the generator in the image processing model, and T(·) is the result of inter-array motion prediction for the image.

6. The method for editing face attributes according to claim 1, characterized in that: The regional fusion image is determined by the following formula: I out =M pred ×I pred +(255-M pred )×I in Among them, M pred is the face attribute mask, I pred To generate an image, a is the preset maximum pixel value, I im For regional images.

7. The method for editing face attributes according to claim 1, characterized in that: Before fusing the regional image and the generated image using the facial attribute mask as a fusion weight to obtain a regional fused image, the method further includes: Performing transformation processing on the facial attribute mask according to a preset transformation coefficient; The step of fusing the regional image and the generated image using the facial attribute mask as a fusion weight to obtain a regional fused image includes: The transformed facial attribute mask is used as a fusion weight to fuse the regional image and the generated image to obtain a regional fused image.

8. A facial attribute editing device, characterized in that: It includes an image extraction module, an image processing module, an image fusion module and an image generation module, wherein: The image extraction module is configured to obtain an image to be processed and extract a regional image from the image to be processed; The image processing module is configured to generate a generated image corresponding to a preset facial attribute editing effect and a facial attribute mask according to the regional image; The image fusion module is configured to use the facial attribute mask as a fusion weight to perform fusion processing on the regional image and the generated image to obtain a regional fusion image; The image generation module is configured to replace the corresponding area in the image to be processed with the regional fusion image to obtain a target image.

9. A facial attribute editing device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the facial attribute editing method as described in any one of claims 1-7.

10. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions, when executed by a computer processor, are used to execute the facial attribute editing method as described in any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the facial attribute editing method described in any one of claims 1 to 7 is implemented.

Images