Face image editing method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides a face image editing method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, the to-be-processed local image is extracted from the to-be-processed image, the image residual corresponding to the preset image effect is generated according to the to-be-processed local image, and the target local image is generated according to the image residual and the to-be-processed local image. The target local image is added to the to-be-processed image to obtain the target image, the whole face image does not need to be input into a generative model for face editing, the area, outside the to-be-processed local part, of the face in the image is not changed, the face details of the image can be reserved, and meanwhile the face editing efficiency is improved. The influence of face image editing on the image definition is effectively reduced, and the face image editing effect is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a facial image editing method, device, equipment, storage medium and product. Background Art

[0002] With the development of Internet technology and image processing technology, people are using digital entertainment products more and more widely, and digital entertainment products are providing more and more functions and gameplay.

[0003] For example, in video applications, face image editing functions are often provided, such as converting the face in the video into a different face (for example, changing the face into a child's face, changing the face style, etc.), or editing the facial features (for example, changing the single eyelids into double eyelids, double eyelids into single eyelids, etc.). Traditional face image editing solutions generally input the entire face image into a generative model, and the generative model outputs the converted face image. However, the face images generated by this face image editing method are prone to losing facial details and have a greater impact on image clarity, resulting in poor face image editing effects. Summary of the invention

[0004] The embodiments of the present application provide a facial image editing method, device, equipment, storage medium and product to solve the technical problems in the related art that facial image editing is prone to lose facial details, has a significant impact on image clarity, and has a poor facial image editing effect. The method can effectively retain facial details, reduce the impact on image clarity, and improve the facial image editing effect.

[0005] In a first aspect, an embodiment of the present application provides a facial image editing method, comprising:

[0006] Extracting a local image to be processed from the image to be processed;

[0007] generating an image residual corresponding to a preset image effect according to the partial image to be processed;

[0008] Generate a target local image according to the image residual and the local image to be processed;

[0009] The target partial image is added to the image to be processed to obtain a target image.

[0010] In a second aspect, an embodiment of the present application provides a face image editing device, including an image extraction module, a residual generation module, a local processing module and an image generation module, wherein:

[0011] The image extraction module is configured to extract a local image to be processed from the image to be processed;

[0012] The residual generation module is configured to generate an image residual corresponding to a preset image effect according to the partial image to be processed;

[0013] The local processing module is configured to generate a target local image according to the image residual and the local image to be processed;

[0014] The image generation module is configured to add the target partial image to the image to be processed to obtain a target image.

[0015] In a third aspect, an embodiment of the present application provides a facial image editing device, including: a memory and one or more processors;

[0016] The memory is used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the facial image editing method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the facial image editing method as described in the first aspect.

[0019] In the fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the facial image editing method described in the first aspect.

[0020] The embodiment of the present application extracts a local image to be processed from the image to be processed, generates an image residual corresponding to a preset image effect according to the local image to be processed, generates a target local image according to the image residual and the local image to be processed, and adds the target local image to the image to be processed to obtain the target image. It does not need to input the entire face image into the generative model for face editing, and will not change the area of ​​the face in the image outside the local area to be processed. While retaining the facial details of the image, it can effectively reduce the impact of face image editing on image clarity and improve the face image editing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a facial image editing method provided by an embodiment of the present application;

[0022] Figure 2 is a flowchart of another facial image editing method provided by an embodiment of the present application;

[0023] Figure 3 is a training diagram of an image processing model provided in an embodiment of the present application;

[0024] Figure 4 It is a loss calculation diagram of an image processing model provided in an embodiment of the present application;

[0025] Figure 5 It is a structural schematic diagram of a facial image editing device provided in an embodiment of the present application;

[0026] Figure 6 It is a structural schematic diagram of a face image editing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0028] The face image editing method provided in this application can be applied to face editing of images. For example, in scenarios such as mobile live broadcast, video editing, and image editing, the face beauty and special effects functions can be realized based on the face image editing method. The face beauty and special effects functions can be the adjustment of the facial features (eyes, ears, mouth, nose, and eyebrows), such as changing a single eyelid to a double eyelid. The purpose is to generate an image residual corresponding to the local image to be processed extracted from the image to be processed, and obtain the target image based on the image residual. It is not necessary to input the entire face image into the generative model for face editing, thereby improving the face image editing effect.

[0029] In existing face image editing solutions, the entire target face is usually generated by inputting the entire face image into a generative model. This face image editing method that generates the entire face is prone to losing some details of the face and image clarity. In addition, the entire face image has too many pixels, and the computational complexity of the generative model is too large. For mobile devices with low performance, it is easy for the device to heat up and drop frames. Based on this, a face image editing method of an embodiment of the present application is provided to solve the technical problems that the existing face image editing is prone to losing facial details, has a great impact on image clarity, and has a poor face image editing effect.

[0030] Figure 1 A flowchart of a facial image editing method provided in an embodiment of the present application is given. The facial image editing method provided in an embodiment of the present application can be executed by a facial image editing device, which can be implemented by hardware and / or software and integrated in a facial image editing device.

[0031] The following description is made by taking the face image editing device executing the face image editing method as an example. Figure 1 , the face image editing method comprises:

[0032] S110: extracting a partial image to be processed from the image to be processed.

[0033] Exemplarily, an image to be processed that needs to be edited for a human face is obtained, wherein the image to be processed may be a live video frame collected in real time during a live video broadcast, may be a video frame in a recorded video, or may be a single image.

[0034] In one embodiment, after obtaining the image to be processed, a partial image to be processed is extracted from the image to be processed. The partial image to be processed is an area in the image to be processed that needs to be edited for the face, which may be an area corresponding to the facial features in the image to be processed. For example, the area to be edited for the face may be an area corresponding to a combination of one or more of the eyes, ears, mouth, nose, and eyebrows in the image to be processed.

[0035] Optionally, the partial image to be processed extracted from the image to be processed may be an image corresponding to a region corresponding to a combination of multiple types of eyes, ears, mouth, nose, and eyebrows, that is, one partial image to be processed may include multiple facial features, in which case facial image editing of multiple facial features can be realized simultaneously through one partial image to be processed. The partial image to be processed extracted from the image to be processed may also be an image corresponding to a region corresponding to one type of eyes, ears, mouth, nose, and eyebrows, that is, one partial image to be processed corresponds to one type of facial features, in which case facial image editing of one facial feature can be realized through one partial image to be processed.

[0036] S120: Generate an image residual corresponding to a preset image effect according to the partial image to be processed.

[0037] Exemplarily, after obtaining the partial image to be processed, an image residual corresponding to a preset image effect is generated according to the partial image to be processed. The preset image effect can be determined according to the effect of editing the face image. For example, the preset image effect can be facial feature editing effects such as changing single eyelids to double eyelids, double eyelids to single eyelids, adding lipstick, changing lipstick color, making ears pointed, eyebrows thicker, and nose narrower. The preset image effect can be one or more different facial feature editing effects.

[0038] Optionally, the local image to be processed can be processed by a pre-completed image processing model (such as a GAN (Generative Adversarial Network) model) to generate an image residual corresponding to a preset image effect. In one embodiment, the image processing model can adopt a lightweight model structure, which can effectively reduce the amount of calculation of the model and the computing overhead of the device, and the image processing model can be subjected to residual learning, which can effectively retain the local details of the facial features while reducing the difficulty of model training and improving the adaptability of face image editing on different terminals.

[0039] S130: Generate a target partial image according to the image residual and the partial image to be processed.

[0040] Exemplarily, after determining the image residual corresponding to the partial image to be processed, the target partial image can be generated according to the image residual and the partial image to be processed. For example, the image residual and the partial image to be processed are summed to obtain the target partial image. The size of the image residual is consistent with the size of the partial image to be processed, and the size of the target partial image is consistent with the size of the partial image to be processed.

[0041] Among them, the target local image can be understood as an image of facial features that has been edited and processed for the corresponding facial features. For example, when the local image to be processed is an image of a human eye with single eyelids, and the preset image effect is to change the single eyelid into a double eyelid, the target local image is an image of a human eye with double eyelids.

[0042] In a possible embodiment, the facial image editing method provided by the present solution generates a target local image based on the image residual and the local image to be processed, and can also be performed by summing the product of a preset effect weight and the image residual with the local image to be processed, and determining the summation result as the target local image.

[0043] Exemplarily, after determining the image residual corresponding to the local image to be processed, the preset effect weight and the image residual are multiplied, and then the product of the preset effect weight and the image residual is added to the local image to be processed to obtain the target local image. For example, assuming that the preset effect weight is α, the image residual is R, and the local image to be processed is I in , the target local image can be expressed as: I out =I in +αR. This scheme can flexibly configure the adjustment degree of face image editing by presetting effect weights, and more flexibly adjust the output of the target image.

[0044] S140: Add the target partial image to the image to be processed to obtain the target image.

[0045] Exemplarily, after obtaining the target partial image, the target partial image is added to the image to be processed to obtain the target image, that is, the target partial image replaces or covers the area corresponding to the partial image to be processed in the image to be processed to obtain the target image.

[0046] This solution can obtain a target image that has achieved face editing by replacing the area corresponding to the local image to be processed in the image to be processed with the target local image that has completed editing. There is no need to input the entire face image into the generative model for image processing, which effectively reduces the data processing volume of the device. In addition, only the area corresponding to the preset facial features needs to be changed, and there is no need to change the area outside the preset facial features. The facial details of the area outside the preset facial features can be effectively retained, and the face image editing effect is more natural.

[0047] In the above, by extracting the local image to be processed from the image to be processed, generating an image residual corresponding to a preset image effect according to the local image to be processed, generating a target local image according to the image residual and the local image to be processed, and adding the target local image to the image to be processed to obtain the target image, there is no need to input the entire face image into the generative model for face editing, and the area of ​​the face in the image outside the local area to be processed will not be changed. While retaining the facial details of the image, the impact of face image editing on image clarity can be effectively reduced, thereby improving the face image editing effect.

[0048] Based on the above embodiments, Figure 2 A flowchart of another face image editing method provided in an embodiment of the present application is given, and the face image editing method is a specific embodiment of the face image editing method described above. Figure 2 , the face image editing method comprises:

[0049] S210: Determine the facial key points in the image to be processed, and extract the local image to be processed from the image to be processed according to the facial key points.

[0050] Exemplarily, an image to be processed is obtained, and facial key points are detected on the image to be processed to determine multiple facial key points in the image to be processed. Among them, facial key points are important feature points of various parts of the face in the image, usually contour points and corner points, including eyebrows, eyes, nose, mouth and other points. Facial key points can use 68, 106 or 240 points to represent the current facial features. For example, in order to balance the accuracy and time-consuming efficiency of key point detection, 106 points can be used to mark facial key points. Optionally, a pre-trained face detection model can be used to detect facial key points of the image to be processed.

[0051] In one embodiment, after determining a plurality of facial key points in the image to be processed, a partial image to be processed is extracted from the image to be processed according to the facial features to be edited and the plurality of facial key points. For example, a plurality of target facial key points corresponding to the facial features to be edited are determined, and a bounding box corresponding to the facial features to be edited is determined based on these target facial key points (such as the maximum coordinate point and the minimum coordinate point on the horizontal and vertical axes of the target facial key points) (it can also be scaled on the basis of the bounding box), and the pixel point range corresponding to the bounding box of the image to be processed is extracted to obtain the partial image to be processed. This solution accurately extracts the partial image to be processed according to the facial key points in the image to be processed, and accurately performs facial editing on the partial image to be processed. While retaining the facial details of the image, it effectively reduces the impact of facial image editing on image clarity, thereby ensuring the effect of facial image editing.

[0052] In one embodiment, when the facial features that need to be edited include two left and right parts (for example, the eyes include the left and right eyes, and the ears include the left and right ears), the bounding box can enclose the two left and right parts at the same time, or two local images to be processed can be determined for the facial key points corresponding to the left and right parts respectively.

[0053] Optionally, when two local images to be processed are determined for the facial key points corresponding to the left and right parts respectively, image residuals can be generated for the two local images to be processed respectively, and target local images at corresponding positions are generated according to the corresponding image residuals respectively, and the target local images are used to replace the corresponding areas in the image to be processed to obtain the target image.

[0054] Optionally, when two local images to be processed are determined respectively for the facial key points corresponding to the left and right parts, the two local images to be processed can be synthesized into one local image to be processed, and then the image residual corresponding to the local image to be processed is determined, and a target local image containing the editing effects corresponding to the two parts is generated from the synthesized local image to be processed according to the image residual, and then the target local image is split to obtain target local images corresponding to the two parts, and the two target local images are used to replace the corresponding areas in the image to be processed to obtain the target image.

[0055] In a possible embodiment, the face image editing method provided by the present solution extracts a partial image to be processed from the image to be processed according to the face key points, including:

[0056] S211: Extracting a first partial image and a second partial image from the image to be processed according to facial key points.

[0057] S212: Combining the first partial image and the second partial image into a partial image to be processed.

[0058] Exemplarily, when the facial features to be edited include two parts on the left and right, the first partial image and the second partial image can be extracted from the image to be processed according to the facial key points corresponding to the two parts on the left and right. For example, taking the facial features to be edited as eyes as an example, the first partial image is extracted from the image to be processed according to the facial key points corresponding to the left eye of the face, and the second partial image is extracted from the image to be processed according to the facial key points corresponding to the right eye of the face.

[0059] In one embodiment, after obtaining the first partial image and the second partial image, the first partial image and the second partial image may be combined into a partial image to be processed. Optionally, the first partial image and the second partial image may be spliced ​​up and down to obtain the partial image to be processed, or the first partial image and the second partial image may be spliced ​​left and right to obtain the partial image to be processed.

[0060] This solution can retain the images of the facial features that need to be edited to the greatest extent by extracting the first partial image and the second partial image from the image to be processed according to the key points of the face, reduce the changes to the areas other than the facial features that need to be edited, and retain the details and clarity of the face to a greater extent. At the same time, the first partial image and the second partial image are synthesized into the partial image to be processed, and the partial images to be processed are processed uniformly, reducing the additional computational overhead caused by the image processing model to process the first partial image and the second partial image separately, greatly reducing the input size of the image processing model, and can significantly reduce the computational overhead of the device, improve the editing efficiency of the face image, and only process the facial features, reducing the loss of detail information of other facial parts except the facial features, and retaining the details and clarity of the face to a greater extent.

[0061] In a possible embodiment, the face image editing method provided by the present solution synthesizes the first partial image and the second partial image into a partial image to be processed, including:

[0062] S2121: Perform affine transformation alignment processing on the first partial image and the second partial image.

[0063] S2122: Combining the first partial image and the second partial image after affine transformation alignment processing into a partial image to be processed.

[0064] Exemplarily, after obtaining the first local image and the second local image, the first local image and the second local image may be subjected to affine transformation alignment processing, for example, the first local image and / or the second local image may be rotated, translated, and cropped so that the first local image and the second local image are aligned in height and / or width, and the first local image and the second local image subjected to affine transformation alignment processing are synthesized into the local image to be processed. For example, taking the scene of a face with single eyelids becoming double eyelids as an example, the center of the eyes in the first local image and the second local image, as well as the size to be cropped, may be calculated respectively according to the key points of the eyes, the two eye images may be aligned through affine transformation, and finally the two eyes may be spliced ​​up and down to obtain the combined image to be processed. This solution effectively fixes the image distribution of the input image processing model by performing affine transformation alignment processing on the first local image and the second local image, improves the processing efficiency and accuracy of the image processing model, and effectively reduces the network learning difficulty of the image processing model, thereby improving the training efficiency and accuracy of the image processing model.

[0065] S220: Input the local image to be processed into the trained image processing model, and generate an image residual corresponding to a preset image effect according to the local image to be processed through the image processing model. The objective function of the trained image processing model is determined according to the generative adversarial network loss function, the absolute value error loss function and the perceptual loss function.

[0066] Exemplarily, this solution generates an image residual corresponding to a local image to be processed through a trained image processing model. After obtaining the local image to be processed, the local image to be processed is input into the image processing model, and the image processing model analyzes and processes the local image to be processed to obtain an image residual corresponding to a preset image effect.

[0067] The objective function of the image processing model trained in this solution is determined according to the generative adversarial network loss function (GANLoss), the absolute value error loss function (L1 Loss) and the perceptual loss function (Perceptual loss). For example, the sum of the adversarial network loss function, the absolute value error loss function and the perceptual loss function, or the weighted sum of the adversarial network loss function, the absolute value error loss function and the perceptual loss function, is used as the objective function of the image processing model training.

[0068] This scheme trains the image processing model by determining the objective function according to the generative adversarial network loss function, the absolute value error loss function and the perceptual loss function. The adversarial network loss function and the absolute value error loss function can make the output of the image processing model closer to the control sample, and the perceptual loss function calculates the loss of the convolution output of the original image and the convolution output of the generated image on the basis of calculating the low-level feature loss (pixel color, edge, etc.), that is, it uses the ability of the convolution layer to abstract high-level features and perceive the image from a global high-dimensional level. The use of perceptual loss can retain the overall information of the image, effectively ensure the authenticity of the generated image, avoid generating unrealistic facial features, and improve the quality of face image editing.

[0069] like Figure 3 As shown in the training diagram of an image processing model provided, taking the scene of a single eyelid on a human face becoming double eyelid as an example, the purpose of image generation is to generate a target facial feature image (such as a double eyelid image) from the input source facial feature image (such as a single eyelid image) through the image processing model inference. The image processing model uses the conditional generative adversarial network (cGAN) as a benchmark, provides [original image x, reference image y (Ground Truth, GT)] as a training data pair, and trains the generator G and discriminator D in the image processing model. Optionally, the adversarial network loss function can be defined as:

[0070] L cGAN (G, D) = E x，y [log D(x,y)]+E x，z [log(1-D(x,G(x,z)))]

[0071] Where z represents the random noise added to the input image, D(·)∈[0,1] is the discriminator output, which represents the probability that the image is real, G(x) is the generated predicted image, and G(x,z) is the predicted image generated when (random) noise z is added. x，y [log D(x, y)] corresponds to Figure 3 On the right, when the discriminator inputs the [x, y] data pair, the discriminator should output 1, that is, make D(x, y) as large as possible. x，z [log(1-D(x, G(x, z)))] corresponds to Figure 3 On the left, it means that when the discriminator inputs the data pair [x, G(x, z)], the discriminator should output 0, that is, make D(x, G(x, z)) as small as possible. For training the generator G, we hope that the generated image is closer to the real one, that is, make D(x, G(x, z)) as large as possible, then L cGAN (G, D) will be larger, so we need to find the minimum min G. For training the discriminator D, the stronger the discriminative ability of D, the larger D(x, y), and the smaller D(x, G(x, z)), then L cGAN (G, D) will be smaller, so we need to find the maximum max D. Based on this, the adversarial network loss function can be expressed as:

[0072]

[0073] In one embodiment, in order to make the image G(x, z) output by the image processing model closer to the GT image y, the L1 distance can be calculated pixel by pixel, that is, the absolute value error loss function:

[0074] L pixel (G) = E x，y，z [||yG(x,z)|| 1 ]

[0075] At the same time, the perceptual loss can use the ability of the convolutional layer to abstract high-level features and perceive the image from a global high-dimensional level. Using perceptual loss can retain the overall information of the image, effectively ensure the authenticity of the generated image, and avoid generating unrealistic facial features. Optionally, the VGG-19 network can be used to calculate the perceptual loss. Optionally, the perceptual loss can be expressed as:

[0076]

[0077] Among them, this scheme uses the L1 distance of multi-layer convolution as the perceptual loss, F i (·) represents the convolutional feature map of the i-th layer, λ i is the loss weight of the i-th convolutional feature map. Based on this, the objective function of the image processing model trained in this scheme can be expressed as:

[0078]

[0079] Among them, λ pixel is the first loss weight of the absolute value error loss function, λ perceptual is the second loss weight of the perceptual loss.

[0080] In a possible embodiment, the training process of the image processing model provided by the present solution may include: inputting a sample image into the image processing model, generating a sample residual corresponding to a preset image effect based on the sample image through the image processing model; obtaining a generated image based on the sample residual and the sample image, and optimizing parameters of the image processing model based on the objective function and the label images corresponding to the generated image and the sample image.

[0081] Exemplarily, when training an image processing model, a sample image is input into the image processing model, and the image processing model generates a sample residual corresponding to a preset image effect according to the sample image. After obtaining the sample residual, a generated image is obtained according to the sample residual and the sample image, for example, the sample residual and the sample image are added to obtain the generated image.

[0082] In one embodiment, after the generated image is obtained, based on the objective function, the model loss is calculated according to the generated image and the label image corresponding to the sample image, and the parameters of the image processing model are optimized according to the model loss.

[0083] like Figure 4 As shown in the loss calculation diagram of an image processing model provided, the image processing model processes a sample image (input image x in the figure) through a generator, and the generator outputs a sample residual (network output Net{out} in the figure), and the sample residual is added to the sample image to obtain a generated image (model output G(x, z) in the figure). Based on the objective function, the loss can be calculated according to the generated image and the label image, and the parameters of the image processing model can be optimized according to the loss calculation results.

[0084] This scheme improves the quality of image generation by learning the difference (i.e., residual) between the input image and the target image. Since it is difficult to directly generate the target image, especially when the image details are complex, this scheme can effectively reduce the difficulty of network learning by learning the residual. The image processing model only needs to learn how to obtain the missing details and features in the target image from the input image, rather than generating the entire image from scratch. At the same time, residual learning can help the image processing model pay more attention to the difference in details, so that the image obtained based on the output residual is closer to the target image in details, effectively improving the overall quality of image generation. Since the image processing model learns the changes or details of the image, rather than the entire image, the image processing model can better adapt to the changes in the input image and provide better generalization ability. The generator in the image processing model learns the image residual, keeps the overall network structure of the image processing model unchanged, adds the input image to the result of the forward propagation as the network output, and calculates the loss function, that is, the original G(x, z) = Net out becomes G(x, z)=x+Net out , still represents the output of the model, and can be used to calculate the loss with the label image to optimize the parameters of the image processing model.

[0085] In one embodiment, in order to meet the needs of real-time face editing functions on mobile devices, it is necessary to consider the computational overhead of the model to avoid occupying too much computing resources of the mobile device. This solution will introduce a depth-separable convolution module based on the U-Net network structure commonly used by the generator G in the generative adversarial network, while reducing the number of convolution channels and the convolution depth, effectively reducing the number of floating-point operations (FLOPs) of model calculations, reducing the computational overhead of the model before image processing, and ensuring that the image processing model runs well on mobile devices.

[0086] S230: Generate a target partial image according to the image residual and the partial image to be processed.

[0087] S240: Add the target partial image to the image to be processed to obtain the target image.

[0088] In one embodiment, the face image editing method provided by the present solution adds a target partial image to an image to be processed to obtain a target image, including:

[0089] S241: when the partial image to be processed is obtained based on the first partial image and the second partial image, obtain a first target partial image and a second target partial image according to the target partial image.

[0090] S242: Add the first target partial image and the second target partial image to the image to be processed to obtain a target image.

[0091] Exemplarily, when the local image to be processed is obtained based on the first local image and the second local image, after obtaining the target local image, the target local image can be split into a first target local image and a second target local image according to the splicing method of the first local image and the second local image, wherein the sizes of the first target local image and the second target local image are consistent with the sizes of the first local image and the second local image.

[0092] For example, when the first local image and the second local image are spliced ​​together vertically to obtain the local image to be processed, the target local image is split vertically into the first target local image and the second target local image; when the first local image and the second local image are spliced ​​together horizontally to obtain the local image to be processed, the target local image is split horizontally into the first target local image and the second target local image.

[0093] In one embodiment, after obtaining the first target partial image and the second target partial image, the first target partial image and the second target partial image are added to the image to be processed according to their positions on the image to be processed to obtain the target image. At this time, the images at the positions corresponding to the first partial image and the second partial image in the target image are replaced with the images corresponding to the first target partial image and the second target partial image. This solution can retain the images of the facial features that need to be edited to the greatest extent possible, reduce the changes to the areas other than the facial features that need to be edited, and retain the details and clarity of the face to the greatest extent possible by splitting the target partial image into the first target partial image and the second target partial image and adding them to the image to be processed when the partial image to be processed is obtained based on the first partial image and the second partial image.

[0094] In a possible embodiment, the face image editing method provided by the present solution adds the first target partial image and the second target partial image to the image to be processed to obtain the target image, which can be: performing inverse affine transformation alignment processing on the first target partial image and the second target partial image; adding the first target partial image and the second target partial image after the inverse affine transformation alignment processing to the image to be processed to obtain the target image.

[0095] Among them, the local image to be processed provided by the present solution is synthesized from the first local image and the second local image after affine transformation alignment processing. Correspondingly, after splitting the target local image to obtain the first target local image and the second target local image, the present solution performs inverse affine transformation alignment processing on the first target local image and the second target local image based on the affine transformation alignment processing method for the first local image and the second local image, thereby converting the specifications of the first target local image and the second target local image to be consistent with the specifications of the first local image and the second local image before affine transformation alignment. After completing the inverse affine transformation alignment processing of the first target local image and the second target local image, the first target local image and the second target local image after the inverse affine transformation alignment processing can be added to the image to be processed to obtain the target image. At this time, the first target local image and the second target local image can correctly replace the areas corresponding to the first local image and the second local image in the image to be processed, and obtain a target image in which face editing is performed on the facial features while completely retaining the details of the face.

[0096] As described above, by extracting the local image to be processed from the image to be processed, generating an image residual corresponding to a preset image effect based on the local image to be processed, generating a target local image based on the image residual and the local image to be processed, and adding the target local image to the image to be processed to obtain the target image, it is not necessary to input the entire face image into the generative model for face editing, and the area of ​​the face in the image outside the local area to be processed will not be changed. While retaining the facial details of the image, the impact of face image editing on image clarity can be effectively reduced, thereby improving the face image editing effect. At the same time, by splitting the target local image into a first target local image and a second target local image and adding them to the image to be processed to obtain the target image when the local image to be processed is obtained based on the first local image and the second local image, the image of the facial features that need to be edited can be retained to a greater extent, reducing the changes to the areas outside the facial features that need to be edited, and retaining the details and clarity of the face to a greater extent. Furthermore, when the partial image to be processed is obtained based on the first partial image and the second partial image, the target partial image is split into the first target partial image and the second target partial image and added to the image to be processed to obtain the target image, the image of the facial features to be edited can be retained to a greater extent, and the changes to the areas other than the facial features to be edited can be reduced, thereby retaining the details and clarity of the face to a greater extent. This solution only changes the high-frequency changes and edge details of the facial features of the image to be processed. Compared with the existing face editing solutions that generate the entire face image, this solution can better retain the detailed information of the facial features, while completely retaining the information of other facial parts except the facial features.

[0097] Figure 5Schematic diagram of the structure of a face image editing device provided by an embodiment of the present application. Figure 5 The face image editing device includes an image extraction module 51, a residual generation module 52, a local processing module 53 and an image generation module 54.

[0098] Among them, the image extraction module 51 is configured to extract the local image to be processed from the image to be processed; the residual generation module 52 is configured to generate an image residual corresponding to a preset image effect based on the local image to be processed; the local processing module 53 is configured to generate a target local image based on the image residual and the local image to be processed; the image generation module 54 is configured to add the target local image to the image to be processed to obtain the target image.

[0099] In the above, by extracting the local image to be processed from the image to be processed, generating an image residual corresponding to a preset image effect according to the local image to be processed, generating a target local image according to the image residual and the local image to be processed, and adding the target local image to the image to be processed to obtain the target image, there is no need to input the entire face image into the generative model for face editing, and the area of ​​the face in the image outside the local area to be processed will not be changed. While retaining the facial details of the image, the impact of face image editing on image clarity can be effectively reduced, thereby improving the face image editing effect.

[0100] In a possible embodiment, the image extraction module 51 extracts a partial image to be processed from the image to be processed, and is configured to: determine facial key points in the image to be processed, and extract the partial image to be processed from the image to be processed according to the facial key points.

[0101] In a possible embodiment, the image extraction module 51 extracts the local image to be processed from the image to be processed according to the key points of the face, and is configured as follows:

[0102] Extracting a first partial image and a second partial image from the image to be processed according to the facial key points;

[0103] The first partial image and the second partial image are combined into a partial image to be processed.

[0104] In a possible embodiment, the image extraction module 51 synthesizes the first partial image and the second partial image into a partial image to be processed, and is configured as follows:

[0105] Performing affine transformation alignment processing on the first partial image and the second partial image;

[0106] The first partial image and the second partial image after affine transformation alignment processing are synthesized into a partial image to be processed.

[0107] In a possible embodiment, the image generation module 54 adds the target partial image to the image to be processed to obtain the target image, and is configured as follows:

[0108] In a case where the partial image to be processed is obtained based on the first partial image and the second partial image, obtaining a first target partial image and a second target partial image according to the target partial image;

[0109] The first target partial image and the second target partial image are added to the image to be processed to obtain a target image.

[0110] In a possible embodiment, the image generation module 54 adds the first target partial image and the second target partial image to the image to be processed to obtain the target image, and is configured as follows:

[0111] Performing inverse affine transformation alignment processing on the first target partial image and the second target partial image;

[0112] The first target partial image and the second target partial image which have been processed by inverse affine transformation alignment are added to the image to be processed to obtain the target image.

[0113] In a possible embodiment, the residual generation module 52 generates an image residual corresponding to a preset image effect according to the partial image to be processed, and is configured as follows:

[0114] The local image to be processed is input into the trained image processing model, and the image residual corresponding to the preset image effect is generated according to the local image to be processed by the image processing model. The objective function of the trained image processing model is determined according to the generative adversarial network loss function, the absolute value error loss function and the perceptual loss function.

[0115] In a possible embodiment, the face image editing device further includes a model training module configured to train an image processing model, and the training process of the image processing model is configured as follows:

[0116] The sample image is input into the image processing model, and the sample residual corresponding to the preset image effect is generated according to the sample image by the image processing model;

[0117] A generated image is obtained according to the sample residual and the sample image, and parameters of the image processing model are optimized based on the objective function and the label images corresponding to the generated image and the sample image.

[0118] In a possible embodiment, the local processing module 53 generates a target local image based on the image residual and the local image to be processed, and is configured to: sum the product of the preset effect weight and the image residual and the local image to be processed, and determine the summation result as the target local image.

[0119] It is worth noting that in the above-mentioned embodiment of the face image editing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0120] The embodiment of the present application also provides a facial image editing device, which can integrate the facial image editing apparatus provided in the embodiment of the present application. Figure 6 Schematic diagram of the structure of a face image editing device provided by an embodiment of the present application. Figure 6 The face image editing device is configured as follows: an input device 63, an output device 64, a memory 62, and one or more processors 61; the memory 62 is used to store one or more programs; when one or more programs are executed by one or more processors 61, the one or more processors 61 implement the face image editing method provided in the above embodiment. The face image editing device, equipment, and computer provided above can be used to execute the face image editing method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0121] The embodiments of the present application also provide a non-volatile storage medium storing computer executable instructions, which are used to execute the facial image editing method provided in the above embodiments when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiments of the present application, whose computer executable instructions are not limited to the facial image editing method provided above, can also execute the related operations in the facial image editing method provided in any embodiment of the present application. The facial image editing apparatus, device and storage medium provided in the above embodiments can execute the facial image editing method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the facial image editing method provided in any embodiment of the present application.

[0122] Based on the above embodiments, the embodiments of the present application also provide a computer program product. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, including a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the facial image editing method provided in each embodiment of the present application.

Claims

1. A facial image editing method, characterized in that: include: Extracting a local image to be processed from the image to be processed; generating an image residual corresponding to a preset image effect according to the partial image to be processed; Generate a target local image according to the image residual and the local image to be processed; The target partial image is added to the image to be processed to obtain a target image.

2. The facial image editing method according to claim 1, characterized in that: The step of extracting the local image to be processed from the image to be processed comprises: Determine the key points of the face in the image to be processed, and extract the local image to be processed from the image to be processed according to the key points of the face.

3. The facial image editing method according to claim 2, characterized in that: The step of extracting a partial image to be processed from the image to be processed according to the facial key points includes: Extracting a first partial image and a second partial image from the image to be processed according to the facial key points; The first partial image and the second partial image are combined into a partial image to be processed.

4. The facial image editing method according to claim 3, characterized in that: The step of synthesizing the first partial image and the second partial image into a partial image to be processed includes: Performing affine transformation alignment processing on the first partial image and the second partial image; The first local image and the second local image after affine transformation alignment processing are synthesized into a local image to be processed.

5. The facial image editing method according to claim 1, characterized in that: The step of adding the target partial image to the image to be processed to obtain the target image comprises: In a case where the partial image to be processed is obtained based on the first partial image and the second partial image, obtaining a first target partial image and a second target partial image according to the target partial image; The first target partial image and the second target partial image are added to the image to be processed to obtain a target image.

6. The facial image editing method according to claim 5, characterized in that: The step of adding the first target partial image and the second target partial image to the image to be processed to obtain a target image includes: Performing inverse affine transformation alignment processing on the first target partial image and the second target partial image; The first target partial image and the second target partial image which have been processed by inverse affine transformation alignment are added to the image to be processed to obtain a target image.

7. The facial image editing method according to claim 1, characterized in that: The step of generating an image residual corresponding to a preset image effect according to the partial image to be processed includes: The local image to be processed is input into the trained image processing model, and the image processing model generates an image residual corresponding to a preset image effect according to the local image to be processed. The objective function of training the image processing model is determined according to the generative adversarial network loss function, the absolute value error loss function and the perceptual loss function.

8. The facial image editing method according to claim 7, characterized in that: The training process of the image processing model includes: Inputting the sample image into the image processing model, and generating a sample residual corresponding to a preset image effect according to the sample image through the image processing model; A generated image is obtained according to the sample residual and the sample image, and parameters of the image processing model are optimized based on the objective function and label images corresponding to the generated image and the sample image.

9. The facial image editing method according to claim 1, characterized in that: The step of generating a target local image according to the image residual and the local image to be processed comprises: The product of the preset effect weight and the image residual is summed with the local image to be processed, and the summation result is determined as the target local image.

10. A facial image editing device, characterized in that: It includes an image extraction module, a residual generation module, a local processing module and an image generation module, wherein: The image extraction module is configured to extract a local image to be processed from the image to be processed; The residual generation module is configured to generate an image residual corresponding to a preset image effect according to the partial image to be processed; The local processing module is configured to generate a target local image according to the image residual and the local image to be processed; The image generation module is configured to add the target partial image to the image to be processed to obtain a target image.

11. A facial image editing device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the facial image editing method as described in any one of claims 1-9.

12. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to execute the facial image editing method as described in any one of claims 1-9 when executed by a computer processor.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the facial image editing method according to any one of claims 1 to 9 is implemented.

Images