Beauty method, program product, storage medium and electronic device

Abstract

Embodiments of the present application provide a beautifying method, a program product, a storage medium and an electronic device. The beautifying method comprises: obtaining a to-be-beautified image containing a face; performing edge-preserving smoothing filtering on the to-be-beautified image to obtain a base image; inputting the base image into a generator in a generative adversarial network to obtain a skin texture image; and performing image fusion on the base image and the skin texture image to obtain a beautified image. The beautified image obtained by the method has both beautifying effect and skin texture.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a beautification method, program product, storage medium, and electronic device. Background Technology

[0002] In recent years, the beautification functions of electronic devices have become increasingly popular among users. Beautification refers to methods that use algorithms (such as skin smoothing algorithms) to enhance facial images. However, existing beautification methods often remove the skin texture along with blemishes, acne, and noise when removing them, resulting in an overly smooth face in the final image that loses the original texture of the skin. Summary of the Invention

[0003] The purpose of this application is to provide a beautification method, program product, storage medium, and electronic device to improve the above-mentioned technical problems.

[0004] To achieve the above objectives, this application provides the following technical solution:

[0005] In a first aspect, embodiments of this application provide a beautification method, comprising: acquiring an image to be beautified containing a human face; performing edge-preserving smoothing filtering on the image to be beautified to obtain a base image; inputting the base image into a generator in a generative adversarial network to obtain a skin texture image; and performing image fusion on the base image and the skin texture image to obtain a beautified image.

[0006] In the above method, the base image can be understood as the initial beautification image, which effectively removes defects such as dark spots, pimples, and noise on the face while preserving the basic contours of the face. However, it also loses skin texture. This method further generates a skin texture image based on the base image and obtains the beautification image by fusing the base image and the skin texture image. This is equivalent to enhancing the skin texture in the base image, so that the final beautification image has both beautification effect and skin texture, and its visual effect is significantly improved.

[0007] In one implementation of the first aspect, the method further includes: performing skin segmentation on the image to be beautified to obtain a skin mask; wherein the pixel value of each pixel in the skin mask represents the probability that the corresponding pixel in the image to be beautified belongs to the skin; the step of image fusion of the base image and the skin texture image to obtain a beautified image includes: performing image fusion on the image to be beautified, the base image, and the skin texture image to obtain the beautified image; wherein the fusion coefficients corresponding to the base image and the skin texture image both include the pixel values ​​in the skin mask, and the fusion coefficient corresponding to the base image is greater than the fusion coefficient corresponding to the skin texture image, and the fusion coefficient corresponding to the image to be beautified includes the complementary values ​​of the pixel values ​​in the skin mask.

[0008] In the above implementation, image fusion is guided by calculating a skin mask, so that different fusion methods can be presented in the skin area and non-skin area of ​​the face (for example, the skin area is more fused with the base image and skin texture image, and the non-skin area is more fused with the image to be beautified), which is beneficial to improving the beautification effect.

[0009] In one implementation of the first aspect, the step of performing edge-preserving smoothing filtering on the image to be beautified to obtain a base image includes: decomposing the image to be beautified into multi-frame frequency band images; wherein the multi-frame frequency band images correspond to image information within multiple frequency bands of the image to be beautified, and the frequency band image corresponding to the lowest frequency band among the multiple frequency bands is the base image, and the method of decomposing and generating the base image is edge-preserving smoothing filtering; the step of performing image fusion on the base image and the skin texture image to obtain a beautified image includes: performing image fusion on the multi-frame frequency band images and the skin texture image to obtain the beautified image.

[0010] In the above implementation, in addition to the base image and skin texture image, at least one frame of frequency band image decomposed from the image to be beautified will be used to fuse and generate the beautified image. Since the frequency band image contains the details of the image to be beautified in the corresponding frequency band, this is beneficial to enrich the details in the beautified image and make the beautification effect more realistic.

[0011] In one implementation of the first aspect, the step of decomposing and generating multiple frequency band images from the image to be beautified includes: performing edge-preserving smoothing filtering on the image to be beautified using at least one set of smoothing parameters to obtain at least one corresponding intermediate image; wherein each set of smoothing parameters represents a corresponding smoothing degree, and the intermediate image with the highest smoothing degree is the base image; subtracting the intermediate image with the lowest smoothing degree from the image to be beautified to obtain a frequency band image corresponding to the highest frequency band among the multiple frequency bands; if the intermediate images are not less than two frames, then subtracting adjacent intermediate images in ascending order of smoothing degree to obtain frequency band images corresponding to the remaining frequency bands among the multiple frequency bands, excluding the highest and lowest frequency bands.

[0012] In the above implementation method, by using at least one set of smoothing parameters to perform edge-preserving smoothing filtering on the image to be beautified, the required multi-frame frequency band image can be quickly decomposed, and the processing method is simple and efficient.

[0013] In one implementation of the first aspect, the method further includes: performing skin segmentation on the image to be beautified to obtain a skin mask; wherein the pixel value of each pixel in the skin mask represents the probability that the corresponding pixel in the image to be beautified belongs to the skin; the step of image fusion of the multi-frame frequency band image and the skin texture image to obtain a beautified image includes: performing image fusion on the image to be beautified, the multi-frame frequency band image, and the skin texture image to obtain the beautified image; wherein the fusion coefficients corresponding to the multi-frame frequency band image and the skin texture image all include the pixel values ​​in the skin mask, and at least one of the fusion coefficients corresponding to the multi-frame frequency band image and the skin texture image includes the product of the pixel values ​​in the skin mask and an adjustable coefficient, and the fusion coefficient corresponding to the base image in the multi-frame frequency band image is greater than the fusion coefficient corresponding to the skin texture image, and the fusion coefficient corresponding to the image to be beautified includes the complementary value of the pixel values ​​in the skin mask.

[0014] In the above implementation, by adding adjustable coefficients that can be adjusted by the user to at least some of the fusion coefficients, it is beneficial to flexibly adjust the proportion of different images in the fusion result, thereby achieving different beautification effects according to user needs.

[0015] In one implementation of the first aspect, the step of inputting the base image into a generator in a generative adversarial network to obtain a skin texture image includes: determining the texture type of the skin in the image to be beautified; and inputting the texture type and the base image into the generator to obtain the skin texture image.

[0016] In the above implementation, the texture type of the skin extracted from the image to be beautified is used to guide the generation of the skin texture image, which helps the generated skin texture image to better match the real skin texture in the image to be beautified, rather than just skin texture with good visual effect.

[0017] In one implementation of the first aspect, the step of inputting the base image into a generator in a generative adversarial network to obtain a skin texture image includes: determining multiple skin regions based on the image to be beautified; and for each skin region, inputting a portion of the base image within that skin region into the generator corresponding to that skin region to obtain a portion of the skin texture image within that skin region.

[0018] In the above implementation, different generators can be used to generate different skin textures for different skin areas, making the skin texture image more detailed, which helps to improve the realism of the skin and enhance the beautification effect.

[0019] Secondly, embodiments of this application provide a beautification device, comprising: an image acquisition module for acquiring an image to be beautified containing a human face; an image filtering module for performing edge-preserving smoothing filtering on the image to be beautified to obtain a base image; an image generation module for inputting the base image into a generator in a generative adversarial network to obtain a skin texture image; and an image fusion module for performing image fusion on the base image and the skin texture image to obtain a beautified image.

[0020] Thirdly, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0022] Fifthly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory stores computer program instructions, and the computer program instructions are read and executed by the processor to perform the method provided in the first aspect or any possible implementation of the first aspect. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a beautification method provided in this application embodiment;

[0025] Figure 2 A schematic diagram illustrating the skin segmentation steps;

[0026] Figure 3 This is a schematic diagram of applying a bilateral filter with radius R1 to the image to be beautified.

[0027] Figure 4 This is a schematic diagram of applying a bilateral filter with a radius of R2 to the image to be beautified;

[0028] Figure 5 A module diagram of a beautification device provided in an embodiment of this application;

[0029] Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security and prevention, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement. The beautification method in this application embodiment also utilizes relevant artificial intelligence technologies.

[0031] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0032] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0033] Figure 1 A flowchart illustrating a beautification method provided in an embodiment of this application. This method may, but is not limited to, [the method described herein]. Figure 6The electronic device in the process performs the operation; for possible structures of this electronic device, please refer to the following section. Figure 6 The explanation. (Refer to...) Figure 1 Beauty enhancement methods include:

[0034] Step S110: Obtain the image to be beautified containing a human face.

[0035] Step S120: Perform edge-preserving smoothing filtering on the image to be beautified to obtain the base image.

[0036] Step S130: Input the base image into the generator in the generative adversarial network to obtain the skin texture image.

[0037] Step S140: Perform image fusion on the base image and the skin texture image to obtain the beautified image.

[0038] Below, we will first introduce how to implement the above steps S110 to S140 based on a skin mask, and then introduce the method of implementation without a skin mask.

[0039] The image to be beautified in step S110 can contain a complete face or only a portion of a face. There are various ways to acquire the image to be beautified: for example, if the electronic device includes a camera, it can capture the image in real time; or, for example, the electronic device can receive the image to be beautified from another electronic device and store it, then read the image from the storage medium when executing step S110, and so on. The image to be beautified can be either a grayscale image or a color image.

[0040] After obtaining the image to be beautified, skin segmentation can be performed to obtain a skin mask. The size of the skin mask is the same as that of the image to be beautified. The pixel value of each pixel in the skin mask represents the probability that the corresponding pixel (pixel at the same position) in the image to be beautified belongs to the skin. For example, the pixel value in the skin mask can be a real number in [0,1] or an integer in [0,255]. Taking the former as an example, the closer the pixel value of a pixel in the skin mask is to 0, the lower the probability that the corresponding pixel in the image to be beautified belongs to the skin; the closer the pixel value of a pixel in the skin mask is to 1, the higher the probability that the corresponding pixel in the image to be beautified belongs to the skin.

[0041] Specifically, the skin mask can be a binary mask. For example, the pixel value of each pixel in the skin mask can only be 0 or 1. A value of 0 indicates that the corresponding pixel in the image to be beautified does not belong to the skin, and a value of 1 indicates that the corresponding pixel in the image to be beautified belongs to the skin. Optionally, directly performing skin segmentation on the image to be beautified may generate a non-binary mask. Then, performing a binarization operation on the non-binary mask will yield the corresponding binary mask.

[0042] Skin segmentation is a type of image segmentation, with the skin as the segmentation target. Therefore, skin segmentation can be achieved using various image segmentation algorithms, including, but not limited to, neural network-based methods such as UNet, FCN, and DeepLab, or traditional algorithms.

[0043] Figure 2 This is a schematic diagram of skin segmentation. (Refer to...) Figure 2 The image on the left, denoted as S, is the image to be beautified. For privacy reasons, the eyes are blurred. The image on the right, denoted as Mask, is a non-binary mask. Black or dark gray portions (pixel values ​​equal to or close to 0) in the mask indicate that pixels in that portion of the image are likely not skin, while white or light gray portions (pixel values ​​equal to or close to 1) indicate that pixels in that portion of the image are likely skin. Observation. Figure 2 It can be seen that the black or dark gray parts are mainly concentrated around the eyes, while the white or light gray parts are mainly located on the cheeks, nose, and forehead. In other words, the pixel values ​​in the skin mask are basically consistent with the actual distribution of human facial skin.

[0044] Note that although the skin mask can be calculated after step S110, the skin mask calculation does not necessarily have to be performed immediately after step S110. For example, in some implementations, the skin mask is not used in steps S120 and S130, but only in step S140. In this case, the skin mask calculation only needs to be completed before step S140 is executed.

[0045] The edge-preserving smoothing filter in step S120 refers to a filter that can both preserve the edges in the image and smooth the rest of the image. Examples include bilateral filtering and guided filtering. The smoothness (or the degree to which edges are preserved) of the edge-preserving smoothing filter depends on the smoothing parameters of the filter. For example, for bilateral filtering, this parameter can be the radius of the filter. The edges that are to be preserved in step S120 may include some major contours of the face, such as the nose contour. Preserving these contours should not cause significant distortion in the beautification effect. By appropriately setting the smoothing parameters of the filter, the purpose of preserving these contours can be achieved.

[0046] Edge-preserving smoothing filtering can be applied to the entire image to be beautified, or it can be applied to only a portion of the image. For now, we can understand it as applying it to the entire image, and the case of applying it to a portion of the image will be explained later.

[0047] Although edge-preserving smoothing filtering retains some edges, it is essentially still smoothing filtering. Therefore, minor skin defects such as dark spots, acne, and noise in the image to be beautified are removed (or at least weakened) during the filtering process, resulting in a skin-smoothing effect in step S120. The base image can also be considered the initial beautified image. The problem with the initial beautified image is that skin texture (e.g., fine skin lines, pores, etc.) is also removed (or at least weakened) along with the skin defects, making the skin appear too smooth and lacking in texture. This is precisely what needs to be remedied by subsequent steps. For ease of explanation later, the base image will be denoted as A1.

[0048] The generative adversarial network (GAN) in step S130 consists of a generator and a discriminator. The GAN is trained before step S130 is executed, and only the generator can be used in step S130. After the base image (which may also include other information) is input into the generator, the generator outputs a skin texture image with the same size as the base image. This image contains the skin texture predicted by the generator, and the skin texture image is denoted as G.

[0049] Optionally, the generative adversarial network used in step S130 can be obtained, but is not limited to, in the following ways: First, a batch of high-resolution images with good skin texture quality but no (or few) skin defects are collected and labeled with "1" as a labeled "real" dataset; Second, a batch of base images obtained through steps S110 to S120 of this application (these images are not necessarily related to the images collected in the previous step) are obtained and fed into the generator to generate a "fake" dataset during the training process; Finally, the discriminator and the generator are trained alternately until the model converges.

[0050] As mentioned earlier, since skin defects have been removed from the base image, resulting in relatively "clean" skin, the skin texture image generated in step S130 based on the base image has good quality and is less prone to imperfections.

[0051] In step S140, the base image and the skin texture image are the images that must be fused at least. Other images can also be fused. The beautified image is the beautified result of the image to be beautified, denoted as Result.

[0052] The implementation method is as follows: The image to be beautified, the base image, and the skin texture image are fused using a skin mask to obtain the beautified image. The fusion method can be weighted fusion, where the fusion coefficients (weights) for the base image and the skin texture image both include pixel values ​​from the skin mask, while the fusion coefficients (weights) for the image to be beautified include complementary values ​​from the pixel values ​​in the skin mask. Furthermore, the fusion coefficient for the base image is greater than that for the skin texture image. Of course, other fusion methods are also possible.

[0053] In this skin mask, the sum of a pixel's pixel value and its complementary value is a fixed value. This fixed value can be the upper limit of the pixel value; for example, if the range of pixel values ​​is [0,1], then the upper limit is 1. Based on the conversion relationship between complementary values ​​and pixel values, the complementary value of each pixel in the skin mask represents the probability that the corresponding pixel in the image to be beautified does not belong to the skin.

[0054] The image fusion process can be represented by the following formula:

[0055] Result = S*(1-Mask)+A1*Mask+(G-A1)*ratio*Mask (1)

[0056] Formula (1) is actually a pixel-by-pixel formula. That is to say, the pixel value of each pixel in Result is calculated by substituting the pixel values ​​of the corresponding pixels in S, A1, G, and Mask into Formula (1).

[0057] In formula (1), the fusion coefficient corresponding to S is 1-Mask, which is the complementary value of the pixel value in Mask, while the fusion coefficient corresponding to A1 is Mask, which is the pixel value in Mask. The fusion coefficient corresponding to G (or G-A1) is ratio*Mask, where ratio can be an adjustable coefficient with a value in the range (0,1) to control the extent to which G-A1 participates in the fusion, or the degree of skin texture in Result. At the same time, adding ratio makes the fusion coefficient corresponding to G less than the fusion coefficient corresponding to A1.

[0058] In formula (1), G is not directly weighted and fused, but is subtracted from A1 before participating in the weighted fusion. This is because G generated in step S130 contains skin components rather than just skin texture. Therefore, G-A1 must be calculated first to obtain the "pure" skin texture before fusion. Otherwise, since A1 also contains skin components, the skin components will be fused twice. Of course, if the generator trained in a certain way can directly generate the "pure" skin texture, then G-A1 in formula (1) can also be replaced with G.

[0059] If the Mask is a binary mask, the meaning of formula (1) is easy to understand. No beautification is performed in non-skin areas (e.g., areas composed of pixels with a value of 0 in the Mask), and the pixel values ​​in S are directly used. Beautification is performed in skin areas (e.g., areas composed of pixels with a value of 1 in the Mask), fusing the pixel values ​​in A1 and G-A1. Furthermore, when fusing in skin areas, since Mask > ratio * Mask, A1 takes precedence, while G-A1 takes a secondary role. This is because beautification is a fundamental requirement, and adding skin texture only further improves the beautification effect.

[0060] If the Mask is a non-binary mask, it can be understood in the same way as a binary mask: in areas that are likely not skin (e.g., areas composed of pixels with values ​​close to 0 in the Mask), the pixel values ​​in S are mainly used; in areas that are likely skin (e.g., areas composed of pixels with values ​​close to 1 in the Mask), the pixel values ​​in A1 and G-A1 are mainly fused, with more fusion of pixel values ​​in A1 than in G-A1.

[0061] Regardless of whether the Mask is a binary mask, Formula (1) expresses the following meaning: when beautifying the skin of a human face, in addition to the base image A1, the skin texture G-A1 should also be added to make up for the problem that A1 is too smooth.

[0062] Furthermore, in some implementations, step S140 may not use the image to be beautified for image fusion, but only the base image and the skin texture image for image fusion. For example, when performing edge-preserving smoothing filtering in step S120, a binarized skin mask can be used to filter only the skin region. In this way, the pixel values ​​in the non-skin region of the image to be beautified are naturally preserved in the base image. According to formula (1), when using a binarized skin mask, the pixel values ​​in the skin region of the image to be beautified do not participate in image fusion, so there is no need for the image to be beautified to participate in image fusion.

[0063] Furthermore, in some implementations, a skin mask may not be calculated or used for image fusion. For example, in step S120, for a pixel in the image to be beautified, if the pixel values ​​of most of its neighboring pixels are close to the skin color, then edge-preserving smoothing filtering is applied; if the pixel values ​​of most of its neighboring pixels are not close to the skin color, then no processing is performed. Although the above operation does not explicitly output a binarized skin mask, it still achieves the effect of filtering in the skin region and not filtering in the non-skin region, so that the pixel values ​​in the non-skin region of the image to be beautified are preserved in the base image. In step S130, the generative adversarial network can be trained to automatically avoid the non-skin region in the base image, generating skin texture only in the skin region, and outputting 0 or very small values ​​in the non-skin region as much as possible. Thus, in step S140, a skin mask is not necessary, and the base image and the skin texture image can be directly fused.

[0064] Alternatively, if the image to be beautified is only a part of a face, containing only skin areas and no non-skin areas (for example, the image to be beautified is only a cheek image), then there is no need to calculate the skin mask.

[0065] Brief summary Figure 1 The method described in this paper uses a base image, which can be understood as an initial beautification image. While preserving the basic contours of the face, it effectively removes defects such as dark spots, acne, and noise on the face, but also loses skin texture. However, this method further generates a skin texture image based on the base image and fuses the base image and the skin texture image to obtain the beautification image. This is equivalent to re-enhancing the skin texture in the base image, so that the final beautification image has both beautification effect and skin texture, and its visual effect is significantly improved.

[0066] Furthermore, in some implementations of this method, a skin mask can be computed to guide image fusion, so that different fusion methods can be presented in the skin area and non-skin area of ​​the face (e.g., the skin area is more fused with the base image and skin texture image, and the non-skin area is more fused with the image to be beautified), thereby improving the beautification effect.

[0067] Based on the above embodiments, step S120 can optionally be implemented in the following manner:

[0068] Multi-frame frequency band images are generated from the image to be beautified. Each multi-frame frequency band image corresponds to image information within multiple frequency bands of the image to be beautified. For example, three multi-frame frequency band images can be generated from the image to be beautified, corresponding to the high, mid, and low frequency bands of the image, respectively denoted as B2 (low frequency), D2 (mid frequency), and D1 (high frequency). The multiple frequency bands corresponding to the multi-frame frequency band images may or may not overlap (the latter case will be mainly discussed). Furthermore, these frequency bands may or may not cover the entire spectrum of the image to be beautified (the latter case will be mainly discussed).

[0069] In these multi-frame frequency band images, the frequency band image corresponding to the lowest frequency band among the multiple frequency bands is the base image (e.g., B2), and the method used to generate the base image is edge-preserving smoothing filtering. The frequency band images corresponding to other frequency bands among the multiple frequency bands (e.g., D2, D1) may or may not be generated through edge-preserving smoothing filtering. It should be noted that the base image only corresponds to the image information within the lowest frequency band of the image to be beautified. However, because edge-preserving smoothing filtering retains certain edges, it cannot be assumed that the base image is entirely composed of low-frequency image information; it can only be said that it mainly contains low-frequency image information.

[0070] In one implementation, multi-frame frequency band images can be easily and quickly decomposed through the following steps:

[0071] Step A: Use at least one set of smoothing parameters to perform edge-preserving smoothing filtering on the images to be beautified, and obtain at least one corresponding intermediate image.

[0072] Each set of smoothing parameters represents a corresponding degree of smoothing. If there are multiple sets of smoothing parameters, the values ​​of each set of smoothing parameters are different, and they correspond to different degrees of smoothing. The greater the degree of smoothing, the lower the frequency components are retained after filtering. Therefore, the intermediate image with the greatest degree of smoothing is the base image (the frequency band image corresponding to the lowest frequency band of the image to be beautified).

[0073] For example, if the edge-preserving smoothing filter is a bilateral filter, the smoothing parameter can be the radius of the filter. Here, we assume there are two radii, R1 and R2, where R1 < R2. According to the principle of bilateral filtering, R2 represents a greater degree of smoothness than R1. Therefore, in step A, we can perform bilateral filtering with radius R1 and bilateral filtering with radius R2 on the image to be beautified, respectively, to obtain intermediate images B1 and B2, as shown below. Figure 3 , Figure 4 As shown. Comparison Figure 3 and Figure 4 It's easy to see that B2's image is smoother than B1's, meaning it contains fewer frequency components.

[0074] Step B: Subtract the intermediate image with the least smoothness from the image to be beautified to obtain the frequency band image corresponding to the highest frequency band among multiple frequency bands.

[0075] For example, if the intermediate image with the least smoothness is B1, and the frequency band image corresponding to the highest frequency band among multiple frequency bands is D1, then D1 = S - B1.

[0076] Step C: If there are at least two intermediate images, subtract the intermediate images of two adjacent frames in order of increasing smoothness to obtain the frequency band images corresponding to the other frequency bands in the multiple frequency bands, excluding the highest and lowest frequency bands.

[0077] Step C is an optional step. If there is only one frame in the intermediate image, step C is not executed, and step B can be executed instead. For example, since B1 and B2 are both intermediate images, the execution condition of step C is met. The frequency band image corresponding to the remaining frequency bands in the multiple frequency bands, excluding the highest and lowest frequency bands, is D2. Then, D2 = B1 - B2.

[0078] The following is a simple explanation of the principle of steps A to C in conjunction with the calculation process of frequency band images B2, D2, and D1: For the high, mid, and low frequency bands of the image to be beautified, B2 corresponds to the image information in the low frequency band. Since B1 has a lower smoothness than B2 and is also generated by edge-preserving smoothing filtering (generally a low-pass filtering), B1 corresponds to the low frequency band plus the image information in the mid frequency band. Thus, S-B1 gives the image information in the high frequency band, and B1-B2 gives the image information in the mid frequency band.

[0079] Steps A through C are not the only way to obtain multi-frame frequency band images. For example, after obtaining B2 through edge-preserving smoothing filtering, a high-pass filter can be designed to decompose D1 from the image to be beautified, and then S-B2-D1 can be used to obtain D2, and so on.

[0080] In step S140, image fusion can be performed on multi-frame frequency band images and skin texture images to obtain a beautified image.

[0081] For example, considering both the image to be beautified and the skin mask, image fusion can be performed using the following formula:

[0082] Result=S*(1-Mask)+B2*Mask+D2*ratio2*Mask

[0083] +D1*ratio1*Mask+(G-B2)*ratio3*Mask(2)

[0084] The first, second, and fifth terms on the right side of formula (2) are similar to those in formula (1), except that the base image has changed from A1 to B2. The third and fourth terms on the right side are newly added, which also incorporate the mid- and high-frequency band images D2 and D1.

[0085] In formula (2), ratio1, ratio2, and ratio3 can be three adjustable coefficients with values ​​in the interval (0,1). Their values ​​can be fixed or adjusted according to user needs. If all three are adjustable, then the fusion coefficients ratio1*Mask, ratio2*Mask, and ratio3*Mask corresponding to D1, D2, and G-B2 are also adjustable. This allows for flexible control over the extent to which D1, D2, and G-B2 participate in the fusion process to achieve different beautification effects. It should be understood that in different implementation methods, the fusion coefficients corresponding to D1, D2, and G-B2 may contain only one or two adjustable coefficients, or none of them may contain adjustable coefficients, or the fusion coefficients corresponding to S or B2 may also contain adjustable coefficients.

[0086] Furthermore, adding ratio1, ratio2, and ratio3 ensures that the fusion coefficients ratio1*Mask, ratio2*Mask, and ratio3*Mask corresponding to D1, D2, and G-B2 are all less than the fusion coefficient Mask corresponding to B2. This means that when fusing in skin areas or areas that are likely skin, B2 should be the primary factor, with D1, D2, and G-B2 playing only auxiliary roles. It is essential that ratio3*Mask < Mask (i.e., ratio3 < 1). Otherwise, if ratio3 = 1, then B2*Mask + (G-B2)*Mask = G*Mask, causing the base image B2 to be canceled out, thus failing to achieve the beautification effect.

[0087] Furthermore, similar to the previous analysis, when performing image fusion based on multi-frame frequency band images and skin texture images, the image to be beautified may not be included, or a skin mask may not be calculated or used, which will not be elaborated again.

[0088] According to formula (3), in the above implementation, in addition to the base image B2 and the skin texture image G, at least one frame of frequency band image (D2, D1) decomposed from the image to be beautified will be used to fuse and generate the beautified image. Since these frequency band images contain the details of the image to be beautified in the corresponding frequency band (D2 contains mid-frequency details, and D1 contains high-frequency details), this is beneficial to enrich the details in the beautified image and make the beautification effect more realistic.

[0089] It is important to note that skin texture can also be considered as skin details, so D2 and D1 actually contain the skin texture in S. However, fusing G-B2 is also essential. The reason is that, taking formula (3) as an example, in addition to skin texture, D2 and D1 also contain skin defects. Therefore, if too much content from D2 and D1 is fused during image fusion, it is equivalent to adding back the skin defects that have already been removed from B2. Therefore, a more reasonable approach is to lightly fuse some content from D2 and D1 by adjusting ratio2 and ratio1 (for example, setting them to smaller values ​​such as 0.1 and 0.2), so that B2 can retain a good smoothing effect while increasing details, while the skin texture is mainly provided by G-B2.

[0090] In step S130, although the generator in the generative adversarial network can generate skin texture images, the skin texture in these images may not be the actual skin texture in the image to be beautified. It may simply be a skin texture that the generator deems visually appealing. For example, suppose the skin texture on a human face is divided into horizontal and vertical lines. The face in the image to be beautified has horizontal lines, but the skin texture image generated by the generator may have vertical lines. Thus, the skin texture in the final beautified image will also be vertical lines. Although it may look visually good, it alters the original skin texture in the image to be beautified, which may not meet the user's expectations.

[0091] Therefore, based on the above embodiments, step S130 can optionally be implemented in the following manner:

[0092] First, determine the skin texture type in the image to be beautified. For example, horizontal and vertical lines in the example above are two skin texture types. However, it should be understood that horizontal and vertical lines are just examples and do not represent the actual skin texture type. Determining the skin texture type can be done using traditional algorithms or deep learning algorithms. For example, the image to be beautified can be input into a pre-trained neural network, which can then predict the skin texture type in the image.

[0093] Then, the base image and the skin texture type are input into the generator to obtain a skin texture image. The skin texture type serves as a generation guide, directing the generator to produce the skin texture image according to this pattern. For example, if the skin texture type is horizontal stripes, then a skin texture image containing horizontal stripes will be generated. In this way, the final beautified image not only has good skin texture effects but also matches the real skin texture in the image to be beautified, resulting in a beautification effect that better meets user expectations.

[0094] For different skin regions in the image to be beautified, different skin textures may need to be generated in the skin texture image. Therefore, in order to obtain a more refined and realistic beautification effect, step S130 can also be implemented in the following way:

[0095] First, multiple skin regions are determined based on the image to be beautified. The rules for determining these skin regions are not limited: for example, they can be divided according to skin location, such as including at least two regions from the cheek, forehead, nose, and chin areas; or they can be divided according to skin texture type, with horizontal lines forming one skin region and vertical lines forming another, and so on. For these two examples, methods such as facial landmark detection and skin texture type detection can be used to determine the skin regions, respectively.

[0096] Then, for each skin region, a portion of the base image within that skin region is input into the generator corresponding to that skin region to obtain a portion of the skin texture image within that skin region. Note that although the skin region is determined based on the image to be beautified, the region itself can be considered not to be bound to the image to be beautified, but only to refer to a set of pixel locations.

[0097] In this implementation, a corresponding Generative Adversarial Network (GAN) is set up for each skin region (although it's possible for several skin regions with common characteristics to correspond to the same GAN), specifically designed to generate skin textures suitable for that skin region. For example, a GAN is set up for the cheek region, a GAN for the forehead region, and so on. Therefore, by inputting a portion of the base image within a certain skin region into the generator of the corresponding GAN for that skin region, a portion of the skin texture image within that skin region can be generated. And by obtaining the portions of the skin texture image within each skin region, the entire skin texture image is naturally obtained.

[0098] Optionally, when generating skin textures based on a portion of the base image within the corresponding skin region, each generator can also combine the skin texture type extracted from the portion of the base image within the corresponding skin region to obtain more accurate generation results.

[0099] Figure 5 This is a block diagram of the beautification device 200 provided in an embodiment of this application. (Refer to...) Figure 5 The beauty device 200 includes:

[0100] Image acquisition module 210 is used to acquire an image to be beautified containing a human face;

[0101] Image filtering module 220 is used to perform edge-preserving smoothing filtering on the image to be beautified to obtain a base image;

[0102] Image generation module 230 is used to input the base image into the generator in the generative adversarial network to obtain a skin texture image;

[0103] The image fusion module 240 is used to perform image fusion on the base image and the skin texture image to obtain a beautified image.

[0104] In one implementation of the beautification device 200, the device further includes: an image segmentation module for performing skin segmentation on the image to be beautified to obtain a skin mask; wherein, the pixel value of each pixel in the skin mask represents the probability that the corresponding pixel in the image to be beautified belongs to the skin; and an image fusion module 240 for performing image fusion on the base image and the skin texture image to obtain a beautified image, including: performing image fusion on the image to be beautified, the base image, and the skin texture image to obtain the beautified image; wherein, the fusion coefficients corresponding to the base image and the skin texture image both include the pixel values ​​in the skin mask, and the fusion coefficient corresponding to the base image is greater than the fusion coefficient corresponding to the skin texture image, and the fusion coefficient corresponding to the image to be beautified includes the complementary values ​​of the pixel values ​​in the skin mask.

[0105] In one implementation of the beautification device 200, the image filtering module 220 performs edge-preserving smoothing filtering on the image to be beautified to obtain a base image, including: decomposing and generating multi-frame frequency band images from the image to be beautified; wherein, the multi-frame frequency band images correspond to image information within multiple frequency bands of the image to be beautified, and the frequency band image corresponding to the lowest frequency band among the multiple frequency bands is the base image, and the method of decomposing and generating the base image is edge-preserving smoothing filtering; the image fusion module 240 performs image fusion on the base image and the skin texture image to obtain a beautified image, including: performing image fusion on the multi-frame frequency band images and the skin texture image to obtain the beautified image.

[0106] In one implementation of the beautification device 200, the image filtering module 220 decomposes the image to be beautified into multiple frames of frequency band images, including: performing edge-preserving smoothing filtering on the image to be beautified using at least one set of smoothing parameters to obtain at least one corresponding intermediate image; wherein, each set of smoothing parameters represents a corresponding smoothing degree, and the intermediate image with the highest smoothing degree is the base image; subtracting the intermediate image with the lowest smoothing degree from the image to be beautified to obtain a frequency band image corresponding to the highest frequency band among the multiple frequency bands; if there are at least two intermediate images, then subtracting adjacent intermediate images in ascending order of smoothing degree to obtain frequency band images corresponding to the remaining frequency bands among the multiple frequency bands, excluding the highest and lowest frequency bands.

[0107] In one implementation of the beautification device 200, the device further includes: an image segmentation module for performing skin segmentation on the image to be beautified to obtain a skin mask; wherein, the pixel value of each pixel in the skin mask represents the probability that the corresponding pixel in the image to be beautified belongs to the skin; and an image fusion module 240 for performing image fusion on the multi-frame frequency band image and the skin texture image to obtain a beautified image, including: performing image fusion on the image to be beautified, the multi-frame frequency band image, and the skin texture image to obtain the beautified image; wherein, the fusion coefficients corresponding to the multi-frame frequency band image and the skin texture image all include the pixel values ​​in the skin mask, and at least one of the fusion coefficients corresponding to the multi-frame frequency band image and the skin texture image includes the product of the pixel values ​​in the skin mask and an adjustable coefficient, and the fusion coefficient corresponding to the base image in the multi-frame frequency band image is greater than the fusion coefficient corresponding to the skin texture image, and the fusion coefficient corresponding to the image to be beautified includes the complementary value of the pixel values ​​in the skin mask.

[0108] In one implementation of the beautification device 200, the image generation module 230 inputs the base image into a generator in a generative adversarial network to obtain a skin texture image, including: determining the texture type of the skin in the image to be beautified; and inputting the texture type and the base image into the generator to obtain the skin texture image.

[0109] In one implementation of the beautification device 200, the image generation module 230 inputs the base image into a generator in a generative adversarial network to obtain a skin texture image, including: determining multiple skin regions based on the image to be beautified; for each skin region, inputting a portion of the base image within that skin region into the generator corresponding to that skin region to obtain a portion of the skin texture image within that skin region.

[0110] The beautification device 200 provided in this application embodiment can be used to execute the beautification method provided in this application embodiment. The implementation principle and technical effects of the device have been described in the foregoing method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the method embodiment.

[0111] Figure 6 The structure of the electronic device 300 provided in an embodiment of this application is shown. (Refer to...) Figure 6 The electronic device 300 includes a processor 310, a memory 320, and a communication interface 330. These components are interconnected and communicate with each other via a communication bus 340 and / or other forms of connection mechanism (not shown).

[0112] The processor 310 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 310 can be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; it can also be a special-purpose processor, including a Graphics Processing Unit (GPU), a Neural-network Processing Unit (NPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Furthermore, when there are multiple processors 310, some can be general-purpose processors and others can be special-purpose processors.

[0113] The memory 320 includes one or more (only one is shown in the figure), which may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0114] Processor 310 and other possible components can access memory 320, read and / or write data therein. In particular, one or more computer program instructions can be stored in memory 320, and processor 310 can read and execute these computer program instructions to implement the beautification method provided in the embodiments of this application.

[0115] Communication interface 330 includes one or more (only one is shown in the figure) that can be used to communicate directly or indirectly with other devices to exchange data. Communication interface 330 may include interfaces for wired and / or wireless communication.

[0116] Understandable. Figure 6 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 6 The more or fewer components shown, or having the same Figure 6 Different configurations are shown. For example, if the electronic device 300 does not communicate with other devices, it may not include the communication interface 330. As another example, the electronic device 300 may also include a camera for capturing the image to be beautified in step S110.

[0117] Figure 6 The components shown can be implemented using hardware, software, or a combination thereof. Electronic device 300 may be a physical device, such as a mobile phone, camera, tablet computer, wearable device, laptop computer, PC, robot, server, etc., or a virtual device, such as a virtual machine, container, etc. Furthermore, electronic device 300 is not limited to a single device; it can also be a combination of multiple devices or a cluster of a large number of devices.

[0118] This application also provides a computer-readable storage medium storing computer program instructions. These computer program instructions are read and executed by a processor to perform the beautification method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 6 The memory 320 in the electronic device 300.

[0119] This application also provides a computer program product, which includes computer program instructions. These computer program instructions are read and executed by a processor to perform the beautification method provided in this application.

[0120] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A beautification method, characterized in that, include: Obtain the image to be beautified, which contains a human face; The image to be beautified is subjected to edge-preserving smoothing filtering to obtain the base image; The base image is input into the generator in the generative adversarial network to obtain a skin texture image; The base image and the skin texture image are fused to obtain a beautified image; The step of performing edge-preserving smoothing filtering on the image to be beautified to obtain the base image includes: Multi-frame frequency band images are generated from the image to be beautified; wherein, the multi-frame frequency band images correspond to image information in multiple frequency bands of the image to be beautified, and the frequency band image corresponding to the lowest frequency band among the multiple frequency bands is the base image, and the method for generating the base image is edge-preserving smoothing filtering; The step of fusing the base image and the skin texture image to obtain the beautified image includes: The beautified image is obtained by fusing the multi-frame frequency band images and the skin texture image.

2. The beautification method according to claim 1, characterized in that, The method further includes: The image to be beautified is segmented to obtain a skin mask; wherein, the pixel value of each pixel in the skin mask represents the probability that the corresponding pixel in the image to be beautified belongs to the skin; The step of fusing the base image and the skin texture image to obtain the beautified image includes: The beautified image is obtained by image fusion of the image to be beautified, the base image, and the skin texture image; wherein the fusion coefficients corresponding to the base image and the skin texture image both include the pixel values ​​in the skin mask, and the fusion coefficient corresponding to the base image is greater than the fusion coefficient corresponding to the skin texture image, and the fusion coefficient corresponding to the image to be beautified includes the complementary values ​​of the pixel values ​​in the skin mask.

3. The beautification method according to claim 1, characterized in that, The step of decomposing and generating multi-frame frequency band images from the image to be beautified includes: The image to be beautified is subjected to edge-preserving smoothing filtering using at least one set of smoothing parameters to obtain at least one corresponding intermediate image; wherein, each set of smoothing parameters represents a corresponding smoothing degree, and the intermediate image with the highest smoothing degree is the base image; Subtract the intermediate image with the least smoothness from the image to be beautified to obtain a frequency band image corresponding to the highest frequency band among the multiple frequency bands; If there are at least two intermediate images, then the intermediate images of two adjacent frames are subtracted in order of increasing smoothness to obtain the frequency band images corresponding to the remaining frequency bands in the multiple frequency bands, excluding the highest and lowest frequency bands.

4. The beautification method according to claim 1, characterized in that, The method further includes: The image to be beautified is segmented to obtain a skin mask; wherein, the pixel value of each pixel in the skin mask represents the probability that the corresponding pixel in the image to be beautified belongs to the skin; The step of fusing the multi-frame frequency band images and the skin texture image to obtain a beautified image includes: The beautified image is obtained by fusing the image to be beautified, the multi-frame frequency band image, and the skin texture image. Wherein, the fusion coefficients corresponding to the multi-frame frequency band images and the skin texture images both include the pixel values ​​in the skin mask, and at least one of the fusion coefficients corresponding to the multi-frame frequency band images and the skin texture images includes the product of the pixel values ​​in the skin mask and the adjustable coefficient, and the fusion coefficient corresponding to the base image in the multi-frame frequency band images is greater than the fusion coefficient corresponding to the skin texture image, and the fusion coefficient corresponding to the image to be beautified includes the complementary value of the pixel values ​​in the skin mask.

5. The beautification method according to any one of claims 1-4, characterized in that, The step of inputting the base image into the generator in the generative adversarial network to obtain a skin texture image includes: Determine the texture type of the skin in the image to be beautified; The texture type and the base image are input into the generator to obtain the skin texture image.

6. The beautification method according to any one of claims 1-4, characterized in that, The step of inputting the base image into the generator in the generative adversarial network to obtain a skin texture image includes: Multiple skin regions are determined based on the image to be beautified; For each skin region, a portion of the base image within that skin region is input into the generator corresponding to that skin region to obtain a portion of the skin texture image within that skin region.

7. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-6.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores computer program instructions, which are read and executed by the processor to perform the method of any one of claims 1-6.

Images