Beauty processing method, system, storage medium and terminal device

By training a generative adversarial network and combining it with image processing steps, the problems of noise amplification and imprecise style transfer in image enhancement are solved, achieving high-quality image beautification while preserving the details of the original image.

CN112862712BActive Publication Date: 2024-11-29GUANGZHOU FANGTU TECH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202110136729.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-02-01
Publication Date
2024-11-29
Estimated Expiration
2041-02-01

AI Technical Summary

Technical Problem

Existing techniques tend to amplify low-contrast images and noise points during image enhancement, and it is difficult to achieve fine color correction through unpaired inter-domain style transfer.

Method used

By employing a pre-trained generative adversarial network (GAN), the parameters of the GAN are updated by calculating the GAN objective function, distance function, and color loss function, and then the image is processed for blemish removal, skin smoothing, whitening, and filtering, ensuring the natural preservation of image details.

Benefits of technology

It effectively avoids the destruction of the original image during image processing, provides high-quality preservation of image detail features, and generates natural images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112862712B_ABST
    Figure CN112862712B_ABST
Patent Text Reader

Abstract

The present invention discloses a facial beauty processing method, system, storage medium and terminal device, the method comprising the following steps: obtaining an image to be processed; performing acne removal, skin grinding, whitening and filter processing on the image to be processed respectively to generate a target image; wherein a pre-trained generative adversarial network is used to perform filter processing on the image to be processed; the training steps of the generative adversarial network include: obtaining a number of training images as a training set; using the training set to calculate the generative adversarial network objective function, distance function and color loss function in the generative adversarial network; calculating the target parameters according to the calculation results of the generative adversarial network objective function, distance function and color loss function; updating the parameters of the generative adversarial network to the target parameters. The present invention can avoid the situation of destroying the original image, thereby providing a high-quality natural image that retains the detailed features of the original image to a certain extent. The present invention can be widely used in the field of image processing technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a beauty processing method, system, storage medium and terminal device. Background Art

[0002] There are many different technologies involved in beauty, such as algorithms for skin resurfacing, whitening, acne removal, and filters. Among them, there are many ways to implement filters. At present, filter effects include methods based on color adjustment and spatial position transformation. With the rise of deep learning, various style transfer, object deformation, and image enhancement technologies have emerged in filter processing. For example, image enhancement technology based on deep convolutional neural networks uses mobile phone photos as input and high-quality photos taken by SLR cameras as targets. It trains a feature map in an end-to-end manner through a generative adversarial network, and integrates multiple loss functions such as content, texture, color, and details to achieve different image filter purposes, so that mobile phone photos can be generated into pictures with SLR camera quality.

[0003] However, the current generative adversarial methods have obvious shortcomings in their application. For example, for image enhancement technology based on deep convolutional neural networks, for low-contrast images and isolated noise points, the noise points will be amplified after enhancement, thereby destroying the original image; based on cycle consistency technology, for style transfer and conversion between unpaired image domains, images with different styles that deviate greatly from the original image will be generated, making it difficult to perform fine color correction. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides a beauty processing method, system, storage medium and terminal device, which can provide a high-quality natural image that retains the detailed features of the original image to a certain extent.

[0005] A facial beautification method according to a first aspect of the present invention includes the following steps:

[0006] Get the image to be processed;

[0007] Performing acne removal, skin smoothing, whitening and filter processing on the image to be processed to generate a target image;

[0008] Wherein, a pre-trained generative adversarial network is used to perform filter processing on the image to be processed; the training steps of the generative adversarial network include:

[0009] Obtain several training images as a training set;

[0010] Using the training set to calculate a generative adversarial network objective function, a distance function, and a color loss function in the generative adversarial network;

[0011] Calculating target parameters according to calculation results of the generative adversarial network objective function, distance function and color loss function;

[0012] The parameters of the generative adversarial network are updated to the target parameters.

[0013] A beautification processing method according to an embodiment of the present invention has at least the following beneficial effects: this embodiment first uses a training set composed of several training images to calculate the generative adversarial network objective function, distance function and color loss function in the generative adversarial network, then calculates the target parameters according to the calculation results of the generative adversarial network objective function, distance function and color loss function, and updates the parameters of the generative adversarial network to the target parameters, and then performs acne removal, skin refining and whitening processing on the processed image respectively, and uses the generative adversarial network updated to the target parameters to filter the processed image to obtain the target image, thereby avoiding the destruction of the original image, thereby providing a high-quality natural image that retains the detailed features of the original image to a certain extent.

[0014] According to some embodiments of the present invention, the generative adversarial network includes a codec generation network and a discriminant network, and the codec generation network includes a plurality of encoding units and a plurality of decoding units;

[0015] The encoding unit compresses the input image and extracts high-level semantic features;

[0016] The decoding unit upsamples the input image and combines the information obtained by the decoding unit from processing the input image to obtain the spatial dimension and details of the input image.

[0017] According to some embodiments of the present invention, the plurality of training images include original images and target true value images; the generative adversarial network objective function is as shown in Formula 1:

[0018]

[0019] Among them, L cGAN (G, D) is the target function value of the generative adversarial network; G represents the codec generation network; D represents the discriminant network; x represents the original image; y represents the target true value image; z represents random noise; and They represent the mathematical expectations about (x, y) and (x, z) respectively; p data (x,y) represents the joint data distribution about (x,y);

[0020] The distance function is shown in Formula 2:

[0021]

[0022] Among them, L L1 (G) is the distance function value; represents the mathematical expectation about ((x,y),z);

[0023] The color loss function is shown in Formula 3:

[0024]

[0025] Among them, L color (G) is the color loss value; [G(x,z)] b and Y b They are the results of Gaussian filtering of the generated image G(x,z) and the target true value image y;

[0026] The calculation formula of the target parameter is shown in Formula 4:

[0027]

[0028] Among them, G * is the target parameter value; G represents the encoding and decoding generation network; D represents the discriminant network; L cGAN (G, D) is the objective function value of the generative adversarial network; L L1 (G) is the distance function value; L color (G) is the color loss value; λ is the balance loss weight.

[0029] According to some embodiments of the present invention, the step of performing acne removal processing on the image to be processed includes:

[0030] Extracting facial key points of the image to be processed;

[0031] Determine the facial elliptical area according to the facial key points;

[0032] Constructing a mask image of the elliptical area of ​​the face;

[0033] Graying the image to be processed;

[0034] Performing AND calculation on the grayscale image and the mask image to obtain a face part image;

[0035] Performing binarization operation and connected domain analysis on the face part image in sequence to obtain isolated point regions;

[0036] Determine the position coordinates of the acne spot according to the isolated point area;

[0037] The image to be processed is repaired according to the position coordinates.

[0038] According to some embodiments of the present invention, the step of performing skin resurfacing processing on the image to be processed is specifically:

[0039] A surface filtering algorithm is used to perform skin resurfacing processing on the image to be processed.

[0040] According to some embodiments of the present invention, the step of whitening the image to be processed includes:

[0041] The mapping table is constructed using the logarithmic curve formula;

[0042] A region of the image to be processed is selectively enhanced according to the mapping table.

[0043] A facial beautification processing system according to a second aspect of an embodiment of the present invention includes:

[0044] An acquisition module, used for acquiring an image to be processed;

[0045] A processing module, used for performing acne removal, skin grinding, whitening and filter processing on the image to be processed to generate a target image;

[0046] Wherein, a pre-trained generative adversarial network is used to perform filter processing on the image to be processed; the training steps of the generative adversarial network include:

[0047] Obtain several training images as a training set;

[0048] Using the training set to calculate a generative adversarial network objective function, a distance function, and a color loss function in the generative adversarial network;

[0049] Calculating target parameters according to calculation results of the generative adversarial network objective function, distance function and color loss function;

[0050] The parameters of the generative adversarial network are updated to the target parameters.

[0051] A facial beautification processing system according to a third aspect of an embodiment of the present invention includes:

[0052] at least one memory for storing a program;

[0053] At least one processor is used to load the program to execute the beauty processing method of the embodiment of the first aspect.

[0054] A storage medium according to an embodiment of a fourth aspect of the present invention stores a program executable by a processor, and when the program executable by the processor is executed by the processor, it is used to execute the beauty processing method of the embodiment of the first aspect.

[0055] According to a terminal device of a fifth aspect embodiment of the present invention, the terminal device controls the image beautification processing process through the beautification processing method of the first aspect embodiment.

[0056] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments, wherein:

[0058] Figure 1 A flowchart of a training method for a generative adversarial network according to an embodiment of the present invention;

[0059] Figure 2 A module block diagram of a codec generation network according to an embodiment of the present invention;

[0060] Figure 3 Module block diagram of a discriminant network according to an embodiment of the present invention;

[0061] Figure 4 The present invention is a flowchart of a beauty processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0063] In the description of the present invention, "several" means more than one, "many" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0064] In the description of the present invention, unless otherwise clearly defined, words such as "setting" should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above words in the present invention in combination with the specific content of the technical solution.

[0065] In the description of the present invention, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0066] Reference Figure 1 The embodiment of the present invention provides a training method of a generative adversarial network applied to an image beautification process. This embodiment can be applied to a processor of a server, a platform or each beautification device terminal.

[0067] like Figure 2 and Figure 3 As shown, the generative adversarial network includes a codec generation network and a discriminant network, wherein the codec generation network includes several coding units and several decoding units. The coding unit includes a convolution layer, a downsampling layer, a normalization layer and a nonlinear activation layer. The decoding unit includes an upsampling layer, a convolution layer, a normalization layer and a nonlinear activation layer. In the application process, the coding unit compresses the input image and extracts high-level semantic features; the decoding unit upsamples the input image and combines the information obtained by the decoding unit from the input image processing to obtain the spatial dimension and details of the input image. The codec generation network is mainly used to generate images, and the backbone structure is a lightweight network MobileNetV2, so as to achieve a smaller parameter amount and a faster speed while ensuring accuracy. The discriminant network is a convolutional neural network, including two convolution layers, two batch normalization layers, two nonlinear activation layers, two pooling layers, two fully connected layers and a binary classification output, which is used to perform a binary classification task to distinguish between images generated by the codec generation network and real images.

[0068] In implementation, the training method includes the following steps:

[0069] S11, obtaining a plurality of training images as a training set, wherein the plurality of images may be about 20,000 images collected from the terminal, and the 20,000 images may be composed of fully matched original images and manually processed target true value images.

[0070] S12, using the training set to calculate the generative adversarial network objective function, distance function and color loss function in the generative adversarial network. Since the entire generative adversarial network optimization goal includes three items, namely the generative adversarial network objective function, distance function and color loss function, therefore, in this step, only the data of these three goals need to be calculated during the training process.

[0071] Specifically, the generative adversarial network objective function in the generative adversarial network is calculated according to the training set by formula 1:

[0072]

[0073] Among them, L cGAN (G, D) is the target function value of the generative adversarial network; G represents the codec generation network; D represents the discriminant network; x represents the original image; y represents the target true value image; z represents random noise; and They represent the mathematical expectations about (x, y) and (x, z) respectively; p data (x,y) represents the joint data distribution about (x,y).

[0074] The distance function in the generative adversarial network is calculated based on the training set using formula 2:

[0075]

[0076] Among them, L L1 (G) is the distance function value, represents the mathematical expectation about ((x,y),z).

[0077] The color loss function in the generative adversarial network is calculated based on the training set using Formula 3:

[0078]

[0079] Among them, L color (G) is the color loss value; [G(x,z)] b and Y b They are the results of Gaussian filtering of the generated image G(x,z) and the target true value image y respectively.

[0080] After obtaining the above optimization objective function, execute step S13.

[0081] S13, calculating the target parameters according to the calculation results of the generative adversarial network objective function, distance function and color loss function. Among them, S13 can be calculated by formula 4:

[0082]

[0083] Among them, G * is the target parameter value; G represents the encoding and decoding generation network; D represents the discriminant network; L cGAN (G, D) is the objective function value of the generative adversarial network; L L1 (G) is the distance function value; L color (G) is the color loss value; λ is the balance loss weight.

[0084] S14, updating the parameters of the generative adversarial network to the target parameters, indicating that the generative adversarial network has completed the training process. Then, the trained generative adversarial network is applied to the actual image beautification process to avoid the situation where the original image is destroyed during the image processing process, thereby providing a high-quality natural image that retains the detailed features of the original image to a certain extent.

[0085] Reference Figure 4 An embodiment of the present invention provides a beauty processing method, which can be applied to a background processor of a server, platform or terminal device.

[0086] During implementation, this embodiment includes the following steps:

[0087] S21, obtaining an image to be processed. The processed image is a real-time image captured by the user during the process of taking photos using a terminal device.

[0088] S22, perform acne removal, skin grinding, whitening and filter processing on the image to be processed to generate a target image. The filter processing process is to use Figure 1 The method is processed by the trained generative adversarial network.

[0089] In some embodiments, the step of performing acne removal processing on the image to be processed can be implemented in the following manner:

[0090] Extract the facial key points of the image to be processed, where the facial key points can include 4, one key point each on the top, bottom, left and right sides of the face. Then determine the facial elliptical area based on the facial key points, and construct a mask image of the facial elliptical area. At the same time, grayscale the image to be processed. Then perform AND calculation on the grayscale image and the mask image to obtain the facial part image; perform adaptive threshold binarization operation and connected domain analysis on the facial part image in turn to find the isolated point area. Determine the position coordinates of the acne spot based on the isolated point area; finally, repair the image to be processed based on the position coordinates.

[0091] In some embodiments, the step of performing skin resurfacing processing on the image to be processed may be performed by using a surface filtering algorithm to perform skin resurfacing processing on the image to be processed. The surface filtering algorithm is shown in Formula 5:

[0092]

[0093] v res is the output pixel value; w is the current pixel value; r is the field radius, w i is the pixel value within radius r; Q is a user parameter, ranging from [0,255].

[0094] In some embodiments, the step of whitening the image to be processed can be performed by constructing a mapping table using a logarithmic curve formula; then, the image to be processed is selectively enhanced according to the mapping table, for example, pixels that are too low or too high are slightly increased, and pixels in the middle are increased more significantly. Specifically, the processing process of this embodiment is shown in Formula 6:

[0095]

[0096] Among them, w(x,y) is the original image pixel; v(x,y) is the value after brightening; β is the logarithmic curve parameter.

[0097] In summary, the above embodiments perform acne removal, skin smoothing, and whitening processing on the processed image respectively, and use the trained generative adversarial network to filter the processed image to obtain the target image, thereby avoiding the destruction of the original image, thereby providing a high-quality natural image that retains the detailed features of the original image to a certain extent.

[0098] The embodiment of the present invention provides a Figure 4 The beauty processing system corresponding to the method includes:

[0099] An acquisition module, used for acquiring an image to be processed;

[0100] A processing module, used for performing acne removal, skin grinding, whitening and filter processing on the image to be processed to generate a target image;

[0101] Wherein, a pre-trained generative adversarial network is used to perform filter processing on the image to be processed; the training steps of the generative adversarial network are as follows: Figure 1 As shown, it includes:

[0102] Obtain several training images as a training set;

[0103] Using the training set to calculate the generative adversarial network objective function, distance function and color loss function in the generative adversarial network;

[0104] Calculate the target parameters according to the calculation results of the generative adversarial network objective function, distance function and color loss function;

[0105] The parameters of the generative adversarial network are updated to the target parameters.

[0106] The contents of the method embodiments of the present invention are all applicable to the system embodiments. The functions specifically implemented by the system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method.

[0107] The embodiment of the present invention provides a beauty processing system, including:

[0108] at least one memory for storing a program;

[0109] At least one processor is used to load the program to execute the beauty processing method.

[0110] The contents of the method embodiments of the present invention are all applicable to the system embodiments. The functions specifically implemented by the system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0111] An embodiment of the present invention provides a storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to perform a beauty processing method.

[0112] An embodiment of the present invention provides a terminal device, wherein the terminal device Figure 4 The beauty processing method shown controls the image beauty processing process.

[0113] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in the relevant technical field without departing from the purpose of the present invention. In addition, the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

Claims

1. A facial beautification method, characterized in that: The following steps are involved: Get the image to be processed; Performing acne removal, skin smoothing, whitening and filter processing on the image to be processed to generate a target image; Wherein, a pre-trained generative adversarial network is used to perform filter processing on the image to be processed; the training steps of the generative adversarial network include: Obtain several training images as a training set; Using the training set to calculate a generative adversarial network objective function, a distance function, and a color loss function in the generative adversarial network; Calculating target parameters according to calculation results of the generative adversarial network objective function, distance function and color loss function; Updating the parameters of the generative adversarial network to the target parameters; The step of whitening the image to be processed comprises: A mapping table is constructed by using a logarithmic curve formula; and a selective enhancement area is performed on the image to be processed according to the mapping table; The mapping table is constructed by using a logarithmic curve formula; the process of selectively enhancing the area of ​​the image to be processed according to the mapping table is as follows: is the original image pixel; is the value after brightening; is the logarithmic curve parameter; The step of performing skin resurfacing processing on the image to be processed is specifically as follows: Using a surface filtering algorithm to perform skin resurfacing processing on the image to be processed; The surface filtering algorithm is as follows: In the formula, is the output pixel value; is the current pixel value; r is the area radius, The radius r The pixel value within; Q is a user parameter, ranging from [0,255].

2. A facial beautification method according to claim 1, characterized in that: The generative adversarial network includes a codec generation network and a discriminant network, and the codec generation network includes a plurality of encoding units and a plurality of decoding units; The encoding unit compresses the input image and extracts high-level semantic features; The decoding unit upsamples the input image and combines the information obtained by the decoding unit from processing the input image to obtain the spatial dimension and details of the input image.

3. A facial beautification method according to claim 2, characterized in that: The plurality of training images include original images and target true value images; the generative adversarial network objective function is as shown in Formula 1: Formula 1 in, is the target function value of the generative adversarial network; G represents the codec generation network; D represents the discriminant network; x represents the original image; y represents the target true value image; z represents random noise; and Represent the mathematical expectations about (x,y) and (x,z) respectively; represents the joint data distribution about (x, y); The distance function is shown in Formula 2: Formula 2 in, is the distance function value; represents the mathematical expectation about ((x,y),z); The color loss function is shown in Formula 3: Formula 3 in, is the color loss value; and Generate images for And the result after Gaussian filtering of the target true image Y; The calculation formula of the target parameter is shown in Formula 4: Equation 4 in, is the target parameter value; G represents the encoding and decoding generation network; D represents the discriminant network; is the objective function value of the generative adversarial network; is the distance function value; is the color loss value; To balance the loss weight.

4. A facial beautification method according to claim 1, characterized in that: The step of performing acne removal processing on the image to be processed includes: Extracting facial key points of the image to be processed; Determine the facial elliptical area according to the facial key points; Constructing a mask image of the elliptical area of ​​the face; Graying the image to be processed; Performing AND calculation on the grayscale image and the mask image to obtain a face part image; Performing binarization operation and connected domain analysis on the face part image in sequence to obtain isolated point regions; Determine the position coordinates of the acne spot according to the isolated point area; The image to be processed is repaired according to the position coordinates.

5. A facial beautification processing system, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A processing module, used for performing acne removal, skin grinding, whitening and filter processing on the image to be processed to generate a target image; Wherein, a pre-trained generative adversarial network is used to perform filter processing on the image to be processed; the training steps of the generative adversarial network include: Obtain several training images as a training set; Using the training set to calculate a generative adversarial network objective function, a distance function, and a color loss function in the generative adversarial network; Calculating target parameters according to calculation results of the generative adversarial network objective function, distance function and color loss function; Updating the parameters of the generative adversarial network to the target parameters; The step of whitening the image to be processed comprises: A mapping table is constructed by using a logarithmic curve formula; and a selective enhancement area is performed on the image to be processed according to the mapping table; The mapping table is constructed by using a logarithmic curve formula; the process of selectively enhancing the area of ​​the image to be processed according to the mapping table is as follows: is the original image pixel; is the value after brightening; is the logarithmic curve parameter; The step of performing skin resurfacing processing on the image to be processed is specifically as follows: Using a surface filtering algorithm to perform skin resurfacing processing on the image to be processed; The surface filtering algorithm is as follows: In the formula, is the output pixel value; is the current pixel value; r is the area radius, The radius r The pixel value within; Q is a user parameter, ranging from [0,255].

6. A facial beautification processing system, characterized in that: include: at least one memory for storing a program; At least one processor is used to load the program to execute the beauty processing method according to any one of claims 1 to 4.

7. A storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to execute the beauty processing method according to any one of claims 1 to 4 when executed by the processor.

Citation Information

Patent Citations

  • Image brightness adjustment method and device, and terminal equipment

    CN106897963A

  • Photo processing method, device, computer readable storage medium and electronic equipment

    CN107147851A

  • Automatic pimple detection method

    CN107274452A

  • Stylization system and method for ultra high-definition resolution patterns

    CN108960408A

  • An automatic face acne removing method based on image inpainting model

    CN109345480A