Skin beautification method for facial images based on additive lee filtering and skin resurfacing

Abstract

The present invention relates to a skin beautification method for facial images based on additive lee filter skin grinding, comprising the following steps: S1: obtaining an original facial image, using an additive lee filter skin grinding operator to perform skin grinding processing on the original facial image, and obtaining image B; S2: inputting image B into a degreasing operator to remove greasy light to obtain image C; S3: inputting image C into a whitening operator to perform whitening processing to obtain a final skin beautification image D, and outputting image D. The method of the present invention is to perform precise adjustment on human skin, and through this automatic adjustment method, the workload of the photo retoucher can be greatly reduced, and the quality of the adjusted photos is better.

Description

Technical Field

[0001] The invention relates to a facial beautification method, and in particular to a facial image skin beautification method based on additive lee filtering and skin resurfacing. Background Art

[0002] With the rapid iteration of hardware technology in recent years, whether it is a digital camera or a smart phone, the camera function is constantly improving, the imaging pixels are getting larger and larger, and the clarity of the portraits taken is getting higher and higher. As a result, spots, acne marks, wrinkles and other blemishes on the face skin will also be photographed. To address this problem, in the field of portrait beauty, it is necessary to perform skin smoothing on the face skin to effectively remove noise, black spots, and blemishes in the portrait image to achieve smoothness, texture, and softening of the face. It is necessary to perform whitening on dark and yellowish skin to make the skin fair and rosy.

[0003] Skin beautification algorithms based on digital image processing include skin smoothing algorithms, whitening algorithms, etc. ] et al. proposed a skin resurfacing algorithm based on nonlinear filters to remove facial wrinkles and spots. They compared the effects of simple filters and extended filters on skin processing. Simple filters can remove roughness and small spots, while extended filters can remove larger spots. Lee C et al. segmented the face and neck regions based on face detection and face alignment technology, and applied a smoothing filter to enhance the skin. Liang L et al. used a region-aware mask to extract the skin region, and divided the skin into three layers, namely, a smoothing layer, an illumination layer, and a color layer, by using an edge-preserving operator. The skin was beautified by adjusting the parameters. Velusamy S et al. proposed an attribute-aware dynamic smoothing filter guided by the number of skin blemishes and the degree of texture roughness. The skin texture was restored by wavelet band processing, and the beautification effect was good. S Liu et al. used deep learning to divide beauty makeup into two steps: makeup recommendation and makeup migration. The degree of makeup can be adjusted by adjusting the weight of each cosmetic.

[0004] Some of the common beautification software and tools on the market require manual intervention to adjust the parameters of functions such as skin smoothing and whitening, while some use the same parameters for all input photos. The problems caused by this are: ① Tools or software that require manual intervention cannot automatically complete batch image beautification processing; ② For different photos, it is impossible to make targeted adjustments based on the shooting scene, light and shadow structure, and skin condition; ③ For the same picture, if repeated operations are performed, the picture will be repeatedly smoothed and whitened, resulting in an increasingly "fake" effect and a worse texture. Summary of the invention

[0005] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: to achieve good image processing effect and good image texture after skin grinding and whitening.

[0006] To solve the above technical problems, the present invention adopts the following technical solution: based on the additive lee filter skin grinding face image

[0007] The skin care method includes the following steps:

[0008] S1: Obtain the original face image, and use the additive lee filter skin smoothing operator to smooth the original face image to obtain image B;

[0009] S2: Input image B into the deglare operator to remove the gloss and obtain image C;

[0010] S3: Input image C into the whitening operator to obtain a final skin-beautified image D after whitening, and output image D.

[0011] As an improvement, the process of performing skin smoothing processing on the original face image by the additive Lee filter skin smoothing operator in S1 is as follows:

[0012] S11: Cut the original face image into N*M size to get the image to be processed, and use x ij Represents the pixel value at the coordinate (i, j) of the image to be processed;

[0013] S12: setting a sliding window of size (2*n+1)*(2*m+1), and calculating the local mean and local variance of the pixel values ​​of all coordinate points in the image to be processed within the sliding window;

[0014] S13: Determine the σ value based on the skin quality classification level of the original face image. Each skin quality level corresponds to a σ value. Then, based on the σ value and the local variance of the pixel values ​​of all coordinate points in the image to be processed within the sliding window obtained in S12, the k value is calculated using the following formula:

[0015]

[0016] Among them, σ represents the parameter of the additive lee filter, k represents the degree of skin smoothing of the original image, and v ij Represents the local variance of the image to be processed inside the sliding window;

[0017] S14: Additively denoise the pixel value at the coordinate (i, j) according to the following formula, and use the pixel value after additive denoising as the pixel value at coordinate (i, j);

[0018]

[0019] in, Represents the pixel value after additive denoising at coordinate (i, j);

[0020] S15: Repeat S12-S14, and traverse all coordinate points in the image to be processed to obtain the pixel value of each coordinate point after additive denoising. The pixel values ​​corresponding to all coordinate points after additive denoising are used as the pixel values ​​of the pixel point after skin smoothing, and image B is obtained.

[0021] As an improvement, the method for calculating the local variance of all coordinate pixel values ​​in the image to be processed within the sliding window in S12 is:

[0022]

[0023] Among them, x kl represents the pixel value at the coordinate (k, l) in the sliding window, m ij Represents the local average value of the pixel values ​​of all coordinate points in the image to be processed within the sliding window, v ij Represents the local variance of all coordinate pixel values ​​in the image to be processed within the sliding window.

[0024] As an improvement, the method for classifying skin quality according to the original face image in S13 is:

[0025] S131: define multiple feature points in the original face image, connect all the feature points in sequence to form a polygon, and the resulting mask is the complete face area, which is defined as M points , the skin area mask of the human body is M human , the mask image of the face skin area is M face :

[0026] M face =M points ∩M human (1-5);

[0027] S132: The mask image of the facial skin area is classified into two dimensions according to wrinkles and pores, as follows:

[0028] Wrinkle levels are divided into four levels, three levels, two levels, and one level, and each wrinkle level is assigned a value of 3, 2, 1, and 0 in sequence;

[0029] The pore levels are divided into four levels, three levels, two levels, and one level, and each pore level is assigned a value of 3, 2, 1, and 0 in sequence;

[0030] S133: In the original face image, four parts of the face, namely the forehead, the left cheek, the right cheek and the chin, are selected as regions of interest, and the weights of each region divided into wrinkles and pores are set. Then, the grade assignments of the four parts are calculated using the following formula, and the value of σ is equal to the grade assignment:

[0031]

[0032] in, γ=1,2,3,4 represent the weights of wrinkles in the forehead, left cheek, right cheek and chin respectively.

[0033] γ=1, 2, 3, 4 represent the weights of pores in the forehead, left cheek, right cheek and chin respectively.

[0034] As an improvement, the process of removing the gloss from the image B by the gloss operator in S2 is as follows:

[0035] S21: classifying the skin quality of image B to determine the oiliness level of image B;

[0036] The oil gloss levels are classified into level 4 oil gloss, level 3 oil gloss, level 2 oil gloss and level 1 oil gloss, and each oil gloss level is assigned a value in turn;

[0037] S22: Calculate the maximum chromaticity σ at each pixel of image B max , and store image B as grayscale image I;

[0038] S23: Calculate the maximum value λ of the approximate diffuse reflection chromaticity at each pixel of image B max , and store it as a grayscale image II;

[0039] S24: using the grayscale image II as a guide image, applying a joint bilateral filter to the image grayscale image I, and storing the filtered image as a preprocessed image;

[0040] S25: Calculate σ for each pixel p in the preprocessed image max (p), compare and σ max , take the maximum value as shown in formula 2-17:

[0041]

[0042] S26: Repeat steps S24 and S25 until each pixel Then execute the next step;

[0043] S27: Determine σ for each pixel p in the preprocessed image max (p) is the selected RGB channel, and the pixel of the selected channel of each pixel point p is σ max (p)×255, and then iterate the pixels of the two unselected channels and the pixels of the selected channel for each pixel point p to obtain the preprocessed image;

[0044] S27: Classify the skin quality of the preprocessed image to determine the oiliness level. When the oiliness level of the preprocessed image is lower than the oiliness level of image B in S21 and is not greater than the preset oiliness level threshold, the preprocessed image is output as image C. Otherwise, return to step S22 and update image B with the preprocessed image.

[0045] As an improvement, the maximum chromaticity σ at each pixel of the image B is calculated in S22 max The process is as follows:

[0046] The reflected light color J in the RGB color space is expressed as the diffuse reflection value J D and the specular reflection value J s The linear combination of colors is as follows:

[0047] J=J D +J s (2-5);

[0048] Define chromaticity as the color component σ c The score is as follows:

[0049]

[0050] Where c∈{r,g,b},J c Indicates the color of reflected light;

[0051] Diffuse reflection chromaticity Λ c and lighting chromaticity Γ c Define the following formula 2-7 and formula 2-8:

[0052]

[0053] in, Represents the diffuse component in layer c, Represents the specular reflection component in layer c;

[0054] According to the above formula, the reflected light color J c Defined as formula 2-9:

[0055]

[0056] Among them, u represents the layer, which can be r layer, g layer or b layer. Represents the diffuse component in layer u, Represents the specular reflection component in layer u;

[0057] Assume that the illumination chromaticity is estimated using white, and the input image B is normalized to and Γ r , Γ gand Γ b Represents the illumination chromaticity of the r, g, and b layers respectively, and Represents the specular reflection values ​​of the r, g, and b layers respectively;

[0058] Then according to the previous formula, the diffuse reflection component is as shown in formula 2-10:

[0059]

[0060] in, Represents the diffuse reflection value of the cth layer;

[0061] The maximum chromaticity is defined as Equation 2-11:

[0062] σ max =max(σ r , σ g , σ b ) (2-11);

[0063] Among them, σ r , σ g , σ b Represents the maximum color components of the r, g, and b layers respectively;

[0064] The maximum diffuse reflectance chromaticity is defined as Formula 2-12:

[0065] Λ max =max(Λ r , Λ g , Λ b ) (2-12);

[0066] Among them, Λ r , Λ g , Λ b Represents the maximum diffuse reflection chromaticity of the r, g and b layers respectively;

[0067] The diffuse component can be represented by Λ max Expressed as formula 2-13:

[0068]

[0069] Λ max The value range is

[0070] As an improvement, the maximum value λ of the approximate diffuse reflection chromaticity at each pixel of the image B is calculated in S23. max The process is as follows:

[0071] Let σ min =min(σ r , σ g , σb ), using λ c To estimate Λ c , calculation formula 2-14 is as follows:

[0072]

[0073] λ c It is an intermediate variable and has no actual meaning;

[0074] Approximate diffuse reflection chromaticity λ c and the true diffuse chromaticity Λ c The relationship between them is described as 1) and 2).

[0075] 1) For any two pixels p and q, if Λ c (p) = Λ c (q), then λ c (p) = λ c (q)

[0076] 2) For any two pixels p and q, if λ c (p) = λ c (q), then only if Λ min (p) = Λ min (q), Λ c (p) = Λ c (q)

[0077] The maximum value of the approximate diffuse reflection chromaticity is given by formula 2-15:

[0078]

[0079] Among them, λ r ,λ g ,λ b The calculated variables representing the r, g, and b layers have no actual meaning;

[0080] Using the approximate diffuse reflection maximum chromaticity value as the smoothing parameter, the maximum chromaticity σ after filtering max The calculation formula 2-16 is as follows:

[0081]

[0082] in, It means that the calculation variable of pixel p has no actual meaning. and It is a typical Gaussian distribution with space and distance weighting function.

[0083] As an improvement, the grayscale image II in S24 is used as a guide image, and the process of applying a joint bilateral filter to the grayscale image I is as follows:

[0084]

[0085] Among them, I D (i, j) represents the pixel value of the pixel point with coordinates (i, j) after joint bilateral filtering, (k, l) represents the pixel coordinates of other points in the filtering window, It represents the pixel value of the center point. It represents the pixel values ​​of the remaining nodes, and w(i, j, k, l) is a parameter of the multiplication of the Gaussian distribution space function and the Gaussian function of the pixel intensity similarity;

[0086] The joint bilateral filter is defined as follows:

[0087]

[0088] yes This part is only related to the coordinates of the pixels p(i, j) and q(k, l). Substitute into the formula, σ max (q) is equal to the I(k,l) part in the bilateral filter, which represents the pixel value at point q.

[0089] As an improvement, the process of whitening the image C by the whitening operator in S3 is as follows:

[0090] S31: classify the skin quality of image C. The skin color levels are divided into four categories: level 4, level 3, level 2, and level 1. Each skin color level is assigned a value of β=3, 2, 1, and 0 in sequence.

[0091] S32: Normalize the pixel values ​​of the R, G, and B channels of the image C layer by layer. The normalization method is:

[0092]

[0093] Where f(x,y) represents the pixel value at each pixel of the input image C, and w(x,y)∈[0,1] represents the output image C′. The output image C′ has three layers: R, G, and B.

[0094] S33: Enhance the input image C′ using the following formula, as shown in Formula 3-2:

[0095]

[0096] Among them, w(x, y) is the input image C′, v(x, y) is the output image D after whitening, and β is the parameter that controls the degree of whitening.

[0097] Compared with the prior art, the present invention has at least the following advantages:

[0098] 1. Currently, most of the automatic photo retouching methods on the market are performed on the entire photo, while the method of the present invention is to perform precise retouching on human skin. Through this automatic retouching method, the workload of the photo retoucher can be greatly reduced, and the quality of the retouched photos is better.

[0099] 2. The computational complexity of the existing filter for each pixel is greater than that of the additive lee filter. When we have a large number of images that need to be smoothed and the skin's wrinkles, acne marks and other defects are relatively serious, we need to increase the filter parameters. When the intensity parameter is too large, the algorithm speed will be greatly affected. The additive lee filter provided by the present invention calculates the parameter k of the degree of skin smoothing modification by skin quality classification. Different skin quality levels will affect the final degree of skin smoothing, making our skin smoothing effect and texture texture real. In addition, while the additive lee filter smoothes the skin, the skin processing effect is uniform, the light and dark contrast is retained, the texture is retained, and therefore the texture is real. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 Flowchart for additive lee filter resurfacing.

[0101] Figure 2 Schematic diagram of the face polygonal outline and region of interest.

[0102] Figure 3 This is a schematic diagram of skin color grading.

[0103] Figure 4 This is a schematic diagram of oil gloss classification.

[0104] Figure 5 Schematic diagram of wrinkle classification.

[0105] Figure 6 Schematic diagram of pore classification.

[0106] Figure 7 Comparison of the original face image before and after skin smoothing. Figure 7a It's the original picture. Figure 7b This is the image after being processed by the skin resurfacing operator.

[0107] Figure 8 It is a flowchart of the method of the present invention.

[0108] Figure 9 Comparison of the original face image before and after de-greasing. Figure 9a It's the original picture. Figure 9b This is the image after being processed by the de-glazing operator.

[0109] Fig.10 Whitening parameter curve comparison chart.

[0110] Figure 11 Comparison of the original face image before and after whitening. Fig.11a It's the original picture. Fig.11bThis is the image after whitening operator processing.

[0111] Figure 12 Comparison of the original face image before and after whitening. Fig.12a It's the original picture. Figure 12b This is the image after whitening operator processing.

[0112] Fig.13 Comparison of skin quality scores before and after beautification. DETAILED DESCRIPTION

[0113] The present invention is described in further detail below.

[0114] The method for skin beautification of a face image based on additive lee filtering and skin resurfacing comprises the following steps:

[0115] S1: Obtain the original face image, and use the additive lee filter skin smoothing operator to smooth the original face image to obtain image B;

[0116] S2: Input image B into the deglare operator to remove the gloss and obtain image C;

[0117] S3: Input image C into the whitening operator to obtain a final skin-beautified image D after whitening, and output image D.

[0118] Specifically, the process of performing skin smoothing processing on the original face image by the additive Lee filter skin smoothing operator in S1 is as follows:

[0119] S11: Cut the original face image into N*M size to get the image to be processed, and use x ij Represents the pixel value at coordinate (i, j) of the image to be processed.

[0120] S12: Set a sliding window of size (2*n+1)*(2*m+1), and calculate the local variance of the pixel values ​​of all coordinate points in the image to be processed within the sliding window; any existing method can be used to calculate the local mean and local variance of all coordinate pixel values ​​in the image to be processed within the sliding window. The present invention preferably selects the following method. The following method is preferably used:

[0121] S13: Determine the σ value based on the skin quality classification level of the original face image. Each skin quality level corresponds to a σ value. Then, based on the σ value and the local variance of the pixel values ​​of all coordinate points in the image to be processed within the sliding window obtained in S12, the k value is calculated using the following formula:

[0122]

[0123] Among them, σ represents the parameter of the additive lee filter, k represents the degree of skin smoothing of the original image, and v ijRepresents the local variance of the image to be processed within the sliding window; for example, when the classification is level one, the value of σ is 0. According to the σ value calculation formula It can be obtained that k=1, indicating that the skin condition is good at this time and no skin resurfacing modification is required); the additive lee filter is a prior art.

[0124] σ represents the parameter of additive Lee filtering, which can be used to control the degree of filtering. When the classification is four levels, the value of σ is 3; when the classification is three levels, the value of σ is 2; when the classification is two levels, the value of σ is 1; when the classification is one level, the value of σ is 0.

[0125] S14: Additively denoise the pixel value at the coordinate (i, j) according to the following formula, and use the pixel value after additive denoising as the pixel value at coordinate (i, j);

[0126]

[0127] in, Represents the pixel value after additive denoising at coordinate (i, j).

[0128] S15: Repeat S12-S14, and traverse all coordinate points in the image to be processed to obtain the pixel value of each coordinate point after additive denoising. The pixel values ​​corresponding to all coordinate points after additive denoising are used as the pixel values ​​of the pixel point after skin smoothing, so as to obtain the filtered image and output it.

[0129] Specifically, the method for calculating the local variance of all coordinate pixel values ​​in the image to be processed within the sliding window in S12 is:

[0130]

[0131] Among them, x kl represents the pixel value at the coordinate (k, l) in the sliding window. The coordinates of the pixel being denoised in the image to be processed are (i, j), and the coordinates of one of the surrounding pixels used are (k, l). ij Represents the local average value of the pixel values ​​of all coordinate points in the image to be processed within the sliding window, v ij Represents the local variance of all coordinate pixel values ​​in the image to be processed within the sliding window.

[0132] Specifically, the method for classifying skin quality according to the original face image in S13 is:

[0133] S131: define multiple feature points in the original face image, connect all the feature points in sequence to form a polygon, and the resulting mask is a complete face area, which is defined as M points , the skin area mask of the human body is M human , the mask image of the face skin area is Mface :

[0134] M face =M points ∩M human (1-5);

[0135] Using the TensorFlow-based deep neural network face detection algorithm provided by OpenCV and the face alignment algorithm proposed by Adrian Bulat, 81 aligned feature points are obtained. The points of the outermost border of the face are connected in sequence to form a polygon, and the resulting mask is the complete face area, defined as M points ,like Figure 2 The outer frame is shown as a red polygon.

[0136] Due to the influence of factors such as hair, glasses, accessories, and light shadows, the skin classification of the face may be inaccurate. Therefore, based on the key point positioning and segmentation, it is necessary to find the intersection with the result of the whole body skin segmentation to obtain the final facial skin area.

[0137] S132: The mask image of the facial skin area is classified into two dimensions according to wrinkles and pores, as follows:

[0138] Human skin has various qualities and can be divided into multiple types based on skin color, oiliness, wrinkles, and pores. In the task of beautification, the first thing to do is to determine the skin quality type and then determine the parameters of the algorithm to handle different defects.

[0139] Skin color: At present, the research on human skin color mainly focuses on the fields of medical diagnosis, face comparison, expression recognition, etc. The present invention proposes to subdivide skin color into grades in order to better determine the parameters of the beauty algorithm, which is different from the standard skin color grading standard. In portrait photography, due to differences in lighting, shooting equipment, shooting parameters, etc., the skin color of the same person will also show different results. Therefore, the present invention classifies skin color based on the brightness and color reflected in the image, rather than the human body itself.

[0140] Skin color levels are divided into four categories: level 4, level 3, level 2, and level 1. Each skin color level is assigned 1, 2, 3, and 0. Level 4 dark skin color refers to dark skin color due to dark skin color or light shadow during shooting. Level 3 yellow skin color refers to yellow skin color due to yellow skin color, ambient light, or white balance settings. Level 2 white skin color refers to white skin due to white skin color or overexposure. Level 1 normal skin color refers to skin color type that does not need to be adjusted, such as Figure 3 shown.

[0141] The gloss levels are classified into level 4 gloss, level 3 gloss, level 2 gloss and level 1 gloss, and each gloss level is assigned 1, 2, 3, and 0 respectively.

[0142] In portrait photography, the highlight area of ​​the face refers to the area with the highest L average value in the Lab color space. Based on the L value of the highlight area, the exposure level of the photo can be determined, which is usually divided into underexposure, normal, and overexposure. In the post-production adjustment process, for underexposed and overexposed photos, it is necessary to brighten and suppress the highlights respectively.

[0143] Since oily skin secretes oil, the oil reflects light during the imaging process, causing the highlight area of ​​the face to reflect light. Therefore, the highlight area is often accompanied by the oily area. By classifying the oily level, the parameters of the de-oiling algorithm are determined.

[0144] Level 4 oily shine means that the skin secretes a lot of oil and the portrait has a high degree of reflection; Level 1 oily shine means that the skin secretes a little oil and the portrait has no reflection. Figure 4 shown.

[0145] The wrinkle levels are divided into four levels, three levels, two levels, and one level, and each wrinkle level is assigned 1, 2, 3, and 0 respectively.

[0146] People at different ages will have wrinkles of different grades. Many computer vision-based wrinkle quantitative measurement methods have been proposed at home and abroad, but they are greatly affected by the illumination, shadow, resolution, etc. when the image is taken, and the detection effect is unstable. The skin resurfacing algorithm focuses on the wrinkles in the skin, so the correctness of wrinkle classification directly determines the effect of the skin resurfacing algorithm. Level 4 represents the level with the most wrinkles, the deepest texture, and the final degree, while level 1 represents the level with few wrinkles, very light texture, and the lowest degree. Figure 5 shown.

[0147] The pore levels are divided into level 4, level 3, level 2, and level 1, and each pore level is assigned 1, 2, 3, and 0 respectively.

[0148] Rough skin is also the focus of the skin resurfacing algorithm. The number and size of skin pores reflect whether the skin is smooth and delicate. The skin conditions of different people vary greatly. The skin is divided into four levels: three, two, and one according to the degree of roughness. The fourth level represents the level of roughness and obvious pores, and the first level represents the level of smoothness and delicateness. Figure 6 shown.

[0149] S133: In the original face image, four parts of the face, namely the forehead, the left cheek, the right cheek and the chin, are selected as regions of interest, and the weights of each region divided into wrinkles and pores are set. Then, the grade assignments of the four parts are calculated using the following formula, and the value of σ is equal to the grade assignment:

[0150]

[0151] in, γ=1,2,3,4 represent the weights of wrinkles in the forehead, left cheek, right cheek and chin respectively. γ=1, 2, 3, 4 represent the weights of pores in the forehead, left cheek, right cheek and chin respectively.

[0152] In the portrait photo, after detecting the face rectangle and aligning the face key points, the area of ​​interest is selected, and the parameters of the beautification algorithm are finally determined based on the above-mentioned skin classification indicators.

[0153] When classifying the skin by indicators, the skin classification weights of different areas of the face are different. The highlight area of ​​the forehead is usually an area with heavier oiliness and lighter skin color, the cheek is usually an area with heavier oiliness and heavier wrinkles, and the chin is usually an area with lighter oiliness and lighter wrinkles. In order to always be able to select skin areas that are not affected by factors such as light shadows and shooting angles, the present invention selects the forehead, left cheek, right cheek, and chin of the face as the areas of interest. When calculating the indicators of the four areas, the weight matrix shown in Table 1 below is set according to experience.

[0154] Table 1 Weight table of skin quality indexes in facial areas of interest

[0155] Forehead Left face Right face jaw color 0.35 0.25 0.25 0.15 Oily 0.4 0.2 0.2 0.1 wrinkle 0.2 0.3 0.3 0.2 Pores 0.2 0.3 0.3 0.2

[0156] The forehead, left cheek, right cheek, and chin of the face can be extracted as regions of interest in the following way:

[0157] The expression formula of facial key points is Loc i =(x i ,y i ),i=1,2,…,81,where x i ,y i They represent the horizontal and vertical coordinates of the points respectively, and the specific area representation is shown in Table 2 below.

[0158] Table 2 Areas corresponding to facial key points

[0159]

[0160]

[0161] In the face skin classification task, if the entire area is used as input, it will be disturbed by posture, shadow, etc. Therefore, it is proposed to divide it into four regions of interest (ROI), as shown in the schematic diagram Figure 2 As shown. Set Recti lx ,Recti ly ,Recti rx ,Rectiry , i=1,2,3,4 are the horizontal axis position of the key point where the upper left corner of the four ROI rectangular areas is located, the vertical axis position of the key point where the upper left corner is located, the horizontal axis position of the key point where the lower right corner is located, and the vertical axis position of the key point where the lower right corner is located. i=1,2,3,4 represent the forehead, left cheek, right cheek, and mandible respectively.

[0162] The key points of the upper left corner and lower right corner of the forehead area are: (Rect1 lx ,Rect1 ly )=(x 21 ,max(y 71 ,y 72 ,y 81 )),(Rect1 rx ,Rect1 ry )=(x 24 ,min(y 21 ,y 24 )).

[0163] The key points of the upper left corner and lower right corner of the left cheek area are: (Rect2 lx ,Rect2 ly )=(x 37 ,y 29 ), (Rect2 rx ,Rect2 ry )=(x 32 ,y 32 ).

[0164] The key points of the upper left corner and lower right corner of the right cheek area are: (Rect3 lx ,Rect3 ly )=(x 36 ,y 29 ), (Rect3 rx ,Rect3 ry )=(x 46 ,y 32 ).

[0165] The key points of the upper left and lower right corners of the mandibular area are: (Rect4 lx ,Rect4 ly )=(x8,max(y 57 ,y 58 ,y 59 )),(Rect4 rx ,Rect4 ry )=(x 10 ,min(y8,y9,y 10 )).

[0166] The schematic diagram of the four areas is as follows Figure 2 As shown in the inner rectangle of .

[0167] Specifically, the process of removing the gloss from the image B by the gloss operator in S2 is as follows:

[0168] S21: classifying the skin quality of image B to determine the oiliness level of image B;

[0169] The oil gloss levels are classified into level 4 oil gloss, level 3 oil gloss, level 2 oil gloss and level 1 oil gloss, and each oil gloss level is assigned a value in turn;

[0170] S22: Calculate the maximum chromaticity σ at each pixel of image B max , and store image B as grayscale image I;

[0171] S23: Calculate the maximum value λ of the approximate diffuse reflection chromaticity at each pixel of image B max , and store it as a grayscale image II;

[0172] For example: Generally, a color image has three layers, RGB, and each layer has a pixel value (0-255). For example, if the pixel value of a certain point is (1, 2, 5), then Save this value and treat it as a grayscale image (the grayscale image has only one layer, so this value can be regarded as the value of a pixel, that is, the image B is obtained above. Similarly, according to λ max Formula Still taking the above point as an example, we get λ max for Save this value as a pixel in the grayscale image.

[0173] S24: using the grayscale image II as a guide image, applying a joint bilateral filter to the grayscale image I, and storing the filtered image as a preprocessed image;

[0174] S25: Calculate σ for each pixel p in the preprocessed image max (p), compare and σ max , take the maximum value as shown in formula 2-17:

[0175]

[0176] S26: Repeat steps S24 and S25 until each pixel Then execute the next step;

[0177] S27: Determine σ for each pixel p in the preprocessed image max (p) is the selected RGB channel, and the pixel of the selected channel of each pixel point p is σ max(p)×255, and then iterate the pixels of the two unselected channels and the pixels of the selected channel for each pixel point p to obtain the preprocessed image; the preprocessed image used in the calculation here is a grayscale image, and the preprocessed image is the iteratively updated σ max (p)×255+pixels of the other two channels constitute a three-layer RGB color image. For example, the σ of pixel p max (p) selects the R channel, then the pixel of pixel p in the R channel is σ max (p)×255, and then compare the pixel p in the G channel and the B channel with the pixel σ in the R channel max (p)×255 is iterated, and the above operation is repeated for all pixel points p to obtain a three-layer RGB color image, i.e., the preprocessed image.

[0178] S27: Classify the skin quality of the preprocessed image to determine the oiliness level. When the oiliness level of the preprocessed image is lower than the oiliness level of image B in S21 and is not greater than the preset oiliness level threshold, the preprocessed image is output as image C. Otherwise, return to step S22 and update image B with the preprocessed image.

[0179] Specifically, the reflected light color J in the RGB color space is expressed as the diffuse reflection value J D and the specular reflection value J s The linear combination of colors is as follows:

[0180] J=J D +J s (2-5);

[0181] Define chromaticity as the color component σ c The score is as follows:

[0182]

[0183] Where c∈{r,g,b},J c Indicates the color of reflected light;

[0184] Diffuse reflection chromaticity Λ c and lighting chromaticity Γ c Define the following formula 2-7 and formula 2-8:

[0185]

[0186] in, Represents the diffuse component in layer c, Represents the specular reflection component in layer c;

[0187] According to the above formula, the reflected light color J c Defined as formula 2-9:

[0188]

[0189] Among them, u represents the layer, which can be r layer, g layer or b layer. Represents the diffuse component in layer u, Represents the specular reflection component in layer u;

[0190] Assume that the illumination chromaticity is estimated using white, and the input image B is normalized to and Γ r , Γ g and Γ b Represents the illumination chromaticity of the r, g, and b layers respectively, and Represents the specular reflection values ​​of the r, g, and b layers respectively;

[0191] Then according to the previous formula, the diffuse reflection component is as shown in formula 2-10:

[0192]

[0193] in, Represents the diffuse reflection value of the cth layer;

[0194] The maximum chromaticity is defined as Equation 2-11:

[0195] σ max =max(σ r , σ g , σ b ) (2-11);

[0196] Among them, σ r , σ g , σ b Represents the maximum color components of the r, g, and b layers respectively;

[0197] The maximum diffuse reflectance chromaticity is defined as Formula 2-12:

[0198] Λ max =max(Λ r , Λ g , Λ b ) (2-12);

[0199] Among them, Λ r , Λ g , Λ b Represents the maximum diffuse reflection chromaticity of the r, g and b layers respectively;

[0200] The diffuse component can be represented by Λ max Expressed as formula 2-13:

[0201]

[0202] Λ max The value range is

[0203] Specifically, the maximum value λ of the approximate diffuse reflection chromaticity at each pixel of image B is calculated in S23. max The process is as follows:

[0204] Let σ min =min(σ r , σ g , σ b ), using λ c To estimate Λ c , calculation formula 2-14 is as follows:

[0205]

[0206] λ c It is an intermediate variable and has no actual meaning;

[0207] Approximate diffuse reflection chromaticity λ c and the true diffuse chromaticity Λ c The relationship between them is described as 1) and 2).

[0208] 1) For any two pixels p and q, if Λ c (p) = Λ c (q), then λ c (p) = λ c (q)

[0209] 2) For any two pixels p and q, if λ c (p) = λ c (q), then only if Λ min (p) = Λ min (q), Λ c (p) = Λ c (q).

[0210] The maximum value of the approximate diffuse reflection chromaticity is given by formula 2-15:

[0211]

[0212] Among them, λ r ,λ g ,λ b The calculated variables representing the r, g, and b layers have no actual meaning;

[0213] Using the approximate diffuse reflection maximum chromaticity value as the smoothing parameter, the maximum chromaticity σ after filtering maxThe calculation formula 2-16 is as follows:

[0214]

[0215] in, It means that the calculation variable of pixel p has no actual meaning. and It is a typical Gaussian distribution with space and distance weighting function.

[0216] Specifically, the grayscale image II in S24 is used as a guide image, and a joint bilateral filter is applied to the grayscale image I.

[0217] The filtering process is as follows:

[0218]

[0219] Among them, I D (i, j) represents the pixel value of the pixel point with coordinates (i, j) after joint bilateral filtering, and (k, l) represents the pixel coordinates of other points in the filtering window. It represents the pixel value of the center point. It represents the pixel values ​​of the remaining nodes, and w(i,j,k,l) ​​is a parameter that is the product of the Gaussian distribution space function and the Gaussian function of pixel intensity similarity.

[0220] The joint bilateral filter is defined as follows:

[0221]

[0222] yes This part is only related to the coordinates of the pixels p(i, j) and q(k, l). Substitute into the formula, σ max (q) is equal to the I(k,l) part in the bilateral filter, which represents the pixel value at point q.

[0223] This joint bilateral filter is used in max This grayscale image is constantly iterated and updated (the algorithm is constantly iterating)

[0224] Finally, we find a σ that conforms to the algorithm process. max The grayscale image composed of values ​​is composed of σ max =max(σ r ,σ g ,σ b ) is defined as follows, we can see that σ maxThe value is a decimal between 0 and 1, and comes from the channel with the largest chromaticity value ratio in the RGB channel. For the entire image, the RGB pixel value of each point is changing, so for the entire image, it is equivalent to calculating the σ value of the RGB channel at the same time. After iteration, we get a max The grayscale image is composed of max Multiplying by 255 + the pixels of the remaining two channels corresponding to this pixel will give the image C after removing the highlights.

[0225] Take a face image as the effect test. The skin glossiness is graded as level 3, so the de-glossing parameter is set to 2. Figure 9 shows the original face image and the effect after being processed by the de-glossing operator. The left picture is the original image, and the right picture is the image after being processed by the de-glossing operator. As can be seen from Figure 9, the de-glossing operator implemented by the algorithm for removing specular highlights proposed in the present invention has obvious effects on removing skin reflection and gloss in the forehead and left cheek area.

[0226] Specifically, the process of whitening the image C by the whitening operator in S3 is as follows:

[0227] S31: classify the skin quality of image C. The skin color levels are divided into four categories: level 4, level 3, level 2, and level 1. Each skin color level is assigned a value of β=3, 2, 1, and 0 in sequence.

[0228] S32: Normalize the pixel values ​​of the R, G, and B channels of the image C layer by layer. The normalization method is:

[0229]

[0230] Where f(x,y) represents the pixel value at each pixel of the input image C, and w(x,y)∈[0,1] represents the output image C′. The output image C′ has three layers: R, G, and B.

[0231] S33: Enhance the input image C′ using the following formula, as shown in Formula 3-2:

[0232]

[0233] Where w(x, y) is the input image C′, v(x, y) is the output image D after whitening, and β is a parameter that controls the degree of whitening. When β is 1, 2, or 3, the effect of the algorithm on the brightness component is as follows: Fig.10 As shown. Map this parameter to the classification result of skin color. When the classification is level 4, the value of β is 3; when the classification is level 3, the value of β is 2; when the classification is level 2, the value of β is 1; when the classification is level 1, the value of β is 0 and is not included in the calculation.

[0234] Experiment and analysis on the effect of whitening operator:

[0235] The experimental results are divided into three parts. First, a sampling comparison is performed to show the difference between the before and after skin beautification pictures. Then, the effectiveness of the overall algorithm is verified by defining the algorithm effect rate. Finally, a blind test score by professionals is performed to verify whether the algorithm reaches the quality of professional portrait post-processing.

[0236] ① Sampling comparison

[0237] Take a woman's face and a man's face as display. Fig.11a and Fig.12a It's the original picture. Fig.11b and Figure 12b This is the processed image. It can be seen that the skin beautification model composed of operators such as skin resurfacing, whitening, and degreasing has an obvious processing effect on the image and greatly improves the appearance of skin texture.

[0238] ② Algorithm effect rate

[0239] The test set contains 145 faces in total. The comparison of the data volume of different index classifications before and after beautification is shown in Table 3. For each skin quality index, the number of first-level classifications after beautification / total number is defined to characterize the algorithm's action rate, as shown in Formula 4.18, which is the proportion of the algorithm achieving the skin beautification effect after acting on the current data set. The algorithm action rate can directly quantify the effect achieved by the model proposed in this invention, and whether it can meet the goals and requirements of automatic skin beautification and photo retouching.

[0240]

[0241] Among them, p is the algorithm's action rate, Count1 is the number of first-level classified images, and Count i It is the total number of images classified into the first, second, third, and fourth levels.

[0242] Table 3. Effectiveness of skin beautification algorithm in test set

[0243]

[0244]

[0245] As can be seen from Table 3, the whitening, degreasing and skin resurfacing algorithms have achieved 97.24%, 97.93%, 96.55% and 95.17% respectively for the four indicators of skin color, gloss, wrinkles and pores, indicating that the skin segmentation, skin quality classification and skin beautification algorithms based on deep learning proposed in the present invention have obvious skin beautification effects and are suitable for automatic skin beautification processing tasks in large-scale scenarios.

[0246] ③ Professional blind test

[0247] In order to further verify the effect of the overall algorithm of the present invention, professional photo retouchers were invited to conduct blind testing and scoring on the original images and the beautified images of this test set. Before scoring, all the labels of the images were removed, leaving only the shuffled digital numbers. After scoring, the scores were calculated based on the numbers.

[0248] The scoring criteria are skin beautification, texture preservation, skin whitening, and blemish removal. The scoring results are as follows Fig.13 shown.

[0249] like Fig.13 As shown in the figure, the skin quality of 145 faces is scored on a 10-point scale. The average score of the original image is 7.79, and the average score of the skin-beautified image is 9.09. Due to factors such as the photographer's level, lighting and shadows, and skin conditions, the original image shows a large fluctuation in scores and generally low scores. The scores of the skin-beautified images are high and evenly distributed, indicating that the algorithm accurately classifies skin quality, the parameters guiding the skin-beautification algorithm are unbiased, and the skin-beautification algorithm works well.

[0250] Based on the self-evaluation results and the blind test scores of professional photo retouchers, it is proved that the present invention has achieved obvious effects in skin resurfacing, texture retention, whitening effect, oily shine removal, etc.

[0251] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A skin beautification method for facial images based on additive lee filtering and skin resurfacing, characterized in that: The steps include: S1: Obtain the original face image, and use the additive lee filter skin smoothing operator to smooth the original face image to obtain image B; The process of performing skin smoothing processing on the original face image by the additive lee filter skin smoothing operator in S1 is as follows: S11: Cut the original face image into N*M size to get the image to be processed, and use x ij Represents the pixel value at the coordinate (i, j) of the image to be processed; S12: setting a sliding window of size (2*n+1)*(2*m+1), and calculating the local mean and local variance of the pixel values ​​of all coordinate points in the image to be processed within the sliding window; The method for calculating the local variance of all coordinate pixel values ​​in the image to be processed in the sliding window in S12 is: Among them, x kl represents the pixel value at the coordinate (k, l) in the sliding window, m ij Represents the local average value of the pixel values ​​of all coordinate points in the image to be processed within the sliding window, v ij Represents the local variance of all coordinate pixel values ​​in the image to be processed within the sliding window; S13: Determine the σ value based on the skin quality classification level of the original face image. Each skin quality level corresponds to a σ value. Then, based on the σ value and the local variance of the pixel values ​​of all coordinate points in the image to be processed within the sliding window obtained in S12, the k value is calculated using the following formula: Among them, σ represents the parameter of the additive lee filter, k represents the degree of skin smoothing of the original image, and v ij Represents the local variance of the image to be processed inside the sliding window; The method for classifying skin quality according to the original face image in S13 is: S131: define multiple feature points in the original face image, connect all the feature points in sequence to form a polygon, and the resulting mask is a complete face area, which is defined as M points , the skin area mask of the human body is M human , the mask image of the face skin area is M face : M face =M points ∩M human (1-5); S132: The mask image of the facial skin area is classified into two dimensions according to wrinkles and pores, as follows: Wrinkle levels are divided into four levels, three levels, two levels, and one level, and each wrinkle level is assigned a value of 3, 2, 1, and 0 in sequence; The pore levels are divided into four levels, three levels, two levels, and one level, and each pore level is assigned a value of 3, 2, 1, and 0 in sequence; S133: In the original face image, four parts of the face, namely the forehead, the left cheek, the right cheek and the chin, are selected as regions of interest, and the weights of each region divided into wrinkles and pores are set. Then, the grade assignments of the four parts are calculated using the following formula, and the value of σ is equal to the grade assignment: in, Represents the weights of wrinkles in the forehead, left cheek, right cheek and chin respectively. Represents the weights of pores in the forehead, left cheek, right cheek and chin respectively; S14: Additively denoise the pixel value at the coordinate (i, j) according to the following formula, and use the pixel value after additive denoising as the pixel value at coordinate (i, j); in, Represents the pixel value after additive denoising at coordinate (i, j); S15: Repeat S12-S14, and traverse all coordinate points in the image to be processed to obtain the pixel value of each coordinate point after additive denoising. The pixel values ​​corresponding to all coordinate points after additive denoising are used as the pixel values ​​of the pixel point after skin smoothing, so as to obtain image B; S2: Input image B into the deglare operator to remove the gloss and obtain image C; S3: Input image C into the whitening operator to obtain a final skin-beautified image D after whitening, and output image D.

2. The method for beautifying a facial image based on additive lee filtering and skin resurfacing as claimed in claim 1, characterized in that: The process of removing gloss from image B by the gloss operator in S2 is as follows: S21: classifying the skin quality of image B to determine the oiliness level of image B; The oil gloss levels are classified into level 4 oil gloss, level 3 oil gloss, level 2 oil gloss and level 1 oil gloss, and each oil gloss level is assigned a value in turn; S22: Calculate the maximum chromaticity σ at each pixel of image B max , and store image B as grayscale image I; S23: Calculate the maximum value λ of the approximate diffuse reflection chromaticity at each pixel of image B max , and store it as a grayscale image II; S24: using the grayscale image II as a guide image, applying a joint bilateral filter to the image grayscale image I, and storing the filtered image as a preprocessed image; S25: Calculate σ for each pixel p in the preprocessed image max (p), compare and σ max , take the maximum value as shown in formula 2-17: S26: Repeat steps S24 and S25 until each pixel Then execute the next step; S27: Determine σ for each pixel p in the preprocessed image max (p) is the selected RGB channel, and the pixel of the selected channel of each pixel point p is σ max (p)×255, and then iterate the pixels of the two unselected channels and the pixels of the selected channel for each pixel point p to obtain the preprocessed image; S27: Classify the skin quality of the preprocessed image to determine the oiliness level. When the oiliness level of the preprocessed image is lower than the oiliness level of image B in S21 and is not greater than the preset oiliness level threshold, the preprocessed image is output as image C. Otherwise, return to step S22 and update image B with the preprocessed image.

3. The method for beautifying a facial image based on additive lee filtering and skin resurfacing as claimed in claim 2, characterized in that: In S22, the maximum chromaticity σ at each pixel of the image B is calculated. max The process is as follows: The reflected light color J in the RGB color space is expressed as the diffuse reflection value J D and the specular reflection value J s The linear combination of colors is as follows: J=J D +J s (2-5); Define chromaticity as the color component σ c The score is as follows: Where c∈{r,g,b},J c Indicates the color of reflected light; Diffuse reflection chromaticity Λ c and lighting chromaticity Γ c Define the following formula 2-7 and formula 2-8: in, Represents the diffuse component in layer c, Represents the specular reflection component in layer c; According to the above formula, the reflected light color J c Defined as formula 2-9: Among them, u represents the layer, which can be r layer, g layer or b layer. Represents the diffuse component in layer u, Represents the specular reflection component in layer u; Assume that the illumination chromaticity is estimated using white, and the input image B is normalized to and Γ r , Γ g and Γ b Represents the illumination chromaticity of the r, g, and b layers respectively, and Represents the specular reflection values ​​of the r, g, and b layers respectively; Then according to the previous formula, the diffuse reflection component is as shown in formula 2-10: in, Represents the diffuse reflection value of the cth layer; The maximum chromaticity is defined as Equation 2-11: s max =max(σ r ,s g ,s b ) (2-11); Among them, σ r , σ g , σ b Represents the maximum color components of the r, g, and b layers respectively; The maximum diffuse reflectance chromaticity is defined as Formula 2-12: L max =max(Λ r ,L g ,L b ) (2-12); Among them, Λ r , Λ g , Λ b Represents the maximum diffuse reflection chromaticity of the r, g and b layers respectively; The diffuse component can be represented by Λ max Expressed as formula 2-13: Λ max The value range is 4. The method for beautifying a facial image based on additive lee filtering and skin resurfacing as claimed in claim 3, characterized in that: In S23, the maximum value λ of the approximate diffuse reflection chromaticity at each pixel of image B is calculated. max The process is as follows: Let σ min =min(σ r , σ g , σ b ), using λ c To estimate Λ c , calculation formula 2-14 is as follows: λ c It is an intermediate variable and has no actual meaning; Approximate diffuse reflection chromaticity λ c and the true diffuse chromaticity Λ c The relationship between them is described as 1) and 2); 1) For any two pixels p and q, if Λ c (p) = Λ c (q), then λ c (p) = λ c (q) 2) For any two pixels p and q, if λ c (p) = λ c (q), then only if Λ min (p) = Λ min (q), Λ c (p) = Λ c (q) The maximum value of the approximate diffuse reflection chromaticity is given by formula 2-15: Among them, λ r ,λ g ,λ b The calculated variables representing the r, g, and b layers have no actual meaning; Using the approximate diffuse reflection maximum chromaticity value as the smoothing parameter, the maximum chromaticity σ after filtering max The calculation formula 2-16 is as follows: in, It means that the calculation variable of pixel p has no actual meaning. and It is a typical Gaussian distribution with space and distance weighting function.

5. The method for beautifying a facial image based on additive lee filtering and skin resurfacing as claimed in claim 4, characterized in that: The grayscale image II in S24 is used as a guide image, and the process of applying a joint bilateral filter to the grayscale image I is as follows: Among them, I D (i, j) represents the pixel value of the pixel point with coordinates (i, j) after joint bilateral filtering, (k, l) represents the pixel coordinates of other points in the filtering window, It represents the pixel value of the center point. It represents the pixel values ​​of the remaining nodes, and w(i, j, k, l) is a parameter multiplied by the Gaussian distribution space function and the Gaussian function of pixel intensity similarity; The joint bilateral filter is defined as follows: yes This part is only related to the coordinates of the pixels p(i, j) and q(k, 1). Substitute into the formula, σ max (q) is equal to the I(k, l) part in the bilateral filter, which represents the pixel value at point q.

6. The method for beautifying facial images based on additive lee filtering and skin resurfacing as claimed in claim 1 or 5, characterized in that: The process of whitening the image C by the whitening operator in S3 is as follows: S31: classify the skin quality of image C. The skin color levels are divided into four categories: level 4, level 3, level 2, and level 1. Each skin color level is assigned a value of β=3, 2, 1, and 0 in sequence. S32: Normalize the pixel values ​​of the R, G, and B channels of the image C layer by layer. The normalization method is: Where f(x, y) represents the pixel value at each pixel point of the input image C, and w(x, y)∈[0, 1] represents the output image C′. The output image C′ has three layers: R, G, and B. S33: Enhance the input image C′ using the following formula, as shown in Formula 3-2: Among them, w(x, y) is the input image C′, v(x, y) is the output image D after whitening, and β is a parameter that controls the degree of whitening.

Images