Image processing method, device and computer readable storage medium

Abstract

The disclosed embodiments provide an image processing method, device, and computer-readable storage medium, the image processing method comprising: obtaining an input image containing a face and determining a skin area; detecting a face in the input image to obtain a face area, extracting key points of the face in the face area, and determining a reserved area and a non-reserved area of ​​the face area according to the key points; fusing the image of the reserved area, a first skin-removing image of the non-reserved area, and a second skin-removing image of the skin area to obtain an output image, wherein the skin-removing intensity of the second skin-removing image is less than the skin-removing intensity of the first skin-removing image. The disclosed embodiments distinguish between the reserved area and the non-reserved area of ​​the face area and perform differential processing on different areas of the face, so as to obtain a more natural beauty effect.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to an image processing method, device, and computer-readable storage medium. Background Art

[0002] In order to take satisfactory photos, it is usually necessary to use software with photo retouching functions. As the popularity of photo retouching software increases, people's requirements for the beauty function of photo retouching software are also getting higher and higher. They all hope that the beauty effect is closer to the real self, but better than the real effect.

[0003] However, the current beautification effects are not natural enough and are easily distorted. Summary of the invention

[0004] The embodiments of the present disclosure provide an image processing method, an apparatus, and a computer-readable storage medium to obtain a more natural beautification effect.

[0005] In one aspect, an embodiment of the present disclosure provides an image processing method, comprising:

[0006] Acquire an input image containing a human face and determine a skin area; detect the human face in the input image to obtain a human face area, extract key points of the human face in the human face area, and determine a retained area and a non-retained area of ​​the human face area according to the key points;

[0007] The image of the reserved area, the first skin resurfacing image of the non-reserved area and the second skin resurfacing image of the skin area are fused to obtain an output image, wherein the skin resurfacing intensity of the second skin resurfacing image is less than the skin resurfacing intensity of the first skin resurfacing image.

[0008] On the other hand, an embodiment of the present disclosure further provides a display device, including a processor and a memory storing a computer program that can be run on the processor, wherein the processor implements the steps of the above-mentioned image processing method when executing the program.

[0009] On the other hand, an embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program executable on a processor, wherein the computer program is used to implement the above-mentioned image processing method when executed by the processor.

[0010] The method provided in the embodiment of the present disclosure distinguishes between the reserved area, the non-reserved area and the skin area of ​​the face area, performs differentiated processing on different areas of the face, and performs differentiated skin resurfacing processing on the non-reserved area and the skin area while retaining the texture of the reserved area, thereby obtaining a more natural beautification effect.

[0011] Of course, it is not necessary to achieve all the advantages described above at the same time for any product or method of the present disclosure. Other features and advantages of the present disclosure will be described in the subsequent description embodiments, and partly become apparent from the description embodiments, or be understood by practicing the present disclosure. The purposes and other advantages of the embodiments of the present disclosure can be achieved and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to provide a further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure. The shapes and sizes of the components in the accompanying drawings do not reflect the actual proportions and are only intended to illustrate the contents of the present disclosure.

[0013] Figure 1 This is a flow chart of an image processing method according to an embodiment of the present disclosure;

[0014] Figure 2 This is a flow chart of another image processing method according to an embodiment of the present disclosure;

[0015] Figure 3a An example of an original input image according to an embodiment of the present disclosure;

[0016] Figure 3b for Figure 3a The mask image obtained after skin detection of the original input image;

[0017] Figure 3c Another example of an original input image according to an embodiment of the present disclosure;

[0018] Figure 3d for Figure 3c The mask image obtained after skin detection of the original input image;

[0019] Figure 4a Schematic diagram of facial key points and facial ROI in an embodiment of the present disclosure;

[0020] Figure 4b This is a schematic diagram of a reserved area mask according to an embodiment of the present disclosure;

[0021] Figure 4c A schematic diagram of a mask of a non-reserved area according to an embodiment of the present disclosure;

[0022] Figure 5a An example of an original input image according to an embodiment of the present disclosure;

[0023] Figure 5b for Figure 5a Grayscale image of the original input image in;

[0024] Figure 6a is an initial feature point map of the original input image of the embodiment of the present disclosure;

[0025] Figure 6b It is a feature point map after the feature points of the embodiment of the present disclosure are enhanced;

[0026] Figure 7a and Figure 7b A comparison diagram of the original input image and the image after curve brightening according to the embodiment of the present disclosure;

[0027] Figure 8a This is a picture showing the effect of high-contrast skin resurfacing in the embodiment of the present disclosure;

[0028] Figure 8b This is a picture showing the effect of bilateral filtering skin resurfacing according to the embodiment of the present disclosure;

[0029] Figure 8c This is a picture showing the effect of edge-preserving filtering and skin resurfacing according to the embodiment of the present disclosure;

[0030] Fig. 9 This is a schematic diagram of the sharpening process of the embodiment of the present disclosure;

[0031] Fig.10a This is a high-contrast skin smoothing image before sharpening in the embodiment of the present disclosure;

[0032] Fig.10b This is a high-contrast skin smoothing image after sharpening in the embodiment of the present disclosure;

[0033] Fig.11 For the color ring;

[0034] Fig.12a and 12b is a comparison diagram of the original input image and the image after color adjustment according to the embodiment of the present disclosure;

[0035] Fig.13a and 13b A comparison diagram of the original input image and the image processed by the embodiment of the present disclosure;

[0036] Fig.14 Schematic diagram of the structure of the image processing device according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure. Although many possible feature combinations are shown in the drawings and discussed in the embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0038] The present disclosure includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features, and elements disclosed in the present disclosure may also be combined with any conventional features or elements to form a unique invention scheme defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in the present disclosure may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other limitations. In addition, various modifications and changes may be made within the scope of protection of the attached claims.

[0039] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be appreciated by those of ordinary skill in the art, other orders of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art can easily understand that these orders can be changed and still remain within the spirit and scope of the disclosed embodiments.

[0040] Unless otherwise defined, the technical terms or scientific terms used in this disclosure are the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. In this disclosure, "plurality" can mean two or more numbers. "Include" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0041] In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of some known functions and known components. The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures can refer to the general design.

[0042] A common beautification operation is to operate on the entire image without distinguishing different areas, which results in a decrease in contrast and distortion of the entire image. Even when distinguishing between the face area and the background area, the boundary processing between the face area and the background area is also a problem. At the same time, there are different areas on the face. If the skin is smoothed on the eyes, eyebrows, lips and other areas, the eyes, eyebrows and lips will be lost, resulting in unnatural distortion. If the eye bags and nasolabial folds are not taken seriously, the skin will not be smoothed enough and the beautification effect will not be achieved.

[0043] The present disclosure provides an image processing method, such as Figure 1 As shown, the following steps are included:

[0044] Step 11, obtaining an input image containing a face and determining a skin area; detecting a face in the input image to obtain a face area, extracting key points of the face in the face area, and determining a retained area and a non-retained area of ​​the face area according to the key points;

[0045] The input image containing a human face may be, for example, an image extracted from a video frame.

[0046] For example, the reserved area includes one or more of the following areas: eye area, eyebrow area, lip area; the non-reserved area includes one or more of the following areas: nasolabial fold area, eyebag area.

[0047] In an exemplary embodiment, the obtaining of an input image containing a human face and determining a skin region may include: performing skin detection on the input image, combining the human face region, and determining a skin region in the human face region as the skin region. By combining skin detection and the human face region, the skin in the human face region can be accurately determined, avoiding the problem of mistaking the background for the skin that may occur in simple skin detection, and solving the problem of incoordination between face beautification and background.

[0048] Step 12, fusing the image of the reserved area, the first skin resurfacing image of the non-reserved area, and the second skin resurfacing image of the skin area to obtain an output image, wherein the skin resurfacing intensity of the second skin resurfacing image is less than the skin resurfacing intensity of the first skin resurfacing image.

[0049] Skin resurfacing is an image processing method that can remove spots or other defects in an image, making the face of a person more delicate. The intensity of skin resurfacing reflects the degree of defect removal and the delicacy of the face. The stronger the skin resurfacing intensity, the fewer defects and the more delicate the skin.

[0050] The disclosed embodiment proposes a method for beautifying the face by fine-tuning the partitioning. By distinguishing the reserved area, non-reserved area and skin area of ​​the face area, different areas of the face are processed differentially. While retaining the texture of the reserved area, the non-reserved area and skin area are subjected to differentiated skin resurfacing processing, thereby obtaining a more natural beautification effect.

[0051] In an exemplary embodiment, the step of fusing the image of the reserved area, the first skin resurfacing image of the non-reserved area, and the second skin resurfacing image of the skin area to obtain an output image includes:

[0052] Step 121, performing a first skin resurfacing process to obtain a first skin resurfacing image of the non-reserved area;

[0053] In an exemplary embodiment, the input image may be subjected to a first skin resurfacing process to obtain a first skin resurfacing process result image, and a first skin resurfacing image of the non-reserved area may be obtained according to a non-reserved area mask: the first skin resurfacing image = first skin resurfacing process result image * non-reserved area mask.

[0054] In the disclosed embodiment, the mask is a binary image (including 0 and 1). By multiplying the skin resurfacing result image with the mask, the purpose of retaining the image of the region of interest and shielding the region not involved in the processing can be achieved. For example, for a pixel A at a certain position, its pixel value in the first skin resurfacing result image is P1, but its value in the non-reserved area mask is 1, then the pixel value of the pixel A in the first skin resurfacing image obtained after multiplication is still P1, and for a pixel B at another position, its pixel value in the first skin resurfacing result image is P2, but its value in the non-reserved area mask is 0, then the pixel value of the pixel B in the first skin resurfacing image obtained after multiplication is 0. In step 121, the skin resurfacing result of the non-reserved area is obtained by multiplying the first skin resurfacing result image with the non-reserved area mask, that is, the first skin resurfacing image.

[0055] Step 122, performing a second skin resurfacing process to obtain a second skin resurfacing image of the area other than the non-retained area;

[0056] In an exemplary embodiment, the input image may be subjected to a second skin resurfacing process to obtain a second skin resurfacing process result image, and a second skin resurfacing image of other regions except the non-reserved region may be obtained according to the non-reserved region mask: the second skin resurfacing image = the second skin resurfacing process result image * (1-non-reserved region mask). The intensity of the second skin resurfacing process is less than the intensity of the first skin resurfacing process.

[0057] Step 123, fusing the first skin resurfacing image and the second skin resurfacing image to obtain a third skin resurfacing image;

[0058] That is, the third skin resurfacing image = the first skin resurfacing image + the second skin resurfacing image

[0059] Step 124: Based on the third resurfaced image, an output image is obtained according to the skin region mask and the reserved region mask.

[0060] For example, you can use one of the following methods:

[0061] Based on the third dermabrasion image, a first result image is obtained as an output image according to the skin area mask and the reserved area mask: first result image=third dermabrasion image*(skin area mask-reserved area mask); or

[0062] The third skin-removing image is sharpened to obtain a sharpened third skin-removing image; based on the sharpened third skin-removing image, a first result image is obtained as an output image according to the skin area mask and the reserved area mask: the first result image = the sharpened third skin-removing image * (skin area mask - reserved area mask). Image sharpening is used to compensate for the contour of the image, enhance the edge of the image and the grayscale jump part, and make the image clear. A clearer beauty effect can be obtained through sharpening.

[0063] In the output image obtained by the above method, the skin area in the face area has a light skin smoothing effect, the non-reserved area has a heavy skin smoothing effect, and the reserved area and the background area have no skin smoothing effect.

[0064] In an exemplary embodiment, after obtaining the first result image, the method may further include: performing color adjustment on the input image to obtain a color-adjusted image; obtaining a second result image according to the skin area mask and the reserved area mask: second result image = color-adjusted image * (1-skin area mask + reserved area mask);

[0065] The obtaining of the output image comprises: fusing the first result image and the second result image to obtain the output image: output image = first result image + second result image.

[0066] The color adjustment of the input image may, for example, reduce the cyan color of pixels in a red area, where the pixels in the red area are pixels with the largest red channel component value.

[0067] In this embodiment, by adjusting the color, for example, reducing the cyan in the input image, the red in the image can be increased. For the face, the redness of the lips can be increased to obtain a better beautification effect.

[0068] In an exemplary embodiment, the first microdermabrasion treatment is a high contrast microdermabrasion treatment.

[0069] In an exemplary embodiment, the second microdermabrasion treatment is a high contrast preservation microdermabrasion treatment.

[0070] Optionally, the first microdermabrasion treatment and the second microdermabrasion treatment may adopt the same or different microdermabrasion treatment methods.

[0071] In an exemplary embodiment, the high contrast preservation microdermabrasion treatment comprises:

[0072] Obtain a feature point map where grayscale mutation occurs in the input image;

[0073] Performing multiple feature point enhancement processing on the feature point map to obtain a high contrast retention mask, and controlling the intensity of the skin resurfacing processing by the number of feature point enhancement processing;

[0074] A skin resurfacing image is obtained by calculation according to the high contrast preserving mask.

[0075] Wherein: skin smoothing image = input image * high contrast preservation mask.

[0076] When obtaining a feature point map where the grayscale of an input image suddenly changes, the input image may be firstly smoothed and filtered, and then the grayscale map after smoothing filtering is subtracted from the grayscale map of the input image to obtain the feature point map where the grayscale suddenly changes.

[0077] Feature point enhancement processing is used to increase the contrast of image pixels. In layman's terms, it makes the bright areas in the image brighter and the dark areas darker through one or more function operations.

[0078] The high-contrast-preserving skin smoothing process adopted in this embodiment can achieve a more stunning effect in terms of ensuring the skin smoothing effect and detail fidelity.

[0079] In an exemplary embodiment, after performing multiple feature point enhancement processes, the method further includes: scaling the feature map after the feature point enhancement process to obtain a high contrast retention mask, and controlling the intensity of the skin resurfacing process by the scaling intensity. The skin resurfacing intensity can be further controlled by image scaling.

[0080] In an exemplary embodiment, before calculating and obtaining the skin resurfacing image according to the high contrast preserving mask, the method further comprises: brightening the input image to obtain a brightened image;

[0081] The step of calculating the skin resurfacing image according to the high contrast retention mask comprises: calculating the skin resurfacing image according to the high contrast retention mask and the brightening image:

[0082] Skin smoothing image = input image * high contrast retention mask + brightening image * (1-high contrast retention mask).

[0083] In this embodiment, a whitening effect can be achieved by brightening the image, which can be performed by using a curve adjustment method or by using OpenCV (a computer vision library) to process the brightness and contrast of the image.

[0084] The disclosed embodiment performs a partitioning and refinement operation on the input image. Combining skin detection and face ROI to distinguish the face area and the background area, solve the boundary processing problem of the face area and the background area, and avoid background distortion. In addition, different areas of the face are processed, and the reserved areas (such as the eyes, eyebrows, lips, etc.) are extracted by face key points. The reserved areas are not subjected to skin grinding, which can preserve the eyes, the texture of the eyebrows, the texture of the lips, etc. The non-reserved areas of the face (such as the eye bags area and the nasolabial folds area) are severely ground to achieve a better beauty effect. Using color adjustment can increase the redness of the lips and further enhance the skin beauty effect. Compared with other beauty methods, the high-contrast skin retention grinding and regional processing used in the disclosed embodiment have amazing effects in ensuring the beauty effect and texture clarity of the output image.

[0085] The following is an example of an application to illustrate the present disclosure. Figure 2 As shown, the implementation steps include:

[0086] Step 21, obtaining an image containing a face (hereinafter referred to as an input image, or an original image), performing face detection on the input image, and determining a face region or a face ROI (region of interest);

[0087] For example, face ROI can be obtained through face recognition, such as Figure 4a The area shown in the square frame.

[0088] Step 22, performing skin detection on the input image to determine a skin mask;

[0089] Optionally, the intersection of the face area and the skin detection result can be taken as a skin mask.

[0090] The skin detection algorithm is described below.

[0091] Skin color detection technology is the process of analyzing and calculating human skin color pixels. Skin color detection technology includes methods based on color space, methods based on spectral features, and methods based on skin color reflectance models. The main steps of these methods are to transform the color space first and then establish a skin color model. The color spaces used in skin color detection include RGB, YCrCb, HSV, and Lab. When processing, the RGB color space is transformed into the corresponding color space. The most widely used methods are based on skin color clustering or threshold segmentation. The more commonly used ones are YCbCr, HSV, RGB, or LAB color space threshold segmentation.

[0092] Take the YCbCr color space as an example. The YCbCr color space is a commonly used color model for skin color detection, where Y represents brightness, Cr represents the red component of the light source, and Cb represents the blue component of the light source. The color difference in the appearance of human skin color is caused by chromaticity, and the skin color distribution of different people is concentrated in a smaller area. Using the Otsu (Otsu method - maximum inter-class variance method) algorithm to perform binary segmentation on the Cr component, the clustering result of skin color can be accurately obtained.

[0093] The Otsu algorithm uses the idea of ​​clustering, dividing the grayscale of the image into two parts according to the grayscale level, so that the grayscale value difference between the two parts is the largest, and the grayscale difference between each part is the smallest, and a suitable grayscale level is found for division by calculating the variance. Therefore, the Otsu algorithm can be used to automatically select the threshold for binarization during binarization. The Otsu algorithm is considered to be the best algorithm for threshold selection in image segmentation. It is simple to calculate and is not affected by image brightness and contrast. Therefore, the segmentation that maximizes the inter-class variance means the minimum probability of misclassification.

[0094] Figure 3 shows a schematic diagram of the skin detection Mask. Figure 3a For the original Figure 1 , Figure 3b For the original Figure 1 The mask image obtained after skin detection, Figure 3c For the original Figure 2 , Figure 3d For the original Figure 2 After skin detection, a mask image is obtained. Subsequently, the mask image is combined with the face area to obtain the skin mask in the face area.

[0095] By taking the intersection of the face area and the area obtained by skin detection, an accurate skin area mask can be obtained.

[0096] Step 23, extract key points from the face area, and determine the mask of the reserved area and the mask of the non-reserved area according to the key points;

[0097] The reserved area includes, for example, but is not limited to, one or more of the following areas: an eyebrow area, an eye area, and a lip area.

[0098] The non-reserved area includes, for example, but is not limited to, one or more of the following areas: nasolabial folds area, eye bag area.

[0099] For example, the required area can be obtained by connecting key points.

[0100] This embodiment uses the open source facial feature recognition library - Dlib to implement face detection. Dlib uses machine learning methods to train the neural network architecture by annotating images of facial key points (eyebrows, eyes, nose, lips, and jaw), and integrates and encapsulates the trained model to form a recognition library. The latest Dlib library can generate 81 feature points, numbered 0-80. Figure 4a are the key points of the face detected and marked by Dlib. The box in the figure is the face ROI area, and the skin within the area is the beautification object. By connecting the key points of the face, the reserved area mask and the non-reserved area mask can be extracted at the same time. In this example, the reserved area mask is the mask of the eyes, lips, and eyebrows of the face, such as Figure 4b As shown in the figure, the non-reserved area mask is the mask of nasolabial folds and eye bags. Figure 4c shown.

[0101] Step 24, performing two high-contrast-preserving skin grinding processes with different parameters on the input image, including: performing a first high-contrast-preserving skin grinding process on the input image to obtain a first high-contrast skin grinding image (i.e., a first skin grinding image), performing a second high-contrast-preserving skin grinding process on the input image to obtain a second high-contrast skin grinding image (i.e., a second skin grinding image), the effect of the first high-contrast skin grinding process is stronger than that of the second high-contrast-preserving skin grinding process, and the skin grinding intensity is higher, the first high-contrast-preserving skin grinding process can be regarded as severe skin grinding, and the second high-contrast-preserving skin grinding process can be regarded as mild skin grinding;

[0102] The high contrast preserving microdermabrasion process used in this embodiment is described below.

[0103] Usually, people may have acne marks and some dark and rough skin on their faces. The purpose of beautification is to make the acne marks disappear as much as possible, make the dark skin whiter, and make the rough skin smoother, so as to achieve the beautification effect. To this end, it is necessary to find the feature points on the face that need to be processed, such as acne marks, dark skin, and rough skin, and process these feature points to achieve beautification.

[0104] By careful observation, we can find that the acne marks and dark skin are different from the surrounding normal skin in skin color. From the perspective of the image, the grayscale value of these acne marks or dark skin is low, while the grayscale value of normal skin is high, and the grayscale is a convex change process, that is, from normal skin to the feature point, the grayscale value will drop sharply from the high point. Similarly, local rough skin will also have a grayscale value mutation phenomenon, as shown in Figure 5. Figure 5a is the original input image, Figure 5b The grayscale image of the original input image, from Figure 5b It can be seen that there will be a grayscale mutation at the location of the acne mark.

[0105] To this end, it is necessary to extract the positions of these feature points. The input image can be smoothed and filtered to obtain a filtered image. For example, averaging filtering, Gaussian filtering, bilateral filtering or guided filtering can be used to fill in the positions of these mutations first, so as to extract feature points later. This embodiment uses Gaussian filtering.

[0106] For all pixel positions, the filtered grayscale value is subtracted from the original grayscale value. For feature points, since the filtered grayscale value is higher than the original grayscale value, the grayscale value of the feature point is less than 0, while the grayscale value of the normal skin position is positive.

[0107] In order to facilitate subsequent processing, all results are translated, for example, uniformly adding 0.5, that is:

[0108] Feature point position = original image grayscale value - filtered grayscale value + 0.5 (1)

[0109] Therefore, the value of the feature point position is less than 0.5, while the value of normal skin is greater than 0.5. Figure 6a The figure shows the feature point diagram where the grayscale changes suddenly, which shows the location of the feature points. The black points are the locations of the feature points, that is, the targets that need to be extracted.

[0110] Depend on Figure 6a You can see that the whole picture is gray. In order to make the feature points more obvious, you can enhance the feature points to make the features more obvious. The high-contrast Mask obtained after enhancement is as follows: Figure 6b shown.

[0111] Feature point enhancement is used to enhance feature point information so that the feature points to be extracted are different from normal skin. For example, the feature point image can be subjected to the following strong light processing 3-5 times (the effect is similar to a dazzling spotlight shining on an image, so it is called strong light processing): for a value x1 less than 0.5, the enhanced new value x1'=x1*x1*2, for a value x2 greater than 0.5, the enhanced new value x2'=1-(1-x2)*(1-x2)*2. Repeat this 3-5 times. The above processing function is only an example, and other functions may be used in other embodiments, and the present disclosure does not limit this. The intensity of the dermabrasion can be controlled by the number of feature point enhancement processing. For example, for severe dermabrasion, 5 strong light processings can be performed, and for mild dermabrasion, 3 strong light processings can be performed. The more enhancements are performed, the stronger the dermabrasion effect.

[0112] Optionally, all values ​​x of the feature map may be scaled at once, and the enhancement strength of the feature map may be adjusted by adjusting the scaling strength, thereby achieving control of the skin resurfacing strength. For example, any of the following scaling methods may be used:

[0113] x new = kx (2)

[0114] Among them, k∈(0,1), the larger the k value, the stronger the skin resurfacing effect.

[0115]

[0116] Among them, c1, c2∈[0, 1] are both scaling factors, c2>c1, and controlling c1 and c2 can adjust the scaling strength.

[0117] After obtaining the high contrast Mask, the feature points can be brightened to increase their grayscale values, making them as white as possible compared to the original color, the same as normal skin color. This embodiment uses a curve adjustment method to brighten and whiten the original image as a whole, as shown in Figure 7. Figure 7a For the original picture, Figure 7b The image is brightened by the curve. The curve adjustment method mainly adjusts the grayscale histogram of the image. Set an anchor point, interpolate according to the anchor point, get the curve, and adjust the grayscale histogram according to the shape of the curve. Setting a suitable anchor point can increase the overall brightness of the image, that is, achieve the effect of overall brightening the image.

[0118] After the curve is brightened, the original image and the brightened image are superimposed using a high-contrast mask to obtain a smoothed image:

[0119] High-contrast smoothing image = original image * high-contrast mask + brightening image * (1 – high-contrast mask) (3)

[0120] The above high-contrast preserving skin grinding method is used to perform skin grinding processing on the original image twice, and a first high-contrast skin grinding image and a second high-contrast skin grinding image are obtained respectively. The two skin grinding intensities are different. In this example, the first high-contrast skin grinding is severe skin grinding, and the second high-contrast skin grinding is mild skin grinding.

[0121] If one skin resurfacing intensity is used, if the skin resurfacing intensity is not enough, a good skin resurfacing effect may not be obtained in the nasolabial folds and / or eye bags areas. If the skin resurfacing intensity is strong, the effect is better in the nasolabial folds and / or eye bags areas, but other areas may be distorted due to excessive skin resurfacing. Therefore, in the embodiment of the present disclosure, heavy skin resurfacing is performed on areas with heavier textures such as nasolabial folds and / or eye bags, while light skin resurfacing is used for other parts of the human facial skin, so as to obtain a better skin resurfacing effect while ensuring no distortion.

[0122] In addition to high-contrast skin resurfacing, you can also use bilateral filtering or edge-preserving filtering. Figure 8a This is the effect picture after high-contrast skin resurfacing. Figure 8b This is the effect of bilateral filtering after skin resurfacing. Figure 8c This is the effect of edge-preserving filter resurfacing. In comparison, we can see that the high-contrast resurfacing has better resurfacing effect and detail fidelity. Figure 8a The microdermabrasion strength is stronger than Figure 8b and Figure 8c The intensity of microdermabrasion.

[0123] Optionally, the two microdermabrasion treatments may employ different microdermabrasion methods.

[0124] Step 25, fusing the two skin-grinding images according to the non-reserved area mask to obtain a high-contrast skin-grinding image of the input image (i.e., the third skin-grinding image);

[0125] A first high-contrast skin smoothing image of the non-reserved area is obtained according to the non-reserved area mask, and a second high-contrast skin smoothing image of the remaining areas except the non-reserved area is obtained, and the first high-contrast skin smoothing image of the non-reserved area and the second high-contrast skin smoothing image of the remaining areas except the non-reserved area are fused to obtain a high-contrast skin smoothing image of the input image.

[0126] High-contrast skin smoothing image = the first high-contrast skin smoothing image * non-reserved area (such as nasolabial folds, eye bags area) Mask + the second high-contrast skin smoothing image * (1 – non-reserved area Mask)

[0127] Step 26, sharpening the high-contrast skin smoothing image of the input image to obtain a sharpened high-contrast skin smoothing image, thereby improving the clarity of the image after skin smoothing;

[0128] Sharpening is to use the original image to sharpen the mask after blurring. The sharpening process includes: Gaussian filtering or other low-pass filtering of the input image to obtain a blurred image, subtracting the original image from the blurred image to obtain an edge image, and linearly combining the edge image and the original image to obtain the sharpened image. Fig. 9 As shown in the figure, x(n,m) represents the original image, the filter in the figure is a linear high-pass filter (Linear HP Filter), the filter output z(n,m) is the edge image obtained by subtracting the high-pass filter from the original image, and y(n,m) is the sharpened image after the original image and the edge image are combined. Figure 10 shows the comparison of the high-contrast skin-retaining smoothing before and after sharpening. Fig.10a This is a high-contrast skin smoothing image before sharpening. Fig.10b This is a high-contrast skin smoothing image after sharpening. It can be seen that the clarity of the image is greatly improved after sharpening.

[0129] This step may be an optional step.

[0130] Step 27, performing color adjustment (e.g., red adjustment) on the input image to obtain a color-adjusted input image, wherein the redness can be increased by color adjustment;

[0131] Selective Color or color adjustment is a simulation of the printing process, in which the amount of ink in a certain color area of ​​the image is increased or decreased, thereby changing a certain color.

[0132] The Selective Color command can be used to selectively and subtly modify the ink content of any primary color without affecting other primary colors. The main colors that can be adjusted by Selective Color Adjustment include: R (red), G (green), B (blue), C (cyan), M (magenta, also known as magenta), Y (yellow), white, black and neutral colors (colors other than pure black and white).

[0133] When the adjustment component is one of RGB, the main color is defined as the component with the largest value among the three RGB channels, and only the pixel position where the main color is consistent with the adjustment component can be adjusted. When the adjustment component is one of CMY, the main color is defined as the component with the smallest value among the three RGB channels, because CMY and RGB are complementary, and the channel with the smallest component in RGB is also the channel with the largest component in CMY channels. Only the pixel position where the channel with the smallest component is consistent with the adjustment component can be adjusted. When the adjustment component is white, the adjustable position is the position where the RGB values ​​are all greater than 128. When the adjustment component is black, the adjustable position is the position where the RGB component values ​​are all less than 128. When the adjustment component is a neutral color, except for the positions of pure black and pure white, all other pixels will be adjusted.

[0134] The specific implementation process of color adjustment includes:

[0135] (1) Parameter adjustment principle Follow the principle of complementary colors to adjust the slider. For example, reducing cyan is equivalent to increasing its complementary color red; reducing magenta is equivalent to increasing its complementary color green; reducing yellow is equivalent to increasing its complementary color blue; reducing black is equivalent to brightening; increasing black is equivalent to darkening; the relationship between CMY and RGB can be seen in Fig.11 Color ring shown;

[0136] (2) Determine the main color of the parameter adjustment: the adjustable main color can be RGB, CMY, or black and white. The difference is that the value of the calculated variable range is different;

[0137] (3) Determine the maximum, middle, and minimum values ​​of the adjustable primary color for each pixel:

[0138] Maximum value max=max(R,G,B)

[0139] Mid value mid = mid (R, G, B)

[0140] Minimum value min = min (R, G, B)

[0141] - If max>mid>min, the main color can be adjusted to the color with the value of max, or a mixture of max and mid.

[0142] -If max=mid>min, the main color can be adjusted to a mixture of max and mid

[0143] - If mid = min, the main color can be adjusted to the color with the value of max

[0144] - If the adjustable primary color is one of RGB, the adjustable range is range = max-mid

[0145] - If the adjustable primary color is one of CMY, the adjustable range is range = mid-min

[0146] (4) Calculate CMY = 1-RGB

[0147] Convert the original image from RGB to CMY

[0148] (5) According to the sliders that need to be adjusted (for example, three or four sliders can be set. If there are three sliders, they can be sliders for the three colors of CMY respectively. If there are four sliders, the fourth slider in addition to CMY can adjust the three colors of CMY at the same time), calculate the adjustable ratio of each slider, where the maximum adjustable ratio of the slider for adjusting the C color is pmax=(c), the maximum adjustable ratio of the slider for adjusting the M color is pmax=(m), and the maximum adjustable ratio of the slider for adjusting the Y color is pmax=(y), where c, m, y∈(0,1)

[0149] -When the adjustment direction is negative, that is, reducing the proportion x of one color in cmy, the adjustment proportion is min(pmax,x)

[0150] New slider value = original value - range × min (pmax, x)

[0151] -When the adjustment direction is positive, the ratio x of one color in cmy is increased, and the adjustment ratio is min(1-pmax,x)

[0152] New slider value = original value + range × min (1-pmax, x)

[0153] (6) Convert back to RGB = 1-CMY

[0154] Convert the adjusted CMY image back to RGB image.

[0155] For example, when adjusting the skin, the color component that can be adjusted is red. Then, only the pixel positions with the largest red channel component value can be adjusted, and the values ​​of the remaining pixels remain unchanged. That is, the cyan of the red area pixels can be reduced. The red area pixels are the pixels with the largest red channel component value.

[0156] Fig.12a For the original picture, Figure 12b This is a schematic diagram of the input image after red adjustment. The adjustment intensity is to reduce cyan by 100%. It can be seen that the lip area becomes significantly redder.

[0157] This step may be an optional step.

[0158] Step 28, obtaining a sharpened high-contrast skin grinding result image of other skin areas (such as forehead, cheek, nose, chin, etc.) in all skin areas except the reserved area according to the skin mask, the reserved area mask and the sharpened high-contrast skin grinding image, hereinafter referred to as the first result image;

[0159] The first result image = sharpening (high contrast skin smoothing image) * (skin mask – reserved area mask)

[0160] Among them, sharpening (high contrast skin smoothing image) refers to the image after the high contrast skin smoothing image is sharpened; (skin mask-reserved area mask) refers to all skin areas except the reserved areas (such as eyes, eyebrows and lips).

[0161] Step 29, obtaining a color-adjusted image of an area in the input image except the other skin areas according to the skin mask, the reserved area mask and the color-adjusted input image, hereinafter referred to as a second result image;

[0162] Second result image = color adjustment (original image) * (1-skin mask + reserved area mask)

[0163] Among them, color adjustment (original image) means the image after color adjustment of the original image; (1-skin Mask+reserved area Mask) is equivalent to "1-(skin Mask-reserved area Mask)".

[0164] The objects that can be adjusted in the second result map mainly include the lips, and possibly the background area (depending on the color of the background).

[0165] Step 30: fuse the first result image and the second result image to obtain an output image.

[0166] Output image = first result image + second result image

[0167] After the above steps, the image after skin beautification can be obtained. FIG13 is a comparison diagram of the beautification effect of the embodiment of the present disclosure, wherein Fig.13a For the original picture, Fig.13b This is a diagram showing the effect of image processing using the disclosed embodiment.

[0168] The disclosed embodiment proposes a method for beautifying the skin of a face by partitioning and refining, which solves the problem of incoordination between face beautification and background from two aspects: face detection and skin detection. In terms of face beautification, different areas of the face are processed differently. Amazing results have been achieved in ensuring the texture and clarity of each part of the face. Skin detection and face area are combined to distinguish the skin area and background area of ​​the face, solve the boundary processing problem of the face area and the background area, and avoid background distortion. Different areas of the face are processed differently, and the eyes, eyebrows, lips and other areas are extracted through the key points of the face. No skin grinding is performed, and the eyes, eyebrows and lips are retained. The texture of the eyes, eyebrows and lips are retained. Heavy skin grinding is performed on the eye bags and nasolabial folds of the face to achieve better beauty effects. The skin beautification effect can be further enhanced by using color adjustment to quickly adjust the redness of the lips, etc.

[0169] In an exemplary embodiment, the present disclosure further provides an image processing device, which may include a processor and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the steps of the image processing method in any of the above embodiments of the present disclosure are implemented.

[0170] In an exemplary embodiment, Fig.14 FIG. 1 is a schematic diagram of the structure of a display device in an embodiment of the present disclosure. Fig.14As shown, the device 60 includes: at least one processor 601; and at least one memory 602 and a bus 603 connected to the processor 601; wherein the processor 601 and the memory 602 communicate with each other via the bus 603; the processor 601 is used to call the program instructions in the memory 602 to execute the steps of the image processing method in any of the above embodiments.

[0171] The processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a transistor logic device, etc., and the present disclosure does not limit this.

[0172] The memory may include a read-only memory (ROM) and a random access memory (RAM), and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0173] In addition to the data bus, the bus may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.14 In the text file, various buses are marked as buses.

[0174] In the implementation process, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the processor or the instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0175] In an exemplary embodiment, the present disclosure further provides a non-volatile computer-readable storage medium on which is stored a computer program executable on a processor, wherein the computer program implements the steps of the aforementioned image processing method when executed by the processor.

[0176] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the functional modules / units mentioned in the above description are not divided equally; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0177] Although the embodiments disclosed in the present disclosure are as above, the contents described are only embodiments adopted to facilitate understanding of the present disclosure and are not intended to limit the present disclosure. Any technician in the field to which the present disclosure belongs can make any modifications and changes in the form and details of implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be subject to the scope defined in the attached claims.

Claims

1. An image processing method, characterized in that: include: Acquire an input image containing a human face and determine a skin area; detect the human face in the input image to obtain a human face area, extract key points of the human face in the human face area, and determine a retained area and a non-retained area of ​​the human face area according to the key points; fusing the image of the reserved area, the first skin resurfacing image of the non-reserved area, and the second skin resurfacing image of the skin area to obtain an output image, wherein the skin resurfacing intensity of the second skin resurfacing image is less than the skin resurfacing intensity of the first skin resurfacing image; The step of fusing the image of the reserved area, the first skin resurfacing image of the non-reserved area, and the second skin resurfacing image of the skin area to obtain an output image includes: Performing a first skin resurfacing process to obtain a first skin resurfacing image of the non-reserved area includes: performing a first skin resurfacing process on the input image to obtain a first skin resurfacing process result, and obtaining the first skin resurfacing image of the non-reserved area according to a non-reserved area mask: the first skin resurfacing image = first skin resurfacing process result image * non-reserved area mask; Performing a second skin resurfacing process to obtain a second skin resurfacing image of other areas except the non-reserved area, comprising: performing a second skin resurfacing process on the input image to obtain a second skin resurfacing result image, and obtaining a second skin resurfacing image of other areas except the non-reserved area according to a non-reserved area mask: the second skin resurfacing image = the second skin resurfacing result image * (1-non-reserved area mask); fusing the first skin resurfacing image and the second skin resurfacing image to obtain a third skin resurfacing image; Based on the third dermabrasion image, according to the skin area mask and the reserved area mask, an output image is obtained, including: based on the third dermabrasion image, according to the skin area mask and the reserved area mask, a first result image is obtained as the output image: the first result image = the third dermabrasion image * (skin area mask - reserved area mask).

2. The method according to claim 1, characterized in that Also includes: Performing a sharpening process on the third skin-removing image to obtain a sharpened third skin-removing image; The step of obtaining an output image based on the third resurfacing image and according to the skin region mask and the reserved region mask comprises: Based on the sharpened third resurfaced image, a first result image is obtained according to the skin area mask and the reserved area mask as an output image: The first result image = the third skin resurfacing image after sharpening*(skin area mask-retained area mask).

3. The method according to claim 1 or 2, characterized in that: Also includes: Performing color adjustment on the input image to obtain a color-adjusted image; According to the skin area mask and the reserved area mask, the second result image is obtained: Second result image = color-adjusted image * (1-skin area mask + reserved area mask); The obtaining of the output image comprises: fusing the first result image and the second result image to obtain the output image.

4. The method according to claim 3, characterized in that The step of adjusting the color of the input image comprises: Reduce the cyan color of the red area pixels, where the red area pixels are the pixels with the largest red channel component value.

5. The method according to claim 1, characterized in that: The step of obtaining an input image containing a human face and determining a skin area comprises: Skin detection is performed on the input image, and a skin area in the face area is determined as the skin area in combination with the face area.

6. The method according to claim 1, characterized in that The reserved area includes one or more of the following areas: eye area, eyebrow area, lip area; The non-reserved area includes one or more of the following areas: nasolabial fold area, eye bag area.

7. The method according to claim 1, characterized in that The first dermabrasion treatment is a high-contrast preservation dermabrasion treatment, and the second dermabrasion treatment is a high-contrast preservation dermabrasion treatment.

8. The method according to claim 7, wherein the high contrast preservation microdermabrasion treatment comprises: Obtain a feature point map where grayscale mutation occurs in the input image; Performing multiple feature point enhancement processing on the feature point map to obtain a high contrast retention mask, and controlling the intensity of the skin resurfacing processing by the number of feature point enhancement processing; A skin resurfacing image is obtained by calculation according to the high contrast preserving mask.

9. The method according to claim 8, characterized in that After performing multiple feature point enhancement processes, the method further includes: scaling the feature map after the feature point enhancement process to obtain a high-contrast retention mask, and controlling the intensity of the skin resurfacing process by the scaling intensity.

10. The method according to claim 8, characterized in that The invention also includes: brightening the input image to obtain a brightened image; The step of calculating the skin resurfacing image according to the high contrast retention mask comprises: calculating the skin resurfacing image according to the high contrast retention mask and the brightening image: Skin smoothing image = input image * high contrast retention mask + brightening image * (1-high contrast retention mask).

11. An image processing device, comprising a processor and a memory storing a computer program executable on the processor, wherein: When the processor executes the program, the steps of the image processing method according to any one of claims 1 to 10 are implemented.

12. A computer-readable storage medium, characterized in that: A computer program that can be run on a processor is stored thereon, and when the computer program is executed by the processor, the steps of the image processing method according to any one of claims 1 to 10 are implemented.

Images