Portrait image processing method and apparatus, electronic device, and storage medium

Abstract

The present application relates to a kind of portrait image processing method, device, electronic equipment and storage medium.The method includes: determining the skin region image in the portrait image to be processed, and the skin flat area, spot area and portrait structure area contained in the high-frequency component image and the low-frequency component image of skin region image;In the case of performing protection processing to the portrait structure area in high-frequency component image and low-frequency component image, respectively, skin flat area is executed in high-frequency component image and low-frequency component image and spot area is executed to perform spot-removing treatment, to obtain the high-frequency output image and low-frequency component image after skin-smoothing and spot-removing;Superimpose the high-frequency output image and low-frequency component image after skin-smoothing and spot-removing, to obtain skin region output image.Using the present method can realize skin-smoothing and spot-removing in portrait image while presenting realistic skin texture and protecting clear portrait structure.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a portrait image processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] Photo and video beautification is widely used in mobile phones and other photo and video recording devices. In the process of beautifying the photographed portrait image, a skin smoothing and spot removing algorithm can be used to perform skin smoothing and spot removing processing on the skin area of the portrait image, so that the skin has a better visual effect. An ideal skin smoothing and spot removing algorithm requires not only to remove the defects (such as freckles, acne marks, and pigment block deposition) of the skin of the portrait, but also to present a delicate and realistic skin texture, and cannot lose the clarity of the portrait outline.

[0003] However, most of the existing skin smoothing and spot removing algorithms lose details to a large extent in the process of skin smoothing and spot removing of the portrait skin area, resulting in an unnatural effect. How to naturally and authentically beautify the skin of the portrait has become an urgent need. SUMMARY

[0004] Therefore, it is necessary to provide a portrait image processing method, device, electronic equipment and storage medium capable of performing skin smoothing and spot removing while preserving the real texture of the portrait image.

[0005] A portrait image processing method includes: determining a skin area image in a to-be-processed portrait image, and a skin flat area, a spot area and a portrait structure area contained in a high-frequency component image and a low-frequency component image of the skin area image; performing skin smoothing processing on the skin flat area and spot removing processing on the spot area in the high-frequency component image and the low-frequency component image respectively under the condition that protection processing is performed on the portrait structure area in the high-frequency component image and the low-frequency component image, thereby obtaining a high-frequency output image after skin smoothing and spot removing and a low-frequency output image after skin smoothing and spot removing; and superimposing the high-frequency output image after skin smoothing and spot removing and the low-frequency output image after skin smoothing and spot removing to obtain a skin area output image.

[0006] In one embodiment, the determination of the skin area image in the to-be-processed portrait image, and the skin flat area, the spot area and the portrait structure area contained in the high-frequency component image and the low-frequency component image of the skin area image includes: identifying and extracting a skin area image from the to-be-processed portrait image; segmenting the skin area image into a high-frequency component image and a low-frequency component image; and identifying the skin flat area, the spot area and the portrait structure area contained in the high-frequency component image and the low-frequency component image respectively.

[0007] In one embodiment, the portrait image processing method further comprises: extracting a non-skin region image from the portrait image to be processed; and splicing the non-skin region image and the skin region output image to obtain a portrait output image.

[0008] In one embodiment, the dividing the skin region image into a high-frequency component image and a low-frequency component image comprises: performing Gaussian filtering on the skin region image to obtain a low-frequency component image; and subtracting the low-frequency component image from the skin region image to obtain a high-frequency component image.

[0009] In one embodiment, in the case that the protection processing is performed on the portrait structure region in the high-frequency component image and the low-frequency component image, the skin flat region is subjected to a skin-smoothing processing and the spot region is subjected to a spot-removing processing in the high-frequency component image and the low-frequency component image, respectively, to obtain a high-frequency output image after skin-smoothing and spot-removing and a low-frequency output image after skin-smoothing and spot-removing, comprises: in the case that the protection processing is performed on the portrait structure region in the high-frequency component image, performing a first skin-smoothing processing on the skin flat region in the high-frequency component image and performing a first spot-removing processing on the spot region in the high-frequency component image to obtain the high-frequency output image after skin-smoothing and spot-removing; and in the case that the protection processing is performed on the portrait structure region in the low-frequency component image, performing a second skin-smoothing processing on the skin flat region in the low-frequency component image and performing a second spot-removing processing on the spot region in the low-frequency component image to obtain the low-frequency output image after skin-smoothing and spot-removing.

[0010] In one embodiment, the first skin-smoothing processing is different from the second skin-smoothing processing, and / or the first spot-removing processing is different from the second spot-removing processing.

[0011] In one embodiment, the performing the skin-smoothing processing on the skin flat region in the high-frequency component image comprises: performing a non-linear attenuation on the skin flat region in the high-frequency component image; wherein in the non-linear attenuation, each pixel value in the high-frequency component image is multiplied by a variable attenuation coefficient, and the variable attenuation coefficient decreases with the decrease of the pixel value multiplied by it.

[0012] In an embodiment, performing the spot removal processing on the spot region in the high-frequency component image comprises: performing a mirror four-corners linear interpolation on each pixel in the spot region in the high-frequency component image to fill in each pixel; wherein, in the mirror four-corners linear interpolation, for a target pixel to be filled in, four boundaries of the spot region in four directions of up, down, left and right of the target pixel are determined, four symmetric pixels outside the spot region of the target pixel are determined respectively with the four boundaries as centers, and a value determined by performing a bilinear interpolation on the four symmetric pixels is used as a pixel value of the target pixel to fill in the target pixel.

[0013] In an embodiment, performing the skin smoothing processing on the skin flat region in the low-frequency component image comprises: performing a surface filtering on the skin flat region in the low-frequency component image.

[0014] In an embodiment, performing the spot removal processing on the spot region in the low-frequency component image comprises: performing a nearest-neighbor four-corners linear interpolation on each pixel in the spot region in the low-frequency component image to fill in each pixel; wherein, in the nearest-neighbor four-corners linear interpolation, for a target pixel to be filled in, four nearest-neighbor pixels outside the spot region in four directions of up, down, left and right of the target pixel are determined, and a value determined by performing a bilinear interpolation on the four nearest-neighbor pixels is used as a pixel value of the target pixel to fill in the target pixel.

[0015] In an embodiment, the performing the protection processing on the portrait structure region in the high-frequency component image and in the low-frequency component image comprises: generating a mask of the portrait structure region; wherein, the mask is used to ensure that the skin smoothing processing and the spot removal processing are prohibited to be performed on the portrait structure region when the skin smoothing processing is performed on the skin flat region and the spot removal processing is performed on the spot region in the high-frequency component image and in the low-frequency component image.

[0016] An image processing device for a portrait image, the device comprising: a region image determining module configured to determine a skin region image in a portrait image to be processed, and a skin flat region, a spot region and a portrait structure region contained in a high frequency component image and a low frequency component image of the skin region image; a region image processing module configured to perform a skin polishing process on the skin flat region and a spot removing process on the spot region in the high frequency component image and the low frequency component image respectively, with a protection process performed on the portrait structure region in the high frequency component image and the low frequency component image, so as to obtain a high frequency output image after skin polishing and spot removing and a low frequency output image after skin polishing and spot removing; and a skin region output module configured to superimpose the high frequency output image after skin polishing and spot removing and the low frequency output image after skin polishing and spot removing, so as to obtain a skin region output image.

[0017] An electronic device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program: determining a skin region image in a portrait image to be processed, and a skin flat region, a spot region and a portrait structure region contained in a high frequency component image and a low frequency component image of the skin region image; performing a skin polishing process on the skin flat region and a spot removing process on the spot region in the high frequency component image and the low frequency component image respectively, with a protection process performed on the portrait structure region in the high frequency component image and the low frequency component image, so as to obtain a high frequency output image after skin polishing and spot removing and a low frequency output image after skin polishing and spot removing; and superimposing the high frequency output image after skin polishing and spot removing and the low frequency output image after skin polishing and spot removing, so as to obtain a skin region output image.

[0018] A computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the following steps: determining a skin region image in a portrait image to be processed, and a skin flat region, a spot region and a portrait structure region contained in a high frequency component image and a low frequency component image of the skin region image; performing a skin polishing process on the skin flat region and a spot removing process on the spot region in the high frequency component image and the low frequency component image respectively, with a protection process performed on the portrait structure region in the high frequency component image and the low frequency component image, so as to obtain a high frequency output image after skin polishing and spot removing and a low frequency output image after skin polishing and spot removing; and superimposing the high frequency output image after skin polishing and spot removing and the low frequency output image after skin polishing and spot removing, so as to obtain a skin region output image.

[0019] The portrait image processing method, device, electronic device and storage medium determine the skin region image in the portrait image to be processed, and the skin flat region, spot region and portrait structure region contained in the high-frequency component image and the low-frequency component image of the skin region image, and perform the skin smoothing processing on the skin flat region and the spot region in the high-frequency component image and the low-frequency component image respectively in the case of performing the protection processing on the portrait structure region in the high-frequency component image and the low-frequency component image. Therefore, the different high-frequency components and low-frequency components and different types of skin regions can be processed in a targeted manner while the definition of the portrait structure region is preserved to the greatest extent. Thus, the skin smoothing and spot removal in the portrait image can be realized while the realistic skin texture and the clear portrait structure are protected, so that the beautified portrait image has a more realistic texture. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A schematic diagram of an electronic device in an embodiment;

[0021] Figure 2 A flowchart of a portrait image processing method in an embodiment;

[0022] Figure 3 A flowchart of a portrait image processing method in an embodiment;

[0023] Figure 4 A flowchart of a portrait image processing method in an embodiment;

[0024] Figure 5 A nonlinear attenuation curve in an embodiment;

[0025] Figure 6 A schematic diagram of a spot region mask in a high-frequency component image of a portrait image in an embodiment;

[0026] Figure 7 A schematic diagram of mirror four vertex linear interpolation in an embodiment;

[0027] Figure 8 A schematic diagram of adjacent four vertex linear interpolation in an embodiment;

[0028] Figure 9 A schematic diagram of portrait structure protection in an embodiment;

[0029] Figure 10 A structural block diagram of a portrait image processing device in an embodiment;

[0030] Figure 11 An internal structure diagram of an electronic device in an embodiment;

[0031] Figure 12A structural schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0032] For the purpose, technical solutions and advantages of the present application to be more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0033] The portrait image processing method provided by the present application can be applied to an electronic device 100 as shown in Figure 1 The electronic device 100 can be, but is not limited to, various smart phones, digital cameras, personal computers, notebook computers, tablet computers, etc. The electronic device 100 can be loaded with a camera 101. The electronic device 100 can obtain a to-be-processed portrait image by real-time shooting through the camera 101, and perform the portrait image processing method of the present application on the portrait image to obtain a skin region output image for subsequent application. In an alternative embodiment, the electronic device 100 can obtain the to-be-processed portrait image from other devices through a network or other communication means, or read the to-be-processed portrait image from a local storage, and then perform the portrait image processing method of the present application on the obtained portrait image. The electronic device 100 can also be loaded with a display screen 102, so that the electronic device 100 can display the obtained skin region output image or an image obtained by further processing the skin region output image on the display screen 102 for a user to view.

[0034] In one embodiment, a portrait image processing method is provided, which can be applied to an electronic device 100 as shown in Figure 1 Referring to Figure 2 and Figure 4 , the method comprises the following steps S210-S230:

[0035] Step S210, determining a skin region image in a to-be-processed portrait image, and a skin flat region, a spot region and a portrait structure region contained in a high-frequency component image and a low-frequency component image of the skin region image.

[0036] In one embodiment, referring to Figure 3 , step S210 can comprise the following steps S211-S213:

[0037] Step S211, identifying and extracting a skin region image from the to-be-processed portrait image.

[0038] Referring to Figure 3 , before step S210, the method can further comprise step S201 of obtaining a to-be-processed portrait image.

[0039] The portrait image to be processed can be an original image captured by the electronic device 100 in real time, or an image processed from the original image, or an image read from another device or the local storage of the electronic device 100. The portrait image to be processed contains information of a human face.

[0040] The skin region refers to a region in which human skin is generally located in the image. For example, the skin region can include a face region for beautification applications for the face. However, it can be understood that in actual applications, the skin region can also include ears, necks, hands, and other human skin regions, that is, the portrait image processing method of the present application can also recognize and process ears, necks, arms, hands, and other human skin regions when running.

[0041] The recognition of the skin region can be implemented by using any known technology such as an artificial intelligence (AI) model, for example, a U-NET model described in the document Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation [J]. Springer, Cham, 2015. Therefore, the present application will not be described here. After the skin region is recognized from the portrait image, the skin region image in the skin region can be extracted accordingly.

[0042] In step S212, the skin region image is divided into a high-frequency component image and a low-frequency component image.

[0043] The high-frequency component in the portrait image represents a part of the image with a sharp color texture change, while the low-frequency component in the portrait image represents a part of the image with a slow color texture change. Generally, the skin texture information and the spot defects of the portrait are mainly concentrated in the high-frequency component, while the large color spots of the skin and the information of the facial structure are mainly in the low-frequency component.

[0044] In this step, the skin region image is subjected to frequency division processing to divide into a high-frequency component image and a low-frequency component image, so as to subsequently perform differentiated processing on the high-frequency component image and the low-frequency component image, so that the skin color after the skin polishing and spot removal is more uniform, and the skin quality is more delicate and realistic. The frequency division processing is performed on each pixel in the skin region image, by dividing each pixel in the skin region image into a high-frequency component pixel and a low-frequency component pixel, all high-frequency component pixels constitute a high-frequency component image, and all low-frequency component pixels constitute a low-frequency component image, so as to divide the skin region image into a high-frequency component image and a low-frequency component image. It can be understood that the high-frequency component image and the low-frequency component image obtained after the division will have the same resolution as the skin region image.

[0045] Step S213, respectively identifying the skin flat region, the spot region and the portrait structure region contained in the high-frequency component image and the low-frequency component image.

[0046] The skin flat region refers to a region where the skin distribution is relatively flat and uniform, such as the face, forehead, chin and other regions. For example, the skin flat region can be substantially equal to the part remaining after removing the spot region and the portrait structure region from the skin region. The spot region refers to the region where freckles, acne marks and other imperfections are located in the skin. The portrait structure region refers to the region where the organ structure to be protected in the portrait is located. The portrait structure region can be specifically defined to contain different regions according to the application needs, and for example, in the case of applying to face beautification, the portrait structure region can include the regions where the eyes, eyebrows, nose, mouth and face contour are located. However, in other embodiments, the portrait structure region can also contain more or less regions or different regions. For example, for hand beautification, the portrait structure region can include the regions where the nails and hand contours are located, etc.

[0047] The identification of the skin flat region, the spot region and the portrait structure region can be implemented by using any known technology such as an AI model, for example, the U-NET model described in the document Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation [J]. Springer, Cham, 2015. can also be used for identification, therefore this application will not be repeated here.

[0048] It can be understood that, for steps S212-S213, the skin flat area, the spot area and the portrait structure area can also be identified from the skin area image first, and then the identified skin area image is segmented into the high-frequency component image and the low-frequency component image, or for steps S211-S213, the skin area, the skin flat area, the spot area and the portrait structure area can also be identified from the portrait image to be processed first, and then the skin area image is extracted and the frequency division processing is performed, etc. These equivalent replacement manners can also determine the skin flat area, the spot area and the portrait structure area contained in the high-frequency component image and the low-frequency component image.

[0049] Step S220, in the case of performing the protection processing on the portrait structure area in the high-frequency component image and the low-frequency component image, performing the skin polishing processing on the skin flat area and the spot-removing processing on the spot area in the high-frequency component image and the low-frequency component image respectively, so as to obtain the high-frequency output image after skin polishing and spot removing and the low-frequency output image after skin polishing and spot removing.

[0050] Specifically, the step S220 can include: performing the first skin polishing processing on the skin flat area and the first spot-removing processing on the spot area in the high-frequency component image on the condition that the protection processing is performed on the portrait structure area in the high-frequency component image, so as to obtain the high-frequency output image after skin polishing and spot removing; performing the second skin polishing processing on the skin flat area and the second spot-removing processing on the spot area in the low-frequency component image on the condition that the protection processing is performed on the portrait structure area in the low-frequency component image, so as to obtain the low-frequency output image after skin polishing and spot removing.

[0051] Step S230, superimposing the high-frequency output image after skin polishing and spot removing and the low-frequency output image after skin polishing and spot removing to obtain the skin area output image.

[0052] Wherein, the superimposition refers to the process of adding the pixel values of the corresponding pixels in the two images. The high-frequency output image after skin polishing and spot removing and the low-frequency output image after skin polishing and spot removing have the same resolution, so in this step, the pixel value of each pixel of the high-frequency output image after skin polishing and spot removing can be added to the pixel value of the corresponding pixel at the corresponding position in the low-frequency output image after skin polishing and spot removing to obtain the skin area output image.

[0053] In the portrait image processing method, the skin region image in the portrait image to be processed is determined, and the skin flat region, the spot region and the portrait structure region contained in the high-frequency component image and the low-frequency component image of the skin region image are determined, and the skin flat region and the spot region are respectively subjected to the skin smoothing treatment and the spot removing treatment in the high-frequency component image and the low-frequency component image under the condition that the portrait structure region is subjected to the protection treatment in the high-frequency component image and the low-frequency component image. Thus, the different high-frequency components and low-frequency components and different types of skin regions can be subjected to the targeted optimization treatment while the definition of the portrait structure region is maximally retained. Thus, the realistic skin texture and the clear portrait structure can be simultaneously presented in the skin smoothing and spot removing of the portrait image, and the beautified portrait image has a more realistic texture.

[0054] In one embodiment, referring to Figure 3 and Figure 4 The portrait image processing method further includes the following steps S240-S250:

[0055] Step S240, extracting a non-skin region image from the portrait image to be processed;

[0056] Step S250, splicing the non-skin region image and the skin region output image to obtain a portrait output image.

[0057] The non-skin region refers to a region image that does not generally include skin. The non-skin region may, for example, include a background, clothes and the like. In this embodiment, after the skin region is identified, the remaining region after the skin region is removed from the portrait image is taken as the non-skin region, and in other embodiments, the image recognition can be directly performed on the portrait image to directly identify the non-skin region. After the non-skin region is determined, the non-skin region image in the non-skin region can be extracted accordingly.

[0058] In this embodiment, the non-skin region image is further spliced with the skin region output image obtained after the skin smoothing and spot removing treatment to obtain the portrait output image after the portrait beautification is completed, so that the portrait output image can be displayed to the user through a display screen or the like.

[0059] In one embodiment, the step of splitting the skin region image into a high-frequency component image and a low-frequency component image includes: performing Gaussian filtering on the skin region image to obtain the low-frequency component image; and subtracting the low-frequency component image from the skin region image to obtain the high-frequency component image.

[0060] For example, the Gaussian filtering function of the performed Gaussian filtering is as follows:

[0061]

[0062] In the above formula, Hi,j σ represents the filtering parameter at an offset (i,j) from the center of the Gaussian filter window, where i represents the horizontal offset from the center of the filter window and j represents the vertical offset from the center of the filter window; σ is the standard deviation of the Gaussian filter.

[0063] The size of the filtering window can be determined based on the size of the face in the skin region image, for example, it can be determined using the following formula:

[0064]

[0065] In the above formula, window size is the size of the filtering window, S face This represents the maximum value of the width and height of the detected face bounding boxes in the image.

[0066] It's understandable that as the angle of the face changes, the identified face detection box might be square or rectangular, especially when only half a face is visible in the image, in which case the identified face detection box will be rectangular. Therefore, using the maximum value of the width and height to represent the face size can more realistically reflect the face scale in the actual image. For different images, the face in the image may appear smaller or larger depending on its distance from the camera, so the size of the face detection box will vary relative to different images. The above scheme can adaptively adjust the parameters of the Gaussian filter according to the size of the face in the image.

[0067] In Gaussian filtering, a larger standard deviation σ results in greater skin smoothing intensity, but also a more blurred image. σ can be chosen based on empirical values. For example, σ = 20; however, other suitable values ​​can also be used.

[0068] Here, "subtraction" refers to the process of subtracting the pixel values ​​of corresponding pixels in two images. In this embodiment, the skin region image and the low-frequency component image have the same resolution. Therefore, in this embodiment, the pixel value of each pixel in the skin region image can be subtracted from the pixel value of the corresponding pixel at the corresponding position in the low-frequency component image to obtain the skin region output image.

[0069] In one embodiment, the first skin resurfacing process in step S220 is different from the second skin resurfacing process, and / or the first freckle removal process is different from the second freckle removal process.

[0070] The high-frequency component image and the low-frequency component image respectively represent different display components in the skin. In this embodiment, different skin-smoothing processing and different spot-removing processing are respectively performed on the high-frequency component image and the low-frequency component image, so that the high-frequency component image and the low-frequency component image can be processed more specifically and more appropriately, thereby effectively improving the skin-smoothing and spot-removing effect of the processed image and making the processed image more delicate and realistic.

[0071] The high-frequency component in the skin flat area (non-portrait structure area and non-spot area) is mainly composed of skin texture. For the processing of the skin texture, in some optional solutions, the following can be selected: 1) completely retaining the high-frequency component in the skin flat area (i.e. not performing processing on the skin flat area in the high-frequency component image), in which case the skin texture after processing is completely retained, and the processed skin is not delicate enough; 2) performing linear attenuation processing on the high-frequency component, in which case the skin texture after processing is relatively delicate, but when the same attenuation coefficient is used for relatively strong texture and relatively weak texture, either the relatively strong texture cannot be effectively weakened or the relatively weak texture is completely eliminated, so that the skin texture structure cannot be effectively retained and the delicate skin texture cannot be achieved.

[0072] To solve the defects of the above solutions, in one embodiment, performing skin-smoothing processing (e.g. first skin-smoothing processing) on the skin flat area in the high-frequency component image includes: performing nonlinear attenuation on the skin flat area in the high-frequency component image; wherein in the nonlinear attenuation, each pixel value in the high-frequency component image is multiplied by a variable attenuation coefficient, and the variable attenuation coefficient decreases as the pixel value multiplied by it decreases.

[0073] The technical solution of this embodiment uses nonlinear attenuation method to perform nonlinear attenuation on the skin flat area in the high-frequency component image, so that the relatively weak skin texture is multiplied by a relatively small variable attenuation coefficient (i.e. close to 1.0), and the relatively strong skin texture is multiplied by a relatively large variable attenuation coefficient (i.e. close to 0.0). The nonlinear attenuation corresponds to a nonlinear attenuation curve as shown in Figure 5 The expression of the nonlinear attenuation curve is -0.0025x 2 +x=y, where x is the input pixel value before nonlinear attenuation, and y is the output pixel value after nonlinear attenuation. After nonlinear attenuation, the image texture structure is completely retained, and the relatively strong texture is effectively attenuated, thereby presenting a delicate and realistic skin texture.

[0074] After the image is divided into frequency bands, the imaging of the spot area mask in the high-frequency component image of the portrait image is described with reference to Figure 6As shown, to achieve the desired spot removal effect, it is necessary to repair the pixels in the spot region on the high-frequency component image. In one alternative approach, when performing spot removal processing on the spot region in the high-frequency component image, the spot region in the high-frequency component image (…) can be directly… Figure 6 The pixel values ​​of the spot region mask shown in the right figure are cleared to zero; alternatively, the “nearest neighbor quad vertex linear interpolation” method, which will be described later, can be used to fill each pixel in the spot region. However, the area filled in this way is too smooth. Using this method to process the spot region in the high-frequency component image will cause the skin texture of the spot region to be lost, and the image will be too smooth and unnatural.

[0075] To address the shortcomings of the above-mentioned solutions, in one embodiment, performing spot removal processing (e.g., a first spot removal process) on spot regions in a high-frequency component image includes: performing mirrored four-vertex linear interpolation on each pixel in the spot region of the high-frequency component image to fill each pixel; wherein, in the mirrored four-vertex linear interpolation, for the target pixel to be filled, the four boundaries of the spot region of the target pixel in the four directions of up, down, left, and right are determined, and four symmetrical pixels of the target pixel outside the spot region are determined with the four boundaries as centers, and the values ​​determined by performing bilinear interpolation on the four symmetrical pixels are used as the pixel values ​​of the target pixel to fill the target pixel.

[0076] This embodiment proposes a unique "mirrored four-vertex linear interpolation" method to fill in blotchy regions in high-frequency component images. For example, see... Figure 7 As shown, taking pixel A as the target pixel to be filled as an example, the method is to use a spot region mask ( Figure 7 (In the white pixel area) Find the four boundaries of the spot area mask of pixel A in the four directions of up, down, left, and right. Figure 7 In the diagram, four bold line segments are located above, below, left, and right of pixel A. Using these four boundaries as centers, find the symmetrical pixels B, C, D, and E of pixel A. The pixel value of pixel A can then be obtained by performing bilinear interpolation on pixels B, C, D, and E. The calculation process for this bilinear interpolation is shown in the following formula:

[0077] V a =1 / (D ab +D ac +D ad +D ae )*((D ad +D ae )*(V b *D ac +V c *D ab ) / (D ad +D ac )+(Dab +D ac )*(V d *D ae +V e *D ad ) / (D ab +D ae ))

[0078] wherein, D ab is the Euclidean distance between pixel A and pixel B, D ac is the Euclidean distance between pixel A and pixel C, D ad is the Euclidean distance between pixel A and pixel D, D ae is the Euclidean distance between pixel A and pixel E, V a is the pixel value of pixel A calculated, V b is the pixel value of pixel B, V c is the pixel value of pixel C, V d is the pixel value of pixel D, V e is the pixel value of pixel E.

[0079] Thus, repeating the operation, the filling value of each pixel in the spot area in the high-frequency component image can be obtained. After the "mirror four-vertex linear interpolation", the spot area in the high-frequency component image can better restore the skin texture in the high-frequency component image.

[0080] In one embodiment, performing the skin-smoothing process (e.g., the second skin-smoothing process) on the skin flat area in the low-frequency component image includes: performing surface filtering on the skin flat area in the low-frequency component image.

[0081] In this embodiment, for the skin flat area in the low-frequency component image, a conventional surface filter can be used as an edge-preserving filter to filter the skin flat area in the low-frequency component image, so that a uniform skin base color can be obtained.

[0082] After the skin region image is divided into frequency components, the fine texture and small spots of the skin are concentrated in the high-frequency component image, and the slightly large spots are still presented on the low-frequency component image. In order to achieve the effect of removing spots, the spots can be filled in the low-frequency component image.

[0083] In an embodiment, performing the spot area processing (e.g., the second spot area processing) on the low-frequency component image includes: performing a four-adjacent-vertex linear interpolation on each pixel in the spot area in the low-frequency component image to fill in each pixel; wherein, in the four-adjacent-vertex linear interpolation, for a target pixel to be filled in, four nearest neighbor pixels of the target pixel outside the spot area in four directions of up, down, left and right are determined, and a value determined by performing a bilinear interpolation on the four nearest neighbor pixels is used as a pixel value of the target pixel to fill in the target pixel.

[0084] In the present embodiment, the four-adjacent-vertex linear interpolation is used to fill in the spot area in the low-frequency component image. For example, referring to FIG. 6, pixel A is taken as an example of a target pixel to be filled in. Four nearest neighbor pixels F, G, H and I of pixel A outside the spot area mask in four directions of up, down, left and right are found through the white pixel area in the spot area mask, i.e., four boundaries of the spot area mask in four directions of up, down, left and right of pixel A are found through the spot area mask, and the four nearest neighbor pixels F, G, H and I are the four pixels immediately adjacent to the four boundaries of the spot area mask. A pixel value of pixel A can be obtained by performing a bilinear interpolation on the four nearest neighbor pixels F, G, H and I. Figure 8 Figure 8 Figure 8

[0085] In this way, the filling values of each pixel in the spot area in the low-frequency component image can be obtained. After the four-adjacent-vertex linear interpolation, the skin in the spot area in the low-frequency component image is smooth and uniform, and a good spot area removal effect can be obtained.

[0086] It can be understood that in other embodiments, the mirror four-adjacent-vertex linear interpolation can also be used to fill in the spot area in the low-frequency component image.

[0087] In an embodiment, performing the protection processing on the portrait structure area in the high-frequency component image and the low-frequency component image includes: generating a mask of the portrait structure area; wherein the mask of the portrait structure area is used to ensure that the portrait structure area is prohibited from being subjected to the skin smoothing processing or the spot area processing when the skin smoothing processing is performed on the skin flat area and the spot area processing is performed on the spot area in the high-frequency component image and the low-frequency component image.

[0088] For example, referring to FIG. 7, a mask of a portrait structure area is generated, and the mask of the portrait structure area is used to ensure that the portrait structure area is prohibited from being subjected to the skin smoothing processing or the spot area processing when the skin smoothing processing is performed on the skin flat area and the spot area processing is performed on the spot area in the high-frequency component image and the low-frequency component image. Figure 9 ​​​As shown, for the face beautification application, taking the skin region including the face region as an example, an AI model (for example, the U-NET model described above) can be used to extract the face outer contour region and the face feature (for example, including eyes, eyebrows, nose, and mouth) region as the portrait structure region, generate a portrait structure region mask, and protect the high-frequency components and low-frequency components in the portrait structure region mask, that is, prohibit performing skin smoothing processing or spot removal processing on the portrait structure region in the high-frequency component image and the low-frequency component image, so as to avoid image blur caused by the loss of face feature and contour information, and maximize the clarity of the portrait structure in the image.

[0089] It should be understood that, although Figures 2-4 the steps in the flowchart are shown in the order of the arrows, these steps are not necessarily executed in the order of the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, Figures 2-4 at least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0090] In one embodiment, as Figure 10 shown, a portrait image processing apparatus 1000 is provided, comprising a region image determination module 1010, a region image processing module 1020, and a skin region output module 1030, wherein:

[0091] The region image determination module 1010 is configured to determine a skin region image in a portrait image to be processed, and a skin flat region, a spot region, and a portrait structure region contained in a high-frequency component image and a low-frequency component image of the skin region image;

[0092] The region image processing module 1020 is configured to, in the case of performing protection processing on the portrait structure region in the high-frequency component image and the low-frequency component image, respectively perform skin smoothing processing on the skin flat region and spot removal processing on the spot region in the high-frequency component image and the low-frequency component image, so as to obtain a high-frequency output image after skin smoothing and spot removal and a low-frequency output image after skin smoothing and spot removal;

[0093] The skin region output module 1030 is configured to superimpose the high-frequency output image after skin smoothing and spot removal and the low-frequency output image after skin smoothing and spot removal to obtain a skin region output image.

[0094] In one embodiment, the portrait image processing apparatus 1000 further comprises:

[0095] a non-skin region extraction module 1040, configured to extract a non-skin region image from the portrait image to be processed;

[0096] a portrait output module 1050, configured to splice the non-skin region image and the skin region output image to obtain a portrait output image.

[0097] The specific definition of the portrait image processing apparatus 1000 can refer to the definition of the portrait image processing method in the foregoing, and will not be described herein. The various modules in the portrait image processing apparatus 1000 described above can be implemented by software, hardware, or a combination thereof, in whole or in part. The various modules described above can be embedded in or independent of a processor in a computer device (for example, an electronic device) in hardware form, or can be stored in a memory in a computer device (for example, an electronic device) in software form, so as to be called and executed by a processor to perform the operations corresponding to the various modules.

[0098] In one embodiment, an electronic device is provided, and a block diagram of an internal structure of the electronic device can be as shown in Figure 11 The electronic device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. The processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the electronic device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement an image distortion correction method. The display screen of the electronic device can be, for example, a liquid crystal display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or can be a key, trackball, or touchpad arranged on the shell of the electronic device, or can be an external keyboard, touchpad, or mouse, etc.

[0099] Those skilled in the art can understand that Figure 11 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0100] In one embodiment, as shown in Figure 12 An electronic device 1200 is provided, which includes a memory 1201 and a processor 1202. The memory 1201 is connected to the processor 1202, and the memory 1201 stores a computer program. The processor 1202 implements the following steps when executing the computer program:

[0101] determine a skin region image in the portrait image to be processed, and a skin flat region, a spot region and a portrait structure region contained in a high frequency component image and a low frequency component image of the skin region image;

[0102] In a case where the protection processing is performed on the portrait structure region in the high frequency component image and the low frequency component image, a skin smoothing processing is performed on the skin flat region and a spot removing processing is performed on the spot region in the high frequency component image and the low frequency component image respectively, so as to obtain a high frequency output image after skin smoothing and spot removing and a low frequency output image after skin smoothing and spot removing;

[0103] superimpose the high frequency output image after skin smoothing and spot removing and the low frequency output image after skin smoothing and spot removing, so as to obtain a skin region output image.

[0104] In one embodiment, as shown in Figure 12 The electronic device 1200 can further include a camera 1203 coupled with the electronic device 1200. The "coupled with" is intended to include both a form of directly disposing the camera in the electronic device 1200 and a form of disposing the camera outside the electronic device (for example, the camera is disposed on another terminal device) and the camera is communicatively connected with the electronic device 1200. The camera 1203 is communicatively connected with the processor 1202; the camera 1203 is used to shoot a portrait image and transmit the portrait image to the processor 1202; the processor 1202 receives the portrait image from the camera 1203.

[0105] In one embodiment, as shown in Figure 12 The electronic device 1200 can further include a display 1204, which is connected with the processor 1202; when the processor 1202 executes the computer program, the processor 1202 further controls the display 1204 to display the skin region output image or an image (for example, a portrait output image) obtained after processing the skin region output image.

[0106] In other embodiments, when the processor 1202 executes the computer program, the processor 1202 further implements the steps of the portrait image processing method of each of the above embodiments and has the corresponding beneficial effects.

[0107] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0108] Determine a skin region image in the portrait image to be processed, and a skin flat region, a spot region and a portrait structure region contained in a high frequency component image and a low frequency component image of the skin region image; perform a skin smoothing process on the skin flat region and a spot removing process on the spot region in the high frequency component image and the low frequency component image respectively in a case that a protection process is performed on the portrait structure region in the high frequency component image and the low frequency component image, so as to obtain a high frequency output image after skin smoothing and spot removing and a low frequency output image after skin smoothing and spot removing.

[0109] Superimpose the high frequency output image after skin smoothing and spot removing and the low frequency output image after skin smoothing and spot removing to obtain a skin region output image.

[0110] In other embodiments, the computer program, when executed by the processor, also implements the steps of the portrait image processing method of the various embodiments of the present application, and has the corresponding beneficial effects.

[0111] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0112] The technical features of the above embodiments can be combined in any way. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0113] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific and detailed manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these are all within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A portrait image processing method, the method comprising: determining a skin region image in a portrait image to be processed, and skin flat regions, spot regions and portrait structure regions contained in a high frequency component image and a low frequency component image of the skin region image; performing a first skin smoothing process on the skin flat regions and a first spot removing process on the spot regions in the high frequency component image under a condition that a protection process is performed on the portrait structure regions in the high frequency component image, to obtain a high frequency output image after skin smoothing and spot removing; performing a second skin smoothing process on the skin flat regions and a second spot removing process on the spot regions in the low frequency component image under a condition that a protection process is performed on the portrait structure regions in the low frequency component image, to obtain a low frequency output image after skin smoothing and spot removing; wherein the first skin smoothing process is different from the second skin smoothing process, and / or the first spot removing process is different from the second spot removing process; superimposing the high frequency output image after skin smoothing and spot removing and the low frequency output image after skin smoothing and spot removing to obtain a skin region output image.

2. The method of claim 1, wherein, The determining a skin region image in a portrait image to be processed, and skin flat regions, spot regions and portrait structure regions contained in a high frequency component image and a low frequency component image of the skin region image comprises: identifying and extracting a skin region image from a portrait image to be processed; segmenting the skin region image into a high frequency component image and a low frequency component image; identifying skin flat regions, spot regions and portrait structure regions contained in the high frequency component image and the low frequency component image, respectively.

3. The method of claim 1, wherein, Further comprising: extracting a non-skin region image from the portrait image to be processed; splicing the non-skin region image and the skin region output image to obtain a portrait output image.

4. The method of claim 2, wherein, The segmenting the skin region image into a high frequency component image and a low frequency component image comprises: performing a Gaussian filter on the skin region image to obtain a low frequency component image; subtracting the low frequency component image from the skin region image to obtain a high frequency component image.

5. The method according to any one of claims 1 to 4, characterized in that, The performing a skin smoothing process on the skin flat regions in the high frequency component image comprises: performing a non-linear attenuation on the skin flat regions in the high frequency component image; wherein in the non-linear attenuation, each pixel value in the high frequency component image is multiplied by a variable attenuation coefficient, and the variable attenuation coefficient decreases as the pixel value multiplied by it decreases.

6. The method according to any one of claims 1 to 4, characterized in that, The performing a spot removing process on the spot regions in the high frequency component image comprises: performing a mirror four-vertex linear interpolation on each pixel in the spot regions in the high frequency component image to fill in each pixel; In the mirror four vertex linear interpolation, for a target pixel to be filled, four boundaries of the spot region in four directions of up, down, left and right of the target pixel are determined, four symmetric pixels outside the spot region of the target pixel are determined respectively with the four boundaries as centers, and a value determined by performing bilinear interpolation on the four symmetric pixels is used as a pixel value of the target pixel to fill the target pixel.

7. The method according to any one of claims 1 to 4, characterized in that, The performing of the skin smoothing processing on the skin flat region in the low frequency component image comprises: The performing of surface filtering on the skin flat region in the low frequency component image.

8. The method according to any one of claims 1 to 4, characterized in that, The performing of the spot removing processing on the spot region in the low frequency component image comprises: Performing adjacent four vertex linear interpolation on each pixel in the spot region in the low frequency component image to fill each pixel; In the adjacent four vertex linear interpolation, for a target pixel to be filled, four nearest neighbor pixels outside the spot region in four directions of up, down, left and right of the target pixel are determined, and a value determined by performing bilinear interpolation on the four nearest neighbor pixels is used as a pixel value of the target pixel to fill the target pixel.

9. The method according to any one of claims 1 to 4, characterized in that, The performing of the protection processing on the portrait structure region in the high frequency component image and in the low frequency component image comprises: generating a mask of the portrait structure region; The mask is used to ensure that the skin smoothing processing and the spot removing processing are prohibited from being performed on the portrait structure region when the skin smoothing processing is performed on the skin flat region in the high frequency component image and in the low frequency component image and the spot removing processing is performed on the spot region.

10. A portrait image processing apparatus characterized by comprising: The device comprises: a region image determination module configured to determine a skin region image in a portrait image to be processed, and a skin flat region, a spot region and a portrait structure region contained in a high frequency component image and a low frequency component image of the skin region image; a region image processing module configured to perform first skin smoothing processing on the skin flat region in the high frequency component image and perform first spot removing processing on the spot region in the high frequency component image under the condition that protection processing is performed on the portrait structure region in the high frequency component image, to obtain a high frequency output image after skin smoothing and spot removing; and perform second skin smoothing processing on the skin flat region in the low frequency component image and perform second spot removing processing on the spot region in the low frequency component image under the condition that protection processing is performed on the portrait structure region in the low frequency component image, to obtain a low frequency output image after skin smoothing and spot removing; wherein the first skin smoothing processing is different from the second skin smoothing processing, and / or the first spot removing processing is different from the second spot removing processing; a skin region output module configured to superimpose the high frequency output image after skin smoothing and spot removing and the low frequency output image after skin smoothing and spot removing, to obtain a skin region output image. 11.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 9.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 9.

Images