Facial image processing methods, apparatuses, storage media, and electronic devices

Abstract

This disclosure provides a facial image processing method, apparatus, storage medium, and electronic device, relating to the field of image processing technology. The facial image processing method includes: determining facial key points at facial feature locations in a facial image to be processed; attaching a pre-made mask to the corresponding facial feature locations in the facial image to be processed based on the facial key points, obtaining a pre-processed facial image; splitting the facial skin in the pre-processed facial image to obtain a skin pre-processed image containing the facial skin; performing skin beautification processing on the skin pre-processed image to obtain a skin target image; and fusing the layers other than the skin pre-processed image from the split pre-processed facial image with the skin target image to obtain a processed target facial image. This method solves the technical problem of high performance overhead in current facial beautification methods, achieving the technical effect of reducing performance consumption during the facial beautification process.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method, apparatus, computer-readable storage medium, and electronic device for processing facial images. Background Technology

[0002] There are two common facial beautification and skin smoothing solutions: The first is based on traditional image processing, generally using filtering and smoothing methods such as bilateral filtering. Bilateral filtering can effectively beautify and smooth facial skin while preserving skin details, but it cannot remove isolated points with large color differences, such as blemishes or moles, and the smoothed effect is somewhat unnatural; furthermore, using large-scale filter kernels increases performance overhead. The second approach is an end-to-end AI (Artificial Intelligence) training scheme based on deep learning. This requires pre-preparing datasets of original and beautified portrait images, designing and training a network, and finally post-processing the generated beautified images, typically adding whitening filters to make the images more natural and realistic. While this approach can achieve end-to-end facial beautification and skin smoothing, acquiring the dataset is costly, and the trained neural network model also suffers from high performance overhead.

[0003] Therefore, current facial beautification methods all suffer from significant performance degradation. Summary of the Invention

[0004] This disclosure provides a facial image processing method, a computer-readable storage medium, and an electronic device, thereby reducing the performance consumption during facial beautification.

[0005] In a first aspect, one embodiment of this disclosure provides a facial image processing method, including:

[0006] Identify facial key points that locate facial features in the face image to be processed;

[0007] The pre-made mask is attached to the corresponding facial features in the face image to be processed based on the facial key points to obtain the pre-processed face image.

[0008] The facial skin in the preprocessed face image is split to obtain a preprocessed skin image containing the facial skin;

[0009] Skin beautification processing is performed on the preprocessed skin image to obtain the target skin image;

[0010] After splitting the preprocessed face image, the layers other than the skin preprocessed image are fused with the skin target image to obtain the processed target face image.

[0011] In one optional embodiment of this disclosure, determining facial key points at the location of facial features in the face image to be processed includes:

[0012] The face detection model is used to detect faces in the face image to be processed, and to obtain the facial key points of the facial features and facial contours in the face image to be processed.

[0013] In one optional embodiment of this disclosure, before attaching a prefabricated mask to the corresponding facial features in the face image to be processed based on facial key points to obtain a preprocessed face image, the method further includes:

[0014] Construct a facial mesh from the facial image to be processed based on facial key points;

[0015] Correspondingly, a pre-made mask is attached to the corresponding facial features in the image to be processed based on facial key points to obtain a pre-processed facial image, including:

[0016] Determine the coordinates of each marker point in the face mesh;

[0017] Based on the coordinates, the pre-made mask is mapped onto the corresponding facial features in the face image to be processed, thus obtaining the pre-processed face image.

[0018] In one optional embodiment of this disclosure, constructing a facial mesh from the facial image to be processed based on facial key points includes:

[0019] Linear interpolation is performed on facial key points to obtain preprocessed key points;

[0020] Triangulation is performed on the preprocessed key points to obtain a face mesh that matches the face image to be processed.

[0021] In one optional embodiment of this disclosure, the facial skin in the preprocessed facial image is segmented to obtain a skin preprocessed image containing the facial skin, including:

[0022] Convert the face image to be processed from the first color space to the HSV color space;

[0023] In the HSV color space, the facial skin in the preprocessed facial image is split based on the skin color values ​​of the processed facial image to obtain an initial skin image containing the facial skin;

[0024] The initial skin image is fused with the skin mask layer in the preprocessed face image to obtain a preprocessed skin image containing the facial skin.

[0025] In one optional embodiment of this disclosure, the skin beautification process includes at least one of: skin uniform brightening process, facial noise removal process, and diamond filter removal process.

[0026] In one optional embodiment of this disclosure, the layers other than the preprocessed skin image after splitting the preprocessed face image are fused with the skin target image to obtain the processed target face image, including:

[0027] After splitting the preprocessed face image, the layers other than the skin preprocessed image are fused with the skin target image to obtain the intermediate face image;

[0028] The target face image is obtained by subtracting blending mode and / or linear blending mode from the middle face image.

[0029] In an optional embodiment of this disclosure, after fusing the layers other than the preprocessed skin image with the skin target image to obtain the processed target face image, the method further includes:

[0030] Sharpen the facial features in the target face image.

[0031] Secondly, one embodiment of this disclosure provides a facial image processing apparatus, the apparatus comprising:

[0032] The determination module is used to determine the facial key points where facial features are located in the face image to be processed;

[0033] The attachment module is used to attach a pre-made mask to the corresponding facial features in the face image to be processed based on facial key points, so as to obtain a pre-processed face image.

[0034] The splitting module is used to split the facial skin in the preprocessed facial image to obtain a preprocessed skin image containing the facial skin;

[0035] The processing module is used to perform skin beautification processing on the preprocessed skin image to obtain the target skin image;

[0036] The fusion module is used to fuse the layers other than the skin preprocessed image after splitting the preprocessed face image with the skin target image to obtain the processed target face image.

[0037] Thirdly, one embodiment of this disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described above.

[0038] Fourthly, one embodiment of this disclosure provides an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described method by executing the executable instructions.

[0039] The technical solution disclosed herein has the following beneficial effects:

[0040] The aforementioned facial image processing method first determines the facial key points of the facial features in the image to be processed. Then, a pre-made mask is attached to the corresponding facial feature positions in the image to be processed based on the facial key points to obtain a pre-processed facial image. The facial skin in the pre-processed facial image is then split to obtain a skin pre-processed image containing the facial skin. Skin beautification processing is performed on the skin pre-processed image to obtain a skin target image. Finally, the layers other than the skin pre-processed image of the split pre-processed facial image are merged with the skin target image to obtain the processed target facial image. This method eliminates the need for sample pre-preparation and the use of large filter kernels, resulting in lower performance consumption. This solves the technical problem of high performance consumption in current facial beautification methods, achieving the technical effect of reducing the performance consumption of facial beautification. Furthermore, by splitting the image to be processed into different layers using a mask and performing beautification processing on each layer separately, this method ensures that the texture of facial features and facial contours is not affected when processing the skin layer, and the skin condition is not affected when processing facial features and facial contours, resulting in better facial beautification effects.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0043] Figure 1 A flowchart illustrating a face image processing method in this exemplary embodiment is shown;

[0044] Figure 2 This illustration shows a schematic diagram of facial key points in a facial image processing method according to this exemplary embodiment;

[0045] Figure 3 This illustration shows an image with a mask added in a face image processing method according to this exemplary embodiment;

[0046] Figure 4 This exemplary embodiment illustrates a skin preprocessed image and other preprocessed images in a face image processing method.

[0047] Figure 5 This illustration shows a face grid in a face image processing method according to this exemplary embodiment;

[0048] Figure 6 A flowchart illustrating a face image processing method in this exemplary embodiment is shown;

[0049] Figure 7 A flowchart illustrating a face image processing method in this exemplary embodiment is shown;

[0050] Figure 8 A flowchart illustrating a face image processing method in this exemplary embodiment is shown;

[0051] Figure 9 This illustration shows a LUT diagram in a face image processing method according to this exemplary embodiment;

[0052] Figure 10 A flowchart illustrating a face image processing method in this exemplary embodiment is shown;

[0053] Figure 11 This illustration shows a before-and-after comparison of facial image beautification performed using the facial image processing method of this disclosure in this exemplary embodiment.

[0054] Figure 12 This illustration shows a schematic diagram of the structure of a facial image processing device in this exemplary embodiment;

[0055] Figure 13 A schematic diagram of the structure of an electronic device in this exemplary embodiment is shown. Detailed Implementation

[0056] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0057] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0058] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0059] In related technologies, there are two commonly used facial beautification and skin smoothing solutions: The first is a skin smoothing solution based on traditional image processing, generally using filtering and smoothing methods, such as bilateral filtering. Bilateral filtering can effectively beautify and smooth facial skin while preserving skin details, but it cannot remove isolated points with large color differences, such as blemishes or moles, and the smoothed effect is rather harsh; furthermore, using large-scale filter kernels increases performance overhead. The second approach is an end-to-end AI training solution based on deep learning. This requires pre-preparing datasets of original and beautified portrait images, designing and training a network, and finally post-processing the generated beautified images, typically adding whitening filters to make the images more natural and realistic. While this solution can achieve end-to-end facial beautification and skin smoothing, acquiring the dataset is costly, and the trained neural network model also suffers from high performance overhead. Therefore, current facial beautification methods generally have significant performance overhead.

[0060] In view of the above problems, this disclosure provides a facial image processing method to reduce performance consumption while ensuring the facial image processing effect. The following is a brief introduction to the application environment of the facial image processing method provided in this disclosure:

[0061] The terrain mesh generation method provided in this disclosure is applied to a terminal device, which can be a server, computer, laptop, tablet, or other wearable device. This disclosure does not impose any limitations and the device can be selected according to the actual situation.

[0062] The following example uses the aforementioned terminal device as the execution subject, illustrating how the facial image processing method is applied to beautify the face in the image to be processed on the terminal device. Please refer to... Figure 1 The facial image processing method provided in this embodiment includes the following steps 101-105:

[0063] Step 101: Determine the facial key points in the face image to be processed, which are the locations of facial features.

[0064] The face image to be processed refers to an image containing a face, which can be a human face or the face of other animals with relatively clear outlines; this disclosure does not specifically limit the scope. The facial features refer to, for example, the five senses such as eyes, nose, mouth, ears, and eyebrows, as well as the facial contours. Key points are points used to characterize the location of these facial features, such as… Figure 2 The key points in it.

[0065] Step 102: Attach the pre-made mask to the corresponding facial features in the face image to be processed based on the facial key points to obtain the pre-processed face image.

[0066] The preprocessed facial image refers to the facial image to be processed, which includes a mask. This mask is used to mark the positions of facial features, facilitating the separation of facial features and contours from the facial skin. When adjusting skin tone, it protects the texture details of facial features from being smoothed out; similarly, processing the texture of facial features does not affect the skin tone. For example, please see... Figure 3 In this process, the mask is attached to the face image to be processed in the original image, thus obtaining the image as shown below. Figure 3 Preprocessed facial images.

[0067] Step 103: Decompose the facial skin in the preprocessed face image to obtain a preprocessed skin image containing the facial skin.

[0068] The aforementioned mask can distinguish facial features from skin, and the terminal device can segment the preprocessed facial image into, for example, segments based on the mask's markings. Figure 4 (a) contains only preprocessed skin images of the face, and as shown in the image. Figure 4 (b) contains only the facial features preprocessed image of the face and facial contours.

[0069] Step 104: Perform skin beautification processing on the preprocessed skin image to obtain the target skin image.

[0070] This skin beautification process refers to applying beautification techniques, such as skin smoothing and blemish removal, to the facial skin without affecting facial features or the lines and textures of the face. The target skin image is the pre-processed skin image, that is, the image or layer after beautifying the facial skin. It should be noted that this skin beautification process can be achieved through any method, such as box filtering or diamond filtering, which will not be elaborated upon here.

[0071] Step 105: After splitting the preprocessed face image, merge the layers other than the skin preprocessed image with the skin target image to obtain the processed target face image.

[0072] Other layers include, but are not limited to, the outline layer of the face, the facial features layer, etc., because the above steps split the preprocessed face image into different layers or images, only performing skin beautification processing on the skin layer (i.e., the skin preprocessed image), while other layers are not processed at all, or performing targeted beautification processing on other layers, such as making the eyes bigger, brighter, or slimmer, and then merging all the layers to obtain the processed target face image.

[0073] The facial image processing method provided in this embodiment first determines the facial key points of the facial feature positions in the facial image to be processed, then attaches a pre-made mask to the corresponding facial feature positions in the facial image to be processed based on the facial key points to obtain a pre-processed facial image, then splits the facial skin in the pre-processed facial image to obtain a skin pre-processed image containing the facial skin, then performs skin beautification processing on the skin pre-processed image to obtain a skin target image, and finally merges the other layers of the pre-processed facial image except for the skin pre-processed image with the skin target image to obtain the processed target facial image. This method eliminates the need for sample pre-preparation and large filter kernels, resulting in lower performance consumption. This solves the technical problem of high performance loss in current face beautification methods, achieving the technical effect of reducing the performance consumption of face beautification. Furthermore, this method splits the face image to be processed into different layers using a mask, performing beautification processing on each layer separately. This ensures that the texture of facial features and contours is not affected when processing the skin layer, and the skin condition is not affected when processing facial features and contours, resulting in better face beautification effects.

[0074] In an optional embodiment of this disclosure, step 101, determining the facial key points at the location of facial features in the face image to be processed, includes the following steps:

[0075] The face detection model is used to detect faces in the face image to be processed, and to obtain the facial key points of the facial features and facial contours in the face image to be processed.

[0076] This face detection model employs the MTCNN (Multi-task convolutional neural network) model, comprising a three-layer network structure of P-Net, R-Net, and O-Net, which can perform face region detection and facial landmark detection simultaneously. By inputting the face region ROI (Region of Interest) into this face detection model, approximately 130 facial landmarks can be obtained, for example... Figure 2 The key points in it.

[0077] This disclosure embodiment performs face detection on the face image to be processed based on a face detection model, and obtains the key facial points of the facial features and facial contours in the face image to be processed. The detection results are reliable and more efficient, which can further improve the reliability and efficiency of the face image processing method of this disclosure embodiment.

[0078] In an optional embodiment of this disclosure, before step 102, where the prefabricated mask is attached to the corresponding facial features in the face image to be processed based on facial key points to obtain the preprocessed face image, the face image processing method further includes the following step A:

[0079] Step A: Construct a facial mesh from the facial key points of the image to be processed.

[0080] A facial mesh is used to locate various features on the face, such as... Figure 5 A triangular mesh, a facial mesh constructed based on facial key points that is identical to the facial features.

[0081] Please see Figure 6 In an optional embodiment of this disclosure, step A, constructing a facial mesh of the face image to be processed based on facial key points, includes the following steps 601-602:

[0082] Step 601: Perform linear interpolation on the facial key points to obtain preprocessed key points.

[0083] Step 602: Triangulate the preprocessed key points to obtain a face mesh that matches the face image to be processed.

[0084] Triangulation involves dividing the surface formed by the key points of the preprocessing into triangular fragments to facilitate a better fit with the face image or skin preprocessing image to be processed, thereby further improving the accuracy and effect of skin beautification.

[0085] For the corresponding information, please refer to [link / reference]. Figure 7 Step 102 above, which involves attaching the pre-made mask to the corresponding facial features in the face image to be processed based on facial key points, to obtain the pre-processed face image, includes the following steps 701-702:

[0086] Step 701: Determine the coordinates of each marker point in the face grid.

[0087] The marker point can be, for example... Figure 5 The intersection of the facial grid can also be the location of each triangular fragment. This disclosure does not impose specific limitations on the embodiments, as long as it can characterize the features of each position on the face.

[0088] Step 702: Map the pre-made mask to the corresponding facial features in the face image to be processed according to the coordinate position to obtain the pre-processed face image.

[0089] This mapping can be achieved through a one-to-one correspondence transformation of position coordinates, or by directly drawing a preset mask onto the corresponding facial feature positions in the image to be processed, aligning the mask with the facial features of the face image to be processed. The UV coordinates of each point on the facial mesh are recorded, the mask is calibrated onto the face image to be processed and its facial features are aligned, and the mask is drawn onto the face image to be processed using warp (affine transformation) and UV coordinates.

[0090] In this embodiment, the coordinates of each marker point in the face grid are first determined, and then the pre-made mask is mapped to the corresponding facial features in the face image to be processed according to the coordinates to obtain a pre-processed face image. The pre-processed face image has a more accurate fit between the mask and the facial features in the face image to be processed, which can further improve the face image beautification effect of this embodiment.

[0091] Please see Figure 8 In an optional embodiment of this disclosure, step 103, which involves splitting the facial skin in the preprocessed facial image to obtain a preprocessed skin image containing the facial skin, includes the following steps 801-803:

[0092] Step 801: Convert the face image to be processed from the first color space to the HSV color space.

[0093] The primary color space of the face image to be processed may be RGB color space, etc., which can be uniformly converted to HSV color space through color conversion.

[0094] Step 802: In the HSV color space, based on the skin color values ​​of the processed face image, the facial skin in the preprocessed face image is split to obtain an initial skin image containing the facial skin.

[0095] The facial skin area is relatively large, and its color differs significantly from other facial features. Therefore, the facial skin in the preprocessed facial image can be extracted based on color values ​​to quickly, efficiently, and accurately obtain the initial skin image, which is the initial skin layer. The remaining parts are the facial feature image (or layer) and the facial contour image (or layer).

[0096] Step 803: Fuse the initial skin image with the skin mask layer in the preprocessed face image to obtain a preprocessed skin image containing the facial skin.

[0097] As in the above embodiment, the mask is used to identify facial skin, facial features, and contours, which can be distinguished or separated using the mask. After obtaining the initial skin image, it is fused with the skin mask layer in the mask to obtain a preprocessed skin image containing only the skin portion; in addition, the preprocessed image also contains other preprocessed images containing facial features and contours, which together constitute the aforementioned preprocessed image.

[0098] This embodiment converts the face image to be processed from the first color space to the HSV color space. In the HSV color space, the facial skin in the preprocessed face image is split based on the skin color values ​​of the processed face image to obtain an initial skin image containing the facial skin. Finally, the initial skin image is fused with the skin mask layer in the preprocessed face image to obtain a preprocessed skin image containing the facial skin. This can obtain a preprocessed skin image containing only the facial skin, which facilitates subsequent targeted skin beautification of the facial skin without affecting the texture of facial features and facial contours, resulting in a better facial image beautification effect.

[0099] In one optional embodiment of this disclosure, the above-mentioned skin beautification process includes at least one of: skin uniform brightening process, facial noise removal process, and diamond filter removal process.

[0100] The skin brightening process can utilize LUT filters for whitening, etc. The input to this process is the pre-processed skin image described above, but it can also include other images, such as the face image to be processed, as a reference. Then, it is processed using methods such as... Figure 9 The skin brightening process, including the whitening LUT image and other filters such as box filters, can be characterized by the following formula (1):

[0101] oriImg + mask + BoxFileter + LUT = PreImg(1)

[0102] In formula (1), PreImg represents the image after uniform skin brightening, oriImg represents the face image to be processed, mask represents the mask plate, BoxFilter represents box filtering, and LUT represents LUT whitening processing.

[0103] Facial noise removal can be achieved through BoxFilter, which is highly effective.

[0104] The diamond-shaped filter can remove larger spots and blemishes from the skin. With the same number of parameters, the diamond-shaped filter smooths a wider range and can also remove large noise points, resulting in better image enhancement. In this embodiment, the diamond-shaped filter can be performed once or multiple times. The diamond-shaped filter removal process can be characterized by the following formulas (2) and (3):

[0105] oriImg + mask = imgblur1(2)

[0106] oriImg + mask + imgblur1 = imgblur2(3)

[0107] In formulas (2) and (3), mgblur1 represents the image after the first diamond filtering process, mgblur1 represents the image after the second diamond filtering process, oriImg represents the face image to be processed, and mask represents the mask plate.

[0108] The skin beautification processing in this embodiment includes at least one of the following: skin uniform brightening processing, facial noise removal processing, and diamond filtering removal processing. It can beautify facial skin from multiple dimensions such as skin tone and blemishes, resulting in a better facial image beautification effect.

[0109] Please see Figure 10 In an optional embodiment of this disclosure, step 105, fusing the layers other than the skin preprocessed image after splitting the preprocessed face image with the skin target image to obtain the processed target face image, includes the following steps 1001-1002:

[0110] Step 1001: After splitting the preprocessed face image, merge the layers other than the skin preprocessed image with the skin target image to obtain the intermediate face image.

[0111] This embodiment can obtain the intermediate face image in the same way as step 803 described above, and will not be repeated here.

[0112] Step 1002: Process the intermediate face image using subtraction blending mode and / or linear blending mode to obtain the target face image.

[0113] The subtraction of the mixed mode processing can be characterized by the following formula (4):

[0114] C1= (PreImg –imgblur1) / 2.0 +128 (4)

[0115] In formula (4), C1 is the result color after subtracting the blending mode, PreImg represents the image after skin uniform brightening (the base color of the processing), and imgblur1 represents the image after the first diamond filtering process (the blending color).

[0116] The linear blending mode processing can be characterized by the following formula (5):

[0117] C2 = (255 – 2imgblur2)(C1 / 255)2 + C1 imgblur2 / 128)(5)

[0118] In formula (5), C2 is the result color after linear blending mode processing, imgblur2 represents the image (blended color) after the second diamond filtering process, and C1 is the result color after subtracting the blending mode processing.

[0119] The embodiments disclosed herein achieve a better skin smoothing effect on the target face image by processing the intermediate face image with a subtraction blending mode and / or a linear blending mode.

[0120] In an optional embodiment of this disclosure, after step 105, where the preprocessed face image is split and the layers other than the preprocessed skin image are fused with the skin target image to obtain the processed target face image, the face image processing method further includes the following steps:

[0121] Sharpen the facial features in the target face image.

[0122] For example, an image can be enhanced using a method similar to Unsharp Mask (USM) sharpening to highlight texture details, resulting in a final image with strong detail realism and a smoothing effect. For instance, from... Figure 11 In (a), the image is a partial view of the unprocessed face image to be processed, corresponding to... Figure 11 The sharpened image in (b) is shown. The USM sharpening process can be characterized by the following formula (6):

[0123] C3 = (A – wB) / (1-w)(6)

[0124] In formula (6), C3 represents the sharpened image, A represents the input image (for example, the image after skin smoothing using the above steps), B represents the image after Gaussian blurring of the input image, and w represents the weight (which can be flexibly set according to the actual situation).

[0125] The image after skin smoothing is smoothed as a whole, making the whole image less clear and bright compared to the original image. The embodiments of this disclosure sharpen the facial features in the target face image, making the details of the target face image more prominent and the beautification effect better.

[0126] The embodiments disclosed herein employ a graphics rendering pipeline for the entire beautification process of the skin preprocessed image. The graphics rendering pipeline can convert 3D or 2D scene coordinates into screen pixels. In actual processing, only vertex shaders and pixel shaders need to be programmed to achieve the skin smoothing and beautification process of the portrait. Compared with the traditional image processing process, it has higher real-time performance and can be applied to GPUs (graphics processing units) of any configuration.

[0127] To implement the above-described facial image processing method, one embodiment of this disclosure provides a facial image processing apparatus 1200. Figure 12 A schematic architecture diagram of a face image processing apparatus 1200 is shown, including: a determination module 1210, an attachment module 1220, a splitting module 1230, a processing module 1240, and a fusion module 1250, wherein:

[0128] The determining module 1210 is used to determine the facial key points at the location of facial features in the face image to be processed;

[0129] The attachment module 1220 is used to attach a pre-made mask to the corresponding facial features in the face image to be processed based on facial key points, so as to obtain a pre-processed face image.

[0130] The splitting module 1230 is used to split the facial skin in the preprocessed facial image to obtain a preprocessed skin image containing the facial skin;

[0131] The processing module 1240 is used to perform skin beautification processing on the preprocessed skin image to obtain the target skin image;

[0132] The fusion module 1250 is used to fuse the layers other than the skin preprocessed image after splitting the preprocessed face image with the skin target image to obtain the processed target face image.

[0133] In one optional embodiment of this disclosure, the determining module 1210 is specifically used to perform face detection on the face image to be processed based on the face detection model, and obtain the facial key points of the facial features and facial contours in the face image to be processed.

[0134] In an optional embodiment of this disclosure, the attachment module 1220 is further configured to: construct a facial mesh of the face image to be processed based on facial key points; determine the coordinate position of each marker point in the facial mesh; and map a prefabricated mask to the corresponding facial feature position in the face image to be processed according to the coordinate position, thereby obtaining a preprocessed face image.

[0135] In one optional embodiment of this disclosure, the attachment module 1220 is specifically used to perform linear interpolation on facial key points to obtain preprocessed key points; and to perform triangulation on the preprocessed key points to obtain a facial mesh that matches the facial image to be processed.

[0136] In one optional embodiment of this disclosure, the splitting module 1230 is specifically used to convert the face image to be processed in the first color space to the HSV color space; in the HSV color space, based on the skin color values ​​of the processed face image, split the facial skin in the preprocessed face image to obtain an initial skin image containing the facial skin; and fuse the initial skin image with the skin mask layer in the preprocessed face image to obtain a preprocessed skin image containing the facial skin.

[0137] In one optional embodiment of this disclosure, the skin beautification process includes at least one of: skin uniform brightening process, facial noise removal process, and diamond filter removal process.

[0138] In one optional embodiment of this disclosure, the fusion module 1250 is specifically used to fuse the layers other than the skin preprocessed image after splitting the preprocessed face image with the skin target image to obtain an intermediate face image; and to process the intermediate face image by subtraction blending mode and / or linear blending mode to obtain the target face image.

[0139] In an optional embodiment of this disclosure, the processing module 1240 is further configured to sharpen the facial features in the target face image.

[0140] Exemplary embodiments of this disclosure also provide a computer-readable storage medium that can be implemented as a program product including program code, which, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. In one embodiment, the program product can be implemented as a portable compact disk read-only memory (imgblur D-ROM) including program code and can run on an electronic device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0141] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (imgblurD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0142] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0143] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0144] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and imgblur++, and conventional procedural programming languages ​​such as the "imgblur" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider). In embodiments of this disclosure, the program code stored in a computer-readable storage medium, when executed, can implement any step of the facial image processing method described above.

[0145] Please see Figure 13 Exemplary embodiments of this disclosure also provide an electronic device 1300, which can be a backend server for an information platform. Reference is made below. Figure 13This electronic device 1300 will be described. It should be understood that... Figure 13 The electronic device 1300 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0146] like Figure 13 As shown, the electronic device 1300 is presented in the form of a general-purpose computing device. The components of the electronic device 1300 may include, but are not limited to: at least one processing unit 1310, at least one storage unit 1320, and a bus 1330 connecting different system components (including storage unit 1320 and processing unit 1310).

[0147] The storage unit stores program code, which can be executed by the processing unit 1310 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1310 can perform actions such as... Figure 1 The methods and steps shown are as follows.

[0148] Storage unit 1320 may include volatile storage units, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0149] Storage unit 1320 may also include a program / utility 1324 having a set (at least one) program module 1325, such program module 1325 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0150] Bus 1330 may include a data bus, an address bus, and a control bus.

[0151] Electronic device 1300 can also communicate with one or more external devices 2000 (e.g., keyboards, pointing devices, Bluetooth devices, etc.) via input / output (I / O) interface 1340. Electronic device 1300 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 1350. As shown, network adapter 1350 communicates with other modules of electronic device 1300 via bus 1330. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0152] In this embodiment of the disclosure, the program code stored in the electronic device can be executed to perform any of the steps in the facial image processing method described above.

[0153] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0154] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0155] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. A facial image processing method, characterized in that, include: Identify facial key points that locate facial features in the face image to be processed; A pre-made mask is attached to the corresponding facial features in the face image to be processed based on the facial key points to obtain a pre-processed face image. The facial skin in the preprocessed facial image is segmented to obtain a preprocessed skin image containing the facial skin; The preprocessed skin image is then subjected to skin beautification processing to obtain the target skin image; The preprocessed face image is split into layers other than the preprocessed skin image and then fused with the target skin image to obtain the processed target face image. The step of segmenting the facial skin in the preprocessed facial image to obtain a preprocessed skin image containing the facial skin includes: The face image to be processed, which is in the first color space, is converted to the HSV color space; In the HSV color space, the facial skin in the preprocessed facial image is split based on the skin color values ​​of the processed facial image to obtain an initial skin image containing the facial skin; The initial skin image is fused with the skin mask layer in the preprocessed face image to obtain the preprocessed skin image containing the facial skin.

2. The facial image processing method according to claim 1, characterized in that, The process of determining the facial key points in the face image to be processed, including: The face detection model is used to perform face detection on the face image to be processed, and the key points of the face, including the positions of facial features and the facial contour, are obtained.

3. The facial image processing method according to claim 1, characterized in that, Before attaching the prefabricated mask to the corresponding facial features in the face image to be processed based on the facial key points to obtain the preprocessed face image, the method further includes: A facial mesh is constructed from the facial key points of the face image to be processed. Correspondingly, the step of attaching the pre-made mask to the corresponding facial features in the face image to be processed based on the facial key points to obtain the pre-processed face image includes: Determine the coordinate position of each marker point in the face mesh; The pre-made mask is mapped onto the corresponding facial features in the face image to be processed based on the coordinates, thereby obtaining the pre-processed face image.

4. The facial image processing method according to claim 3, characterized in that, The process of constructing a facial mesh from the facial key points of the face image to be processed includes: Linear interpolation is performed on the facial key points to obtain preprocessed key points; The preprocessed key points are triangulated to obtain the face mesh that matches the face image to be processed.

5. The facial image processing method according to claim 1, characterized in that, The skin enhancement process includes at least one of the following: skin even brightening, facial noise removal, and diamond filtering removal.

6. The facial image processing method according to claim 1, characterized in that, The step of fusing the layers other than the preprocessed skin image from the preprocessed facial image with the target skin image to obtain the processed target facial image includes: The preprocessed face image is split and then merged with the skin target image, except for the skin preprocessed image, to obtain the intermediate face image; The intermediate face image is processed by subtraction blending mode and / or linear blending mode to obtain the target face image.

7. The facial image processing method according to claim 1, characterized in that, After the preprocessed face image is split and all layers other than the preprocessed skin image are fused with the target skin image to obtain the processed target face image, the method further includes: The facial features in the target face image are sharpened.

8. A facial image processing device, characterized in that, The device includes: The determination module is used to determine the facial key points where facial features are located in the face image to be processed; The attachment module is used to attach a pre-made mask to the corresponding facial features in the face image to be processed based on the facial key points, so as to obtain a pre-processed face image. The splitting module is used to split the facial skin in the preprocessed facial image to obtain a preprocessed skin image containing the facial skin; The processing module is used to perform skin beautification processing on the preprocessed skin image to obtain the target skin image; The fusion module is used to fuse the layers other than the skin preprocessed image after splitting the preprocessed face image with the skin target image to obtain the processed target face image; The splitting module is configured as follows: The face image to be processed, which is in the first color space, is converted to the HSV color space; In the HSV color space, the facial skin in the preprocessed facial image is split based on the skin color values ​​of the processed facial image to obtain an initial skin image containing the facial skin; The initial skin image is fused with the skin mask layer in the preprocessed face image to obtain the preprocessed skin image containing the facial skin.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 7 by executing the executable instructions.

Images