Image processing method, device, electronic device and storage medium

Abstract

The present disclosure provides an image processing method, apparatus, electronic device, and storage medium. The method comprises: obtaining an image to be processed, the image to be processed including a facial region; extracting low-frequency images corresponding to different facial structures from the facial region; fusing the low-frequency images into a first reference image according to their corresponding facial structures; obtaining a residual between the facial region and the first low-frequency image corresponding to the entire facial region in the low-frequency image; superimposing the residual onto the first reference image to obtain a second reference image of the facial region; and performing skin resurfacing processing on the image to be processed based on the second reference image to obtain a skin resurfacing image including the facial region. This method can not only remove facial blemishes and improve facial uniformity during the skin resurfacing process of a portrait by fusing the low-frequency images corresponding to different facial structures in the facial region, but also preserve facial structural features through the residual information between the facial region and the low-frequency image, thereby enhancing the three-dimensional effect of the portrait and optimizing the skin resurfacing results.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, device, electronic device, and storage medium. Background Art

[0002] Photo sharing has become a mainstream social media platform for modern people, and before sharing, photos often need to be enhanced. Portrait enhancement typically involves skin texture, face shape, body shape, and image quality. For example, common enhancement methods for problematic skin include resurfacing, blemish removal, and even skin tone.

[0003] Related technologies use high-intensity low-pass filtering to smoothen the skin, improving its smoothness and uniformity. However, this method results in a loss of skin texture and a more artificial appearance. Using global low-pass filtering to smoothen the skin can also disrupt facial structure, resulting in a flattened face and a loss of three-dimensionality.

[0004] Therefore, how to solve the problems of skin texture loss, strong fake feeling and loss of three-dimensional sense caused by the skin resurfacing process in related technologies has become a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The present disclosure provides an image processing method, device, electronic device and storage medium, which are used to remove facial blemishes while preserving facial structural features during the portrait skin refining process, improve facial uniformity and portrait three-dimensionality, and optimize the portrait skin refining results.

[0006] According to a first aspect of an embodiment of the present disclosure, the present disclosure provides an image processing method, the method comprising:

[0007] Acquire an image to be processed, where the image to be processed includes a face area;

[0008] Extract low-frequency images corresponding to different facial structures from the face region;

[0009] fusing the low-frequency images into a first reference image according to their corresponding facial structures;

[0010] Obtaining a residual between the face region and a first low-frequency image corresponding to the entire face region in the low-frequency image;

[0011] Superimposing the residual onto the first reference image to obtain a second reference image of the face area;

[0012] The image to be processed is subjected to skin smoothing processing based on the second reference image to obtain a skin smoothing image containing a human face area.

[0013] In an optional embodiment, obtaining an image to be processed includes: obtaining spatial coordinate information of a face region, the spatial coordinate information including the position coordinates of facial key points and the pitch angle of the face; generating a facial bounding box based on the position coordinates of the facial key points and the pitch angle of the face; and extracting the face region from the image to be processed using the facial bounding box.

[0014] In an optional embodiment, generating a facial bounding box based on the position coordinates of facial key points and the face pitch angle includes:

[0015] The position coordinates of the facial key points are linearly expanded to obtain the position coordinates of the extension points; the width of the facial bounding box is calculated based on the position coordinates of the facial key points and the position coordinates of the extension points; the height of the facial bounding box is calculated based on the face pitch angle; and the facial bounding box is generated based on the width and height.

[0016] In an optional embodiment, extracting low-frequency images corresponding to different facial structures from the face region includes:

[0017] The face area is Gaussian blurred to obtain a first low-frequency image; the first low-frequency image is mean blurred to obtain a second low-frequency image corresponding to the cheeks and / or forehead in the face area; the first low-frequency image is surface blurred to obtain a third low-frequency image corresponding to the facial features in the face area.

[0018] In an optional embodiment, performing Gaussian blur processing on the face region to obtain the first low-frequency image includes: performing Gaussian blur processing on the entire facial structure in the face region according to a first blur radius to obtain the first low-frequency image.

[0019] Performing mean blurring on the first low-frequency image to obtain a second low-frequency image corresponding to the cheek and / or forehead in the face region includes: performing mean blurring on the first low-frequency image according to a second blur radius to obtain the second low-frequency image, wherein the second blur radius is greater than the first blur radius.

[0020] Performing surface blurring processing on the first low-frequency image to obtain a third low-frequency image corresponding to facial features in the face region includes: performing surface blurring processing on the first low-frequency image according to a third blurring radius to obtain the third low-frequency image, wherein the third blurring radius is greater than the first blurring radius.

[0021] In an optional embodiment, fusing the low-frequency images into a first reference image according to their respective corresponding facial structures includes: determining a weight relationship between the second low-frequency image and the third low-frequency image based on their respective corresponding facial structures to obtain a first mask image indicating the weight relationship, wherein the second low-frequency image has a higher weight in a cheek region and / or a forehead region of the first mask image than the third low-frequency image, and the third low-frequency image has a higher weight in a facial feature region of the first mask image than the second low-frequency image; and fusing the second low-frequency image and the third low-frequency image into the first reference image using the first mask image.

[0022] In an optional embodiment, superimposing the residual onto the first reference image to obtain a second reference image of the face area includes: superimposing the residual linear light onto the first reference image to obtain the second reference image.

[0023] A skin smoothing process is performed on the image to be processed based on the second reference image to obtain a skin smoothing image containing a facial region, including: fusing the second reference image and the facial region into a third reference image of the facial region, wherein the second mask image is used to indicate a weight relationship between image information in the second reference image and image information in the facial region; and drawing the third reference image into the facial region of the image to be processed to output the skin smoothing image.

[0024] According to a second aspect of an embodiment of the present disclosure, the present disclosure provides an image processing apparatus, the apparatus comprising:

[0025] An acquisition unit is configured to acquire an image to be processed, where the image to be processed includes a face area;

[0026] a fusion unit configured to extract low-frequency images corresponding to different facial structures from the face region; and fuse the low-frequency images into a first reference image according to the facial structures corresponding to the low-frequency images;

[0027] a superposition unit configured to obtain a residual between the face region and a first low-frequency image corresponding to the entire face region in the low-frequency image; and superimpose the residual onto the first reference image to obtain a second reference image of the face region;

[0028] The skin refining unit is configured to perform skin refining on the image to be processed based on the second reference image to obtain a skin refining image containing a human face area.

[0029] In an optional embodiment, the acquisition unit is specifically configured to: obtain spatial coordinate information of the face area, the spatial coordinate information including the position coordinates of the face key points and the face pitch angle; generate a facial bounding box based on the position coordinates of the face key points and the face pitch angle; and use the facial bounding box to extract the face area from the image to be processed.

[0030] In an optional embodiment, the acquisition unit is specifically configured to generate a facial bounding box based on the position coordinates of facial key points and the pitch angle of the face as follows:

[0031] The position coordinates of the facial key points are linearly expanded to obtain the position coordinates of the extension points; the width of the facial bounding box is calculated based on the position coordinates of the facial key points and the position coordinates of the extension points; the height of the facial bounding box is calculated based on the face pitch angle; and the facial bounding box is generated based on the width and height.

[0032] In an optional embodiment, the fusion unit is specifically configured to extract low-frequency images corresponding to different facial structures from the face region as follows:

[0033] The face area is Gaussian blurred to obtain a first low-frequency image; the first low-frequency image is mean blurred to obtain a second low-frequency image corresponding to the cheeks and / or forehead in the face area; the first low-frequency image is surface blurred to obtain a third low-frequency image corresponding to the facial features in the face area.

[0034] In an optional embodiment, the fusion unit performs Gaussian blur processing on the face area to obtain the first low-frequency image, and is specifically configured to: perform Gaussian blur processing on the overall facial structure in the face area according to the first blur radius to obtain the first low-frequency image.

[0035] In the process of performing mean blurring on the first low-frequency image to obtain a second low-frequency image corresponding to the cheek and / or forehead in the face region, the fusion unit is specifically configured to perform mean blurring on the first low-frequency image according to a second blur radius to obtain the second low-frequency image, where the second blur radius is greater than the first blur radius.

[0036] In the process of performing surface blurring on the first low-frequency image to obtain a third low-frequency image corresponding to the facial features in the face area, the fusion unit is specifically configured as follows: performing surface blurring on the first low-frequency image according to a third blur radius to obtain a third low-frequency image, where the third blur radius is greater than the first blur radius.

[0037] In an optional embodiment, the fusion unit is specifically configured to fuse the low-frequency images into the first reference image according to their corresponding facial structures as follows:

[0038] A weight relationship between the second low-frequency image and the third low-frequency image is determined based on the facial structures corresponding to each of the second low-frequency image and the third low-frequency image to obtain a first mask image indicating the weight relationship, wherein the second low-frequency image has a higher weight in a cheek region and / or a forehead region of the first mask image than the third low-frequency image, and the third low-frequency image has a higher weight in a facial feature region of the first mask image than the second low-frequency image; and the second low-frequency image and the third low-frequency image are fused into a first reference image using the first mask image.

[0039] In an optional embodiment, the superposition unit superimposes the residual onto the first reference image to obtain a second reference image of the face area, and is specifically configured to superimpose the residual linear light onto the first reference image to obtain the second reference image.

[0040] The skin smoothing unit is specifically configured to: fuse the second reference image with the facial area to obtain a third reference image of the facial area, wherein the second mask image is used to indicate the weight relationship between the image information in the second reference image and the image information in the facial area; and draw the third reference image into the facial area of ​​the image to be processed to output a skin smoothing image.

[0041] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor and a memory, wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the image processing method in the first aspect.

[0042] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by an electronic device, the electronic device is enabled to execute at least the image processing method in the first aspect.

[0043] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, which implements the image processing method in the first aspect when executed by a processor.

[0044] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0045] In the present disclosure, an image to be processed is obtained, the image to be processed includes a face region; low-frequency images corresponding to different facial structures are extracted from the face region; the low-frequency images are fused into a first reference image according to their respective corresponding facial structures; a residual between the face region and the first low-frequency image corresponding to the entire face region in the low-frequency image is obtained; the residual is superimposed on the first reference image to obtain a second reference image of the face region; and skin resurfacing processing is performed on the image to be processed based on the second reference image to obtain a skin resurfacing image including the face region. In the present disclosure, facial blemishes can be removed and facial uniformity can be improved during the skin resurfacing process of a portrait by fusing the low-frequency images corresponding to different facial structures in the face region, and the facial structural features can be retained through the residual information between the face region and the low-frequency image, thereby improving the three-dimensional effect of the portrait and optimizing the skin resurfacing result of the portrait. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0047] Figure 1 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0048] Figure 2 The figure is a schematic diagram of the facial structure of a human portrait according to an exemplary embodiment.

[0049] Figure 3 The figure is a schematic structural diagram of an image processing apparatus according to an exemplary embodiment.

[0050] Figure 4 The figure is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0051] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0052] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0053] As mentioned above, in portrait beautification, the scope of beautification usually includes skin texture, face shape, body shape, image quality, etc. For example, common beautification methods for skin problems include skin resurfacing, blemish removal, and skin texture evenness.

[0054] Related technologies use high-intensity low-pass filtering to smoothen the skin of portraits, enhancing its smoothness and uniformity. However, this method can result in a loss of skin texture and a more artificial appearance. To address this issue, related technologies add sharpening after smoothing. This process introduces some high-frequency information, including image noise, which degrades image quality.

[0055] Furthermore, related technologies that use global low-pass filtering to smooth skin in portraits can disrupt facial structure, resulting in flattened faces and a loss of three-dimensionality. To address this issue, a mask can be used to reduce the impact of low-pass filtering on facial structure. However, this approach cannot remove imperfections within the mask, compromising the smoothing effect.

[0056] In summary, a solution is urgently needed to solve the problems of skin texture loss, strong fake feeling and loss of three-dimensional sense caused by the skin resurfacing process in related technologies.

[0057] To solve at least one technical problem existing in the related art, the present disclosure provides an image processing method, apparatus, electronic device and storage medium.

[0058] The core concept of the above technical solution is to obtain an image to be processed, which includes a facial region, such as a portrait in a photograph to be optimized. Low-frequency images corresponding to different facial structures are then extracted from the facial region and fused into a first reference image based on their corresponding facial structures. By fusing the low-frequency images corresponding to different facial structures in the facial region, facial blemishes are removed and facial uniformity is improved during the portrait resurfacing process. Furthermore, a residual is obtained between the facial region and the first low-frequency image corresponding to the entire facial region in the low-frequency image. This residual is then superimposed on the first reference image to obtain a second reference image of the facial region. This image retains as much facial structural information as possible while blurring to remove facial blemishes, achieving a sharpened portrait. This avoids the problems of missing skin texture, a strong fake look, and a loss of three-dimensionality that can occur with a single type of blurring, further enhancing the three-dimensionality of the portrait and optimizing the resurfacing result. Finally, the image to be processed is resurfaced based on the second reference image to obtain a resurfaced image containing the facial region, optimizing the resurfacing result.

[0059] Based on the core idea introduced above, the present disclosure provides an image processing method. Figure 1 FIG. 1 is a flow chart of an image processing method provided by an exemplary embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0060] 101. Acquire an image to be processed, where the image to be processed includes a face area;

[0061] 102. Extract low-frequency images corresponding to different facial structures from the face area;

[0062] 103. Fusing the low-frequency images into a first reference image according to their corresponding facial structures;

[0063] 104. Obtain a residual between the face region and the first low-frequency image corresponding to the entire face region in the low-frequency image;

[0064] 105. Superimpose the residual on the first reference image to obtain a second reference image of the face area;

[0065] 106. Perform skin smoothing on the image to be processed based on the second reference image to obtain a skin smoothed image containing a facial area.

[0066] In actual applications, each step in the above method can be implemented by an electronic device, which can be a terminal device such as a mobile phone, a wearable device (such as a smart bracelet, VR device, etc.), a tablet computer, a PC, a laptop computer, etc. Taking a mobile phone as an example, it can be implemented by calling a dedicated application installed in the mobile phone, or by calling a small program set in an instant messaging application or other types of applications, or by calling a cloud server through a mobile phone application. The steps in the above method can also be implemented by the cooperation of multiple electronic devices. For example, the server can send the execution result to the terminal device so that the terminal device can render and display the execution result. The server can be a physical server containing an independent host, or it can be a virtual server hosted by a host cluster, or it can be a cloud server, which is not limited by the present disclosure.

[0067] It is worth noting that the present disclosure does not limit the execution order of steps 101 to 106. In fact, the above steps can be executed simultaneously or separately in different orders.

[0068] The following describes the various steps in the image processing method in conjunction with specific embodiments.

[0069] First, in 101, an image to be processed is obtained. The image to be processed includes but is not limited to a photo or a video.

[0070] In this embodiment, the image to be processed includes a human face region. In practical applications, the human face region is the main area in the image to be processed that requires skin smoothing. For example, the human face in a selfie, a portrait in a short video, a portrait in a group photo, and an image of a pet. Of course, in addition to the human face region, the skin smoothing process can also be performed on other areas in the image to be processed, such as the arm region, the leg region, etc.

[0071] Specifically, in an optional embodiment, in 101, obtaining the image to be processed can be implemented as follows: obtaining spatial coordinate information of the face area, the spatial coordinate information including the position coordinates of the facial key points and the face pitch angle; generating a facial bounding box based on the position coordinates of the facial key points and the face pitch angle; and extracting the face area from the image to be processed using the facial bounding box.

[0072] Optionally in the present disclosure, a face region can be identified from the image to be processed in 101. Specifically, for example, in response to an object selection instruction issued by the user to the image to be processed, the subject selected by the user is used as the face region. For example, a target recognition strategy is used to identify the face region to be smoothed in the image to be processed. Specifically, a subject that matches the target type can be used as the face region. Alternatively, a subject at a target position in the image to be processed can be used as the face region. The target position can be determined based on image shooting parameters. For example, for an image shot using center-weighted metering, the subject at the geometric center of the image is used as the face region. It is worth noting that this step can be automatically triggered after the image to be processed is input.

[0073] Then, in step 101, spatial coordinate information of the facial region is obtained. In practical applications, this spatial coordinate information includes, but is not limited to, facial key points and facial pitch angle. Optionally, a trained deep neural network can be used to calculate spatial coordinate information of the facial region, such as the position information of facial key points and facial pitch angle. Facial key points include, for example, eyebrows, nose, eyes, mouth, ears, forehead, chin, and jawline. These include the contour points of eyebrows, nose, eyes, and mouth.

[0074] Then, a facial bounding box is generated based on the spatial coordinate information, that is, based on the position coordinates of the facial key points and the pitch angle of the face. Specifically, in an optional embodiment, the width of the facial bounding box is calculated based on the position coordinates of the facial key points. The height of the facial bounding box is calculated based on the position coordinates of the facial pitch angle and the highest and lowest points of the facial key points. The facial bounding box is generated based on the width and height. For example, based on the position information of the facial key points And the face pitch angle To estimate the minimum bounding box of the portrait (i.e., the facial bounding box), the minimum bounding box needs to cover not only the entire facial information of the portrait, but also as much of the neck as possible.

[0075] Furthermore, because common skin resurfacing algorithms in related art typically do not consider facial scale and instead directly apply the same set of parameters to the entire image, resulting in poor image processing stability, the present disclosure employs a method of extracting the facial region from the image to be processed, after generating a facial bounding box. This can be achieved by extracting the facial region corresponding to the facial bounding box from the image to be processed based on the positional coordinates of each point within the facial bounding box. Specifically, after extracting the facial region corresponding to the facial bounding box, the extracted facial region is plotted onto an image window of a fixed size to obtain an initial image of the facial region. For example, based on the positional coordinates of each point within the facial bounding box and its width and height information, the facial region corresponding to the facial bounding box (i.e., the facial region) is extracted from the image to be processed (i.e., the facial region) and plotted onto an image window of a fixed size k to obtain a facial image S (i.e., the initial image). Scaling the facial regions to the same scale thus provides a basis for using the same filtering parameters in subsequent processing, such as using the same step radius for blurring. In this way, portraits of different scales can all be processed and optimized for facial structural features in the same scale space, ensuring the stability of the image processing effect.

[0076] Optionally in the present disclosure, in order to better cover the range where the portrait is located, the facial key points can also be linearly expanded to obtain extension points. Based on this, another optional embodiment of generating a facial bounding box based on the position coordinates of the facial key points and the face pitch angle in 101 is: linearly expand the position coordinates of the facial key points to obtain the position coordinates of the extension points; calculate the width of the facial bounding box based on the position coordinates of the facial key points and the position coordinates of the extension points; calculate the height of the facial bounding box based on the face pitch angle; generate the facial bounding box based on the width and height. Thus, the facial bounding box generated by the position coordinates of the facial key points and the extension points can more effectively cover the range where the portrait is located and improve the smoothness of the portrait outline. Specifically, the position coordinates of the extension points are calculated as follows:

[0077] Based on the location information of facial key points First, linearly expand the facial key points to estimate the forehead key points and the portrait frame points. For example, for N existing facial key points, select the facial key point in the face as the starting point Xo and the point on the eyebrow as Xm. Then define the extension point Xe as the point on the extension line of Xo and Xm, and set the expansion multiplier to n. Based on the following formula:

[0078]

[0079] Based on the above formula, the coordinates of the forehead extension point Xe can be calculated. Similarly, based on the above formula, the coordinates of the portrait frame extension points can be estimated by selecting key points in the middle of the portrait and key points on the cheeks. This will not be further explained here.

[0080] In step 102, low-frequency images corresponding to different facial structures are extracted from the face region. Specifically, the face region includes multiple facial structures. Figure 2 Taking the facial structure shown as an example, the multiple facial structures include but are not limited to at least one of the cheeks, forehead, and facial features. The correspondence between facial structures and low-frequency images can be one-to-one or many-to-one, for example, multiple facial structures correspond to one low-frequency image. In 102, corresponding types of blurring processing can be performed using filtering methods corresponding to the multiple facial structures to obtain low-frequency images corresponding to different facial structures. In practical applications, the types of blurring processing include but are not limited to Gaussian blurring, mean blurring, and surface blurring. Based on this, the multiple filtering modules include but are not limited to: at least one of a Gaussian blurring module corresponding to the overall facial structure, a mean blurring module corresponding to the cheeks and forehead, and a surface blurring module corresponding to the facial features. Of course, in practical applications, low-frequency images corresponding to different facial structures can also be obtained through other methods such as downsampling, which is not limited in the present disclosure.

[0081] In the present disclosure, since different types of blurring can be achieved on the face area through different filtering methods, the images after different blurring can retain the visual characteristics of different facial structures, avoiding problems such as skin texture loss, strong fake feeling and lack of three-dimensional sense caused by a single type of blurring, further removing facial blemishes and improving facial uniformity.

[0082] Specifically, in an optional embodiment of step 102, Gaussian blur processing is performed on the facial region to obtain a first low-frequency image; mean blur processing is performed on the first low-frequency image to obtain a second low-frequency image corresponding to the cheeks and / or forehead in the facial region; and surface blur processing is performed on the first low-frequency image to obtain a third low-frequency image corresponding to the facial features in the facial region. Thus, different types of blur processing are used to extract different types of facial features, thereby removing facial blemishes and improving facial uniformity.

[0083] The following describes a specific implementation method for performing blur processing using different processing granularities for different facial structures in the face area.

[0084] Regarding the overall facial structure of a portrait (i.e., the entire face), in an optional embodiment, Gaussian blurring of the facial region in step 102 to obtain a first low-frequency image corresponding to the entire facial region can be implemented by Gaussian blurring the entire facial structure within the facial region according to a first blur radius to obtain the first low-frequency image corresponding to the entire facial structure. Thus, the Gaussian blurring process removes any imperfections in the overall facial structure of the portrait, providing a foundation for subsequent skin refining.

[0085] For the cheeks and / or forehead of a portrait, since the cheeks and forehead are relatively smooth, the skin color in these areas needs to be made more uniform. Therefore, mean filtering can be used to extract low-frequency information in the image area where the cheeks and forehead are located, thereby achieving the purpose of uniform skin color and texture, and improving the smoothness and uniformity of the portrait skin.

[0086] Based on the first low-frequency image, mean blurring is performed on the first low-frequency image in step 102 to obtain a second low-frequency image corresponding to the cheeks and / or forehead in the facial region. This can be achieved by performing mean blurring on the first low-frequency image according to a second blurring radius to obtain the second low-frequency image corresponding to the cheeks and / or forehead. The second blurring radius is larger than the first blurring radius. Specifically, a mean blurring module can be used to perform mean blurring on relatively smooth regions such as the cheeks and forehead in the first low-frequency image according to the second blurring radius to obtain the second low-frequency image corresponding to these smooth regions.

[0087] Because facial features in portraits are three-dimensional, it's necessary to remove minor imperfections while preserving their structure and details. Therefore, it's necessary to extract high-frequency information from the image region where the facial features are located to preserve their structural characteristics (such as eye contours and nose bridge height). Surface blurring, due to its properties, can better protect details and preserve edge information, making it suitable for processing facial features.

[0088] Based on the first low-frequency image, surface blurring is performed on the first low-frequency image in step 102 to obtain a third low-frequency image corresponding to the facial features in the face region. This can be achieved by performing surface blurring on the first low-frequency image according to a third blur radius to obtain the third low-frequency image corresponding to the facial features. The third blur radius is larger than the first blur radius. Specifically, a surface blurring module can be used to perform surface blurring on a three-dimensional region (e.g., the facial features) in the first low-frequency image according to the third blur radius to obtain the third low-frequency image corresponding to the three-dimensional region.

[0089] By using the various processing granularities and filtering methods used in the above steps (such as the various filters used in actual applications), different types of blurring are performed on different facial structures in the portrait. For example, mean filtering is used on the forehead and cheeks, and surface blurring is used on the facial features. This effectively preserves the texture characteristics of the corresponding facial structures, removing blemishes while enhancing the three-dimensional effect of the portrait.

[0090] Then, at step 103, the low-frequency images are fused into a first reference image based on their corresponding facial structures. Specifically, a weight relationship between the second and third low-frequency images is determined based on their corresponding facial structures to obtain a first mask image indicating the weight relationship. The second and third low-frequency images are fused into a first reference image using the first mask image. The second low-frequency image has a higher weight in the cheek and / or forehead regions of the first mask image than the third low-frequency image, and the third low-frequency image has a higher weight in the facial features regions of the first mask image than the second low-frequency image.

[0091] In simple terms, the first mask image is used to set the proportion of the second low-frequency image and the third low-frequency image in the fusion process. Obviously, the second low-frequency image is obtained for the cheeks and / or forehead of the face area, so the weight of the second low-frequency image in the cheeks and / or forehead area is higher than the weight of the third low-frequency image; the third low-frequency image is obtained for the facial features of the face area, so the weight of the second low-frequency image in the facial features area is lower than the weight of the third low-frequency image. Based on this, in the present disclosure, optionally, the second low-frequency image has the highest weight in the cheeks and / or forehead area in the first mask image, and the third low-frequency image has the highest weight in the facial features area in the first mask image. Therefore, the fusion method of the low-frequency images corresponding to various facial structures can be adjusted through the first mask image, that is, the degree of influence of different types of blur processing methods on each facial structure in the final optimization effect can be flexibly adjusted, thereby obtaining a first reference image that can effectively fuse the blur processing effects of each facial structure in the face area, which helps to improve the texture of the final optimization result.

[0092] In step 104, the residual between the initial image of the facial region and the first low-frequency image corresponding to the entire facial region in the low-frequency image is obtained. This step is primarily based on the principle of unsharp masking (USM) to obtain sharpening information of the facial region to enhance the three-dimensional effect of the portrait. Specifically, in step 104, the residual between the facial region and the first low-frequency image is calculated. This residual is the sharpening information of the facial region. The first low-frequency image is a low-frequency image of the overall facial structure. It can be understood that the principle of USM processing is to achieve image sharpening and edge enhancement by subtracting a blurred image (i.e., an unsharp mask) from the initial image. The unsharp mask is a blurred copy of the image. It is generated by rescaling the image to obtain more low-frequency structural information at the same contrast as the unsharp mask. This is equivalent to adding a high-pass filtered image to achieve image sharpening. Therefore, in order to maximize the retention of the texture information of the facial structure of the portrait, a small-kernel USM is used to superimpose relatively fine texture information into the portrait, so as to remove facial blemishes and achieve the skin smoothing effect while retaining more facial structure information.

[0093] Furthermore, in step 105 , the residual between the face region and the first low-frequency image is added to the first reference image to obtain a second reference image of the face region.

[0094] Specifically, in step 105, the step of superimposing the residual between the facial region and the first low-frequency image onto the first reference image to obtain the second reference image of the facial region can be implemented by linearly superimposing the residual onto the first reference image to obtain the second reference image of the facial region. Specifically, the sharpening information (i.e., the residual between the facial region and the low-frequency image) is superimposed onto the first reference image using linear light superposition to obtain the second reference image used to optimize the facial region. This allows the texture information of the facial structure of the portrait to be superimposed onto the portrait in the form of a residual, thereby removing facial blemishes and achieving a smoothing effect while preserving more facial structural information.

[0095] For example, assuming that the first reference image is R1, and assuming that the sharpening information is the residual Hp obtained by the above steps, then the residual Hp can be superimposed on the first reference image R1 using linear light superposition processing to obtain a face image F with superimposed texture details (i.e., the portrait skin refining result) using the following formula:

[0096]

[0097] Furthermore, in 106, the image to be processed is subjected to skin smoothing processing based on the second reference image to obtain a skin smoothing image including the facial area. In an optional embodiment, the second reference image and the facial area are fused into a third reference image of the facial area through a second mask image; the third reference image is drawn into the facial area of ​​the image to be processed to output a skin smoothing image. Specifically, the third reference image is replaced into the facial area of ​​the image to be processed, and the skin smoothing image of the image to be processed is output, thereby optimizing the skin smoothing effect of the facial area. In the present disclosure, the second mask image is used to indicate the weight relationship between the image information in the second reference image and the image information in the facial area. The facial features area in the second mask image uses a higher proportion of the image information of the facial area. The second mask image can retain more of the original image information of the facial area in the facial features area, thereby making the facial features area clearer.

[0098] pass Figure 1 In the illustrated image processing method, for a facial region in an image to be processed, low-frequency images corresponding to different facial structures are extracted from the facial region. These low-frequency images are then fused into a first reference image based on their respective facial structures. By fusing the low-frequency images corresponding to the different facial structures in the facial region, facial blemishes are removed during the portrait skin refining process, improving facial uniformity. Furthermore, a residual is obtained between the facial region and the first low-frequency image corresponding to the entire facial region in the low-frequency image. This residual is then superimposed on the first reference image to produce a second reference image of the facial region. This method, while blurring the facial blemishes, preserves as much facial structure information as possible, achieving a sharpened portrait. This avoids issues such as loss of skin texture, a strong false appearance, and a loss of three-dimensionality that can occur with a single type of blurring, further enhancing the three-dimensionality of the portrait and optimizing the skin refining result. Finally, skin refining is performed on the image to be processed based on the second reference image, producing a skin refining image containing the facial region, optimizing the skin refining result.

[0099] Figure 3 An image processing device is provided in an embodiment of the present disclosure. Figure 3 As shown, the device includes:

[0100] An acquisition unit 301 is configured to acquire an image to be processed, where the image to be processed includes a face area;

[0101] The fusion unit 302 is configured to extract low-frequency images corresponding to different facial structures from the face region; and fuse the low-frequency images into a first reference image according to the facial structures corresponding to each of the images.

[0102] The superposition unit 303 is configured to obtain a residual between the face region and a first low-frequency image corresponding to the entire face region in the low-frequency image; and superimpose the residual onto the first reference image to obtain a second reference image of the face region;

[0103] The skin smoothing unit 304 is configured to perform skin smoothing on the image to be processed based on the second reference image to obtain a skin smoothed image containing a facial area.

[0104] Optionally, the acquisition unit 301 is specifically configured to: obtain spatial coordinate information of the face area, the spatial coordinate information including the position coordinates of the face key points and the face pitch angle; generate a facial bounding box based on the position coordinates of the face key points and the face pitch angle; and use the facial bounding box to extract the face area from the image to be processed.

[0105] Optionally, in the process of generating the facial bounding box based on the position coordinates of the facial key points and the face pitch angle, the acquiring unit 301 is specifically configured as follows:

[0106] The position coordinates of the facial key points are linearly expanded to obtain the position coordinates of the extension points; the width of the facial bounding box is calculated based on the position coordinates of the facial key points and the position coordinates of the extension points; the height of the facial bounding box is calculated based on the face pitch angle; and the facial bounding box is generated based on the width and height.

[0107] Optionally, in the process of extracting low-frequency images corresponding to different facial structures from the face region, the fusion unit 302 is specifically configured as follows:

[0108] Gaussian blur processing is performed on the face area to obtain a first low-frequency image; mean blur processing is performed on the first low-frequency image to obtain a second low-frequency image corresponding to the cheeks and / or forehead in the face area; surface blur processing is performed on the first low-frequency image to obtain a third low-frequency image corresponding to the facial features in the face area.

[0109] Optionally, in the process of performing Gaussian blur processing on the face area to obtain the first low-frequency image, the fusion unit 302 is specifically configured to: perform Gaussian blur processing on the overall facial structure in the face area according to the first blur radius to obtain the first low-frequency image.

[0110] In the process of performing mean blurring on the first low-frequency image to obtain a second low-frequency image corresponding to the cheek and / or forehead in the face region, the fusion unit 302 is specifically configured to perform mean blurring on the first low-frequency image according to a second blurring radius to obtain the second low-frequency image, where the second blurring radius is greater than the first blurring radius.

[0111] In the process of performing surface blurring on the first low-frequency image to obtain a third low-frequency image corresponding to the facial features in the face region, the fusion unit 302 is specifically configured to perform surface blurring on the first low-frequency image according to a third blurring radius to obtain the third low-frequency image, where the third blurring radius is greater than the first blurring radius.

[0112] Optionally, in the process of fusing the low-frequency images into the first reference image according to their corresponding facial structures, the fusion unit 302 is specifically configured as follows:

[0113] A weight relationship between the second low-frequency image and the third low-frequency image is determined based on the facial structures corresponding to each of the second low-frequency image and the third low-frequency image to obtain a first mask image indicating the weight relationship, wherein the second low-frequency image has a higher weight in a cheek region and / or a forehead region of the first mask image than the third low-frequency image, and the third low-frequency image has a higher weight in a facial feature region of the first mask image than the second low-frequency image; and the second low-frequency image and the third low-frequency image are fused into a first reference image using the first mask image.

[0114] Optionally, in the process of superimposing the residual into the first reference image to obtain the second reference image of the face area, the superimposing unit 303 is specifically configured to: superimpose the residual linear light into the first reference image to obtain the second reference image.

[0115] The skin smoothing unit 304 is specifically configured to: fuse the second reference image with the facial area to obtain a third reference image of the facial area, wherein the second mask image is used to indicate the weight relationship between the image information in the second reference image and the image information in the facial area; and draw the third reference image into the facial area of ​​the image to be processed to output a skin smoothing image.

[0116] The above-mentioned image processing device can execute the systems or methods provided in the aforementioned embodiments. For parts not described in detail in this embodiment, please refer to the relevant descriptions of the aforementioned embodiments and will not be repeated here.

[0117] In a possible design, the structure of the above-mentioned image processing device can be implemented as an electronic device. Figure 4 As shown, the electronic device may include: a processor 21 and a memory 22. The memory 22 stores executable codes, and when the executable codes are executed by the processor 21, the processor 21 can at least implement the image processing method provided in the above embodiments.

[0118] The electronic device may further include a communication interface 23 for communicating with other devices or a communication network.

[0119] The present disclosure also provides a computer-readable storage medium including instructions, wherein the medium stores executable code. When the executable code is executed by a processor of a wireless router, the processor executes the neural network-based feature data processing methods provided in the aforementioned embodiments. Alternatively, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0120] In an exemplary embodiment, a computer program product is also provided, including a computer program. When the computer program is executed by a processor, the neural network-based feature data processing method provided in the aforementioned embodiments is implemented. The computer program is implemented by a program running on a terminal or a server.

[0121] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0122] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed, wherein the image to be processed includes a face area; extracting low-frequency images corresponding to different facial structures from the face region; fusing the low-frequency images into a first reference image according to their corresponding facial structures; obtaining a residual between the face region and a first low-frequency image corresponding to the entire face region in the low-frequency image; superimposing the residual onto the first reference image to obtain a second reference image of the face area; performing skin resurfacing processing on the image to be processed based on the second reference image to obtain a skin resurfacing image containing the facial area; Extracting low-frequency images corresponding to different facial structures from the facial region includes: performing Gaussian blur processing on the facial region to obtain a first low-frequency image; performing mean blur processing on the first low-frequency image to obtain a second low-frequency image corresponding to cheeks and / or forehead in the facial region; and performing surface blur processing on the first low-frequency image to obtain a third low-frequency image corresponding to facial features in the facial region.

2. The method according to claim 1, characterized in that The step of obtaining an image to be processed includes: Acquire spatial coordinate information of the face area, wherein the spatial coordinate information includes position coordinates of key points of the face and a pitch angle of the face; Generate a facial bounding box based on the position coordinates of the facial key points and the facial pitch angle; The face region is extracted from the image to be processed using the face circumscribed frame.

3. The method according to claim 2, characterized in that The generating of a facial bounding box based on the position coordinates of the facial key points and the facial pitch angle includes: Linearly extending the position coordinates of the facial key points to obtain the position coordinates of the extended points; Calculating the width of the facial circumscribed frame according to the position coordinates of the facial key points and the position coordinates of the extension points; Calculating the height of the facial circumscribed frame according to the face pitch angle; The face circumscribed frame is generated based on the width and the height.

4. The method according to claim 1, wherein The performing Gaussian blur processing on the face region to obtain a first low-frequency image includes: Performing Gaussian blur processing on the overall facial structure in the face region according to a first blur radius to obtain the first low-frequency image; The performing mean blurring processing on the first low-frequency image to obtain a second low-frequency image corresponding to the cheek and / or forehead in the face region includes: performing mean blurring on the first low-frequency image according to a second blurring radius to obtain the second low-frequency image, wherein the second blurring radius is larger than the first blurring radius; The performing surface blurring processing on the first low-frequency image to obtain a third low-frequency image corresponding to the facial features in the face region includes: The first low-frequency image is subjected to surface blurring processing according to a third blurring radius to obtain the third low-frequency image, where the third blurring radius is greater than the first blurring radius.

5. The method according to claim 1, wherein The step of fusing the low-frequency images into a first reference image according to their corresponding facial structures includes: determining a weight relationship between the second low-frequency image and the third low-frequency image based on facial structures corresponding to each of the second low-frequency image and the third low-frequency image, to obtain a first mask image indicating the weight relationship, wherein the second low-frequency image has a higher weight in a cheek region and / or a forehead region of the first mask image than the third low-frequency image, and the third low-frequency image has a higher weight in a facial feature region of the first mask image than the second low-frequency image; The second low-frequency image and the third low-frequency image are fused into the first reference image using the first mask image.

6. The method according to claim 1, wherein The step of superimposing the residual onto the first reference image to obtain a second reference image of the face region includes: superimposing the residual linear light onto the first reference image to obtain the second reference image; The step of performing skin smoothing on the image to be processed based on the second reference image to obtain a skin smoothed image containing the facial area includes: fusing the second reference image and the facial region into a third reference image of the facial region using a second mask, wherein the second mask is used to indicate a weight relationship between image information in the second reference image and image information in the facial region; The third reference image is drawn into the face region of the image to be processed to output the skin refining image.

7. An image processing device, characterized in that The device comprises: an acquisition unit, configured to acquire an image to be processed, wherein the image to be processed includes a face area; a fusion unit configured to extract low-frequency images corresponding to different facial structures from the face region; and fuse the low-frequency images into a first reference image according to the facial structures corresponding to each of the images; a superposition unit configured to obtain a residual between the face region and a first low-frequency image corresponding to the entire face region in the low-frequency image; and superimpose the residual onto the first reference image to obtain a second reference image of the face region; a skin refining unit configured to perform skin refining on the image to be processed based on the second reference image to obtain a skin refining image containing the facial region; In the process of extracting low-frequency images corresponding to different facial structures from the facial region, the fusion unit is specifically configured to: perform Gaussian blur processing on the facial region to obtain the first low-frequency image; perform mean blur processing on the first low-frequency image to obtain a second low-frequency image corresponding to the cheeks and / or forehead in the facial region; and perform surface blur processing on the first low-frequency image to obtain a third low-frequency image corresponding to the facial features in the facial region.

8. The device according to claim 7, characterized in that The acquisition unit is specifically configured to: Acquire spatial coordinate information of the face area, wherein the spatial coordinate information includes position coordinates of key points of the face and a pitch angle of the face; Generate a facial bounding box based on the position coordinates of the facial key points and the facial pitch angle; The face region is extracted from the image to be processed using the face circumscribed frame.

9. The device according to claim 8, characterized in that In the process of generating the facial circumscribed frame based on the position coordinates of the facial key points and the facial pitch angle, the acquisition unit is specifically configured as follows: Linearly extending the position coordinates of the facial key points to obtain the position coordinates of the extended points; Calculating the width of the facial circumscribed frame according to the position coordinates of the facial key points and the position coordinates of the extension points; Calculating the height of the facial circumscribed frame according to the face pitch angle; The face circumscribed frame is generated based on the width and the height.

10. The device according to claim 7, characterized in that The fusion unit performs Gaussian blur processing on the face area to obtain the first low-frequency image, and is specifically configured as follows: Performing Gaussian blur processing on the overall facial structure in the face region according to a first blur radius to obtain the first low-frequency image; In the process of performing mean blurring on the first low-frequency image to obtain a second low-frequency image corresponding to the cheek and / or forehead in the face region, the fusion unit is specifically configured as follows: performing mean blurring on the first low-frequency image according to a second blurring radius to obtain the second low-frequency image, wherein the second blurring radius is larger than the first blurring radius; The fusion unit is specifically configured to perform surface blurring on the first low-frequency image to obtain a third low-frequency image corresponding to the facial features in the face region as follows: The first low-frequency image is subjected to surface blurring processing according to a third blurring radius to obtain the third low-frequency image, where the third blurring radius is greater than the first blurring radius.

11. The device according to claim 7, characterized in that In the process of fusing the low-frequency images into the first reference image according to their corresponding facial structures, the fusion unit is specifically configured as follows: determining a weight relationship between the second low-frequency image and the third low-frequency image based on facial structures corresponding to each of the second low-frequency image and the third low-frequency image, to obtain a first mask image indicating the weight relationship, wherein the second low-frequency image has a higher weight in a cheek region and / or a forehead region of the first mask image than the third low-frequency image, and the third low-frequency image has a higher weight in a facial feature region of the first mask image than the second low-frequency image; The second low-frequency image and the third low-frequency image are fused into the first reference image using the first mask image.

12. The device according to claim 7, characterized in that In the process of the superimposing unit superimposing the residual onto the first reference image to obtain the second reference image of the face area, the superimposing unit is specifically configured as follows: superimposing the residual linear light onto the first reference image to obtain the second reference image; The skin refining unit performs skin refining on the image to be processed based on the second reference image to obtain the skin refining image containing the facial area, and is specifically configured as follows: fusing the second reference image and the facial region into a third reference image of the facial region using a second mask, wherein the second mask is used to indicate a weight relationship between image information in the second reference image and image information in the facial region; The third reference image is drawn into the face region of the image to be processed to output the skin refining image.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 6. 14 . A computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by an electronic device, the electronic device is enabled to execute the image processing method according to claim 1 .

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images