Face image optimization method and device, storage medium and electronic device

Abstract

The present disclosure relates to the field of image processing technology, and specifically to a facial image optimization method and device, a computer-readable storage medium, and an electronic device, the method comprising: obtaining a facial image to be beautified, processing the facial image to be beautified using a pre-trained image optimization model, and obtaining a reference fusion degree image; obtaining a reference facial image and a target skin smoothing degree corresponding to the facial image to be beautified; wherein the reference facial image is an image with the maximum skin smoothing degree corresponding to the facial image to be beautified; calculating a target fusion degree matrix corresponding to the target skin smoothing degree according to the reference fusion degree image; and determining a target facial image according to the target fusion degree matrix, the facial image to be beautified, and the reference facial image. The technical solution of the embodiments of the present disclosure improves the image optimization effect.

Description

Background Art

[0002] Beauty refers to the use of image processing technology to beautify portraits in images or videos to better meet the user's aesthetic needs.

[0003] In the related art, image optimization generally includes two steps, namely, blemish removal and skin smoothing. However, the accuracy of the skin smoothing process in the related art is poor, resulting in poor image optimization effect.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a face image optimization method, a face image optimization device, a computer-readable medium and an electronic device, thereby improving the image optimization effect at least to a certain extent.

[0006] According to a first aspect of the present disclosure, a facial image optimization method is provided, comprising: obtaining a facial image to be beautified, processing the facial image to be beautified using a pre-trained image optimization model to obtain a baseline fusion degree image; obtaining a reference facial image and a target skin smoothing degree corresponding to the facial image to be beautified; wherein the reference facial image is an image with the maximum skin smoothing degree corresponding to the facial image to be beautified; calculating a target fusion degree matrix corresponding to the target skin smoothing degree according to the baseline fusion degree image; and determining a target facial image according to the target fusion degree matrix, the facial image to be beautified, and the reference facial image.

[0007] According to a second aspect of the present disclosure, a facial image optimization device is provided, including: a first acquisition module, used to acquire a facial image to be beautified, and to process the facial image to be beautified using a pre-trained image optimization model to obtain a baseline fusion degree image; a second acquisition module, used to acquire a reference facial image and a target skin smoothing degree corresponding to the facial image to be beautified; wherein the reference facial image is an image with the maximum skin smoothing degree corresponding to the facial image to be beautified; a matrix calculation module, used to calculate a target fusion degree matrix corresponding to the target skin smoothing degree according to the baseline fusion degree image; and an image optimization module, used to determine a target facial image according to the target fusion degree matrix, the facial image to be beautified, and the reference facial image.

[0008] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which enables the one or more processors to implement the above method when the one or more programs are executed by the one or more processors.

[0010] A facial image optimization method provided by an embodiment of the present disclosure calculates a baseline fusion degree image corresponding to the facial image to be beautified through an image optimization model, and calculates a target fusion degree matrix corresponding to the target skin smoothing degree based on the baseline fusion degree image. The calculation of the target fusion degree matrix is ​​realized through the processing of a deep neural network. On the one hand, the flexibility of image optimization is increased, and on the other hand, different fusion coefficients can be used for different areas, thereby enhancing the skin smoothing accuracy and thus improving the optimization effect of the facial image.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0013] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0014] Figure 2 A flowchart schematically illustrates a method for optimizing a facial image in an exemplary embodiment of the present disclosure;

[0015] Figure 3 A flowchart of obtaining a facial image to be beautified in an exemplary embodiment of the present disclosure is schematically shown;

[0016] Figure 4 A schematic diagram of the structure of an image optimization model in an exemplary embodiment of the present disclosure is schematically shown;

[0017] Figure 5 Schematically illustrates a flow chart of processing an image using an image optimization model in an exemplary embodiment of the present disclosure;

[0018] Figure 6 A flowchart for obtaining a target blemish-free face image in an exemplary embodiment of the present disclosure is schematically shown;

[0019] Figure 7A flowchart for calculating a target fusion degree matrix in an exemplary embodiment of the present disclosure is schematically shown;

[0020] Figure 8 A flowchart for adjusting high-frequency image pixel values ​​in an exemplary embodiment of the present disclosure is schematically shown;

[0021] Fig. 9 A schematic diagram schematically illustrates a gradual change process of a boundary area in an exemplary embodiment of the present disclosure;

[0022] Fig.10 A schematic flow chart showing a method for beautifying the face in this exemplary embodiment;

[0023] Fig.11 A schematic diagram schematically shows the composition of a face image optimization device in an exemplary embodiment of the present disclosure;

[0024] Fig.12 A schematic diagram of an electronic device to which the embodiments of the present disclosure can be applied is shown. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0027] The skin refining method commonly used in related technologies usually requires calculating the skin probability image, and then obtaining a 100% smooth image based on the fusion calculation of the skin probability image, that is, the image with the greatest skin refining degree, and obtaining the skin refining image based on the fusion calculation of the skin refining degree set by the user. The skin refining degree set by the user is consistent in all skin areas of the original image, that is, the degree of fusion of the rough skin area and the smooth skin area of ​​the original image is the same, which can easily lead to problems such as insufficient skin refining in the rough area and still rough, and excessive skin refining in the smooth area and blurring. It cannot adapt to users with different skin types at the same time. And because the skin probability image can easily misidentify skin-like hair and clothing, it leads to blurry hair and clothing effects.

[0028] In view of the above problems, the exemplary embodiments of the present disclosure first provide an image optimization method, and its application scenarios include but are not limited to: the terminal device is installed with an image optimization App (Application), and the user selects an image in the local album in the App for optimization processing, or optimizes the currently captured image; the terminal device executes the image optimization processing method of this exemplary embodiment, or the terminal device sends the image to the server, and the server executes the image optimization processing method of this exemplary embodiment to optimize the image. Alternatively, the video selected by the user or the currently captured video can also be optimized, specifically, the frames containing portraits in the video can be optimized, for example, in a live broadcast scenario, the real-time video stream can be optimized. The specific system architecture can be referred to. Figure 1 shown.

[0029] Figure 1 The schematic diagram of the system architecture is shown, and the system architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, etc. The server 120 generally refers to a background system that provides image optimization related services in this exemplary embodiment, and may be a server or a cluster formed by multiple servers. The terminal 110 and the server 120 may be connected via a wired or wireless communication link to perform data exchange.

[0030] In one implementation, the facial image optimization method may be performed by the terminal 110. For example, after a user uses the terminal 110 to take a photo or the user selects a facial image to be beautified from an album of the terminal 110, the terminal 110 performs image optimization processing on the image and outputs a target facial image.

[0031] In one implementation, the facial image optimization method may be performed by the server 120. For example, after a user uses the terminal 110 to take a picture or the user selects a facial image to be beautified in the photo album of the terminal 110, the terminal 110 uploads the image to the server 120, and the server 120 performs image optimization processing on the image and returns the target facial image to the terminal 110.

[0032] As can be seen from the above, the execution entity of the facial image optimization method in this exemplary embodiment may be the above-mentioned terminal 110 or server 120, and the present disclosure does not limit this.

[0033] Combine the following Figure 2 The face image optimization method in this exemplary embodiment is described. Figure 2 An exemplary process of the face image optimization method is shown, which may include:

[0034] Step S210, obtaining a face image to be beautified, and processing the face image to be beautified using a pre-trained image optimization model to obtain a reference fusion degree image;

[0035] Step S220, obtaining a reference face image corresponding to the face image to be beautified and a target skin smoothing degree; wherein the reference face image is an image with the maximum skin smoothing degree corresponding to the face image to be beautified;

[0036] Step S230, calculating a target fusion degree matrix corresponding to the target skin resurfacing degree according to the reference fusion degree image;

[0037] Step S240: determining a target facial image according to the target fusion degree matrix, the facial image to be beautified, and the reference facial image.

[0038] Based on the above method, the baseline fusion degree image corresponding to the facial image to be beautified is calculated through the image optimization model, and the target fusion degree matrix corresponding to the target skin smoothing degree is calculated according to the baseline fusion degree image. The calculation of the target fusion degree matrix is ​​realized through the processing of the deep neural network. On the one hand, the flexibility of image optimization is increased, and on the other hand, different fusion coefficients can be used for different areas, thereby enhancing the skin smoothing accuracy and thus improving the optimization effect of the facial image.

[0039] Below Figure 2 Each step in the procedure is described in detail.

[0040] refer to Figure 2 In step S210, a face image to be beautified is obtained, and the face image to be beautified is processed using a pre-trained image optimization model to obtain a reference fusion degree image.

[0041] In this example implementation, the above-mentioned benchmark fusion degree image is used to characterize the degree of fusion between each pixel point in the face image to be beautified corresponding to the benchmark skin smoothing degree and the reference face image, wherein the benchmark skin smoothing degree image can be a single-channel image, wherein the pixel value of each pixel in the benchmark fusion degree image characterizes the degree of fusion between the pixel points of the face image to be beautified and the reference face image.

[0042] Among them, the above-mentioned reference face image represents the image after the above-mentioned face image to be beautified is subjected to the maximum skin smoothing degree. The maximum skin smoothing degree can be 100% or 90%, and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0043] In an exemplary embodiment, obtaining the facial image to be beautified may include step S310 and step S320.

[0044] In step S310, an image to be processed is obtained, and at least one face image to be optimized is extracted from the image to be processed according to a preset rule.

[0045] In this example implementation, the image to be processed may be an image selected by the user, such as an image selected by the user in an album, or an image automatically designated by the system, such as a currently captured image.

[0046] The face image to be optimized is a sub-image obtained by intercepting the face part in the image to be processed. This exemplary embodiment mainly optimizes the face in the image to be processed, and does not limit the number of faces in the image to be processed. For example, when the image to be processed includes multiple faces, multiple face images to be optimized can be extracted.

[0047] In one implementation, extracting at least one face image to be optimized from the image to be processed according to a preset rule may include:

[0048] Generate one or more face frames in the image to be processed according to the facial key points identified in the image to be processed;

[0049] The face frame whose area is greater than or equal to the face area threshold is retained, and the image in the face frame is intercepted to obtain one or more face images to be optimized. Among them, the key points of the face may include the key parts of the face and the points on the edge of the face, and the key points of each face are in the face frame. The face frame can be a rectangular frame, or a circular frame, a hexagonal frame, etc., and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0050] In one implementation, the face frame may be a minimum rectangular frame including key points of the face.

[0051] Generally, all faces can be detected in the image to be processed by the face detection algorithm, which may include faces that do not need beautification (such as the faces of passers-by in the distance). Considering that in the scene of image beautification, it is usually necessary to beautify larger faces (the effect of beautification on smaller faces is not obvious), the face frame can be filtered by the face area threshold. Specifically, the face area threshold can be set according to experience or the size of the image to be processed. For example, the face area threshold can be the size of the image to be processed * 0.05; if the area of ​​the face frame is greater than or equal to the face area threshold, it is a face that needs beautification and the face frame is retained; if the area of ​​the face frame is less than the face area threshold, it is a face that does not need beautification and the face frame is deleted.

[0052] After filtering the face frames, the retained face frames are the face frames of valid faces. The images in each face frame are intercepted to obtain the same number of face images to be optimized as the number of face frames.

[0053] In one embodiment, in order to facilitate the subsequent combination of face images to be optimized, an upper limit on the number of face images to be optimized, that is, an upper limit on the number of face frames, can be set. For example, it can be set to 4. If the number of retained face frames is greater than 4 after filtering by the above face area threshold, 4 face frames can be selected from them, such as the 4 face frames with the largest area, or the 4 face frames closest to the center of the image to be processed, and 4 face images to be optimized are correspondingly intercepted, and the faces in other face frames are not optimized; or multiple optimization processes can be performed, in which 4 face frames are selected and face images to be optimized are intercepted for optimization in this process, and other face frames are selected and face images to be optimized are intercepted for optimization in the next process, so as to complete the optimization of faces in all face frames in the image to be processed whose area is greater than the face area threshold.

[0054] In one embodiment, before capturing the image in the face frame, the face frame may be enlarged so that the face frame includes a small amount of area other than the face, so as to facilitate gradient processing during subsequent image fusion. When performing the enlargement process, the face frame may be enlarged in one or more directions according to a preset ratio. For example, the preset ratio is 1.1, and the face frame is uniformly enlarged in all directions so that the size of the enlarged face frame is 1.1 times the original size. It should be noted that when the face frame is enlarged, if one or more boundaries of the face frame reach the boundary of the image to be processed, the boundary of the face frame is stopped at the boundary of the image to be processed.

[0055] In step S320, the face image to be optimized is preprocessed to obtain an initial face image.

[0056] In this example implementation, the output size of a model may be set, the size of each face image to be optimized obtained above may be determined, and the initial face image may be preprocessed according to the output size to obtain the face image to be beautified. The preprocessing may include one or more of the following.

[0057] When the size relationship between the width and height of the face image to be optimized is different from the size relationship between the width and height of the output size of the model, the face image to be optimized is rotated 90 degrees. Specifically, in the input size of the face image to be optimized and the model, the width is greater than the height or the width is less than the height, then the size relationship between the width and height of the face image to be optimized and the input size of the model is the same, and there is no need to rotate the face image to be optimized; otherwise, the size relationship between the width and height of the face image to be optimized and the input size of the model is different, and the face image to be optimized needs to be rotated 90 degrees (either clockwise or counterclockwise). For example, when the input size of the model is 320 wide and 448 high, that is, the width is less than the height, if the face image to be optimized is wider than the height, the face image to be optimized is rotated 90 degrees.

[0058] In one implementation, in order to maintain the angle of the human face in the human face image to be optimized, the human face image to be optimized may not be rotated.

[0059] When the size of the face image to be optimized is larger than the input size of the model, the face image to be optimized is downsampled according to the input size of the model. The size of the face image to be optimized is larger than the input size of the model, which means that the width of the face image to be optimized is larger than the width of the input size of the model, or the height of the face image to be optimized is larger than the height of the input size of the model. In the image beautification scenario, the image to be processed is generally a clear image taken by the terminal device, and its size is large. Therefore, it is a common situation that the size of the face image to be optimized is larger than the input size of the model, that is, it is usually necessary to downsample the face image to be optimized.

[0060] Down sampling can be achieved by using methods such as bilinear interpolation and nearest neighbor interpolation, which is not limited in the present disclosure.

[0061] After downsampling, at least one of the width and height of the face image to be optimized is aligned with the input size of the model, including the following situations:

[0062] The width and height of the face image to be optimized are the same as the input size of the model;

[0063] The width of the face image to be optimized is the same as the width of the input size of the model, and the height is smaller than the height of the input size of the model;

[0064] The height of the face image to be optimized is the same as the height of the input size of the model, and the width is smaller than the width of the input size of the model.

[0065] It should be noted that if the facial image to be optimized has been rotated as mentioned above to obtain a rotated facial image to be optimized, then when the size of the facial image to be optimized is larger than the input size of the model, it is downsampled according to the input size of the model. The specific implementation method is the same as the downsampling method of the facial image to be optimized as mentioned above, so it will not be repeated here.

[0066] On the contrary, when the size of the face image to be optimized (or the rotated face image to be optimized) is less than or equal to the input size of the model, the downsampling processing step may not be performed.

[0067] When the size of the face image to be optimized is smaller than the input size of the model, the face image to be optimized is padded according to the difference between the input size of the face image to be optimized and the model, so that the size of the padded face image to be optimized is equal to the input size of the model. The size of the face image to be optimized is smaller than the input size of the model, which means that at least one of the width and height of the face image to be optimized is smaller than the input size of the model, and the other is not larger than the input size of the model, which specifically includes the following situations:

[0068] The width of the face image to be optimized is smaller than the width of the model's input size, and the height is also smaller than the height of the model's input size;

[0069] The width of the face image to be optimized is smaller than the width of the model's input size, and the height is equal to the height of the model's input size;

[0070] The height of the face image to be optimized is smaller than the height of the input size of the model, and the width is equal to the height of the input size of the model.

[0071] Preset pixel values ​​can be used for filling, which are usually pixel values ​​that are very different from the color of the face, such as (R0, G0, B0), (R255, G255, B255), etc.

[0072] Generally, the face image to be optimized can be filled around, for example, the center of the face image to be optimized is overlapped with the center of the input size of the model, and the difference part around the face image to be optimized is filled, so that the size of the face image to be optimized after filling is consistent with the input size of the model. Of course, the face image to be optimized can also be aligned with one side edge of the input size of the model, and the other side is filled. This disclosure does not limit this.

[0073] It should be noted that if the facial image to be optimized has been subjected to at least one of the above-mentioned rotation and downsampling processing to obtain a facial image to be optimized that has been subjected to at least one of the rotation and downsampling processing, then when the size of the facial image to be optimized is smaller than the input size of the model, it is filled according to the difference between the size of the facial image to be optimized and the input size of the model. The specific implementation method is the same as the filling method of the above-mentioned facial image to be optimized, and therefore it will not be repeated here.

[0074] In an example implementation of the present disclosure, corresponding transformation information may be saved, including but not limited to: the direction and angle of rotation of each face image to be optimized, the ratio of downsampling, and the coordinates of the filled pixels, so as to facilitate subsequent inverse transformation according to the transformation information.

[0075] After the above preprocessing, the face image to be optimized can obtain the above face image to beautify.

[0076] In an example implementation of the present disclosure, the processing of the to-beautify face image using a pre-trained image optimization model to obtain a reference fusion degree image may include processing the to-beautify face image using a pre-trained image optimization model to obtain a reference fusion degree image and a target blemish-free face image;

[0077] In this example implementation, in one implementation, the above image optimization model can use a lightweight deep neural network to reduce the amount of calculation and realize the learning and processing of image depth features. Exemplarily, the deep neural network can use an end-to-end structure to realize pixel-level processing of beauty face images.

[0078] Figure 4 An exemplary structure of a deep neural network is shown. Figure 4 As shown, the above-mentioned image optimization model can be a fully convolutional network, including: a first pixel rearrangement layer, at least one convolutional layer, at least one transposed convolutional layer, and a second pixel rearrangement layer.

[0079] based on Figure 4 The deep neural network shown in Figure 5 As shown, the pre-trained image optimization model is used to process the face image to be beautified, and the baseline fusion degree image can be obtained by Figure 5 Steps S510 to S540 in the embodiment are implemented as follows:

[0080] Step S510: using a first pixel rearrangement layer to perform pixel rearrangement processing on the face image to be beautified from a single channel to multiple channels to obtain a first feature image.

[0081] It should be noted that the facial image to be beautified may be a single-channel image (such as a grayscale image) or a multi-channel image (such as an RGB image). The first pixel rearrangement layer may rearrange each channel of the facial image to be beautified into multiple channels.

[0082] In an optional implementation, step S510 includes:

[0083] Inputting the face image to be beautified with a channel number a into the first pixel rearrangement layer;

[0084] The pixels in each n*n neighborhood in each channel of the facial image to be beautified are rearranged to the same positions in the n*n channels, and a first feature image with a*n*n channel number is output.

[0085] Where a represents the number of channels of the face image to be beautified, which is a positive integer; n represents the parameter of pixel rearrangement, which is a positive integer not less than 2. Taking n=2 as an example, traverse the first channel of the face image to be beautified, usually starting from the upper left corner, extract the pixels of each 2*2 grid, and rearrange them to the same position in the four channels, thereby turning one channel into four channels, and at the same time reducing the width and height of the image to half, and record the rearranged image as the first feature image; the same method is used to process other channels. If the face image to be beautified is a single-channel image, the first feature image of the four channels is obtained after the pixels are rearranged; if the face image to be beautified is a three-channel image, the first feature image of the twelve channels is obtained after the pixels are rearranged.

[0086] The first pixel rearrangement layer can be implemented by using the space_to_depth function in TensorFlow (a machine learning implementation framework) to convert the spatial features in the face image to be beautified into depth features. It can also be implemented by using a convolution operation with a step size of n. In this case, the first pixel rearrangement layer can be regarded as a special convolution layer.

[0087] Step S520: performing convolution processing on the first feature image using a convolution layer to obtain a second feature image.

[0088] The present disclosure does not limit the number of convolutional layers, the size of convolutional kernels, the specific structure of convolutional layers, etc. The convolutional layer is used to extract image features from different scales and learn depth information. The convolutional layer may include a matching pooling layer for downsampling the convolutional image to achieve information abstraction, increase the receptive field, and reduce parameter complexity.

[0089] When multiple convolutional layers are set, a step-by-step convolution and downsampling method can be used. For example, the image can be reduced by a factor of 2 until the last convolutional layer outputs a second feature image. The second feature image can be the feature image with the smallest size in the deep neural network processing process.

[0090] Step S530: Use a transposed convolution layer to perform a transposed convolution process on the second feature image to obtain a third feature image.

[0091] The present disclosure does not limit the number of transposed convolution layers, the size of the transposed convolution kernel, the specific structure of the transposed convolution layer, etc. The transposed convolution layer is used to upsample the second feature image, which can be regarded as the reverse process of convolution, thereby restoring the size of the image.

[0092] When multiple transposed convolutional layers are set, a stepwise upsampling method may be used, for example, the image may be upsampled by a factor of 2 until the last transposed convolutional layer outputs the third feature image.

[0093] In an optional implementation, the convolution layer and the transposed convolution layer are completely symmetrical structures, and the third feature image has the same size and the same number of channels as the first feature image.

[0094] In an optional implementation, a direct connection can be established between the convolutional layer and the transposed convolutional layer, such as Figure 5 As shown, a direct connection is established between the convolution layer and the transposed convolution layer corresponding to the feature image of the same size, thereby directly connecting the feature image information of the convolution link to the feature image in the transposed convolution link, which is conducive to obtaining a third feature image with more comprehensive information.

[0095] Step S540: using the second pixel rearrangement layer to perform pixel rearrangement processing on the third feature image from multi-channel to single-channel, to obtain a reference fusion degree image.

[0096] It should be noted that the facial image to be beautified may be a single-channel image (such as a grayscale image) or a multi-channel image (such as an RGB image). The second pixel rearrangement layer may rearrange each channel of the facial image to be beautified into multiple channels. In an optional implementation, step S640 includes:

[0097] Input the third feature image with a channel number of b*n*n into the second pixel rearrangement layer;

[0098] Rearrange the pixels at the same position in each n*n channel of the third feature image to the n*n neighborhood in a single channel, and output a reference fusion degree image with b channels;

[0099] Wherein, b is a positive integer.

[0100] The second pixel rearrangement layer can be implemented by using the depth_to_space function in TensorFlow to convert the depth features in the third feature image into spatial features. It can also be implemented by using a transposed convolution operation with a step size of n. In this case, the second pixel rearrangement layer can be regarded as a special transposed convolution layer.

[0101] If the convolution layer and the transposed convolution layer are completely symmetrical structures, that is, the size and number of channels of the third feature image are the same as those of the first feature image, then a = b. Furthermore, the number of channels of the reference fusion degree image and the face image to be beautified is also the same, that is, the processing of the deep neural network does not change the image size and number of channels.

[0102] In this example implementation, the image optimization model can also perform blemish removal processing, that is, the output of the image optimization model can also be used to calculate the target blemish-free face image corresponding to the face image to be beautified. The blemish removal effect depends on the quality of the data set and the training effect, and does not depend on the artificially designed image feature calculation.

[0103] In the summary of this example implementation, after being fully trained with a comprehensive data set, the deep neural network can cope with almost all situations in practical applications, including different lighting conditions, different skin conditions, etc., to accurately and fully detect and remove portrait blemishes, and solve the problem of unclean blemish removal in related technologies. At the same time, the obtained baseline fusion degree image can also characterize the degree of skin refining of each pixel, which may include different lighting conditions, different skin conditions, etc., and can achieve accurate skin refining processing.

[0104] In an example implementation of the present disclosure, processing the facial image to be beautified using a pre-trained image optimization model to obtain a reference fusion degree image and a target blemish-free facial image may include step S610 and step S620.

[0105] In step S610, the facial image to be beautified is input into the image optimization model to obtain the reference fusion degree image and the initial blemish-free facial image.

[0106] The specific execution process of the above-mentioned image optimization model has been introduced in detail above and therefore will not be repeated here. That is, after the beauty face image is input into the above-mentioned image optimization model, the image optimization model outputs a baseline fusion degree image and an initial blemish-free face image after executing the above-mentioned steps S510 to S540.

[0107] In step S620, the initial blemish-removed facial image and the face image to be beautified are fused according to the blemish-removal degree set by the user to obtain the target blemish-removed facial image.

[0108] In this exemplary embodiment, the degree of blemish removal can be a parameter for the current setting, a system default parameter, or a parameter used in the last beautification. After determining the degree of blemish removal, the initial blemish-removed face image and the face image to be beautified can be fused using the degree of blemish removal as a weight. For example, assuming that the range of the degree of blemish removal is 0 to 1 and the currently set value is a, refer to the following formula:

[0109] img=ori_img*(1.0-a)+deblemsih_img*a

[0110] Where img represents the target blemish-free face image, ori img_ represents the face image to be beautified, and deblemish_img represents the initial blemish-free face image. When a is 0, it means no blemish removal is performed, and the face image to be beautified is used completely; when a is 100, it means full blemish removal, and the initial blemish-free face image is used completely. Therefore, formula (1) indicates that an image between the face image to be beautified and the initial blemish-free face image is obtained through fusion. The larger a is, the closer the obtained image is to the initial blemish-free face image, that is, the higher the degree of beautification is, the more obvious the beautification effect is.

[0111] In this example implementation, the above-mentioned benchmark fusion degree image is used to determine the degree of fusion between each pixel in the face image to be beautified corresponding to the benchmark skin smoothing and the reference face image, wherein the benchmark fusion degree can be 60%, 30%, 40%, etc., and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0112] In this example implementation, the values ​​of the pixels in the reference fusion degree image may be between -255 and 255, or may be normalized values ​​directly used to represent the degree of skin resurfacing, which is not specifically limited in this example implementation.

[0113] Specifically, the value range of the skin smoothing fusion degree image can be set to [-1.0, 1.0]. Pixel value <0.0 represents non-skin area; pixel value = 0.0 represents the minimum skin smoothing, that is, the opacity is 0%, and the output pixel = the original image pixel; usually, the pixels of the facial features edge area of ​​the original image skin area will remain at 0.0, and the extremely smooth area of ​​the original image skin area will remain at 0.0; the original image value = 1.0 represents the maximum skin smoothing.

[0114] Similarly, the value range of the above-mentioned skin smoothing fusion degree image can also be set to [-255, 255]. Pixel value <0 represents non-skin area; pixel value = 0 represents the minimum skin smoothing, that is, the opacity is 0%, and the output pixel = the original image pixel; usually the pixels of the facial features edge area of ​​the original image skin area will remain at 0.0, and the extremely smooth area of ​​the original image skin area will remain at 0.0; the original image value = 255 represents the maximum skin smoothing.

[0115] In this example implementation, the image optimization model is obtained through training, and the training data may include a plurality of initial face images and reference skin resurfacing degree images corresponding to the initial face images. The reference skin resurfacing degree images may be obtained in the following manner.

[0116] The following takes the above-mentioned reference skin resurfacing degree of 60% as an example, that is, the reference fusion degree image is a 60% fusion degree image, and the value range of the skin resurfacing fusion degree image is set to [-1.0, 1.0], and the process of obtaining the above-mentioned reference fusion degree image is described in detail.

[0117] Specifically, you can first initialize and obtain a skin grinding 0% fusion degree image, that is, assign skin pixels to 0.0 and non-skin area pixels to -1.0; from the 0% fusion degree image, you can obtain a skin grinding 30% fusion degree image. Specifically, increase the opacity of the skin area pixels. Increase the opacity of the rough skin area pixels of the initial face image more, that is, increase the fusion degree, and get closer to a smooth 100% image, so as to achieve the purpose of smoothness. The pixels in the smooth skin area of ​​the original image are increased less. The skin area of ​​the original image that is sufficient for skin grinding to 30% smoothness is kept at 0.0, the facial features and their edge areas are kept at 0.0, and the non-skin area pixels are kept at -1.0. The above 30% can be used as the default skin grinding degree of the program.

[0118] Then, the image with 60% fusion degree of skin grinding is obtained from the image with 30% fusion degree of skin grinding. For the non-0.0 area of ​​the image with 30% fusion degree of skin grinding, 0.3 is added uniformly to obtain the image with 60% fusion degree of skin grinding. At this time, the non-0.0 area can be regarded as the standard consistent 60% fusion degree of skin grinding. Then, the opacity of the area with 0.0 value of the preliminary image with 60% fusion degree of skin grinding is adjusted. If the roughness of a sub-area in the area with 0.0 value is stronger than the pixel with 60% fusion degree of skin grinding in the current non-0.0 area, the pixel value of the sub-area is increased to achieve a consistent smoothness, and the increased value is not greater than 0.3. The other sub-areas in the area with 0.0 value still remain at 0.0. Thus, the image with 60% fusion degree of skin grinding is obtained.

[0119] Then, the image optimization model is trained based on the initial face image and the 60% fusion degree image corresponding to the initial face image.

[0120] In step S220, a reference facial image corresponding to the facial image to be beautified and a target skin smoothing degree are obtained; wherein the reference facial image is an image with the maximum skin smoothing degree corresponding to the facial image to be beautified.

[0121] In this example implementation, a reference facial image with the greatest degree of skin smoothing corresponding to the facial image to be beautified can be obtained, wherein the degree of skin smoothing of the reference facial image can be an image with 100% skin smoothing or an image with 90% skin smoothing, and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0122] In this example implementation, the target skin resurfacing degree may be a parameter currently set, a system default parameter, or a parameter used in the last skin resurfacing, etc., which is not specifically limited in this example implementation.

[0123] In step S230, a target fusion degree matrix corresponding to the target skin resurfacing degree is calculated according to the reference fusion degree image.

[0124] In an exemplary implementation of the present disclosure, the above steps may specifically include steps S710 to S730.

[0125] In step S710, a reference fusion degree matrix corresponding to the reference fusion image is determined.

[0126] In this example implementation, if the pixel value range of the above-mentioned benchmark fusion image is [-1.0, 1.0], the above-mentioned benchmark fusion image can be directly used as the benchmark fusion degree matrix. Otherwise, the pixel points in the above-mentioned benchmark fusion degree image can be normalized to obtain the above-mentioned benchmark fusion degree matrix. This example implementation does not make specific limitations on the above-mentioned normalization process.

[0127] In step S720, a target fusion degree matrix is ​​calculated according to the reference fusion degree matrix and the target skin resurfacing degree.

[0128] In this example implementation, a mapping relationship may be first determined according to the reference fusion degree matrix and the target skin resurfacing degree; and then the target fusion degree matrix may be calculated according to the reference fusion degree matrix, the mapping relationship and the target skin resurfacing degree.

[0129] Specifically, the absolute value of the difference between the target skin resurfacing degree and the reference skin resurfacing degree can be first obtained. When the reference skin resurfacing degree is greater than the target skin resurfacing degree, the pixel values ​​in the reference fusion degree matrix are subtracted from the absolute value to obtain the target fusion degree matrix. When the reference skin resurfacing degree is lower than the target skin resurfacing degree, the pixel values ​​in the reference fusion degree matrix are subtracted from the absolute value to obtain the target fusion degree matrix.

[0130] In step S730, the elements in the target fusion degree matrix are truncated so that the values ​​of the elements in the target fusion degree matrix are all within a first preset range.

[0131] In this example implementation, the elements in the target fusion degree matrix may be truncated so that the values ​​of the elements in the target fusion degree matrix are all within a first preset range, wherein the first preset range may be greater than or equal to 0 and less than or equal to 1. Here, the truncation process may be replacing elements less than or equal to 0 with 0 and replacing elements greater than or equal to 1 with 1.

[0132] In this exemplary implementation, the first preset range may also be greater than or equal to 0 and less than or equal to 0.9, and may also be customized according to user needs, which is not specifically limited in this exemplary implementation.

[0133] Step S240: determining a target facial image according to the target fusion degree matrix, the facial image to be beautified, and the reference facial image.

[0134] In an example implementation of the present disclosure, determining the target facial image according to the target fusion degree matrix, the facial image to be beautified, and the reference facial image may include step S710 and step S720.

[0135] In step S710, the target blemish-removed face image is inversely processed to obtain a face image to be fused, wherein the inverse processing is a reverse operation of the preprocessing.

[0136] In this example implementation, the face image to be beautified is obtained by optimizing the face image to be optimized. Before fusion, the target blemish-free face image can be inversely processed to obtain the face image to be fused, wherein the inverse processing is the reverse operation of the preprocessing. The preprocessing process has been described in detail, so it will not be repeated here.

[0137] In step S720, the face image to be fused is fused with the reference face image according to the target fusion degree matrix to obtain the target face image.

[0138] In an example implementation of the present disclosure, the target facial image can be calculated based on the target blemish-free facial image and the reference facial image. The inverse processing can be performed on the target blemish-free facial image to obtain the blemish-free facial image to be fused. The target blemish-free facial image to be fused is then fused with the reference facial image to obtain the target facial image. Adding the blemish removal process can make the beautification effect of the obtained target facial image better.

[0139] In this example implementation, error information in the pre-processing and inverse processing processes may be obtained, and then the error information may be used to update the face image to be fused.

[0140] Specifically, a high-frequency image of the facial image to be optimized can be first obtained based on the error information. Specifically, the inverse processing operation can be performed on the facial image to be beautified to obtain an intermediate facial image. The difference between the facial image to be optimized and the intermediate facial image can be used as the high-frequency image. The target facial image can be updated based on the high-frequency image.

[0141] Among them, since the above preprocessing process includes the resizing and downsampling process, the high-frequency information of the image will inevitably be lost in the downsampling process. The downsampled image is upsampled so that the obtained intermediate face image has the same resolution as the face to be optimized. It should be noted that if the face image to be optimized is rotated before downsampling, it can also be rotated in the opposite direction after upsampling so that the obtained intermediate face image has the same direction as the face image to be optimized.

[0142] Upsampling can be performed by bilinear interpolation, nearest neighbor interpolation, and other methods. Although upsampling can restore the resolution, it is difficult to restore the lost high-frequency information, that is, the intermediate face image can be a low-frequency image of the face image to be optimized. Thus, the difference between the face image to be optimized and the intermediate face image is determined, for example, the difference between the face image to be optimized and the intermediate face image can be used as the high-frequency image.

[0143] When the high-frequency image is used to update the target facial image, the high-frequency image can be added to the unupdated target facial image to obtain an updated target facial image, so that high-frequency information such as detailed texture is added to the target facial image to make it more realistic.

[0144] In an example implementation of the present disclosure, if the facial image to be fused is deblemish_img, the user sets the skin smoothing degree to alpha_cus, the benchmark fusion degree image is alpha_60_img, the reference facial image is 100_soften_img, and the skin smoothing image with the skin smoothing degree set by the user is cus_soften_img, then the pixel-level skin smoothing fusion processing is as follows:.

[0145] alpha_100_img=clip(alpha_60_img+0.4,-1.0,1.0)*(alpha_60_img≠0.0)

[0146] alpha_cus_img=clip(alpha_100_img-1.0+alpha_cus,0.0,1.0)cus_soften_img

[0147] =100_soften_img*alpha_cus_img+deblemish_img*(1.0-alpha_cus_img)

[0148] Among them, clip is the value truncation processing, the lower limit is 0.0, the upper limit is 1.0, and the area less than or equal to 0.0 in the benchmark fusion image, including non-skin areas, facial features and edge areas, extremely smooth skin areas, etc., will remain at 0.0 in aplpha_cus_img, that is, no fusion and skin grinding are required. The above target face image can be calculated by the above formula.

[0149] Since the face image to be optimized and the intermediate face image are usually very similar, the pixel values ​​in the high-frequency image obtained based on their difference are generally small, such as the RGB channel values ​​not exceeding 4. However, for the mutation positions in the face image to be optimized, such as small black moles on the face, they have strong high-frequency information, so the pixel values ​​of the corresponding positions in the high-frequency image may be relatively large. When the high-frequency image is fused to the face image to be optimized, the pixel values ​​at these positions may have adverse effects, such as sharp edges such as "mole marks", resulting in unnatural visual perception.

[0150] In view of the above problems, in one implementation, reference is made to Figure 8 As shown, the image optimization method may further include the following steps S810 and S820:

[0151] Step S810: Determine defect points in the high-frequency image.

[0152] Among them, the defect point is a pixel point with strong high-frequency information, and the point with a larger pixel value in the high-frequency image can be determined as a defect point.

[0153] Alternatively, in one embodiment, the defect point may be determined by:

[0154] Subtract the face image to be optimized from the corresponding target defect-free face image to obtain the difference value of each pixel;

[0155] When it is determined that the difference value of a certain pixel point meets the preset defect condition, the pixel point corresponding to the pixel point in the high-frequency image is determined as the defect point.

[0156] The preset defect condition is used to measure the difference between the face image to be optimized and the target defect-free face image, so as to determine whether each pixel is a defect point to be removed. In the defect removal process, small moles and acne on the face are usually removed and the face skin color is filled. At this position, the face image to be optimized and the target defect-free face image are very different, so the defect points can be identified by setting the preset defect condition.

[0157] Exemplarily, the preset defect condition may include: the difference of each color channel is greater than the first color difference threshold, and at least one of the difference of each color channel is greater than the second color difference threshold. The first color difference threshold and the second color difference threshold may be empirical thresholds. For example, when the color channel includes RGB, the first color difference threshold may be 20, and the second color difference threshold may be 40. Thus, after obtaining the difference between the face image to be optimized and the target defect-free face image for each pixel, the specific difference of the three RGB color channels in the difference is judged to determine whether the difference of each color channel is greater than 20, and whether the difference of at least one color channel is greater than 40. When these two conditions are met, it means that the preset defect condition is met, and the pixel at the corresponding position in the high-frequency image is determined as a defect point.

[0158] Step S820: adjusting the pixel values ​​in a preset area around the defect point in the high-frequency image to within a second preset range.

[0159] After determining the defect point, the preset area around the defect point can be further determined in the high-frequency image. For example, it can be a 5*5 pixel area centered on the defect point. The specific size can be determined according to the size of the high-frequency image, and the present disclosure does not limit it. The pixel values ​​in the preset area are adjusted to the second preset range. The second preset range is generally a smaller numerical range, which can be determined based on experience and actual needs. The pixel value usually needs to be reduced during adjustment. Exemplarily, the second preset range can be -2 to 2, and the pixel values ​​around the defect point may exceed -5 to 5. Adjusting it to -2 to 2 is actually a limit processing. In this way, sharp edges such as "mole marks" can be weakened, increasing the visual natural feeling.

[0160] In this example implementation, after obtaining the target facial image, the target facial image can be used to replace the facial image to be optimized in the image to be processed, and the boundary area between the unreplaced area in the image to be processed and the target facial image is gradient processed to form a smooth transition in the boundary area.

[0161] The unreplaced area in the image to be processed is the area in the image to be processed except the face image to be optimized. The boundary area between the unreplaced area and the target face image actually includes two parts: the boundary area in the unreplaced area adjacent to the target face image, and the boundary area in the target face image adjacent to the unreplaced area. This exemplary embodiment can perform gradient processing on any part of them, or can perform gradient processing on both parts at the same time.

[0162] refer to Fig. 9As shown, a certain proportion (such as 10%) of the boundary area can be determined in the target face image, which extends inward from the edge of the target face image. It should be noted that the boundary area usually needs to avoid the face part to avoid changing the color of the face part during the gradient processing. For example, after the above-mentioned enlargement of the face frame, the face in the face image to be optimized is at a certain distance from the boundary, and the face in the target face image is also at a certain distance from the boundary, so that the face part can be better avoided during the gradient processing. After determining the boundary area, the inner edge color of the boundary area is obtained, which is recorded as the first color; the inner edge color of the unreplaced area is obtained, which is recorded as the second color; and the boundary area is subjected to a gradient processing of the first color and the second color. As a result, the boundary between the unreplaced area and the target face image is a gradient color area ( Fig. 9 This creates a smooth transition and prevents color mutations that could cause visual disharmony.

[0163] It should be noted that when there are multiple target face images, each target face image can be used to replace the corresponding face image to be optimized in the image to be processed, and gradient processing can be performed on the boundary area to give it a natural and harmonious visual experience.

[0164] Fig.10 The schematic flow of the image optimization method is shown, including:

[0165] Step S1001: extract at least one face image to be optimized from the images to be processed according to a preset rule.

[0166] Step S1002: preprocessing the face image to be optimized to obtain a face image to be beautify.

[0167] Step S1003: input the face image to be beautified into the image optimization model to obtain an initial blemish-free face image and a reference fusion degree image.

[0168] Step S1004 , according to the blemish removal degree set by the user, the initial blemish-removed face image and the face image to be beautified are merged to obtain the target blemish-removed face image.

[0169] Step S1005 , performing inverse processing on the facial image to be beautified to obtain an intermediate facial image.

[0170] Step S1006, subtracting the intermediate face image from the face image to be optimized to obtain a high-frequency image.

[0171] Step S1007, inversely process the target blemish-free face image to obtain a face image to be fused.

[0172] Step S1008: using the high-frequency image to update the face image to be fused.

[0173] Step S1009: determining a reference fusion degree matrix corresponding to the reference fusion image.

[0174] Step S1010, calculating a target fusion degree matrix according to the reference fusion degree matrix and the target skin resurfacing degree.

[0175] Step S1011 , fusing the face image to be fused with the reference face image according to the target fusion degree matrix to obtain the target face image.

[0176] In summary, in this exemplary embodiment, the reference fusion degree image corresponding to the facial image to be beautified is calculated by the image optimization model, and the target fusion degree matrix corresponding to the target skin smoothing degree is calculated based on the reference fusion degree image. The calculation of the target fusion degree matrix is ​​realized by the processing of the deep neural network. On the one hand, the flexibility of image optimization is increased. On the other hand, different fusion coefficients can be used for different areas, which is suitable for various lighting conditions or skin conditions, and the skin smoothing accuracy is enhanced, thereby improving the optimization effect of the facial image. On the other hand, the blemish removal function is realized by the processing of the deep neural network to replace the multiple fixed algorithm processes in the related technology, reducing the time consumption and memory usage. On the other hand, when the image to be processed includes multiple faces, the optimization of multiple faces can be realized through one processing after combining the multiple faces, without the need for multiple optimizations, thereby improving the processing efficiency.

[0177] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0178] For further reference, Fig.11 As shown, in the implementation of this example, a face image optimization device 1100 is also provided, including a first acquisition module 1110, a second acquisition module 1120, a matrix calculation module 1130, and an image optimization module 1140. Among them:

[0179] The first acquisition module 1110 can be used to acquire a facial image to be beautified, and process the facial image to be beautified using a pre-trained image optimization model to obtain a reference fusion degree image.

[0180] In an example implementation, the first acquisition module 1110 is configured to:

[0181] Acquire an image to be processed, and extract at least one face image to be optimized from the image to be processed according to a preset rule;

[0182] The face image to be optimized is preprocessed to obtain the face image to be beautified.

[0183] In this example implementation, the image optimization module 1140 is configured to:

[0184] Performing inverse processing on the facial image to be beautified to obtain an intermediate facial image, wherein the inverse processing is a reverse operation of the preprocessing;

[0185] The intermediate face image is fused with the reference face image according to the target fusion degree matrix to obtain the target face image.

[0186] In this example implementation, the image optimization device further includes an updating module, which can be used to obtain error information in the preprocessing and inverse processing processes;

[0187] The target face image is updated based on the error information.

[0188] In another example implementation, the first acquisition module 1110 is configured to: process the to-beautify facial image using a pre-trained image optimization model to obtain a reference fusion degree image and a target blemish-free facial image.

[0189] Specifically: the facial image to be beautified is input into the image optimization model to obtain the baseline fusion degree image and the initial blemish-free facial image; the initial blemish-free facial image and the facial image to be beautified are fused according to the blemish removal degree set by the user to obtain the target blemish-free facial image.

[0190] In this example implementation, the image optimization module 1140 is configured to:

[0191] The target facial image is determined according to the target fusion degree matrix, the target blemish-free facial image and the reference facial image.

[0192] The second acquisition module 1120 may be configured to acquire a reference facial image corresponding to the facial image to be beautified and a target skin smoothing degree; wherein the reference facial image is an image with the maximum skin smoothing degree corresponding to the facial image to be beautified.

[0193] The matrix calculation module 1130 may be configured to calculate a target fusion degree matrix corresponding to the target skin resurfacing degree according to the reference fusion degree image.

[0194] In one example implementation, the matrix calculation module 1130 is configured to:

[0195] A reference fusion degree matrix corresponding to the reference fusion image is determined. Specifically, the pixel value of each pixel point of the reference fusion degree image is normalized to obtain the reference fusion degree matrix.

[0196] Calculating a target fusion degree matrix according to the reference fusion degree matrix and the target skin resurfacing degree;

[0197] The elements in the target fusion degree matrix are truncated so that the values ​​of the elements in the target fusion degree matrix are all within a first preset range.

[0198] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0199] The exemplary embodiment of the present disclosure also provides an electronic device for executing the above-mentioned face image optimization method, and the electronic device may be the above-mentioned terminal 110 or the server 120. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned face image optimization method by executing the executable instructions.

[0200] Below Fig.12 The mobile terminal 1200 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Fig.12 The construction in can also be applied to fixed type equipment.

[0201] like Fig.12 As shown, the mobile terminal 1200 may specifically include: a processor 1201, a memory 1202, a bus 1203, a mobile communication module 1204, an antenna 1, a wireless communication module 1205, an antenna 2, a display screen 1206, a camera module 1207, an audio module 1208, a power module 1209 and a sensor module 1210.

[0202] The processor 1201 may include one or more processing units, for example, the processor 1201 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc. The image optimization method in this exemplary embodiment may be executed by an AP, a GPU or a DSP, and when the method involves processing related to a neural network, it may be executed by an NPU.

[0203] The encoder can encode (i.e. compress) an image or video, for example, the target face image can be encoded into a specific format to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of the image or video to restore the image or video data, such as reading the encoded data of the target face image, decoding it through the decoder to restore the data of the target face image, and then perform relevant processing for image optimization on the data. The mobile terminal 1200 can support one or more encoders and decoders. In this way, the mobile terminal 1200 can process images or videos in multiple encoding formats, such as: JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap) and other image formats, MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, HEVC (High Efficiency Video Coding) and other video formats.

[0204] The processor 1201 may be connected to the memory 1202 or other components via a bus 1203 .

[0205] The memory 1202 may be used to store computer executable program codes, which include instructions. The processor 1201 executes various functional applications and data processing of the mobile terminal 1200 by running the instructions stored in the memory 1202. The memory 1202 may also store application data, such as images, videos and other files.

[0206] The communication function of the mobile terminal 1200 can be implemented by the mobile communication module 1204, antenna 1, wireless communication module 1205, antenna 2, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 1204 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 1200. The wireless communication module 1205 can provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. applied to the mobile terminal 200.

[0207] The display screen 1206 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The camera module 1207 is used to implement shooting functions, such as shooting images, videos, etc. The audio module 1208 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 1209 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 1210 may include a depth sensor 12101, a pressure sensor 12102, a gyroscope sensor 12103, an air pressure sensor 12104, etc., to implement corresponding sensing detection functions.

[0208] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0209] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification.

[0210] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0211] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0212] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0213] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0214] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A facial image optimization method, characterized in that: include: Acquire a facial image to be beautified, and process the facial image to be beautified using a pre-trained image optimization model to obtain a reference fusion degree image, wherein the reference fusion degree image is used to characterize the fusion degree between each pixel in the facial image to be beautified corresponding to a reference skin resurfacing degree and the reference facial image; Obtaining a reference face image and a target skin smoothing degree corresponding to the face image to be beautified; wherein the reference face image is an image with the maximum skin smoothing degree corresponding to the face image to be beautified; Calculating a target fusion degree matrix corresponding to the target skin resurfacing degree according to the reference fusion degree image; Determine a target facial image according to the target fusion degree matrix, the facial image to be beautified, and the reference facial image; The method of calculating a target fusion degree matrix corresponding to the target skin refining degree according to the benchmark fusion degree image includes: determining a benchmark fusion degree matrix corresponding to the benchmark fusion degree image; determining an absolute value of a difference between the target skin refining degree and the benchmark skin refining degree; when the benchmark skin refining degree is greater than the target skin refining degree, subtracting the absolute value from the pixel value in the benchmark fusion degree matrix to obtain a target fusion degree matrix; and when the benchmark skin refining degree is less than the target skin refining degree, adding the absolute value to the pixel value in the benchmark fusion degree matrix to obtain the target fusion degree matrix; and truncating elements in the target fusion degree matrix so that the values ​​of the elements in the target fusion degree matrix are all within a first preset range.

2. The method according to claim 1, characterized in that The process of processing the face image to be beautified by using the pre-trained image optimization model to obtain a reference fusion degree image includes: Processing the face image to be beautified using a pre-trained image optimization model to obtain a baseline fusion degree image and a target blemish-free face image; Determining a target facial image according to the target fusion degree matrix, the facial image to be beautified, and the reference facial image comprises: The target facial image is determined according to the target fusion degree matrix, the target blemish-free facial image and the reference facial image.

3. The method according to claim 2, characterized in that The process of processing the face image to be beautified by using the pre-trained image optimization model to obtain a reference fusion degree image and a target blemish-free face image comprises: Inputting the face image to be beautified into the image optimization model to obtain the reference fusion degree image and the initial blemish-free face image; The initial blemish-removed face image and the face image to be beautified are fused according to the blemish-removal degree set by the user to obtain the target blemish-removed face image.

4. The method according to claim 2, characterized in that: Obtaining the initial face image includes: Acquire an image to be processed, and extract at least one face image to be optimized from the image to be processed according to a preset rule; The face image to be optimized is preprocessed to obtain the face image to be beautified.

5. The method according to claim 4, characterized in that The determining of the target facial image according to the target fusion degree matrix, the facial image to be beautified and the reference facial image comprises: Performing inverse processing on the target blemish-free face image to obtain a face image to be fused, wherein the inverse processing is a reverse operation of the preprocessing; The face image to be fused is fused with the reference face image according to the target fusion degree matrix to obtain the target face image.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining error information during the preprocessing and inverse processing; The face image to be fused is updated based on the error information.

7. The method according to claim 6, characterized in that Determining the reference fusion degree matrix corresponding to the reference fusion degree image includes: The pixel value of each pixel of the reference fusion degree image is normalized to obtain the reference fusion degree matrix.

8. A facial image optimization device, characterized in that: include: A first acquisition module is used to acquire a face image to be beautified, and to process the face image to be beautified using a pre-trained image optimization model to obtain a reference fusion degree image, wherein the reference fusion degree image is used to characterize the fusion degree between each pixel in the face image to be beautified corresponding to a reference skin resurfacing degree and the reference face image; A second acquisition module is used to acquire a reference face image corresponding to the face image to be beautified and a target skin smoothing degree; wherein the reference face image is an image with the maximum skin smoothing degree corresponding to the face image to be beautified; A matrix calculation module, used for calculating a target fusion degree matrix corresponding to the target skin resurfacing degree according to the reference fusion degree image; An image optimization module, configured to determine a target facial image according to the target fusion degree matrix, the facial image to be beautified, and the reference facial image; The method of calculating a target fusion degree matrix corresponding to the target skin refining degree according to the benchmark fusion degree image includes: determining a benchmark fusion degree matrix corresponding to the benchmark fusion degree image; determining an absolute value of a difference between the target skin refining degree and the benchmark skin refining degree; when the benchmark skin refining degree is greater than the target skin refining degree, subtracting the absolute value from the pixel value in the benchmark fusion degree matrix to obtain a target fusion degree matrix; and when the benchmark skin refining degree is less than the target skin refining degree, adding the absolute value to the pixel value in the benchmark fusion degree matrix to obtain the target fusion degree matrix; and truncating elements in the target fusion degree matrix so that the values ​​of the elements in the target fusion degree matrix are all within a first preset range.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the facial image optimization method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the facial image optimization method as described in any one of claims 1 to 7.

Images