Image beautification processing method, device, storage medium and electronic device

Abstract

The present disclosure provides an image beautification processing method, an image beautification processing device, a computer-readable storage medium and an electronic device, and relates to the field of imaging technology. The image beautification processing method comprises: obtaining an image to be beautified; obtaining a global face image and a local face image corresponding to the image to be beautified; the resolution of the global face image is smaller than the resolution of the local face image; processing the global face image by a first machine learning model to obtain a blemish-free global face image; processing the local face image by a second machine learning model to obtain a blemish-free local face image; based on the blemish-free global face image and the blemish-free local face image, obtaining a target beautification image corresponding to the image to be beautified. The present disclosure performs multiple beauty processing according to different resolution requirements in the image, improves the image beautification effect, and improves the processing efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of imaging technology, and in particular to an image beautification processing method, an image beautification processing device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Beauty refers to the use of image processing technology to beautify portraits in images or videos to better meet the user's aesthetic needs.

[0003] In the related art, the image can be beautified as a whole. Since there are many types of defects in the image that needs to be beautified, the use of the above method for overall beautification may result in unsatisfactory beautification effect and poor image quality.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides an image beautification processing method, an image beautification processing device, a computer-readable storage medium and an electronic device, thereby improving the image beautification effect at least to a certain extent.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, there is provided an image beautification processing method, comprising: obtaining an image to be beautified; obtaining a global face image and a local face image corresponding to the image to be beautified; the resolution of the global face image is smaller than the resolution of the local face image; processing the global face image by a first machine learning model to obtain a blemish-free global face image; processing the local face image by a second machine learning model to obtain a blemish-free local face image; and obtaining a target beautification image corresponding to the image to be beautified based on the blemish-free global face image and the blemish-free local face image.

[0008] According to a second aspect of the present disclosure, there is provided an image beautification processing device, comprising: an image acquisition module, for acquiring an image to be beautified; an image segmentation module, for acquiring a global face image and a local face image corresponding to the image to be beautified; the resolution of the global face image is smaller than the resolution of the local face image; a first image processing module, for processing the global face image through a first machine learning model to obtain a blemish-free global face image; a second image processing module, for processing the local face image through a second machine learning model to obtain a blemish-free local face image; and an image generation module, for acquiring a target beautification image corresponding to the image to be beautified based on the blemish-free global face image and the blemish-free local face image.

[0009] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image beautification processing method of the first aspect and possible implementation methods thereof are implemented.

[0010] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image beautification processing method of the above-mentioned first aspect and its possible implementation method by executing the executable instructions.

[0011] The technical solution provided in the embodiments of the present disclosure, on the one hand, performs global beautification on a low-resolution global face image through a first machine learning model, performs local beautification on a high-resolution local face image through a second machine learning model, and performs beautification processing on a global face image and a local face image with different resolution processing requirements in an image to be beautified through different models, respectively, so as to achieve multiple beautification processing in different regions, avoiding the limitation of only one model for overall beautification processing in the related art, increasing the flexibility and pertinence of image beautification processing, and improving the beautification effect and image quality. On the other hand, since the resolution requirements of images processed by different models are different, compared with using only one model for processing, the time consumption and memory usage are reduced, the beautification efficiency is improved, and the power consumption is reduced.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0014] Figure 1 A schematic diagram showing the structure of the system architecture for implementing image beautification processing in this exemplary embodiment;

[0015] Figure 2 A flowchart of an image beautification processing method according to the exemplary embodiment is shown;

[0016] Figure 3 A schematic diagram showing a global image of a human face in this exemplary embodiment;

[0017] Figure 4 A first schematic diagram showing a partial face image in this exemplary embodiment;

[0018] Figure 5 A second schematic diagram showing a partial face image in this exemplary embodiment;

[0019] Figure 6 A schematic diagram showing the structure of a first machine learning model in this exemplary embodiment is shown;

[0020] Figure 7 A schematic diagram of a process of obtaining a global image of a blemish-free face by using a first machine learning model in this exemplary embodiment is shown;

[0021] Figure 8 A schematic diagram of a process of obtaining a blemish-free partial face image by using a second machine learning model in this exemplary embodiment is shown;

[0022] Fig. 9 A flowchart of obtaining a target beauty image in this exemplary embodiment is shown;

[0023] Fig.10 A flowchart of determining a limit value high frequency face image in this exemplary embodiment is shown;

[0024] Fig.11 A schematic diagram showing a method of acquiring a high-frequency image in this exemplary embodiment;

[0025] Fig.12 A schematic diagram showing a filling process in this exemplary embodiment is shown;

[0026] Fig.13 A schematic diagram showing the gradient fusion in this exemplary embodiment is shown;

[0027] Fig.14 A specific flow chart of an image beautification processing method in this exemplary embodiment is shown;

[0028] Fig.15 A schematic diagram showing the structure of an image beautification processing device in this exemplary embodiment is shown;

[0029] Fig.16 A schematic structural diagram of an electronic device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0031] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0032] Portrait blemish removal is part of image beautification processing and is usually the first stage of image beautification processing. Portrait blemish removal includes but is not limited to removing spots and acne, removing eye bags, dirty mouth corners, light and shadow smoothing, dry lip lines, etc. After portrait blemish removal, you can continue with skin smoothing, skin color adjustment, facial features deformation, brightness adjustment, etc.

[0033] In the related art, the face area image is usually cropped, scaled to a fixed size, and processed in a single step. However, due to the different minimum processing resolution requirements for different types of defects and noise, the minimum processing resolution requirements for acne, eye bags, color blocks, etc. are low, while the minimum processing resolution requirements for bloodshot eyes / noise, dirty nose wings / nose hair, dirty corners of the mouth / noise, etc. are high. Therefore, using a single-processing deep neural network model processing method, if the input face size is too small, that is, the resolution is less than the minimum processing resolution requirement for certain defects, the defect cannot be fully removed and the defect removal effect is poor. If the input face size is large enough, it will directly lead to long processing time and high power consumption.

[0034] In order to solve the above technical problems, the exemplary embodiment of the present disclosure first provides an image beautification processing method, and its application scenarios include but are not limited to: the terminal device is installed with an image beautification App (Application), and the user selects an image in the local album in the App for beautification processing, or performs beautification processing on the currently captured image; the terminal device executes the image beautification processing method of this exemplary embodiment, or the terminal device sends the image to the server, and the server executes the image beautification processing method of this exemplary embodiment to perform beautification processing on the image. Alternatively, the video selected by the user or the currently captured video can also be beautified, specifically, the frames containing portraits in the video can be beautified, for example, in a live broadcast scenario, the real-time video stream can be beautified.

[0035] like Figure 1 In the system architecture diagram shown, the terminal 100 may be an intelligent device with an image processing function, such as a smart phone, a computer, a tablet computer, an intelligent speaker, a smart watch, a vehicle-mounted device, a wearable device, a monitoring device, and the like. The terminal may include a camera, and the type of the camera may be any type as long as it can perform photo processing. The number of cameras may be at least one, such as one, four, and the like. The image to be processed may be a captured image, or each frame of a captured video, or an image obtained from a terminal or other storage device.

[0036] In the embodiment of the present disclosure, the terminal 100 may include an imaging unit 101, a memory 102, a processor 103, and an input-output unit 104. The imaging unit 101 is used to take pictures. The memory 102 is used to store images, and the processor is used to process images, such as performing beauty processing, etc. The processor generally includes one or more processors of a CPU, a GPU, a DSP, and an NPU. The memory 102 may store an image to be beautified 105. The input-output unit is used to display images.

[0037] The terminal 101 obtains the image to be beautified 105 from the memory 102 and sends it to the processor 103. The global face image and the partial face image corresponding to the image to be beautified are obtained in the processor 103. The global face image is processed by the first machine learning model, and the partial face image of the image to be beautified is processed by the second learning model to perform beautification processing in different ranges, thereby generating a target beautified image 106 after beautification and displaying it through the input and output unit 104.

[0038] It should be noted that the image beautification processing method provided in the embodiment of the present disclosure may be executed by the terminal 100. The image beautification processing method may also be set in the terminal 100.

[0039] Next, combine Figure 2 The specific steps of the image beautification processing method are explained. Figure 2 An exemplary process of the image beautification processing method is shown, which may include the following steps S210 to S250, wherein:

[0040] Step S210, obtaining the image to be beautified;

[0041] Step S220, obtaining a global face image and a partial face image corresponding to the image to be beautified; the resolution of the global face image is smaller than the resolution of the partial face image;

[0042] Step S230, processing the global face image by using a first machine learning model to obtain a blemish-free global face image;

[0043] Step S240, processing the partial face image by a second machine learning model to obtain a blemish-free partial face image;

[0044] Step S250: acquiring a target beautified image corresponding to the image to be beautified based on the blemish-removed global face image and the blemish-removed partial face image.

[0045] In the disclosed embodiments, the image beautification processing may be a universal beautification processing, which refers to a relatively fixed and unified link in the entire beautification processing process, such as blemish removal, which is necessary for most users and does not require personalized settings.

[0046] The image to be beautified refers to an image that needs to be beautified, such as a face image in an image, etc. Since there may be multiple types of defects in a face image, and the minimum resolutions corresponding to different types of defects are different, using images with the same resolution to process different types of defects may result in poor defect removal results. Therefore, the image to be beautified can be divided into a global face image and a local face image. Among them, the global face image is used for global defect removal, and the local face image is used for local defect removal. Moreover, the resolution of the global face image is different from that of the local face image. Specifically, the resolution of the global face image is smaller than that of the local face image. By obtaining the global face image and the local face image corresponding to the image to be beautified, different types of defects can be beautified respectively.

[0047] The first machine learning model can remove defects from the global face image to obtain a defect-free global face image, and the second machine learning model can remove defects from the local face image to obtain a defect-free local face image. Using different machine learning models to process different types of defects can improve accuracy and pertinence.

[0048] After obtaining the beautified images corresponding to different types of defects, the global image of the blemish-free face and the local image of the blemish-free face can be filled into the corresponding positions in the image to be beautified in sequence to obtain the target beautified image corresponding to the image to be beautified, that is, the image after removing multiple types of defects from the image to be beautified.

[0049] Next, Figure 2 Each step in the procedure is described in detail.

[0050] refer to Figure 2 As shown in , in step S210 , an image to be beautified is obtained.

[0051] In the disclosed embodiment, the image to be beautified may be an image that needs beautification, and may be obtained from an image to be processed. The image to be processed may be an image selected by a user, such as an image selected by a user in an album or other files, or an image obtained from a network or other applications or other storage locations, or an image automatically designated by a system, such as a currently captured image. The image to be beautified may be an image containing a face image, or may be a part of the image to be processed, for example, may be a part of the image to be processed containing a face. Specifically, it may be obtained by preprocessing the image to be processed.

[0052] After obtaining the image to be processed, the image to be processed may be preprocessed to obtain the image to be beautified. The image to be beautified may be a rectangular face image to be beautified extracted from the image to be processed. The preprocessing may be extraction or other operations. The number of images to be beautified may be one or more, which is specifically determined according to the number of face images included in the image to be processed, and the number of images to be beautified may be less than or equal to the number of face images included in the image to be processed, which is specifically determined according to the size of each face image and the upper limit of the number.

[0053] In some embodiments, the specific implementation of step S210 may be: extracting the image to be beautified from the image to be processed. Specifically comprising the following steps: generating one or more face frames in the image to be processed according to the facial key points identified in the image to be processed; retaining the face frames whose areas are greater than or equal to the face area threshold, and intercepting the image in the face frames to obtain the image to be beautified. Among them, the facial key points may include points at the key parts of the face and the edges of the face. The face frame may be a rectangular frame, and the facial key points of each face are within the face frame. In one implementation, the face frame may be the smallest rectangular frame including the facial key points.

[0054] Generally, all faces can be detected in the image to be processed by the face detection algorithm, but some faces may not need beautification (such as the faces of passers-by in the distance). Since in the image beautification scenario, it is usually necessary to beautify larger faces (the effect of beautification is not obvious for smaller faces), the face frame can be filtered by the face area threshold to avoid misoperation caused by faces with smaller areas. Specifically, the face area threshold can be the size of the image to be processed * 0.05, which can be determined according to the size of the image to be processed; if the area of ​​the face frame is greater than or equal to the face area threshold, it is determined as a face that needs beautification and the face frame is retained; if the area of ​​the face frame is less than the face area threshold, the face frame is deleted.

[0055] After filtering the face frames, the retained face frames are the face frames of valid faces. The images in each face frame are intercepted to obtain the same number of images to be beautified as the number of face frames.

[0056] In one embodiment, an upper limit on the number of images to be beautified can be set, which can be achieved by setting an upper limit on the number of face frames. The upper limit can be set to 4, for example. If the number of retained face frames is greater than 4 after filtering by the above-mentioned face area threshold, 4 face frames can be selected therefrom. The 4 face frames can be, for example, the 4 face frames arranged in the top 4 order from large to small in area, or the 4 face frames closest to the center of the image to be processed, or the 4 face frames selected according to actual needs, corresponding to 4 original face images are intercepted, and the faces in other face frames are not beautified; or multiple beautification processes can be performed, in which 4 face frames are selected and the corresponding face images are intercepted for beautification in this process, and other face frames are selected and other face images are intercepted for beautification in the next process, thereby completing the beautification of faces in all face frames in the image to be processed whose areas are greater than the face area threshold.

[0057] In some embodiments, before capturing the image in the face frame, the face frame may be enlarged so that the face frame includes a small amount of area other than the face, so as to facilitate gradient processing during subsequent image fusion. When performing the enlargement process, the face frame may be enlarged in one or more directions according to a preset ratio. For example, the preset ratio is 1.1, and the face frame is uniformly enlarged in all directions so that the size of the enlarged face frame is 1.1 times the original size. It should be noted that when the face frame is enlarged, if one or more boundaries of the face frame reach the boundary of the image to be processed, the boundary of the face frame is stopped at the boundary of the image to be processed, that is, the boundary of the image to be processed is used as the boundary of the face frame.

[0058] Next, in step S220, a global face image and a partial face image corresponding to the image to be beautified are obtained; the resolution of the global face image is smaller than the resolution of the partial face image.

[0059] In the disclosed embodiments, various types of defects may exist during the image beautification process. Different types of defects may have different minimum processing resolution requirements. Various types of defects may be determined based on the difficulty of processing the defects, and the various types of defects may be distributed in different parts of the face image. For example, the various types of defects may be facial spot defects and eyeball defects, etc. For example, for facial spot defects, the minimum resolution requirement is relatively low; for eyeball defects, the minimum resolution requirement is relatively high.

[0060] In the process of processing the image to be beautified, if the size of the face in the image to be beautified is too small, that is, the resolution is less than the minimum processing resolution requirement of a certain type of defect among multiple types of defects, then the defect cannot be fully removed and the defect removal effect is poor. If the size of the face in the image to be beautified is too large, that is, the resolution is greater than the minimum processing resolution requirement of a certain type of defect among multiple types of defects, it may result in a long processing time and high power consumption. In order to avoid the problem of different image resolution requirements corresponding to different types of defects, different types of defects can be beautified separately.

[0061] In some embodiments, the beautified image may be subjected to multiple blemish removal processes respectively according to multiple beautification attribute parameters of the image to be beautified. When performing multiple blemish removal processes, multiple images with different resolutions may be first acquired to implement multiple blemish removal processes. For example, a global face image and a local face image corresponding to the image to be beautified may be acquired to perform classified beautification, thereby obtaining corresponding global blemish removal results and local blemish removal results.

[0062] It should be noted that in order to achieve accurate beautification, the resolution of the global face image is smaller than the resolution of the local face image. Resolution is used to indicate the minimum resolution required for beautifying the global face image or the local face image. That is, the minimum resolution for beautifying the global face image is smaller than the minimum resolution for beautifying the local face image.

[0063] In the process of classification beautification, the specific implementation of step S220 may be: the image to be beautified is divided into a global face image and a local face image by using beautification attribute parameters. The beautification attribute parameters are used to indicate the defect type of the image to be beautified and the resolution corresponding to the defect type. The resolution may be the minimum resolution corresponding to the defect type. Exemplarily, the beautification attribute parameters may include: a first type of defect and a corresponding resolution (e.g., a first resolution), and a second type of defect and a corresponding resolution (e.g., a second resolution). Among them, the first type of defect may be a defect that can be processed based on the low resolution represented by the first resolution, for example, including but not limited to acne, eye bags, color blocks, etc.; the second type of defect may be a defect that cannot be processed based on the low resolution represented by the first resolution, and the second type of defect is a locally distributed defect, for example, including but not limited to bloodshot eyes / noise, dirty nose wings / nose hair, dirty corners of the mouth / noise, etc. It should be noted that the resolution corresponding to the first type of defect is less than the resolution corresponding to the second type of defect, that is, the minimum processing resolution requirement corresponding to the first type of defect is low, and the minimum processing resolution requirement corresponding to the second type of defect is high.

[0064] Continue to refer Figure 2 As shown, in step S230, the global face image is processed by a first machine learning model to obtain a blemish-free global face image.

[0065] In the disclosed embodiments, since different types of defects have different corresponding resolutions, in order to avoid the limitations caused by processing all types of defects in the same way, the beautification image may be beautified through multiple different models to obtain a blemish removal result. Exemplarily, the global face image corresponding to the beautification image may be processed by a first machine learning model, and the local face image corresponding to the beautification image may be processed by a second machine learning model. For the first machine learning model and the second machine learning model, since the types of defects processed are different and the resolution requirements corresponding to different defect types are different, the input image sizes of different machine learning models may also be different. Based on this, in order to improve the accuracy of model processing, the corresponding image may be processed based on the input image size of the machine learning model to keep it consistent with the input image size of the machine learning model.

[0066] In some embodiments, since the input image sizes of different machine learning models are different, the image to be beautified can be processed into a global face image and a local face image according to the input image sizes of different machine learning models. Among them, the global face image can be an image that matches the first machine learning model, and the local face image can be an image that matches the second machine learning model. In the process of determining the global face image and the local face image, the image to be beautified can first be divided into a global image and local images of multiple parts according to the beauty attribute parameters. The beauty attribute parameters specify the defect type, on this basis, the global image can be an image corresponding to the first type of defect, specifically the image to be beautified itself, and can also be called a complete rectangular original face image; the local image can be a partial image represented by multiple parts corresponding to the second type of defect, such as a local image corresponding to the eyes, a local image corresponding to the nose, and a local image corresponding to the mouth.

[0067] In the process of acquiring the global face image, the global image corresponding to the image to be beautified may be adjusted. The adjustment operation may be size transformation, which may include but is not limited to one or more of rotation, scaling, and padding.

[0068] In some embodiments, it may first be determined whether the size parameters of the global image meet the size condition. The size condition may be the height and width of the global image, which match the height and width of the input image size of the first machine learning model. If they do not match, the global image may be resized based on the input image size of the first machine learning model to obtain a global face image.

[0069] In some embodiments, resizing the global image in the beautified image may include any one or more of the following methods:

[0070] (1) When the size relationship between the width and height of the global image of the image to be beautified (long and short sides) is different from the size relationship between the width and height of the input image size of the first machine learning model, the global image is rotated 90 degrees. Specifically, if the global image of the image to be beautified and the input image size of the first machine learning model are both larger in width than in height or smaller in width than in height, then the size relationship between the width and height of the global image and the input image size of the first machine learning model is the same, and there is no need to rotate the global image; otherwise, the size relationship between the width and height of the global image and the input image size of the first machine learning model is different, and the global image needs to be rotated 90 degrees (either clockwise or counterclockwise) to keep the size relationship between the two consistent. For example, when the input image size of the first machine learning model is 704 in width and 448 in height, that is, the width is greater than the height, if the global image is smaller in width than the height, the global image is rotated 90 degrees. In addition, in order to maintain the angle of the face in the image to be beautified, the global image may not be rotated.

[0071] (2) When the size of the global image is larger than the size of the input image, the global image is scaled according to the size of the input image of the first machine learning model. The scaling here may be downsampling. The size of the global image is larger than the size of the input image, which means that the width of the global image is larger than the width of the input image size, or the height of the global image is larger than the height of the input image size. In the image beautification scenario, the image to be beautified is generally provided by a clear image taken by a terminal device, and its size is large. Therefore, it is a common situation that the size of the global image is larger than the size of the input image, that is, it is usually necessary to downsample the global image. Downsampling can be used to represent size reduction processing. Downsampling can be implemented by methods such as bilinear interpolation and nearest neighbor interpolation, which is not limited in the present disclosure.

[0072] After downsampling, the global image is aligned with at least one of a width and a height of an input image size of the first machine learning model, specifically including the following situations: the width and height of the global image are the same, the width of the global image is the same as the width of the input image size, and the height is less than the height of the input image size;

[0073] The height of the global image is the same as the height of the input image size, and the width is smaller than the width of the input image size.

[0074] It should be noted that if the global image is rotated as described above to obtain a rotated global image, when the size of the rotated global image is larger than the size of the input image, it is downsampled according to the size of the input image. The specific implementation method is the same as the downsampling method of the above-mentioned global image, so it is not repeated here. On the contrary, when the size of the global image (or the rotated global image) is less than or equal to the size of the input image, the downsampling processing step may not be performed. After the downsampling is performed, the aspect ratio of the global image after the scaling process is required to be consistent with the aspect ratio of the global image before the size scaling process, and the width and height after the scaling process are both less than or equal to the width and height of the corresponding input image size, and the width and height of the global image after the scaling process should be as close as possible to the width and height of the corresponding input image size.

[0075] (3) When the size of the global image is smaller than the size of the input image, the global image is padded according to the difference between the sizes of the global image and the input image, so that the size of the padded global image is equal to the size of the input image. The size of the global image is smaller than the size of the input image, which means that at least one of the width and height of the global image is smaller than the size of the input image, and the other is not larger than the size of the input image, including the following cases: the width of the global image is smaller than the width of the input image, and the height is also smaller than the height of the input image; the width of the global image is smaller than the width of the input image, and the height is equal to the height of the input image; the height of the global image is smaller than the height of the input image, and the width is equal to the height of the input image.

[0076] When filling, you can fill in the four sides of the global image, for example, overlap the center of the global image with the center of the input image size, fill in the difference part around the global image, so that the size of the global image after filling is consistent with the input image size. Of course, you can also align the edge of one side of the global image with the input image size and fill in the other side, which is not limited in the present disclosure. When filling, you can use a preset pixel value, which is usually a pixel value that is significantly different from the color of the face, such as (R0, G0, B0), (R255, G255, B255), etc. The unfilled part can be assigned a value of 0 or no additional value is required, as long as the size of the global image after filling is consistent with the input image size.

[0077] In the above transformation, by changing the direction, size, etc. of the global image, the unified processing of the first machine learning model is facilitated. Subsequently, the face image after beautification needs to be inversely transformed to restore it to a size consistent with the direction, size, etc. of the global image. Therefore, the corresponding size transformation information can be saved, including but not limited to: the direction and angle of rotation of each global image, the downsampling ratio (scaling ratio), the coordinates of the filled pixels, and the filled pixel values.

[0078] In the embodiment of the present disclosure, by rotating, scaling, and filling the global image according to the input image size of the first machine learning model, the global image can be converted into a global face image that can be directly applied to the first machine learning model. The global face image can be Figure 3 as shown in .

[0079] In the process of obtaining a partial face image, the partial image corresponding to the image to be beautified may be adjusted to obtain a partial face image. The partial face image may be obtained by combining partial images of multiple parts, or may be obtained separately according to the partial image of each part. Based on this, obtaining a partial face image may include the following two methods:

[0080] The first one is to adjust and combine the local images of the multiple parts based on an input image size of the second machine learning model to obtain the local image of the face.

[0081] In the disclosed embodiment, the partial face image is an image that matches the input image size of the second machine learning model. The input of the second machine learning model is an image composed of partial images of multiple parts.

[0082] In the process of obtaining a partial image of the face, the partial images of multiple parts corresponding to the image to be beautified can be combined and adjusted. That is, by combining the partial images of multiple parts, a combined partial image can be obtained, and the combined partial image can be further adjusted to obtain a partial image of the face. Combining partial images of multiple parts refers to splicing the partial images of multiple parts together according to a preset format. The preset format can be set according to actual needs and is not specifically limited here. For example, the preset format can be that multiple parts are arranged in sequence according to their distribution positions on the face, and spliced ​​to obtain a combined partial image. The lengths of multiple parts can be equal, and the width of each part can be set according to actual needs. For example, when the multiple parts are eyes, nose and mouth, the combined partial image is obtained by arranging them in the order of left eye, right eye, nose, left corner of the mouth, and right corner of the mouth.

[0083] Furthermore, the combined local image may be adjusted to obtain a local face image. The adjustment operation may be a size transformation. In some embodiments, when the size parameters of the local image do not meet the size condition, the local image may be resized based on the input image size of the second machine learning model to obtain the local face image. The size condition may be the height and width of an image combined from local images of multiple parts, which matches the height and width of the input image size of the second machine learning model. The size transformation may include but is not limited to any one or more of rotation, scaling, and padding. The specific method for performing the size transformation is the same as the specific method for performing the size transformation on the global image described above, and will not be repeated here.

[0084] In addition, the local images of multiple parts can also be resized according to the size of each part of the input image size in the second machine learning model, and the local images of multiple parts after the size transformation can be spliced ​​according to the preset format in the input image size, as long as the local image of the face that matches the input image size of the second machine learning model can be obtained.

[0085] Figure 4 A schematic diagram of a partial face image is shown schematically in FIG. Figure 4 In the embodiment, a partial face image 400 is obtained according to partial images 401, 402, 403, 404 and 405 of the face to be beautified. Figure 4 In the image, the width of the eyes represented by partial images 401 and 402 is 192 pixels, the width of the nose represented by partial image 403 is 192 pixels, and the width of the corners of the mouth represented by partial images 404 and 405 is 96 pixels; while in the global face image, the width of the eyes is about 90 pixels, and the width of the nose is about 90 pixels. Therefore, for the second type of defects with high minimum processing resolution requirements such as bloodshot eyes / noise, dirty nose wings / nose hair, dirty corners of the mouth / noise, the resolution of the local face image is higher than that of the global face image, which can improve the defect removal effect.

[0086] The second method is to adjust the local images of multiple parts based on the input image sizes of multiple second machine learning models to generate multiple local face images.

[0087] In the disclosed embodiment, the partial image of the face is an image that matches the input image size of each second machine learning model. The input of each second machine learning model is obtained by adjusting the partial image of the corresponding part, and each second machine learning model can correspond one to one to the partial image of a part. Based on this, a second machine learning model can be used to independently adjust the partial image of a part. In some embodiments, when the size parameters of the partial image of each part do not meet the size conditions of the corresponding machine learning model, the partial image of the part can be resized based on the input image size of the corresponding second machine learning model to obtain the corresponding partial image of the face. The size condition can be the height and width of the partial image of a part, which matches the height and width of the input image size of the corresponding second machine learning model. The size transformation may include but is not limited to any one or more of rotation, scaling and padding. The specific method for performing the size transformation is the same as the specific method for performing the size transformation on the global image mentioned above, and will not be repeated here. For details, please refer to Figure 5 As shown in , a partial face image 5011 is obtained according to partial image 501, a partial face image 5021 is obtained according to partial image 502, a partial face image 5031 is obtained according to partial image 503, a partial face image 5041 is obtained according to partial image 504, and a partial face image 5051 is obtained according to partial image 505. It should be noted that the input image size of each second machine learning model can be the same or different, and is not specifically limited here.

[0088] In the disclosed embodiments, by processing the image to be beautified into a global face image and a local face image with different resolutions, the problem of inability to fully remove defects due to the input image resolution being lower than the minimum processing resolution requirement for certain defects is avoided, and the problem of long processing time and high power consumption due to the input image resolution being higher than the minimum processing resolution requirement is also avoided. The resolutions required for multiple defect types in the image to be beautified can be classified and processed, thereby improving the specificity and accuracy.

[0089] In addition, when a partial image is transformed into a partial face image, multiple tasks such as blemish removal, denoising, and super-resolution can be performed simultaneously. For example, when an eye image is transformed into a partial face image, the downsampling ratio is usually low or no downsampling is performed, and multiple tasks such as blemish removal, denoising, and super-resolution can be processed simultaneously to improve processing efficiency.

[0090] Next, continue to refer to Figure 2 As shown in , in step S230, the global face image is processed by a first machine learning model to obtain a blemish-free global face image.

[0091] In the disclosed embodiments, the global face image can be beautified by the first machine learning model to generate the blemish-free global face image. The first machine learning model can be a DNN (Deep Neural Network, DNN), a deep convolutional neural network. The deep convolutional neural network can be a fully convolutional network, which is a neural network with a large number of layers. By increasing the number of network layers (i.e., network depth) to reduce the number of parameters, it can learn the deep features of the image and achieve pixel-level processing. In some embodiments, the global face image can be pixel rearranged and feature extracted by the first machine learning model to generate a blemish-free global face image corresponding to the global face image. The blemish-free global face image can be an image obtained by beautifying the first type of blemishes based on the entire image to be beautified.

[0092] In one embodiment, a lightweight machine learning model can be used to reduce the amount of computation and achieve learning and processing of image depth features. Exemplarily, the machine learning model can adopt an end-to-end structure to achieve pixel-level processing of beauty images. Figure 6 The structure diagram of the first machine learning model is schematically shown in FIG. Figure 6 As shown in , the first machine learning model may include a first pixel rearrangement layer 610, at least one convolutional layer 620, at least one transposed convolutional layer 630, and a second pixel rearrangement layer 640. In one embodiment, the first machine learning model may be a full convolutional network, and the full convolutional network can process images of different sizes. In this case, the first machine learning model has no requirements for the input image size, and the size has an impact on the amount of calculation, memory usage, and beauty refinement. The input image size can be determined according to the beauty refinement set by the user or the performance of the terminal device. As a result, the machine learning model can be deployed on devices with different performances such as high, medium, and low, and has a wide range of applications. There is no need to deploy different machine learning models for different devices, which reduces the training cost of the network. Exemplarily, when lightweight computing is suitable for a mobile terminal, the input image size can be determined as a smaller value, such as 640 wide and 448 high.

[0093] based on Figure 6 The specific implementation of the first machine learning model shown in step S230 to obtain the global image of the blemish-free face can be as follows Figure 7 As shown in , it mainly includes steps S710 to S740, wherein:

[0094] Step S710: Perform a first pixel rearrangement process on the global face image using the first pixel rearrangement layer to obtain a first feature image.

[0095] It should be noted that the global face image can be a single-channel image (such as a grayscale image) or a multi-channel image (such as an RGB image). The first pixel rearrangement layer can rearrange each channel of the global face image into multiple channels.

[0096] In an optional embodiment, step S710 includes the following steps: inputting a global face image with a channel number a into a first pixel rearrangement layer; rearranging the pixels in each n*n neighborhood in each channel of the global face image to the same positions in n*n channels, and outputting a first feature image with a channel number a*n*n.

[0097] Where a is a positive integer, indicating the number of channels of the global face image; n is a parameter for pixel rearrangement, which is a positive integer not less than 2. Taking n=2 as an example, traverse the first channel of the global face image, usually starting from the upper left corner, extract the pixels of each 2*2 grid, and rearrange them to the same position in the four channels, thereby turning one channel into four channels, and at the same time reducing the width and height of the image to half of the original, and recording the rearranged image as the first feature image; the same method is used to process other channels. If the global face image is a single-channel image, the first feature image of the four channels is obtained after the pixels are rearranged; if the original face composite image is a three-channel image, the first feature image of the twelve channels is obtained after the pixels are rearranged. The first pixel rearrangement layer can be implemented using the space_to_depth function in TensorFlow (a machine learning implementation framework), or it can be implemented using a convolution operation with a step size of n. In this case, the first pixel rearrangement layer can be regarded as a special convolution layer. The first pixel rearrangement layer can rearrange multiple pixels in adjacent domains to the same position in different channels, increasing the number of channels. It can convert the spatial features of the global face image into deep features, thereby improving the accuracy and comprehensiveness of the features.

[0098] Step S720: Perform convolution processing on the first feature image using a convolution layer to obtain a second feature image.

[0099] The present disclosure does not limit the number of convolutional layers, the size of convolutional kernels, the specific structure of convolutional layers, etc. The convolutional layer is used to extract image features from different scales and learn depth information. The convolutional layer may include a matching pooling layer for downsampling the convolutional image to achieve information abstraction, increase the receptive field, and reduce parameter complexity.

[0100] When multiple convolutional layers are set, a stepwise convolution and downsampling method can be used. For example, the image can be reduced by a factor of 2 until the last convolutional layer outputs a second feature image. The second feature image can be the feature image with the smallest size in the processing of the machine learning model.

[0101] Step S730: Use a transposed convolution layer to perform a transposed convolution process on the second feature image to obtain a third feature image.

[0102] The present disclosure does not limit the number of transposed convolution layers, the size of the transposed convolution kernel, the specific structure of the transposed convolution layer, etc. The transposed convolution layer is used to upsample the second feature image, which can be regarded as the reverse process of convolution, thereby restoring the size of the image.

[0103] When multiple transposed convolutional layers are set, a stepwise upsampling method may be used, for example, the image may be upsampled by a factor of 2 until the last transposed convolutional layer outputs the third feature image.

[0104] In an optional implementation, the convolution layer and the transposed convolution layer are completely symmetrical structures, and the third feature image has the same size and the same number of channels as the first feature image.

[0105] In an optional implementation, a direct connection can be established between the convolutional layer and the transposed convolutional layer, such as Figure 6 As shown in the structure in , a direct connection is established between the convolution layer and the transposed convolution layer corresponding to the feature images of the same size, thereby directly connecting the feature image information of the convolution link to the feature image in the transposed convolution link, which is conducive to obtaining a third feature image with more comprehensive information.

[0106] Step S740: Using the second pixel rearrangement layer, the third feature image is subjected to pixel rearrangement processing from multi-channel to single-channel, so as to obtain a global face image with blemishes removed.

[0107] The second pixel rearrangement layer has the opposite process to the first pixel rearrangement layer, and the second pixel rearrangement layer can rearrange multiple channels of the third feature image corresponding to the global face image into one channel. In an optional implementation, step S740 includes the following steps: inputting the third feature image with a channel number of b*n*n into the second pixel rearrangement layer; rearranging the pixels at the same position in each n*n channel of the third feature image into an n*n neighborhood in a single channel, and outputting a blemish-free global face image with a channel number of b. Where b is a positive integer.

[0108] The second pixel rearrangement layer can be implemented by using the depth_to_space function in TensorFlow to convert the depth features in the third feature image into spatial features. It can also be implemented by using a transposed convolution operation with a step size of n. In this case, the second pixel rearrangement layer can be regarded as a special transposed convolution layer.

[0109] If the convolution layer and the transposed convolution layer are completely symmetrical structures, that is, the size and number of channels of the third feature image are the same as those of the first feature image, then a=b. Furthermore, the number of channels of the blemish-free face global image and the face global image is also the same, that is, the processing of the first machine learning model does not change the image size and the number of channels. It should be noted that the processing of the machine learning model also does not change the number of faces.

[0110] The blemish-free global face image may be an image obtained by performing blemish removal processing on the global face image using the first machine learning model. The blemish removal processing includes but is not limited to blemish removal, acne removal, eye bag removal, even skin tone, smooth light and shadow, dirty mouth corner processing, lip processing, and other blemish removal processing performed on the first type of blemishes.

[0111] In this exemplary embodiment, in order to improve the accuracy of the network, the machine learning model can be trained to obtain a first machine learning model. The first machine learning model can be trained as a model for performing a first type of defect beautification on an image. The training process may include the following steps: obtaining a plurality of images without beautification as sample global face images; obtaining a sample defect-free image obtained by manually removing defects from the sample global face image; constructing a data set with the sample global face image and the sample defect-free image to train the first machine learning model, and the obtained first machine learning model can be used to perform defect removal processing of the first type of defects. If when obtaining the above-mentioned sample defect-free image, it is replaced by obtaining a sample beauty image after artificially removing defects and smoothing the sample global face image, and constructing a data set and training the machine learning model, the obtained first machine learning model can be used to perform defect removal and smoothing processing at the same time. Therefore, according to actual application requirements, sample images that have undergone specific beauty processing can be obtained and a data set can be constructed to train a first machine learning model that can realize a specific beauty processing function. In addition, the first machine learning model can integrate a variety of different beauty processing functions, which is more convenient in solution implementation and has higher processing efficiency than setting multiple algorithm modules.

[0112] In step S240, the partial face image is processed by a second machine learning model to obtain a blemish-free partial face image.

[0113] In the disclosed embodiment, the partial face image may also be processed by a second machine learning model to obtain a blemish-free partial face image. The second machine learning model may also be a deep convolutional neural network. The partial face image matches the input image size of the second machine learning model. In some embodiments, the partial face image may be subjected to pixel rearrangement and feature extraction by at least one second machine learning model to generate at least one blemish-free partial face image corresponding to the partial face image. The blemish-free partial face image may be an image obtained by beautifying the second type of blemishes based on part of the image to be beautified.

[0114] The second machine learning model may include: a third pixel rearrangement layer, at least one convolution layer, at least one transposed convolution layer, and a fourth pixel rearrangement layer. The specific structure may also refer to Figure 6 As shown in . Figure 6 The difference between the first machine learning model shown in is that, in order to improve the processing speed and the amount of data processed by the second machine learning model is small, the number of network layers of the second machine learning model can be less than the number of network layers of the first machine learning model. When there are multiple second machine learning models, the number of network layers of the multiple second machine learning models can be the same or different, which is not specifically limited here.

[0115] Based on the structure of the second machine learning model, the specific implementation method of obtaining the blemish-free partial face image in step S240 can be as follows: Figure 8 As shown in , it mainly includes the following steps:

[0116] Step S810: Perform a third pixel rearrangement process on the partial face image using the third pixel rearrangement layer to obtain a first partial feature image.

[0117] In this step, the partial face image can be a partial face image composed of partial images of multiple parts, or a partial face image corresponding to a partial image of each part. The third pixel rearrangement layer can rearrange each channel of the partial face image into multiple channels, and the specific processing method is the same as that in step S710, which will not be repeated here.

[0118] Step S820: Perform convolution processing on the first local feature image using a convolution layer to obtain a second local feature image.

[0119] In this step, the convolution layer may include a matching pooling layer for downsampling the convolved image through a convolution operation. When multiple convolution layers are set, a step-by-step convolution and downsampling method may be used, for example, the image may be downsampled by a factor of 2 until the last convolution layer outputs a second local feature image, which may be the feature image with the smallest size during the processing of the second machine learning model.

[0120] Step S830: Use a transposed convolution layer to perform a transposed convolution process on the second local feature image to obtain a third local feature image.

[0121] In this step, the transposed convolution layer is used to upsample the second local feature image, which can be regarded as the reverse process of convolution, thereby restoring the size of the image. When multiple transposed convolution layers are set, a stepwise upsampling method can be adopted, for example, the image can be increased at a rate of 2 until the last transposed convolution layer outputs the third local feature image.

[0122] In an optional embodiment, the convolution layer and the transposed convolution layer are completely symmetrical structures, and the size and number of channels of the third feature image are the same as those of the first feature image. A direct connection can be established between the convolution layer and the transposed convolution layer corresponding to feature images of the same size, so as to obtain a third local feature image with more comprehensive information.

[0123] Step S840: Using the fourth pixel rearrangement layer, the third local feature image is subjected to pixel rearrangement processing from multi-channel to single-channel, so as to obtain a blemish-free local face image.

[0124] Among them, the fourth pixel rearrangement layer is equivalent to the second pixel rearrangement layer, and its process is opposite to that of the third pixel rearrangement layer. The fourth pixel rearrangement layer can rearrange multiple channels of the third local feature image corresponding to the local face image into one channel. The blemish-free local face image can be a blemish-free image obtained by combining local images of multiple parts, or it can be a blemish-free image corresponding to the local image of each part, which is determined according to the number of the second machine learning models and specific needs. The specific implementation method is the same as that in step S740 and will not be repeated here.

[0125] The blemish-removed partial facial image may be an image obtained by performing blemish removal processing on the partial facial image using a second machine learning model. The blemish removal processing includes but is not limited to blemishes such as red bloodshot eyes and noise at corners of the mouth, and blemish removal processing performed on second type blemishes.

[0126] refer to Figure 8 According to the steps shown in , the partial face image 805 can be used as input to the third pixel rearrangement layer 801, and then the partial face image 805 is processed by at least one convolution layer 802, at least one transposed convolution layer 803 and the fourth pixel rearrangement layer 804 to remove defects, thereby obtaining a defect-removed partial face image 808 corresponding to the partial face image 805. The partial face image 806 and the partial face image 807 can also be used as inputs to obtain a defect-removed partial face image 809 corresponding to the partial face image 806 and a defect-removed partial face image 810 corresponding to the partial face image 807.

[0127] In the disclosed embodiment, the second machine learning model is used to perform blemish removal on the partial face image that matches the input image size, and a corresponding blemish-free partial face image is obtained. Since the second machine learning model has fewer network layers, the efficiency of blemish removal on the partial face image can be improved, the time consumption of blemish removal processing is reduced, and the memory usage is reduced.

[0128] In one embodiment, in order to improve the accuracy of the network, the machine learning model can be trained to obtain a second machine learning model. The specific training process is the same as the training process of the first machine learning model, and will not be repeated here.

[0129] It should be noted that after being fully trained with a comprehensive data set, the first machine learning model and the second machine learning model can cope with almost all situations in practical applications, including different lighting conditions, different skin conditions, different types of blemishes, etc., to accurately and fully detect and remove portrait blemishes, and solve the problems of incomplete blemish removal and certain limitations in related technologies.

[0130] In the disclosed embodiments, the image to be beautified is transformed into a global face image and a local face image with different resolutions according to the minimum processing resolution requirements for different types of defects / noise on the image to be beautified. The global face image is globally beautified by a first machine learning model, and the local face image is locally beautified by a second machine learning model. By using multiple machine learning models to classify the images to be beautified and perform beautification and blemish removal, the limitations caused by different resolutions in a single beautification and blemish removal process are avoided, the blemish removal effect can be improved, the overall power consumption is reduced, and the processing efficiency is improved.

[0131] Continue to refer Figure 2 As shown in , in step S250, based on the blemish-removed global face image and the blemish-removed local face image, a target beautified image corresponding to the image to be beautified is obtained.

[0132] In the embodiment of the present disclosure, after obtaining the global image of the blemish-free face and the local image of the blemish-free face, the two can be integrated to obtain a target beauty image. The target beauty image can be a blemish-free image corresponding to the image to be beauty-enhanced, which is specifically determined according to the outputs of the first machine learning model and the second machine learning model.

[0133] The specific implementation of step S250 may be: optimizing the global image of the blemish-free face and the local image of the blemish-free face to obtain a target beauty image. Fig. 9 The flowchart for generating the target beauty image is schematically shown in FIG. Fig. 9 As shown in , it mainly includes the following steps:

[0134] In step S910, the blemish-removed global face image is optimized to generate an intermediate global face image, and the blemish-removed local face image is optimized to obtain an intermediate local face image.

[0135] In the disclosed embodiment, the intermediate global face image refers to the final global result obtained by performing global blemish removal on the global face image, and the intermediate local face image refers to the final local result obtained by performing local blemish removal on the local face image. Image optimization includes but is not limited to one or more of resizing, beauty processing, and fusion processing.

[0136] When the blemish-removed global face image is optimized to obtain an intermediate global face image, the blemish-removed transformed face image corresponding to the blemish-removed global face image can be obtained, and the blemish-removed transformed face image is fused with the image to be beautified to obtain the intermediate global face image.

[0137] Among them, according to the padding and combination information in the saved size transformation information, the low-resolution resized rectangular face image after blemish removal can be taken out from the low-resolution fixed input size image after blemish removal represented by the blemish-removed global face image as the blemish-removed transformed face image. Furthermore, the fusion processing can be performed based on the blemish-removed transformed face image and the image to be beautified to obtain an intermediate global face image. Among them, the image to be beautified used for performing the fusion processing refers to the original image represented by the resized image to be beautified corresponding to the rectangular face image after blemish removal and size transformation processing. Among them, the implementation method of fusing the blemish-removed transformed face image with the image to be beautified to obtain the intermediate global face image can include the following steps one to three, specifically:

[0138] Step 1: determining a difference between the blemish-removed transformed face image and the image to be beautified, and determining a limit face high-frequency image of the image to be beautified according to the difference.

[0139] In this step, the difference may be the difference between pixels at corresponding positions. Fig.10 The schematic diagram shows the process of determining and limiting the high-frequency information of the face, refer to Fig.10 As shown in , it mainly includes the following steps:

[0140] In step S1001, the blemish-removed transformed face image is subtracted from the corresponding image to be beautified to obtain a difference value of each pixel;

[0141] In step S1002, if the difference of the pixel point meets the defect condition, the pixel position of the pixel point is determined;

[0142] In step S1003, in the high-frequency image of the beautified image, the pixel values ​​of the pixel points within the area of ​​the pixel position are adjusted to within the value range to obtain a limited-value high-frequency image of the face.

[0143] In the disclosed embodiment, the corresponding image to be beautified may be the original image after size transformation. The pixels at the corresponding positions of the blemish-removed transformed face image and the corresponding original image after size transformation may be subtracted, and the subtracted value may be used as the difference of each pixel. Since the image to be beautified and the blemish-removed transformed face image are usually very similar, the pixel values ​​in the high-frequency image obtained based on their difference are generally small, such as the values ​​of each RGB channel do not exceed 4. However, for the mutation position in the image to be beautified, such as a small black mole on the face, it has strong high-frequency information, so the pixel value of the corresponding position in the high-frequency image may be relatively large. When performing image fusion, the pixel values ​​at these positions may have adverse effects, such as producing sharp edges such as "mole marks", resulting in unnatural visual perception. Therefore, it is possible to further determine whether the difference of the pixel points meets the defect condition. If so, the pixel position of the pixel point that meets the defect condition can be determined.

[0144] Among them, the defect condition is used to measure the difference between the image to be beautified and the face image to be transformed after blemish removal, so as to determine whether each pixel needs to be processed. Exemplarily, the defect condition may include: the difference of each color channel is greater than the first color difference threshold, and at least one of the difference of each color channel is greater than the second color difference threshold. The first color difference threshold and the second color difference threshold may be empirical thresholds. For example, when the color channel includes RGB, the first color difference threshold may be 20, and the second color difference threshold may be 40. Thus, after obtaining the difference of each pixel, the specific difference of the three color channels of RGB in the difference is judged to determine whether the difference of each color channel is greater than 20, and whether the maximum value of the difference of the three color channels is greater than 40. When the pixel difference of a certain pixel is determined by the double threshold method to meet these two conditions, it means that the defect condition is met, and the pixel position of the pixel can be determined.

[0145] Furthermore, the pixel position found by the double threshold method has a sharp edge in the high-frequency image, which may produce an unwanted effect and reduce the image quality. Based on this, the pixel values ​​of the pixel points in the regional range of the pixel position in the high-frequency image of the beauty image can be limited. It can be adjusted to the value range to obtain a limited face high-frequency image. When performing the limit processing, these pixel positions can be calculated according to the saved size transformation information to obtain their corresponding positions in the high-frequency image of the beauty image. The pixel points in the regional range can be the pixel points around the pixel position, for example, it can be a 5*5 pixel area centered on the pixel position. The specific size can be determined according to the size of the high-frequency image, and the present disclosure does not limit it. The value range is generally a smaller numerical range, which can be set according to actual needs, for example, it can be (-2, 2). Exemplarily, the value range can be -2 to 2, and the pixel values ​​in the regional range may exceed -5 to 5, so it can be limited to adjust it to -2 to 2. By limiting the pixel values ​​of the pixels around the pixel position, a high-frequency image of the image to be beautified after local limit processing is obtained, which can avoid the appearance of unnatural marks and other sharp edges, improve image quality, and increase the visual natural feeling.

[0146] It should be supplemented that the high-frequency image can be acquired according to the following method: when acquiring the global face image based on the image to be beautified, if the image to be beautified is scaled, the scaled face image obtained by the size scaling is inversely transformed according to the size transformation information to obtain the inversely transformed face image to determine the low-frequency image of the image to be beautified; based on the difference between the image to be beautified and the low-frequency image of the image to be beautified, the high-frequency image of the image to be beautified is acquired; the low-frequency image of the image to be beautified has the same resolution as the image to be beautified.

[0147] The resizing information may include the scaling direction and the scaling range. The resolution of the scaled face image is lower than that of the original image to be beautified, and the high-frequency information of the image will be lost during the resizing process. It should be noted that if the image to be beautified is rotated before being resized, the scaled face image can be reversely rotated after the inverse transformation so that the directions of the two images are the same.

[0148] The inverse transformation may be the inverse transformation of the resizing process, such as size enlargement. The inverse transformation may adopt methods such as bilinear interpolation and nearest neighbor interpolation. Although the resolution can be restored by the inverse transformation, it is difficult to restore the lost high-frequency information, that is, the inverse transformed face image can be regarded as a low-frequency image of the image to be beautified.

[0149] refer to Fig.11 The flowchart of obtaining a high-frequency image is shown in , and obtaining the high-frequency image of the image to be beautified mainly includes the following steps:

[0150] In step S1101, an image to be beautified is obtained;

[0151] In step S1102, the image to be beautified is resized;

[0152] In step S1103, the low-frequency image is obtained by enlarging the size according to the size transformation information;

[0153] In step S1104, the image to be beautified is subtracted from the low-frequency image to obtain a high-frequency image.

[0154] In another implementation, the high-frequency information may be extracted by filtering the image to be beautified to obtain the high-frequency image. It should be noted that if the image to be beautified is not resized, the high-frequency information does not need to be calculated.

[0155] Step 2: A rectangular low-frequency face image can be determined based on the blemish-removed transformed face image. The specific steps of obtaining the rectangular low-frequency face image include: fusing the blemish-removed transformed face image with the image to be beautified according to the beautification degree parameter to obtain a degree rectangular face image; and enlarging the degree rectangular face image to its original size to obtain a rectangular low-frequency face image.

[0156] The beautification degree parameter may be a beautification intensity parameter under a specific beautification function, such as the degree of blemish removal. Specifically, it may be used to represent the current beautification parameter, such as the currently set parameter, the system default parameter, or the parameter used in the last beautification. After determining the beautification degree parameter, the beautification degree parameter may be used as a weight to fuse the image to be beautified with the blemish removal transformed face image to obtain a rectangular face image that is beautified according to the beautification degree parameter.

[0157] Specifically, assuming that the range of the blemish removal degree is 0 to 100 and the currently set beautification degree parameter is a, the degree of rectangular face image can be determined according to formula (1):

[0158]

[0159] Wherein, image represents the degree rectangular face image, ori_image represents the image to be beautified, and deblemish_image represents the blemish-removed face image. When a is 0, it means no blemish removal is performed, and the image to be beautified is used completely; when a is 100, it means complete blemish removal, and the blemish-removed face image is used completely. Therefore, through the fusion method in formula (1), images corresponding to different beautification degree parameters can be obtained, and the rectangular face low-frequency image is between the image to be beautified and the blemish-removed face image, that is, the larger the beautification degree parameter, the more obvious the beautification effect.

[0160] On this basis, the rectangular face image can be enlarged to its original size to obtain a rectangular face low-frequency image. Specifically, the enlargement can be performed according to the size transformation information used for the blemish-removed face image, and the size transformation information can be information for scaling the original image to be beautified.

[0161] Step three, fusing the rectangular face low-frequency image with the limit face high-frequency image to obtain a blemish-free global fused face image, and determining the blemish-free global fused face image as the intermediate face global image.

[0162] The pixel values ​​of the low-frequency rectangular face image of the original size after the blemishes are removed according to the beautification degree parameter and the high-frequency face image of the limit value can be added to obtain the global fused face image after the blemishes are removed. The global fused face image after the blemishes are removed refers to the rectangular face image after the blemishes are removed according to the beautification degree parameter. And the global fused face image after the blemishes are removed can be used as the result of the overall beautification.

[0163] In the disclosed embodiment, by performing image optimization on the blemish-free global face image output by the first machine learning model, the overall beautification effect and image quality can be improved.

[0164] The specific implementation method of performing image optimization on the defect-removed partial face image to generate an intermediate partial face image may include the following steps 1 to 4, wherein:

[0165] Step 1: resize the blemish-free partial face image to obtain a blemish-free transformed partial face image. Exemplarily, the blemish-free partial face image output by the second machine learning model can be resized according to the saved resize information for resizing the partial image in the beautified image.

[0166] Step 2: According to the beautification degree parameter, the blemish removal transformed local image and the to-beautify image are fused to obtain a degree face local image. The to-beautify image used for fusion refers to a local image in the to-beautify image, that is, the original image after size transformation corresponding to the blemish removal transformed local image. The image formed by combining local images of multiple parts can be fused once, or the local images of multiple parts can be fused separately, which is not limited here. By way of example, the fusion can be performed in the manner of formula (1), which will not be repeated here.

[0167] Step 3: Enlarging the partial face image to its original size to obtain a partial low-frequency face image. The partial face image may be enlarged to its original size according to the previously saved size transformation information to obtain a partial low-frequency face image.

[0168] Step 4: Fuse the local low-frequency image of the face with the high-frequency image of the local face image to obtain a blemish-free local fused face image as the intermediate local face image. The high-frequency image of the local face image can be determined based on the difference between the local image in the image to be beautified and the low-frequency image obtained by scaling the local image. The specific determination method can be the same as Fig.11 The steps in are the same and will not be repeated here. Based on this, the pixel values ​​of the local low-frequency image of the original size of the face after blemish removal according to the beautification degree parameter and the high-frequency image of the local image can be added to obtain the local fused face image after blemish removal, and the local fused face image after blemish removal can be used as the intermediate local face image to represent the result of local beautification. The local fused face image after blemish removal refers to the rectangular face image after local blemish removal according to the beautification degree parameter.

[0169] The technical solution in the disclosed embodiment can perform the above-mentioned processing on each face to obtain the overall beautification result and the local beautification result corresponding to each face, thereby improving accuracy and pertinence.

[0170] Next, continue to refer to Fig. 9 As shown in , in step S920, the target beauty image is obtained according to the intermediate human face partial image and the intermediate human face global image.

[0171] In the disclosed embodiment, since multiple beautifications are performed, it is necessary to fill the intermediate face partial image and the intermediate face global image into the image to be beautified through multiple fillings, thereby obtaining the target beautified image. When filling, the whole image can be filled first and then the partial image, or the partial image can be filled first and then the whole image, as long as all the beautification results are filled. In the disclosed embodiment, the whole image is filled first and then the partial image is filled for specific description.

[0172] Fig.12 A flowchart for filling is schematically shown in FIG. Fig.12 As shown in , it mainly includes the following steps:

[0173] In step S1201, the intermediate global face image is filled into a corresponding position in the image to be processed corresponding to the image to be beautified, so as to generate an intermediate beautified image.

[0174] In the disclosed embodiment, the overall beautification result represented by the global image of the middle face can be filled into the corresponding position in the image to be processed to which the image to be beautified belongs, that is, filled into the overall image where the image to be beautified is located, so as to obtain the overall image represented by the middle beautification image. Exemplarily, the boundary area of ​​the global image of the middle face can be gradually merged with the image to be processed. Specifically, the global image of the middle face can be divided into a global central area and a global boundary area. The global boundary area can be a partial area including the boundary. The width of the global boundary area is, for example, 10% or 10 pixels of the global image of the middle face, and the global boundary area extends inward from the outer edge of the global image of the middle face. The global central area refers to the remaining area in the global image of the middle face except the global boundary area. The global boundary area generally does not cover the face area. During the filling process, the global image of the middle face in the global central area can be directly replaced with the image at the corresponding position in the image to be processed and the image to be beautified. Furthermore, the global image of the middle face in the global boundary area can be gradually merged with the image at the corresponding position in the image to be processed and the image to be beautified, so that the global boundary area achieves a smooth transition. The gradient fusion here means that the ratio of the pixel values ​​of the pixel points in the global boundary area far from the global center area to the image to be processed is greater than the ratio of the pixel values ​​of the pixel points in the global boundary area close to the global center area to the image to be processed, and the pixel values ​​of the pixel points at the outer edge of the global boundary area relative to the global center area are the pixel values ​​of the image to be processed. For example, after determining the global boundary area, the inner edge pixels of the global boundary area are obtained, recorded as the first pixel; the outer edge pixels are obtained, recorded as the second pixel; the first pixel and the second pixel are gradiently processed on the global boundary area until the pixels at the outer edge are the pixel values ​​of the image to be processed.

[0175] In step S1202, the intermediate partial face image is filled into the intermediate beautification image to generate the target beautification image.

[0176] In the disclosed embodiment, the middle face partial image can be divided into a local center area and a local boundary area. The width of the local boundary area can be, for example, 10% of the middle face partial image or 10 pixels, etc., which is specifically determined according to the size of the image. The local center area refers to the remaining area in the middle face partial image except the local boundary area. During the filling process, the middle face partial image in the local center area can be directly replaced with the image at the corresponding position in the middle beauty image. Furthermore, the middle face partial image in the local boundary area can be gradually merged with the image at the corresponding position in the middle beauty image, so that the local boundary area achieves a smooth transition. The pixel value of the pixel point of the local boundary area relative to the outer edge of the local center area is the pixel value of the middle beauty image.

[0177] Fig.13 The specific process of gradient fusion is schematically shown in Fig.13 As shown in , for the image to be processed 1301 and the intermediate global face image 1302 and the intermediate partial face image 1303, the intermediate global face image 1302 can first be gradually merged with the image to be processed in region 1310, and replaced in region 1311 to obtain an intermediate beauty image 1304. Further, the intermediate partial face image 1303 can be gradually merged with the intermediate beauty image 1304 in region 1320, and replaced in region 1321 to obtain a target beauty image 1305.

[0178] It should be noted that when there are multiple faces, the corresponding image to be beautified in the image to be processed can be replaced by the global image of the middle face corresponding to each face, and the global boundary area can be gradient processed to obtain an intermediate beautified image; further, the corresponding area in the intermediate beautified image can be replaced by the local image of the middle face corresponding to each face, and the local boundary area can be transitionally smoothed to obtain the target beautified image corresponding to each face. It should be noted that for the image to be beautified containing multiple faces, they can be filled in the order from large to small in area, or in other orders, which are not limited here.

[0179] By achieving transition processing through overall and local gradient fusion, it is possible to avoid unnatural boundaries between the image after blemish removal and the image to be filled in at the edge, prevent color mutations that cause visual disharmony, and improve image quality.

[0180] Fig.14 The schematic flow of the image beautification processing method is shown in FIG. Fig.14 As shown in , it mainly includes the following steps:

[0181] Step S1401, generate multiple face frames in the image to be processed according to the recognized face key points, and retain the face frames whose area is not less than the face area threshold. Assume that the image to be processed includes a main face, for example, face frame 1 is generated accordingly.

[0182] Step S1402 , enlarging the face frame 1 , for example, to 1.1 times, and then capturing the image in the face frame to obtain the image 1 to be beautified.

[0183] Step S1403, resize the image 1 to be beautified and resize the local image 2 in the image to be beautified to obtain a global face image 1 and a local face image 2. The resize resize may be scaling, rotating, or filling.

[0184] Step S1404, processing the global face image 1 according to the first machine learning model to obtain the blemish-free global face image 1, and processing the local face image 2 according to the second machine learning model to obtain the blemish-free local face image 2.

[0185] Step S1405: scaling the image 1 to be beautified and performing an inverse transformation to obtain a low-frequency image 1. scaling the local image 2 and performing an inverse transformation to obtain a low-frequency image 2.

[0186] Step S1406 , obtaining a high-frequency image 1 by subtracting the image to be beautified 1 from the low-frequency image 1 ; and obtaining a high-frequency image 2 by subtracting the local image 2 from the low-frequency image 2 .

[0187] Step S1407 , obtaining an intermediate global face image 1 according to the blemish-removed global face image 1 and the high-frequency image 1 ; obtaining an intermediate local face image 2 according to the blemish-removed local face image 2 and the high-frequency image 2 .

[0188] Step S1408 , fusing the intermediate global face image 1 with the image to be processed to obtain an intermediate beautified image 1 .

[0189] Step S1409 , fusing the intermediate beautified image 1 with the intermediate face partial image 2 to obtain a target beautified image.

[0190] The technical solution provided by the above-mentioned image beautification processing method, on the one hand, performs global beautification on the global face image through the first machine learning model, performs local beautification on the local face image through the second machine learning model, and performs multiple beautification processes on areas with different resolution processing requirements in an image through different models, thereby avoiding the limitation of only one model being used for beautification processing in the related art, increasing the flexibility and pertinence of image beautification processing, and improving the beautification effect and image quality. On the other hand, since the resolution requirements of images processed by different models are different, compared with using only one model for processing, it reduces time consumption and memory usage, improves beautification efficiency and reduces power consumption.

[0191] In one embodiment, Figure 2 The image beautification method shown in the figure is used as the first stage of beautification processing. Figure 2 Before or after the image beautification processing method, other stages of beautification processing are added. For example, the above machine learning model is used to remove blemishes from the image. After obtaining the image to be processed, Figure 2 The image beautification processing method is used to process the target beautification image, and the obtained target beautification image is the blemish-free image. The beautification processing in other stages may be personalized beautification processing, so the blemish-free image may be subsequently subjected to personalized beautification processing to obtain the final beautification image.

[0192] Generally, blemish removal is necessary for image beautification, and users have a relatively fixed demand for blemish removal. Figure 2 The image beautification processing method realizes a universal blemish removal beautification processing flow. In contrast, skin smoothing, deformation, three-dimensional, skin color adjustment, light and shadow adjustment and other processing are not necessary, and the specific needs of users for these processing are also personalized. These processing can be called personalized beauty processing, which usually requires users to make specific settings before processing. For example, the user selects one or more beauty functions and sets parameters such as skin smoothing degree and deformation degree, and then the terminal device processes according to the user's settings.

[0193] It should be noted that this disclosure is for Figure 2 The order of image beautification processing and other stages of beautification processing is not limited. For example, the image to be processed can be first subjected to personalized beautification processing to obtain an intermediate beautification image, and then the intermediate beautification image can be used as the image to be processed and executed. Figure 2 The target beautified image is the final output beautified image.

[0194] The technical solution provided in the embodiment of the present disclosure adopts a two-stage portrait image beautification processing method based on a portrait blemish removal DNN. The first stage performs portrait blemish removal processing, and cuts out the face area image according to the face frame information. According to the minimum processing resolution requirements for different types of blemish processing and noise processing. The size of the complete face area image is transformed to the global face image corresponding to the first machine learning model, and input into the first machine learning model for blemish removal processing. The local face images with high minimum processing resolution requirements such as bloodshot eyes / noise, dirty nose wings / nose hair, dirty corners of the mouth / noise are resized and combined to the input image size of the second machine learning model, and input into the second machine learning model for blemish removal. Using multiple machine learning models, for different types of blemishes and noise, according to the minimum processing resolution requirements corresponding to the blemish type, independent scaling processing and model processing of different degrees are performed multiple times, so as to obtain the best processing effect for various types of blemishes, and reduce power consumption compared to single processing of the complete face with high resolution. Multiple models all use lightweight convolutional neural networks with short processing time and low power consumption. The processing time and power consumption can be further controlled by setting different fixed sizes, making them applicable to mobile devices with different computing power levels, such as high, medium, and low, thereby improving versatility and application scope.

[0195] The exemplary embodiment of the present disclosure also provides an image beautification processing device. Fig.15 As shown, the image beautification processing device 1500 may include:

[0196] Image acquisition module 1501, used to acquire the image to be beautified;

[0197] The image splitting module 1502 is used to obtain a global face image and a partial face image corresponding to the image to be beautified; the resolution of the global face image is smaller than the resolution of the partial face image;

[0198] A first image processing module 1503 is used to process the global face image by using a first machine learning model to obtain a blemish-free global face image;

[0199] A second image processing module 1504 is used to process the partial face image by using a second machine learning model to obtain a blemish-free partial face image;

[0200] The image generation module 1505 is used to obtain a target beautified image corresponding to the image to be beautified based on the blemish-removed global face image and the blemish-removed local face image.

[0201] In an exemplary embodiment of the present disclosure, the first machine learning model includes: a first pixel rearrangement layer, at least one convolution layer, at least one transposed convolution layer, and a second pixel rearrangement layer; the first image processing module includes: a first pixel rearrangement module, used to perform a first pixel rearrangement process on the global face image using the first pixel rearrangement layer to obtain a first feature image; a first convolution module, used to perform a convolution process on the first feature image using the convolution layer to obtain a second feature image; a first transposed convolution module, used to perform a transposed convolution process on the second feature image using the transposed convolution layer to obtain a third feature image; a second pixel rearrangement module, used to perform a second pixel rearrangement process on the third feature image using the second pixel rearrangement layer to obtain the blemish-free global face image.

[0202] In an exemplary embodiment of the present disclosure, the number of the second machine learning models is at least one; the second image processing module includes: a combined processing module, used to process the partial face image composed of multiple parts through a second machine learning model, to generate the blemish-free partial face image; or, a separate processing module, used to process the partial face images corresponding to multiple parts respectively through multiple second machine learning models, to generate multiple blemish-free partial face images; wherein the second machine learning model corresponds one-to-one to the partial face image.

[0203] In an exemplary embodiment of the present disclosure, the second machine learning model includes: a third pixel rearrangement layer, at least one convolution layer, at least one transposed convolution layer, and a fourth pixel rearrangement layer; the combined processing module includes: a third pixel rearrangement module, used to perform a third pixel rearrangement process on the local face image using the third pixel rearrangement layer to obtain a first local feature image; a second convolution module, used to perform a convolution process on the first local feature image using the convolution layer to obtain a second local feature image; a second transposed convolution module, used to perform a transposed convolution process on the second local feature image using the transposed convolution layer to obtain a third local feature image; a fourth pixel rearrangement module, used to perform a fourth pixel rearrangement process on the third local feature image using the fourth pixel rearrangement layer to obtain the blemish-free local face image.

[0204] In an exemplary embodiment of the present disclosure, an image splitting module includes: a splitting control module, used to determine a global image and local images of multiple parts corresponding to the image to be beautified based on multiple beauty attribute parameters; an adjustment module, used to adjust the global image to obtain the global face image, and adjust the local image to obtain the local face image.

[0205] In an exemplary embodiment of the present disclosure, the adjustment module includes: a global adjustment module, which is used to adjust the global image based on the input image size of the first machine learning model to obtain the global face image.

[0206] In an exemplary embodiment of the present disclosure, the adjustment module includes: a combined adjustment module, which is used to combine and adjust the local images of the multiple parts based on an input image size of the second machine learning model to obtain the local face image; or, a separate adjustment module, which is used to adjust the local images of the multiple parts separately based on the input image sizes of multiple second machine learning models to generate multiple local face images.

[0207] In an exemplary embodiment of the present disclosure, the global adjustment module includes: a size transformation module, which is used to transform the size of the global image based on the input image size of the first machine learning model to obtain a global face image if the size parameters of the global image do not meet the size conditions; wherein the size transformation includes one or more of rotation, resizing and padding.

[0208] In an exemplary embodiment of the present disclosure, the first machine learning model and the second machine learning model are both deep convolutional neural networks, and the number of network layers of the second machine learning model is less than the number of network layers of the first machine learning model.

[0209] In an exemplary embodiment of the present disclosure, the image generation module includes: an image optimization module, which is used to perform image optimization on the blemish-removed global face image to generate an intermediate global face image, and to perform image optimization on the blemish-removed local face image to obtain an intermediate local face image; and an image integration module, which is used to obtain the target beauty image based on the intermediate local face image and the intermediate global face image.

[0210] In an exemplary embodiment of the present disclosure, the image optimization module includes: a transformation module, used to obtain the blemish-removed transformed face image corresponding to the blemish-removed global face image; and a fusion processing module, used to fuse the blemish-removed transformed face image with the image to be beautified to obtain the intermediate global face image.

[0211] In an exemplary embodiment of the present disclosure, the fusion processing module includes: a pixel limit module, which uses a crossbow to determine the difference between the blemish-removed transformed face image and the image to be beautified, and determines the limit face high-frequency image of the image to be beautified based on the difference; a low-frequency acquisition module, which is used to determine a rectangular face low-frequency image based on the blemish-removed transformed face image; and a fusion module, which is used to fuse the rectangular face low-frequency image with the limit face high-frequency image to obtain a global fused face image after blemish removal, and determine the global fused face image after blemish removal as the intermediate face global image.

[0212] In an exemplary embodiment of the present disclosure, the low-frequency acquisition module is configured to: fuse the blemish-removed transformed face image with the image to be beautified according to the beautification degree parameter to obtain a degree rectangular face image; and enlarge the degree rectangular face image to its original size to obtain a rectangular face low-frequency image.

[0213] In an exemplary embodiment of the present disclosure, the pixel limit module includes: a difference determination module, which is used to subtract the blemish-removed transformed face image from the corresponding image to be beautified to obtain the difference of each pixel; a pixel position determination module, which is used to determine the pixel position of the pixel if the difference of the pixel meets the defect condition; and a range limitation module, which is used to adjust the pixel values ​​of the pixel points within the area range of the pixel position in the high-frequency image of the image to be beautified to within the value range to obtain a limited face high-frequency image.

[0214] In an exemplary embodiment of the present disclosure, the device also includes: an inverse transformation module, which is used to, when obtaining the global face image according to the image to be beautified, if the image to be beautified is resized, inversely transform the scaled face image obtained by the rescaling according to the size transformation information to obtain an inversely transformed face image, so as to determine a low-frequency image of the image to be beautified; and a high-frequency image acquisition module, which is used to acquire a high-frequency image of the image to be beautified according to a difference between the image to be beautified and the low-frequency image of the image to be beautified; the low-frequency image of the image to be beautified has the same resolution as the image to be beautified.

[0215] In an exemplary embodiment of the present disclosure, the image optimization module includes: a size transformation module, configured to perform size transformation on the blemish-removed partial face image to obtain a blemish-removed transformed partial face image;

[0216] The beautification degree determination module is used to fuse the blemish-removed transformed local image with the local image in the image to be beautified according to the beautification degree parameter to obtain a degree-specific human face local image; the enlargement module is used to enlarge the degree-specific human face local image to the original size to obtain a human face local low-frequency image; the fusion module is used to fuse the human face local low-frequency image with the high-frequency image of the local image to obtain a blemish-removed local fused human face image as an intermediate human face local image.

[0217] In an exemplary embodiment of the present disclosure, the image integration module includes: a first filling module, used to fill the intermediate global face image into the corresponding position of the image to be processed corresponding to the image to be beautified, so as to generate an intermediate beautified image; a second filling module, used to fill the intermediate partial face image into the intermediate beautified image, so as to generate the target beautified image.

[0218] In an exemplary embodiment of the present disclosure, a first filling module includes: an area determination module, used to divide the intermediate human face global image into a global central area and a global boundary area; a replacement module, used to replace the image to be processed according to the intermediate human face global image in the global central area; a gradient fusion module, used to gradiently fuse the intermediate human face global image in the global boundary area with the image to be processed to obtain the intermediate beauty image; wherein the pixel value of the pixel point at the outer edge of the global boundary area relative to the global central area is the pixel value of the image to be processed.

[0219] In an exemplary embodiment of the present disclosure, the second filling module is configured to: replace the intermediate beauty image according to the local central area of ​​the intermediate partial face image; gradually merge the local boundary area of ​​the intermediate partial face image with the intermediate beauty image to obtain the target beauty image; wherein the pixel values ​​of the pixel points of the local boundary area relative to the outer edge of the local central area are the pixel values ​​of the intermediate beauty image.

[0220] In an exemplary embodiment of the present disclosure, the target beautification image is a blemish-removed beautification image. After obtaining the blemish-removed beautification image, the device further includes: a personalized beautification module, configured to perform personalized beautification processing on the blemish-removed beautification image to obtain the beautification image.

[0221] The specific details of each part of the above-mentioned image beautification processing device 1500 have been described in detail in the implementation method of the image beautification processing method. The undisclosed details can be found in the implementation method of the method part, and thus will not be repeated here.

[0222] The exemplary embodiment of the present disclosure also provides an electronic device for executing the above-mentioned image beautification processing method. The electronic device may be the above-mentioned terminal, including but not limited to a smart phone, a tablet computer, a wearable device, a computer, etc. In general, the electronic device includes a processor and a memory. The memory is used to store executable instructions of the processor, and may also store application data, such as image data, game data, etc.; the processor is configured to execute the image beautification processing method in this exemplary embodiment by executing executable instructions.

[0223] Below Fig.16 The mobile terminal 1600 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Fig.16 The construction in can also be applied to fixed type equipment.

[0224] like Fig.16 As shown, the mobile terminal 1600 may specifically include: a processor 1601, a memory 1602, a bus 1603, a mobile communication module 1604, an antenna 1, a wireless communication module 1605, an antenna 2, a display screen 1606, a camera module 1607, an audio module 1608, a power module 1609 and a sensor module 1610.

[0225] The processor 1601 may include one or more processing units, for example, the processor 1601 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc. The image beautification processing method in this exemplary embodiment may be executed by an AP, a GPU or a DSP. When the method involves processing related to a neural network, it may be executed by an NPU, for example, the NPU may load neural network parameters and execute algorithm instructions related to the neural network.

[0226] The encoder can encode (i.e. compress) an image or video to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of the image or video to restore the image or video data. The mobile terminal 1600 can support one or more encoders and decoders, such as: image formats such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), and video formats such as MPEG (Moving Picture Experts Group) 1, MPEG10, H.1063, H.1064, and HEVC (High Efficiency Video Coding).

[0227] The processor 1601 may be connected to the memory 1602 or other components via a bus 1603 .

[0228] The memory 1602 may be used to store computer executable program codes, which may include instructions. The processor 1601 executes various functional applications and data processing of the mobile terminal 1600 by running the instructions stored in the memory 1602. The memory 1602 may also store application data, such as images, videos, and other files.

[0229] The communication function of the mobile terminal 1600 can be implemented by the mobile communication module 1604, antenna 1, wireless communication module 1605, antenna 2, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 1604 can provide 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 1600. The wireless communication module 1605 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near field communication applied to the mobile terminal 1600.

[0230] The display screen 1606 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The camera module 1607 is used to implement shooting functions, such as shooting images, videos, etc., and the camera module may include a color temperature sensor array. The audio module 1608 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 1609 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 1610 may include one or more sensors for implementing corresponding sensing detection functions. For example, the sensor module 1610 may include an inertial sensor, which is used to detect the motion posture of the mobile terminal 1600 and output inertial sensing data.

[0231] It should be noted that a computer-readable storage medium is also provided in an embodiment of the present disclosure. The computer-readable storage medium may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

[0232] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0233] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0234] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.

[0235] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0236] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0237] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0238] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims. It should be understood that the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image beautification processing method, characterized in that: include: Get the image to be beautified; Acquire a global face image and a partial face image corresponding to the image to be beautified; the resolution of the global face image is smaller than the resolution of the partial face image; Processing the global face image by a first machine learning model to obtain a blemish-free global face image; Processing the partial face image by a second machine learning model to obtain a blemish-free partial face image; Performing image optimization on the blemish-free global face image to generate an intermediate global face image, and performing image optimization on the blemish-free local face image to obtain an intermediate local face image; acquiring a target beauty image based on the intermediate local face image and the intermediate global face image; The step of performing image optimization on the defect-removed partial face image to generate an intermediate partial face image includes: Resizing the blemish-removed partial face image to obtain a blemish-removed transformed partial face image; According to the beautification degree parameter, the blemish removal transformed partial image is merged with the partial image in the image to be beautified to obtain a partial image of the face; Enlarging the partial face image to its original size to obtain a partial low-frequency face image; The local low-frequency image of the face is fused with the high-frequency image of the local image to obtain a defect-free local fused face image as an intermediate local face image.

2. The image beautification processing method according to claim 1, characterized in that: The first machine learning model includes: a first pixel rearrangement layer, at least one convolution layer, at least one transposed convolution layer, and a second pixel rearrangement layer; the process of processing the global face image by the first machine learning model to obtain a blemish-free global face image includes: Using the first pixel rearrangement layer to perform a first pixel rearrangement process on the global face image to obtain a first feature image; Using the convolution layer to perform convolution processing on the first feature image to obtain a second feature image; Using the transposed convolution layer to perform a transposed convolution process on the second feature image to obtain a third feature image; The second pixel rearrangement layer is used to perform a second pixel rearrangement process on the third feature image to obtain the blemish-free global face image.

3. The image beautification processing method according to claim 1, characterized in that: The number of the second machine learning models is at least one; and the processing of the partial face image by the second machine learning model to obtain the defect-free partial face image includes: Processing the partial face image composed of multiple parts through a second machine learning model to generate the blemish-free partial face image; or, The partial facial images corresponding to the multiple parts are processed respectively through multiple second machine learning models to generate multiple blemish-free partial facial images; wherein the second machine learning models correspond one-to-one to the partial facial images.

4. The image beautification processing method according to claim 3, characterized in that: The second machine learning model includes: a third pixel rearrangement layer, at least one convolution layer, at least one transposed convolution layer, and a fourth pixel rearrangement layer; the second machine learning model is used to process the partial face image composed of multiple parts to generate the blemish-free partial face image, including: Using the third pixel rearrangement layer to perform a third pixel rearrangement process on the partial face image to obtain a first partial feature image; Using the convolution layer to perform convolution processing on the first local feature image to obtain a second local feature image; Using the transposed convolution layer to perform a transposed convolution process on the second local feature image to obtain a third local feature image; The fourth pixel rearrangement layer is used to perform a fourth pixel rearrangement process on the third local feature image to obtain the blemish-free human face local image.

5. The image beautification processing method according to claim 1, characterized in that: The step of obtaining a global face image and a partial face image corresponding to the image to be beautified includes: Based on multiple beautification attribute parameters, determining a global image and local images of multiple parts corresponding to the image to be beautified; The global image is adjusted to obtain the global face image, and the local image is adjusted to obtain the local face image.

6. The image beautification processing method according to claim 5, characterized in that: The step of adjusting the global image to obtain the global face image includes: Based on the input image size of the first machine learning model, the global image is adjusted to obtain the global face image.

7. The image beautification processing method according to claim 5, characterized in that: The number of the second machine learning models is at least one; and the step of performing an adjustment operation on the partial image to obtain the partial face image includes: Based on an input image size of the second machine learning model, combining the partial images of the plurality of parts and performing an adjustment operation to obtain the partial face image; or, Based on the input image sizes of multiple second machine learning models, the local images of multiple parts are adjusted respectively to generate multiple local face images.

8. The image beautification processing method according to claim 6, characterized in that: The step of adjusting the global image based on the input image size of the first machine learning model to obtain the global face image includes: If the size parameter of the global image does not meet the size condition, resize the global image based on the input image size of the first machine learning model to obtain a global face image; The size transformation includes one or more of rotation, size scaling and filling.

9. The image beautification processing method according to claim 1, characterized in that: Both the first machine learning model and the second machine learning model are deep convolutional neural networks, and the number of network layers of the second machine learning model is smaller than that of the first machine learning model.

10. The image beautification processing method according to claim 1, characterized in that: The step of performing image optimization on the defect-removed global face image to generate an intermediate global face image comprises: Obtaining a blemish-removed transformed face image corresponding to the blemish-removed global face image; The blemish-removed transformed face image is fused with the image to be beautified to obtain the intermediate global face image.

11. The image beautification processing method according to claim 10, characterized in that: The step of fusing the blemish-removed transformed face image with the image to be beautified to obtain the intermediate global face image includes: Determine the difference between the blemish-removed transformed face image and the image to be beautified, and determine a limit face high-frequency image of the image to be beautified according to the difference; Determine a rectangular human face low-frequency image based on the blemish-removed transformed human face image; The rectangular face low-frequency image is fused with the limit face high-frequency image to obtain a blemish-free global fused face image, and the blemish-free global fused face image is determined as the intermediate face global image.

12. The image beautification processing method according to claim 11, characterized in that: The step of determining a rectangular human face low-frequency image based on the blemish-removed transformed human face image comprises: The blemish-removed transformed face image and the image to be beautified are merged according to the beautification degree parameter to obtain a degree rectangular face image; The rectangular face image is enlarged to its original size to obtain a rectangular face low-frequency image.

13. The image beautification processing method according to claim 11, characterized in that: The step of determining a difference between the blemish-removed transformed face image and the image to be beautified, and determining a limit face high-frequency image of the image to be beautified according to the difference, comprises: Subtracting the blemish-removed transformed face image from the corresponding image to be beautified to obtain a difference value for each pixel; If the difference of the pixel point meets the defect condition, determining the pixel position of the pixel point; In the high-frequency image of the beautified image, the pixel values ​​of the pixel points within the area range of the pixel position are adjusted to within the value range to obtain a limited-value high-frequency image of the face.

14. The image beautification processing method according to claim 13, characterized in that: The method further comprises: When acquiring the global face image according to the image to be beautified, if the image to be beautified is resized, the scaled face image obtained by the resizing is inversely transformed according to the size transformation information to obtain an inversely transformed face image, so as to determine a low-frequency image of the image to be beautified; A high-frequency image of the image to be beautified is acquired according to a difference between the image to be beautified and a low-frequency image of the image to be beautified; the low-frequency image of the image to be beautified has the same resolution as the image to be beautified.

15. The image beautification processing method according to claim 1, characterized in that: The step of acquiring the target beauty image according to the intermediate human face partial image and the intermediate human face global image comprises: Filling the intermediate global face image into a corresponding position in the image to be processed corresponding to the image to be beautified, to generate an intermediate beautified image; The intermediate partial face image is filled into the intermediate beautification image to generate the target beautification image.

16. The image beautification processing method according to claim 15, characterized in that: Filling the intermediate global face image into a corresponding position in the image to be processed corresponding to the image to be beautified to generate the intermediate beautified image includes: Dividing the intermediate face global image into a global center region and a global boundary region; Replace the image to be processed according to the middle human face global image in the global central area; Gradually merge the intermediate global face image in the global boundary area with the image to be processed to obtain the intermediate beauty image; The pixel values ​​of the pixel points at the outer edge of the global boundary area relative to the global central area are the pixel values ​​of the image to be processed.

17. The image beautification processing method according to claim 15, characterized in that: Filling the intermediate partial face image into the intermediate beautification image to generate the target beautification image includes: replacing the intermediate beautification image according to the local central area of ​​the intermediate partial face image; Gradually merge the local boundary area of ​​the intermediate partial face image with the intermediate beautification image to obtain the target beautification image; The pixel values ​​of the pixel points at the outer edge of the local boundary area relative to the local central area are the pixel values ​​of the intermediate beauty image.

18. The image beautification processing method according to claim 1, characterized in that: The target beautification image is a blemish-free beautification image. After obtaining the blemish-free beautification image, the method further includes: Performing personalized beautification processing on the blemish-removed beautification image to obtain a beautification image.

19. An image beautification processing device, characterized in that: include: An image acquisition module, used to acquire the image to be beautified; An image splitting module, used to obtain a global face image and a local face image corresponding to the image to be beautified; the resolution of the global face image is smaller than the resolution of the local face image; A first image processing module, configured to process the global face image by using a first machine learning model to obtain a blemish-free global face image; A second image processing module, used for processing the partial face image by using a second machine learning model to obtain a blemish-free partial face image; An image generation module is used to optimize the blemish-free global face image to generate an intermediate global face image, and optimize the blemish-free local face image to obtain an intermediate local face image; and obtain a target beauty image based on the intermediate local face image and the intermediate global face image; The step of performing image optimization on the defect-removed partial face image to generate an intermediate partial face image includes: Resizing the blemish-removed partial face image to obtain a blemish-removed transformed partial face image; According to the beautification degree parameter, the blemish removal transformed partial image is merged with the partial image in the image to be beautified to obtain a partial image of the face; Enlarging the partial face image to its original size to obtain a partial low-frequency face image; The local low-frequency image of the face is fused with the high-frequency image of the local image to obtain a defect-free local fused face image as an intermediate local face image.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image beautification processing method according to any one of claims 1 to 18 is implemented.

21. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the image beautification processing method according to any one of claims 1 to 18 by executing the executable instructions.

Images