Image beautification processing method, device, storage medium and electronic device

Abstract

The present disclosure provides an image beautification processing method, an image beautification processing device, a storage medium and an electronic device, and relates to the field of imaging technology. The image beautification processing method includes: obtaining a face image to be beautified; performing feature extraction on the face image to be beautified through the main network of a multi-head machine learning model to obtain a feature image; performing beautification processing on the feature image through the basic branch network of the multi-head machine learning model to obtain a basic image of a face with blemishes removed; performing incremental beautification processing on the feature image through the local branch network of the multi-head machine learning model to obtain at least one difference image of a face with blemishes removed; fusing the face image to be beautified, the basic image of the face with blemishes removed and the at least one difference image of the face with blemishes removed to obtain a target beautification image corresponding to the face image to be beautified. The present disclosure performs different types of beautification processing on images, thereby improving the image beautification effect.

Description

Technical Field

[0001] The present disclosure relates to the field of imaging technology, and in particular to an image beautification processing method, an image beautification processing device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Beauty refers to the use of image processing technology to beautify portraits in images or videos to better meet the user's aesthetic needs.

[0003] In related technologies, images can be beautified overall. However, since images requiring beautification often contain a wide variety of blemishes, the aforementioned method produces an image with all blemishes removed. This method cannot individually control specific blemishes, resulting in limited flexibility, inaccurate images, and poor beautification results.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The present disclosure provides an image beautification processing method, an image beautification processing device, a computer-readable storage medium, and an electronic device, thereby improving the image beautification effect at least to a certain extent.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a method for image beautification processing is provided, comprising: obtaining a facial image to be beautified; performing feature extraction on the facial image to be beautified through a main network of a multi-head machine learning model to obtain a feature image; performing beautification processing on the feature image through a basic branch network of the multi-head machine learning model to obtain a basic image of a blemish-removed face; performing incremental beautification processing on the feature image through a local branch network of the multi-head machine learning model to obtain at least one difference image of a blemish-removed face; and fusing the facial image to be beautified, the basic image of the blemish-removed face, and the at least one difference image of the blemish-removed face to obtain a target beautification image corresponding to the facial image to be beautified.

[0008] According to a second aspect of the present disclosure, an image beautification processing device is provided, comprising: an image acquisition module for acquiring a facial image to be beautified; a main processing module for performing feature extraction on the facial image to be beautified through a main network of a multi-head machine learning model to acquire a feature image; a basic beautification module for performing beautification processing on the feature image through a basic branch network of the multi-head machine learning model to obtain a basic image of a blemish-removed face; an incremental beautification module for performing incremental beautification processing on the feature image through a local branch network of the multi-head machine learning model to obtain at least one difference image of a blemish-removed face; and an image fusion module for fusing the facial image to be beautified, the basic image of the blemish-removed face and the at least one difference image of the blemish-removed face to acquire a target beautification image corresponding to the facial image to be beautified.

[0009] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image beautification processing method described in the first aspect is implemented.

[0010] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image beautification processing method described in the first aspect above by executing the executable instructions.

[0011] The technical solution provided in the embodiments of the present disclosure, on the one hand, uses the basic branch network and multiple local branch networks in the multi-head machine learning model to respectively perform different types of beauty processing such as beauty processing and incremental beauty processing on the beautified face image, and combines the results of different types of beauty processing to obtain the target beauty image, avoiding the limitation of the related art of using only one model for overall beauty processing, and can separate the processing process of different types of defects. Through different types of beauty processing, different types of defects can be controlled separately to obtain a defect-free face difference image with at least one type of defect removed, and the image can be fused according to needs, which can increase the flexibility and pertinence of image beauty processing, improve the beauty effect and image quality, and increase the scope of application and operability. On the other hand, since the different branch networks process different types of images, compared with using only one model for all types of beauty processing, the time consumption and memory usage are reduced, the operation efficiency is improved, and the power consumption is reduced. On the other hand, all types of defects can be processed by the multi-head machine learning model, which improves the comprehensiveness of the beauty processing.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0014] Figure 1 A schematic diagram showing the structure of a system architecture for implementing image beautification processing in this exemplary embodiment is shown;

[0015] Figure 2 A flowchart of an image beautification processing method according to this exemplary embodiment is shown;

[0016] Figure 3 A schematic diagram showing a facial image to be beautified in this exemplary embodiment;

[0017] Figure 4 A schematic diagram showing the structure of a machine learning model in this exemplary embodiment is shown;

[0018] Figure 5 A schematic diagram of the process of obtaining a basic face image with blemishes removed is shown in this exemplary embodiment;

[0019] Figure 6 A schematic diagram showing the process of model training in this exemplary embodiment is shown;

[0020] Figure 7 A schematic diagram illustrating image processing by a multi-head machine learning model in this exemplary embodiment is shown;

[0021] Figure 8 A flowchart of acquiring a target beauty image in this exemplary embodiment is shown;

[0022] Figure 9 A schematic diagram illustrating obtaining a fused image of a basic mode in this exemplary embodiment is shown;

[0023] Figure 10 A schematic diagram illustrating obtaining a fused image in a beauty mode in this exemplary embodiment is shown;

[0024] Figure 11 A schematic diagram illustrating a method of acquiring a high-frequency image in this exemplary embodiment is shown;

[0025] Figure 12 A schematic diagram showing a filling process in this exemplary embodiment is shown;

[0026] Figure 13 A specific flow chart of an image beautification processing method according to this exemplary embodiment is shown;

[0027] Figure 14 A schematic structural diagram of an image beautification processing device according to this exemplary embodiment is shown;

[0028] Figure 15 A schematic structural diagram of an electronic device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0030] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0031] Blemish removal for portraits is part of image enhancement and is typically the first stage of the process. This includes, but is not limited to, removing blemishes and acne, removing eye bags, treating dirty corners of the mouth, smoothing light and shadows, and addressing dry lip lines. Following blemish removal, you can continue with skin smoothing, skin tone adjustment, facial feature transformation, and brightness adjustment.

[0032] Existing portrait blemish removal algorithms used in mobile devices typically employ traditional processing methods, including but not limited to processing based on facial key points, computation of manually designed image features, spatial filtering, and layer fusion, or methods based on deep neural network models. First, due to the complex and diverse lighting conditions encountered in real-world scenarios, as well as the diverse skin blemishes and textures of the subjects, traditional methods are often unable to accurately and completely detect skin blemishes, resulting in incomplete portrait blemish removal and the potential for abnormal effects. Second, existing deep neural network model-based processing methods typically crop the face region image, scale it to a fixed size, and complete the processing in a single step, resulting in an image with all types of blemishes removed. However, they are unable to isolate processing information for specific types of blemishes, meaning they are unable to control the degree of removal of a specific type or types of blemishes, such as achieving 100% skin tone uniformity and 50% eye bag removal. At the same time, during the model learning process, it is difficult to independently adjust the priority and learning difficulty of different types of blemishes, making it difficult to obtain the optimal blemish removal effect. For example, assuming that removing acne is more important than even skin tone, it is difficult to directly control it during the model learning process.

[0033] In order to solve the above technical problems, the exemplary embodiment of the present disclosure first provides an image beautification processing method, the application scenarios of which include but are not limited to: the terminal device is installed with an image beautification App (Application), the user selects an image in the local album in the App for beautification processing, or performs beautification processing on the currently captured image; the terminal device executes the image beautification processing method of this exemplary embodiment, or the terminal device sends the image to the server, and the server executes the image beautification processing method of this exemplary embodiment to beautify the image. Alternatively, the video selected by the user or the currently captured video can also be beautified, specifically, the frames containing portraits in the video can be beautified, for example, in a live broadcast scenario, the real-time video stream can be beautified.

[0034] like Figure 1 In the system architecture diagram shown, terminal 100 can be a smart device with image processing capabilities, such as a smartphone, computer, tablet, smart speaker, smartwatch, in-vehicle device, wearable device, monitoring device, or other smart device. The terminal can include a camera, and the camera type can be of any type as long as it can process images. The number of cameras can be at least one, for example, one, four, or so. The image to be processed can be a captured image, each frame of a captured video, or an image obtained from the terminal or other storage device.

[0035] In the disclosed embodiments, terminal 100 may include an imaging unit 101, a memory 102, a processor 103, and an input / output unit 104. Imaging unit 101 is used for taking photos. Memory 102 is used for storing images, and the processor is used for image processing, such as performing beautification. The processor typically includes one or more of a CPU, GPU, DSP, and NPU. Memory 102 may store a facial image 105 to be beautified. The input / output unit is used for displaying the image.

[0036] Terminal 101 obtains a facial image 105 to be beautified from memory 102 and sends it to processor 103. Processor 103 obtains the facial image to be beautified and processes it using a multi-head machine learning model to perform different types of beautification processing, thereby generating a target beautified image 106 and displaying it via input / output unit 104.

[0037] It should be noted that the image beautification processing method provided in the embodiment of the present disclosure can be executed by the terminal 100. The image beautification processing method can also be set in the terminal 100.

[0038] Next, combine Figure 2 The specific steps of the image beautification processing method are explained. Figure 2 An exemplary process of the image beautification processing method is shown, which may include the following steps S210 to S250, wherein:

[0039] Step S210, obtaining a face image to be beautified;

[0040] Step S220, extracting features from the face image to be beautified using a main network of a multi-head machine learning model to obtain a feature image;

[0041] Step S230, performing beauty processing on the feature image through the basic branch network of the multi-head machine learning model to obtain a basic image of the face without blemishes;

[0042] Step S240, performing incremental beautification processing on the feature image through the local branch network of the multi-head machine learning model to obtain at least one blemish-free face difference image;

[0043] Step S250 , fusing the face image to be beautified, the basic face image after blemish removal, and the at least one difference face image after blemish removal to obtain a target beautified image corresponding to the face image to be beautified.

[0044] In the disclosed embodiment, image beautification processing may be universal beautification processing, which refers to a relatively fixed and unified link in the entire beautification processing process, such as blemish removal, which is necessary for most users and does not require personalized settings.

[0045] The face image to be beautified refers to an image that needs to be beautified, for example, a face image in an image, etc. Since a face image may contain various types of defects, the target beautification parameters may be different in different scenarios.

[0046] The multi-head machine learning model can perform blemish removal processing on a beautified facial image to obtain a base image of the blemish-free face, and can also perform incremental beautification processing on the beautified facial image to obtain at least one difference image of the blemish-free face. The multi-head machine learning model can include a main network and multiple branch networks, each of which can include a base branch network and at least one local branch network. By performing different types of beautification processing using the base branch network and local branch networks in the multi-head machine learning model, it is possible to achieve separate control of each type or types of blemishes, thereby improving accuracy and flexibility.

[0047] Furthermore, the face image to be beautified, the base face image after blemish removal, and the difference image after blemish removal can be incrementally fused to obtain an intermediate beautified image, and the intermediate beautified image can be inserted into the corresponding position of the face image to be beautified to obtain a target beautified image corresponding to the face image to be beautified, that is, the image of the face image to be beautified after removing various types of blemishes. Specifically, the target beautification parameters can be determined according to actual needs, and the face image to be beautified, the base face image after blemish removal, and the difference image after blemish removal can be partially or fully fused according to the target beautification parameters to obtain the corresponding target beautified image, thereby improving flexibility and authenticity.

[0048] Next, Figure 2 Each step is described in detail.

[0049] refer to Figure 2 As shown in , in step S210 , a face image to be beautified is obtained.

[0050] In the disclosed embodiments, the facial image to be beautified may be one or more facial images requiring beautification, and may be obtained from the image to be processed. The image to be processed may be a user-selected image, such as an image selected from an album or other file, or an image obtained from the network, another application, or another storage location, or an image automatically assigned by the system, such as a currently captured image. The facial image to be beautified may be an image containing a facial image, or may be a portion of the image to be processed, such as a portion containing a face. Specifically, the image to be processed may be obtained by preprocessing the image to be processed.

[0051] After obtaining the image to be processed, the image can be preprocessed to obtain a facial image to be beautified. The facial image to be beautified can be a rectangular facial image extracted from the image to be processed. The preprocessing can be extraction or other operations. The number of facial images to be beautified can be one or more, specifically determined by the number of facial images contained in the image to be processed. The number of facial images to be beautified can be less than or equal to the number of facial images contained in the image to be processed, specifically determined by the size of each facial image and the upper limit of the number.

[0052] In some embodiments, the specific implementation of step S210 may be: extracting a facial image to be beautified from the image to be processed. Specifically, the steps include: generating one or more facial frames in the image to be processed based on the facial key points identified in the image to be processed; retaining facial frames whose areas are greater than or equal to a facial area threshold, and capturing the image within the facial frames to obtain the facial image to be beautified. Facial key points may include key facial parts and points on the edges of the face. The facial frame may be a rectangular frame, with the facial key points of each face located within the frame. In one embodiment, the facial frame may be the smallest rectangular frame that includes the facial key points.

[0053] Generally, all faces can be detected in the image to be processed by the face detection algorithm, but some of them may include faces that do not need beautification (such as the faces of passers-by in the distance). Since in the image beautification scenario, it is usually necessary to beautify larger faces (the beautification effect is not obvious on smaller faces), the face frame can be filtered by the face area threshold to avoid misoperation caused by faces with smaller areas. Specifically, the face area threshold can be the size of the image to be processed * 0.05, which can be determined according to the size of the image to be processed; if the area of ​​the face frame is greater than or equal to the face area threshold, it is determined as a face that needs beautification and the face frame is retained; if the area of ​​the face frame is less than the face area threshold, the face frame is deleted.

[0054] After filtering the face frames, the remaining face frames are the valid face frames. The image within each face frame is captured to obtain the same number of face images to be beautified as the number of face frames.

[0055] In one embodiment, an upper limit on the number of facial images to be beautified can be set, which can be achieved by setting an upper limit on the number of face frames. The upper limit can be set to 4, for example. If the number of retained face frames is greater than 4 after filtering by the above-mentioned face area threshold, 4 face frames can be selected from them. The 4 face frames can be, for example, the 4 face frames ranked in descending order of area, or the 4 face frames closest to the center of the image to be processed, or the 4 face frames selected based on actual needs, corresponding to 4 original face images being intercepted, and faces in other face frames are not beautified; or multiple beautification processes can be performed, in which 4 face frames are selected and the corresponding face images are intercepted for beautification in the current process, and other face frames are selected and other face images are intercepted for beautification in the next process, thereby completing the beautification of faces in all face frames in the image to be processed whose area is greater than the face area threshold.

[0056] In some embodiments, before capturing the image within the face frame, the face frame may be expanded to include a small amount of area outside the face, facilitating gradient processing during subsequent image fusion. During the expansion process, the face frame may be expanded in one or more directions according to a preset ratio. For example, if the preset ratio is 1.1, the face frame is uniformly expanded in all directions, resulting in the expanded face frame size being 1.1 times its original size. It should be noted that during the expansion process, if one or more boundaries of the face frame reach the boundaries of the image to be processed, the boundaries of the face frame are caused to remain at the boundaries of the image to be processed, i.e., the boundaries of the image to be processed serve as the boundaries of the face frame.

[0057] Continue to refer Figure 2 As shown, in step S220, the main network of the multi-head machine learning model is used to extract features of the face image to be beautified to obtain a feature image.

[0058] In the disclosed embodiments, to avoid the problem of only being able to perform overall beautification using a single model in related technologies, a multi-head machine learning model can be used to perform different types of beautification processing on beautified facial images to achieve classified beautification, resulting in different types of beautified images. The different types of beautification processing can include basic beautification and incremental beautification. Incremental beautification can be local beautification using incremental logic. Specifically, different types of beautified images can include a base image with blemishes removed and a difference image with blemishes removed.

[0059] In some embodiments, because different machine learning models require different input image sizes, before inputting the facial image to be beautified into the multi-head machine learning model, the facial image to be beautified may be processed to match the input image size of the multi-head machine learning model to adjust the facial image to be beautified. The adjusted facial image to be beautified may be an image that matches the input image size of the multi-head machine learning model.

[0060] In the process of adjusting the facial image to be beautified, the facial image to be beautified can be adjusted. The adjustment operation can be a size change, and the size change can include but is not limited to one or more of rotation, scaling, and padding. In the embodiment of the present disclosure, each facial image to be beautified can be resized separately to obtain at least one facial image to be beautified. Alternatively, multiple facial images to be beautified can be resized and then merged to obtain an image of a fixed input size that matches the input image size of the multi-head machine learning model. By merging multiple faces, it can be ensured that the power consumption of multiple face processing is basically the same as that of single face processing, so it can be applied to platforms with limited computing power.

[0061] In some embodiments, it may first be determined whether the size parameters of the facial image to be beautified meet a size condition. The size condition may be that the height and width of the facial image to be beautified match the height and width of the input image size of the multi-head machine learning model. If they do not match, the facial image to be beautified may be resized based on the input image size of the multi-head machine learning model to obtain the facial image to be beautified.

[0062] In some embodiments, resizing the beautified facial image may include any one or more of the following methods:

[0063] (1) When the size relationship between the width and height of the face image to be beautified (the long and short sides) is different from the size relationship between the width and height of the input image size of the multi-head machine learning model, the face image to be beautified is rotated 90 degrees. Specifically, if the width of the face image to be beautified and the input image size of the multi-head machine learning model are both greater than the height or less than the height, then the size relationship between the width and height of the face image to be beautified and the input image size of the multi-head machine learning model is the same, and there is no need to rotate the face image to be beautified; otherwise, the size relationship between the width and height of the face image to be beautified and the input image size of the machine learning model is different, and the face image to be beautified needs to be rotated 90 degrees (either clockwise or counterclockwise) to keep the size relationship between the two consistent. For example, when the input image size of the multi-head machine learning model is 704 in width and 448 in height, that is, the width is greater than the height, if the width of the face image to be beautified is less than the height, then the face image to be beautified is rotated 90 degrees. In addition, in order to maintain the angle of the face in the face image to be beautified, the face image to be beautified may not be rotated.

[0064] (2) When the size of the face image to be beautified is larger than the size of the input image, the face image to be beautified is scaled according to the input image size of the multi-head machine learning model. The scaling here can be downsampling. The size of the face image to be beautified is larger than the size of the input image, which means that the width of the face image to be beautified is larger than the width of the input image size, or the height of the face image to be beautified is larger than the height of the input image size. In the image beautification scenario, the face image to be beautified is generally provided by a clear image taken by a terminal device, and its size is large. Therefore, it is a common situation that the size of the face image to be beautified is larger than the size of the input image, that is, it is usually necessary to downsample the face image to be beautified. Downsampling can be used to represent size reduction processing. Downsampling can be achieved by methods such as bilinear interpolation and nearest neighbor interpolation, which is not limited in this disclosure.

[0065] After downsampling, the facial image to be beautified is aligned with at least one of the width and height of the input image size of the machine learning model, specifically including the following situations: the width and height of the two are the same, the width of the facial image to be beautified is the same as the width of the input image size, and the height is smaller than the height of the input image size; the height of the facial image to be beautified is the same as the height of the input image size, and the width is smaller than the width of the input image size.

[0066] It should be noted that if the above-mentioned rotation is performed on the face image to be beautified to obtain a rotated face image to be beautified, then when the size of the rotated face image to be beautified is larger than the size of the input image, it is downsampled according to the size of the input image. The specific implementation method is the same as the downsampling method of the face image to be beautified, so it will not be repeated. On the contrary, when the size of the face image to be beautified (or the rotated face image to be beautified) is less than or equal to the size of the input image, the downsampling processing step may not be performed. After the downsampling is performed, the aspect ratio of the face image to be beautified after the scaling process is required to be consistent with the aspect ratio of the face image to be beautified before the size scaling process, and the width and height after the scaling process are both less than or equal to the width and height of the corresponding input image size, and the width and height of the face image to be beautified after the scaling process should be as close as possible to the width and height of the corresponding input image size.

[0067] (3) When the size of the face image to be beautified is smaller than the size of the input image, the face image to be beautified is padded according to the size difference between the face image to be beautified and the input image, so that the size of the padded face image to be beautified is equal to the size of the input image. The size of the face image to be beautified is smaller than the size of the input image, which means that at least one of the width and height of the face image to be beautified is smaller than the size of the input image, and the other is not larger than the size of the input image. Specifically, it includes the following situations: the width of the face image to be beautified is smaller than the width of the input image, and the height is also smaller than the height of the input image; the width of the face image to be beautified is smaller than the width of the input image, and the height is equal to the height of the input image; the height of the face image to be beautified is smaller than the height of the input image, and the width is equal to the height of the input image.

[0068] When filling, you can fill in the four sides of the face image to be beautified, for example, make the center of the face image to be beautified coincide with the center of the input image size, and fill in the difference part around the face image to be beautified, so that the size of the face image to be beautified after filling is consistent with the input image size. Of course, you can also align the edge of one side of the face image to be beautified with the input image size, and fill in the other side, which is not limited in this disclosure. Preset pixel values ​​can be used for filling, usually pixel values ​​that are significantly different from the face color, such as (R0, G0, B0), (R255, G255, B255), etc. The unfilled part can be assigned a value of 0 or no additional value is required, as long as the size of the face image to be beautified after filling is consistent with the input image size.

[0069] In the above transformation, the orientation and size of the face image to be beautified are changed to facilitate unified processing by the machine learning model. Subsequently, the beautified face image will need to be inversely transformed to restore it to the same orientation and size as the face image to be beautified. Therefore, the corresponding size transformation information can be saved, including but not limited to: the rotation direction and angle of each face image to be beautified, the downsampling ratio (scaling ratio), the coordinates of the padded pixels, and the padded pixel values.

[0070] In the embodiment of the present disclosure, by adjusting the input image size of the multi-head machine learning model, the face image to be beautified is rotated, scaled, and filled in one or more ways, and the face image to be beautified can be converted into a face image to be beautified that can be directly applied to the multi-head machine learning model. The face image to be beautified can be Figure 3 As shown in .

[0071] In the embodiment of the present disclosure, in order to achieve different types of beautification processing, the facial image to be processed can be processed by multiple branch networks in a multi-head machine learning model to obtain a blemish-free facial basic image and at least one blemish-free facial difference image. The multi-head machine learning model can be a DNN (Deep Neural Network, DNN), a deep convolutional neural network. The deep convolutional neural network can be a fully convolutional network, which is a neural network with a large number of layers. By increasing the number of network layers (i.e., network depth) to reduce the number of parameters, it can learn the deep features of the image and achieve pixel-level processing.

[0072] The multi-head machine learning model can process images of different sizes. In this case, the machine learning model has no requirements for the input image size, and the size has an impact on the amount of calculation, memory usage, and beautification precision. The input image size can be determined according to the beautification precision set by the user or the performance of the terminal device. As a result, the multi-head machine learning model can be deployed on devices with different performance, such as high, medium, and low, and has a wide range of applications. There is no need to deploy different machine learning models for different devices, which reduces the training cost of the network. For example, when lightweight computing is suitable on a mobile terminal, the input image size can be determined as a smaller value, such as 640 width * 448 height.

[0073] In one embodiment, a lightweight machine learning model can be used to reduce the amount of computation and realize the learning and processing of image depth features. Exemplarily, the multi-head machine learning model can adopt an end-to-end structure to realize pixel-level processing of beautified facial images. The multi-head machine learning model can include a main network and multiple branch networks. Therefore, the multi-head machine learning model can be composed of a lightweight main network and several lightweight branch networks (i.e., head networks) to reduce the amount of computation.

[0074] The main network of a multi-head machine learning model is the shared component, and the number of network layers included in this shared component varies in different scenarios. This means the main network can vary depending on the scenario. The branch network refers to the non-shared component, and each branch network can be used to enhance a specific type of blemish or several types of blemishes. For example, a branch network might only contain the last layer, with all other layers considered the main network; or a branch network might contain the last four layers, with all other layers considered the main network. The number of network layers in the main network is determined based on actual needs. For example, this could be the difference between the processing objects corresponding to different types of enhancement, such as blemishes. For example, if the differences between the blemish types are large, the main network should contain fewer network layers; if the differences between the blemish types are small, the main network should contain more network layers.

[0075] In some embodiments, multiple branch networks can be further divided into base branch networks and local branch networks based on reference information, so that the multiple branch networks are configured to output a base blemish-removed face image and several blemish-removed face difference images containing one or more other blemish removal categories. The number of local branch networks can be at least one. Reference information may include, but is not limited to, combinations of blemish removal functions under different application modes, user control requirements for different blemish removals, and the desired effect and model learning difficulty for specific blemish removal types. Based on this reference information, the outputs of multiple branch networks can be flexibly used during the model application phase.

[0076] In a multi-head machine learning model, the main network can be a multi-layer network starting from the input end of the multi-head machine learning model. There is no specific restriction on the specific number of convolutional layers and the number of channels in each layer of the main network. Figure 4 As shown in the network structure diagram, the main network 410 may include a first pixel rearrangement layer 411, at least one convolution layer 412, and at least one transposed convolution layer 413. Based on the above network structure, the specific implementation method of processing the beautified face image through step S220 can be referred to. Figure 5 As shown in , it mainly includes steps S510 to S530, wherein:

[0077] In step S510, the first pixel rearrangement layer is used to perform a first pixel rearrangement process on the facial image to be beautified to obtain a first feature image.

[0078] The facial image to be beautified can be a single-channel image (e.g., a grayscale image) or a multi-channel image (e.g., an RGB image). The first pixel rearrangement layer can be a preprocessing layer, which reduces the height and width of the input image to 0.5 times the original size and quadruples the number of channels by performing pixel value rearrangement processing, rearranging four adjacent neighborhood pixels to the same position in different channels. Based on this, it can be considered that the first pixel rearrangement layer rearranges each channel of the facial image to be beautified into multiple channels.

[0079] In an optional embodiment, step S510 includes the following steps: inputting the facial image to be beautified with a channel number a into the first pixel rearrangement layer; rearranging the pixel points in each n*n neighborhood in each channel of the facial image to be beautified to the same position in the n*n channels, and outputting a first feature image with a channel number a*n*n.

[0080] Where a is a positive integer representing the number of channels in the facial image to be beautified; n represents the pixel permutation parameter, a positive integer not less than 2. Taking n = 2 as an example, the first channel of the facial image to be beautified is traversed, typically starting from the top left corner. Pixels in each 2x2 grid are extracted and permuted to the same position in each of the four channels. This transforms one channel into four channels, while simultaneously reducing the image's width and height to half. This permuted image is recorded as the first feature image. The same process is applied to the remaining channels. If the facial image to be beautified is a single-channel image, the pixel permutation yields a four-channel first feature image. If the facial image to be beautified is a three-channel image, the pixel permutation yields a twelve-channel first feature image. The first pixel permutation layer can be implemented using the space_to_depth function in TensorFlow (a machine learning framework) or a convolution operation with a stride of n. In this case, the first pixel permutation layer can be considered a special convolutional layer. The first pixel rearrangement layer can rearrange multiple pixels in adjacent domains to the same position in different channels, increasing the number of channels. It can convert the spatial features of the facial image to be beautified into deep features, thereby improving the accuracy and comprehensiveness of the features.

[0081] Step S520: Perform convolution processing on the first feature image using a convolution layer to obtain a second feature image.

[0082] This disclosure does not limit the number of convolutional layers, convolution kernel size, or specific structure of convolutional layers. Convolutional layers are used to extract image features at different scales and learn depth information. Convolutional layers can include supporting pooling layers to downsample the convolved image to achieve information abstraction, increase the receptive field, and reduce parameter complexity.

[0083] When multiple convolutional layers are set, a step-by-step convolution and downsampling method can be used. For example, the height and width of the feature layer can be reduced by a factor of 2 through the convolutional layer and the stride or the pooling operation Pool. The image can be reduced by a factor of 2 until the last convolutional layer outputs the second feature image. The second feature image can be the feature image with the smallest size in the processing of the multi-head machine learning model.

[0084] Step S530: Perform transposed convolution processing on the second feature image using a transposed convolution layer to obtain the feature image.

[0085] The present disclosure does not limit the number of transposed convolution layers, the size of the transposed convolution kernel, the specific structure of the transposed convolution layer, etc. The transposed convolution layer is used to upsample the second feature image and can be regarded as the reverse process of convolution, thereby restoring the size of the image.

[0086] When setting multiple transposed convolutional layers, a step-by-step upsampling method can be used. For example, the height and width of the image can be increased by a factor of 2 until the last transposed convolutional layer outputs the feature image, so that the height, width and number of channels of the input and output layers are consistent.

[0087] In an optional embodiment, the convolution layer and the transposed convolution layer are completely symmetrical structures, and the feature image and the first feature image have the same size and number of channels.

[0088] In addition, a direct connection can be established between the convolution layer and the transposed convolution layer through a direct connection operation, such as Figure 4 The structure is shown in . Direct connections are established between convolutional layers and transposed convolutional layers corresponding to feature images of the same size. That is, a direct connection operation occurs between several feature layers of the same height and width during both the height and width reduction and increase processes. This allows the feature image information in the convolution phase to be directly connected to the feature image in the transposed convolution phase, facilitating the generation of a more comprehensive feature image. The direct connection operation can be a concatenation operation, an addition operation, or other operations, and is not specifically limited here.

[0089] In the disclosed embodiments, the first pixel rearrangement layer, convolution layer, and transposed convolution layer included in the main network of the multi-head machine learning model are used to extract features from the beautified facial image, thereby improving the accuracy of the feature image. Furthermore, obtaining the feature image through the main network can improve the consistency of the feature image.

[0090] Next, continue to refer to Figure 2 As shown in , in step S230, the feature image is beautified by the basic branch network of the multi-head machine learning model to obtain a blemish-free basic face image.

[0091] In the disclosed embodiment, the blemish-removed basic face image refers to the image obtained by performing basic beautification on the face image to be beautified, and there will only be one result. Basic beautification can include but is not limited to: smoothing light and shadow, and removing dirty or old corners of the mouth.

[0092] The branch network can be a layer or layer network starting from the output end of the multi-head machine learning model. The branch network can be used to implement different types of beautification processing, which can correspond to different types of beautification or blemish removal. The branch network can specifically correspond to a basic branch network and at least one local branch network according to the reference information. In the embodiment of the present disclosure, there is no specific restriction on the specific number of convolutional layers of the main network and the number of channels of each layer. There is no specific restriction on the number of branch networks and the network structure of the branch network. It is only required that each branch network contains a post-processing layer and the output size is consistent with the fixed input size. In addition, if there is a 2-fold increase in the height and width before the post-processing layer, a direct connection can be established between it and several feature layers of the same height and width corresponding to the descending process in the main network.

[0093] Among them, the basic branch network is used to perform basic beautification on the beautified face image, such as smoothing the light and shadow, processing the dirty and old corners of the mouth, etc., so as to obtain a basic face image without blemishes. The basic branch network at least includes a second pixel rearrangement layer, which can be Figure 4 The basic head post-processing layer 421 in the basic branch network 420 is shown in FIG.

[0094] The second pixel rearrangement layer has the opposite process to the first pixel rearrangement layer. The second pixel rearrangement layer can upsample the feature image corresponding to the face image to be beautified in the opposite way to the first pixel rearrangement layer, and rearrange the multiple channels of the feature image into one channel. In an optional embodiment, the following steps may be included: inputting a feature image with a channel number of b*n*n into the second pixel rearrangement layer; upsampling the feature image by 2 times, and rearranging the pixels at the same position in each n*n channel of the feature image to the n*n neighborhood in a single channel, and outputting a basic image of the face with a channel number of b. Wherein, b is a positive integer. In addition, the size of the basic image of the face with a blemish removed output by the second pixel rearrangement layer is consistent with the fixed input size, that is, consistent with the size of the face image to be beautified input into the multi-head machine learning model.

[0095] The second pixel rearrangement layer can be implemented using the depth_to_space function in TensorFlow to convert the depth features in the feature image into spatial features. It can also be implemented using a transposed convolution operation with a step size of n. In this case, the second pixel rearrangement layer can be regarded as a special transposed convolution layer.

[0096] If the convolutional layer and the transposed convolutional layer are completely symmetrical, meaning the feature image and the first feature image have the same size and number of channels, then a = b. Furthermore, the number of channels in the base image of the blemish-removed face and the image to be enhanced is also the same, meaning the multi-head machine learning model's processing does not change the image size or number of channels. It's important to note that the multi-head machine learning model's processing also does not change the number of faces.

[0097] In addition, the basic branch network may also include a transposed convolution layer and a convolution layer. The transposed convolution layer is located before the second pixel rearrangement layer, and the convolution layer is located between the transposed convolution layer and the convolution layer. There is no specific limit on the number of transposed convolution layers and convolution layers. Based on this, when processing the beautified face image through the basic branch network to obtain the blemish-free face base image, the following steps may also be included:

[0098] Upsampling the feature image using the transposed convolution layer in the basic branch network to obtain an intermediate feature image;

[0099] The intermediate feature image is convolved by the convolution layer in the basic branch network.

[0100] Among them, the transposed convolution layer can be, for example, Figure 4 The base head convolution layer 422 in the base branch network 420 shown in FIGURE 4 increases the height and width by a factor of 2 to obtain an intermediate feature image, and directly connects it to the corresponding feature layer in the descending process in the main network. Several convolution layers may also be included between the transposed convolution layer and the post-processing layer, such as convolution layer 423 in the base branch network. This convolution layer is used to perform convolution processing on the intermediate feature image to obtain an intermediate result again without changing the image size and resolution.

[0101] It should be noted that other network layers may be included before the post-processing layer, and there is no specific limitation on the number of layers and channels of other network layers, which are determined according to actual needs.

[0102] Continue to refer Figure 2 As shown in , in step S240, the feature image is incrementally beautified by the local branch network of the multi-head machine learning model to obtain at least one blemish-free face difference image.

[0103] In the embodiment of the present disclosure, the number of local branch networks in the multi-head machine learning model is at least one. The blemish-removed face difference image refers to an image obtained by performing incremental beautification on the beautified face image, and specifically can be a face difference image with at least one type of blemish removed obtained by incremental beautification processing. Incremental beautification processing can be different types of local beautification processing, which refers to local removal of different types of blemishes to remove one or more types of blemishes. The type of blemish-removed face difference image is determined according to actual needs, and the types of blemish-removed face difference images corresponding to different scenes may be the same or different. In some embodiments, incremental beautification processing can include different types of local beautification, and there is incremental logic between different types of local beautification. Incremental beautification processing can, for example, include but is not limited to freckle and acne removal processing, texture removal (eye bag removal, nasolabial fold removal processing), and skin color processing (such as large-scale skin color uniformity processing). Among them, the incremental beautification processing is arranged in order of priority from high to low as follows: basic beauty, freckle and acne removal, texture removal, and large-scale skin color processing. In addition, the incremental beautification process may also include other blemish removal processes, which are determined based on actual needs, such as fat and thin processing, etc. The blemish-removed face difference image may be a face difference image with at least one type of blemish removed, such as a face difference image corresponding to blemish removal or skin color processing.

[0104] Each local branch network is used to perform local beautification on a specific type of blemish in the facial image being beautified, such as removing blemishes, acne, eye bags, nasolabial folds, even skin tone over a large area, and creating three-dimensional highlights. This allows for different types of incremental beautification processing to be performed through the local branch network, resulting in at least one blemish-free face difference image, i.e., the blemish-free face difference image corresponding to each local branch network. There is a one-to-one correspondence between the blemish-free face difference image and the local branch network.

[0105] For example, the local branch network may include but is not limited to a local branch network for freckle and acne removal; a local branch network for texture processing (eye bag removal, nasolabial fold removal); a local branch network for skin color processing (such as large-scale skin color processing); in addition, the local branch network can also add corresponding semantic information images, including but not limited to skin segmentation mask images, etc.

[0106] Since different branch networks are used to perform different types of beautification processing, the network types contained in different branch networks can be the same, but the network parameters corresponding to each branch network are different. Based on this, multiple local branch networks can include at least a third pixel rearrangement layer, and the parameters of the third pixel rearrangement layer of each local branch network are different, and the parameters of the third pixel rearrangement layer of each local branch network are different from the second pixel rearrangement layer of the basic branch network. Based on the third pixel rearrangement layer, different types of local beautification can be performed on the feature image through each local branch network, that is, one local branch network is used to perform one type of incremental beautification processing. Since the processing objects of each local branch network are different, different types of incremental beautification processing are used to represent beauty processing of different ranges and different blemish removal on the feature image through each local branch network.

[0107] refer to Figure 4 As shown in , the third pixel rearrangement layer can be the nth head post-processing layer 431 in the local branch network 430. The process of the third pixel rearrangement layer is opposite to that of the first pixel rearrangement layer. The third pixel rearrangement layer can upsample the feature image corresponding to the facial image to be beautified in the opposite manner to that of the first pixel rearrangement layer, and rearrange multiple channels of the feature image into one channel. In an optional embodiment, the following steps may be included: inputting a feature image with a channel number of b*n*n into the third pixel rearrangement layer; upsampling the feature image by a factor of 2, and rearranging the pixels at the same position in each n*n channel of the feature image to an n*n neighborhood in a single channel, and outputting a blemish-free face difference image with a channel number of b. Wherein, b is a positive integer.

[0108] In addition, the local branch network may further include a transposed convolution layer and a convolution layer. The transposed convolution layer is located before the third pixel rearrangement layer, and the convolution layer is located between the transposed convolution layer and the convolution layer. There is no specific limit on the number of transposed convolution layers and convolution layers. Based on this, when performing incremental beauty processing through each local branch network to obtain a blemish-free face difference image, the following steps may also be included:

[0109] Upsampling the feature image using the transposed convolutional layer in each of the local branch networks to obtain an intermediate local feature image;

[0110] The intermediate local feature image is convolved by the convolution layer in each of the local branch networks.

[0111] Among them, the transposed convolution layer can be, for example, Figure 4The nth head convolution layer 432 in the local branch network 430 shown in FIG. 4 may further include several convolution layers between the transposed convolution layer and the post-processing layer, such as the convolution layer 433 in the local branch network, so that the height and width dimensions are increased by a factor of 2 to obtain an intermediate local feature image, and a direct connection is performed with the corresponding feature layer in the descending process of the main network; this convolution layer is used to perform convolution processing on the intermediate local feature image to obtain an intermediate result again without changing the image size and resolution. It should be noted that the intermediate local feature images corresponding to different local branch networks can be different.

[0112] In some embodiments, a multi-head machine learning model including a main network and four branch networks, i.e., four head networks, is used as an example for explanation. Among them, the first head network is a basic blemish removal head network, i.e., a basic branch network, which outputs a blemish-removed face basic image out_img including smooth light and shadow, dirty and old mouth corner processing. base The second head network outputs the blemish-free face difference image (out_diff_img1) after blemish and acne removal. The third head network outputs the blemish-free face difference image (out_diff_img2) after eye bags and nasolabial folds are removed. The fourth head network outputs the blemish-free face difference image (out_diff_img3) after a large-scale skin tone uniformity treatment.

[0113] refer to Figure 4 The structural diagram of the multi-head machine learning model shown in , includes a main network 410 and a branch network 400, and each branch network corresponds to a type of blemish removal processing. The branch network 400 may include a basic branch network 420, multiple local branch networks 430, such as the second head network, the third head network, and the fourth head network. The basic branch network is used for basic beautification, and the local branch network is used for separate local beautification of a specific type of blemish. Based on this, the number of blemish-removed face difference images can be the same as the number of local branch networks in the multi-head machine learning model.

[0114] In the disclosed embodiments, multiple branch networks within a multi-head machine learning model can be used to perform beautification processing on facial images with different types of blemishes. This avoids the limitation of unified control and enables separate control of each type of blemish through multiple branch networks, increasing flexibility and broadening the scope of application. Furthermore, the ability to use multiple branch networks to perform different types of beautification processing improves the accuracy of the processing.

[0115] In this exemplary embodiment, in order to improve network accuracy, a multi-head machine learning model can be trained to update model parameters. Specifically, sample images of each branch network can be obtained, and loss functions and weight parameters can be independently calculated for each branch network to train the multi-head machine learning model.

[0116] Figure 6 The flowchart for model training is shown schematically. Figure 6 As shown in , the training process can include the following steps:

[0117] In step S610, a first loss function of the basic branch network and a second loss function of the local branch network are obtained, and an overall loss function is determined according to the first loss function and the second loss function;

[0118] In step S620, according to the overall loss function, the network parameters of the main network, the basic branch network and the local branch network are adjusted for joint training until the overall loss function meets the convergence condition to obtain the multi-head machine learning model.

[0119] In the embodiment of the present disclosure, for each branch network, its loss function can be the same or different, which is specifically determined according to the difference between the labeled image and the sample beauty image of the sample image corresponding to each branch network.

[0120] Exemplarily, a plurality of unbeautified images can be obtained as sample images for each branch network; the sample images are images of different people to be beautified collected in complex and diverse real-world scenes. Annotated images are obtained by manually removing blemishes from the sample images. Annotated images are images obtained by manually or automatically annotating the image using image editing tools to perform processing on the beautified images, including but not limited to light and shadow smoothing, dirty and old corner processing of the mouth, freckle and acne removal, eye bag removal, nasolabial fold removal, large-scale skin tone uniformity, and high-light stereoscopic processing. The operations on each type of blemish above are all performed using independent layers, and the corresponding difference image (diff_img) for each blemish treatment can be obtained, which is then provided to the model for learning. The sample images and the annotated images are used to construct a dataset to train a multi-head machine learning model, and the resulting multi-head machine learning model can be used to perform multiple types of beautification processing. If, when obtaining the above-mentioned sample images, it is replaced by obtaining sample beautification images after manual blemish removal and skin smoothing of the sample images, and a dataset is constructed and a multi-head machine learning model is trained, the resulting multi-head machine learning model can be used to perform blemish removal and skin smoothing simultaneously. Therefore, based on actual application requirements, sample images that have undergone specific beauty processing can be obtained and a dataset can be constructed to train a multi-head machine learning model capable of implementing specific beauty processing functions. Furthermore, the multi-head machine learning model can integrate a variety of different beauty processing functions, making it more convenient to implement the solution and achieving higher processing efficiency compared to setting up multiple algorithm modules. The disclosed embodiments are explained using the example of only performing blemish removal.

[0121] For different branch networks, their sample images are also different, which can be specifically determined by the type of beautification processing that the branch network can perform, that is, the type of blemishes that can be processed. In some embodiments, a first sample image corresponding to the base branch network can be first obtained and input into the base branch network to be trained to obtain a first sample beautification image corresponding to the first sample image of the base branch network, that is, the first sample beautification image obtained by automatically processing the first sample image by the base branch network. Furthermore, the difference between the first sample beautification image and its corresponding first annotated image can be compared, and the first loss function of the base branch network can be determined based on this difference. Similarly, a second sample image corresponding to each local branch network can be obtained, and a second annotated image and a second sample beautification image corresponding to the second sample image can be obtained, and the second loss function can be determined based on the difference between the second sample beautification image and the second annotated image. It should be noted that the second sample image of each local branch network can be the same, but the second annotated image is different. The second annotated image is specifically obtained based on the blemish removal processing performed by the local branch network, that is, corresponding to the beautification processing performed by the local branch network. Based on this, the corresponding loss function can be independently calculated for each branch network.

[0122] By combining the difference images of various defects, the labeled images corresponding to each branch network can be obtained as shown in formula (1):

[0123]

[0124] Among them, the first labeled image of the basic branch network is gt_img base , the second labeled image of the first local branch network is gt_diff_img1, the second labeled image of the second local branch network is gt_diff_img2, and the second labeled image of the third local branch network is gt_diff_img3.

[0125] It should be noted that each branch network can select a corresponding loss function based on the characteristics of the defect-free face difference image it outputs. The first loss function and the second loss function may include but are not limited to any one of the loss functions such as L1, L2, SSIM (Structural Similarity) and any combination. Among them, the L1 norm is the minimum absolute deviation, which minimizes the sum of the absolute differences between the target value and the estimated value. The L2 norm refers to the minimum square error, which minimizes the sum of the squares of the differences between the target value and the estimated value. The SSIM loss function is used to measure the gap between two images.

[0126] When the first loss function or the second loss function can include a combination of multiple types of loss functions, the multiple loss functions can be combined according to the weight parameters corresponding to each loss function, and the loss function of the nth branch network (i.e., the first loss function or the second loss function) can be as shown in formula (2), wherein a1 represents the weight of the L1 norm and a2 represents the weight of the SSIM.

[0127]

[0128] After obtaining the loss function of each branch network, the overall loss function can be calculated based on the loss function of each branch network, that is, the overall loss function can be determined based on the first loss function of the basic branch network and the second loss function of the local branch network. Specifically, the weight parameter of the first loss function of the basic branch network can be determined, and the second loss function and weight parameter of the local branch network can be determined. Specifically, the weight parameters of the basic branch network and the local branch network can be determined based on information such as the priority of the corresponding branch network and the training difficulty. Here, the weight parameter is determined according to the priority as an example, and the weight parameter is positively correlated with the priority. That is, the higher the priority, the larger the weight parameter. In the embodiment of the present disclosure, there is no special limitation on the priority between branch networks, and it is determined according to actual needs. For example, if the priority of the basic branch network is greater than the priority of the first local branch network, the weight parameter of the first loss function is greater than the weight parameter of the second loss function of the first local branch network. Further, the first loss function and the second loss function can be combined according to the weight parameter to determine the overall loss function. Specifically, the first loss function and the second loss function can be weighted according to the corresponding weight parameters, as shown in reference formula (3):

[0129] loss sum =λ base *loss base +λ1*loss2+…+λ n *loss n Formula (3)

[0130] After determining the overall loss function, the main network and at least one branch network can be jointly trained according to the overall loss function until the overall loss function meets the convergence condition, thereby obtaining a multi-head machine learning model. Exemplarily, the overall loss function can be used as a benchmark to synchronously update the network parameters of the main network, the basic branch network, and the local branch network. That is, the four branch networks are jointly trained together, each branch network uses the corresponding sample image and loss function, and during the joint training, the basic branch network and all branch networks formed by at least one local branch network are comprehensively considered for unified iteration to update the network parameters according to the overall loss function. When the accuracy of the multi-head machine learning model on the validation set reaches the set standard (such as a preset threshold), it can be considered that the convergence condition is met, and the model training is determined to be complete, thereby obtaining a multi-head machine learning model.

[0131] It should be noted that after being fully trained with a comprehensive data set, the multi-head machine learning model can cope with almost all situations in practical applications, including different lighting conditions, different skin conditions, different types of blemishes, etc., to accurately and fully detect and remove portrait blemishes, and solve the problems of incomplete blemish removal and certain limitations in related technologies.

[0132] The model training method in the embodiment of the present disclosure performs model training by setting a corresponding loss function for each branch network and combining the overall loss function determined by the loss functions of all branch networks. It can independently calculate the loss function and corresponding weight parameters of each branch network, and perform separate parameter adjustment control on the basic branch network and the local branch network. Compared with a single model, it can achieve better results, improve the accuracy of model training, obtain more accurate main networks, basic branch networks and local branch networks, and improve the accuracy of multi-head machine learning models.

[0133] Figure 7 : A schematic diagram of image processing by a multi-head machine learning model is shown in the figure. The face image to be beautified 701 can be input into the multi-head machine learning model 702, and the blemish-removed face basic image 705 is obtained through the main network 703 and the basic branch network 704. The blemish-removed face difference image 707 is obtained through the main network 703 and the local branch network 706. The blemish-removed face difference image 709 is obtained through the main network 703 and the local branch network 708.

[0134] In the disclosed embodiment, the basic branch network in the multi-head machine learning model and multiple branch networks composed of multiple local branch networks can be used to perform beautification processing on different types of defects of the beautified facial image respectively, so that each type of defect can be independently controlled by each branch network, avoiding the limitation of related technologies that overall beautification control can only be performed by one model, and improving the flexibility and accuracy of beauty control.

[0135] Next, continue to refer to Figure 2 As shown in , in step S250, the face image to be beautified, the blemish-removed face basic image and the at least one blemish-removed face difference image are fused to obtain a target beautified image corresponding to the face image to be beautified.

[0136] In the disclosed embodiments, different users, scenarios, and modes have different requirements for image blemish removal, resulting in different target beauty parameters. To avoid the limitations of using a single-head model for blemish removal, which can only perform overall processing, multiple different branch networks can be used to perform beauty processing on the face image to be beautified. The required image can then be obtained from the blemish-removed face base image and at least one blemish-removed face difference image according to different target beauty parameters, generating a target beauty image that meets the target beauty parameters, and achieving personalized configuration of the target beauty image to make the target beauty image more in line with actual needs.

[0137] In some embodiments, the target beautification image, the blemish-removed base face image, and the at least one blemish-removed face difference image can be fused based on target beautification parameters to obtain the target beautification image. In the disclosed embodiments, different target beautification parameters may exist for the same image during the image beautification process. The target beautification parameters represent the image beautification requirements and may include, for example, the beautification type, beautification area, and the degree of beautification for each beautification type. The target beautification parameters can be determined based on one or a combination of parameters, such as the beautification processing mode, user requirements, and usage scenario. If the target beautification parameters are determined based on the usage scenario, for example, if the usage scenario is a photo-type image, the target beautification parameters are larger, indicating a higher beautification requirement; if the usage scenario is a normal scene, the corresponding target beautification parameters are smaller, indicating a lower beautification requirement. If the target beautification parameters are determined based on user requirements, higher requirements lead to higher target beautification parameters, while lower requirements lead to lower target beautification parameters.

[0138] Next, we will explain how the target beauty parameters are determined based on the beauty processing mode. The beauty processing mode can include one mode or multiple modes, and the target beauty parameters can be the beauty parameters of one mode or the beauty parameters of multiple modes. When the target beauty parameters include beauty parameters of multiple modes, they can include beauty parameters of the normal mode and beauty parameters of the beauty mode. Among them, the normal mode can include a non-beautification mode or a partial beauty mode. The non-beautification mode can be understood as the basic image quality without beauty, that is, a real portrait. For example, the non-beautification mode refers to only performing light and shadow smoothing and skin tone uniformity; the partial beauty mode refers to only performing partial beauty of all types corresponding to the multi-head machine learning model, for example, only performing light and shadow smoothing, skin tone uniformity, and blemish and acne removal without eye bag treatment. The beauty mode can be understood as a full beauty mode, that is, performing all types of beauty. When the beauty processing mode is one mode, it can be a full beauty mode. In the embodiments of the present disclosure, the beauty processing mode and the degree of beauty for various blemish removal can be configured according to actual needs and are not specifically limited here.

[0139] In some embodiments, the outputs of each branch network and the original image represented by the face image to be beautified can be fused according to the target beautification parameters to obtain a target beautification image corresponding to the target beautification parameters. For example, after obtaining the blemish-removed face base image and the blemish-removed face difference image, the face image to be beautified, the blemish-removed face base image, and the blemish-removed face difference image can be integrated to obtain the target beautification image. The target beautification image can be the blemish-removed image corresponding to the face image to be beautified, which is specifically determined based on the output of the machine learning model. For any mode, the final output image will include the original image and the blemish-removed face base image, so the target beautification parameters are mainly used to select the required blemish-removed face difference image. Based on this, the fusion process can be understood as: incrementally fusing the face image to be beautified, the blemish-removed face base image, and the blemish-removed face difference image corresponding to the target beautification parameters to obtain the target beautification image corresponding to the target beautification parameters. For example, the blemish-removed face base image can be fused with part or all of the first blemish-removed face difference image, the second blemish-removed face difference image, and the third blemish-removed face difference image according to incremental logic to obtain a target beautification image that matches the target beautification parameters.

[0140] Figure 8 The flowchart for obtaining the target beauty image is shown schematically in FIG. Figure 8 As shown in , the implementation method of fusing the face image to be beautified and the result output by the multi-head machine learning model to obtain the target beautified image may include the following steps, specifically:

[0141] In step S810, the face image to be beautified, the basic face image with blemishes removed, and the difference image of the face with blemishes removed corresponding to the target beautification parameters are fused according to corresponding degree weights to obtain a fixed-size fused image.

[0142] In the embodiment of the present disclosure, the face image to be beautified refers to the original image with a fixed input size, which can be obtained by using img ori To express it. The face image to be beautified, the basic face image to be blemish-removed, and the difference image to be blemish-removed each correspond to a degree weight, which can be used to represent the proportion of each image, such as 50% or 100%, etc. For the blemish-removed face difference images corresponding to multiple local branch networks, their degree weights are independent of each other and can be the same or different, depending on actual needs. For the basic face image to be blemish-removed corresponding to the basic branch network, the degree weight can be determined according to actual needs, and the sum of the degree weights of the face image to be beautified and the basic face image to be blemish-removed is 1.

[0143] Since the target beauty parameters may include beauty parameters for one mode or multiple modes, based on this, if the target beauty parameters are beauty parameters for the base mode, the face image to be beautified, the blemish-removed face base image, and the partially blemish-removed face difference image can be incrementally fused according to the corresponding degree weights outside or inside the multi-head machine learning model to obtain the fixed-size fused image. Incremental fusion can be used to indicate fusion according to incremental logic, that is, the blemish-removed face base image is sequentially fused with at least a partially blemish-removed face difference image to perform fusion according to the increments. The partially blemish-removed face difference image can, for example, be the blemish-removed face difference image output by the local branch network of the third head. Alternatively, the partially blemish-removed face difference image can also include the blemish-removed face difference image output by the local branch network of the second head, specifically determined according to the beauty type corresponding to the target beauty parameters. This is not particularly limited here, as long as it does not include the blemish-removed face difference images of all local branch networks.

[0144] In basic mode, the image quality improvement function for beautified facial images can be achieved, including light and shadow smoothing / dirty mouth corner processing by the first head basic branch network and large-scale skin smoothing processing by the fourth head (third local branch network), but does not include processing of obvious blemishes such as acne and eye bags. The basic mode processing can achieve image quality improvement of portraits without beautification function. The target beautification image corresponding to the beauty parameters of the basic mode is output by the basic branch network out_img base , the defect-free face difference image out_diff_img3 output by the third local branch network is summed with degree weights for incremental fusion to obtain a fixed-size fusion image in the basic mode, which can be specifically shown as formula (4):

[0145] out_img 无美颜(基础画质) =(1-b)*img ori +b*out_img base +c3*out_diff_img3 formula (4)

[0146] b represents the degree of basic blemish removal, with a value range of [0.0, 1.0]. c3 represents the degree of overall skin tone evenness, with a value range of [0.0, 1.0]. There are no specific restrictions on the degree weights here; for example, both b and c3 can be set to 1.

[0147] If the target beautification parameter is a beautification parameter of the beautification mode, the face image to be beautified, the basic image of the face with blemishes removed, and all difference images of the face with blemishes removed can be incrementally fused according to the corresponding degree weights outside the multi-head machine learning model to obtain the fixed-size fused image. The output image of the beautification mode is usually defined as the user's beautified portrait. In this beautification mode, the beautification function of the portrait can be realized, including light and shadow smoothing / dirty mouth corner processing of the first head, freckle and acne removal processing of the second head, eye bags / nasolabial fold processing of the third head, and large-scale skin smoothing processing of the fourth head. The fixed-size fused image of the beautification mode is shown in formula (5):

[0148]

[0149] b represents the degree of basic blemish removal, with a value range of [0.0, 1.0]. c1 represents the degree of blemish and acne removal, c2 represents the degree of eye bag and nasolabial fold removal, and c3 represents the degree of overall skin tone smoothing. The degree weights all have a value range of [0.0, 1.0]. There are no specific restrictions on the degree weights here; for example, b and c3, or c1 and c2, can both be set to 1.

[0150] In addition, if the target beautification parameter is a beautification parameter of a mode, it can be defaulted to the beautification parameter of the beautification mode, which can be applied to scenarios where more branch networks, i.e., smaller-granularity branch networks, need to be subdivided in the model learning stage, but only coarser-granularity (larger-granularity) combinations are needed in the model application stage. At this time, the multi-head machine learning model can be used to incrementally fuse the face image to be beautified, the blemish-removed face basic image, and all blemish-removed face difference images to obtain the fixed-size fused image, that is, the images output by the basic branch network and all local branch networks are fused according to incremental logic within the multi-head machine learning model to output a fixed-size fused image. Specifically, the images output by the basic branch network and the local branch network can also be incrementally fused according to the corresponding degree weights, which will not be repeated here. When the target beautification parameter is a beautification parameter of a mode, different types of beauty processing can be performed through multiple branch networks in the multi-head machine learning model, which can improve the accuracy of each type of beauty processing, and fusing them within the model can reduce computing power.

[0151] Figure 9 A schematic diagram of obtaining a fused image of the basic mode is shown schematically in FIG. Figure 9As shown in , a facial image to be beautified 901 is input into a multi-head machine learning model 902. A base image 905 of a blemish-free face is obtained through a main network 903 and a basic branch network 904. A difference image 907 of a blemish-free face is obtained through the main network 903 and a first local branch network 906. A difference image 909 of a blemish-free face is obtained through the main network 903 and a second local branch network 908. A difference image 911 of a blemish-free face is obtained through the main network 903 and a third local branch network 910. Furthermore, the facial image to be beautified 901, the base image 905 of the blemish-free face, and the difference image 911 of the blemish-free face are incrementally fused according to their corresponding degree weights to obtain a fixed-size fused image 912.

[0152] Figure 10 A schematic diagram of obtaining a fused image in the beauty mode is shown schematically in FIG. Figure 10 As shown in , a facial image to be beautified 1001 is input into a multi-head machine learning model 1002. A blemish-free facial base image 1005 is obtained through the main network 1003 and the basic branch network 1004. A blemish-free facial difference image 1007 is obtained through the main network 1003 and the first local branch network 1006. A blemish-free facial difference image 1009 is obtained through the main network 1003 and the second local branch network 1008. A blemish-free facial difference image 1011 is obtained through the main network 1003 and the third local branch network 1010. Further, the facial image to be beautified 1001, the blemish-free facial base image 1005, the blemish-free facial base image 1007, the blemish-free facial base image 1009, and the blemish-free facial difference image 1011 are incrementally fused according to the corresponding degree weights to obtain a fixed-size fused image 1012.

[0153] In the disclosed embodiment, the target beauty parameters can be used to fuse the images required by the target beauty parameters according to the degree weights, thereby realizing separate control of different types of blemishes. At the same time, separate control of the degree weights of different types of blemishes is realized, and a face difference image with at least one type of blemish removed can be obtained. Different types of beauty processing can be achieved through a multi-head machine learning model, so that the acquired image is more in line with the requirements of the target beauty parameters, thereby improving the targetedness and flexibility.

[0154] In step S820, a high-frequency image of the facial image to be beautified is obtained, and an intermediate beautified image is obtained by combining the fixed-size fusion image and the high-frequency image of the facial image to be beautified.

[0155] In the embodiment of the present disclosure, the high-frequency image can be used to represent the detail information in the facial image to be beautified. Figure 11 The flowchart for obtaining high-frequency images is shown in FIG. Figure 11 As shown in , it mainly includes the following steps:

[0156] In step S1101, the face image to be beautified is resized to obtain a resized face image;

[0157] In step S1102, the transformed facial image is enlarged to its original size to obtain a low-frequency facial image of the facial image to be beautified;

[0158] In step S1103, the high-frequency image is determined according to the difference between the face image to be beautified and the low-frequency face image.

[0159] In an embodiment of the present disclosure, if the facial image to be beautified is resized, the transformed facial image obtained by the resize transformation is inversely transformed according to the resize transformation information, and the transformed facial image is enlarged to its original size to obtain an inversely transformed facial image, so as to determine a low-frequency image of the facial image to be beautified; based on the difference between the facial image to be beautified and the low-frequency image of the facial image to be beautified, a high-frequency image of the facial image to be beautified is obtained; the low-frequency image of the facial image to be beautified has the same resolution as that of the facial image to be beautified.

[0160] The resizing information may include scaling information for the original facial image to be beautified. The resizing information may include scaling direction and range. The resolution of the resized facial image is lower than that of the original facial image to be beautified, and high-frequency information in the image may be lost during the resizing process. It should be noted that if the facial image to be beautified is rotated before being resized, the resizing process can be reversed after the inverse transformation, so that the two images are oriented in the same direction.

[0161] The inverse transformation can be the inverse of scaling, such as enlarging the image. Methods such as bilinear interpolation and nearest neighbor interpolation can be used for the inverse transformation. While the inverse transformation can restore resolution, it is difficult to recover the lost high-frequency information. In other words, the inverse-transformed facial image can be considered a low-frequency image of the face to be enhanced.

[0162] Furthermore, the high-frequency image is determined based on the difference between the facial image to be beautified and the low-frequency image. Specifically, the original rectangular facial image to be beautified is subtracted from the low-frequency image to obtain the original high-frequency information of the original rectangular facial image. The high-frequency image is used to describe the details in the facial image to be beautified, thereby enhancing its integrity and authenticity.

[0163] In another embodiment, the facial image to be beautified may be filtered to extract high-frequency information to obtain a high-frequency image. It should be noted that if the facial image to be beautified is not resized, there is no need to calculate the high-frequency information.

[0164] After obtaining the high-frequency image of the facial image to be beautified, the fixed-size fused image and the high-frequency image can be combined to obtain an intermediate beautified image. After obtaining the fixed-size fused image, the fixed-size fused image can be resized to obtain an intermediate low-frequency image; the intermediate low-frequency image can be fused with the high-frequency image to obtain an intermediate beautified image. Among them, the fixed-size fused image output by the multi-head machine learning model can be enlarged in size according to the saved size transformation information to enlarge it back to its original size, thereby obtaining a low-frequency image of the final blemish-free rectangular facial image of the original size, which can be represented as an intermediate low-frequency image. Furthermore, the intermediate low-frequency image can be fused with the high-frequency image of the facial image to be beautified to obtain the final blemish-free rectangular facial image of the original size, that is, the intermediate beautified image. The intermediate beautified image can be a blemish-free image of the original size.

[0165] In step S830 , the intermediate beautification image is filled into a corresponding position in the image to be processed corresponding to the face image to be beautified, so as to obtain the target beautification image.

[0166] In the disclosed embodiment, the intermediate beauty image can be filled into the corresponding position in the image to be processed to which the facial image to be beautified belongs, that is, filled into the overall image where the facial image to be beautified is located, to obtain the target beauty image. Exemplarily, the boundary area of ​​the intermediate beauty image can be gradually fused with the image to be processed. Specifically, the intermediate beauty image can be divided into a central area and a boundary area. The boundary area can be a partial area including the boundary. The width of the boundary area is, for example, 10% or 10 pixels of the intermediate beauty image, and the boundary area extends inward from the outer edge of the intermediate beauty image. The central area refers to the remaining area of ​​the intermediate beauty image excluding the boundary area. The boundary area generally does not cover the facial area. During the filling process, the intermediate beauty image in the central area can be directly replaced with the image at the corresponding position in the image to be processed that corresponds to the facial image to be beautified. At the same time, the intermediate beauty image in the boundary area can also be gradually fused with the image at the corresponding position in the image to be processed that corresponds to the facial image to be beautified, so that the boundary area achieves a smooth transition. Gradual fusion here means that the ratio of the pixel values ​​of the pixel points in the boundary area away from the central area to the ratio of the pixel values ​​of the pixel points in the boundary area close to the central area to the ratio of the pixel values ​​of the pixel points in the boundary area to the image to be processed is greater, and the pixel values ​​of the pixel points on the outer edge of the boundary area relative to the central area are the pixel values ​​of the image to be processed. For example, after determining the boundary area, the inner edge pixels of the boundary area are obtained and recorded as the first pixel; the outer edge pixels are obtained and recorded as the second pixel; and the boundary area is subjected to a gradient process of the first pixel and the second pixel until the outer edge pixels have the pixel values ​​of the image to be processed.

[0167] Figure 12 The specific process of gradient fusion is schematically shown in Figure 12 As shown in , the intermediate beautification image 1202 is gradually fused with the image to be processed 1201 in region 1210 and replaced in region 1211 to obtain the target beautification image 1203 .

[0168] It should be noted that when there are multiple faces, the intermediate beautification image corresponding to each face can be used to replace the corresponding face image to be beautified in the processed image, and gradient processing can be performed on the boundary area to obtain a target beautification image. It should be noted that for a face image to be beautified containing multiple faces, the filling can be performed in order of area from largest to smallest, or in other orders, which is not limited here.

[0169] By implementing transition processing through gradient fusion, we can avoid unnatural boundaries between the edges of the image after blemish removal and the image to be filled in, prevent color mutations that cause visual disharmony, and improve image quality.

[0170] The technical solution in the disclosed embodiment can perform the above-mentioned processing on each face to obtain a blemish-removed basic face image and at least one blemish-removed face difference image corresponding to each face, and incrementally fuse the face image to be beautified, the blemish-removed basic face image, and the at least one blemish-removed face difference image corresponding to each face according to the target beautification parameters, thereby improving the accuracy, pertinence and flexibility of image beautification.

[0171] Figure 13 The schematic flow of the image beautification processing method is shown in FIG. Figure 13 As shown in , it mainly includes the following steps:

[0172] Step S1301: Generate multiple face frames based on the recognized facial key points in the image to be processed, and retain face frames whose area is not less than the face area threshold. Assume that the image to be processed includes a main face, for example, face frame 1 is generated accordingly.

[0173] In step S1302 , the face frame 1 is enlarged, for example, to 1.1 times, and then the image within the face frame is captured to obtain the face image 1 to be beautified.

[0174] Step S1303: resize the face image 1 to be beautified, and resize the face image to be beautified to obtain a resized face image 1. Resize resize may include scaling, rotating, or padding.

[0175] Step S1304 , performing multiple types of defect processing on the transformed face image 1 according to the main network and multiple branch networks in the multi-head machine learning model to obtain a defect-removed face basic image 1 and a defect-removed face difference image 2 .

[0176] Step S1305 , fusing the face image to be beautified 1 , the base face image 1 after blemish removal, and the difference face image 2 after blemish removal to obtain a fixed-size fused image 1 .

[0177] Step S1306 : scaling the face image 1 to be beautified, and performing an inverse transformation to obtain a low-frequency image 1 .

[0178] Step S1307 : Subtract the face image to be beautified 1 from the low-frequency image 1 to obtain the high-frequency image 1 .

[0179] Step S1308 : enlarge the fixed-size fused image 1 to obtain an intermediate low-frequency image 1 .

[0180] Step S1309 : Obtain an intermediate beautified image 1 according to the intermediate low-frequency image 1 and the high-frequency image 1 .

[0181] Step S1310 : Fusing the intermediate beautified image 1 with the image to be processed to obtain a target beautified image.

[0182] The technical solution provided by the above-mentioned image beautification processing method, on the one hand, uses the basic branch network and multiple local branch networks in the multi-head machine learning model to perform different types of beautification processing on the beautified face image, such as beautification processing and incremental beautification processing, respectively. This avoids the limitation of the related art of using only one model for overall beautification processing, and can achieve separate control of different types of beautification processing, which can increase the flexibility and pertinence of image beautification processing, and improve the beautification effect and image quality. On the other hand, because different branch networks process different types of images, compared with using only one model or using multiple completely different models for processing, it reduces time consumption and memory usage, improves beautification efficiency, and reduces power consumption.

[0183] In one embodiment, Figure 2 The image beautification method shown is the first stage of beautification processing. Figure 2 Before or after the image beautification processing method, other stages of beautification processing are added. For example, the above machine learning model is used to remove blemishes from the image. After obtaining the image to be processed, Figure 2 The image is processed using the image beautification processing method, and the target beautified image obtained is the blemish-removed image. The beautification processing in other stages can be personalized beautification processing, so personalized beautification processing can be performed on the blemish-removed image to obtain the final beautified image.

[0184] Generally, blemish removal is necessary for image beautification, and users have relatively fixed requirements for blemish removal. Figure 2 This method of image beautification implements a universal blemish removal and beautification process. In contrast, skin smoothing, deformation, 3D, skin tone adjustment, and light and shadow adjustment are not essential, and users' specific needs for these processes are also personalized. These processes can be referred to as personalized beautification. They usually require specific user settings before processing. For example, the user selects one or more beauty functions and sets parameters such as the degree of skin smoothing and deformation. The terminal device then performs the processing based on the user's settings.

[0185] It should be noted that this disclosure is for Figure 2 The order of image beautification processing and other stages of beautification processing is not limited. For example, the personalized beautification processing can be performed on the face image to be beautified first to obtain the intermediate beautification image, and then the intermediate beautification image can be used as the face image to be beautified. Figure 2 The image beautification processing method is used, and the target beautification image obtained is the final output beautification image.

[0186] The technical solution provided in the embodiment of the present disclosure adopts a two-stage portrait image beautification processing method based on the portrait blemish removal DNN. The first stage performs portrait blemish removal processing, and crops the face area image according to the face frame information. According to the target beautification parameter requirements corresponding to different types of blemishes, the size of the complete face area image is transformed into the face image to be beautified corresponding to the multi-head machine learning model, and input into the multi-head machine learning model, and blemish removal processing is performed separately through multiple branch networks. Using multiple branch networks, different types of blemishes and noise points are subjected to independent scaling processing and model processing of different degrees in multiple times, which can obtain the best processing effect for various types of blemishes, and reduce power consumption compared to single processing of the complete face. In the model application stage, through the fusion between the original image and the output images of multiple heads, it is possible to realize the setting of blemish removal in different modes and by users in different modes, and to realize independent control of different modes and specific types of blemishes, thereby improving flexibility and increasing the scope of application. Multiple models utilize lightweight convolutional neural networks, resulting in fast processing time and low power consumption. This can be further controlled by setting different fixed sizes, enabling compatibility with mobile devices of varying computing power levels, including high, medium, and low, thus increasing versatility and applicability. During the model learning phase, independent loss functions are calculated and weights are assigned for the multiple branch networks comprised of the basic branch network and local branch networks. This improves the accuracy of the branch networks and makes it easier to achieve optimal blemish removal results than with a single-head network.

[0187] The exemplary embodiment of the present disclosure also provides an image beautification processing device. Figure 14As shown, the image beautification processing device 1400 may include:

[0188] Image acquisition module 1401, used to acquire the face image to be beautified;

[0189] The subject processing module 1402 is used to extract features of the face image to be beautified using a subject network of a multi-head machine learning model to obtain a feature image;

[0190] A basic beautification module 1403 is configured to perform beautification processing on the feature image using the basic branch network of the multi-head machine learning model to obtain a basic face image without blemishes;

[0191] An incremental beautification module 1404 is configured to perform incremental beautification processing on the feature image using a local branch network of the multi-head machine learning model to obtain at least one blemish-free face difference image;

[0192] The image fusion module 1405 is configured to fuse the facial image to be beautified, the basic facial image after blemish removal, and the at least one difference facial image after blemish removal to obtain a target beautified image corresponding to the facial image to be beautified.

[0193] In an exemplary embodiment of the present disclosure, the main network includes: a first pixel rearrangement layer, at least one convolution layer, and at least one transposed convolution layer; the main processing module includes: a first pixel rearrangement module, used to use the first pixel rearrangement layer to perform a first pixel rearrangement process on the face image to be beautified to obtain a first feature image; a convolution module, used to use the convolution layer to perform a convolution process on the first feature image to obtain a second feature image; and a transposed convolution module, used to use the transposed convolution layer to perform a transposed convolution process on the second feature image to obtain the feature image.

[0194] In an exemplary embodiment of the present disclosure, the basic beautification module includes: a second pixel rearrangement module, configured to perform a second pixel rearrangement process on the feature image using the second pixel rearrangement layer to obtain the blemish-free basic face image.

[0195] In an exemplary embodiment of the present disclosure, the basic branch network also includes a transposed convolution layer and a convolution layer; the basic beautification module also includes: an upsampling module for using the transposed convolution layer in the basic branch network to upsample the feature image to obtain an intermediate feature image; a convolution processing module for performing convolution processing on the intermediate feature image through the convolution layer in the basic branch network.

[0196] In an exemplary embodiment of the present disclosure, there is at least one local branch network, and the local branch network includes at least a third pixel rearrangement layer; the incremental beautification module includes: a third pixel rearrangement module, which is used to use the third pixel rearrangement layer in each of the local branch networks to perform third pixel rearrangement processing on the feature image to perform incremental beautification processing, and obtain the blemish-free face difference image of each of the local branch networks.

[0197] In an exemplary embodiment of the present disclosure, the local branch network also includes a transposed convolution layer and a convolution layer; the incremental beautification module also includes: an upsampling processing module, which is used to use the transposed convolution layer in each of the local branch networks to upsample the feature image to obtain an intermediate local feature image; and a convolution processing module, which is used to perform convolution processing on the intermediate local feature image through the convolution layer in each of the local branch networks.

[0198] In an exemplary embodiment of the present disclosure, the image fusion module includes: a fusion control module, which is used to fuse the face image to be beautified, the blemish-removed face basic image and the at least one blemish-removed face difference image based on the target beautification parameters to obtain the target beautification image.

[0199] In an exemplary embodiment of the present disclosure, the fusion control module includes: a parameter fusion module, which is used to fuse the face image to be beautified, the blemish-removed face basic image and the blemish-removed face difference image corresponding to the target beautification parameters to obtain the target beautification image corresponding to the target beautification parameters.

[0200] In an exemplary embodiment of the present disclosure, the parameter fusion module includes: a fusion image acquisition module, which is used to fuse the face image to be beautified, the blemish-removed face basic image and the blemish-removed face difference image corresponding to the target beautification parameter according to corresponding degree weights to obtain a fixed-size fusion image; an intermediate beautification image acquisition module, which is used to obtain a high-frequency image of the face image to be beautified, and combine the fixed-size fusion image and the high-frequency image of the face image to be beautified to obtain an intermediate beautification image; and a filling module, which is used to fill the intermediate beautification image into a corresponding position in the image to be processed corresponding to the face image to be beautified to obtain the target beautification image.

[0201] In an exemplary embodiment of the present disclosure, the target beautification parameters are beautification parameters of multiple modes, and the multiple modes include a basic mode and a beautification mode; the fused image acquisition module includes: a first fusion module, which is used to, if the target beautification parameters are the beautification parameters of the basic mode, fuse the face image to be beautified, the blemish-removed face basic image, and some blemish-removed face difference images according to corresponding degree weights to obtain the fixed-size fused image; a second fusion module, which is used to, if the target beautification parameters are the beautification parameters of the beauty mode, fuse the face image to be beautified, the blemish-removed face basic image, and all blemish-removed face difference images according to corresponding degree weights to obtain the fixed-size fused image.

[0202] In an exemplary embodiment of the present disclosure, the target beautification parameters are beautification parameters of a pattern; the fused image acquisition module includes: a third fusion module, used to fuse the face image to be beautified, the blemish-removed face basic image and all blemish-removed face difference images through a multi-head machine learning model to obtain the fixed-size fused image.

[0203] In an exemplary embodiment of the present disclosure, the intermediate image acquisition module includes: a size conversion module, used to perform size conversion on the facial image to be beautified to obtain a transformed facial image; an inverse conversion module, used to enlarge the transformed facial image to its original size to obtain a low-frequency facial image of the facial image to be beautified; and a high-frequency image acquisition module, used to determine the high-frequency image based on the difference between the facial image to be beautified and the low-frequency facial image.

[0204] In an exemplary embodiment of the present disclosure, the intermediate beautification image acquisition module includes: a low-frequency image acquisition module, used to resize the fixed-size fused image to obtain an intermediate low-frequency image; and a fusion module, used to fuse the intermediate low-frequency image with the high-frequency image to obtain an intermediate beautification image.

[0205] In an exemplary embodiment of the present disclosure, the filling module includes: a region division module for dividing the intermediate beautification image into a central region and a boundary region; a replacement module for replacing the image to be processed corresponding to the face image to be beautified according to the intermediate beautification image in the central region;

[0206] A gradient fusion module is used to gradually fuse the intermediate beauty image in the boundary area with the image to be processed to obtain the target beauty image; wherein the pixel values ​​of the pixel points at the outer edge of the boundary area relative to the central area are the pixel values ​​of the image to be processed.

[0207] In an exemplary embodiment of the present disclosure, the device also includes: a loss function determination module, which is used to obtain the first loss function of the basic branch network and the second loss function of the local branch network, and determine the overall loss function based on the first loss function and the second loss function; a model training module, which is used to adjust the network parameters of the main network, the basic branch network and the local branch network according to the overall loss function for joint training until the overall loss function meets the convergence condition, so as to obtain the multi-head machine learning model.

[0208] In an exemplary embodiment of the present disclosure, the loss function determination module is configured to: obtain the first sample image and the first annotated image corresponding to the basic branch network, and obtain the second sample image and the second annotated image of the local branch network; process the corresponding first sample image through the basic branch network to obtain the first sample beautification image of the basic branch network, and process the corresponding second sample image through the local branch network to obtain the second sample beautification image of the local branch network; determine the first loss function corresponding to the basic branch network based on the difference between the first sample beautification image and the first annotated image; determine the second loss function corresponding to the local branch network based on the difference between the second sample beautification image and the second annotated image.

[0209] In an exemplary embodiment of the present disclosure, the loss function determination module includes: a loss function combination module, configured to combine the first loss function and the second loss function according to corresponding weight parameters to obtain the overall loss function.

[0210] In an exemplary embodiment of the present disclosure, the target beautification image is a blemish-removed beautification image. After obtaining the blemish-removed beautification image, the device further includes: a personalized beautification module, configured to perform personalized beautification processing on the blemish-removed beautification image to obtain a beautification image.

[0211] The specific details of each part of the above-mentioned image beautification processing device 1400 have been described in detail in the implementation of the image beautification processing method. For undisclosed details, please refer to the implementation content of the method part, and thus will not be repeated here.

[0212] The exemplary embodiments of the present disclosure further provide an electronic device for executing the aforementioned image beautification processing method. The electronic device may be the aforementioned terminal, including but not limited to a smartphone, tablet computer, wearable device, computer, etc. Generally, an electronic device includes a processor and a memory. The memory is used to store executable instructions for the processor and may also store application data, such as image data, game data, etc.; the processor is configured to execute the image beautification processing method of this exemplary embodiment by executing the executable instructions.

[0213] Below is Figure 15 The structure of the electronic device is exemplarily described by taking the mobile terminal 1500 in FIG. 1 as an example. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 15 The construction in can also be applied to fixed type equipment.

[0214] like Figure 15 As shown, the mobile terminal 1500 may specifically include: a processor 1501, a memory 1502, a bus 1503, a mobile communication module 1504, an antenna 1, a wireless communication module 1505, an antenna 2, a display screen 1506, a camera module 1507, an audio module 1508, a power module 1509 and a sensor module 1510.

[0215] The processor 1501 may include one or more processing units, for example, the processor 1501 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-Network Processing Unit). The image beautification processing method in this exemplary embodiment may be executed by an AP, a GPU, or a DSP. When the method involves neural network-related processing, it may be executed by an NPU. For example, the NPU may load neural network parameters and execute neural network-related algorithm instructions.

[0216] The encoder can encode (i.e., compress) an image or video to reduce the data size for easy storage or transmission. The decoder can decode (i.e., decompress) the encoded data of the image or video to restore the image or video data. The mobile terminal 1500 can support one or more encoders and decoders, such as: image formats such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), and video formats such as MPEG (Moving Picture Experts Group) 1, MPEG10, H.1063, H.1064, and HEVC (High Efficiency Video Coding).

[0217] The processor 1501 may be connected to the memory 1502 or other components via a bus 1503 .

[0218] Memory 1502 can be used to store computer-executable program code, which includes instructions. Processor 1501 executes the instructions stored in memory 1502 to execute various functional applications and data processing of mobile terminal 1500. Memory 1502 can also store application data, such as images, videos, and other files.

[0219] The communication functions of mobile terminal 1500 are implemented through mobile communication module 1504, antenna 1, wireless communication module 1505, antenna 2, a modem processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 1504 can provide 3G, 4G, and 5G mobile communication solutions for mobile terminal 1500. Wireless communication module 1505 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for mobile terminal 1500.

[0220] The display screen 1506 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The camera module 1507 is used to implement shooting functions, such as shooting images, videos, etc., and the camera module may include a color temperature sensor array. The audio module 1508 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 1509 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 1510 may include one or more sensors for implementing corresponding sensing detection functions. For example, the sensor module 1510 may include an inertial sensor, which is used to detect the motion posture of the mobile terminal 1500 and output inertial sensing data.

[0221] It should be noted that a computer-readable storage medium is also provided in an embodiment of the present disclosure. The computer-readable storage medium may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the electronic device.

[0222] Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device.

[0223] Computer-readable storage media can transmit, propagate, or transfer programs for use by or in conjunction with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0224] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.

[0225] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0226] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0227] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0228] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims. It should be understood that the present disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image beautification processing method, characterized in that: include: Get the face image to be beautified; Extract features from the facial image to be beautified using a main network of a multi-head machine learning model to obtain a feature image; Performing beauty processing on the feature image through the basic branch network of the multi-head machine learning model to obtain a basic image of the face without blemishes; Performing incremental beautification processing on the feature image through a local branch network of the multi-head machine learning model to obtain at least one blemish-free face difference image; fusing the facial image to be beautified, the basic facial image after blemish removal, and the at least one difference image after blemish removal to obtain a target beautified image corresponding to the facial image to be beautified; The step of fusing the face image to be beautified, the basic face image after blemish removal, and the at least one difference face image after blemish removal to obtain a target beautified image corresponding to the face image to be beautified includes: The face image to be beautified, the basic face image after blemish removal, and the difference image after blemish removal corresponding to the target beautification parameters are fused according to corresponding degree weights to obtain a fixed-size fused image; Acquire a high-frequency image of the face image to be beautified, and combine the fixed-size fusion image and the high-frequency image of the face image to be beautified to acquire an intermediate beautified image; The intermediate beautification image is filled into a corresponding position in the image to be processed corresponding to the face image to be beautified, so as to obtain the target beautification image.

2. The image beautification processing method according to claim 1, characterized in that: The main network includes: a first pixel rearrangement layer, at least one convolutional layer, and at least one transposed convolutional layer; The method of extracting features of the face image to be beautified by using the main network of the multi-head machine learning model to obtain a feature image includes: performing a first pixel rearrangement process on the facial image to be beautified using the first pixel rearrangement layer to obtain a first feature image; Performing convolution processing on the first feature image using the convolution layer to obtain a second feature image; The transposed convolution layer is used to perform a transposed convolution process on the second feature image to obtain the feature image.

3. The image beautification processing method according to claim 1, characterized in that: The basic branch network includes at least a second pixel rearrangement layer; the basic branch network of the multi-head machine learning model performs beautification processing on the feature image to obtain a blemish-free basic face image, including: The second pixel rearrangement layer is used to perform a second pixel rearrangement process on the feature image to obtain the blemish-removed basic face image.

4. The image beautification processing method according to claim 3, characterized in that: The basic branch network further includes a transposed convolution layer and a convolution layer; the basic branch network of the multi-head machine learning model performs beautification processing on the feature image to obtain a blemish-free face basic image, and further includes: Upsampling the feature image using the transposed convolution layer in the basic branch network to obtain an intermediate feature image; The intermediate feature image is convolved by the convolution layer in the basic branch network.

5. The image beautification processing method according to claim 1, characterized in that: There is at least one local branch network, and the local branch network includes at least a third pixel rearrangement layer; The incremental beautification processing is performed on the feature image by using a local branch network of the multi-head machine learning model to obtain at least one blemish-free face difference image, including: The third pixel rearrangement layer in each of the local branch networks is used to perform a third pixel rearrangement process on the feature image to perform incremental beautification processing, thereby obtaining the blemish-free face difference image of each of the local branch networks.

6. The image beautification processing method according to claim 5, characterized in that: The local branch network further includes a transposed convolution layer and a convolution layer; the local branch network of the multi-head machine learning model performs incremental beautification processing on the feature image to obtain at least one defect-removed face difference image, and further includes: Upsampling the feature image using the transposed convolution layer in each of the local branch networks to obtain an intermediate local feature image; The intermediate local feature image is convolved by the convolution layer in each of the local branch networks.

7. The image beautification processing method according to claim 1, characterized in that: The fusing the facial image to be beautified, the basic facial image after blemish removal, and the at least one difference image after blemish removal to obtain a target beautified image corresponding to the facial image to be beautified includes: Based on the target beautification parameters, the to-beautify face image, the blemish-removed face basic image, and the at least one blemish-removed face difference image are fused to obtain the target beautification image.

8. The image beautification processing method according to claim 7, characterized in that: The step of fusing the face image to be beautified, the blemish-removed face base image, and the at least one blemish-removed face difference image based on the target beautification parameter to obtain the target beautification image includes: The face image to be beautified, the basic face image after blemish removal, and the difference image after blemish removal corresponding to the target beautification parameters are fused to obtain a target beautification image corresponding to the target beautification parameters.

9. The image beautification processing method according to claim 1, characterized in that: The target beauty parameters are beauty parameters of multiple modes, and the multiple modes include a basic mode and a beauty mode; The step of fusing the face image to be beautified, the basic face image after blemish removal, and the difference face image after blemish removal corresponding to the target beautification parameters according to corresponding degree weights to obtain a fixed-size fused image includes: If the target beautification parameter is a beautification parameter of a basic mode, the face image to be beautified, the blemish-removed basic face image, and the partial blemish-removed face difference image are fused according to corresponding degree weights to obtain the fixed-size fused image; If the target beautification parameter is a beautification parameter of a beautification mode, the face image to be beautified, the blemish-removed face basic image, and all blemish-removed face difference images are fused according to corresponding degree weights to obtain the fixed-size fused image.

10. The image beautification processing method according to claim 1, characterized in that: The target beautification parameter is a beautification parameter of a mode; fusing the face image to be beautified, the blemish-removed face base image, and the blemish-removed face difference image corresponding to the target beautification parameter according to corresponding degree weights to obtain a fixed-size fused image, including: The face image to be beautified, the basic face image after blemish removal, and all difference images after blemish removal are fused through a multi-head machine learning model to obtain the fixed-size fused image.

11. The image beautification processing method according to claim 1, characterized in that: The step of obtaining a high-frequency image of the face image to be beautified includes: Performing size transformation on the facial image to be beautified to obtain a transformed facial image; Enlarging the transformed facial image to its original size to obtain a low-frequency facial image of the facial image to be beautified; The high-frequency image is determined according to a difference between the face image to be beautified and the low-frequency face image.

12. The image beautification processing method according to claim 1, characterized in that: The step of combining the fixed-size fused image and the high-frequency image of the face image to be beautified to obtain an intermediate beautified image includes: performing size transformation on the fixed-size fused image to obtain an intermediate low-frequency image; The intermediate low-frequency image is fused with the high-frequency image to obtain an intermediate beautification image.

13. The image beautification processing method according to claim 1, characterized in that: Filling the intermediate beautification image into a corresponding position in the image to be processed corresponding to the face image to be beautified to obtain a target beautification image corresponding to the target beautification parameter includes: Dividing the intermediate beautification image into a central area and a boundary area; replacing the image to be processed corresponding to the facial image to be beautified according to the intermediate beautification image in the central area; Gradually fuse the intermediate beautification image in the boundary area with the image to be processed to obtain the target beautification image; The pixel values ​​of the pixel points at the outer edge of the boundary area relative to the central area are the pixel values ​​of the image to be processed.

14. The image beautification processing method according to claim 1, characterized in that: The method further comprises: Obtaining a first loss function of the basic branch network and a second loss function of the local branch network, and determining an overall loss function based on the first loss function and the second loss function; According to the overall loss function, the network parameters of the main network, the basic branch network and the local branch network are adjusted for joint training until the overall loss function meets the convergence condition to obtain the multi-head machine learning model.

15. The image beautification processing method according to claim 14, characterized in that: The obtaining of the first loss function of the basic branch network and the second loss function of the local branch network includes: Acquire a first sample image and a first annotated image corresponding to the basic branch network, and acquire a second sample image and a second annotated image corresponding to the local branch network; Processing the corresponding first sample image through the basic branch network to obtain a first sample beautification image of the basic branch network, and processing the corresponding second sample image through the local branch network to obtain a second sample beautification image of the local branch network; Determining a first loss function corresponding to the basic branch network according to a difference between the first sample beautified image and the first annotated image; Determine a second loss function corresponding to the local branch network based on a difference between the second sample beautified image and the second annotated image.

16. The image beautification processing method according to claim 14, characterized in that: The determining of an overall loss function according to the first loss function and the second loss function includes: The first loss function and the second loss function are combined according to corresponding weight parameters to obtain the overall loss function.

17. An image beautification processing device, characterized in that: include: An image acquisition module is used to acquire the face image to be beautified; A subject processing module is used to extract features of the face image to be beautified through a subject network of a multi-head machine learning model to obtain a feature image; A basic beautification module, configured to perform beautification processing on the feature image through the basic branch network of the multi-head machine learning model to obtain a basic image of the face without blemishes; An incremental beautification module, configured to perform incremental beautification processing on the feature image through a local branch network of the multi-head machine learning model to obtain at least one blemish-free face difference image; an image fusion module, configured to fuse the face image to be beautified, the base face image after blemish removal, and the at least one difference image after blemish removal to obtain a target beautified image corresponding to the face image to be beautified; The step of fusing the face image to be beautified, the basic face image after blemish removal, and the at least one difference face image after blemish removal to obtain a target beautified image corresponding to the face image to be beautified includes: The face image to be beautified, the basic face image after blemish removal, and the difference image after blemish removal corresponding to the target beautification parameters are fused according to corresponding degree weights to obtain a fixed-size fused image; Acquire a high-frequency image of the face image to be beautified, and combine the fixed-size fusion image and the high-frequency image of the face image to be beautified to acquire an intermediate beautified image; The intermediate beautification image is filled into a corresponding position in the image to be processed corresponding to the face image to be beautified, so as to obtain the target beautification image.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image beautification processing method according to any one of claims 1 to 16 is implemented.

19. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the image beautification processing method according to any one of claims 1 to 16 by executing the executable instructions.

Images