Portrait image enhancement model training method, image enhancement method, device and medium

Abstract

Embodiments of the present disclosure provide a portrait image enhancement model training method, a portrait image enhancement method, an apparatus, an electronic device, and a storage medium. The method includes: obtaining a training data set including a plurality of image training samples; generating a corresponding face gradient image based on a face image to be enhanced; performing image enhancement processing on the face image to be enhanced based on the face gradient image by a face image enhancement model to be trained to obtain an output face image; generating a loss function according to the output face image and a high-quality face image; and training the face image enhancement model to be trained according to the loss function to obtain a target face image enhancement model. The method introduces the face gradient image as a control condition into the model, guides the model to better maintain the structural consistency of the original image in the image enhancement process, and improves the fidelity of the output image.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a method for training a human image enhancement model, a method for enhancing human images, an apparatus, an electronic device, a computer program product, and a storage medium. Background Technology

[0002] With the widespread adoption of mobile internet and short video applications, users' demands for portrait image quality are increasing. In actual shooting and dissemination, limitations imposed by shooting equipment, ambient lighting, and quality loss during video encoding and decoding often result in blurry, noisy, and detail-lossed original images, affecting visual quality. Especially for portraits, which are the visual focus, the clarity, detail retention, and identity consistency of the facial area directly impact the viewing experience and content quality. Currently, image enhancement methods based on Generative Adversarial Networks (GANs) have improved image quality to some extent, but problems remain, such as uncontrollable generation results and reduced image fidelity, particularly in scenarios requiring precise control over the fidelity and generation intensity of different facial regions. At the technical implementation level, video restoration and enhancement algorithms often face a trade-off between restoration and fidelity. Therefore, achieving both high fidelity and excellent restoration results in portrait restoration scenarios has become a significant challenge in the current technological field. Summary of the Invention

[0003] This disclosure provides a method for training a portrait image enhancement model, a portrait image enhancement method, an apparatus, an electronic device, a computer program product, and a storage medium.

[0004] According to a first aspect of the present disclosure, a method for training a face image enhancement model is provided, comprising: acquiring a training dataset including multiple image training samples; the image training samples including a face image to be enhanced and a high-quality face image that are in a corresponding relationship; generating a corresponding face gradient image based on the face image to be enhanced; performing image enhancement processing on the face image to be enhanced based on the face gradient image using the face image enhancement model to be trained, to obtain an output face image; generating a loss function based on the output face image and the high-quality face image; and training the face image enhancement model to be trained based on the loss function to obtain a target face image enhancement model.

[0005] In some exemplary embodiments of this disclosure, the step of performing image enhancement processing on the face image to be enhanced based on the face gradient image using the face image enhancement model to be trained, to obtain an output face image, includes: encoding the face image to be enhanced based on the face gradient image using an encoder to obtain a face feature vector to be enhanced; quantizing the face feature vector to be enhanced using a quantization codebook to obtain a quantized face feature vector; and decoding the quantized face feature vector based on the face gradient image using a decoder to obtain the output face image.

[0006] In some exemplary embodiments of this disclosure, generating a corresponding face gradient image based on the face image to be enhanced includes: performing image convolution processing on the face image to be enhanced to obtain a first face gradient image; generating a corresponding face mask image based on the face image to be enhanced; and performing mask processing on the first face gradient image based on the face mask image to obtain the face gradient image.

[0007] In some exemplary embodiments of this disclosure, generating a corresponding face mask based on the face image to be enhanced includes: generating a first mask that completely covers the face image to be enhanced; generating a second mask that partially covers the face image to be enhanced; and randomly determining the face mask as either the first mask or the second mask based on a preset selection probability.

[0008] In some exemplary embodiments of this disclosure, generating a corresponding face mask based on the face image to be enhanced includes: parsing the face image to be enhanced using a face parsing model to divide the face image to be enhanced into at least one face region; and generating the face mask based on the at least one face region.

[0009] In some exemplary embodiments of this disclosure, generating the face mask based on the at least one face region includes: determining the region probability corresponding to the face region based on the face region category corresponding to the face region; the region probability corresponds one-to-one with the face region; randomly determining the corresponding face region as an covered region or an uncovered region based on the region probability; and generating the face mask based on each of the face regions.

[0010] In some exemplary embodiments of this disclosure, generating the face mask map based on the at least one face region includes: determining the region mask intensity corresponding to the face region based on the face region category corresponding to the face region; the region mask intensity corresponds one-to-one with the face region; and generating the face mask map based on the region mask intensity corresponding to each of the face regions.

[0011] In some exemplary embodiments of this disclosure, the method further includes: performing scene recognition on the face image to be enhanced to obtain scene category information; and determining the region probability or region mask intensity corresponding to each face region based on the scene category information.

[0012] In some exemplary embodiments of this disclosure, the method further includes: obtaining region configuration information; and determining the region probability or region mask intensity corresponding to each of the face regions based on the region configuration information.

[0013] According to a second aspect of the present disclosure, a face image enhancement method is provided, comprising: acquiring an image to be processed; inputting the image to be processed into a face image enhancement model; the face image enhancement model being trained by any of the face image enhancement model training methods described herein; and the face image enhancement model performing image enhancement processing on the image to be processed to obtain a target image.

[0014] According to a third aspect of the present disclosure, a face image enhancement model training apparatus is provided, comprising: a sample acquisition module configured to acquire a training dataset including multiple image training samples; the image training samples including a face image to be enhanced and a high-quality face image that are in a corresponding relationship; a gradient image module configured to generate a corresponding face gradient image based on the face image to be enhanced; an image enhancement module configured to perform image enhancement processing on the face image to be enhanced based on the face gradient image using the face image enhancement model to be trained, to obtain an output face image; a loss function module configured to generate a loss function based on the output face image and the high-quality face image; and a model training module configured to train the face image enhancement model to be trained according to the loss function, to obtain a target face image enhancement model.

[0015] According to a fourth aspect of the present disclosure, a face image enhancement apparatus is provided, comprising: an image acquisition module configured to acquire an image to be processed; an image input module configured to input the image to be processed into a face image enhancement model; the face image enhancement model being trained by any of the face image enhancement model training methods described herein; and an image processing module configured to perform image enhancement processing on the image to be processed by the face image enhancement model to obtain a target image.

[0016] According to a fifth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement any of the face image enhancement model training methods or any of the face image enhancement methods described in the present disclosure.

[0017] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the face image enhancement model training methods or any of the face image enhancement methods described in the present disclosure.

[0018] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program that is executed by a processor using any of the face image enhancement model training methods or any of the face image enhancement methods described in the present disclosure.

[0019] The portrait image enhancement model training method provided in this disclosure generates a corresponding face gradient image based on the face image to be enhanced; the face image enhancement model to be trained performs image enhancement processing on the face image to be enhanced based on the face gradient image to obtain an output face image; a loss function is generated based on the output face image and the high-quality face image; and the face image enhancement model to be trained is trained based on the loss function to obtain a target face image enhancement model. This method introduces the face gradient image as a control condition into the model, guiding the model to better maintain the structural consistency of the original image during the image enhancement process and improving the fidelity of the output image.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 A schematic diagram of an exemplary system architecture to which the methods of embodiments of this disclosure can be applied is shown.

[0023] Figure 2 This is a flowchart illustrating a method for training a portrait image enhancement model according to an exemplary embodiment.

[0024] Figure 3 This is a schematic diagram of the training process of a portrait image enhancement model, as illustrated in the example. Figure 1 .

[0025] Figure 4 This is a flowchart illustrating a face gradient image generation method.

[0026] Figure 5This is a schematic diagram of the training process of a portrait image enhancement model, as illustrated in the example. Figure 2 .

[0027] Figure 6 This is based on the flow of a face mask generation method illustrated in the example. Figure 1 .

[0028] Figure 7 This is based on the flow of a face mask generation method illustrated in the example. Figure 2 .

[0029] Figure 8 This is a flowchart illustrating a face image enhancement method according to an exemplary embodiment.

[0030] Figure 9 This is a block diagram illustrating a face image enhancement model training apparatus according to an exemplary embodiment.

[0031] Figure 10 This is a block diagram illustrating a face image enhancement device according to an exemplary embodiment.

[0032] Figure 11 This is a schematic diagram illustrating the structure of an electronic device suitable for implementing exemplary embodiments of the present disclosure, according to an exemplary embodiment. Detailed Implementation

[0033] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0034] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0035] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in at least one hardware module or integrated circuit, or in different network and / or processor devices and / or microcontroller devices.

[0036] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0037] In this specification, the terms “a,” “an,” “the,” “the,” and “at least one” are used to indicate the presence of at least one element / component / etc.; the terms “comprising,” “including,” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first,” “second,” and “third,” etc., are used only as markings and are not a limitation on the number of objects.

[0038] Figure 1 A schematic diagram of an exemplary system architecture to which the methods of embodiments of this disclosure can be applied is shown.

[0039] like Figure 1 As shown, the system architecture may include server 101, network 102, terminal device 103, terminal device 104, and terminal device 105. Network 102 serves as the medium for providing a communication link between terminal device 103, terminal device 104, or terminal device 105 and server 101. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0040] Server 101 can be a server that provides various services, such as a back-end management server that supports the devices operated by users using terminal devices 103, 104, or 105. The back-end management server can analyze and process received requests and other data, and feed back the processing results to terminal devices 103, 104, or 105.

[0041] Terminal devices 103, 104, and 105 can be smartphones, tablets, laptops, desktop computers, smart speakers, wearable smart devices, virtual reality devices, augmented reality devices, etc., but are not limited to these.

[0042] It should be understood that Figure 1 The number of terminal devices 103, 104, 105, network 102, and server 101 in the diagram is merely illustrative. Server 101 can be a single physical server, a server cluster consisting of multiple servers, or a cloud server. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0043] The steps of the method in the exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings and examples.

[0044] Figure 2 This is a flowchart illustrating a method for training a portrait image enhancement model according to an exemplary embodiment. Figure 3 This is a schematic diagram of the training process of a portrait image enhancement model, as illustrated in the example. Figure 1 . Figure 2 The method provided in the embodiments can be executed by any electronic device, such as the one described above. Figure 1 Terminal devices in or Figure 1 The server in, or Figure 1 The terminal devices and servers in the process are executed together, but this disclosure does not limit this.

[0045] In step S210, a training dataset including multiple image training samples is obtained; the image training samples include corresponding face images to be enhanced and high-quality face images.

[0046] In this embodiment of the disclosure, a large number of image training samples are pre-acquired to construct a training dataset for training the portrait image enhancement model. Each image training sample includes at least one set of corresponding face images to be enhanced and high-quality face images. The face image to be enhanced serves as the face image to be enhanced in the image training samples. The high-quality face image serves as the target face image in the image training samples.

[0047] In step S220, a corresponding face gradient image is generated based on the face image to be enhanced.

[0048] In this embodiment, a face gradient image is obtained by performing gradient calculations on the face image to be enhanced. This face gradient image represents the spatial variation of pixel intensity in the image. Since the pixel intensity variation is relatively large at the edge contours of the face in the face image, the face gradient image essentially carries key structural information and contour details of the face image, especially the facial features and boundaries, which have strong representational capabilities. This disclosure introduces the face gradient image as a control condition into the model, guiding the model to better maintain the structural consistency of the original image during the image enhancement process and improve the fidelity of the output image.

[0049] In an exemplary embodiment, the face gradient image can be obtained by performing image convolution processing on the face image to be enhanced. For example, a convolution kernel based on the Sobel operator or the Laplacian operator can be used.

[0050] In step S230, the face image to be enhanced is processed by the face gradient image based on the face image enhancement model to be trained, and an output face image is obtained.

[0051] In this embodiment, a pre-constructed face image enhancement model is used to perform image enhancement processing on the face images to be enhanced in each training image sample to obtain the corresponding output face image. Based on the comparison between the output face image and the corresponding high-quality face image, the face image enhancement model is trained. During the image enhancement process of the face image enhancement model, the aforementioned face gradient image is introduced as a control condition, so that the model can better maintain the structural consistency of the original image during the image enhancement process.

[0052] In an exemplary embodiment, the face image enhancement model to be trained can be constructed using a Generative Adversarial Network (GAN) or Vector Quantization Generative Adversarial Network (VQGAN) architecture.

[0053] Figure 3 This is a schematic diagram of the training process of a portrait image enhancement model, as illustrated in the example. Figure 1 .like Figure 3 As shown, the portrait image enhancement model performs image enhancement processing on the face image to be enhanced through the following steps.

[0054] The face image to be enhanced is encoded by an encoder based on the face gradient image to obtain the face feature vector to be enhanced. The face feature vector to be enhanced is quantized using a quantization codebook to obtain a quantized face feature vector. The quantized facial feature vector is decoded based on the facial gradient image by a decoder to obtain the output facial image.

[0055] In this embodiment, the face image enhancement model is constructed based on the encoder-quantization-decoder VQGAN model framework. The face gradient image is introduced into the encoder and decoder respectively to participate in the corresponding encoding and decoding processes, thereby controlling the image enhancement process. First, the encoder extracts and encodes features from the input face image to be enhanced and the face gradient image simultaneously or separately, and fuses the two features to obtain a face feature vector rich in content and structural information. Next, the face feature vector to be enhanced is discretely quantized through a quantization codebook, mapping continuous features to discrete vectors in the codebook to learn high-quality face priors and stabilize the training, resulting in a quantized face feature vector. Finally, the decoder performs upsampling and image reconstruction based on the quantized face feature vector that incorporates the corresponding control information from the face gradient image, generating the enhanced output face image. In this process, the face gradient image participates as a control condition signal throughout, constraining the encoding and decoding process and ensuring that the output image maintains a high degree of consistency with the input in terms of structural contour, achieving high fidelity.

[0056] In an exemplary embodiment, the encoder fuses the feature vectors of the face image to be enhanced and the face gradient image by feature splicing, which may specifically include the following steps.

[0057] The face gradient image is encoded by an encoder to obtain a first face feature vector; The face image to be enhanced is encoded by an encoder to obtain a second face feature vector; The first face feature vector and the second face feature vector are concatenated to obtain the face feature vector to be enhanced.

[0058] In step S240, a loss function is generated based on the output face image and the high-quality face image.

[0059] In this embodiment, a loss function is generated by comparing the difference between the output face image generated by the model to be trained and the corresponding high-quality face image. The loss function is typically composed of a weighted combination of multiple sub-loss terms to guide the direction of model training and optimization. It should be noted that this loss function can be constructed using various methods, all of which should be considered within the scope of this disclosure.

[0060] In an exemplary embodiment, the portrait image enhancement model includes a generator and a discriminator during training. The generator is responsible for generating realistic data samples, while the discriminator is used to distinguish between real and generated data. The two are trained alternately in an adversarial manner: the discriminator aims to maximize its ability to distinguish between real and fake data, that is, to classify real samples as "real" and generated samples as "fake" as much as possible; the generator, on the other hand, aims to generate realistic fake samples sufficient to "deceive" the discriminator, causing it to mistakenly classify them as "real." Through this continuous adversarial game, the generator's generation ability continuously improves, ultimately producing highly realistic data.

[0061] In step S250, the face image enhancement model to be trained is trained according to the loss function to obtain the target face image enhancement model.

[0062] In this embodiment, the face image enhancement model is trained according to the loss function. In each training iteration, the model receives a batch of training data, executes the forward process of steps S210 to S240 to obtain the loss value, then calculates the gradient of the loss relative to the model parameters, and updates the parameters along the gradient descent direction, so that the loss value gradually decreases. This process is repeated until the model performance converges on the validation set or reaches the preset number of training rounds. Through continuous training, the face image enhancement model can learn the difference between the output face image and the high-quality face image, thereby enhancing the robustness and adaptability of the model, ensuring that the high-quality signal in the output image remains highly stable even in complex and ever-changing image processing tasks.

[0063] The portrait image enhancement model training method provided in this disclosure generates a corresponding face gradient image based on the face image to be enhanced; the face image enhancement model to be trained performs image enhancement processing on the face image to be enhanced based on the face gradient image to obtain an output face image; a loss function is generated based on the output face image and the high-quality face image; and the face image enhancement model to be trained is trained based on the loss function to obtain a target face image enhancement model. This method introduces the face gradient image as a control condition into the model, guiding the model to better maintain the structural consistency of the original image during the image enhancement process and improving the fidelity of the output image.

[0064] Figure 4 This is a flowchart illustrating a face gradient image generation method. Figure 5 This is a schematic diagram of the training process of a portrait image enhancement model, as illustrated in the example. Figure 2 .like Figure 4 , 5As shown in this embodiment, the face gradient image generation process in step S220 can include the following steps.

[0065] In step S410, the face image to be enhanced is subjected to image convolution processing to obtain a first face gradient image.

[0066] In this embodiment, by introducing the face gradient image as a control condition into the face image enhancement model, although it can enable the model to better maintain the consistency of the original image during the image enhancement process and improve the fidelity of the output image, it also inhibits the model's image generation capability and affects the image generation effect. Because different application scenarios may require different capabilities for different regions of the face image. For example, the facial features area has a relatively high fidelity requirement, needing to maintain consistency with the original image. However, areas such as hair do not require high fidelity and can be enhanced with richer, more natural textures. Therefore, it is necessary to retain gradient images differently for different regions of the face image, so that the model learns to adopt different generation capabilities and fidelity levels for different regions of the face image.

[0067] In this embodiment, a first face gradient image is obtained by performing image convolution processing on the face image to be enhanced. For example, a convolution kernel based on the Sobel operator or the Laplacian operator can be used. This first face gradient image is a gradient image with balanced gradients in all regions, without involving the preservation or overlay of gradient images for different regions.

[0068] In step S420, a corresponding face mask image is generated based on the face image to be enhanced.

[0069] In this embodiment of the disclosure, a corresponding face mask is generated based on the face image to be enhanced. This face mask is a weight map of the same size as the original image, with each pixel having a value (typically between 0 and 1). The weight values ​​in this face mask represent the mask intensity of the gradient image of the corresponding region.

[0070] In an exemplary embodiment, the face mask is a binary mask. The weight values ​​of each pixel in the mask determine whether a region is covered or uncovered. For example, a weight value of 1 indicates that the region is not covered, preserving its gradient image and ensuring high fidelity through enhanced gradient control. A weight value of 0 indicates that the region is covered, obscuring its gradient image and enhancing content generation in that region to improve image quality by suppressing gradient control.

[0071] In an exemplary embodiment, the weight value corresponding to each pixel of the face mask image corresponds to the mask intensity. This mask intensity is used to represent the degree of suppression of the gradient image. For example, a weight value of 0.5 indicates partial coverage of the region, that is, the mask intensity of the gradient image of the region is determined based on the weight value, thereby suppressing gradient control to a certain extent, so that the generation capability and fidelity capability of the region are appropriately balanced.

[0072] In step S430, the first face gradient image is masked based on the face mask image to obtain the face gradient image.

[0073] In this embodiment, the first face gradient image obtained in step S410 is combined with the face mask image generated in step S420, and the final face gradient image is obtained through masking processing. Masking processing is essentially a pixel-level weighted fusion operation. For example, each pixel value of the first face gradient image is multiplied by the corresponding weight value in the face mask image. After this operation, the gradient values ​​of regions with high weights in the mask image are preserved, while the gradient values ​​of regions with low weights are weakened or reduced to zero. Through the above masking processing, the gradient control information is modulated in a partitioned manner, transforming a unified global structure map into a gradient map with variable intensity. This face gradient image enables the model to learn to apply different generation capabilities and fidelity preservation capabilities to different regions of the face image, thereby achieving more refined control over the enhancement capabilities of the face image.

[0074] The portrait image enhancement model training method provided in this disclosure, through the aforementioned masking process, achieves partitioned modulation of gradient control information, transforming a unified global structure map into a gradient map with variable intensity. This facial gradient image enables the model to learn to apply different generation and fidelity capabilities to different regions of the facial image, thereby achieving more refined control over the enhancement capabilities of the facial image.

[0075] Figure 6 This is based on the flow of a face mask generation method illustrated in the example. Figure 1 .like Figure 6 As shown in this embodiment, the face gradient image generation process in step S420 can include the following steps.

[0076] In step S610, a first mask image is generated that completely covers the face image to be enhanced.

[0077] In this embodiment of the disclosure, at least two candidate mask images are generated based on the face image to be enhanced. One of these mask images is a first mask that completely covers the face image to be enhanced. That is, the weight values ​​of all pixel positions in this mask image are set to 0. Based on this first mask image, the entire region of the first face gradient image will be covered, and no related gradient control will be performed.

[0078] In step S620, a second mask map is generated that covers a portion of the face image to be enhanced.

[0079] In this embodiment of the disclosure, at least two candidate mask images are generated based on the face image to be enhanced. One of the mask images is a second mask that covers a portion of the face image to be enhanced. Specifically, the weight values ​​of some pixel positions in this mask image are set to 0, while the weight values ​​of other pixel positions are set to 1. Based on this second mask image, a portion of the first face gradient image is selectively covered, while other regions retain their corresponding gradient control.

[0080] In step S630, based on a preset selection probability, the face mask is randomly determined to be either the first mask or the second mask.

[0081] In this embodiment, a specific masking strategy is not used consistently during model training. Instead, based on a preset selection probability, a mask is randomly selected from two candidate masks (full coverage and partial coverage) to serve as the face mask used in the forward propagation. This random switching training mechanism allows the same model to alternate between "strong control" and "weak control" constraint environments in different training batches. This enables the model to learn to adaptively adjust its enhancement processing methods. Through this long-term, alternating adversarial training, the model gradually masters the ability to dynamically balance "fidelity" and "generation intensity" based on the strength of the input control information. Furthermore, by adjusting the selection probability, the type of face gradient image received by the model during training can be adjusted to meet the needs of different application scenarios.

[0082] Figure 7 This is based on the flow of a face mask generation method illustrated in the example. Figure 2 .like Figure 7 As shown in this embodiment, the face gradient image generation process in step S420 can include the following steps.

[0083] In step S710, the face image to be enhanced is analyzed using a face analysis model, and the face image to be enhanced is divided into at least one face region.

[0084] In this embodiment, a face analysis model is used to perform refined semantic segmentation on the face image to be enhanced, thereby dividing the image into several face regions with clear semantic meanings. These include, for example, the eyes, eyebrows, nose, mouth, ears, and hair. Furthermore, common accessories and environmental backgrounds surrounding the face can also be included, such as backgrounds, hats, earrings, and glasses. As mentioned earlier, due to visual habits, people have different visual sensitivities to different regions of a face image. For example, people are more sensitive to structural changes in facial features, thus requiring high fidelity in these areas. Conversely, people are less sensitive to hair, eyebrows, and accessories, thus requiring lower fidelity in these areas and improved texture details.

[0085] In an exemplary embodiment, the face resolution model is a model pre-built based on a neural network (such as a U2-Net network) and trained using a specific training data set. After training, the face resolution model parses face images, determines the face region category to which each pixel in the face image belongs, and labels each pixel with a face region value corresponding to that category. Each face region value corresponds one-to-one with a face region category.

[0086] For example, a face image is pre-defined into 16 face region categories, including: background, facial skin, right eyebrow, left eyebrow, right eye, left eye, glasses, right ear, left ear, earring, nose, mouth, upper lip, lower lip, hair, and hat. Each of these 16 face region categories is assigned a face region value from 0 to 15. During training, the face analysis model identifies and judges these 16 face regions in the face image using the training data set, and assigns the corresponding face region value to each pixel in the face image based on the judgment results.

[0087] It should be noted that, depending on the actual application needs, the above-mentioned face region categories may be enriched or simplified, and all such simplifications should be considered within the scope of protection of this disclosure.

[0088] In step S720, the face mask map is generated based on the at least one face region.

[0089] In this embodiment of the disclosure, based on the aforementioned division of the face image into multiple face regions, a differentiated masking strategy is determined according to the characteristics of each face region. Based on the masking strategy corresponding to each face region, a mask map for that face region is determined. For example, it may be an overlaid area, an uncovered area, or the mask intensity of that area. Based on the mask maps of each face region, a final mask map is generated that is the same size as the original image, but each pixel value represents the gradient control intensity at that location.

[0090] In an exemplary embodiment, independent region probabilities can be assigned according to the category of face regions, and the coverage of corresponding face regions can be controlled through region probabilities. Specifically, this may include the following steps: Based on the face region category corresponding to the face region, the region probability corresponding to the face region is determined; the region probability corresponds one-to-one with the face region. Based on the region probability, the corresponding face region is randomly determined to be a covered region or an uncovered region. The face mask map is generated based on each of the face regions.

[0091] In this embodiment, based on the aforementioned face parsing model, the face region category corresponding to each face region is identified, and a corresponding region probability is independently assigned to each face region. This region probability corresponds one-to-one with the face region. The region probability controls the probability of the face region being covered in the face mask image. During model training, based on this region probability, the corresponding face region is randomly determined to be covered or uncovered during the training process. Based on this, whether each region in the face image is covered is determined, thereby generating a complete face mask image. In this way, on the one hand, by alternating whether face regions are covered, the model can learn to adaptively adjust its enhancement processing method; on the other hand, by independently assigning the region probability of each face region, the model can adapt to the differentiated requirements of different types of face regions for fidelity and generation capabilities.

[0092] In an exemplary embodiment, scene recognition can be performed on the face image to be enhanced in advance to obtain scene category information. Then, based on the scene category information of the image, the region probability corresponding to each face region is determined. Since images of different scene categories have different requirements for fidelity and generation capabilities, for example, if the original image itself has low clarity, the model needs to improve its generation capability to compensate for the lack of clarity. Or, for example, if the original image is a close-up image of a person, it requires high facial fidelity, thus the model needs to improve fidelity and suppress generation capability. Therefore, by performing scene recognition on the face image to be enhanced and determining the required region probability for each face region corresponding to the scene category, the model's adaptability to different scene categories is improved.

[0093] In an exemplary embodiment, the region probabilities of each face region can be pre-configured to determine the corresponding region configuration information. During model training, the probability of a face region being covered is controlled based on the region probabilities corresponding to each face region in the region configuration information. This region configuration information can be automatically configured based on the scene or manually configured through interaction.

[0094] In an exemplary embodiment, independent region mask intensities can be assigned based on the category of the face region. These region mask intensities control the degree of gradient image suppression for the corresponding face region. Specifically, this may include the following steps: Based on the face region category corresponding to the face region, the region mask intensity corresponding to the face region is determined; the region mask intensity corresponds one-to-one with the face region. The face mask map is generated based on the region mask intensity corresponding to each of the face regions.

[0095] In this embodiment, the face region category corresponding to each face region is identified based on the aforementioned face parsing model, and a corresponding region mask intensity is independently assigned to each face region. This region mask intensity corresponds one-to-one with the face region. The gradient image suppression level of the corresponding face region is controlled by this region mask intensity. During model training, the differentiated gradient image suppression level of each face region is adjusted based on this region mask intensity, thereby generating a complete face mask map. In this way, by adjusting the gradient image suppression level of different face regions, the model can adapt to the differentiated requirements of fidelity and generation capabilities for different types of face regions.

[0096] In an exemplary embodiment, scene recognition can be performed on the face image to be enhanced in advance to obtain scene category information. Then, based on the scene category information of the image, the region mask intensity corresponding to each face region is determined. Since images of different scene categories have different requirements for fidelity and generation capabilities, for example, if the original image itself has low clarity, the model needs to improve its generation capability to compensate for the lack of clarity. Or, for example, if the original image is a close-up image of a person, it requires high facial fidelity, thus the model needs to improve fidelity and suppress generation capability. Therefore, by performing scene recognition on the face image to be enhanced and determining the required region mask intensity for each face region corresponding to the scene category, the model's adaptability to different scene categories is improved.

[0097] In an exemplary embodiment, the region mask intensity for each face region can be pre-configured to determine the corresponding region configuration information. During model training, the gradient image suppression level for the corresponding face region is controlled based on the region mask intensity corresponding to each face region in the region configuration information. This region configuration information can be automatically configured based on the scene or manually configured through interaction.

[0098] The portrait image enhancement model training method provided in this disclosure utilizes a face parsing model to perform refined semantic segmentation on the face image to be enhanced, thereby dividing the image into several face regions with clear semantic meanings. Differentiated masking strategies are determined based on the characteristics of each face region, thus determining the mask map for that face region. In this way, on the one hand, the model can learn to adaptively adjust its enhancement processing method; on the other hand, by independently assigning the region probability or mask intensity of each face region, the model can adapt to the differentiated requirements of different types of face regions for fidelity and generation capabilities.

[0099] Figure 8 This is a flowchart illustrating a face image enhancement method according to an exemplary embodiment. Figure 8 The method provided in the embodiments can be executed by any electronic device, such as the one described above. Figure 1 Terminal devices in or Figure 1 The server in, or Figure 1 The terminal devices and servers in the process are executed together, but this disclosure does not limit this.

[0100] In step S810, the image to be processed is acquired.

[0101] In this embodiment of the disclosure, a user's image enhancement request is obtained. This image enhancement request includes at least one image to be processed. The image to be processed is the input image to be subjected to image enhancement processing.

[0102] In step S820, the image to be processed is input into the portrait image enhancement model; the portrait image enhancement model is obtained by training through any of the aforementioned portrait image enhancement model training methods.

[0103] In step S830, the portrait image enhancement model performs image enhancement processing on the image to be processed to obtain the target image.

[0104] In this embodiment of the disclosure, the image to be processed in the image enhancement request is input into a portrait image enhancement model trained by any of the aforementioned portrait image enhancement model training methods. This portrait image enhancement model can perform image enhancement processing on the image to be processed based on the model parameters adjusted during the aforementioned training process. Because the robustness and adaptability of the model are enhanced during training, the output image can still maintain good stability of high-quality signals even in complex and ever-changing image processing tasks.

[0105] The portrait image enhancement method provided in this disclosure includes: acquiring an image to be processed; inputting the image to be processed into a face image enhancement model; and performing image enhancement processing on the image to be processed by the face image enhancement model to obtain a target image. This method introduces the face gradient image as a control condition into the model, guiding the model to better maintain the structural consistency of the original image during the image enhancement process and improving the fidelity of the output image.

[0106] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0107] Figure 9 This is a block diagram illustrating a portrait image enhancement model training apparatus according to an exemplary embodiment. (Refer to...) Figure 9 The device 900 may include: a sample acquisition module 910, a gradient image module 920, an image enhancement module 930, a loss function module 940, and a model training module 950.

[0108] The sample acquisition module 910 is configured to acquire a training dataset including multiple image training samples; the image training samples include corresponding face images to be enhanced and high-quality face images.

[0109] The gradient image module 920 is configured to generate a corresponding face gradient image based on the face image to be enhanced.

[0110] The image enhancement module 930 is configured to perform image enhancement processing on the face image to be enhanced based on the face gradient image using the face image enhancement model to be trained, and obtain an output face image.

[0111] The loss function module 940 is configured to generate a loss function based on the output face image and the high-quality face image.

[0112] The model training module 950 is configured to train the face image enhancement model to be trained according to the loss function to obtain the target face image enhancement model.

[0113] In some exemplary embodiments of this disclosure, the image enhancement module 930 is further configured to encode the face image to be enhanced based on the face gradient image using an encoder to obtain a face feature vector to be enhanced; quantize the face feature vector to be enhanced using a quantization codebook to obtain a quantized face feature vector; and decode the quantized face feature vector based on the face gradient image using a decoder to obtain the output face image.

[0114] In some exemplary embodiments of this disclosure, the gradient image module 920 is further configured to perform image convolution processing on the face image to be enhanced to obtain a first face gradient image; generate a corresponding face mask image based on the face image to be enhanced; and perform mask processing on the first face gradient image based on the face mask image to obtain the face gradient image.

[0115] In some exemplary embodiments of this disclosure, the gradient image module 920 is further configured to generate a first mask that completely covers the face image to be enhanced; generate a second mask that partially covers the face image to be enhanced; and randomly determine the face mask as the first mask or the second mask based on a preset selection probability.

[0116] In some exemplary embodiments of this disclosure, the gradient image module 920 is further configured to parse the face image to be enhanced using a face parsing model, divide the face image to be enhanced into at least one face region, and generate the face mask map based on the at least one face region.

[0117] In some exemplary embodiments of this disclosure, the gradient image module 920 is further configured to determine the region probability corresponding to the face region based on the face region category corresponding to the face region; the region probability corresponds one-to-one with the face region; based on the region probability, the corresponding face region is randomly determined to be a covered region or an uncovered region; and the face mask map is generated based on each face region.

[0118] In some exemplary embodiments of this disclosure, the gradient image module 920 is further configured to determine the region mask intensity corresponding to the face region based on the face region category corresponding to the face region; the region mask intensity corresponds one-to-one with the face region; and generate the face mask map based on the region mask intensity corresponding to each face region.

[0119] In some exemplary embodiments of this disclosure, the gradient image module 920 is further configured to perform scene recognition on the face image to be enhanced to obtain scene category information; and determine the region probability or region mask intensity corresponding to each face region based on the scene category information.

[0120] In some exemplary embodiments of this disclosure, the gradient image module 920 is further configured to acquire region configuration information and determine the region probability or region mask intensity corresponding to each of the face regions based on the region configuration information.

[0121] Figure 10 This is a block diagram illustrating a portrait image enhancement device according to an exemplary embodiment. (Refer to...) Figure 10The device 1000 may include: an image acquisition module 1010, an image input module 1020, and an image processing module 1030.

[0122] Image acquisition module 1010 is configured to acquire the image to be processed.

[0123] The image input module 1020 is configured to input the image to be processed into a portrait image enhancement model; the portrait image enhancement model is obtained by training using any of the portrait image enhancement model training methods described above.

[0124] The image processing module 1030 is configured to perform image enhancement processing on the image to be processed by the portrait image enhancement model to obtain the target image.

[0125] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0126] The following reference Figure 11 To describe an electronic device 1100 according to such an embodiment of the present disclosure. Figure 11 The electronic device 1100 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0127] like Figure 11 As shown, the electronic device 1100 is manifested in the form of a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to: at least one processing unit 1110, at least one storage unit 1120, a bus 1130 connecting different system components (including storage unit 1120 and processing unit 1110), and a display unit 1140.

[0128] The storage unit stores program code, which can be executed by the processing unit 1110 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1110 can perform actions such as... Figure 2 The steps shown.

[0129] For example, electronic devices can achieve such Figure 2 or Figure 8 The steps shown.

[0130] Storage unit 1120 may include readable media in the form of volatile storage units, such as random access memory (RAM) 1121 and / or cache memory 1122, and may further include read-only memory (ROM) 1123.

[0131] Storage unit 1120 may also include a program / utility 1124 having a set (at least one) program module 1125, such program module 1125 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0132] Bus 1130 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0133] Electronic device 1100 can also communicate with one or more external devices 1170 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1100, and / or any device that enables electronic device 1100 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1150. Furthermore, electronic device 1100 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1160. As shown, network adapter 1160 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0134] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0135] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions that can be executed by a processor of the device to perform the described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0136] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods described above.

[0137] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0138] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for training a face image enhancement model, characterized in that, include: Obtain a training dataset that includes multiple image training samples; The image training samples include corresponding face images to be enhanced and high-quality face images; Generate a corresponding face gradient image based on the face image to be enhanced; The face image to be enhanced is processed by the face gradient image based on the face image enhancement model to be trained, and an output face image is obtained. A loss function is generated based on the output face image and the high-quality face image; The face image enhancement model to be trained is trained according to the loss function to obtain the target face image enhancement model.

2. The method according to claim 1, characterized in that, The step of performing image enhancement processing on the face image to be enhanced based on the face gradient image using the face image enhancement model to be trained, to obtain an output face image, includes: The face image to be enhanced is encoded by an encoder based on the face gradient image to obtain the face feature vector to be enhanced. The face feature vector to be enhanced is quantized using a quantization codebook to obtain a quantized face feature vector. The quantized facial feature vector is decoded based on the facial gradient image by a decoder to obtain the output facial image.

3. The method according to claim 1, characterized in that, The step of generating a corresponding face gradient image based on the face image to be enhanced includes: The face image to be enhanced is subjected to image convolution processing to obtain a first face gradient image; The corresponding face mask image is generated based on the face image to be enhanced; The first face gradient image is masked based on the face mask image to obtain the face gradient image.

4. The method according to claim 3, characterized in that, The step of generating a corresponding face mask image based on the face image to be enhanced includes: Generate a first mask that completely covers the face image to be enhanced; Generate a second mask map that covers a portion of the face image to be enhanced; Based on a preset selection probability, the face mask is randomly determined to be either the first mask or the second mask.

5. The method according to claim 3, characterized in that, The step of generating a corresponding face mask image based on the face image to be enhanced includes: The face image to be enhanced is analyzed using a face analysis model, and the face image to be enhanced is divided into at least one face region; The face mask map is generated based on the at least one face region.

6. The method according to claim 5, characterized in that, Generating the face mask map based on the at least one face region includes: Based on the face region category corresponding to the face region, the region probability corresponding to the face region is determined; the region probability corresponds one-to-one with the face region. Based on the region probability, the corresponding face region is randomly determined to be a covered region or an uncovered region. The face mask map is generated based on each of the face regions.

7. The method according to claim 5, characterized in that, Generating the face mask map based on the at least one face region includes: Based on the face region category corresponding to the face region, the region mask intensity corresponding to the face region is determined; the region mask intensity corresponds one-to-one with the face region. The face mask map is generated based on the region mask intensity corresponding to each of the face regions.

8. The method according to claim 6 or 7, characterized in that, The method further includes: Scene recognition is performed on the face image to be enhanced to obtain scene category information; Based on the scene category information, determine the region probability or region mask intensity corresponding to each of the face regions.

9. The method according to claim 6 or 7, characterized in that, The method further includes: Get region configuration information; Based on the region configuration information, the region probability or region mask intensity corresponding to each face region is determined.

10. A method for enhancing facial images, characterized in that, include: Obtain the image to be processed; The image to be processed is input into the face image enhancement model; the face image enhancement model is obtained by training using the face image enhancement model training method as described in any one of claims 1 to 9; The face image enhancement model performs image enhancement processing on the image to be processed to obtain the target image.

11. A face image enhancement model training device, characterized in that, include: The sample acquisition module is configured to acquire a training dataset that includes multiple image training samples; The image training samples include corresponding face images to be enhanced and high-quality face images; The gradient image module is configured to generate a corresponding face gradient image based on the face image to be enhanced; The image enhancement module is configured to perform image enhancement processing on the face image to be enhanced based on the face gradient image using the face image enhancement model to be trained, and obtain an output face image; The loss function module is configured to generate a loss function based on the output face image and the high-quality face image; The model training module is configured to train the face image enhancement model to be trained according to the loss function to obtain the target face image enhancement model.

12. A face image enhancement device, characterized in that, include: The image acquisition module is configured to acquire the image to be processed. An image input module is configured to input the image to be processed into a face image enhancement model; the face image enhancement model is obtained by training using the face image enhancement model training method as described in any one of claims 1 to 9. The image processing module is configured to perform image enhancement processing on the image to be processed by the face image enhancement model to obtain the target image.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the executable instructions to implement the face image enhancement model training method as described in any one of claims 1 to 9 or the face image enhancement method as described in claim 10.

14. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the face image enhancement model training method as claimed in any one of claims 1 to 9 or the face image enhancement method as claimed in claim 10.

15. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the face image enhancement model training method as described in any one of claims 1 to 9 or the face image enhancement method as described in claim 10.

Images