Unsupervised enhancement method for low-quality face images and storage medium

Abstract

The application discloses a kind of low-quality face image's unattended enhancement method and storage medium, belong to image enhancement technical field, the present application is by constructing enhancement model, the face image information and illumination state are parsed, the brightness enhancement of non-uniform face image is realized, enhancement model is based on deep learning and neural network technology, can automatically learn the feature and enhancement method of face image, improve the quality of image, then by using the discriminant model of generative adversarial network model is constructed, the output of enhancement model is handled in combination with constraint function, to keep the real details and natural image state of enhancement result, and effectively constrain enhancement result, the face feature information in the image of the present application is fully exploited, and the natural exposure face state is maintained, so that the face information of low exposure area is close to natural state, and the clear and fine feature texture is better expressed, so as to effectively improve the reconstruction performance of model image.

Description

Technical Field

[0001] This invention belongs to the field of image enhancement technology, specifically relating to an unsupervised enhancement method and storage medium for low-quality face images. Background Technology

[0002] As times change, we are transitioning from a materialistic society to a diversified, information-driven era, where images have become an indispensable part, especially facial images, which hold significant commemorative value. However, in everyday indoor photography, uncontrollable environmental factors such as weather and lighting affect image quality, resulting in diverse non-uniform exposures in facial images, leading to overall low quality and poor visual effects. Image enhancement for non-uniform exposure is a challenging task because it is difficult to simultaneously maintain uniform brightness and a natural appearance under non-uniform exposure conditions.

[0003] Existing low-light image enhancement algorithms have some problems when dealing with images in non-uniform lighting. Due to the lack of image information about normal exposure, existing algorithms often enhance the contrast of the entire image, resulting in over-enhancing / over-exposing bright and sharp areas, rather than flexibly improving the overall image quality, and often causing uneven exposure. Summary of the Invention

[0004] To address the problems mentioned in the background section, this invention provides an unsupervised enhancement method and storage medium for low-quality face images, thereby solving the problem that existing technologies cannot flexibly improve overall image quality and often result in uneven exposure.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] An unsupervised enhancement method for low-quality face images includes the following steps:

[0007] S1: Construct a dataset of face images under non-uniform lighting;

[0008] S2: Construct an augmentation model, which includes a VGG network, a first U-Net network, two second U-Net networks, and a third U-Net network;

[0009] S3: Transfer the face images I from the dataset in The features are sequentially input into the VGG network and the first U-Net network for feature extraction to obtain the first feature.

[0010] S4: Input the first feature into two second U-Net networks simultaneously to perform image decomposition and obtain face information a0 and illumination information S0;

[0011] S5: Using facial information a0, illumination information S0, and facial image I in The residual data res0 is calculated, and then the residual data res0 is compared with the face image I. in Perform feature concatenation to obtain the concatenation result;

[0012] S6; The stitching result is fed into the third U-Net network to obtain the face image I0;

[0013] S7: Use the face image I0 as the input to the VGG network, repeat S3-S6, and obtain face information a1, illumination information S1, residual data res1 and face image I1;

[0014] S8: Construct the loss function loss0 based on face image I1 and face image I0:

[0015] ;

[0016] The loss function loss0 is used as a constraint on the unsupervised augmentation model to make the augmentation model converge and obtain the trained model.

[0017] S9: Input the non-uniformly lit face image into the trained model to obtain the enhanced image, construct a discriminative network, and input the enhanced image into the discriminative network to obtain the final image;

[0018] The discriminant network is used to obtain face images with uniform illumination. h Constrained representation of the state and details of the enhanced image.

[0019] Preferably, in S4, the first feature is represented as a0*S0.

[0020] Preferably, in S5, res0=I in -a0*S0, the residual data res0 is used as a compensation parameter to compensate for the information bias caused by non-uniform face images during the decomposition process.

[0021] Preferably, the discriminant network in S9 is a VGG network structure, which processes the face image I... h The enhanced image and the image are respectively input into the discriminative network to generate corresponding features f. h And f1, and construct the loss function loss1 to measure the feature f. h Consistency with f1.

[0022] Preferably, the first U-Net network, the second U-Net network, and the third U-Net network are all composed of eight symmetrical convolutional modules, each of which includes cascaded convolutional layers and a max pooling layer or a deconvolutional layer.

[0023] A computer-readable storage medium includes computer instructions for instructing a computer to perform an unsupervised augmentation method for a low-quality face image.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] This application constructs an enhancement model to analyze facial image information and illumination conditions, achieving brightness enhancement of non-uniform facial images. The enhancement model, based on deep learning and neural network technology, can automatically learn facial image features and enhancement methods, improving image quality. Then, a discriminative model is constructed using a generative adversarial network model, and the output of the enhancement model is processed by a constraint function to maintain the realistic details and natural image state of the enhancement result, and to effectively constrain the enhancement result. This application fully mines the facial feature information in the image and maintains the natural exposure facial state, making the facial information in low-exposure areas close to the natural state and better expressing clear and fine feature textures, thereby effectively improving the reconstruction performance of the model image. Attached Figure Description

[0026] Figure 1 To enhance the structural diagram of the model;

[0027] Figure 2 A schematic diagram to identify the network structure;

[0028] Figure 3 This is a schematic diagram of the U-Net network structure. Detailed Implementation

[0029] To facilitate understanding of the technical content of this invention by those skilled in the art, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the scope of the invention.

[0030] Example 1:

[0031] This embodiment uses one NVIDIA GTX3060ti GPU for the experiment. The training data is collected from the Internet, and the dataset includes more than 5,000 low-quality non-uniform face images, covering image data of various poses, face scales and background information. The image resolution is set to 256x256. In this embodiment, 3,000 faces are randomly selected for unsupervised face enhancement. 2,000, 100 and 900 images are used as training set, validation set and test sample data, respectively. Since the performance of deep learning face enhancement methods is related to the similarity between test images and training images, additional tests are performed on the actual non-uniform face images to ensure robustness and generalization.

[0032] like Figure 2 As shown, an unsupervised enhancement method for low-quality face images includes the following steps:

[0033] Step 1: Pre-training a high-quality network for efficient decomposition of face image information. This step uses a network framework for image parsing and feature extraction, and includes the following sub-steps:

[0034] Step 1.1: Randomly select face images of different poses and face scales from the high-quality dataset and divide them into training, validation, and test sets. These images contain face images and their corresponding feature data, including the face itself and facial information affected by lighting.

[0035] Step 1.2: Use a VGG network and a U-net network to extract depth information features from the input low-quality face image to obtain the features of the low-quality face image.

[0036] Step 1.3: Extract the generated image features and input them into two U-Net networks to perform image decomposition and generate corresponding feature information for the training data.

[0037] like Figure 3 As shown, each U-Net network consists of eight symmetrical convolutional modules. Each convolutional module includes cascaded convolutional layers and a max-pooling layer or deconvolutional layer to decompose image information. Step 1 aims to pre-train the feature decomposition module as an initial network model for low-quality, non-uniform brightness face images.

[0038] Step 2: Feature extraction for low-quality face images. This step uses a residual network for image parsing and feature extraction, and includes the following sub-steps:

[0039] Step 2.1: Randomly select face images with different poses, face scales, and brightness states from the actual non-uniform brightness dataset, and divide them into training and test sets.

[0040] Step 2.2: Use a pre-trained network to process the input low-quality face image I in Feature extraction of depth information is performed to obtain the features of low-quality face images.

[0041] Step 2.3: Extract the generated image features and input them into two U-Net networks to perform image decomposition, obtaining the face information a0 and the face information S0 affected by illumination.

[0042] Step 3: Fuse and further express the information a0 and S0 obtained from the decomposition of low-quality face images.

[0043] This step specifically includes the following sub-steps:

[0044] Step 3.1: The face information a0 and the face information S0 affected by illumination are initialized from the input face image I through the model initialization process. in The decomposition is performed, the decomposition results are multiplied, and then subtracted from the input image to obtain the residual map res0=I. in -a0*S0. Where res0 is used to measure the lighting conditions of the face image and to guide the enhancement of the face image.

[0045] Step 3.2: Combine the obtained compensation parameters res0 and the input image I in The images are concatenated and then input into the U-Net network to generate an enhanced face image result I0. By incorporating the data state of the compensation parameters, the expression of non-uniform brightness in the input image is enhanced.

[0046] Step 4: The face image I0 generated by the enhancement model is used as input and fed into the pre-trained model framework to generate the corresponding face information a1 and face information S1 affected by illumination. By repeating the enhancement result expression in Step 3, iterative compensation parameters res1 and enhancement result I1 are generated respectively. According to image attributes, when the input and output compensation parameters are close to 0, the brightness of the face image becomes more uniform and the image quality is improved. Therefore, a loss function is constructed. As a constraint for unsupervised augmentation models.

[0047] Step 5: As Figure 2 As shown, a discriminative network is constructed, which is then combined with actual high-quality, natural, and uniform face images I. h The overall state and details of the enhanced face image I1 are constrained and expressed. The face image I... h The enhanced result I1 is input into the VGG network to generate the corresponding feature f. h We calculate f1 and construct a loss function to measure the consistency of features between the two, in order to maintain the image state of the augmented result.

[0048] Step 6, Image performance evaluation for face enhancement:

[0049] Step 6.1: The image performance of face enhancement is tested using the test set obtained in Step 2.1. The test data mainly includes actual low-quality face images. Due to the lack of high-quality face images, this technique uses non-reference image quality evaluation metrics such as BRISQUE, PIQE, CEIQ, and LOE as objective evaluation standards for the enhancement effect.

[0050] Step 6.2 compares the non-uniform brightness enhancement algorithms NPE, SIRE, EnlightenGAN, and Zero-DCE. After sufficient training, the corresponding test sets are tested, and objective metrics are compared. Tests show that the experimental results of this invention result in clearer facial details and less interference noise.

[0051] As shown in Table 1, it outperforms other comparison algorithms in objective metrics such as BRISQUE, PIQE, CEIQ, and LOE.

[0052] Table 1. Objective Indicators of Face Enhancement Algorithms

[0053]

[0054] As can be seen, the enhancement model proposed in this application for non-uniform low-brightness and low-quality face images can significantly represent low-quality face images and can achieve the reconstruction of non-uniform low-brightness and low-quality face images.

[0055] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0056] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. An unsupervised enhancement method for low-quality face images, characterized in that, Includes the following steps: S1: Construct a dataset of face images under non-uniform lighting; S2: Construct at least one augmentation model, which includes a VGG network, a first U-Net network, two second U-Net networks, and a third U-Net network; S3: Extract face images from the dataset I in The first feature is obtained by sequentially inputting the VGG network and the first U-Net network in an augmentation model for feature extraction. S4: Input the first feature into two second U-Net networks simultaneously to perform image decomposition and obtain face information. a 0 and lighting information S 0 ; S5: Through facial information a 0 Illumination information S 0 and facial images I in The residual data were calculated. res 0 Then the residual data res 0 With facial images I in Perform feature concatenation to obtain the concatenation result; S6; will The stitched result is fed into the third U-Net network to obtain the face image. I 0 ; S7: Transfer facial image I 0 As input to the VGG network, S3-S6 are repeated to obtain face information. a 1 Illumination information S 1 residual data res 1 and facial images I 1 ; S8: Based on facial image I 1 and facial images I 0 Constructing the loss function loss 0 : ； loss function loss 0 Used as a constraint for unsupervised augmentation models to make the augmentation models converge and obtain the trained models; S9: Input the non-uniformly lit face image into the trained model to obtain the enhanced image, construct a discriminative network, and input the enhanced image into the discriminative network to obtain the final image; The discriminant network is used to obtain face images with uniform illumination. I h Constrained representation of the state and details of the enhanced image; In S4, the first feature is represented as a 0 * S 0 ; In S5, res 0 = I in - a 0 * S 0 residual data res 0 It is used as a compensation parameter to compensate for information deviations caused by non-uniform face images during the decomposition process.

2. The unsupervised enhancement method for low-quality face images according to claim 1, characterized in that, The discriminant network in S9 uses a VGG network structure to analyze face images. I h The enhanced image and the image are respectively input into the discriminative network to generate corresponding features. f h and f 1 And construct the loss function. loss 1 Measuring features f h and f 1 Consistency.

3. The unsupervised enhancement method for low-quality face images according to claim 1, characterized in that, The first, second, and third U-Net networks each consist of eight symmetrical convolutional modules. Each convolutional module includes cascaded convolutional layers and a max-pooling layer or a deconvolutional layer.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions for instructing a computer to perform an unsupervised enhancement method for a low-quality face image as described in any one of claims 1-3.

Images