Face image restoration method based on semantic features

Abstract

Disclosed in the present invention is a face image restoration method based on semantic features. The method comprises the following steps: acquiring a face image to be restored; and inputting said face image into a trained restoration model to obtain a restored face image. The restoration model comprises: an encoder, used for generating low-quality face semantic features on the basis of a face image to be restored; a reference image generator, used for generating, on the basis of the low-quality face semantic features, a plurality of high-quality face reference images by inputting random noise; a feature transfer device, used for constructing a face component feature dictionary on the basis of the plurality of high-quality face reference images, and transferring the high-quality component features in the face component feature dictionary into the low-quality face semantic features to obtain high-quality face semantic features; and a decoder, used for generating a restored face image on the basis of the high-quality face semantic features. The consistency of face semantic information is ensured while the details of a restoration result and the texture are ensured, and a high evaluation index score and a high-quality visual effect can be obtained in a real scene.

Description

A face image restoration method based on semantic features Technical Field

[0001] The present invention relates to a facial image restoration method based on semantic features, and belongs to the technical field of image processing. Background Art

[0002] With the advancement of technology and improvements in living standards, image and video data is increasing exponentially, including a vast amount of human-themed images and videos. Furthermore, applications such as facial recognition, face matching, and face tracking in drones and surveillance cameras are constantly emerging. However, due to imaging device instabilities, such as focus failure and camera shake; imaging environmental factors, such as low light, high exposure, and subject motion; and network transmission conditions, such as compression, scaling, and codec formats, images can suffer from various types and degrees of image degradation, such as blur and noise. This is particularly true for facial images, where complex degradation not only reduces human perception but also poses serious risks for tasks involving the safety of people and property, such as facial access control, payment, and security monitoring. Therefore, facial image restoration aims to restore clear facial images from degraded images to aid complex facial visual tasks, such as face detection and recognition. Compared to natural images, facial images contain not only visually perceptible details but also individual facial features and identity information.

[0003] Currently, face restoration methods can be divided into three main categories: methods based on geometric priors, methods based on references, and methods based on generative priors. Existing methods based on geometric priors often struggle to capture effective geometric information from low-quality face images; existing reference-based methods typically pre-build a dictionary with a certain capacity, resulting in low generalization performance when faced with real degraded face images; and existing methods based on generative priors ignore the face's identity information when performing restoration operations.

[0004] Most of the above methods only focus on the restoration of facial geometric structure and detailed texture, but ignore the mining and preservation of semantic information, resulting in the generated facial images looking very real, but their basic features have changed.

[0005] Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a facial image restoration method based on semantic features, a reference image generator and a feature transferor based on the restoration model, which can improve the problem of existing methods ignoring semantic information, ensure the consistency of facial semantic information while ensuring the details and texture of the restoration results, and achieve higher evaluation index scores and high-quality visualization effects in real scenarios.

[0007] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0008] The present invention discloses a facial image restoration method based on semantic features, comprising the following steps:

[0009] Obtaining a face image to be restored;

[0010] Inputting the facial image to be restored into a trained restoration model to obtain a restored facial image;

[0011] The restoration model includes:

[0012] An encoder, configured to generate low-quality facial semantic features based on the facial image to be restored;

[0013] A reference image generator, configured to generate a plurality of high-quality face reference images by inputting random noise according to the low-quality face semantic features;

[0014] A feature transferor is configured to construct a face component feature dictionary based on the plurality of high-quality face reference images; and transfer the high-quality component features in the face component feature dictionary to the low-quality face semantic features to obtain high-quality face semantic features;

[0015] The decoder is used to generate a restored face image based on high-quality facial semantic features.

[0016] Furthermore, the encoder includes 5 residual convolution units and 4 2x downsampling units, and the residual convolution units and the 2x downsampling units are alternately connected in series.

[0017] Furthermore, the reference image generator includes a first generator module, a second generator module and a third generator module connected in series in sequence;

[0018] The first generator module includes a semantic feature fusion unit, a residual convolution unit and a 2x upsampling unit connected in series;

[0019] The second generator module includes a semantic feature fusion unit, a residual convolution unit and a 2x upsampling unit connected in series;

[0020] The third generator module includes a semantic feature fusion unit and a residual convolution unit connected in series.

[0021] Furthermore, the semantic feature fusion unit includes one normalization layer and two convolutional layers connected in series; the semantic feature fusion unit is used to fuse low-quality facial semantic features and random noise.

[0022] Furthermore, the feature transferor includes a dictionary construction module and a feature transfer module;

[0023] The dictionary construction module includes a facial feature extraction unit, a first facial component detection unit and a dictionary construction unit connected in series;

[0024] The facial feature extraction unit includes a plurality of residual convolution units and 2x downsampling units alternately connected in series, and is used to obtain a plurality of high-quality facial reference features based on the plurality of high-quality facial reference images;

[0025] The first face component detection unit is configured to obtain high-quality component features of different component categories based on the multiple high-quality face reference features;

[0026] The dictionary construction unit is used to construct a face component feature dictionary based on the high-quality component features of the different component categories;

[0027] The feature transfer module includes a second face component detection unit, a dictionary lookup unit and a feature fusion unit connected in series.

[0028] The second facial component detection unit is configured to obtain low-quality component features of corresponding component categories based on the low-quality facial semantic features;

[0029] The dictionary search unit is configured to obtain high-quality component features of corresponding component categories based on the low-quality component features and the constructed face component feature dictionary;

[0030] The feature fusion unit is used to fuse the low-quality facial semantic features with the high-quality component features of the corresponding component category to obtain high-quality facial semantic features.

[0031] Furthermore, the decoder includes 5 residual convolution units and 4 2x upsampling units, and the residual convolution units and the 2x upsampling units are alternately connected in series.

[0032] Furthermore, the training method of the restoration model includes the following steps:

[0033] Obtain a training set, the training set including the face training image to be restored and the corresponding real face restoration image,

[0034] Inputting the face training image to be restored into a pre-built restoration model to obtain a restored face training image;

[0035] Calculating a restoration model loss function based on the restored face training image and the corresponding real face restoration image;

[0036] The restoration model is iteratively updated and trained based on the gradient descent method, and the restoration model with the minimum restoration model loss function is used as the trained restoration model.

[0037] Furthermore, obtaining a training set includes the following steps:

[0038] Acquire a high-quality face image, and use the high-quality face image as a true face restoration image;

[0039] Performing pixel adjustment on the high-quality face image to obtain a degraded face image; using the degraded face image as a face training image to be restored;

[0040] The expression of the pixel adjustment operation is as follows: lq ={JPEG q ((I hq *k σ )↓ s +n δ )}↑ s

[0041] Where, I lq Represents the degraded face image, that is, the face training image to be restored; JPEG q Indicates JPEG compression with compression quality q; I hq represents a high-quality face image, i.e., a true restored face image; * represents a convolution operation; k σ represents the blur kernel with sigma = σ;↓ S Indicates downsampling s times operation; n δ Indicates Gaussian noise with sigma = δ; ↑ s Indicates an upsampling operation by a factor of s.

[0042] Furthermore, the restoration model loss function is expressed as follows: L = L l1 +λ per L per +λ adv L adv

[0043] Where, L represents the loss function value of the restoration model; L l1 represents the L1 loss value; λ per represents the perceptual loss weight; L per Represents the perceptual loss value based on the VGG network; λ adv represents the adversarial loss weight; L adv Represents the adversarial loss value based on adversarial training.

[0044] Furthermore, the L1 loss value L l1 The expression is as follows:

[0045] Where, I hq Represents the true restored image of the face; represents the restored face training image; |·|1 represents the mean absolute error;

[0046] The perceptual loss value L based on the VGG network per The expression is as follows:

[0047] Where, Represents the feature maps of the 1st to 5th convolutional layers in the pre-trained VGG model; represents the square of the 2-norm;

[0048] The adversarial loss value L based on adversarial training adv The expression is as follows:

[0049] Where D(·) represents the output of the discriminator in adversarial training; Indicates about The expected value of the distribution; softplus represents the softplus function, and the expression of the softplus function is: softplus(x)=ln(1+e x ).

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The face image restoration method based on semantic features of the present invention first generates a plurality of high-quality face reference images by inputting random noise and taking semantic information as guidance based on a reference image generator based on a restoration model; then, based on a feature transferor based on the restoration model, a semantically guided lightweight face component feature dictionary that is easy to quickly search is constructed, which can improve the problem of existing methods ignoring semantic information, enhance the generalization performance of the face restoration model, ensure the consistency of facial semantic information while ensuring the details and texture of the restoration results, and achieve higher evaluation index scores and high-quality visualization effects in real scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] FIG1 is a flow chart of a method for restoring a face image based on semantic features provided by an embodiment;

[0053] FIG2 is a schematic structural diagram of a restoration model provided in an embodiment;

[0054] FIG3 is a schematic structural diagram of an encoder provided in an embodiment;

[0055] FIG4 is a schematic structural diagram of a reference image generator provided by an embodiment;

[0056] FIG5 is a schematic structural diagram of a feature transferor provided in an embodiment;

[0057] FIG6 is a schematic structural diagram of a decoder provided in an embodiment;

[0058] FIG7 is a schematic diagram of the structure of the residual convolution unit provided in an embodiment. DETAILED DESCRIPTION

[0059] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0060] The present invention discloses a facial image restoration method based on semantic features, as shown in FIG1 and FIG2 , comprising the following steps:

[0061] Obtaining a face image to be restored;

[0062] Input the face image to be restored into the trained restoration model to obtain the restored face image;

[0063] The restoration model includes:

[0064] An encoder, used to generate low-quality facial semantic features based on the face image to be restored;

[0065] A reference image generator is used to generate multiple high-quality face reference images based on low-quality facial semantic features by inputting random noise;

[0066] A feature transferor is used to construct a face component feature dictionary based on multiple high-quality face reference images; transfer the high-quality component features in the face component feature dictionary to low-quality face semantic features to obtain high-quality face semantic features;

[0067] The decoder is used to generate a restored face image based on high-quality facial semantic features.

[0068] The technical concept of the present invention is as follows: a reference image generator based on a restoration model generates multiple high-quality face reference images by inputting random noise and using semantic information as a guide; a feature transferor based on the restoration model constructs a lightweight and fast-searchable face component feature dictionary, which can improve the problem of existing methods ignoring semantic information, enhance the generalization performance of the face restoration model, and achieve higher evaluation index scores and high-quality visualization effects in real scenarios.

[0069] As shown in Figure 3, the encoder includes 5 residual convolution units and 4 2x downsampling units, and the residual convolution units and the 2x downsampling units are alternately connected in series.

[0070] As shown in FIG4 , the reference image generator includes a first generator module, a second generator module, and a third generator module connected in series in sequence;

[0071] The first generator module includes a semantic feature fusion unit, a residual convolution unit, and a 2x upsampling unit connected in series.

[0072] The second generator module includes a semantic feature fusion unit, a residual convolution unit, and a 2x upsampling unit connected in series.

[0073] The third generator module includes a semantic feature fusion unit and a residual convolution unit connected in series.

[0074] Among them, the semantic feature fusion unit contains one normalization layer and two convolutional layers connected in series; the normalization layer adopts layer normalization (Layer Norm, LN).

[0075] The semantic feature fusion unit is used to fuse low-quality facial semantic features with random noise. Specifically, multiple sets of Gaussian noise generated based on random seeds are fused with the low-quality facial semantic features in the semantic feature fusion unit. This multi-scale, multi-stage fusion method is used to generate multiple high-quality facial reference images with the same semantic information but different expressions.

[0076] As shown in Figure 5, the feature transfer module includes a dictionary construction module and a feature transfer module.

[0077] The dictionary construction module is used to construct a face component feature dictionary that conforms to the current semantic information based on the multiple high-quality face reference images generated above, and includes a face feature extraction unit, a first face component detection unit, and a dictionary construction unit connected in series in sequence;

[0078] The facial feature extraction unit includes multiple residual convolution units and 2x downsampling units that are alternately connected in series, and is used to obtain multiple high-quality facial reference features based on multiple high-quality facial reference images; specifically, each high-quality facial reference image is learned to obtain a high-quality facial reference feature.

[0079] The first face component detection unit is used to obtain high-quality component features of different component categories based on multiple high-quality face reference features.

[0080] The dictionary construction unit is used to construct a facial component feature dictionary based on high-quality component features of different component categories. Specifically, it consists of three residual convolution units and two 2x downsampling units connected in series.

[0081] The feature transfer module includes a second face component detection unit, a dictionary lookup unit and a feature fusion unit connected in series.

[0082] A second face component detection unit is used to obtain low-quality component features of corresponding component categories based on low-quality face semantic features;

[0083] A dictionary lookup unit is used to obtain high-quality component features of corresponding component categories based on low-quality component features and a constructed face component feature dictionary;

[0084] The feature fusion unit is used to fuse low-quality facial semantic features with high-quality component features of the corresponding component categories to obtain high-quality facial semantic features.

[0085] Specifically, the first face component detection unit and the second face component detection unit both use the public dlib library; the dictionary construction unit uses the existing K clustering method; the dictionary search unit calculates the similarity between each low-quality component feature to be searched and each item in the face component feature dictionary, and selects the high-quality component feature with the largest similarity value, where the similarity calculation uses Euclidean distance.

[0086] Existing dictionary-learning-based face restoration methods typically pre-construct dictionaries. This redundant dictionary requires significant storage and computational overhead during face restoration, resulting in high costs, and the initial capacity limits the model's generalization performance. Therefore, the present invention constructs a semantically guided, lightweight dictionary of facial component features, avoiding the issues of pre-constructed dictionaries. This facilitates search and ensures the consistency of facial semantic information while preserving the detail and texture of the restored results. This invention can achieve high evaluation scores and high-quality visualization in real-world scenarios.

[0087] As shown in Figure 6, the decoder includes 5 residual convolution units and 4 2x upsampling units, and the residual convolution units and the 2x upsampling units are alternately connected in series.

[0088] It should be noted that the residual convolution unit involved in this embodiment uses residual learning to fully mine feature information. As shown in Figure 7, it is specifically composed of two convolutional layers connected in series. There is also an activation layer connected in series in the two convolutional layers. The input features enter the two convolutional layers and the activation layer, and the features obtained after the input features enter the two convolutional layers and the activation layer are added to the input features as the unit output.

[0089] The 2x downsampling unit is a convolutional layer with a kernel size of 2 and a stride of 2, which reduces the size of the input features by a factor of 2. The 2x upsampling unit increases the size of the input features by a factor of 2. Specifically, it consists of a convolutional layer with a kernel size of 1 and a sub-pixel layer that upsamples by a factor of 2, connected in series.

[0090] The training method of the restoration model in this embodiment includes the following steps:

[0091] 1. Obtain a training set, which includes the face training image I to be restored lq And the corresponding face real restoration image I hq .

[0092] Specifically, obtaining a training set includes the following steps:

[0093] From the public face dataset, such as Flickr-Faces-Hight-Quality (Flickr-Faces-HQ, FFHQ) high-definition face dataset, high-quality face images are obtained and used as the real face restoration image I hq ;

[0094] Perform pixel adjustment on the high-quality face image to obtain a degraded face image; use the degraded face image as the face training image to be restored I lq .

[0095] Specifically, we extract each high-quality face image from the FFHQ dataset and adjust its length and width to 512 pixels to obtain the degraded face image. The expression of the pixel adjustment operation is as follows: hq ={JPEG q ((I hq *k σ )↓ s +n δ )}↑ s

[0096] Where, I lq Represents the degraded face image, that is, the face training image to be restored; JPEG q Indicates JPEG compression with compression quality q; I hp represents a high-quality face image, i.e., a true restored face image; * represents a convolution operation; k σ represents the blur kernel with sigma = σ;↓ s Indicates downsampling s times operation; n δ Indicates Gaussian noise with sigma = δ; ↑ s Indicates an upsampling operation by a factor of s.

[0097] The specific parameters of the pixel adjustment operation can be adjusted according to the actual image and are not limited here.

[0098] 2. The face training image to be restored I lq Input into the pre-built restoration model to obtain the restored face training image

[0099] 3. Training images based on restored faces And the corresponding face real restoration image I hq , calculate the restoration model loss function L.

[0100] The expression of the restoration model loss function is as follows: L = L l1 +λ per L per +λ adv L adv

[0101] Where, L represents the loss function value of the restoration model; L l1 represents the L1 loss value; λ per Represents the perceptual loss weight, in this embodiment, λ per The value is 0.1; L per Represents the perceptual loss value based on the VGG network; λ adv Represents the adversarial loss weight, in this embodiment, λ adv The value is 0.01; L abv Represents the adversarial loss value based on adversarial training.

[0102] L1 loss value L l1 The expression is as follows:

[0103] Where, I hq Represents the true restored image of the face; represents the restored face training image; |·|1 represents the mean absolute error.

[0104] Perceptual loss value L based on VGG network per The expression is as follows:

[0105] Where, Represents the feature maps of the 1st to 5th convolutional layers in the pre-trained VGG model; Represents the square of the 2-norm.

[0106] Adversarial loss value L based on adversarial training abv The expression is as follows:

[0107] Where D(·) represents the output of the discriminator in adversarial training; Indicates about The expected value of the distribution; softplus represents the softplus function, and the expression of the softplus function is: softplus(x)=ln(1+e x ).

[0108] 4. The restoration model is iteratively updated and trained based on the gradient descent method, and the restoration model with the minimum restoration model loss function is used as the trained restoration model.

[0109] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0110] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0111] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0113] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A facial image restoration method based on semantic features, characterized in that: The steps include: Obtaining a face image to be restored; Inputting the facial image to be restored into a trained restoration model to obtain a restored facial image; The restoration model includes: An encoder, configured to generate low-quality facial semantic features based on the facial image to be restored; A reference image generator, configured to generate a plurality of high-quality face reference images by inputting random noise according to the low-quality face semantic features; A feature transferor is configured to construct a face component feature dictionary based on the plurality of high-quality face reference images; and transfer the high-quality component features in the face component feature dictionary to the low-quality face semantic features to obtain high-quality face semantic features; The decoder is used to generate a restored face image based on high-quality facial semantic features.

2. The method for facial image restoration based on semantic features according to claim 1, wherein: The encoder includes five residual convolution units and four 2x downsampling units, and the residual convolution units and the 2x downsampling units are alternately connected in series.

3. The facial image restoration method based on semantic features according to claim 1, characterized in that: The reference image generator comprises a first generator module, a second generator module and a third generator module connected in series in sequence; The first generator module includes a semantic feature fusion unit, a residual convolution unit and a 2x upsampling unit connected in series; The second generator module includes a semantic feature fusion unit and a residual convolution unit connected in series. and 2x upsampling unit; The third generator module includes a semantic feature fusion unit and a residual convolution unit connected in series.

4. The method for restoring a face image based on semantic features according to claim 3, wherein: The semantic feature fusion unit comprises one normalization layer and two convolutional layers connected in series; the semantic feature fusion unit is used to fuse low-quality facial semantic features and random noise.

5. The method for facial image restoration based on semantic features according to claim 1, wherein: The feature transferor includes a dictionary construction module and a feature transfer module; The dictionary construction module includes a facial feature extraction unit, a first facial component detection unit and a dictionary construction unit connected in series; The facial feature extraction unit includes a plurality of residual convolution units and 2x downsampling units alternately connected in series, and is used to obtain a plurality of high-quality facial reference features based on the plurality of high-quality facial reference images; The first face component detection unit is configured to obtain high-quality component features of different component categories based on the multiple high-quality face reference features; The dictionary construction unit is used to construct a face component feature dictionary based on the high-quality component features of the different component categories; The feature transfer module includes a second face component detection unit, a dictionary lookup unit and a feature fusion unit connected in series. The second facial component detection unit is configured to obtain low-quality component features of corresponding component categories based on the low-quality facial semantic features; The dictionary search unit is configured to obtain high-quality component features of corresponding component categories based on the low-quality component features and the constructed face component feature dictionary; The feature fusion unit is used to fuse the low-quality facial semantic features with the high-quality component features of the corresponding component category to obtain high-quality facial semantic features.

6. The method for facial image restoration based on semantic features according to claim 1, wherein: The decoder includes five residual convolution units and four 2x upsampling units, and the residual convolution units and the 2x upsampling units are alternately connected in series.

7. The method for facial image restoration based on semantic features according to claim 1, wherein: The training method of the restoration model includes the following steps: Obtain a training set, the training set including the face training image to be restored and the corresponding real face restoration image, Inputting the face training image to be restored into a pre-built restoration model to obtain a restored face training image; Calculating a restoration model loss function based on the restored face training image and the corresponding real face restoration image; The restoration model is iteratively updated and trained based on the gradient descent method, and the restoration model with the minimum restoration model loss function is used as the trained restoration model.

8. The method for facial image restoration based on semantic features according to claim 7, wherein: Obtaining a training set includes the following steps: Acquire a high-quality face image, and use the high-quality face image as a true face restoration image; Performing pixel adjustment on the high-quality face image to obtain a degraded face image; using the degraded face image as a face training image to be restored; The expression of the pixel adjustment operation is as follows: I lq ={JPEG q ((I hq *k σ )↓ s +n δ )}↑ s Where, I lq Represents the degraded face image, that is, the face training image to be restored; JPEG q Indicates JPEG compression with compression quality q; I hq represents a high-quality face image, i.e., a true restored face image; * represents a convolution operation; k σ represents the blur kernel with sigma = δ;↓ s Indicates downsampling s times operation; n δ Indicates Gaussian noise with sigma = δ; ↑ s Indicates an upsampling operation by a factor of s.

9. The method for facial image restoration based on semantic features according to claim 7, wherein: The restoration model loss function is expressed as follows: L = L l1 +λ per L per +λ adv L adv Where, L represents the loss function value of the restoration model; L l1 represents the L1 loss value; λ per represents the perceptual loss weight; L per Represents the perceptual loss value based on the VGG network; λ adv represents the adversarial loss weight; L adv Represents the adversarial loss value based on adversarial training.

10. The method for facial image restoration based on semantic features according to claim 9, wherein: The L1 loss value L l1 The expression is as follows: Where, I hq Represents the true restored image of the face; represents the restored face training image; |·|1 represents the mean absolute error; The perceptual loss value L based on the VGG network per The expression is as follows: Where, Represents the feature maps of the 1st to 5th convolutional layers in the pre-trained VGG model; represents the square of the 2-norm; The adversarial loss value L based on adversarial training adv The expression is as follows: Where D(·) represents the output of the discriminator in adversarial training; Indicates about expected value of a distribution; Softplus represents the softplus function, and the expression of the softplus function is: softplus(x)=ln(1+e x ).

Images