A face image restoration method based on face part prior

Abstract

The present invention provides a facial image restoration method based on facial part priors, which specifically includes a high-quality facial part prior module, a low-quality facial part mapping module and an output module. The facial image restoration method based on facial part priors constructs a priori networks of various parts through training, obtains three part sub-images given a low-quality facial image, maps them to high-quality feature z spaces corresponding to various parts through respective encoders, then obtains restored part results through the part prior network, restores various restored parts to the overall facial image, and obtains the final restored results. The method is suitable for high-quality restoration of low-quality facial images in real scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of image restoration, and in particular relates to a facial image restoration method based on facial part priors. Background Art

[0002] As a carrier of information, images are becoming more and more important in daily life, and facial images occupy a large proportion of images and play an important role. With the improvement of photographic equipment and technology, the quality of facial images is becoming clearer and clearer. However, due to the old photographing equipment and the instability of the photographing environment, low-quality facial images are easily blurred, which affects the visual perception, thereby leading to the failure of subsequent tasks such as face recognition. For real low-quality facial images, such as old photos, due to their complex degradation types, the existing technology cannot fully simulate the degradation process, resulting in the inability of data-driven neural networks to learn high-quality restoration mappings well. Existing methods usually use encoders and decoders to train restoration models. However, since the real degradation situation is difficult to simulate, the existing methods are usually unable to effectively restore unseen degradation types, and cannot accurately map low-quality facial images with diverse and complex degradation types to high-quality facial feature space, resulting in the restoration results easily changing the facial structure, and being limited by the computing power of mobile terminals, it is impossible to process ultra-high-definition facial image restoration. In addition, due to the complexity of training, it is easy to cause the restoration results to be unrealistic. For this reason, the present invention proposes a facial image restoration method based on facial part prior. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the present invention provides a facial image restoration method based on facial part prior. The facial image restoration method based on facial part prior constructs a priori networks of various parts through training. Given a low-quality facial image, three part sub-images are obtained, which are respectively mapped to the high-quality feature z space corresponding to each part through respective encoders. Subsequently, the restored part results are obtained through the part prior network, and each restored part is restored to the overall facial image to obtain the final restored result. The method is suitable for high-quality restoration of low-quality facial images in real scenes.

[0004] In order to achieve the above object, the present invention is implemented through the following technical scheme: a method for restoring a face image based on a priori knowledge of face parts, specifically comprising:

[0005] High-quality face part prior module, each salient part of the face is composed of a StyleGAN network structure, random noise z is input, and high-quality part images are output after training to obtain the prior network of each part;

[0006] The low-quality face part mapping module consists of three low-quality part encoders, which are used to map the various part images of a given low-quality face image of any degradation type to the high-quality feature z space of each part, and obtain the z space features of the high-quality parts corresponding to each low-quality part;

[0007] The output module is used to fuse the restored parts to obtain the restored face image.

[0008] In this embodiment, the high-quality face part prior module construction process is as follows:

[0009] ①: Input 1*P random noise z, pass through Q layers of fully connected layers, and obtain the features of the salient parts of the face; where P and Q are both integers greater than 1;

[0010] ②: The input 1*P-dimensional fixed constant and the style features of the salient parts of the face are modulated for the first time, and the learnable weight noise and activation function are embedded to obtain the features of the salient parts of the face;

[0011] ③: After convolution and activation functions, the features of the salient parts of the face are used to obtain the image of the salient parts of the face;

[0012] ④: For the features of the salient parts of the face and the style features of the salient parts of the face, after modulated convolution, embedding the learnable weight noise, activation function, obtain the features of the salient parts of the face, and return to ③;

[0013] ⑤: Repeat steps ③ and ④ nine times, and the obtained facial salient parts image is used as the final output facial salient parts image.

[0014] In this embodiment, all activation operations are FusedLeakyReLU functions, and all convolution operations are 3*3 convolution operations with a step size of 1.

[0015] In this embodiment, the construction process of the low-quality face part mapping module includes:

[0016] Input a low-quality face image and obtain the salient area of ​​the face;

[0017] The low-quality face salient area is sequentially subjected to eight convolution, normalization, activation operations, and the first full connection, activation, and second full connection operations to obtain the image mapping to high-quality spatial features;

[0018] The low-quality facial salient part mapping features are passed through the face prior module to obtain high-quality results of the restored facial salient parts;

[0019] The restored face salient area is integrated into the overall image to obtain the final restoration result;

[0020] Among them, the activation operation is the FusedLeakyReLU function.

[0021] In this embodiment, the feature consistency loss constraint during training is:

[0022] For the same high-quality image I h Degraded image I obtained with different degradation parameters d1 and I d2 , the feature space z vector of the salient parts of the face obtained by the low-quality part mapping module needs to meet the consistency:

[0023]

[0024]

[0025]

[0026] in is the low-quality face salient part mapping network, Θ is the network learnable parameter, I d1 and I d2 For high quality images I h The low-quality face salient regions after different degradations, I h For high quality images I h Cropped face salient area, z 1 , z 2 and z 0 For the corresponding mapping space, the consistency loss of the z-space face salient area is defined as:

[0027]

[0028] Degraded face salient parts image I d1 and I d2 The corresponding high-quality restoration result is:

[0029]

[0030]

[0031] in is the face salient part prior network, Θ D is the prior parameter of pre-training, I 1 'and I 2 In order to restore the high-quality salient areas of the face, the spatial consistency loss of the salient parts of the face image is defined as:

[0032]

[0033] In this embodiment, the multi-scale reconstruction loss during training is:

[0034] The general reconstruction loss is defined as:

[0035]

[0036] in and I P Corresponding to the output part image and the corresponding high-quality part image, Φ is the pre-trained VGG network, and i represents the corresponding scale space;

[0037] The multi-scale loss of the face part prior module is defined as:

[0038]

[0039] Beneficial effects of the present invention:

[0040] 1. This facial image restoration method based on facial part prior constructs a priori networks for each part through training. Given a low-quality facial image, three part sub-images are obtained, which are mapped to the high-quality feature z space corresponding to each part through their respective encoders. Subsequently, the restored part results are obtained through the part prior network, and each restored part is restored to the overall facial image to obtain the final restored result. This method is suitable for high-quality restoration of low-quality facial images in real scenarios.

[0041] 2. This face image restoration method based on face part prior proposes to refine the face image into various salient areas for processing. According to the characteristic that face images have a strong prior structure, namely eyes, nose and mouth, arbitrary random noise z is input through the part prior network to ensure the output of high-quality and clear part images. Any low-quality part image is used to find the corresponding noise space z through the part mapping module, and then the high-quality part result is output through the part prior module as the part restoration result. It can output high-quality and clear part results for any low-quality part image, which can steadily improve the restoration performance while solving the problem of limited computing power on mobile terminals. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a high-quality face part prior module of a face image restoration method based on face part prior;

[0043] Figure 2 A facial image restoration framework diagram based on a facial part prior facial image restoration method. DETAILED DESCRIPTION

[0044] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0045] See also Figure 1 to Figure 2 The present invention provides a technical solution: a method for restoring a face image based on a priori knowledge of face parts, specifically comprising:

[0046] High-quality face part prior module, each salient part of the face is composed of a StyleGAN network structure, random noise z is input, and high-quality part images are output after training to obtain the prior network of each part;

[0047] The low-quality face part mapping module consists of three low-quality part encoders, which are used to map the various part images of a given low-quality face image of any degradation type to the high-quality feature z space of each part, and obtain the z-space features of the high-quality parts corresponding to each low-quality part;

[0048] The output module is used to fuse the restored parts to obtain the restored face image.

[0049] Through training, a priori networks for various parts are constructed. Given a low-quality face image, three part sub-images are obtained, which are mapped to the high-quality feature z space corresponding to each part through their respective encoders. Subsequently, the restored part results are obtained through the part prior network, and each restored part is restored to the overall face image to obtain the final restored result, which is suitable for high-quality restoration of low-quality face images in real scenes.

[0050] In this embodiment, the salient parts of the face include eyes, nose and mouth, which are denoted as E, N and M respectively. The construction process of the high-quality face part prior module is as follows:

[0051] ①: Construction of high-quality eye part prior module

[0052] Input 1*P E The random noise z E , through Q E Layers of fully connected layers to obtain eye style features; where P E and Q E are all integers greater than 1;

[0053] Input 1*P E Dimensional fixed constant and eye style features, perform E1th modulated convolution, embed learnable weight noise, activation function, and obtain eye part features with scale 1;

[0054] For the eye feature with scale 1, after the E1th convolution and activation function, an eye image with scale 1 is obtained;

[0055] For the eye part features and eye style features with scale 1, after the E2th modulated convolution, the learnable weight noise and activation function are embedded to obtain the eye part features with scale 2;

[0056] For the eye features with a scale of 2, after the E2th convolution and activation function, an eye image with a scale of 2 is obtained;

[0057] For the eye part features and eye style features with a scale of 2, after the E3th modulated convolution, the learnable weight noise and activation function are embedded to obtain the eye part features with a scale of 3;

[0058] For the eye features with a scale of 3, after the E3th convolution and activation function, an eye image with a scale of 3 is obtained;

[0059] For the eye part features and eye style features with a scale of 3, after the E4th modulated convolution, the learnable weight noise and activation function are embedded to obtain the eye part features with a scale of 4;

[0060] For the eye features with a scale of 4, after the E4th convolution and activation function, an eye image with a scale of 4 is obtained;

[0061] For the eye part features and eye style features with a scale of 4, after the E5th modulated convolution, the learnable weight noise and activation function are embedded to obtain the eye part features with a scale of 5;

[0062] For the eye feature with a scale of 5, after the E5th convolution and activation function, an eye image with a scale of 5 is obtained;

[0063] For the eye part features and eye style features with a scale of 5, after the E6th modulated convolution, the learnable weight noise and activation function are embedded to obtain the eye part features with a scale of 6;

[0064] For the eye features with a scale of 6, after the E6th convolution and activation function, an eye image with a scale of 6 is obtained;

[0065] For the eye part features and eye style features with a scale of 6, after the E7th modulated convolution, the learnable weight noise and activation function are embedded to obtain the eye part features with a scale of 7;

[0066] For the eye feature with a scale of 7, after the E7th convolution and activation function, an eye image with a scale of 7 is obtained;

[0067] For the eye part features and eye style features with a scale of 7, after the E8th modulated convolution, the learnable weight noise and activation function are embedded to obtain the eye part features with a scale of 8;

[0068] For the eye features with a scale of 8, after the E8th convolution and activation function, an eye image with a scale of 8 is obtained;

[0069] For the eye part features and eye style features with a scale of 8, after the E9th modulated convolution, the learnable weight noise and activation function are embedded to obtain the eye part features with a scale of 9;

[0070] For the eye part feature with a scale of 9, after the E9th convolution and activation function, an eye part image with a scale of 9 is obtained as the final output eye part image;

[0071] ②: Construction of high-quality nose part prior module

[0072] Input 1*P N The random noise z N , through Q N Layer fully connected layer, to obtain the nose style features; where P N and Q N are all integers greater than 1;

[0073] Input 1*P N Dimensional fixed constant and nose style features, perform N1th modulated convolution, embed learnable weight noise, activation function, and obtain nose part features with scale 1;

[0074] For the nose feature with a scale of 1, after the N1th convolution and activation function, an image of the nose with a scale of 1 is obtained;

[0075] For the nose part features and nose style features with a scale of 1, after the N2th modulated convolution, the learnable weight noise and activation function are embedded to obtain the nose part features with a scale of 2;

[0076] For the nose feature with a scale of 2, after the N2th convolution and activation function, an image of the nose with a scale of 2 is obtained;

[0077] For the nose part features and nose style features with a scale of 2, after the N3th modulated convolution, the learnable weight noise and activation function are embedded to obtain the nose part features with a scale of 3;

[0078] For the nose feature with a scale of 3, after the N3th convolution and activation function, an image of the nose with a scale of 3 is obtained;

[0079] For the nose part features and nose style features with a scale of 3, after the N4th modulated convolution, the learnable weight noise and activation function are embedded to obtain the nose part features with a scale of 4;

[0080] For the nose feature with a scale of 4, after the N4th convolution and activation function, an image of the nose with a scale of 4 is obtained;

[0081] For the nose part features and nose style features with a scale of 4, after the N5th modulated convolution, the learnable weight noise and activation function are embedded to obtain the nose part features with a scale of 5;

[0082] For the nose feature with a scale of 5, after the N5th convolution and activation function, an image of the nose with a scale of 5 is obtained;

[0083] For the nose part features and nose style features with a scale of 5, after the N6th modulated convolution, the learnable weight noise and activation function are embedded to obtain the nose part features with a scale of 6;

[0084] For the nose feature with a scale of 6, after the N6th convolution and activation function, an image of the nose with a scale of 6 is obtained;

[0085] For the nose part features and nose style features with a scale of 6, after the N7th modulated convolution, the learnable weight noise and activation function are embedded to obtain the nose part features with a scale of 7;

[0086] For the nose feature with a scale of 7, after the N7th convolution and activation function, an image of the nose with a scale of 7 is obtained;

[0087] For the nose part features and nose style features with a scale of 7, after the N8th modulated convolution, the learnable weight noise and activation function are embedded to obtain the nose part features with a scale of 8;

[0088] For the nose feature with a scale of 8, after the N8th convolution and activation function, an image of the nose with a scale of 8 is obtained;

[0089] For the nose part features and nose style features with a scale of 8, after the N9th modulated convolution, the learnable weight noise and activation function are embedded to obtain the nose part features with a scale of 9;

[0090] For the nose feature with a scale of 9, after the N9th convolution and activation function, a nose image with a scale of 9 is obtained as the final output nose image;

[0091] ③: Construction of high-quality mouth part prior module

[0092] Input 1*P MThe random noise z M , through Q M Layer fully connected layer, to obtain the mouth style features; where P M and Q M are all integers greater than 1;

[0093] Input 1*P M Dimensional fixed constant and mouth style features, perform M1th modulated convolution, embed learnable weight noise, activation function, and obtain mouth part features with scale 1;

[0094] For the mouth feature with scale 1, after M1-th convolution and activation function, a mouth image with scale 1 is obtained;

[0095] For the mouth part features and mouth style features with scale 1, after M2-th modulated convolution, learnable weight noise and activation function are embedded to obtain the mouth part features with scale 2;

[0096] For the mouth features with a scale of 2, after the M2th convolution and activation function, a mouth image with a scale of 2 is obtained;

[0097] For the mouth part features and mouth style features with a scale of 2, after the M3th modulated convolution, the learnable weight noise and activation function are embedded to obtain the mouth part features with a scale of 3;

[0098] For the mouth features with a scale of 3, after the M3th convolution and activation function, a mouth image with a scale of 3 is obtained;

[0099] For the mouth part features and mouth style features with a scale of 3, after the M4th modulated convolution, the learnable weight noise and activation function are embedded to obtain the mouth part features with a scale of 4;

[0100] For the mouth features with a scale of 4, after the M4th convolution and activation function, a mouth image with a scale of 4 is obtained;

[0101] For the mouth part features and mouth style features with a scale of 4, after the M5th modulated convolution, the learnable weight noise and activation function are embedded to obtain the mouth part features with a scale of 5;

[0102] For the mouth features with a scale of 5, after the M5th convolution and activation function, a mouth image with a scale of 5 is obtained;

[0103] For the mouth part features and mouth style features with a scale of 5, after the M6th modulated convolution, the learnable weight noise and activation function are embedded to obtain the mouth part features with a scale of 6;

[0104] For the mouth features with a scale of 6, after the M6th convolution and activation function, a mouth image with a scale of 6 is obtained;

[0105] For the mouth part features and mouth style features with a scale of 6, after the M7th modulated convolution, the learnable weight noise and activation function are embedded to obtain the mouth part features with a scale of 7;

[0106] For the mouth feature with a scale of 7, after the M7th convolution and activation function, a mouth image with a scale of 7 is obtained;

[0107] For the mouth part features and mouth style features with a scale of 7, after the M8th modulated convolution, the learnable weight noise and activation function are embedded to obtain the mouth part features with a scale of 8;

[0108] For the mouth features with a scale of 8, after the M8th convolution and activation function, a mouth image with a scale of 8 is obtained;

[0109] For the mouth part features and mouth style features with a scale of 8, after the M9th modulated convolution, the learnable weight noise and activation function are embedded to obtain the mouth part features with a scale of 9;

[0110] For the mouth part features with a scale of 9, after the M9th convolution and activation function, a mouth part image with a scale of 9 is obtained as the final output mouth part image.

[0111] In this embodiment, all activation operations are FusedLeakyReLU functions, and all convolution operations are 3*3 convolution operations with a step size of 1.

[0112] In this embodiment, the construction process of the low-quality face part mapping module includes:

[0113] Input a low-quality face image and obtain the salient areas of the face, namely the eye, nose and mouth areas. The eye, nose and mouth areas are denoted as E, N and M respectively.

[0114] The low-quality eye area is sequentially subjected to eight convolution, normalization, activation operations, and the first full connection, activation, and second full connection operations to obtain an image mapping to a high-quality z E’ Spatial characteristics;

[0115] Among them, the E'1th convolution is performed as 64 3*3 convolution operations with a stride of 2;

[0116] The E'2th convolution operation is 128 3*3 convolution operations with a stride of 2;

[0117] The E'3th convolution operation is 256 3*3 convolution operations with a stride of 2;

[0118] The E'4th to E'8th convolution operations are all 512 3*3 convolution operations with a stride of 2;

[0119] The low-quality nose area is sequentially subjected to eight convolution, normalization, activation operations, and the first full connection, activation, and second full connection operations to obtain an image mapping to a high-quality z N’ Spatial characteristics;

[0120] Among them, the N'1th convolution is performed as 64 3*3 convolution operations with a stride of 2;

[0121] The N'2th convolution operation is 128 3*3 convolution operations with a stride of 2;

[0122] The N'3th convolution operation is 256 3*3 convolution operations with a stride of 2;

[0123] The N'4th to N'8th convolution operations are all 512 3*3 convolution operations with a stride of 2;

[0124] The low-quality mouth area is sequentially subjected to eight convolution, normalization, activation operations, and the first full connection, activation, and second full connection operations to obtain an image mapping to a high-quality z M’ Spatial characteristics;

[0125] Among them, the M'1th convolution is performed as 64 3*3 convolution operations with a stride of 2;

[0126] The M'2th convolution operation is 128 3*3 convolution operations with a stride of 2;

[0127] The M'3th convolution operation is 256 3*3 convolution operations with a stride of 2;

[0128] The M'4th to N'8th convolution operations are all 512 3*3 convolution operations with a stride of 2;

[0129] Low quality left eye part mapping feature z E’ After the eye part prior module is used, the high-quality result of the restored left eye is obtained;

[0130] The low-quality right eye image is flipped left to right and then passed through the low-quality mapping module. The restored eye image is obtained through the eye part prior module, and then flipped left to right to obtain a high-quality restoration result for the right eye.

[0131] Low quality nose part mapping feature z N’ After the nose part prior module is used, a high-quality result of the restored nose is obtained;

[0132] Low quality mouth part mapping feature z M’ After the mouth part prior module is used, high-quality results of the restored mouth are obtained;

[0133] The restored left eye, right eye, nose and mouth regions are fused into the overall image to obtain the final restoration result;

[0134] All activation operations are FusedLeakyReLU functions.

[0135] In this embodiment, the eyes, nose, and mouth are denoted as E, N, and M, respectively, and the feature consistency loss constraint during training is:

[0136] For the same high-quality image I h Degraded image I obtained with different degradation parameters d1 and I d2 , the feature space z vector obtained by the eye low-quality part mapping module needs to meet the consistency:

[0137]

[0138]

[0139]

[0140] in For low-quality eye part mapping network, Θ E are the network learnable parameters, and For high quality images I h Low-quality eye regions after different degradations, For high quality images I h Cropped eye area, and For the corresponding mapping space, the z-space eye area consistency loss is defined as:

[0141]

[0142] Degenerate eye image and The corresponding high-quality restoration result is:

[0143]

[0144]

[0145] in is the eye part prior network, is the pre-trained prior parameter, and In order to restore the high-quality eye area, the eye image spatial consistency loss is defined as:

[0146]

[0147] For the same high-quality image I h Degraded image I obtained with different degradation parameters d1 and I d2 , the feature space z vector of the nose part obtained by the nose low-quality part mapping module needs to meet the consistency:

[0148]

[0149]

[0150]

[0151] in For low-quality nose part mapping network, Θ N are the network learnable parameters, and For high quality images I h The low-quality nose region after different degradations, For high quality images I h The cropped nose area, and For the corresponding mapping space, the consistency loss of the nose area in z space is defined as:

[0152]

[0153] Degraded nose image and The corresponding high-quality restoration result is:

[0154]

[0155]

[0156] in is the nose part prior network, is the pre-trained prior parameter, and In order to restore the high-quality nose area, the nose image spatial consistency loss is defined as:

[0157]

[0158] For the same high-quality image I h Degraded image I obtained with different degradation parameters d1 and I d2 , the feature space z vector of the mouth part obtained by the mouth low-quality part mapping module needs to meet the consistency:

[0159]

[0160]

[0161]

[0162] in For low-quality mouth part mapping network, Θ M are the network learnable parameters, and For high quality images I h The low-quality mouth area after different degradations, For high quality images I h The cropped mouth area, and For the corresponding mapping space, the consistency loss of the z-space mouth area is defined as:

[0163]

[0164] Degraded nose image and The corresponding high-quality restoration result is:

[0165]

[0166]

[0167] in is the mouth part prior network, is the pre-trained prior parameter, and In order to restore the high-quality mouth area, the spatial consistency loss of the mouth image is defined as:

[0168]

[0169] In this embodiment, the eyes, nose, and mouth are denoted as E, N, and M, respectively, and the multi-scale reconstruction loss during training is:

[0170] The general reconstruction loss is defined as:

[0171]

[0172] in and I P Corresponding to the output part image and the corresponding high-quality part image, Φ is the pre-trained VGG network, and i represents the corresponding scale space;

[0173] The multi-scale loss of the eye part prior module is defined as:

[0174]

[0175] The multi-scale loss of the nose part prior module is defined as:

[0176]

[0177] The multi-scale loss of the mouth part prior module is defined as:

[0178]

[0179] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the attached claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved.

[0180] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A facial image restoration method based on facial part priors, characterized in that: Specifically include: Construct a high-quality face part prior module. Each salient face part is composed of a StyleGAN network structure. Random noise z is input. After training, high-quality part images are output respectively to obtain the prior network of each part. Construct a low-quality face part mapping module, which consists of three low-quality part encoders, to map each part image of a given low-quality face image of any degradation type to the high-quality feature z space of each part, and obtain the z-space features of the high-quality parts corresponding to each low-quality part; The output module fuses the restored parts to obtain the restored face image; The construction process of the high-quality face part prior module is as follows: ①: Input 1*P random noise z, pass through Q layers of fully connected layers, and obtain the features of the salient parts of the face; where P and Q are both integers greater than 1; ②: The input 1*P-dimensional fixed constant and the style features of the salient parts of the face are modulated for the first time, and the learnable weight noise and activation function are embedded to obtain the features of the salient parts of the face; ③: After convolution and activation functions, the features of the salient parts of the face are used to obtain the image of the salient parts of the face; ④: For the features of the salient parts of the face and the style features of the salient parts of the face, after modulated convolution, embedding the learnable weight noise, activation function, obtain the features of the salient parts of the face, and return to ③; ⑤: Repeat steps ③ and ④ nine times, and the obtained facial salient parts image is used as the final output facial salient parts image.

Images