Adaptive face restoration method and device based on facial attribute information prediction

Abstract

The present invention proposes an adaptive face restoration network and device based on facial attribute information prediction. Through fine-tuning of the facial parsing network, the complete facial attribute information is predicted from the missing face to provide facial structure information, such as the posture, expression and angle of the missing face. In facial restoration, the existing method of facial prior information is distinguished. The structural information is adaptively extracted and normalized through a separable adaptive normalization module. The predicted attribute information and the feature information of the input face image are adaptively fused to guide the generation of the facial structure with accurate structure. Finally, the identity information of the generated image and the original image is obtained through the identity extractor of the multi-head discriminator. The identity loss constraint is used to ensure the accuracy of the identity in the generation. Experimental results show that compared with the existing method, the method of the present invention can restore facial images with accurate structure and clear texture even in the case of complex occlusion parts.

Description

Technical Field

[0001] The present invention relates to the field of face image restoration, and in particular to an adaptive face restoration method and device based on facial attribute information prediction. Background Art

[0002] As an application of deep learning, face restoration has the ability to repair damaged portrait photos and restore the blurred facial features of people who violate the law during criminal investigations. Image completion (also known as inpainting) is a reconstruction process that infers the complete content from a cropped image. It is a fundamental task in the field of visual images, such as image editing, image relocation, object removal, face recognition, and photo restoration. For image restoration, contextual information is crucial to create visual texture and reasonable structure for the needs of the missing area. Traditional restoration methods supplement damaged images by searching for contextual content similar to the missing content through diffusion or patch-based methods. Many of these methods have achieved good visual results for simple natural images with repeated textures and similar structures, but cannot be applied to complex scenes. Most existing image restoration methods focus on the texture of the image rather than the structural information. Directly applying these methods to face restoration mostly generates unnatural and blurred face images. In particular, for face images, the attribute information of the face (face structure, contour, content information) is not fully utilized, resulting in unsatisfactory results. In recent years, due to the rapid development of deep learning and computer vision technology, various technologies related to human faces have become mature and are gradually being applied in people's lives.

[0003] Traditional image restoration methods often perform well on simple masked image restoration, but they are often poorly applicable when the missing area is too large. Therefore, many deep learning-based methods improve image restoration by increasing network depth and parameters to ensure the semantic coherence of the generated image. For example, context encoder (CE) introduces GAN into image restoration and achieves semantic hole filling by establishing a relationship between contextual information and damaged areas. However, since the consistency of structure and distribution is not considered, artifacts appear at the boundaries of the missing area. To improve the quality of image restoration, Yu et al. combined contextual attention (CA) with the restoration network to learn to utilize long-range information from unmasked pixels. However, when the structure and distribution inside and outside the image mask vary greatly, the results of CA suffer from severe structural distortion. In 2019, the edge-connect designed by Nazeri K et al. predicted the masked edge information to guide image restoration, but failed to generate harmonious and natural internal structure and content. In 2020, Liu et al. combined structure and texture information to generate high-quality images through mutual encoder-decoder. In 2021, Liao et al. proposed a multi-task learning framework based on coherent priors to obtain the structure and texture of semantic objects to promote the joint optimization of semantic segmentation and image restoration. In 2022, Li et al. generated multivariate results through a new Transformer, which included a new variant of multi-head self-attention and a novel style operation module. These image restoration methods do not fully utilize domain information (such as edge information and structural information). Therefore, they still have problems with structural distortion or obvious artifacts in face restoration.

[0004] Going deeper into the field of face restoration, most image restoration methods are not applicable to faces because faces have special textures and symmetrical structures. If these methods are used directly for face restoration, the resulting facial images will be unnatural and blurry. Early face restoration algorithms such as Pathak et al. proposed in 2016 used encoder-decoder models to fill the gaps. Subsequently, more complex networks, including unit variants, contextual attention mechanisms, and empty convolutions, emerged to improve the quality of generated faces. GFCnet in 2017 proposed an encoder-decoder model that used face parsing as a loss to constrain the face structure. However, it performed poorly in texture generation. In 2018, Lee et al. proposed a collaborative adversarial learning method to supervise face restoration by predicting semantic information and facial landmarks. But it did not focus on the reconstruction of the face structure, resulting in asymmetric generation results. In 2020, Zhou et al. designed a dual spatial attention module injected by oracle supervision to learn the correlation between facial textures at multiple scales. However, this method relies heavily on too many discriminators to supervise the generation of facial components, which increases the number of parameters and is difficult to train. In addition, multi-stage generation has also received widespread attention, in which intermediate cues are used to refine the structural information of the generated face. EdgeConnect proposed an end-to-end network consisting of an edge extractor and an image restoration network, which complements the edge information and contextual information of the image through combination. In 2021, Liu et al. proposed an end-to-end two-stage face restoration network based on dense multi-scale fusion of local binary patterns to provide structural information for image restoration. However, the above restoration methods cannot handle large mask problems and produce diverse completion results.

[0005] Deficiencies of existing technologies:

[0006] 1. Traditional methods have achieved good visual results for simple natural images with repeated textures and similar structures, but these methods cannot guarantee the structural consistency of the foreground and background of the generated image in the case of large-scale missing images.

[0007] Most traditional image restoration methods use the undamaged part of the current image or related images as a superset. Then they select the patch that is most similar to the boundary of the missing area to complete the image by migration or diffusion. However, when large areas of information are missing, these methods cannot even guarantee the consistency of the image structure or even forge a complete image structure. But when repairing large-scale missing areas, even if a complete image structure is generated, the structural consistency of the foreground and background cannot be guaranteed.

[0008] The network often directly adopts simple feature fusion methods (such as feature channel splicing), which easily causes semantic conflicts and leads to loss of effective information when fusing features of different scales.

[0009] Feature fusion after feature extraction is a common method to improve face restoration performance. However, common feature fusion structures often directly use simple feature fusion methods, which may lead to challenges such as semantic conflicts and context information loss, thus limiting the further improvement of network performance. Therefore, it is necessary to adopt a strategy to effectively filter out other useless information in the space and alleviate semantic conflicts.

[0010] 3. Face images have special textures and symmetrical structures. If the natural image restoration method is used to directly restore the missing face, it will result in unnatural and blurred face images.

[0011] Unlike natural images, face images need to consider facial structure and special texture information. However, most existing face restoration methods find it difficult to achieve both accurate and reasonable structural results when dealing with large-scale face loss. The networks that use facial attribute information to complete face restoration do not fully utilize facial attribute information (facial structure, contour and content), which leads to some structural distortions and facial asymmetry. Summary of the invention

[0012] In view of the shortcomings of the prior art, the present invention proposes an adaptive face restoration method based on facial attribute information prediction, characterized in that, for a restoration network for a large area of ​​missing faces, an adaptive face restoration network is constructed, comprising a facial parsing network FPN, a facial restoration network FIN and a multi-head discriminator including an identity detector IE, wherein the facial parsing network FPN is used to predict and obtain complete facial attribute information from the missing face to provide facial structure information; the facial restoration network FIN uses a separable adaptive normalization module to extract the facial structure information output by the facial parsing network and normalize it, and guides the generation of the facial structure with the facial structure information; the identity extractor IE of the multi-head discriminator obtains the identity information of the generated predicted face image and the original face image to ensure that the distribution of the predicted result is close to the distribution of the original face image, and the method specifically comprises:

[0013] Step 1: Prepare open face datasets for face restoration, namely the CelebA dataset and the CelebA-HQ dataset, and mask the face images in the two datasets to obtain the mask image Im;

[0014] Specifically, the masking ratio of the CelebA dataset is set to 64*64, and the masking ratio of the CelebA-HQ dataset is set to 128*128.

[0015] Step 2: Divide the data set obtained in step 1 into a training set, a validation set, and a test set according to the ratio of 8:1:1;

[0016] Step 3: construct and initialize an adaptive face restoration network based on facial attribute information prediction, wherein the adaptive face restoration network consists of a first-stage facial parsing network, a second-stage facial restoration network, and a multi-head discriminator;

[0017] In the first stage, the face parsing network includes a spatial path and a context path. The spatial path uses a small-stride convolution to retain spatial information and generate high-resolution features. The context path uses a fast downsampling strategy and an attention optimization module to obtain a large receptive field. A feature fusion module is used between the two paths to combine features.

[0018] In the second stage, the face restoration network uses the LinkNet network as the backbone network of the generator, including an encoder and a decoder, both of which are connected by residuals. The encoder is composed of a spectral normalization module, and the decoder is composed of a separable adaptive normalization module and a spectral normalization module.

[0019] The multi-head discriminator discriminates the predicted face result output by the face restoration network and the original face image;

[0020] Step 4: Inputting the training set into the constructed face restoration network for training, specifically including:

[0021] Step 41: inputting the mask image Im into the face parsing network to obtain the face area information of the human face, sequentially performing 4 times, 18 times, 16 times and 32 times downsampling operations of the context path to obtain the first feature map D1, the second feature map D2, the third feature map D3 and the fourth feature map D4, obtaining the spatial feature map Is through the spatial path, and inputting the third feature map D3 and the fourth feature map D4 into the attention optimization module respectively to obtain the optimized feature map U3 and the optimized feature map U4;

[0022] Step 42: Add the optimized feature map U3, the optimized feature map U4 and the fourth feature map D4 element by element to obtain a context feature map It, and finally input the spatial feature map Is and the context feature map It into a feature fusion module to obtain facial structure information Pm;

[0023] Step 43: input the mask image Im of the training set into the encoder of the face restoration network. The encoder obtains the attribute features of the face through the spectral normalization module and the downsampling module connected in sequence. Specifically, the mask image generates a coding feature map through the first spectral normalization module, and then downsamples by 2 times and inputs it into the spectral normalization module of the next level to obtain coding feature maps F1, coding feature maps F2, coding feature maps F3 and coding feature maps F4 respectively.

[0024] Step 44: The encoded feature map F4 is upsampled by 2 times, and then residually connected and summed with the encoded feature map F3, and then input into the spectral normalization module of the decoder to obtain a decoded feature map F1m; the decoded feature map F1m and the facial structure information Pm are input into the separable adaptive normalization module to obtain an intermediate weighted feature map W1;

[0025] Then, the intermediate weight feature map W1 is upsampled by 2 times, summed and fused with the encoding feature map F2 through residual connection, and then input into the next spectral normalization module to obtain the decoding feature map F2m. The decoding feature map F2m and the facial structure information Pm are input into the next separable adaptive normalization module to generate the intermediate weight feature map W2.

[0026] Finally, the intermediate weight feature map W2 is upsampled by 2 times, summed and fused with the encoding feature map F1 through residual connection, and then input into the next spectral normalization module to obtain the decoding feature map F3m. The decoding feature map F3m and the facial structure information Pm are input into the next separable adaptive normalization module to obtain the output feature map Fout.

[0027] Step 5: The output feature map Fout generated in step 44 is used as a predicted face image and input into a multi-head discriminator for discrimination. The multi-head discriminator uses loss to bring the distribution of the predicted face image closer to that of the original face image.

[0028] Step 6: Step 4 and step 5 are trained sequentially for a preset number of times, and the model weight is retained every 10 rounds. After the training is completed, the test set is passed to the trained face restoration network for testing to obtain whether the objective evaluation index of the face restoration currently tested is stable, and whether the restoration result is successfully generated by subjective visual observation. If so, the current model parameters are saved, otherwise skip and test the next model weight;

[0029] Step 7: Select the network with the best performance during the test, then feed the validation set into the network, calculate the objective evaluation index, and save the final predicted face image.

[0030] According to a preferred embodiment, the data processing of the feature fusion module in step 42 is as follows:

[0031] Step 421: Input the context feature map It and the spatial feature map Is into the feature fusion module, first perform feature splicing to extract spatial and context information, and then input the first branch and the second branch;

[0032] Step 422: the first branch obtains global information through global pooling, and then obtains local information through sequentially connected 1×1 convolution, relu activation and 1×1 convolution, sigmoid activation to obtain a fused feature map I1;

[0033] Step 423: The second branch obtains the fused feature map I2 through batch normalization, 1×1 convolution, and relu activation, and then multiplies the fused feature map I1 element by element to obtain the fused feature map I3, and then sums the features of the fused feature map I3 and the fused feature map I2 to obtain the facial structure information Pm.

[0034] According to a preferred embodiment, the data processing of the separable adaptive normalization module in step 44 is specifically as follows:

[0035] Step 441: In the separable adaptive normalization module, the input decoded feature map F1m and the face structure information Pm are sent to the separable adaptive normalization layer;

[0036] Step 442: In the separable adaptive normalization layer, the face structure information Pm first passes through a 1×1 convolution to extract features, and then passes through two 1×1 convolutions in parallel to balance the distribution of the image, and respectively extracts the mean value and variance of the face structure information Pm;

[0037] Step 443: In the separable adaptive normalization layer, the decoded feature map F1m is first normalized to a standard normal distribution, then multiplied by the variance of the face structure information Pm, and then added to its average value to form an intermediate feature map X1;

[0038] Step 444: The intermediate feature map X1 enters the next separable adaptive normalization layer through relu activation and convolution and combines with the face structure information Pm to obtain the intermediate feature map X2, which is again activated and convolved through relu and residually connected with the decoded feature map F1m to generate the intermediate weighted feature map W1;

[0039] Similarly, the decoded feature map F2m and the facial structure information Pm are input into the separable adaptive normalization module to obtain the intermediate weighted feature map W2.

[0040] An adaptive face restoration device based on facial attribute information prediction, the device is used to implement the adaptive face restoration method according to claim 1, including a data set preprocessing module, a facial analysis module, a facial restoration module and a multi-head discriminator, wherein:

[0041] The data set preprocessing module is used to mask the complete face to obtain a mask image Im, and divide it into a training set, a validation set and a test set according to the ratio;

[0042] The facial analysis module is used to obtain complete facial attribute information from the mask image Im, including a spatial path, a context path and a feature fusion module, obtains spatial features through the spatial path, obtains context features through the context path, and outputs facial structure information Pm through the feature fusion module;

[0043] The facial restoration module includes a spectral normalization module and a separable adaptive normalization module, which is used to adaptively extract the features of the facial structure information Pm and the mask image Im to guide the generation of the facial structure;

[0044] The multi-head discriminator is used to discriminate the predicted face result and the original face image output by the face restoration module, obtain the identity information of the predicted face image and the original face image through the identity extractor, and use the identity loss constraint to ensure the accuracy of the identity during the generation process.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. This paper proposes semantic segmentation-guided face restoration, which can generate visually realistic results and preserve facial specific structures in large-area masked faces.

[0047] 2. In order to fully preserve the facial structure, a separable adaptive normalization block is designed. In the face restoration generator, the predicted semantic information is used to ensure the consistency of the appearance and structure of the face area.

[0048] 3. In order to ensure the accuracy of the identity information of the generated image and the original image, an identity extractor is proposed to ensure the consistency of the texture information between the completed face and the original face. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the structure of the network adaptive face restoration network proposed in the present invention;

[0050] Figure 2 It is a schematic diagram of the structure of the facial parsing network of the first stage of the network proposed in the present invention;

[0051] Figure 3 Schematic diagram of the structure of the face restoration network in the second stage of the network proposed in the present invention;

[0052] Figure 4 is a structural schematic diagram of the attention optimization module of the present invention;

[0053] Figure 5 It is a structural schematic diagram of the feature fusion module of the present invention;

[0054] Figure 6 (a) is a schematic diagram of the structure of the separable adaptive normalization module;

[0055] Figure 6 (b) Schematic diagram of the structure of the separable adaptive normalization layer

[0056] Figure 7 This is a comparison chart of the experimental results of different methods on the CelebA-HQ dataset;

[0057] Figure 8 This is a comparison chart of the results of the ablation experiment on the CelebA-HQ dataset of the present invention;

[0058] Fig. 9 This is a comparison chart of the results of different mask positions on the CelebA-HQ dataset of the present invention;

[0059] Fig.10 It is a structural schematic diagram of the invented face repair device. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0061] AFINet of the present invention refers to: Adaptive Face Inpainting Network, adaptive face repair network.

[0062] The FPN of the present invention refers to: Face Parsing Network, facial analysis network.

[0063] The FIN of the present invention refers to: Face Inpainting Network, face repair network.

[0064] The ARM of the present invention refers to: Attention Refinement Module, attention optimization module.

[0065] The FFM of the present invention refers to: Feature Fusion Module, feature fusion module.

[0066] The SANBlock of the present invention refers to: Separable Adaptive Normalization Block, which is a separable adaptive normalization module.

[0067] The SNBlock of the present invention refers to: Apectral Normalization Block, spectral normalization module.

[0068] The M-PD of the present invention refers to: Multi-Patch Discriminator, a multi-head discriminator (composed of 3 different losses).

[0069] The IE of the present invention refers to: Identity Extractor. (It is composed of a 5-layer residual network Resnet and a fully connected layer Fully Connected Layer)

[0070] The present invention relates to the research field of mask face image restoration, which mainly solves the problems of distorted facial structure, blurred details, edge artifacts, etc. in existing face restoration methods. As one of the mainstream methods in mask face image restoration tasks, deep learning has shown superior performance far exceeding traditional methods. However, most neural network architectures in deep learning are difficult for ordinary convolutional neural networks to learn the global structure of the image, and it is difficult to balance the accuracy and rationality of the structure. The network that uses facial attribute information to complete face restoration does not make full use of facial attribute information (facial structure, contour and content), which leads to some structural distortions and facial asymmetry.

[0071] In view of the problems of distorted face structure, blurred details, edge artifacts, etc. in existing face restoration methods, this paper proposes an adaptive face restoration network based on facial attribute information prediction. Figure 1 This is the network structure diagram of the adaptive face restoration network proposed in the present invention. Its input data is the mask image Im, where the original unmasked image Io is used to guide the generation of facial structure information; the output is the predicted generated image Ig. The resolution of the original image Io is 3×256×256, 3 represents the number of channels, and 256 represents the height and width of the image respectively.

[0072] The core and main innovation of the adaptive face restoration network of the present invention lies in the feature fusion module FFM, which can separate the adaptive normalization module SANBlock and the identity extractor IE.

[0073] The method proposed in the present invention constructs a restoration network for large-area missing faces. The constructed adaptive face restoration network AFINet includes three parts: a face parsing network FPN, a face restoration network FIN and a multi-head discriminator based on an identity detector IE, wherein the face parsing network FPN (structure diagram as shown in FIG. Figure 2 The facial restoration network FIN (structure diagram as shown) is used to predict and obtain complete facial attribute information from the missing face to provide facial structure information, such as the posture, expression and angle of the missing face; Figure 3As shown in the figure, the structure information of the output of the facial parsing network FPN is adaptively extracted and normalized, and the generation of the facial structure is guided by the accurate structure. The distribution of the generated result is ensured to be closer to the distribution of the original face image through the multi-head discriminator M-PD. Among them, the identity extractor IE obtains the identity information of the generated face image and the original face image, and uses the identity loss constraint to ensure the accuracy of the identity during the generation process, and successfully adaptively fuses the predicted attribute information and the feature information of the input image, thereby improving the accuracy performance of face restoration. The method specifically includes:

[0074] Step 1: Prepare two open face datasets for face restoration, namely the CelebA dataset and the CelebA-HQ dataset, and mask the face images in the two datasets to obtain the mask image Im.

[0075] Step 11: Download the divided CelebA dataset and CelebA-HQ dataset from the official website of the dataset; the image resolution of the CelebA dataset is set to 256*256, and the mask ratio is 64*64; CelebA-HQ is a higher quality face image of CelebA, with a resolution of 512 and a mask ratio of 128*128.

[0076] Step 2: Divide the data set obtained in step 1 into training set, validation set and test set according to the ratio of 8:1:1.

[0077] Specifically, the CelebA dataset contains 202,599 images of 10,177 identities, which are divided into three subsets: training set, validation set, and test set; containing 162,770, 19,867, and 19,962 samples respectively. The CelebA-HQ dataset has a total of 30,000 images, and this method divides it into a training set of 28,000 samples, a validation set of 1,000 samples, and a test set of 1,000 samples.

[0078] Step 3: Build and initialize the adaptive face restoration network AFINet based on facial attribute information prediction. The structural diagram is as follows: Figure 1 As shown in the figure, the face restoration network consists of two stages: the first stage face parsing network FPN and the second stage face restoration network FIN. In the first stage, the face parsing network FPN consists of a bilateral segmentation network containing two paths. The spatial path uses a convolution with a small stride to retain spatial information and generate high-resolution features. The context path uses a fast downsampling strategy and an attention optimization module ARM to obtain a sufficient receptive field. The feature fusion module FFM is used between the two paths to combine features. In the second stage, the face restoration network FIN (structure diagram as shown in the figure) is constructed. Figure 3As shown in Figure 3, the LinkNet network is used as the backbone network of the generator, which consists of three parts: residual connection, separable adaptive normalization module SANBlock and spectral normalization module SNBlock.

[0079] In order to make fuller and deeper use of attribute features in the generation process, this method expresses the facial attribute information obtained in the encoding stage of the generator as a multi-level map and feeds it into the high-dimensional layer corresponding to the decoding part. Finally, considering that identity information contains many specific texture and structural features, this method innovatively uses the identity discriminator IE to constrain the identity of the generated image based on the multi-head discriminator M-PD with three losses (adversarial loss, reconstruction loss, and identity loss), ensuring that the identity attributes and feature distribution of the generated image are close to the original image Io.

[0080] Step 4: Input the training set processed in step 3 into the constructed face restoration network to train the network, specifically including:

[0081] Step 41: Input the mask image Im in the training set into the facial parsing network to obtain the regional information of the face, and obtain the first feature map, the second feature map D2, the third feature map D3 and the fourth feature map D4 through 4 times, 8 times, 16 times and 32 times downsampling operations of the context path in sequence, obtain the spatial feature map Is through the spatial path, and input the third feature map D3 and the fourth feature map D4 into the attention optimization module respectively to obtain the optimized feature map U3 and the optimized feature map U4 respectively.

[0082] The data processing flow of the attention optimization module is as follows:

[0083] Step 411: Input the third feature map D3 and the fourth feature map D4 into the attention optimization module, extract global information through global pooling in the attention optimization module, reduce the dimension, and then use 1×1 convolution to adjust the channel of the output feature. Finally, batch normalization and sigmoid activation function are used to make the weight of the region of interest higher, and output feature vector W3 and feature vector W4.

[0084] Step 413: Then, the feature vector W3 and the feature vector W4 are element-wise multiplied with the third feature map D3 and the fourth feature map D4 to obtain the optimized feature map U3 and the optimized feature map U4 focusing on the information of different facial regions.

[0085] Figure 4 This is a schematic diagram of the structure of the attention optimization module (ARM) of the present invention. Through continuous convolution, normalization and sigmoid activation, the optimized weights for different facial regions are obtained, and the feature information is enriched through residual connection to ensure sufficient receptive field.

[0086] Step 42: The optimized feature map U3, the optimized feature map U4 and the fourth feature map D4 are added element by element to obtain the context feature map It. Finally, the spatial feature map Is and the context feature map It are passed to the feature fusion module FFM to obtain the face structure information Pm. The data processing of the feature optimization module is as follows:

[0087] Step 421: Input the context feature map It and the spatial feature map Is into the feature fusion module, first perform feature splicing so that the feature map contains richer spatial and context information, and then input the first branch and the second branch;

[0088] Step 422: The first branch obtains global information through global pooling, and then obtains different local information through the series connection of 1×1 convolution, relu activation and 1×1 convolution, sigmoid activation to obtain the fused feature map I1.

[0089] Step 423: The second branch obtains the fused feature map I2 through batch normalization, 1×1 convolution, and relu activation, and then multiplies it element by element with the first branch to obtain the fused feature map I3. The fused feature map I3 is then summed with the fused feature map I1 to finally obtain the facial structure information Pm.

[0090] Figure 5 The figure is a schematic diagram of the structure of the feature fusion module FFM of the present invention. Feature fusion is often an effective method to improve the accuracy of face restoration. However, common feature fusion structures usually directly aggregate and splice global features. The direct fusion of features from different regions will lead to the loss of valuable clues due to the non-negligible semantic differences. The feature fusion module FFM enriches the contextual information of the network by inputting different branch convolutions, thereby alleviating semantic conflicts.

[0091] Figure 2 This is a schematic diagram of the structure of the facial parsing network FPN in the first stage of the adaptive face restoration network AFInet proposed in the present invention. This module takes the mask image Im as input and the facial structure information Pm as output. With the help of the mapping relationship between the original face image Io and the face mask image Im, the facial parsing network is used to obtain the regional information of the face, and this is used to guide the generation of the missing part of the face.

[0092] Step 43: The mask image Im of the training set is input into the encoder of the face restoration network. In the encoding stage, the attribute features of the face are obtained through the spectral normalization module and the downsampling module connected in sequence. Specifically, the mask image is generated into a coding feature map through the first spectral normalization module, and then downsampled by 2 times and input into the spectral normalization module of the next level to obtain the coding feature map F1, coding feature map F2, coding feature map F3 and coding feature map F4 respectively.

[0093] Figure 3The schematic diagram of the structure of the face restoration network FIN in the second stage of the adaptive face restoration network AFINet is shown. The face restoration network takes the face structure information Pm and the mask image Im obtained in the first stage as input. The network uses the LinkNet network as the backbone network of the generator, including an encoder and a decoder, both of which are connected by residuals. The encoder is mainly composed of a spectral normalization module, and the decoder is mainly composed of a separable adaptive normalization module and a spectral normalization module.

[0094] In the encoder, we input the cropped image into an encoder consisting of spectral normalization to obtain the attribute features of the face. In order to utilize the attribute features in the generation process, we represent them as multi-level maps and send them to the higher dimensional layers corresponding to the decoder. Specifically, the missing image Im is sent to the generator to obtain the corresponding encoded feature map F1, encoded feature map F2, encoded feature map F3 and encoded feature map F4. The first three encoded feature maps are added element by element with the up-sampled results of the encoded feature map F4, the intermediate weight feature map W1, and the intermediate weight feature map W2 through residual connections to enrich the missing information and then continue to be input to the next stage of the decoder.

[0095] The decoder part includes a spectral normalization module and a separable adaptive normalization module SANBlock. The separable adaptive normalization module SANBlock fuses facial attribute information and context feature information extracted from the missing image at the channel level to guide facial generation. This can effectively utilize structural information while ensuring the semantic continuity and structural accuracy of the generated image. The resulting face image is fed into the multi-head discriminator M-PD to close the distribution of the generated image and the original image through loss.

[0096] Considering that identity information contains many specific texture and structural features, but some face restoration algorithms do not take their role into account, this method uses an identity extractor IE to constrain the identity of the generated image. Specifically, the identity vector is extracted from the complete original unmasked image Io and the generated image Ig as an additional feature constraint. Specifically, when generating a face image, Io and Ig are respectively fed into the identity extractor to obtain the corresponding identity vectors Vo and Vg. Then the cosine similarity of the two vectors is calculated and used as a loss to further close the feature distribution of the generated image and the original image. It can ensure that the identity attributes and feature distribution of the generated image are close to the original image.

[0097] Step 44: The encoded feature map F4 outputted from step 42 is upsampled by 2 times, and then residually connected and fused with the encoded feature map F3, and then input into the spectral normalization module of the decoding stage to obtain the decoded feature map F1m. The decoded feature map F1m and the facial structure information Pm are input into the separable adaptive normalization module to obtain the intermediate weighted feature map W1.

[0098] Then, the intermediate weighted feature map W1 is upsampled by 2 times and then fused with the encoded feature map F2 through a residual connection and input into the next spectral normalization module to obtain the decoded feature map F2m. The decoded feature map F2m and the facial structure information Pm are repeatedly input into the separable adaptive normalization module to output the intermediate fused feature map W2.

[0099] Finally, the intermediate fusion feature map W2 is upsampled by 2 times and then fused with the encoded feature map F1 through a residual connection and input into the spectral normalization module of the decoding stage to obtain the decoded feature map F3m. The decoded feature map F3m and the facial structure information Pm are repeatedly input into the separable adaptive normalization module to obtain the output feature map Fout.

[0100] Figure 6 (a) is a schematic diagram of the structure of the separable adaptive normalization module. Figure 6 (b) is a schematic diagram of the structure of the separable adaptive normalization layer, and its data processing is specifically as follows:

[0101] Step 441: In the separable adaptive normalization module of step 44, the input decoded feature map F1m and face structure information Pm are sent to the separable adaptive normalization layer;

[0102] Step 442: In the separable adaptive normalization layer, the face structure information Pm first passes through a 1×1 convolution to extract features, and then passes through two 1×1 convolutions in parallel to balance the distribution of the image, and respectively extracts the mean value and variance of the face structure information Pm;

[0103] Step 443: In the separable adaptive normalization layer, the decoded feature map F1m is first normalized to a standard normal distribution, then multiplied by the variance of the facial structure information Pm, and then added to its average value to form an intermediate feature map X1.

[0104] Step 444: The intermediate feature map X1 enters the next separable adaptive normalization layer through relu activation and convolution and is combined with the facial structure information Pm to form the intermediate feature map X2. After relu activation and convolution again, it is residually connected with the decoded feature map F1m to generate the intermediate weight feature map W1.

[0105] Similarly, the decoded feature map F2m and the facial structure information Pm are input into the separable adaptive normalization module to obtain the intermediate weighted feature map W2.

[0106] In the separable adaptive normalization layer, the decoded feature map F1m or the decoded feature map F2m is used as input, and first batch normalization is used to convert it into a standard normal distribution. Then, the attribute information Pm extracted by the face parsing network is sent to two 1×1 parallel convolutional layers to balance the distribution of the image. This method calculates the mean and variance of each corresponding facial region by learning its pixel-level mean and standard deviation, multiplies its mean with the mean of the corresponding decoded feature map F1m or the corresponding decoded feature map F2m, and then adds its corresponding variance. Then, the generated face structure is regularized to complete the adaptive fusion of structural information and context information, and finally the attribute information of each facial region is learned by inverse regularization.

[0107] Step 5: The output feature map Fout is used as the predicted face image and input into the multi-head discriminator M-PD for discrimination, and the distribution of the generated image and the original image is brought closer through loss. Specifically, there are three types of losses: reconstruction loss, adversarial loss, and identity loss.

[0108] Among them, the adversarial loss is used to improve the realism of the generated face image, the reconstruction loss ensures the similarity between the repaired area and the original image at the pixel level, and the identity loss is used to maintain the individual characteristics and consistency of the repaired face. Only one discriminator is used to obtain multi-level information from the images generated by the three loss approximations, and provides a precisely located facial feature distribution, ultimately generating visually realistic and high-quality facial images.

[0109] Step 6: Step 4 and step 5 are trained sequentially for a preset number of times, and the model weight is retained every 10 rounds. After the training is completed, the test set is passed to the trained face restoration network for testing to determine whether the network is stable in the current test face restoration objective evaluation indicators PSNR (Peak Signal to Noise Ratio, measuring the similarity between images), SSIM (Structural Similarity measuring the structural similarity between images), and FID (Fréchet Inception Distance measuring the feature similarity of generated images). Whether the generated result is successfully restored is observed through subjective visual observation. If so, the current model parameters are saved, otherwise the next model weight is skipped for testing.

[0110] Step 7: Select the network with the best performance during the test, then send the test set to the network, calculate objective evaluation indicators such as PSNR, SSIM, FID, and save the final detection effect diagram.

[0111] To further illustrate the effectiveness of the proposed method, the proposed method is compared with other existing methods. In order to make a fair comparison, the official codes released by other methods are used and their experimental settings are followed, where all methods are implemented in the same computing environment and quantitative and qualitative analysis is performed simultaneously. The 10 methods are as follows: Method 1 is the CSA method, which proposes to use the attention mechanism to focus on the semantically relevant areas in the image to address the problem of local pixel discontinuity caused by ignoring the semantic relevance and feature continuity of the hole / missing area; Method 2 is the PEN method, which is a convolutional neural network composed of a partial encoder and a decoder. The innovation lies in the introduction of partial encoding strategies, perceptual losses and self-attention mechanisms to improve the consistency, detail preservation and context understanding capabilities of image restoration; Method 3 is the RN method, which proposes a regional normalization method to perform specific normalization on the restoration area so that the restoration result better matches the surrounding environment and reduces the artifacts and discontinuities introduced by the restoration; Method 4 is the MED method, which aims at the problem of local feature capture and utilization in image restoration, by segmenting the image and assigning a dedicated embedding network to each local area, the accuracy and naturalness of the image restoration result are improved; Method 5 is the RFR method, which is a method based on cyclic feature reasoning , by gradually reasoning and integrating local and global features, more accurate, detailed and natural image restoration results are achieved; Method 6 is the PIC method, which accurately and carefully completes the image with good continuity and naturalness by utilizing the existing photometric information and image edge features in the image; Method 7 is the ICT method, which innovatively achieves high-fidelity image completion by introducing the Transformer model and multimodal sampling, and generates diversified restoration results with rich details and natural authenticity, significantly improving the performance and effect of image restoration tasks; Method 8 is the MAT method, which proposes a model that integrates Transformer and convolution for the problem of large mask image restoration, and achieves high-fidelity and diverse image restoration through customized Transformer blocks and dynamic masks; Method 9 is the CMT method, which uses a continuous mask to represent the amount of error in the mark, predicts the initial restoration result through multi-layer mask self-attention and mask update layers, and reconstructs a more realistic image through a refinement process.

[0112] Since the PIC method, ICT method and MAT method can produce a variety of different types of results among these methods, this method selects the best results among them for comparison. In order to quantify the quality of face images restored by each SOTA method from the perspective of indicators, the network uses the following three indicators for evaluation: Peak Signal to Noise Ratio (PSNR) is used to measure the similarity between images; Structural Similarity (SSIM) measures the structural similarity between images; FID value (Fréchet Inception Distance) measures the feature similarity of the generated image. For structural similarity PSNR and structural similarity SSIM, the higher the value, the better the performance, while for FID value, the lower the value, the more realistic the image appears visually. In order to prove that the network proposed in this invention has better performance, the generated image quality is better, clearer, more realistic and structurally accurate, and can be fairly compared with these deep learning-based methods, this paper uses the same dataset partition and the same training environment to train and test these methods, and selects the best results of each method according to its final indicators.

[0113] Table 1 shows the quantitative indicators of the nine compared methods and the adaptive face restoration network proposed in this invention on the CelebA-HQ dataset and the CelebA dataset.

[0114] Table 1 Quantitative evaluation indicators of different methods on different datasets

[0115]

[0116] From the values ​​in Table 1, it can be seen that the FID values ​​of method 5 in both datasets are better than those of other methods. It designs an inter-encoder to jointly focus on the structure and texture at different levels, balance the features, and achieve image restoration. The adaptive face restoration network AFINet of the present invention achieves a higher structural similarity SSIM (0.9180), peak signal-to-noise ratio PSNR (28.2910) and a relatively high FID value (10.69), which shows that the results of this method are superior to other methods. In addition, compared with method 6, method 7 has a structural similarity SSIM of 0.8784, a peak signal-to-noise ratio PSNR of 25.3825, and an FID value of 12.5007 in face completion. The results of this method are also better than them, which can further prove that the method of this paper is more effective than most existing face restoration methods.

[0117] In order to more intuitively illustrate the effectiveness of the method of the present invention, the face restoration effects of the existing method and the method of the present invention are compared. Figure 7This is a comparison of the experimental results of different methods on the CelebA-HQ dataset. Figure 7 (a) is the input image, Figure 7 (b) is the experimental result of method 1. Figure 7 (c) is the experimental result of method 2, Figure 7 (d) is the experimental result of method 3, Figure 7 (e) is the experimental result of method 4, Figure 7 (f) is the experimental result of method 5, Figure 7 (g) is the experimental result of method 6, Figure 7 (h) is the experimental result of method 7, Figure 7 (i) is the experimental result of method 8, Figure 7 (j) is the experimental result of the method of the present invention, Figure 7 (k) is a real face image.

[0118] according to Figure 7 From the comparison results in , it can be seen that the facial features of method 1 have artifacts, especially the eyes and lips; the texture of method 2 is too smooth, and the generated features are relatively fuzzy; method 6 achieves the diversity of the process, but does not retain reasonable structural information; method 3 has clear texture, but lacks facial details, and the eyes are relatively fuzzy; method 4 lacks some structures removed by the mask, and a distorted mouth structure appears; method 7 can generate clearer textures, but the generated face structure is somewhat unreasonable. At the same time, since method 7 uses the Transformer method, the number of calculation parameters is large, and the time required to generate an image is relatively slow compared to other methods. Method 8 can generate a clearer facial structure, but there is a boundary between the inside and outside of the mask, which is quite different from the original image; the completion result of method 9 performs poorly in capturing the lip structure. The method of the present invention has better performance in face restoration.

[0119] In order to prove the effectiveness of the adaptive face restoration network AFINet proposed in this paper, the performance of different parts of the model is analyzed. The experimental results are shown in Table 2. The comparison results of the ablation experiment are shown in Table 2. Figure 8As shown in the figure, ablation experiments on the CelebA-HQ dataset. The following different combinations are specifically analyzed. As shown in Table 2, a) W / i represents the adaptive face restoration network AFINet without identity loss, which results in the lack of clear and reasonable structure of the generated image; b) W / s represents the adaptive face restoration network AFINet does not use SNABlock for feature fusion, which results in unstable structure of the generated image; c) W / u means that our network structure replaces the main structure of Linknet with Unet, which achieves poor performance in face restoration tasks; d) W / p means that the face parsing network FPN is removed from our method, resulting in unreasonable facial structure of the generated image.

[0120] Table 2 Quantitative ablation results of adaptive face restoration network AFINet

[0121]

[0122] like Figure 8 As shown, Figure 8 (a) is the input image, Figure 8 (b) is the experimental result of W / i network. Figure 8 (c) is the experimental result of W / s network. Figure 8 (d) is the experimental result of W / u network, Figure 8 (e) is the experimental result of W / p network, Figure 8 (f) is the experimental result of the network of the present invention, Figure 8 (g) is a real face image.

[0123] From the ablation experiment, it can be seen that the adaptive face restoration network AFINet designed in the present invention achieves better performance results in face restoration in both quantitative and qualitative aspects.

[0124] In addition, to illustrate the robustness of our method, in addition to the quantitative and qualitative results of center-missing images, we also show the face restoration results using different masks. Fig. 9 The results of face restoration with different masks are shown, and the face images used are from the public CelebA-HQ dataset. Fig. 9 As shown, the gray box represents the masked area. Fig. 9 (a) is the left eye mask image, Fig. 9 (b) is the right eye mask image, Fig. 9 (c) Eye mask image, Fig. 9 (d) Left face mask image, Fig. 9 (e) Right face mask image, Fig. 9 (f) Mouth mask image, Fig. 9 (g) Masked image of the face excluding the hair. Fig. 9(i) is the inpainted image of the left eye mask, Fig. 9 (j) corresponds to the repaired image of the right eye mask. Fig. 9 (k) Inpainted image corresponding to the eye mask, Fig. 9 (l) The inpainted image corresponding to the left face mask, Fig. 9 (m) The inpainted image corresponding to the right face mask, Fig. 9 (n) The inpainted image corresponding to the mouth mask, Fig. 9 (o) Corresponding inpainted image of the face mask excluding the hair.

[0125] Table 3 is the quantitative evaluation index of the image restoration method of the present invention at different occluded parts. It can be seen from Table 3 that the method of the present invention still has a relatively high objective evaluation index and obtains a better restoration effect even when half or full face is occluded.

[0126] Table 3 Quantitative indicators of the restoration results of the method of the present invention in seven different occluded parts

[0127] Fig.10 FIG. 1 is a schematic diagram of the structure of the face repair device of the present invention. Fig.10 As shown, the restoration device is used to implement the adaptive face restoration method of the present invention, including a data set preprocessing module, a face analysis module, a face restoration module and a multi-head discriminator, wherein:

[0128] The data set preprocessing module is used to mask the complete face to obtain the mask image Im, and at the same time divide it into a training set, a verification set and a test set according to the ratio.

[0129] The facial analysis module is used to obtain complete facial attribute information from the mask image Im, including a spatial path, a context path and a feature fusion module. The spatial features are obtained through the spatial path, the context features are obtained through the context path, and the facial structure information Pm is output through the feature fusion module.

[0130] The facial restoration module includes a spectral normalization module and a separable adaptive normalization module, which is used to adaptively extract the features of the facial structure information Pm and the mask image Im to guide the generation of the facial structure.

[0131] The multi-head discriminator is used to discriminate the predicted face result and the original face image output by the face restoration module. In addition to using the reconstruction loss and the adversarial loss to adjust the prediction, the identity information of the predicted face image and the original face image is obtained through the identity extractor, and the identity loss constraint is used to ensure the accuracy of the identity during the generation process.

[0132] It should be noted that the above specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall within the protection scope of the present invention. Those skilled in the art should understand that the present invention description and its drawings are illustrative and do not constitute a limitation of the claims. The protection scope of the present invention is defined by the claims and their equivalents.

Claims

1. An adaptive face restoration method based on facial attribute information prediction, characterized in that: For a repair network of a large area of ​​missing face, an adaptive face repair network is constructed, including a face parsing network, a face repair network and a multi-head discriminator including an identity detector, wherein the face parsing network is used to predict and obtain complete facial attribute information from the missing face to provide facial structure information; the face repair network uses a separable adaptive normalization module to extract and normalize the facial structure information output by the face parsing network, and guides the generation of the facial structure with the facial structure information; the identity extractor of the multi-head discriminator obtains the identity information of the generated predicted face image and the original face image to ensure that the distribution of the predicted result is close to the distribution of the original face image. The method specifically includes: Step 1: Prepare open face datasets for face restoration, namely the CelebA dataset and the CelebA-HQ dataset, and mask the original face images in the two datasets to obtain the mask image Im; Step 2: Divide the data set obtained in step 1 into a training set, a validation set, and a test set according to the ratio of 8:1:1; Step 3: construct and initialize an adaptive face restoration network based on facial attribute information prediction, wherein the adaptive face restoration network is composed of a facial parsing network in the first stage and a facial restoration network in the second stage; In the first stage, the face parsing network includes a spatial path and a context path. The spatial path uses a small-stride convolution to retain spatial information and generate high-resolution features. The context path uses a fast downsampling strategy and an attention optimization module to obtain a large receptive field. A feature fusion module is used between the two paths to combine features. In the second stage, the face restoration network uses the LinkNet network as the backbone network of the generator, including an encoder and a decoder, both of which are connected by residuals. The encoder is composed of a spectral normalization module, and the decoder is composed of a separable adaptive normalization module and a spectral normalization module. The multi-head discriminator discriminates the predicted face result output by the face restoration network and the original face image; Step 4: Inputting the training set into the constructed face restoration network for training, specifically including: Step 41: inputting the mask image Im into the face parsing network to obtain the regional information of the face, sequentially performing 4 times, 18 times, 16 times and 32 times downsampling operations of the context path to obtain the first feature map D1, the second feature map D2, the third feature map D3 and the fourth feature map D4, obtaining the spatial feature map Is through the spatial path, and inputting the third feature map D3 and the fourth feature map D4 into the attention optimization module respectively to obtain the optimized feature map U3 and the optimized feature map U4; Step 42: Add the optimized feature map U3, the optimized feature map U4 and the fourth feature map D4 element by element to obtain a context feature map It, and finally input the spatial feature map Is and the context feature map It into a feature fusion module to obtain facial structure information Pm; Step 43: input the mask image Im of the training set into the encoder of the face restoration network. The encoder obtains the attribute features of the face through the spectral normalization module and the downsampling module connected in sequence. Specifically, the mask image generates a coding feature map through the first spectral normalization module, and then downsamples by 2 times and inputs it into the spectral normalization module of the next level to obtain coding feature maps F1, coding feature maps F2, coding feature maps F3 and coding feature maps F4 respectively. Step 44: The encoded feature map F4 is upsampled by 2 times, and then residually connected and summed with the encoded feature map F3, and then input into the spectral normalization module of the decoder to obtain a decoded feature map F1m; the decoded feature map F1m and the facial structure information Pm are input into the separable adaptive normalization module to obtain an intermediate weighted feature map W1; Then, the intermediate weight feature map W1 is upsampled by 2 times, summed and fused with the encoding feature map F2 through residual connection, and then input into the next spectral normalization module to obtain the decoding feature map F2m. The decoding feature map F2m and the facial structure information Pm are input into the next separable adaptive normalization module to generate the intermediate weight feature map W2. Finally, the intermediate weight feature map W2 is upsampled by 2 times, summed and fused with the encoding feature map F1 through residual connection, and then input into the next spectral normalization module to obtain the decoding feature map F3m. The decoding feature map F3m and the facial structure information Pm are input into the next separable adaptive normalization module to obtain the output feature map Fout. Step 5: The output feature map Fout generated in step 44 is used as a predicted face image and input into a multi-head discriminator for discrimination. The multi-head discriminator uses loss to bring the distribution of the predicted face image closer to that of the original face image. Step 6: Step 4 and step 5 are trained sequentially for a preset number of times, and the model weight is retained every 10 rounds. After the training is completed, the test set is passed to the trained face restoration network for testing to determine whether the objective evaluation index of the face restoration currently tested is stable, and whether the restoration result is successfully generated through subjective visual observation. If so, the current model parameters are saved, otherwise skip and test the next model weight; Step 7: Select the network with the best performance during the test, then feed the validation set into the network, calculate the objective evaluation index, and save the final predicted face image.

2. The adaptive face restoration method as claimed in claim 1, characterized in that: The data processing of the feature fusion module in step 42 is as follows: Step 421: Input the context feature map It and the spatial feature map Is into the feature fusion module, first perform feature splicing to extract spatial and context information, and then input the first branch and the second branch; Step 422: the first branch obtains global information through global pooling, and then obtains local information through sequentially connected 1×1 convolution, relu activation and 1×1 convolution, sigmoid activation to obtain a fused feature map I1; Step 423: The second branch obtains the fused feature map I2 through batch normalization, 1×1 convolution, and relu activation, and then multiplies the fused feature map I1 element by element to obtain the fused feature map I3, and then sums the features of the fused feature map I3 and the fused feature map I2 to obtain the facial structure information Pm.

3. The adaptive face restoration method as claimed in claim 2, characterized in that: Step 44 may be used to separate the data processing of the adaptive normalization module as follows: Step 441: In the separable adaptive normalization module, the input decoded feature map F1m and the face structure information Pm are sent to the separable adaptive normalization layer; Step 442: In the separable adaptive normalization layer, the face structure information Pm first passes through a 1×1 convolution to extract features, and then passes through two 1×1 convolutions in parallel to balance the distribution of the image, and respectively extracts the mean value and variance of the face structure information Pm; Step 443: In the separable adaptive normalization layer, the decoded feature map F1m is first normalized to a standard normal distribution, then multiplied by the variance of the face structure information Pm, and then added to its average value to form an intermediate feature map X1; Step 444: The intermediate feature map X1 enters the next separable adaptive normalization layer through relu activation and convolution and combines with the face structure information Pm to obtain the intermediate feature map X2, which is again activated and convolved through relu and residually connected with the decoded feature map F1m to generate the intermediate weighted feature map W1; Similarly, the decoded feature map F2m and the facial structure information Pm are input into the separable adaptive normalization module to obtain the intermediate weighted feature map W2.

4. An adaptive face restoration device based on facial attribute information prediction, characterized in that: The device is used to implement the adaptive face restoration method according to claim 1, comprising a data set preprocessing module, a face analysis module, a face restoration module and a multi-head discriminator, wherein: The data set preprocessing module is used to mask the complete face to obtain a mask image Im, and divide it into a training set, a validation set and a test set according to the ratio; The facial analysis module is used to obtain complete facial attribute information from the mask image Im, including a spatial path, a context path and a feature fusion module, obtains spatial features through the spatial path, obtains context features through the context path, and outputs facial structure information Pm through the feature fusion module; The facial restoration module includes a spectral normalization module and a separable adaptive normalization module, which is used to adaptively extract the features of the facial structure information Pm and the mask image Im to guide the generation of the facial structure; The multi-head discriminator is used to discriminate the predicted face result and the original face image output by the face restoration module, obtain the identity information of the predicted face image and the original face image through the identity extractor, and use the identity loss constraint to ensure the accuracy of the identity during the generation process.

Images