A face image restoration method based on contour and semantic guidance

Abstract

The invention discloses a facial image restoration method based on contour and semantic guidance, comprising the following steps: S1: preprocessing a facial image to obtain a facial image to be restored; S2: graying the facial image to be restored in step S1 to obtain a facial gray image to be restored; S3: extracting facial contour information from the facial gray image to be restored in S2 through an edge detection algorithm, thereby obtaining a facial contour image to be restored; S4: establishing a contour restoration model based on a multi-scale residual block of an adversarial network structure, inputting the facial gray image to be restored in step S2 and the facial contour image to be restored in step S3 into the contour restoration model, and obtaining a facial contour restoration image; S5: establishing an image restoration model based on an attention pyramid and a multi-scale residual block of an adversarial network structure, inputting the facial image to be restored in step S1 and the facial contour restoration image in step S4 into the image restoration model, and obtaining a restored complete image.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a face image restoration method based on contour and semantic guidance. Background Art

[0002] In practical applications, due to active occlusions such as glasses and accessories, as well as obstacles in the environment, it is often impossible to collect facial images with complete facial features, resulting in low face recognition accuracy. Compared with ordinary image restoration, the face has stronger semantics and more complex texture details. It is necessary not only to consider the rationality of the face structure, but also to retain the character information during the restoration process. Therefore, due to the particularity of face image restoration, it is still a very difficult task to generate clear, reasonable and accurate face images through image restoration algorithms.

[0003] Before deep learning was widely used in the field of image restoration, traditional methods can be divided into two categories: diffusion-based and patch-based image restoration methods. Diffusion-based methods use the edge information of the area to be repaired to determine the diffusion direction and diffuse known information into the edge, such as the TV-Stokes method. Sample-based methods iteratively search for image blocks with similar textures from known areas to fill in missing areas, such as the PatchMatch method. However, these methods rely too much on known information and cannot achieve satisfactory results when repairing some complex and non-replicable image structures.

[0004] With the rapid development of deep learning, methods based on generative adversarial networks have become mainstream in the field of image restoration. Pathak et al. introduced the encoding-decoding network structure into image restoration work and proposed the Context Encoder network for the first time in combination with the adversarial idea of ​​GAN; Iizuka et al. introduced the local-global dual discriminator based on the Context Encoder and proposed the GLCIC network using dilated convolution; Yu et al. proposed the deepfill network, which borrows or copies feature information from known background patches through the contextual attention mechanism to generate missing foreground patches; Nazeri et al. adopted a two-stage restoration network, first predicting the complete edge information of the image and then using it as a structural prior to guide content restoration; Wang et al. proposed a two-stage visual consistency network VCNet that includes a mask prediction module and a robust restoration module. Compared with traditional methods, image restoration methods based on deep learning have powerful image feature expression learning capabilities and data probability distribution fitting capabilities. Although existing methods have good restoration effects on most images, the restoration effect on face images is not satisfactory. For face restoration tasks, deep learning methods are still highly random, and existing methods still have great shortcomings in achieving image structure coherence, clear semantics, and clear texture. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a facial image restoration method based on contour and semantic guidance, which solves the problems of blurred boundaries, inconsistent texture and semantics, and unclear restoration details when restoring images, and effectively improves the accuracy of face recognition. This method obtains the facial contour map as structural prior information through the contour restoration network, which improves the rationality of the position and shape of facial parts; uses the attention pyramid structure to reconstruct the complete semantics of different levels from deep to shallow in the encoding process as prior information to guide the decoding process, which improves the consistency of vision and semantics; uses a multi-scale fusion residual structure to enhance the feature extraction ability of the network and improve the image restoration quality.

[0006] To achieve the above object, the present invention adopts the following technical solution:

[0007] A face image restoration method based on contour and semantic guidance comprises the following steps:

[0008] S1: preprocessing the input face image, scaling or stretching it to a uniform size, and obtaining the face image to be repaired;

[0009] S2: grayscale the face image to be restored obtained in step S1 to obtain a grayscale image of the face to be restored;

[0010] S3: extracting facial contour information from the grayscale image of the face to be repaired obtained in S2 through an edge detection algorithm, thereby obtaining a binary facial contour image to be repaired;

[0011] S4: establishing a contour restoration model based on a multi-scale residual block of an adversarial network structure, wherein the contour restoration model is composed of a generator and a discriminator, and a multi-scale residual structure is introduced into the generator, and the grayscale image of the face to be restored obtained in step S2 and the contour image of the face to be restored obtained in step S3 are input into the contour restoration model based on the multi-scale residual block to obtain a complete face contour restoration image;

[0012] S5: Establish an image restoration model based on an attention pyramid and multi-scale residual blocks with an adversarial network structure. The image restoration model consists of a generator and a discriminator, and the attention pyramid structure and the multi-scale residual structure are introduced into the generator. The face image to be restored obtained in step S1 and the face contour restoration image in step S4 are input into the image restoration model based on the attention pyramid and multi-scale residual blocks to restore the missing area of ​​the face image and obtain a restored complete image.

[0013] As a further improvement of the present invention: the multi-scale residual structure uses a multi-scale fusion residual block to use dilated convolutions of different scales to extract feature information under different receptive fields and realize multi-scale feature fusion. The multi-scale fusion residual block uses dilated convolutions with dilation factors of 1, 2, 4, and 8 respectively through four branches to extract features x1, x2, x3, and x4 of different scales from the input feature information, and associates different sparse multi-scale feature combinations through the adjacent accumulation method, and then performs feature fusion.

[0014] As a further improvement of the present invention: x1, x2, x3, and x4 in the extracted multi-scale features are combined and associated using ordinary convolution to obtain features X1, X2, X3, and X4, and then the four groups of features are fused through 1×1 ordinary convolution, and the formula is as follows:

[0015]

[0016] y=Conv 1×1，d =1(X1+X2+X3+X4),

[0017] Among them, y is the feature information of the output of the multi-scale fusion residual block, Conv 3×3，d＝1 and Conv 1×1，d＝1 They represent ordinary convolutions with kernel sizes of 3 and 1 and dilation factor of 1 respectively.

[0018] As a further improvement of the present invention: the contour restoration model converts the grayscale image I of the portrait in，gray、Contour map C in And the mask map M of the missing area is used as input, and the repaired contour image is obtained through the contour repair network:

[0019] C out =G1(I in，gray , C in , M),

[0020] Among them, C out It is the complete portrait contour map restored by the network, and G1 represents the generator of the contour restoration model.

[0021] As a further improvement of the present invention: the attention pyramid structure fills in the feature information of the missing area through a pyramid-like attention from deep to shallow during the generator encoding process, so that the semantically complete feature maps reconstructed at different levels guide the generator decoding process through jump links.

[0022] As a further improvement of the present invention: the attention pyramid structure uses the similarity inside and outside the missing region obtained in the high-level feature map to fill the low-level feature map through the attention module during the encoding process, and its formula is as follows:

[0023]

[0024] Among them, f l ,f f-1 ,…,f 1 It represents the feature map from deep to shallow in the encoding process, and attention represents the attention module.

[0025] As a further improvement of the present invention: the attention module uses a 3×3 ordinary convolution to extract patches inside and outside the missing area and calculates the cosine similarity:

[0026]

[0027] in, represents a patch centered at (i,j) extracted from the background region of the high-level feature map,

[0028] Represents a patch centered at (i′, j′) extracted from the foreground area of ​​the high-level feature map. The similarity is then measured by softmax to obtain the attention score of each patch:

[0029]

[0030] The missing area is reconstructed by weighting the attention score obtained in the high-level feature map to its adjacent low-level feature map, and the formula is as follows:

[0031]

[0032] in, represents a patch centered at (i,j) extracted from the background region of the low-level feature map, represents the patch centered at (i′, j′) extracted from the foreground region of the low-level feature map, and N is the number of patches extracted from the background region.

[0033] As a further improvement of the present invention: the image restoration model converts the portrait I in Composite contour map C output by the contour repair model comp As input, the restored complete face image is obtained through the image restoration network:

[0034]

[0035] Among them, I out is the complete face image restored by the network, G2 represents the generator of the image restoration model, and C comp =C gt e(1-M)+C out e M,C gt is the contour map of the real face image, and e represents element-by-element multiplication.

[0036] As a further improvement of the present invention: the edge detection algorithm is a Canny edge detection algorithm, which is used to extract and output contour information of a face image.

[0037] Different from the prior art, the above technical solution has the following advantages:

[0038] 1. Use the contour restoration model to repair the facial contour, and use it as prior information to guide the face restoration work of the image restoration model. This can improve the rationality of the position and shape of important facial parts such as facial features after restoration when a large area of ​​the image is missing, so that the image has a better structural restoration effect.

[0039] 2. The image restoration model introduces an attention pyramid structure. During the encoding process, a pyramid-like attention mechanism from deep to shallow is used to fill in the content of the missing area to obtain a semantically complete feature map, which is used as a prerequisite to constrain the decoding and restoration of the image, thereby improving the semantic credibility of the restored image and ensuring the visual and semantic consistency of the missing area.

[0040] 3. A multi-scale fusion residual structure is used for feature extraction. The combination of dilated convolutions of different scales enables the network to expand the area of ​​the receptive field without increasing the number of parameters, thereby enhancing the feature extraction capability and improving the image restoration quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the face image restoration method based on contour and semantic guidance;

[0042] Figure 2 It is the network structure diagram of the contour repair model;

[0043] Figure 3 It is the network structure diagram of the image restoration model;

[0044] Figure 4 It is the structural diagram of the multi-scale fusion residual block;

[0045] Figure 5 This is a diagram of the working mechanism of attention in the attention pyramid;

[0046] Figure 6 This is the effect diagram of face image restoration under different missing areas;

[0047] Figure 7 This is a comparison of face recognition accuracy before and after restoration under different missing areas; DETAILED DESCRIPTION

[0048] In order to explain the technical content, structural features, achieved objectives and effects of the technical solution in detail, the following is a detailed description in conjunction with specific embodiments and accompanying drawings.

[0049] The present invention proposes a face image restoration method based on contour and semantic guidance, comprising the following steps:

[0050] S1: preprocess the input face image, scale or stretch the input image, adjust it to a uniform size of 256×256, and obtain the face image to be repaired;

[0051] S2: grayscale the face image to be restored obtained in step S1 to obtain a grayscale image of the face to be restored;

[0052] S3: extracting facial contour information from the grayscale image of the face to be repaired obtained in S2 through the Canny edge detection algorithm, thereby obtaining a binary facial contour image to be repaired;

[0053] S4: Establishing a contour restoration model based on a multi-scale residual block of an adversarial network structure, wherein the contour restoration model consists of a generator and a discriminator, and a multi-scale residual structure is introduced into the generator, and further feature extraction is achieved by constructing a multi-scale fusion residual block;

[0054] See also Figure 4As shown in the figure, it is the structure diagram of the multi-scale residual block. The channel of the input feature is reduced to 1 / 4 using ordinary convolution with a convolution kernel of 3×3. The feature information x1, x2, x3, and x4 under different receptive fields are extracted through four branches using dilated convolution with dilation factors of 1, 2, 4, and 8 respectively. Then, the four sets of multi-scale features extracted are combined and associated using 3×3 ordinary convolution using the adjacent accumulation method to obtain features X1, X2, X3, and X4. Then, X1, X2, X3, and X4 are simply concat and then fused using 1×1 ordinary convolution. The feature fusion process is shown in the following formula:

[0055]

[0056] y=Conv 1×1,d＝1 (X1+X2+X3+X4),

[0057] Where y is the feature information of the output of the multi-scale fusion residual block, Conv 3×3,d＝1 and Conv 1×1,d＝1 They represent ordinary convolutions with kernel sizes of 3 and 1 and dilation factor of 1 respectively;

[0058] After the contour restoration model based on the multi-scale residual block is established, the grayscale image of the face to be restored obtained in step S2 and the contour image of the face to be restored obtained in step S3 are input into the contour restoration model, and the edge information of the unknown area is predicted with the image mask as a precondition to obtain a complete face contour restoration image;

[0059] See also Figure 2 As shown in the figure, it is the structure diagram of the contour restoration model, in which k is the convolution kernel size, n is the number of output channels, and s is the step size of the convolution operation. TConv is the transposed convolution for upsampling, and Residual block is the multi-scale fusion residual block;

[0060] The grayscale image I of the face image in,gray 、Contour map C in The mask image M of the missing area is used as input, and the restored face contour image C is obtained through the contour restoration network G1. out :

[0061] C out =G1(I in,gray ,C in ,M),

[0062] The contour restoration model uses adversarial loss and feature matching loss to optimize model performance during training. The total loss function of contour restoration is as follows:

[0063]

[0064]

[0065]

[0066] in is the total loss of the contour restoration model, L adv,1 is the adversarial loss of the contour inpainting model, L FM is the feature matching loss, λ adv,1 and λ FM are the regularization parameters of the adversarial loss and feature matching loss, respectively. D1 represents the discriminator of the contour restoration model. C gt is the contour map of the real face image, I gt,gray is the grayscale image of the real face image, represents the activation map of the discriminator at layer i, N i represents the number of elements of the i-th activation map of the discriminator, and K is the number of convolutional layers of the discriminator;

[0067] S5: An image restoration model based on an attention pyramid and a multi-scale residual block of an adversarial network structure is established. The image restoration model consists of a generator and a discriminator, and the attention pyramid structure and the multi-scale residual structure are introduced in the generator. Further feature extraction is achieved by constructing an attention pyramid and a multi-scale fusion residual block. The multi-scale residual block structure introduced in the image restoration model is consistent with that described in S4.

[0068] In the present invention, the attention pyramid structure introduced in the image restoration model fills in the feature information of the missing area through a pyramid-like attention from deep to shallow during the generator encoding process, and then enables the semantically complete feature graphs reconstructed at different levels to guide the generator decoding process through a skip link method;

[0069] See also Figure 5 As shown, it is a schematic diagram of the attention working mechanism in the attention pyramid. During the encoding process, the attention pyramid structure obtains the similarity inside and outside the missing area from the high-level feature map through the attention mechanism and uses it to fill the content of the low-level feature map. The formula is as follows:

[0070]

[0071] Among them, f l ,f f-1 ,…,f 1 Represents the feature map from deep to shallow in the encoding process, and attention represents the attention module;

[0072] This attention module uses a 3×3 normal convolution to extract patches inside and outside the missing area and calculates the cosine similarity between the inner and outer patches:

[0073]

[0074] in, represents a patch centered at (i,j) extracted from the background region of the high-level feature map,

[0075] Represents a patch centered at (i′, j′) extracted from the foreground area of ​​the high-level feature map. The similarity is then measured by softmax to obtain the attention score of each patch:

[0076]

[0077] The missing area is reconstructed by weighting the attention score obtained in the high-level feature map to its adjacent low-level feature map, and the formula is as follows:

[0078]

[0079] in, represents a patch centered at (i,j) extracted from the background region of the low-level feature map,

[0080] represents the patch centered at (i′, j′) extracted from the foreground region of the low-level feature map, and N is the number of patches extracted from the background region;

[0081] The image restoration model uses adversarial loss, l1 reconstruction loss, perceptual loss and style loss to optimize model performance during training. The total loss function of image restoration is as follows:

[0082]

[0083]

[0084]

[0085] L style =E[||ψ i (I gt )-ψ i (I out )||1],

[0086] in, is the total loss of the image restoration model, L adv,2 is the adversarial loss of the image inpainting model, is the l1 reconstruction loss, L perc is the perceptual loss, L style is the style loss, D2 represents the discriminator of the contour restoration model, and λ adv,2 ,l1 , perc and λ style are the regularization parameters of adversarial loss, l1 reconstruction loss, perceptual loss and style loss, respectively. gt is the real face image (groundtruth), φ i Represents the activation map of relu1_1, relu2_1, relu4_1, relu4_1, relu5_1 of the pre-trained VGG19 network, N i represents the number of elements in the i-th activation map, ψ i Represented by the activation map φ i The Gram matrix formed.

[0087] The image restoration model based on the attention pyramid and the multi-scale residual block is established. The face image to be restored obtained in step S1 and the face contour restoration image in step S4 are input into the image restoration model to restore the missing area of ​​the face image and obtain the restored complete image.

[0088] See also Figure 3 As shown in the figure, it is the structure diagram of the image restoration model. in Composite contour map C output by the contour repair model comp As input, the restored complete face image I is obtained through the image restoration network G2. out :

[0089] I out =G2(I in ,C comp ),

[0090] Among them C comp =C gt e(1-M)+C out e M, e means element-by-element multiplication.

[0091] Experimental results and analysis

[0092] In order to test the restoration performance of the algorithm, the algorithm of the present invention is trained and tested on the CelebA face dataset and the CelebA-HQ face dataset. CelebA is a large face attribute dataset with 202,599 face images of 10,177 identities. CelebA-HQ is a face dataset generated by selecting 60,000 images from the CelebA dataset and enhancing their clarity. In the experiment, 200,000 images from the CelebA dataset and 48,000 training images from the CelebA-HQ dataset were selected to form the training set, and the remaining images formed the test set.

[0093] Experimental environment and parameters

[0094] The experimental environment of the present invention is a graphics workstation equipped with two GTX 2080Ti 11G graphics cards and an Intel Xeon Silver 4110 2.10GHz 8-core CPU. Based on the Ubuntu 16.04 operating system, the Python programming language is used to build a network model on the PyTorch deep learning framework to implement algorithm training and testing. During the training process, the batch size of each batch is 4, and the maximum number of iterations is set to 1e 6 , Adam optimizer is used to optimize the model with parameters β1=0, β2=0.9.

[0095] Performance Evaluation

[0096] Six groups of images were randomly selected from the image test set as experimental objects, each group of 200 face images, corresponding to six different areas of image content loss of 10% to 60%. The algorithm of the present invention was used to repair the six groups of images respectively, and the peak signal-to-noise ratio and structural similarity between the repaired images and the real images were calculated to reflect the repair effect of the algorithm on face images with different loss degrees.

[0097] Peak signal-to-noise ratio (PSNR) is a reference value for measuring the image quality between the maximum signal and the background noise. The larger its value, the less distortion the restored image has. Structural similarity (SSIM) is an indicator that measures the structural similarity between two images in terms of brightness, contrast, and structure. The larger the value, the more similar the restored image is to the real image. The expressions for peak signal-to-noise ratio and structural similarity are:

[0098]

[0099]

[0100] Where MAX (x,y) represents the maximum value of the image pixels, and MSE(x,y) represents the mean square error between the real image x and the restored image y. l(x,y), c(x,y) and s(x,y) reflect the brightness, contrast and structure between the two images respectively. x , μ y are the means of x and y respectively, are the variances of x and y, σ xy is the covariance of x and y. c1, c2, c3 are three constants, among which Table 2 shows the PSNR and SSIM values ​​of the face restoration images of the present invention at different degrees of loss.

[0101] Table 2 PSNR and SSIM values ​​of restored face images

[0102]

[0103] Compared with previous image restoration methods, the method of the present invention demonstrates a restoration effect comparable to that of previous excellent restoration methods when the missing area of ​​the face image is less than 20%. For images with a missing area greater than 20%, especially an area greater than 40%, the face images restored by this method have obvious improvements in the evaluation indicators PSNR and SSIM, indicating that this method has a certain degree of improvement in image restoration quality.

[0104] See also Figure 6 and Figure 7 Shown are the facial image restoration effects under different missing areas and the comparison of face recognition accuracy before and after restoration. Figure 6 The good restoration ability of the method of the present invention can be demonstrated from the visual effect. Even when the facial features are missing by more than 50% in a large area, a restoration result with clear semantics and reasonable structure can still be obtained. The FaceNet face recognition algorithm was used to conduct face recognition experiments on the above 6 groups of images before and after restoration. Figure 7 It can be seen that when the missing area is less than 10%, since the recognition rate has reached a high level before restoration, the improvement after restoration is not large, but it is still about 8%; when the missing area exceeds 10%, the recognition rate before restoration begins to drop sharply, but the recognition rate drops less after restoration. When the missing area is less than 40%, the recognition rate can still maintain about 90% after restoration; when the missing area exceeds 50%, the face recognition algorithm can no longer normally recognize the face before restoration, so the recognition accuracy is extremely low. Although the accuracy is still not high after restoration, it is about 5 times higher than before restoration.

[0105] The face image restoration method based on contour and semantic guidance proposed in the present invention first repairs the face contour in the algorithm to improve the rationality of the position and shape of the facial parts of the restored image; introduces an attention pyramid structure into the image restoration network, and uses the attention module in the encoder to reconstruct semantic features from deep to shallow as prior information to guide the decoding work, thereby enhancing the visual and semantic consistency of the restored face image; adopts a multi-scale fusion residual structure to enhance the feature extraction capability of the network and improve the image restoration quality.

[0106] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "include..." or "comprise..." do not exclude the existence of other elements in the process, method, article or terminal device including the elements. In addition, in this article, "greater than", "less than", "exceed" and the like are understood to exclude the number itself; "above", "below", "within" and the like are understood to include the number itself.

[0107] Although the above embodiments have been described, once those skilled in the art know the basic creative concepts, they can make additional changes and modifications to these embodiments. Therefore, the above description is only an embodiment of the present invention and does not limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A face image restoration method based on contour and semantic guidance, characterized in that: The following steps are involved: S1: preprocessing the input face image, scaling or stretching it to a uniform size, and obtaining the face image to be repaired; S2: grayscale the face image to be restored obtained in step S1 to obtain a grayscale image of the face to be restored; S3: extracting facial contour information from the grayscale image of the face to be repaired obtained in S2 through an edge detection algorithm, thereby obtaining a binary facial contour image to be repaired; S4: establishing a contour restoration model based on a multi-scale residual block of an adversarial network structure, wherein the contour restoration model is composed of a generator and a discriminator, and a multi-scale residual structure is introduced into the generator, and the grayscale image of the face to be restored obtained in step S2 and the contour image of the face to be restored obtained in step S3 are input into the contour restoration model based on the multi-scale residual block to obtain a complete face contour restoration image; S5: Establish an image restoration model based on an attention pyramid and multi-scale residual blocks with an adversarial network structure. The image restoration model consists of a generator and a discriminator, and the attention pyramid structure and the multi-scale residual structure are introduced into the generator. The face image to be restored obtained in step S1 and the face contour restoration image in step S4 are input into the image restoration model based on the attention pyramid and multi-scale residual blocks to restore the missing area of ​​the face image and obtain a restored complete image.

2. The method for face image restoration based on contour and semantic guidance according to claim 1, characterized in that: The multi-scale residual structure uses a multi-scale fusion residual block and uses dilated convolutions of different scales to extract feature information under different receptive fields and realize multi-scale feature fusion. The multi-scale fusion residual block uses dilated convolutions with dilation factors of 1, 2, 4, and 8 respectively to extract features x1, x2, x3, and x4 of different scales from the input feature information through four branches, and associates different sparse multi-scale feature combinations through the adjacent accumulation method, and then performs feature fusion.

3. The method for face image restoration based on contour and semantic guidance according to claim 2, characterized in that: The extracted multi-scale features x1, x2, x3, and x4 are combined and associated using ordinary convolution to obtain features X1, X2, X3, and X4. Then, the four sets of features are fused through 1×1 ordinary convolution. The formula is as follows: y=Conv 1×1,d＝1 (X1+X2+X3+X4), Among them, y is the feature information of the output of the multi-scale fusion residual block, Conv 3×3,d＝1 and Conv 1×1,d＝1 They represent ordinary convolutions with kernel sizes of 3 and 1 and dilation factor of 1 respectively.

4. The method for face image restoration based on contour and semantic guidance according to claim 1, characterized in that: The contour restoration model converts the grayscale image I of the portrait in,gray 、Contour map C in And the mask map M of the missing area is used as input, and the repaired contour image is obtained through the contour repair network: C out =G1(I in,gray ,C in ,M), Among them, C out It is the complete portrait contour map restored by the network, and G1 represents the generator of the contour restoration model.

5. The method for face image restoration based on contour and semantic guidance according to claim 1, characterized in that: The attention pyramid structure fills in the feature information of the missing area through a pyramid-like attention from deep to shallow during the generator encoding process, so that the semantically complete feature maps reconstructed at different levels guide the generator decoding process through jump links.

6. The method for facial image restoration based on contour and semantic guidance according to claim 5, characterized in that: The attention pyramid structure uses the attention module to fill the low-level feature map with the similarity of the missing area inside and outside the high-level feature map during the encoding process. The formula is as follows: where f l ,f f-1 ,…,f 1 , represents the feature map from deep to shallow in the encoding process, and attention represents the attention module.

7. The method for face image restoration based on contour and semantic guidance according to claim 6, characterized in that: The attention module uses a 3×3 normal convolution to extract patches inside and outside the missing region and calculates their cosine similarity: in, represents a patch centered at (i,j) extracted from the background region of the high-level feature map, represents a patch centered at (i′, j′) extracted from the foreground area of ​​the high-level feature map, and then the similarity is measured by softmax to obtain the attention score of each patch: The missing area is reconstructed by weighting the attention score obtained in the high-level feature map to its adjacent low-level feature map, and the formula is as follows: in, represents a patch centered at (i,j) extracted from the background region of the low-level feature map, represents the patch centered at (i′, j′) extracted from the foreground region of the low-level feature map, and N is the number of patches extracted from the background region.

8. The method for face image restoration based on contour and semantic guidance according to claim 1, characterized in that: The image restoration model transforms the portrait I in Composite contour map C output by the contour repair model comp As input, the restored complete face image is obtained through the image restoration network: I out =G2(I in ,C comp ), Among them, I out is the complete face image restored by the network, G2 represents the generator of the image restoration model, and C comp =C gt e(1-M)+C out e M,C gt is the contour map of the real face image, and e represents element-by-element multiplication.

9. The method for face image restoration based on contour and semantic guidance according to claim 1, characterized in that: The edge detection algorithm is a Canny edge detection algorithm, which is used to extract and output contour information of a face image.

Images