Face image inpainting method based on structure and texture pair generation

Abstract

The application discloses a face image repairing method based on structure and texture dual generation, relates to the technical field of image repairing, and realizes repairing of a damaged face image through a deep learning method, solves the problem of inconsistency between structure and texture after face image repairing, and improves the repairing effect of a large-area damaged image, and comprises the following steps: step S1, pre-processing of an input image to obtain a face image to be repaired; step S2, establishing a face image repairing model based on structure and texture dual generation, and inputting the image obtained in step S1 into the image repairing model for training; step S3, continuously iteratively training until the network finally converges, and obtaining the face image repairing model; and step S4, inputting a damaged face image into the trained face image repairing model to obtain a repaired face image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image inpainting, more particularly to a face image inpainting method based on structure and texture pair generation. BACKGROUND

[0002] Image inpainting aims to restore the pixels of the damaged area in the image and make the filled image consistent with the original image as much as possible in visual and semantic level. It is not only crucial in computer vision tasks, but also an important cornerstone for other image processing tasks. Face inpainting, as an important branch, plays an important role in practical applications. Compared with ordinary image inpainting, face has stronger semantics and more complex texture details, not only the rationality of face structure needs to be considered, but also the information of the person needs to be preserved in the inpainting process.

[0003] From the early traditional methods to the current deep learning-based methods, image inpainting has made great progress. Traditional methods are only suitable for the inpainting of single simple small area missing pictures, and lack of semantic consistency. Therefore, deep learning-based methods have become the mainstream.

[0004] Pathak first proposed Context Encoders, which uses an encoder-decoder network to extract features and output the reconstruction result, and it is also the first inpainting method based on GAN; Iizuka et al. introduced a local-global dual discriminator on the basis of Context Encoder, and proposed GLCIC network using dilated convolution; Yu et al. proposed deepfill network, which borrows or copies feature information from known background patches through context attention mechanism to generate missing foreground patches; Nazeri et al. designed EdgeConnect using a two-stage model, which first generates an edge hallucination map for the irregular missing area by an edge generator as an a priori result, and then fills the missing area based on this edge hallucination map using an image inpainting network.

[0005] However, these methods do not simultaneously use structure and texture features, resulting in inconsistent structure and texture of the output image. Defect repair involves both high-level semantic knowledge and low-level pixel information. Only by highly structuring the fusion of the two parts of information can the image repair level of the human visual system be approached. To this end, Guo et al. proposed a novel dual-flow network for image inpainting, which models structure-constrained texture synthesis and texture-guided structure reconstruction in a coupled manner to obtain more reasonable output. Although this method improves the consistency between structure and texture, there are still two problems, 1) the relationship between structure and texture is not fully considered, resulting in limited consistency between them. 2) Lack of consideration of global context reasoning and global and local pixel continuity, resulting in defects such as structure distortion and texture blur in the repaired image, especially when the area of damage is large. Based on these two shortcomings, the present scheme proposes a face image inpainting method based on structure and texture dual generation. This method can enhance the texture and structure consistency of the repaired face image while achieving large-area damaged face image repair. SUMMARY

[0006] The purpose of the present application is to solve the above technical problems, and the present application provides a face image inpainting method based on structure and texture dual generation.

[0007] In order to achieve the above purpose, the present application specifically adopts the following technical scheme:

[0008] A face image inpainting method based on structure and texture dual generation, comprising the following steps:

[0009] Step S1: pre-processing the input image to obtain a face image to be repaired;

[0010] Step S2: establishing a face image inpainting model based on structure and texture dual generation, and inputting the image obtained in step S1 into the image inpainting model for training;

[0011] Step S3: continuously iteratively training until the network finally converges to obtain a face image inpainting model;

[0012] Step S4: inputting a damaged face image into the trained face image inpainting model to obtain a repaired face image.

[0013] As an optional technical scheme, in step 2, the face image inpainting model is a structure of a generative adversarial network, which consists of a generator and a discriminator;

[0014] The generator includes a dual encoder-decoder and a feature fusion part, and the discriminator consists of a texture discriminator and a structure discriminator.

[0015] As an optional technical solution, the convolutional layer of the dual encoder-decoder adopts gated convolution to encode and decode features, and a batch normalization layer is added after each gated convolution layer, denoted as:

[0016] Gating=∑∑W g ·I

[0017] Feature=∑∑W f ·I

[0018] Output=BN(φ(Feature)⊙σ(Gating))

[0019] Wherein, I represents a feature map; Gating represents gating; Feature represents a feature map after convolution; Output represents a final output feature map, W g and W f represent different convolution kernels respectively; φ is a LeakyReLU activation function, σ is a Sigmoid activation function, and ⊙ represents element-level multiplication. Compared with hard gating, the gating value of gated convolution is between 0 and 1. The closer the gating value is to 1, the more effective pixels there are. BN represents batch normalization.

[0020] As an optional technical solution, the dual encoder-decoder,

[0021] In the encoding stage, the left and right two encoders respectively receive the damaged image and the damaged structure image to encode the texture and structure features,

[0022] In the decoding stage, the texture decoder synthesizes the structure-constrained texture by borrowing the structure features from the structure encoder, and the structure decoder restores the texture-guided structure by obtaining the texture features from the texture encoder.

[0023] As an optional technical solution, the discriminator is a double-flow discriminator with texture branch and structure branch, and the structure branch of the discriminator further has an additional edge detector for edge extraction, wherein the two discriminator trunks are composed of ordinary convolution, and the edge detector is composed of a convolutional neural network residual block.

[0024] As an optional technical solution, the preprocessing of step SS1 is:

[0025] Firstly, the size of the image is adjusted, and the image is adjusted to 256x256 size through cropping and padding,

[0026] Then, a binary mask M is obtained from the irregular mask dataset provided by NVIDIA to artificially damage the image, and a damaged image is obtained; the damaged image is processed to obtain a damaged grayscale image;

[0027] Finally, the damaged gray image is extracted by the Canny edge detection algorithm to obtain the damaged edge image.

[0028] As an optional technical solution, the step 3 adopts the CelebA-HQ dataset for training, including training images and test images, and the experimental equipment adopts the NVIDIA V100, and the whole model is realized by PyTorch; when the model is trained, the batch size is set to 8, and the Adam optimizer is used for optimization.

[0029] As an optional technical solution, first, the initial training is performed by using a learning rate of 2x10 -4 , and then the model is fine-tuned by using a learning rate of 5x10 -5 ; the learning rate fine-tuned model is trained by using a joint loss, including a reconstruction loss, a perception loss, a style loss and an adversarial loss.

[0030] As an optional technical solution, the four loss functions are as follows:

[0031] The reconstruction loss function is L rec =E[||I out -I gt ||1]

[0032] Wherein E represents expectation, I out represents the generated picture, I gt represents the real picture, and ||·||1 represents the L1 norm

[0033] The perception loss function is:

[0034] The perception loss pre-trained on ImageNet by VGG-16 is used to simulate human visual perception of image quality. Wherein E represents expectation, I out represents the generated picture, I gt represents the real picture, ||·||1 represents the L1 norm, and φ i represents the activation map of the i-th pooling layer of Vgg16, and in the actual process, i∈[1,3].

[0035] The style loss function is:

[0036] Wherein E represents expectation, I out represents the generated picture, I gt represents the real picture, It represents the Gram matrix of the activation map φ i .

[0037] The adversarial loss function is:

[0038] Wherein, E represents expectation, G represents generator, D represents discriminator, I gt represents real picture, E gt represents real edge map, I out represents generated picture, E out represents generated edge map.

[0039] As an optional technical solution, in order to guide the dual encoder-decoder to generate structure and texture features, the intermediate loss is introduced on F s and F t

[0040] L inter =L structure +L texture =BCE(E gt ,P s (F s ))+l1(I gt ,P t (F t ))

[0041] Wherein, I gt represents real picture, E gt represents real edge map, P s and P t are mapping functions composed of convolution kernel residual blocks, which respectively map structure features F s and texture features F t to corresponding edge maps and RGB pictures.

[0042] The beneficial effects of the present application are as follows:

[0043] 1. The current image inpainting model cannot simultaneously and sufficiently use the information of structure and texture features for inpainting, resulting in the problem of inconsistency of structure and texture in the inpainted image, and the present application can fully utilize the structure and texture features of the image, and the consistency of the inpainting result is better.

[0044] 2. The existing research still has the problems of structure distortion or texture blur when repairing large area and irregular missing area, which is mainly due to the fact that the context of the image is not fully utilized, resulting in insufficient connection from local features to overall consistency. The present application can fully utilize the context information of the image, and has better repair effect when the large area is damaged. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is the flow chart of the method of the present application.

[0046] Figure 2 is the network structure diagram of the method of the present application. ​

[0047] Figure 3 An adaptive dual feature fusion module (ADFF) in the method generator of the present application.

[0048] Figure 4 A gated aggregated contextual transformations (GACT) module in the method generator of the present application

[0049] Figure 5 A qualitative comparison effect diagram of the method of the present application and other methods. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0052] EMBODIMENT

[0053] A face image inpainting method based on structure and texture dual generation, referring to Figure 1 , comprising the following steps:

[0054] Step S1: pre-processing the input image to obtain a face image to be repaired. First, adjust the size of the image, and adjust the image to 256x256 size by cropping and padding. Then, obtain a binary mask M from the irregular mask dataset provided by NVIDIA to artificially damage the image, obtaining a damaged image. The damaged image is grayed to obtain a damaged gray image, and finally the damaged gray image is extracted by the Canny edge detection algorithm to obtain the face contour information, obtaining a damaged edge map.

[0055] Step S2: establish a face image inpainting model based on structure and texture dual generation, input the image obtained in S1 into the image inpainting model for training.

[0056] The face image inpainting model based on structure and texture dual generation is as followsFigure 2 As shown, the model is based on the structure of a generative adversarial network, which consists of a generator and a discriminator. The generator includes a dual encoder-decoder and feature fusion, and the discriminator consists of a texture discriminator and a structure discriminator.

[0057] Specifically, the dual encoder-decoder adopts a connection mode similar to U-net, and the left and right two encoders respectively receive the damaged image and the damaged structure image to encode the texture and structure features. In the decoding stage, the texture decoder synthesizes the structure-constrained texture by borrowing the structure features from the structure encoder, and the structure decoder restores the texture-guided structure by obtaining the texture features from the texture encoder. Using this dual structure, the structure and texture can complement each other well, thereby improving the consistency of the texture and structure.

[0058] The convolutional layers of the dual encoder-decoder use gated convolution to encode and decode features. Compared with partial convolution, gated convolution learns features in an end-to-end manner and dynamically updates the mask, which can effectively adapt to the case of uneven pixel distribution, making the repair result clearer and more consistent with the semantic context. At the same time, a batch normalization layer is added after each gated convolution layer to prevent gradient vanishing during training. This operation can be represented as:

[0059] Gating=∑∑W g ·I

[0060] Feature=∑∑W f ·I

[0061] Output=BN(φ(Feature)⊙σ(Gating))

[0062] where I represents the feature map; Gating represents the gating; Feature represents the feature map after convolution; Output represents the final output feature map, W g and W f represent different convolution kernels; BN represents batch normalization; φ is the LeakyReLU activation function, σ is the Sigmoid activation function, and represents element-wise multiplication. Compared with hard gating, the gating value of gated convolution is between 0 and 1, and the closer the gating value is to 1, the more effective pixels there are.

[0063] In addition, the dual encoder-decoder of the present scheme introduces six Gated Aggregated Contextual Transformations (GACT) modules, which are embedded between the encoder and the decoder in a gated residual connection manner, and can capture long-distance contextual information and rich interested patterns. The GACT module is as follows: Figure 3As shown, the design of GACT module adopts the strategy of splitting, transforming, and aggregating. (i) Splitting: the input 256-channel feature map x1 is reduced to 4 64-channel sub-feature maps using 4 3x3 gated convolutions. (ii) Transforming: the convolution kernels of each gated convolution have different hole rates. Larger hole rates enable the convolution kernel to focus on larger areas of the input image, while convolution kernels with smaller hole rates focus on local patterns with smaller receptive fields. (iii) Aggregating: the 4 context-transformed features from different receptive fields are finally aggregated by channel dimension splicing and standard gated convolution to obtain the fused feature x2. In addition to this, the residual connection structure is also used. First, a 3x3 standard gated convolution and a Sigmoid operation are used on x1 to form a threshold g, and then the transformed fused feature and the original feature are gated and weighted to obtain the final output feature. The weighting formula is: x1xg+x2x(1-g).

[0064] After generating the complete structure feature and texture feature through the dual encoder-decoder part of the generator, an adaptive dual feature fusion module (ADFF) is used to further fuse the two features. He adaptively fuses the two semantic features by controlling the fusion ratio of texture and structure, so as to enhance the structure continuity and texture consistency, making the repair result more reasonable. The ADFF module is as shown in Figure 4 .

[0065] Specifically, the texture feature map output by the decoder is represented as F t , and the structure feature map is represented as F s . In order to construct texture-aware structure features, the soft gating G t is formulated as:

[0066] G t =σ(SE(g([F s ,F t ])))

[0067] where [·] represents channel dimension splicing, g(·) represents convolution with a convolution kernel size of 3. SE(·) represents a channel attention mechanism for obtaining important channel dimension information. σ(·) is a sigmoid activation function, and G t , the texture feature can be dynamically fused into the structure feature, and the fusion formula is as follows:

[0068]

[0069] where α is a learnable parameter, and ☉ and ⊕ represent pixel-wise multiplication and pixel-wise addition, respectively. The same method is used to calculate the texture feature aware of the structure. The fusion formula is as follows:

[0070] Gs = sigma (SE (h ([F s , F t ])))

[0071]

[0072] Finally, the texture and structure features are fused by the following formula to obtain the final fusion feature.

[0073] F b = SK (k ([F S ', F t ']))

[0074] Wherein SK is a convolution kernel attention mechanism, which can adaptively select a suitable convolution kernel, and is helpful to repair the consistency of image structure and texture.

[0075] Finally, the obtained fusion feature is sent to a Contextual Feature Aggregation (CFA) module, which generates more vivid details by modeling long-term spatial dependencies.

[0076] The discriminator is a double-flow discriminator with a texture branch and a structure branch, and the structure branch of the discriminator further has an additional edge detector for edge extraction. The two discriminator trunks are composed of ordinary convolution, and in order to improve the stability of the generative adversarial network, a PReLU normalization is also used. The edge detector is composed of a convolutional neural network residual block.

[0077] Step S3: through continuous iteration training, until the network finally converges, and a face image repair model is obtained.

[0078] The CelebA-HQ dataset is used for training, including 28000 training images and 2000 test images. The experimental equipment adopts NVIDIA V100, and the whole model is realized by PyTorch. When training the model, the batch size is set to 8, and the Adam optimizer is used for optimization. First, an initial training is performed by using a learning rate of 2x10 -4 , and then the model is fine-tuned by using a learning rate of 5x10 -5 .

[0079] The model is trained using a joint loss, including a reconstruction loss, a perception loss, a style loss and an adversarial loss, to obtain a visually real and semantically reasonable repair result.

[0080] Reconstruction loss: L rec = E [||I out -I gt ||1]

[0081] where E denotes expectation, I out denotes generated image, I gt denotes real image, ||·||1denotes L1 norm

[0082] Perceptual loss:

[0083] The perceptual loss pre-trained by VGG-16 on ImageNet is used to simulate human visual perception of image quality. where E denotes expectation, I out denotes generated image, I gt denotes real image, ||·||1denotes L1 norm, φ i denotes the activation map of the i-th pooling layer of Vgg16, in practice, i∈[1, 3].

[0084] Style loss:

[0085] where E denotes expectation, I out denotes generated image, I gt denotes real image, It represents the Gram matrix of the activation map φ i .

[0086] Adversarial loss:

[0087] where E denotes expectation, G denotes generator, D denotes discriminator, I gt denotes real image, E gt denotes real edge map, I out denotes generated image, E out denotes generated edge map.

[0088] In order to guide the dual encoder-decoder to generate structure and texture features, intermediate loss L s and L t are introduced on F inter and F structure . texture = BCE (E gt , P s (F s )) + l1 (I gt , P t (F t ))

[0089] where I gt denotes real image, E gt denotes real edge map, P s and P t are mapping functions composed of convolution kernel residual blocks, which extract structure features Fs and texture features F t These are mapped to the corresponding edge maps and RGB images, respectively.

[0090] The total loss is: L joint =λ rec L rec +λ perc L perc +λ style L style +λ adv L adv +λ inter L inter

[0091] Where λ rec ,λ perc ,λ style ,λ adv and λ inter These represent the calculation parameters for the corresponding losses. They are set as follows:

[0092] λ rec =10,λ perc =0.1,λ style =250,λ adv =0.1 and λ inter =1

[0093] Step S4: Input the damaged face image into the trained face image restoration model to obtain the restored face image. To verify the effectiveness of the algorithm, the experiment used the test set of the CelebA-HQ dataset. Under different mask region ratios, the algorithm was qualitatively and quantitatively compared with EdgeConnect, RFR-inpainting, and CTSDG algorithms.

[0094] Qualitative analysis: such as Figure 5 As shown, Figure 5 'a' represents a damaged facial image that needs to be repaired. Figure 5 In the case of large-area damage, EdgeConnect repairs distorted and severely deformed facial structures, and only produces good results when the damage is small. Figure 5 In C, RFR-inpainting produces overly smoothed content, and in cases of large-area damage, issues such as color inconsistency, artifacts, and texture blurring occur. Figure 5 In d, CTSDG also suffers from problems such as texture blurring and structural distortion. Figure 5 e represents the restoration result of this invention. As can be seen, the restored facial structure and texture are more consistent, the semantics are more reasonable, and it can still generate good restoration results even when there is a large area of ​​damage. Figure 5 f represents the actual image corresponding to the ground damage image.

[0095] Quantitative analysis: Experiments were conducted on the CelebA-HQ dataset, using different proportions of 10% to 50% masks to represent the size of the damaged area, and the generated results were quantitatively compared. The main evaluation indicators are PSNR, SSIM and MAE, as shown in the table below. The method of the present application achieves the best results in the three indicators compared with other methods. (↑ indicates that the larger the value, the better, ↓ indicates that the smaller the value, the better, and bold indicates the best result)

[0096] Table 1: Comparison of objective evaluation indicators of experimental results of CelebA-HQ dataset

[0097]

[0098] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A face image inpainting method based on structure and texture pair generation, characterized in that, The method comprises the following steps: Step S1: preprocessing the input image to obtain a face image to be repaired; Step S2: establishing a face image repair model based on structure and texture dual generation, and inputting the image obtained in step S1 into the image repair model for training; Step S3: continuously iterating training until the network finally converges, and obtaining a face image repair model; Step S4: inputting a damaged face image into the trained face image repair model to obtain a repaired face image; In step 2, the face image repair model is a structure of a generative adversarial network, which comprises a generator and a discriminator; The generator comprises a dual encoder-decoder and feature fusion, and the discriminator comprises a texture discriminator and a structure discriminator; The convolution layer of the dual encoder-decoder adopts gated convolution to encode and decode features, and a batch normalization layer is added after each gated convolution layer, which is represented as: wherein I represents a feature map; Gating represents gating; Feature represents a feature map after convolution; Output represents a final output feature map, and respectively represent different convolution kernels; is a LeakyReLU activation function, is a Sigmoid activation function, and represents element-level multiplication. Compared with hard gating, the gating value of the gating convolution is between 0 and 1. The closer the gating value is to 1, the more effective pixels there are. BN represents batch normalization; The dual encoder-decoder, In the encoding stage, the left and right two encoders respectively receive a damaged image and a damaged structure image to encode texture and structure features, In the decoding stage, the texture decoder synthesizes structure-constrained texture by borrowing structure features from the structure encoder, and the structure decoder restores texture-guided structure by obtaining texture features from the texture encoder; The discriminator is a double-flow discriminator with a texture branch and a structure branch, and the structure branch of the discriminator further has an additional edge detector for edge extraction, wherein the two discriminator trunks are composed of ordinary convolution, and the edge detector is composed of a convolutional neural network residual block.

2. The face image inpainting method based on structure and texture pair generation according to claim 1, characterized in that, The preprocessing of step S1 is as follows: Firstly, the size of the image is adjusted, and the image is adjusted to 256*256 by cropping and padding, Then, a binary mask M is obtained from an irregular mask dataset provided by NVIDIA to artificially damage the image, and a damaged image is obtained; the damaged image is processed to obtain a damaged grayscale image; Finally, the damaged grayscale image is extracted by a Canny edge detection algorithm to obtain a damaged edge image.

3. The face image inpainting method based on structure and texture pair generation according to claim 1, characterized in that, Step 3 uses the CelebA-HQ dataset for training, which includes training images and test images, and the experimental equipment uses NVIDIA V100, and the whole model is realized by PyTorch; when training the model, the batch size is set to 8, and the Adam optimizer is used for optimization.

4. The face image inpainting method based on structure and texture pair generation according to claim 3, characterized in that, Firstly, the initial training is performed using a learning rate of 2*10-4, and then the model is fine-tuned using a learning rate of 5*10-5; the learning rate fine-tuned model is trained using a joint loss, including a reconstruction loss, a perception loss, a style loss and an adversarial loss.

5. The face image inpainting method based on structure and texture pair generation according to claim 4, characterized in that, The four loss functions are as follows: Reconstruction loss function: where E denotes expectation, denotes the generated picture, denotes the real picture, denotes the L1 norm; Perceptual loss function: The perceptual loss pre-trained by VGG-16 on ImageNet is used to simulate the visual perception of human beings on image quality, where E represents expectation, represents the generated picture, represents the real picture, represents the L1 norm, represents the activation map of the i-th pooling layer of Vgg16, and in the actual process, i∈[1, 3]; Style loss function: where E denotes expectation, denotes the generated picture, denotes the real picture, denotes the activation map Gram matrix; Adversarial loss function: wherein E denotes expectation, denotes the generator, denotes the discriminator, denotes a real picture, denotes a real edge map, denotes a generated picture, denotes a generated edge map.

6. The face image inpainting method based on structure and texture pair generation according to claim 1, characterized in that, To guide the dual encoder-decoder to generate structure and texture features, in and an intermediate loss is also introduced: wherein, represents a real picture, represents a real edge map, and is a mapping function consisting of a convolution kernel residual block, mapping structural features and texture features to corresponding edge maps and RGB pictures, respectively.

Images