A face blindness restoration method based on 3D decomposition

Abstract

The present invention belongs to the field of digital image processing technology, and is specifically a method for blind face restoration based on three-dimensional decomposition. The present invention comprises: using a three-dimensional reconstruction model of a face to decompose a face image into three parts: a texture layer, a bone structure layer, and environment and posture parameters; using an encoder-decoder to coarse-grained fill the bone structure layer, first using different parameter encoder-decoders to fine-grained fill the texture layer, and then using a face enhancement module to enhance the details of the face; using a renderer in a three-dimensional reconstruction network to render the two restored components in combination with other parameters to obtain a restored and reconstructed three-dimensional face image; inputting the restored three-dimensional face image and the occluded input image into a face refinement module to obtain a restored face image. Experimental results show that when the occlusion range of the image is not manually marked, the present invention can accurately repair the face while keeping the unoccluded part consistent with the input image.

Description

Technical Field

[0001] The invention belongs to the technical field of digital image processing, and in particular relates to a method for repairing blind faces. Background Art

[0002] In recent years, with the continuous development of digital image processing technology, face restoration, as one of the important tasks, has been increasingly used in daily life. Face restoration tasks can repair occluded or physically damaged face photos to obtain unoccluded complete face images. Traditional face restoration methods require the area to be restored as a label to be input into the network, and manual labeling of the occluded or damaged areas of the image to be restored is a cumbersome process. Blind face restoration is a face restoration method that does not require manual labeling of occluded areas and automatically restores occluded faces by the network. Current blind face restoration methods are mainly based on occlusion area detection methods. However, occlusion detection performs well on post-physical damage to photos, but performs poorly in images that contain occluders in the original photos. Inaccurate detection of occluded edges can also cause artifacts at the edges of the restoration results that affect the visual effect. Based on the above background, the present invention combines face three-dimensional decomposition and blind face restoration, utilizes the inherent characteristics of face images to decompose the image to be restored, restores the different face components obtained by decomposition in different ways, and finally synthesizes and refines the components into an unobstructed face.

[0003] In recent years, the three-dimensional reconstruction of human faces has achieved more and more results in the research of digital image processing. The traditional three-dimensional reconstruction method of human faces needs to match the face image to be reconstructed with multiple scanned three-dimensional human heads to obtain the reconstruction result. In recent years, researchers have begun to try to decompose the face image into multiple different components and synthesize these components into a three-dimensional face model. Reference [1] proposed a method for decomposing non-aligned faces into multiple components in an unsupervised manner and finally rendering and synthesizing them into a three-dimensional model. Based on the research on three-dimensional reconstruction in recent years, the present invention uses the prior knowledge of human faces and the symmetry characteristics of human faces to apply the three-dimensional decomposition part to face restoration. By repairing each decomposed face in different ways, a complete face image with occlusion and damage removed is obtained. Summary of the invention

[0004] In order to overcome the shortcomings of existing blind face restoration technologies in occlusion detection and edge processing of restoration results, the purpose of the present invention is to provide a blind face restoration method based on three-dimensional decomposition, which is mainly aimed at occluded or damaged face images, and realizes the network to automatically repair the damaged face without giving the range of manually labeled occlusion area, so as to obtain an unoccluded and undamaged face image.

[0005] The blind face restoration method based on three-dimensional decomposition proposed in the present invention uses an unsupervised three-dimensional face reconstruction model to perform three-dimensional decomposition on the face, and decomposes the input face image into three parts: texture layer, bone structure layer, and environment and posture parameters; combining the inherent characteristics of the two layers obtained by decomposition, different networks are used to repair them respectively; wherein, the bone structure layer is directly filled with coarse-grained using an encoder-decoder architecture; the texture layer is first filled with fine-grained using an encoder-decoder architecture with different parameters, and then the face enhancement module is used to enhance the facial details of the texture layer; the renderer in the three-dimensional reconstruction network is used to combine the two repaired components with other parameters to render the repaired and reconstructed three-dimensional face image; the repaired three-dimensional face image and the occluded input image are input into the face refinement module together, thereby obtaining the repaired face image. In addition, the present invention constructs a three-dimensional occlusion dataset using three-dimensional occlusion sticker rendering. The specific steps are as follows:

[0006] (1) 3D face decomposition

[0007] For a given occluded face image to be restored, it is input into an unsupervised 3D face decomposition network, which is used to decompose the input image into a skeleton structure layer (shape, s), a texture layer (texture, t), an environment (environment, e) and pose (pose, p) parameters; wherein the skeleton structure layer is a single-channel grayscale image, the texture layer is a three-channel color image, and the remaining parameters are a set of values ​​representing the lighting environment and face angle of the face. The specific decomposition process can be expressed as:

[0008] Ω(t,s,e,p)=F 3D (I)#(1)

[0009] Where Ω represents the result set of the 3D decomposition model, and the decomposition results are the skeleton structure layer (s), texture layer (t), environment (e) and pose (p) parameters, I represents the input image to be decomposed, and F 3D is the three-dimensional decomposition function.

[0010] In the 3D face decomposition network, the skeleton structure layer and texture layer are mainly generated by the encoder-decoder structure, and the environment and posture parameters are encoded by the encoder structure. Among them, different encoder-decoders or encoder structures do not share parameters. The network structure for generating the skeleton structure layer is a network for processing single-channel images, and the network for generating the texture layer is a network for processing three-channel images.

[0011] Among them, the 3D face reconstruction model obtained by rendering the decomposed components can be projected according to the posture parameters of the input image to obtain a 2D face image with the same posture and ambient lighting as the input image. This is a 3D face reconstruction rendering process and a projection process of the 3D face reconstruction model; the 3D face reconstruction rendering process is expressed as:

[0012] M=R 0 (t,s,e)#(2)

[0013] The projection process of the 3D face reconstruction model is expressed as:

[0014] I ′ = ∏(M,p) = ∏(R 0 (t,s,e),p)#(3)

[0015] Among them, R 0 represents the renderer, and ∏ is the projection function.

[0016] In this method, the face 3D decomposition network model used is based on the unsupervised 3D face reconstruction model proposed by Wu et al. [1]. On this basis, the image scale is adjusted to 128×128 to improve the image resolution, and the face identity loss is added during the network training process to improve the degree of preservation of the face identity features in the decomposition result. The identity loss L ID It can be expressed as follows:

[0017]

[0018] Among them, the function f(·) is the transformation of the network on the image, I is the input image, I ′ is the processed image.

[0019] In the process of 3D face decomposition, the face symmetry probability map proposed in the literature [1] is used to ensure that all components of the face are symmetrical under standard conditions. This symmetry probability map can repair asymmetric occlusion and damage in various structural layers of the face to a certain extent.

[0020] (2) Repair of bone structure layer and texture layer

[0021] The bone structure layer and texture layer of the image to be repaired obtained by decomposition in the 3D face decomposition network are repaired so that the repaired bone structure layer and texture layer have a complete face structure. For the symmetrical area to be repaired, the bone structure layer and texture layer are first repaired using the encoder-decoder (context encoder) network respectively; for this part of the network, the pixel-level reconstruction loss, the generator adversarial loss and the corresponding discriminator adversarial loss are used for training respectively; where:

[0022] Pixel-level reconstruction loss L pix It can be expressed as:

[0023] L pix =∥t gt -G(t occ )∥#(5)

[0024] Adversarial loss L adv It can be expressed as:

[0025]

[0026] The corresponding discriminator adversarial loss L D It can be expressed as:

[0027]

[0028] In formulas (5), (6), and (7), t occ represents the bone structure graph obtained by decomposing the occlusion graph, t gt is the corresponding groundtruth image, G is the generator, and D is the discriminator.

[0029] Since the bone structure layer only has relatively simple contour information but does not contain more accurate facial detail information, the bone structure layer only needs to fill in the structural deficiencies through its special repair network. The filling process mainly fills the missing facial structure part with the facial prior contour in the repair network; the texture layer has more facial details and colors, and has relatively accurate facial details, which will be damaged to a certain extent during the facial symmetry decomposition and initial facial repair process, so it is necessary to use the face enhancement module for further enhancement.

[0030] The grayscale image of the bone structure layer can be restored to a relatively accurate structure only after the encoder-decoder structure is repaired, while the texture layer needs to use the face enhancement module to enhance the relatively blurred details in the repair process. The face enhancement module here directly uses the face enhancement module proposed in the literature [6]. After this step, the repaired bone structure image t inpaint and texture maps inpaint , respectively expressed as:

[0031] t inpaint =E(G(t occ ))#(8)

[0032] s inpaint =G(S occ )#(9)

[0033] Among them, t inpaint and t occThey are the restored bone structure image and the unrestored bone structure image directly decomposed from the occluded image; inpaint and occ They are the restored texture map and the unrestored texture map directly decomposed from the occluded image.

[0034] At this stage, the pixel-level reconstruction loss L pix The coefficient is set to λ 1 , adversarial loss L adv The coefficient is set to λ 2 , the corresponding discriminator adversarial loss L D The coefficient is set to λ 3 ; The total loss is:

[0035] L M =λ 1 L pix +λ 2 L adv +λ 3 L D .

[0036] (3) The results of bone structure layer and texture layer restoration are rendered into a synthetic 3D face model

[0037] The bone structure layer and texture layer repaired in step (2) are combined with the environmental parameters and posture parameters decomposed in step (1), and the repaired three-dimensional face result and the two-dimensional projection with the same environmental lighting and position parameters as the input image are obtained through a renderer and a projection function. The specific rendering and projection process is the same as that described in formulas (2) and (3); this process does not involve model parameter training.

[0038] (4) Face Retouching

[0039] The two-dimensional projection of the three-dimensional restoration face model obtained in step (3) is combined with the original image to be restored, and the two-dimensional projection and the original image to be restored are input into the face refinement module to obtain the final face restoration result. The combination of the two-dimensional projection of the three-dimensional face model and the original image to be restored is expressed as:

[0040] I recon =I occ ⊕I ′ 3D =I occ ⊕∏(R 0 (t inpaint ,s inpaint ,e),p)#(10)

[0041] The face reconstruction image obtained by combining formula (10) has a complete face structure but loses a small amount of details of the face. It is necessary to use some details in the original image to be repaired to combine it to obtain a repair result with a good visual effect. That is, this step requires learning a mapping from a reconstructed image and the image to be repaired to the repaired face, which can be expressed as:

[0042] (I recon ,I occ )→I gt #(11)

[0043] The face refinement module adopts the U-Net network architecture and uses pixel-to-pixel loss, perceptual loss, adversarial loss and discriminator adversarial loss to train the network. for:

[0044]

[0045] Perceptual loss L percep for:

[0046]

[0047] Adversarial loss L adv_U It is expressed as:

[0048]

[0049] Adversarial loss of the discriminator It can be expressed as:

[0050]

[0051] (5) Overall network training process

[0052] Finally, in order to ensure the continuity between the various modules of the network and optimize the parameters contained in each module for the entire network, the face refinement module and the face component repair module (including the overall module of the bone structure layer and the texture layer repair) in the entire network are fine-tuned using the end-to-end reconstruction loss. Since the training data of the 3D face decomposition network is different from that of the other modules, in the final end-to-end training, the parameters of the 3D decomposition network are fixed, and only the parameters of the face component repair module and the face refinement module network are fine-tuned. The loss can be expressed as:

[0053] L recon =|I refine -I gt |#(16)

[0054] The training loss of the entire network can be expressed as:

[0055]

[0056] The first, second, and third terms correspond to the losses expressed by equations (5), (6), and (7), and the fourth, fifth, sixth, seventh, and eighth terms correspond to the losses expressed by equations (12), (14), (15), (13), and (16). 1 ,λ 2 ,λ 3 ,λ 4 ,λ 5 , 6 ,λ 7 is the corresponding item weight coefficient.

[0057] The modules involved in steps (1), (2) and (4) are all trained using pairs of non-occluded face images and corresponding occluded face images.

[0058] The beneficial effects of the present invention are:

[0059] The method of the present invention avoids the artifact problem of occlusion edge caused by inaccurate occlusion detection in the past blind face restoration methods based on occlusion detection, and uses three-dimensional face decomposition technology to repair the various levels of the face in different ways. The present invention makes full use of the prior knowledge of the face in the face image, the symmetry of the face itself and the three-dimensional characteristics of the face, decomposes the face, repairs it in a targeted manner and refines the synthesis result, and obtains a more accurate and detailed blind face restoration result. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of the present invention.

[0061] Figure 2 The experimental results of using the present invention to perform face blind restoration on 128×128 images. The first row is the input image to be restored, the second row is the restoration result of the present invention; the two columns on the left are the experimental results on the synthetic image and the two columns on the right are the experimental results on the real data set. DETAILED DESCRIPTION

[0062] Taking an obstructed or damaged face picture as an example, the face part is restored to a complete face image according to the information provided by the remaining unobstructed parts by the method of the present invention. The specific process is as follows:

[0063] (1) Use a 3D face decomposition network to decompose the face image to be repaired into a standard symmetrical texture map, a bone structure map, and environment and posture parameters. The network is trained for 30 epochs, with the input image to be decomposed and the output texture map and bone structure map all being 128×128 pixels in size. The Adam optimizer is used for training at this stage and the batch size is set to 16.

[0064] (2) Repair the texture map and bone structure map obtained by decomposition. The networks for repairing the two components were trained for 200 epochs, with the learning rate set to 0.0002 and the batch size set to 8. The pixel-by-pixel reconstruction loss coefficient λ 1 Set to 999, the coefficient λ of the adversarial loss 2 Set to 1, the dual discriminator adversarial loss coefficient λ 3 Set to 1; the face enhancement module involved in the texture map restoration process directly reuses the face enhancement module in the literature [6];

[0065] (3) The restoration result is rendered into a 3D face image. The rendering process is the inverse transformation of the 3D decomposition process. For the specific process, please refer to the rendering method in reference [1]. This rendering process does not involve network training.

[0066] (4) The synthesized 3D face image and the input image to be repaired are simultaneously input into the face refinement module to obtain the repaired face image. The network is trained for 200 epochs with a learning rate of 0.0002 and a batch size of 128. In this stage, the pixel-by-pixel reconstruction loss coefficient λ is 4 Set to 100, the generator and discriminator adversarial loss coefficient λ 5 , 6 are set to 1 respectively, and the perceptual loss coefficient λ 7 Set to 10;

[0067] (5) All modules of the entire network are finally fine-tuned in an end-to-end manner. In this stage, the parameters of the face 3D decomposition module are fixed, and the parameters of the remaining modules are fine-tuned in this stage.

[0068] Figure 2 This is a comparison of examples of the present invention, where the first row is the input image to be repaired, and the second row is the repair result diagram of the present invention; the two columns on the left are the experimental results on the synthetic image, and the two columns on the right are the experimental results on the real data set; it can be seen that the method of the present invention can perform blind repair on the occluded face image in both synthetic occlusion and real occlusion, and obtain a more realistic repair result while maintaining the features of the unoccluded part.

[0069] References

[0070] [1]S.Wu,C.Rupprecht,and A.Vedaldi.“Unsupervised Learning of ProbablySymmetric Deformable3DObjects from Images in the Wild”.In CVPR,pages 1–10.Computer Vision Foundation / IEEE,2020.

[0071] [2]X.Yuan and I.Park.“Face de-occlusion using3d morphable model andgenerative adversarial network”.InICCV,pages 10061–10070.IEEE,2019.

[0072] [3]Y.Wang,Y.Chen,X.Tao,and J.Jia.“Vcnet:ARobust Approach to BlindImage Inpainting”.In ECCV(25),volume 12370of Lecture Notes in ComputerScience,pages752–768.Springer,2020.

[0073] [4]A.Hertz,S.Fogel,R.Hanocka,R.Giryes,andD.Cohen-Or.“Blind VisualMotif Removal from aSingleImage”.In CVPR,pages 6858–6867.Computer VisionFoundation / IEEE,2019.

[0074] [5]D.Pathak,P.Krahenbuhl,J.Donahue,T.Darrell,and A.Efros.“ContextEncoders:Feature Learning by Inpainting”.In CVPR,pages 2536–2544.IEEEComputerSociety,2016

[0075] [6]Z.Wan,B.Zhang,D.Chen,P.Zhang,D.Chen,J.Liao,and F.Wen.“Bringing OldPhotos Back toLife”.In CVPR,pages 2744–2754.Computer Vision Foundation / IEEE,2020。

Claims

1. A method for face blindness restoration based on three-dimensional decomposition, characterized in that: The specific steps are as follows: (1) 3D face decomposition The given occluded face image to be repaired is input into the unsupervised 3D face decomposition network, and the network is used to decompose the input image into the skeleton structure layer (s), texture layer (t), environment (e) and posture (p) parameters; among which, the skeleton structure layer is a single-channel grayscale image, the texture layer is a three-channel color image, and the remaining parameters are a set of values ​​representing the lighting environment and face angle of the face. The specific decomposition process is expressed as: Ω(t,s,e,p)=F 3D (I)#(1) Where Ω represents the result set of the 3D decomposition model, and the decomposition results are the skeleton structure layer (s), texture layer (t), environment (e) and pose (p) parameters, I represents the input image to be decomposed, and F 3D is the three-dimensional decomposition function; In the 3D face decomposition network, the skeleton structure layer and the texture layer are generated by the encoder-decoder structure, and the encoder structure encodes the environment and posture parameters; different encoder-decoders or encoder structures do not share parameters, the network structure for generating the skeleton structure layer is a network for processing single-channel images, and the network for generating the texture layer is a network for processing three-channel images; The decomposed components are rendered by a renderer to obtain a 3D face reconstruction model, and the 3D face reconstruction model is projected according to the pose parameters of the input image to obtain a 2D face image with the same pose and ambient lighting as the input image. The 3D face reconstruction rendering process is expressed as: M=R0(t,s,e)#(2) The projection process of the 3D face reconstruction model is expressed as: I ′ =∏(M,p)=∏(R0(t,s,e),p)#(3) Among them, R0 represents the renderer, ∏ is the projection function; The face 3D decomposition network model used is based on the unsupervised 3D face reconstruction model. On this basis, the image scale is adjusted to 128×128 to improve the image resolution, and the face identity loss is added during the network training process to improve the degree of preservation of the face identity features in the decomposition result. The identity loss L ID It is expressed as follows: Among them, the function f(·) is the transformation of the network on the image, I is the input image, I ′ is the processed image; The face symmetry probability map is used in the process of 3D face decomposition to ensure that all components of the face are symmetrical under standard conditions. The symmetry probability map can repair asymmetric occlusion and damage in various structural layers of the face to a certain extent. (2) Repair of bone structure layer and texture layer Repairing the bone structure layer and texture layer of the image to be repaired obtained by decomposition in the three-dimensional face decomposition network, so that the repaired bone structure layer and texture layer have a complete face structure; For the symmetrical area to be repaired, the skeleton structure layer and the texture layer are first repaired using the encoder-decoder network respectively; for this part of the network, the pixel-level reconstruction loss, the generator adversarial loss and the corresponding discriminator adversarial loss are used for training respectively; where: Pixel-level reconstruction loss L pix It is expressed as: L pix =∥t gt -G(t occ )∥#(5) Adversarial loss L adv It is expressed as: The corresponding discriminator adversarial loss L D It is expressed as: In formulas (5), (6), and (7), t occ represents the bone structure graph obtained by decomposing the occlusion graph, t gt is the corresponding groundtruth image, G is the generator, and D is the discriminator; At this stage, the total training loss is: L M =λ1L pix +λ2L adv +λ3L D ; Parameters λ1, λ2, λ3 are the corresponding weight coefficients; For the grayscale image of the bone structure layer, only the encoder-decoder structure can be used to restore a relatively accurate structure. For the texture layer, the face enhancement module is also used to enhance the details that are relatively blurred during the restoration process; after this step, the restored bone structure image t inpaint and texture maps inpaint , respectively expressed as: t inpaint =E(G(t occ ))#(8) s inpaint =G(s occ )#(9) Among them, t inpaint and t occ They are the restored bone structure image and the unrestored bone structure image directly decomposed from the occluded image; inpaint and occ They are the restored texture map and the unrestored texture map directly decomposed from the occluded image; (3) Rendering of the bone structure layer and texture layer repair results to synthesize a 3D face model The bone structure layer and texture layer repaired in step (2) are combined with the environmental parameters and posture parameters decomposed in step (1), and the repaired three-dimensional face result and the two-dimensional projection with the same environmental lighting and position parameters as the input image are obtained through a renderer and a projection function. The specific rendering and projection process is the same as that described in formulas (2) and (3); (4) Face Retouching The two-dimensional projection of the three-dimensional restoration face model obtained in step (3) is combined with the original image to be restored, and the two-dimensional projection and the original image to be restored are input into the face refinement module to obtain the final face restoration result; the combination method of the two-dimensional projection of the three-dimensional face model and the original image to be restored is expressed as: I recon =I occ ⊕I ′ 3D =I occ ⊕∏(R0(t inpaint ,s inpaint ,e),p)#(10) The face reconstruction image obtained by combining formula (10) has a complete face structure but loses a small amount of details of the face. It is necessary to use some details in the original image to be repaired to combine it to obtain a repair result with a good visual effect. This step requires learning a mapping from a reconstructed image and the image to be repaired to the repaired face, which is expressed as: (I recon ,I occ )→I gt #(11) The face refinement module adopts the architecture of the U-Net network and uses pixel-to-pixel loss, perceptual loss, adversarial loss and discriminator adversarial loss to train the network, where: Pixel-to-pixel loss for: Perceptual loss L percep for: Adversarial loss L adv_U It is expressed as: The adversarial loss L of the discriminator DU It can be expressed as: (5) Overall network training Finally, in order to ensure the continuity between the various modules of the network and optimize the parameters of each module for the entire network, the face refinement module and the face component repair module are fine-tuned using the end-to-end reconstruction loss; since the training data of the 3D face decomposition network is different from that of the other modules, in the final end-to-end training, the parameters of the 3D decomposition network are fixed, and only the parameters of the face component repair module and the face refinement module network are fine-tuned; its loss L recon It is expressed as: L recon =|I refine -I gt |#(16) The training loss of the entire network is expressed as: Parameters λ1, λ2, λ3, λ4, λ5, λ6, and λ7 are the corresponding item weight coefficients.

Images