A method for restoring old photos based on different damage types

Abstract

This paper discloses a method for restoring old photos based on different types of damage. The method includes: constructing an old photo defect detection dataset and an old photo restoration dataset; proposing a multi-task old photo defect detection network and an old photo restoration network. The old photo restoration network consists of ScrathNet and PatchNet. A method for fusing crack and block damage features is proposed, hierarchically fusing the output of ScrathNet with the encoder of PatchNet, allowing PatchNet to obtain richer contextual reference information. This method can restore complex old photos with different types of damage and achieve good results.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to a method for restoring old photos based on different damage types. Background Art

[0002] In recent years, several deep learning-based photo restoration methods have been proposed, but they all have their own shortcomings. For example, a triplet-domain old photo restoration module can effectively restore large block areas but not cracked areas. This is because the module stacks too many residual modules during the feature extraction phase, learning global information while losing details. Conversely, a context-assisted cracked photo restoration module can effectively restore old photos with cracks but lacks performance on large block areas. This is because the module is designed primarily to learn local receptive fields and does not consider global information. Summary of the Invention

[0003] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and propose a method for repairing old photos based on different types of damage. It can achieve comprehensive repair of images with different defect types by hierarchically fusing the crack repair results and block area repair results, and obtain excellent repair results.

[0004] To achieve the above-mentioned purpose, the present invention provides a technical solution: a method for restoring old photos based on different types of damage, comprising the following steps:

[0005] S1. Collect old photo data and obtain old photo detection dataset and old photo restoration dataset through annotation and photo retouching.

[0006] S2. Using the old photo detection dataset obtained in step S1, a multi-task old photo defect detection network is trained. The multi-task old photo defect detection network is a generative adversarial network composed of an encoder and three decoders. The three decoders are respectively used for crack area detection, block missing area detection, and overall missing area detection. The network outputs three types of defect masks for old photos, namely, a crack mask, a block missing mask, and an overall damage mask. The multi-task old photo defect detection network can obtain the three types of defect masks for old photos, thereby realizing defect location of different damage types in old photos.

[0007] S3. The old photo restoration dataset obtained in step S1 and the three types of defect masks of the old photos obtained in step S2 are used to train an old photo restoration network for different types of damage. The old photo restoration network consists of ScrathNet and PatchNet. The ScratchNet is responsible for repairing the cracked areas of old photos. It uses a multi-branch partial convolution module composed of 3×3, 5×5, and 7×7 convolution kernels of different scales to achieve multi-scale crack image completion. The ScrathNet obtains results of different scales of repaired cracked areas, and the results of different scales are hierarchically integrated into PatchNet. The PatchNet is responsible for repairing block-like missing images of old photos. It uses the Transformer module as the basic unit. In the encoding stage, it uses the Non-local Inpainting Attention Module instead of the original Multi-attention Module, so that the block areas can be completed using global information. In the decoding stage, it uses the Mask-aware InstanceNorm instead of the Layer in the original Transformer module. Norm can make the foreground and background areas that need to be repaired consistent in style, making the repaired old photo image more harmonious. The encoding and decoding stages of the PatchNet are connected using the U-Net method to minimize information loss. Finally, the harmonized old photo image features are input into the 3×3 convolution module to obtain the final old photo restoration result with excellent performance.

[0008] Furthermore, in step S2, the encoder and decoder are both composed of a channel attention mechanism module, the encoder and decoder are connected using a U-Net method, and three BCE losses are used to guide the training of the multi-task old photo defect detection network to achieve end-to-end multi-task old photo detection.

[0009] Furthermore, in step S3, the old photo restoration data obtained in step S1 and the crack mask Ms, block loss mask Mp, and overall damage mask Ma obtained in step S2 are used to train old photo restoration networks for different damage types. First, the old photo image is multiplied by 1-Mp to cover the block loss area, and then Ms is fused by concatenation to retain the contextual information of the crack area. The fused result is then input into the first twin encoder to transform the three-channel image into multi-channel features to achieve high-dimensional feature extraction. The low-dimensional features and Ms are input into ScratchNet. The ScratchNet consists of four Multi-branch Partical Convolutions, which can learn local area knowledge and is suitable for repairing crack problems of different sizes. Inside ScratchNet, jump connections are used to reduce information loss. Finally, four crack area repair results of different scales are obtained through ScratchNet.

[0010] Further, in step S3, the old photo image is multiplied by 1-M a The input is sent to the second twin encoder to realize coarse feature extraction. The parameters of the second twin encoder and the first twin encoder are shared, which can realize the interaction of crack features and block missing features. The features passed by the second twin encoder and the corresponding Mp are input to PatchNet to realize block area repair. The encoder of PatchNet is responsible for repairing the retouched photo image in the encoding stage. It is a global attention completion based on the Transformer module. The encoder consists of four scale Transfill modules. The output results of each module are hierarchically fused with the ScratchNet results of each stage to provide PatchNet with rich contextual information, so that it can obtain more semantically consistent repair results; the decoder of PatchNet consists of four Transhomon modules of different sizes. The decoder uses Mask-aware Instance Norm instead of Layer Norm in the original Transformer module in the decoding stage, which can transfer the background style to the foreground, so that the style and semantic consistency of the repair area and the background area are better maintained. The above process is guided by Ma; the decoded features are input into the 3×3 convolution module to obtain a 3-channel excellent old photo repair result.

[0011] Furthermore, in order to optimize the parameters of the old photo restoration network and obtain the best restoration results, the following loss function is defined:

[0012] a. In order to reduce the difference between the pixels of the repaired image and the true value image, the reconstruction loss function L is defined rec As shown below:

[0013] Lrec =||(I o -I gt )||1

[0014] Where, I o Represents the repair result diagram, I gt represents the true value graph;

[0015] b. To reduce the difference between the features of the restored image and the true image, define the perceptual loss function L per As shown below:

[0016]

[0017] Where N i represents the feature map of the i-th layer of VGG-16, and Φ represents the activation function;

[0018] c. To reduce the difference between the styles of the restored image and the true image, define the style loss function L sty As shown below:

[0019]

[0020] Where, It is C j ×C j Gram matrix, C j Represents the j×j feature matrix, the Gram matrix can obtain the style knowledge of the image;

[0021] d. To further narrow the distance between the repaired image and the true image, define the adversarial loss function L adv As shown below:

[0022]

[0023] Where D represents the discriminator and E represents the expectation;

[0024] e. Loss function L of the entire old photo restoration network su As shown below:

[0025] L su =λ rec L rec +λ per L per +λ sty L sty +λ adv L adv

[0026] Where λ rec ,λ per ,λ sty ,λ adv is the empirical coefficient.

[0027] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0028] 1. The present invention can realize multi-task old photo mask prediction. Compared with the original old photo mask detection module, the present invention can simultaneously predict masks of different damage types, thereby realizing the repair of different defect types.

[0029] 2. The present invention designs ScratchNet for crack repair and PatchNet for block area repair respectively, and hierarchically fuses the crack repair results and block repair data streams, which can effectively solve the repair problems of different damage types.

[0030] 3. In the encoder stage of PatchNet, the present invention uses Non-local Inpainting instead of Multi-attention in the original Transformer module, which can effectively solve the global repair problem.

[0031] 4. In the decoder stage of PatchNet, the present invention uses Mask-aware Instance Norm instead of the Layer Norm in the original Transformer model, which can effectively solve the problem of style consistency between the foreground area and the background area to be repaired. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the logic flow of the method of the present invention.

[0033] Figure 2 Schematic diagram of the network structure for old photo restoration.

[0034] Figure 3 Schematic diagram of the structure of Non-local Inpainting Attention in PatchNet.

[0035] Figure 4 Schematic diagram of the structure of Mask-aware Instance Norm in PatchNet. DETAILED DESCRIPTION

[0036] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0037] like Figure 1 As shown, this embodiment discloses a method for repairing old photos based on different types of damage, and the specific implementation steps include:

[0038] 1) Data Preparation

[0039] We collected old photos from several open source websites and asked restoration experts to mark areas of different damage types to obtain three types of masks: crack masks, block loss masks, and overall damage masks; we also asked other experts to restore the old photos to obtain the true value of the restored old photos.

[0040] 2) Data augmentation

[0041] In order to increase the training data, the detection data and restoration data are flipped, rotated, and cropped to obtain a large number of old photo detection datasets and old photo restoration datasets with strong generalization.

[0042] 3) Network construction

[0043] like Figure 2 As shown in Figure 1, this embodiment uses the Pytorch framework to build a multi-task old photo defect detection network and an old photo restoration network for different damage types. The specific steps are as follows.

[0044] 3.1) In order to obtain the defect location of old photos, a multi-task old photo defect detection network is proposed. The multi-task old photo defect detection network is a generative adversarial network, which consists of an encoder and three decoders, and the encoder and decoder are connected in a U-Net manner. Both the encoder and the decoder are composed of a channel attention mechanism module. The multi-task old photo defect detection network is trained using the old photo detection dataset obtained in step 2). When the training converges, the optimal multi-task old photo defect detection network is obtained. The optimal network is then used to detect different types of defect masks for old photos (crack mask Ms, block mask Mp, overall damage mask Ma), thereby realizing the location of defects of different damage types in old photos.

[0045] 3.2) The old photo restoration dataset and the different types of defect masks of the old photos obtained in step 3.1) are used to train an old photo restoration network for different types of damage. The old photo restoration network specifically consists of ScratchNet (also known as a crack repair network) and PatchNet (block repair network). ScratchNet is responsible for repairing crack areas, and PatchNet is responsible for repairing block areas. Since elongated crack areas are easier to repair than larger block areas, and the results of the repaired crack areas can assist in repairing block areas, the data stream of the present invention first passes through ScratchNet and then passes through PatchNet.

[0046] 3.3) Multiply the old photo to be repaired by a block mask to cover the block defect area, and merge the crack area mask to retain the contextual information of the crack area. The processed image is input into the first twin encoder to obtain a high-dimensional feature code. Next, the feature code is input into ScratchNet. Since repairing cracks requires attention to local information, the present invention uses a multi-branch partial convolution module, which is composed of 3×3, 5×5, and 7×7 convolution kernels to achieve multi-scale crack area image completion. And jump connections are added between each multi-branch partial convolution module to reduce information loss. ScratchNet contains a total of four multi-branch partial convolution modules (Multi-branch Partical Convolution), and the results of each module repair are input into the next PatchNet.

[0047] 3.4) Multiply the old photo to be repaired by the overall damage mask to cover the entire defective area, and input the pre-processed image into the second twin encoder to obtain a high-dimensional feature code. Next, the feature code is input into PatchNet. Figure 3 and Figure 4 As shown, PatchNet consists of an encoder and a decoder, and the encoder and decoder are connected in a similar way to U-Net. In order to expand the module learning receptive field, the basic components of PatchNet's encoder and decoder are Transformer modules. In the encoding stage, the present invention modifies the original Transformer's Multi-head Attention into Non-local Inpainting Attention, which is responsible for filling in the block defect areas. The encoder consists of four scale Transfill modules. The output results of each module are hierarchically fused with the ScratchNet results of each stage to provide PatchNet with rich contextual information, so that it can obtain more semantically consistent repair results and assist in the repair of block areas.

[0048] 3.5) In the PatchNet decoder stage, to maintain a consistent style between the damaged foreground and background areas of the old photo, we use Mask-aware Instance Norm (MAN) instead of the original Transformer module's Layer Norm. This process is guided by MA. After the decoding stage, the decoded features are fed into a 3×3 convolutional module to produce the final restored image.

[0049] 4) Define the loss function

[0050] The loss functions defined in this embodiment include reconstruction loss function, perception loss function, style loss function, and adversarial loss function, which are as follows:

[0051] a. In order to reduce the difference between the pixels of the repaired image and the true value image, the reconstruction loss function L is defined rec As shown below:

[0052] L rec =||(I o -I gt )||1

[0053] Where, I o Represents the repair result diagram, I gt Represents the true value graph.

[0054] b. To reduce the difference between the features of the restored image and the true image, define the perceptual loss function L per As shown below:

[0055]

[0056] Where N i represents the feature map of the i-th layer of VGG-16, and Φ represents the activation function.

[0057] c. To reduce the difference between the styles of the restored image and the true image, define the style loss function L sty As shown below:

[0058]

[0059] Where, It is C j ×C j Gram matrix, C j Represents the j×j feature matrix, and the Gram matrix can obtain the style knowledge of the image.

[0060] d. To further narrow the distance between the repaired image and the true image, define the adversarial loss function L adv As shown below:

[0061]

[0062] Where D stands for discriminator and E stands for expectation.

[0063] e. Loss function L of the entire old photo restoration network su As shown below:

[0064] L su =λ rec L rec +λ per L per +λ sty L sty +λ adv L adv

[0065] Where λ rec ,λ per ,λ sty ,λ adv is the empirical coefficient, λ rec =1,λ per =0.2,λ sty =250,λ adv =0.2.

[0066] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for restoring old photos based on different types of damage, characterized in that: The following steps are involved: S1. Collect old photo data and obtain old photo detection dataset and old photo restoration dataset through annotation and photo retouching. S2. Using the old photo detection dataset obtained in step S1, a multi-task old photo defect detection network is trained. The multi-task old photo defect detection network is a generative adversarial network composed of an encoder and three decoders. The three decoders are respectively used for crack area detection, block missing area detection, and overall missing area detection. The network outputs three types of defect masks for old photos, namely, a crack mask, a block missing mask, and an overall missing mask. The multi-task old photo defect detection network can obtain the three types of defect masks for old photos, thereby realizing defect location of different damage types in old photos. S3. The old photo restoration dataset obtained in step S1 and the three types of defect masks of the old photos obtained in step S2: crack mask Ms, block loss mask Mp, and overall loss mask Ma are used to train old photo restoration networks for different types of damage. The old photo restoration network consists of ScratchNet and PatchNet. The ScratchNet is responsible for repairing the cracked areas of old photos. It uses a multi-branch partial convolution module composed of convolution kernels of different scales, 3×3, 5×5, and 7×7, to achieve multi-scale crack image completion. The ScratchNet obtains results of different scales for repairing the cracked areas, and the results of different scales are hierarchically integrated into PatchNet. The PatchNet is responsible for repairing block loss in old photo images. It uses the Transformer module as the basic unit. In the encoding stage, it uses the Non-local InpaintingAttention Module instead of the original Multi-attention Module, which can use global information to complete the block area. In the decoding stage, it uses the Mask-aware Instance Norm replaces the LayerNorm in the original Transformer module, making the styles of the foreground and background areas to be restored consistent. The encoding and decoding stages of the PatchNet are connected using a U-Net approach. Finally, the harmonized old photo image features are input into a 3×3 convolution module to obtain the final old photo restoration result; Among them, the old photo restoration network for different types of damage is trained. First, the old photo image is multiplied by 1-Mp to cover the block-like missing area, and then Ms is fused by concatenation to retain the contextual information of the crack area. The fused result is then input into the first twin encoder, and the three-channel image is transformed into multi-channel features to achieve high-dimensional feature extraction. The high-dimensional features and Ms are input into ScratchNet. The ScratchNet consists of four Multi-branch PartialConvolutions, which can learn local area knowledge and is suitable for repairing crack problems of different sizes. In addition, within ScratchNet, jump connections are used to reduce information loss. Finally, the repair results of crack areas of four different scales are obtained through ScratchNet.

2. The method for restoring old photos based on different damage types according to claim 1, characterized in that: In step S2, the encoder and decoder are both composed of a channel attention mechanism module. The encoder and decoder are connected using a U-Net method, and three BCE losses are used to guide the training of the multi-task old photo defect detection network to achieve end-to-end multi-task old photo detection.

3. The method for restoring old photos based on different damage types according to claim 2, characterized in that: In step S3, the old photo image is multiplied by 1-Ma and input into the second twin encoder to realize coarse feature extraction, wherein the second twin encoder and the first twin encoder share parameters, which can realize the interaction of crack features and block missing features, and the features passed by the second twin encoder and the corresponding Mp are input to PatchNet to realize block area repair. The encoder of PatchNet is responsible for repairing the retouched photo image in the encoding stage. It is based on the Transformer module for global attention completion. The encoder of PatchNet consists of four scale Transfill modules. The output results of each module are hierarchically fused with the ScratchNet results of each stage to provide rich contextual information for PatchNet, so that it can obtain more semantically consistent repair results; the decoder of PatchNet consists of four Transformer modules of different sizes. The decoder of PatchNet uses Mask-aware Instance Norm instead of Layer in the original Transformer module in the decoding stage. Norm can transfer the style of the background to the foreground, so that the style and semantic consistency between the repaired area and the background area are better maintained. The above process is guided by Ma; the decoded features are input into the 3×3 convolution module to obtain a 3-channel old photo restoration result.

Images