Method and system for restoring old photos based on deep neural network

Abstract

The application provides an old photo repairing method and system based on a deep neural network, which comprises the following steps: obtaining an old photo to be repaired; constructing a rough repairing network model and training the model to eliminate non-structural damage of the old photo and obtain a rough-repaired old photo; constructing a fine repairing network model, taking the rough-repaired old photo as an input of the model to train the model, eliminating structural damage in the rough-repaired old photo, and obtaining a repaired old photo. By using the two-stage repairing model from rough to fine, the application divides the complex damage in the old photo into two types, i.e. structural damage and non-structural damage. The rough repairing stage focuses on removing the non-structural damage in the old photo, and the fine repairing stage mainly solves the structural damage. The application adopts a progressive reconstruction strategy, which makes the reconstruction process from overall to local, and gradually reconstructs a clean, structurally complete and detailed repairing result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an old photo restoration method and system based on a deep neural network. BACKGROUND

[0002] Old photo restoration technology is very popular among the public. Many people give restored old photos to the elderly as gifts to friends and relatives. Old photo restoration has become a fashion. At the same time, old photos have the value of documentary historical materials for recording history and reflecting the social appearance at the time. Old photos can also reflect the progress of the history of photography. With the development of the history of photography, the carrier of the photo has also gradually changed. With the popularization of photography, there is a photography experience, and many people are interested in the history of photography because of their own photography experience, and are also interested in the development process of photography. In addition, the artistic value implied in old photos gives people visual enjoyment, and it is also a visual art.

[0003] The image completion problem aims to fill in the missing parts of the picture with reasonable content, which can be used to repair structural damage such as scratches and stains in old photos, but non-structural damage in the photo still cannot be repaired, and may even affect the completion effect, resulting in poor repair quality. However, in the existing old photo restoration technology, such as Bringing Old Photos Back to Life, the generated repair result is too fuzzy and smooth, and only produces blurred and severely lacking in detail content in the area where the damage needs to be completed again. For example, Resolution-robust Large Mask Inpainting with Fourier Convolut ions only targets structural damage in old photos and cannot handle non-structural damage

[0004] Patent document CN115311156A discloses a photo restoration method, including the following steps. Step S1: extracting a photo area in a picture. Step S2: extracting a damaged, creased, and scratched area in the photo area. Step S3: repairing the damaged, creased, and scratched area in the photo area. Step S4: detecting a face area in the photo area. Step S5: removing various degradations in the face area and adding face details in the face area.

[0005] However, patent document CN115311156A cannot handle non-structural damage.

[0006] In summary, there is a need in the market for an old photo restoration method and system based on a deep neural network that can repair both structural damage and non-structural damage. SUMMARY

[0007] Aiming at the defects in the prior art, the present application aims to provide a deep neural network-based old photo restoration method and system.

[0008] According to the present application, a deep neural network-based old photo restoration method is provided, comprising:

[0009] Step S1: obtaining an old photo to be restored;

[0010] Step S2: constructing a rough restoration network model and training the model, eliminating non-structural damage of the old photo, and obtaining a rough-restored old photo;

[0011] Step S3: constructing a fine restoration network model, taking the rough-restored old photo as the input of the model to train the model, eliminating structural damage in the rough-restored old photo, and further obtaining a restored old photo.

[0012] Preferably, the rough restoration network model comprises a variational autoencoder and a mapping network;

[0013] The variational autoencoder comprises a first variational autoencoder and a second variational autoencoder, wherein the first variational autoencoder comprises an encoder, a decoder, a discriminator for a reconstruction result, and a discriminator for an encoding vector, and the second variational autoencoder comprises an encoder, a decoder, and a discriminator for a reconstruction result;

[0014] The fine restoration network model comprises a multi-scale densely connected edge restoration network and a Fourier convolution-based completion network;

[0015] The edge restoration network comprises three layers of the same dense residual network, and each layer of the network takes the outputs of all previous layers as additional inputs through a skip connection, so as to extract local features from the input.

[0016] Preferably, step S2 comprises:

[0017] Step S2.1: obtaining an encoding vector through the encoder of the first variational autoencoder, while extracting image detail information in the intermediate feature map through a detail feature extraction module;

[0018] Step S2.2: converting the encoding vector through the mapping network to convert the degraded picture domain to the clean picture domain;

[0019] Step S2.3: obtaining a reconstructed old photo through the decoder of the second variational autoencoder;

[0020] The detail feature extraction module comprises three layers of gated convolution.

[0021] Preferably, the first variational autoencoder, the second variational autoencoder, and the mapping network are independently trained respectively.

[0022] The first variational autoencoder is trained to map real old photos and degraded pictures synthesized by adding various degradation kernels into the same latent space;

[0023] The second variational autoencoder is trained to learn the latent space of clean pictures in the process of learning to reconstruct images.

[0024] Preferably, step S3 comprises:

[0025] Step S3.1: Splicing the gray image of the coarse repair result, the extracted corresponding edge map and the corresponding mask together in the channel dimension as the input of the edge repair network, and further repairing the edge map after coarse repair;

[0026] Step S3.2: Extracting the structural information of the further repaired edge map through the structural feature extraction module;

[0027] Step S3.3: The completion network completes the repair of the old photo according to the old photo after coarse repair and the structural information;

[0028] Wherein, the structural feature extraction module comprises an encoder-decoder structure composed of gate convolution, and the outputs of the last three up-sampling gate convolutions are taken as the extracted structural information, and are added to the inputs of the first three layers of the completion network through three trainable weights.

[0029] Preferably, the loss in the training process of the coarse repair network model and the fine repair network model comprises a reconstruction loss, an adversarial loss and a feature matching loss;

[0030] The reconstruction loss calculation formula is as follows:

[0031]

[0032] In the formula, represents a mapping network, is the reconstruction result, I gt is a real clean picture, z x is the latent space encoding vector of the degraded picture, z y is the latent space encoding vector of the clean picture, and represent weights;

[0033] The adversarial loss calculation formula is as follows:

[0034]

[0035] In the formula, represents the discriminator of the coarse repair model;

[0036] The feature matching loss calculation formula is as follows:

[0037]

[0038] In the formula, and are the feature maps output by the i-th layer of the discriminator and the VGG network respectively, and represent the number of activation functions in the layer.

[0039] Preferably, the loss in the fine repair network model training process also includes gradient constraint and high receptive field perception loss:

[0040] The calculation formula of the gradient constraint is as follows:

[0041]

[0042] In the formula, is the gradient operator, D ξ represents the discriminator of the coarse repair model, λ GP represents the weight of the gradient constraint loss, I gt represents the real clean picture;

[0043] The perception loss evaluates the distance between the features extracted from the predicted image and the target image through a basic pre-trained network φ(), and the calculation formula is as follows:

[0044]

[0045] In the formula, φ HRF represents a pre-trained ResNet50 network, λ HRF represents the weight of the perception loss, represents the reconstruction result in the fine repair stage.

[0046] Preferably, it also includes a scratch detection step;

[0047] A scratch detection network based on the Unet network as the main architecture is established, which is first trained only using synthetic images. The loss in the training process includes cross-entropy loss and Focal loss.

[0048] The calculation formula of the cross-entropy loss is as follows:

[0049]

[0050] In the formula, represents the cross-entropy loss, s i and y i respectively represent the picture with scratches and the corresponding mask, H and W respectively represent the height and width of the picture, and The mask obtained by network prediction is represented, and a weight a is used to compensate for the imbalance of positive and negative pixel samples, and the formula is as follows:

[0051]

[0052] The calculation formula of the focal loss is as follows:

[0053]

[0054] Wherein, γ represents a super parameter that can be adjusted, The calculation is as follows:

[0055]

[0056] In the formula, The focal loss is represented.

[0057] According to the old photo repair system based on a deep neural network provided by the application, comprising:

[0058] Module M1: obtaining an old photo to be repaired;

[0059] Module M2: constructing a rough repair network model and training the model, eliminating non-structural damage of the old photo, and obtaining a rough repaired old photo;

[0060] Module M3: constructing a fine repair network model, taking the rough repaired old photo as the input of the model to train the model, eliminating structural damage in the rough repaired old photo, and further obtaining a repaired old photo.

[0061] Preferably, the rough repair network model comprises a variational autoencoder and a mapping network;

[0062] The variational autoencoder comprises a first variational autoencoder and a second variational autoencoder, wherein the first variational autoencoder comprises an encoder, a decoder, a discriminator for a reconstruction result and a discriminator for an encoding vector, and the second variational autoencoder comprises an encoder, a decoder and a discriminator for a reconstruction result;

[0063] The fine repair network model comprises a multi-scale densely connected edge repair network and a Fourier convolution-based completion network;

[0064] The edge repair network comprises three layers of the same dense residual network, and each layer of the network takes the outputs of all previous layers as additional inputs through a jump connection, so as to extract local features from the input.

[0065] Compared with the prior art, the application has the following beneficial effects:

[0066] 1、The old photo repair problem is divided into two categories of structural damage and non-structural damage by adopting a two-stage repair model from coarse to fine, the coarse repair stage focuses on removing non-structural damage in the old photo, the fine repair stage mainly solves the structural damage, and the gradual reconstruction strategy is adopted, so that the reconstruction process is first overall and then local, and the structural integrity, detail richness and neatness of the old photo repair result are improved.

[0067] 2、The old photo repair problem is converted into a mapping problem between the degraded picture domain, the real old photo domain and the clean picture domain, the similarity between the degraded picture and the real old photo in appearance is utilized, and the two are mapped to the same hidden space, so that the paired data set available for training is increased, and the repair quality of the old photo is improved.

[0068] 3、The detail feature extraction module is introduced in the coarse repair stage, the selectable characteristics of the gating convolution are utilized, the overall detail information of the input picture is extracted to assist the reconstruction process of the coarse repair stage, and the problem that the output result is too smooth while removing non-structural noise is solved.

[0069] 4、The edge repair network and the completion network are introduced in the fine repair stage, the large damaged area in the coarse repair result is further filled, and the integrity and structural rationality of the repair result are improved. BRIEF DESCRIPTION OF DRAWINGS

[0070] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments, read in conjunction with the accompanying drawings:

[0071] Figure 1 The workflow schematic diagram of the present application. DETAILED DESCRIPTION

[0072] The present application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, a number of changes and improvements can be made. These all belong to the protection scope of the present application.

[0073] The degradation process of old photos is very complex, and the lost information cannot be recovered by a simple single-stage network, so we divide the restoration of old photos into two stages: coarse restoration and fine restoration. From coarse to fine, we remove non-structural damage such as noise, fading, film grain, and structural damage such as scratches, mold spots, and stains in old photos. The restoration stage learns the mapping relationship between the degraded image domain and the clean image domain to recover the non-structural damage in the old photo; the fine restoration stage focuses on filling the missing or damaged areas in the old photo by a multi-scale residual dense connection edge restoration network and a completion network based on a fast Fourier convolution block. The two-stage joint training enables the network to automatically find the balance between structural damage and non-structural damage restoration.

[0074] Embodiment one

[0075] According to the old photo restoration method based on a deep neural network provided by the application, as shown in the formula: Figure 1 As shown in the formula, the method comprises the following steps:

[0076] Step S1: obtaining an old photo to be restored.

[0077] Step S2: constructing a coarse restoration network model and training the model to eliminate the non-structural damage of the old photo and obtain a coarsely restored old photo. The coarse restoration network model comprises a variational autoencoder and a mapping network. The variational autoencoder comprises a first variational autoencoder and a second variational autoencoder, wherein the first variational autoencoder comprises an encoder, a decoder, a discriminator for a reconstruction result, and a discriminator for an encoding vector, and the second variational autoencoder comprises an encoder, a decoder, and a discriminator for a reconstruction result. Step S2 comprises the following steps:

[0078] Step S2.1: obtaining an encoding vector through the encoder of the first variational autoencoder, and simultaneously extracting image detail information in the intermediate feature map through a detail feature extraction module. The detail feature extraction module comprises three layers of gate convolution. The gate convolution can automatically select useful information in the input features for transmission by learning.

[0079] Step S2.2: converting the degraded image domain to the clean image domain through the mapping network.

[0080] Step S2.3: obtaining a reconstructed old photo through the decoder of the second variational autoencoder.

[0081] Specifically, the first variational autoencoder, the second variational autoencoder and the mapping network are trained independently respectively. The first variational autoencoder is trained to map the real old photos and the degraded pictures synthesized by adding various degradation kernels into the same latent space. The second variational autoencoder is trained to learn the latent space of clean pictures in the process of learning to reconstruct images. The mapping network is trained to learn the mapping relationship between the degraded picture domain and the clean picture domain, and the gradients in the two variational autoencoders are frozen, and only the mapping network is set to the training mode. The real old photos and the synthesized degraded pictures are both damaged pictures, and they are similar in appearance. Therefore, the mapping process from the degraded picture domain to the clean picture domain can also be applied to the real old photos. Moreover, the conversion in the low-dimensional latent space is simpler than the conversion in the complex image domain. Since the two variational autoencoders and the mapping network are independently trained separately, they do not interfere with each other, so the result reconstructed from the clean picture domain finally will conform to the characteristics of the clean picture.

[0082] The loss in the training process of the coarse repair network model includes reconstruction loss, adversarial loss and feature matching loss. It should be noted that the loss function used in the training process of the first variational autoencoder and the second variational autoencoder also includes KL divergence. Specifically as follows:

[0083] a. The reconstruction loss calculates the L1 loss between the reconstruction result and the real clean picture, and the L1 loss between the feature vector of the degraded picture converted to the clean picture domain by the mapping network and the feature vector of the real clean picture in the latent space. The calculation formula is as follows:

[0084]

[0085] In the formula, represents the mapping network, is the reconstruction result, I gt is the real clean picture, z x is the latent space encoding vector of the degraded picture, z y is the latent space encoding vector of the clean picture, and represent the weights;

[0086] b. In order to generate natural pictures and more realistic details, the adversarial loss is adopted, and the calculation formula of the adversarial loss is as follows:

[0087]

[0088] In the formula, represents the discriminator of the coarse repair model.

[0089] c. Feature matching loss is used to stabilize GAN training, and L1 distance between the outputs of multiple layers of the pre-trained VGG network and the discriminator input with real and synthetic samples is calculated, and the calculation formula is as follows:

[0090]

[0091] In the formula, and are the feature maps of the i-th layer output of the discriminator and the VGG network, respectively, and represent the number of activation functions in the layer.

[0092] Step S3: constructing a fine repair network model, training the model by taking the coarse repaired old photo as the input of the model, eliminating the structural damage in the coarse repaired old photo, and then obtaining the repaired old photo. The fine repair network model includes a multi-scale densely connected edge repair network and a Fourier convolution-based completion network. The edge repair network includes three layers of the same dense residual network, which can enhance feature propagation, encourage feature reuse, and reduce the number of parameters. Each layer of the network takes the output of all previous layers as additional input through a jump connection, so that local features are extracted from the input. At the same time, the structure of a single dense residual network only extracts image features at one resolution, and the network has a limited receptive field. Since the scales and shapes of the missing edges are different, features of different scales are combined to capture various edge characteristics. Specifically, step S3 includes:

[0093] Step S3.1: concatenating the gray-scale image of the coarse repair result, the extracted corresponding edge map, and the corresponding mask in the channel dimension to form an input of the edge repair network, and further repairing the coarse repaired edge map. At the same time, downsample the input by 4 times and 8 times, respectively, and input it into the upper, middle, and lower layers of the repair network, so that the network can have a larger receptive field by calculating features of different scales, and the edge repair network can be trained using binary cross entropy loss (BCE).

[0094] Step S3.2: extracting structural information of the further repaired edge map through a structural feature extraction module. The edge map is a sparse image, in which most of the content is meaningless 0 value, making it difficult to extract structural features. Therefore, the structural feature extraction module includes an encoder-decoder structure composed of gated convolution, and the outputs of the last three upsampled gated convolutions are taken as the extracted structural information, which is added to the input of the first three layers of the completion network through three trainable weights.

[0095] Step S3.3: The completion network completes the restoration of the old photo based on the coarsely restored old photo and the structural information. The completion network consists of stacked Fast Fourier Convolutional layers. Specifically, the Fast Fourier Convolutional layers are based on channel Fast Fourier Transform (FFT) and have a receptive field covering the entire image range. The Fourier Convolutional layers divide the channels into two parallel branches: a local branch and a global branch. The local branch uses conventional convolution, while the global branch uses real-signal FFT to consider the global context. Therefore, the use of Fourier convolution allows the generator network to consider the global context from early layers, which is crucial for high-resolution image restoration.

[0096] The losses in the training process of the fine-repair network model include reconstruction loss, adversarial loss, feature matching loss, gradient constraint, and high receptive field perception loss.

[0097] Specifically, the reconstruction loss, adversarial loss, and feature matching loss are implemented in the same way as the steps during the training of the coarse-repair network model. It should be noted that when calculating the adversarial loss, the fine-repair stage only treats the regions hidden by the mask as fake samples in the discriminator. Furthermore, the calculation of gradient constraints and high receptive field perceptual loss is as follows:

[0098] The formula for calculating gradient constraints is as follows:

[0099]

[0100] In the formula, For the gradient operator, D ξ The discriminator, λ, represents the coarse-repair model. GP I represents the weights of the gradient constraint loss. gt Represents a truly clean image;

[0101] Perceptual loss evaluates the distance between features extracted from the prediction image and the target image through a basic pre-trained network φ(). In other words, traditional supervised loss requires the generator to accurately reconstruct features from the original image. Figure 1 However, the visible portion of an image often lacks sufficient information to accurately reconstruct the masked portion. Therefore, traditional supervision can lead to models generating an average pattern of multiple seemingly plausible results for the repaired content, resulting in blurry outcomes. In contrast, perceptual loss evaluates the distance between features extracted from the prediction and target images through a basic pre-trained network. It does not require an exact reconstruction relative to the original image, allowing for appropriate variations in the reconstruction result within a reasonable range of content. Large mask inpainting focuses on understanding the overall structure; therefore, using a base network with a rapidly growing receptive field is crucial for calculating the perceptual loss. The calculation formula is as follows:

[0102]

[0103] In the formula, φ HRF represents a pre-trained ResNet50 network, λ HRF represents the weight of the perception loss, represents the reconstruction result of the fine repair stage.

[0104] Further, in the coarse repair stage, during the down-sampling, domain alignment and mapping processes, part of the detail information in the image is inevitably lost, resulting in the generated result being too fuzzy and smooth. In order to solve this problem, three detail feature extraction modules of different scales are added in the first variational autoencoder, and the calculated detail features are added as an amount to the decoder of the second variational autoencoder. Taking the output of the first three layers of the first variational autoencoder ensures that the picture information has not been lost too much, and part of the noise can be attenuated to a certain extent in the down-sampling process. At the same time, the output of the coarse repair model is taken as the input of the fine repair model, and the end-to-end joint training is carried out. Let the reconstruction process be overall first and local second, and gradually reconstruct the clean, complete structure and detailed repair result.

[0105] The old photo repair method based on a deep neural network provided by the application further comprises a scratch detection step;

[0106] The scratch detection network with Unet network as the main architecture is established, and first, only synthetic images are used for training. The loss in the training process includes cross-entropy loss and Focal loss.

[0107] The calculation formula of the cross-entropy loss is as follows:

[0108]

[0109] In the formula, represents the cross-entropy loss, s i and y i respectively represent the picture with scratches and the corresponding mask, H and W respectively represent the height and width of the picture, and represent the mask predicted by the network, and the weight alpha is used to make up for the imbalance of positive and negative pixel samples. Since the area with scratches is usually a small part of the entire image, the weight alpha is used to make up for the imbalance of positive and negative pixel samples. The value of alpha is determined by the positive / negative ratio of y i .

[0110]

[0111] The calculation formula of the Focal loss is as follows:

[0112]

[0113] wherein γ represents a tunable hyperparameter, is calculated as follows:

[0114]

[0115] wherein, denotes the Focal loss.

[0116] In order to further improve the detection performance on real old photos, the collected old photos with scratches are manually labeled, and the detection network is fine-tuned. At the same time, a user interaction interface is set, and the user selects to perform scratch automatic detection, or manually paints the repair area by himself / herself, and completes the repair of the input old photo according to the obtained mask to adapt to different repair and application requirements.

[0117] Embodiment two

[0118] The application further provides an old photo repair system based on a deep neural network. A person skilled in the art can realize the old photo repair system based on the deep neural network by performing the step flow of the old photo repair method based on the deep neural network, that is, the old photo repair method based on the deep neural network can be understood as a preferred embodiment of the old photo repair system based on the deep neural network.

[0119] According to the application, an old photo repair system based on a deep neural network is provided, which comprises:

[0120] Module M1: obtaining an old photo to be repaired.

[0121] Module M2: constructing a coarse repair network model and training the model to eliminate non-structural damage of the old photo and obtain a coarsely repaired old photo. The coarse repair network model comprises a variational autoencoder and a mapping network. The variational autoencoder comprises a first variational autoencoder and a second variational autoencoder, wherein the first variational autoencoder comprises an encoder, a decoder, a discriminator for a reconstruction result and a discriminator for an encoding vector, and the second variational autoencoder comprises an encoder, a decoder and a discriminator for a reconstruction result. Specifically, the first variational autoencoder, the second variational autoencoder and the mapping network are independently trained. The first variational autoencoder is trained to map a real old photo and a degraded picture synthesized by adding various degradation kernels into the same hidden space; the second variational autoencoder is trained to learn the hidden space of a clean picture in the process of learning to reconstruct an image. The following sub-modules are further included:

[0122] Module M2.1: Obtain an encoding vector through the encoder of the first variational autoencoder, and meanwhile extract image detail information in the intermediate feature map through a detail feature extraction module. The detail feature extraction module includes three layers of gated convolution. Module M2.2: Convert the degradation picture domain to the clean picture domain through a mapping network by using the encoding vector. Module M2.3: Obtain a reconstructed old photo through the decoder of the second variational autoencoder.

[0123] Module M3: Construct a fine restoration network model, train the model by using the old photo after the coarse restoration as the input of the model, eliminate structural damage in the old photo after the coarse restoration, and then obtain a restored old photo. The fine restoration network model includes a multi-scale densely connected edge restoration network and a Fourier convolution-based completion network. The edge restoration network includes three layers of the same densely connected residual network, and each layer of the network takes the outputs of all previous layers as additional inputs through a jump connection, so that local features are extracted from the input. Specifically, the following sub-modules are included:

[0124] Module M3.1: Concatenate the gray-scale image of the coarse restoration result, the extracted corresponding edge map, and the corresponding mask in the channel dimension to obtain an input of the edge restoration network, and further restore the edge map after the coarse restoration.

[0125] Module M3.2: Extract structural information of the edge map after the further restoration through a structural feature extraction module. The structural feature extraction module includes an encoder-decoder structure composed of gated convolution, takes the outputs of the last three up-sampling gated convolutions as the extracted structural information, and adds the structural information to the inputs of the first three layers of the completion network through three trainable weights.

[0126] Module M3.3: The completion network completes the restoration of the old photo according to the old photo after the coarse restoration and the structural information.

[0127] Module M4: Establish a scratch detection network with Unet network as the main architecture. First, only synthetic images are used for training, and the loss in the training process includes cross-entropy loss and Focal loss.

[0128] The calculation formula of the cross-entropy loss is as follows:

[0129]

[0130] In the formula, L denotes the cross-entropy loss, s denotes the cross-entropy loss, s i and y i denote the picture with scratches and the corresponding mask, respectively, H and W denote the height and width of the picture, respectively, and The mask obtained by network prediction is represented, and the weight a is used to make up for the imbalance of positive and negative pixel samples, and the formula is as follows:

[0131]

[0132] The calculation formula of the focal loss is as follows:

[0133]

[0134] Wherein, γ represents an adjustable hyperparameter, The calculation is as follows:

[0135]

[0136] In the formula, Indicates the focal loss. Further, the loss in the training process of the coarse repair network model and the fine repair network model includes a reconstruction loss, an adversarial loss and a feature matching loss.

[0137] The calculation formula of the reconstruction loss is as follows:

[0138]

[0139] In the formula, Represents a mapping network, Is the reconstruction result, I gt Is the real clean picture, z x Is the hidden space encoding vector of the degraded picture, z y Is the hidden space encoding vector of the clean picture, And Indicate the weights.

[0140] The calculation formula of the adversarial loss is as follows:

[0141]

[0142] In the formula, Indicates the discriminator of the coarse repair model.

[0143] The calculation formula of the feature matching loss is as follows:

[0144]

[0145] In the formula, And Are the feature maps output by the i-th layer of the discriminator and the VGG network respectively, And Indicate the number of activation functions in the layer.

[0146] In addition, the loss in the fine repair network model training process further includes gradient constraint and high receptive field perception loss: the calculation formula of the gradient constraint is as follows:

[0147]

[0148] In the formula, φ is a gradient operator, D ξ denotes the discriminator of the coarse repair model, λ GP denotes the weight of the gradient constraint loss, I gt denotes a real clean picture.

[0149] The perception loss evaluates the distance between the features extracted from the predicted image and the target image through a basic pre-trained network φ(), and the calculation formula is as follows:

[0150]

[0151] In the formula, φ HRF denotes a pre-trained ResNet50 network, λ HRF denotes the weight of the perception loss, denotes the reconstruction result in the fine repair stage.

[0152] Those skilled in the art know that, in addition to implementing the system, device and each module thereof provided by the present application in the form of pure computer readable program code, the same program can also be realized by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, the system, device and each module thereof provided by the present application can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing methods and structures within the hardware component.

[0153] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for restoring old photos based on a deep neural network, characterized in that, The method comprises the following steps: Step S1: obtaining an old photo to be restored; Step S2: constructing a coarse restoration network model and training the model to eliminate non-structural damage of the old photo, and obtaining a coarsely restored old photo; Step S3: constructing a fine restoration network model, taking the coarsely restored old photo as the input of the model to train the model, eliminating structural damage in the coarsely restored old photo, and further obtaining a restored old photo; The coarse restoration network model comprises a variational autoencoder and a mapping network; The variational autoencoder comprises a first variational autoencoder and a second variational autoencoder, wherein the first variational autoencoder comprises an encoder, a decoder, a discriminator for a reconstruction result, and a discriminator for an encoding vector, and the second variational autoencoder comprises an encoder, a decoder, and a discriminator for a reconstruction result; The fine restoration network model comprises a multi-scale densely connected edge restoration network and a Fourier convolution-based completion network; The edge restoration network comprises three layers of the same dense residual network, and each layer of the network takes the outputs of all previous layers as additional inputs through a jump connection, so that local features are extracted from the input. 2.The deep neural network-based old photo restoration method of claim 1, wherein, Step S2 comprises: Step S2.1: obtaining an encoding vector through the encoder of the first variational autoencoder, and simultaneously extracting image detail information in an intermediate feature map through a detail feature extraction module; Step S2.2: converting the encoding vector through the mapping network to convert a degraded picture domain to a clean picture domain; Step S2.3: obtaining a reconstructed old photo through the decoder of the second variational autoencoder; The detail feature extraction module comprises three layers of gate convolution. 3.The deep neural network-based old photo restoration method of claim 2, wherein, The first variational autoencoder, the second variational autoencoder, and the mapping network are independently trained respectively; The first variational autoencoder is trained to map a real old photo and a degraded picture synthesized by adding various degradation kernels to the same hidden space; The second variational autoencoder is trained to learn the hidden space of a clean picture in the process of learning to reconstruct an image. 4.The deep neural network-based old photo restoration method of claim 1, wherein, Step S3 comprises: Step S3.1: concatenating a gray image of the coarse restoration result, an extracted corresponding edge map, and a corresponding mask in a channel dimension to serve as an input of the edge restoration network, and further restoring the edge map after the coarse restoration; Step S3.2: extracting structural information of the further restored edge map through a structural feature extraction module; Step S3.3: completing the restoration of the old photo according to the old photo after the coarse restoration and the structural information by the completion network; The structural feature extraction module comprises an encoder-decoder structure composed of gate convolution, and the outputs of the last three up-sampling gate convolutions are taken as extracted structural information and added to the inputs of the first three layers of the completion network through three trainable weights.

5. The deep neural network-based old photo restoration method according to any one of claims 1 to 4, characterized in that, The losses in the training processes of the coarse restoration network model and the fine restoration network model comprise a reconstruction loss, an adversarial loss, and a feature matching loss; The calculation formula of the reconstruction loss is as follows: wherein represents a mapping network, is the reconstruction result for the coarse repair stage, I gt is the real clean picture, z x is the latent space encoding vector for the degraded picture, z y is the latent space encoding vector for the clean picture, and denotes a weight; The calculation formula of the adversarial loss is as follows: In the formula, denotes a discriminator representing a coarse repair model; The calculation formula of the feature matching loss is as follows: wherein and are the feature maps of the i-th layer output of the discriminator and VGG network, respectively, and denotes the number of activation functions in this layer. 6.The deep neural network-based old photo restoration method of claim 4, wherein, The losses in the training process of the fine restoration network model further comprise a gradient constraint and a high receptive field perception loss: The calculation formula of the gradient constraint is as follows: In the formula, is a gradient operator, D ξ is a discriminator representing the coarse repair model, λ GP is a weight representing the gradient constraint loss, I gt is a real clean picture; The perceptual loss evaluates the distance between the features extracted from the predicted image and the target image through a basic pre-training network φ(), and the calculation formula is as follows: In the formula, φ HRF represents a pre-trained ResNet50 network, λ HRF represents the weight of the perception loss, represents the reconstruction result of the fine-tuning stage. 7.The deep neural network-based old photo restoration method of claim 1, wherein, The scratch detection step is also included; A scratch detection network with Unet network as the main architecture is established, and first, only synthetic images are used for training, and the loss in the training process includes cross-entropy loss and Focal loss. The calculation formula of the cross-entropy loss is as follows: wherein denotes the cross-entropy loss, s i and y i denote the picture with scratches and the corresponding mask, respectively, H and W denote the height and width of the picture, respectively, and denotes the mask predicted by the network, while the weight a is used to compensate for the imbalance between positive and negative pixel samples, as follows: The calculation formula of the Focal loss is as follows: where γ denotes a tunable hyperparameter, The calculation is as follows: In the formula, represents the Focal loss. 8.A deep neural network-based old photo restoration system, characterized by, It includes: Module M1: obtaining an old photo to be repaired; Module M2: constructing a coarse repair network model and training the model, eliminating non-structural damage of the old photo, and obtaining a coarsely repaired old photo; Module M3: constructing a fine repair network model, taking the coarsely repaired old photo as the input of the model to train the model, eliminating structural damage in the coarsely repaired old photo, and further obtaining a repaired old photo; The coarse repair network model includes a variational autoencoder and a mapping network; The variational autoencoder includes a first variational autoencoder and a second variational autoencoder, wherein the first variational autoencoder includes an encoder, a decoder, a discriminator for a reconstruction result, and a discriminator for an encoding vector, and the second variational autoencoder includes an encoder, a decoder, and a discriminator for a reconstruction result; The fine repair network model includes a multi-scale densely connected edge repair network and a Fourier convolution-based completion network; The edge repair network includes three layers of the same dense residual network, and each layer of the network takes the outputs of all previous layers as additional inputs through a skip connection, so as to extract local features from the input.

Images