Method for comprehensively repairing scratched old photo

Through the combination of deep learning models, the problems of blurred image quality, misalignment of texture and low facial detection efficiency of old photo scratches and portrait photos are solved, and high-quality scratch noise removal and facial integrity repair are achieved, improving photos Recovery effect of the appearance and detail.

CN119006338AActive Publication Date: 2024-11-22HUBEI UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410915044.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-22
Estimated Expiration
2044-07-09

AI Technical Summary

Technical Problem

The existing old photo repair technology has problems such as blurred image quality, misalignment of texture, low facial detection efficiency, and overflow of contour color when dealing with scratches and portrait photos, especially in photos with small facial areas.

Method used

By training deep learning models, using diffusion model, noise prediction network, image super-resolution network, face detection network and image shading network, we process scratches, noise, image details and face recovery respectively to achieve scratch noise removal and image Improved clarity and facial integrity repair.

Benefits of technology

It realizes high-quality repair of scratched old photos, especially in portrait photos, improves the appearance, solves the problems of blurred image quality and texture misalignment, and improves facial detection accuracy and image detail recovery effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006338A_ABST
    Figure CN119006338A_ABST
Patent Text Reader

Abstract

The invention discloses a method for comprehensively repairing a scratched old photo, which comprises the following steps of: detecting scratches and human faces in the photo through different detectors by training a deep learning model network, and respectively repairing the scratches and the human faces through different technical schemes; and through a small target face detection algorithm and a face generation recovery network, even a portrait picture with a damaged face can have a good-impression recovery effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of low-level computer vision technology, and in particular relates to a method for comprehensive restoration of scratched old photos. Background Art

[0002] With the development of intelligent devices and the popularization of computer resources, it has become simple and efficient to convert traditional paper photos into digital formats. Compared with paper photos, the advantage of digital archives is that they are easy to edit, consult and disseminate. As paper photos are damaged or creased over time, most of them will be irreversibly damaged, and most of the scanned photos have the problem of blurred image quality and rough structure, among which the portrait photos are particularly serious. The problem of comprehensive restoration of scratched old photos can be divided into traditional methods and methods based on deep learning. The traditional restoration method mainly manually marks the area to be repaired, and the algorithm calculates the filling pixels according to the edge pixels of the area to be repaired, and finally iterates and fills them one by one to achieve the purpose of restoration. However, this method lacks a deep understanding of the image content, and there are still problems of blur and discontinuous picture structure after restoration. In the past two years, deep learning technology has developed rapidly and has been widely used in image processing tasks. At present, the old photo restoration methods based on deep learning are mostly focused on the restoration of old photos with single degradation information or still have the problems of blurred image quality and texture dislocation. The image restoration method based on the diffusion model can control the structural information of the damaged image from a global perspective to complete the restoration of the damaged image. In portrait photos, due to the incomplete content and low resolution of degraded old photos, the detection and restoration of human faces also face certain challenges. Most of the existing old photo restoration technical solutions do not perform targeted operations and processing on portrait photos. After enhancing the face and image quality, the details of the old photos can be sharper and clearer. Finally, colorizing the old photos can make the repair effect more vivid and better looking. Most of the existing old photo scratch repair technical solutions have the problem of distorted repair effects, especially in portrait photos. For scratches on single portraits and group photos, the appearance is not good after repair. The face detection efficiency in the existing photo repair system is low, and it cannot be applied to group photos or old photos with small facial areas. In addition, when repairing and coloring old photos, due to the low clarity and less texture details of the image, there will be a phenomenon of contour color overflow. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a method for comprehensive restoration of scratched old photos. By training a deep learning model network, scratches and faces in the photos are detected by different detectors and repaired respectively by different technical solutions; through a small target face detection algorithm and a face generation and restoration network, even portrait photos with damaged faces can have a relatively good restoration effect.

[0004] To achieve the above object, the present invention provides a method for comprehensive repair of scratched old photos, comprising:

[0005] S1. Establishing a diffusion model, and adding noise to the old photo with scratches to be repaired based on the noise adding method of the diffusion model to obtain a complete Gaussian noise map; establishing a scratch denoising space according to the old photo with scratches to be repaired;

[0006] S2, establishing a noise prediction network, performing noise prediction on the complete Gaussian noise map and the scratch denoising space, obtaining corresponding pure noise maps respectively, and calculating the scores of each of the pure noise maps;

[0007] S3, weighted calculation of the scores of each of the pure noise images to obtain a state graph; based on the denoising method of the diffusion model, reverse iterative denoising processing is performed on the state graph to obtain an old photo without scratches;

[0008] S4, establishing an image super-resolution network, using a convolutional neural network, extracting a shallow feature map of the scratch-free old photo, and directly inputting the shallow feature map into a reconstruction module through a long skip connection to retain low-frequency information;

[0009] S5, inputting the shallow feature map into a deep feature extraction module, performing deep feature modeling on the shallow feature map, and obtaining a deep feature map for recovering the lost high-frequency information;

[0010] S6. Through long skip connections, the low-frequency information is directly input into the reconstruction module to help the deep feature extraction module focus on the high-frequency information. The low-frequency information and the high-frequency information are upsampled using a sub-pixel convolution layer. The scratch-free old photo is reconstructed by aggregating the upsampled low-frequency information and the high-frequency information, and 4x super-resolution is performed to obtain a clear scratch-free old photo.

[0011] S7, establishing a face detection network, inputting the clear and scratch-free old photo into the face detection network to perform face detection in the photo, and obtaining a cropped image of a local face area;

[0012] S8, establishing a face restoration network, processing the local area cropped image of the face separately to perform face restoration, and obtaining a restored face image;

[0013] S9, performing histogram matching on the repaired face and the face detection position to obtain a clear photo without scratches and with a complete face;

[0014] S10. Establish an image colorization network to colorize the obtained clear, scratch-free photo with a complete face, so as to obtain a color photo with no scratches and a clear face.

[0015] Furthermore, the method to obtain a completely Gaussian noise map is:

[0016] dx=θ t (η-x)dt+ξ t dw

[0017] Among them, x is the state diagram at time t, θ t (η-x) and ξ t are drift and dispersion equations respectively, η is the scratched old photo, dw is Gaussian noise, θ t With ξ t is a positive parameter related to time.

[0018] Furthermore, the method of establishing a scratch denoising space according to the old photo with scratches to be repaired includes:

[0019] Based on an odd-numbered Gaussian convolution kernel, Gaussian blur processing is performed on the old photo with scratches to be repaired, so as to obtain a brightness information feature map that retains the brightness information of the picture and eliminates the scratches;

[0020] Establishing and training a scratch detection network, performing scratch detection on the old photos with scratches to be repaired, and obtaining a normal distribution of scratches;

[0021] The scratch denoising space is established based on the brightness information feature map and the scratch normal distribution.

[0022] Further, the method of establishing and training the scratch detection network includes:

[0023] Based on real scratched old photos and synthetic scratched simulated old photos, a scratched old photo dataset is established;

[0024] The scratch area of ​​each old photo in the scratch old photo data set is marked, and the scratch old photos and the marked photos to be segmented are made to correspond one to one, and an image processing network is trained to obtain scratch detection results;

[0025] Based on the scratch detection result, a cross entropy loss is used to minimize the difference between the detected scratch and the actual scratch;

[0026] Based on the difference, the objective function of the scratch detection network is obtained, and the establishment and training of the scratch detection network are completed.

[0027] Furthermore, the method for obtaining the normal distribution of scratches includes:

[0028] Based on the scratch detection network, scratch detection is performed on the old photo with scratches to be repaired to obtain a binary image mask with scratch position information;

[0029] Performing a cropping operation on the binary image mask and the old photo with scratches to be repaired to obtain a feature image with only scratch information;

[0030] The feature map containing only scratch information is encoded to obtain a normal distribution of scratches on the old photo with scratches to be repaired.

[0031] Furthermore, the method for obtaining the scratch-free old photo includes:

[0032] Performing weighted calculation on the scores of each of the pure noise images to obtain a final score calculation result;

[0033] Based on the final score calculation result, obtaining a state diagram for the next stage;

[0034] Based on the denoising method of the diffusion model, the state diagram is subjected to reverse iterative denoising processing to exclude scratch information and generate global semantic information to obtain the scratch-free old photo.

[0035] Furthermore, a method for extracting a shallow feature map of the scratch-free old photo using a convolutional neural network is as follows:

[0036] F0=H SF (I LQ )

[0037] Among them, F0 is the shallow feature map; I LQ For old photos without scratches; H SF (.) is a convolutional neural network.

[0038] Furthermore, the method of performing deep feature modeling on the shallow feature map to obtain the deep feature map is as follows:

[0039] F DF =H DF (F0)

[0040] Among them, F DF is the deep feature map; H DF (.) is the deep feature extraction module; F0 is the shallow feature map.

[0041] Furthermore, the method for obtaining the clear and scratch-free old photo is:

[0042] I RHQ =H REC (F0+F DF )

[0043] Among them, I RHQ For clear old photos without scratches; H REC (.) is the reconstruction module; F DF is a deep feature map; F0 is a shallow feature map.

[0044] Technical effect of the invention: The present invention discloses a method for comprehensive restoration of old photos with scratches. By training a deep learning model network, scratches and faces in the photos are detected by different detectors and repaired respectively by different technical solutions; through a small target face detection algorithm and a face generation and restoration network, even portrait photos with damaged faces can have a relatively good restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0046] Figure 1 A schematic diagram of a process for comprehensively repairing scratched old photos according to an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the process of the method for removing scratches from old photos according to an embodiment of the present invention;

[0048] Figure 3 A network diagram for noise prediction according to an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of the process of an image super-resolution method according to an embodiment of the present invention;

[0050] Figure 5 This is a structural diagram of face enhancement according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0052] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] like Figure 1 As shown, this embodiment provides a method for comprehensive repair of scratched old photos, including:

[0054] S1. Establish a diffusion model, and add noise to the old photo with scratches to be repaired based on the noise adding method of the diffusion model to obtain a complete Gaussian noise map; and establish a scratch denoising space;

[0055] The specific method of establishing the diffusion model is:

[0056] The signal is added by adding noise using stochastic differential equations, namely Brownian motion, which is a continuous Gaussian process that injects stable, random noise into the system at every continuous moment, turning the discrete noise addition and denoising process into a continuous process. Where w represents Brownian motion (or Wiener process), θ t (η-x) controls the “flow direction” of the entire SDE, ξ t Control the diffusion degree or noise adding speed of SDE. Randomly sample and add Gaussian noise at each step, and the variance gradually increases, and finally it reaches a complete Gaussian noise map. That is, at each continuous moment, inject Gaussian noise into the old photo with scratches to be repaired to obtain a complete Gaussian noise map. The formula is as follows:

[0057] dx=θ t (η-x)dt+ξ t dw,

[0058] Among them, x is the state diagram at time t, θ t (η-x) and ξ t are drift and dispersion equations respectively, η is the scratched old photo, dw is Gaussian noise, θ t With ξ t is a positive parameter related to time; in this process, this embodiment sets the overall time T to 100, and through the reference of the SDE stochastic differential equation diffusion method, the noise addition is changed from a discrete process to a continuous process.

[0059] The reverse iterative denoising method is used to denoise the complete Gaussian noise map and the scratch denoising space. The formula is as follows:

[0060]

[0061] in, is the logarithmic gradient function of the probability density; x is the state diagram at time t (that is, the state diagram at time t of the complete Gaussian noise diagram or the scratch denoising space)

[0062] Based on the stochastic differential equation denoising method and the reverse iterative denoising method, the discrete denoising and denoising process is transformed into a continuous denoising and denoising process to complete the establishment of the diffusion model.

[0063] Based on the old photo with scratches to be repaired, a scratch denoising space is established. The space is constructed by using the distribution of scratches to predict noise in the Gaussian blurred photo. t, the Gaussian blurred photo and x(t) are input into the noise prediction network for noise prediction, because this can retain the brightness information of the original image while excluding the scratch information.

[0064] A further implementation method is that in step S1, the method for establishing the scratch denoising space is:

[0065] like Figure 2 As shown, based on an odd-numbered Gaussian convolution kernel, the weighted average value of the center point of the pixel value of the old photo with scratches to be repaired is calculated to complete the Gaussian blur processing. After this process, a brightness information feature map is obtained that retains the brightness information of the picture and eliminates high-frequency information such as scratches;

[0066] Establishing and training a scratch detection network, performing scratch detection on the old photos with scratches to be repaired, and obtaining a normal distribution of scratches;

[0067] The scratch denoising space is established based on the brightness information feature map and the scratch normal distribution.

[0068] A further implementation method is that the method for obtaining the brightness information feature map is:

[0069] Based on the odd-numbered Gaussian convolution kernel, the weighted average value of the center point of the old photo with scratches to be repaired is calculated, and the Gaussian blur processing is completed to obtain the brightness information feature map.

[0070] This scratch detection and segmentation network uses the widely used image processing network Unet. By fine-tuning and retraining this network, it is used in this scratch detection and segmentation scenario. Unet extracts the scratch features of the image through multiple downsampling and upsampling operations.

[0071] A further implementation method is that the method of establishing and training the scratch detection network is:

[0072] Based on real scratched old photos and synthetic scratched simulated old photos, a scratched old photo dataset is established;

[0073] The scratch area of ​​each old photo in the scratch old photo dataset is marked, and the scratch old photos and the marked photos to be segmented are matched one by one, and the image processing network Unet is trained to obtain the scratch detection results;

[0074] Based on the scratch detection results, the cross entropy loss is used to minimize the difference between the detected scratch mask and the actual scratch mask y;

[0075] Based on the difference, the objective function of the scratch detection network is obtained, and the establishment and training of the scratch detection network are completed.

[0076] A further implementation method is that the method for obtaining the normal distribution of scratches is:

[0077] Based on the scratch detection network, scratch detection is performed on the old photos with scratches to be repaired, and a binary image mask with scratch position information is obtained;

[0078] The binary image mask and the old photo of the scratch to be repaired are cropped to obtain a feature map with only scratch information;

[0079] The encoder is used to encode the feature map with only scratch information to obtain the normal distribution of scratches in the old photo to be repaired. Specifically, through a simple cropping operation, a picture with only scratches can be obtained. By encoding this photo, a set of means and a set of variances are obtained, and these means and variances correspond to a Gaussian distribution respectively.

[0080] S2: Establish a noise prediction network, perform noise prediction on the complete Gaussian noise map and the scratch denoising space, obtain the corresponding pure noise maps respectively, and calculate the scores of each pure noise map. Specifically, perform reverse iterative denoising on the output in S1, where the only unknown is The probability density logarithmic gradient function is the score. A noise prediction network based on U-net is trained by the maximum likelihood estimation method to more accurately predict the current state of noise. In the test generation, the current state of noise is predicted by this network to obtain a pure noise map, and finally the score is calculated. The network diagram is as follows: Figure 3 shown.

[0081] S3: Calculate the scores of each pure noise image by weighted calculation to obtain a state image; wherein the score is a probability density logarithmic gradient function; perform reverse iterative denoising on the state image based on the denoising method of the diffusion model to obtain a scratch-free and clean old photo.

[0082] A further implementation method is that in step S3, the method for obtaining a clean old photo without scratches is:

[0083] Perform weighted calculation on the scores of each pure noise image to obtain the final score calculation result;

[0084] Based on the final score calculation result, obtaining a state diagram for the next stage;

[0085] Based on the denoising method of the diffusion model, the state diagram is subjected to reverse iterative denoising processing to exclude scratch information and generate global semantic information to obtain the scratch-free and clean old photo.

[0086] In this embodiment, the number of normal distributions in S2 is set to 3. This distribution generates three scores in the Gaussian state space. Together with the score predicted by the scratch noise map, the four scores are combined and calculated to obtain the final score. The calculation method generated in reverse in S3 is used to obtain the next state map according to the final score.

[0087] According to the calculated score, x(t), the state diagram x(t-1) of the next stage can be obtained. After repeated iterations, this process can be regarded as the reverse recovery of S1. However, in the reverse recovery process, the information of scratches is excluded and the semantic information that conforms to the global situation is generated. The predicted noise and scratches are regarded as abnormal information and taken away from the noise. The removed noise and x(t) are filled with x(t-1) according to the reverse formula. It can be simply understood as the denoising process moving towards the target image in a vector space. From T to the gradual restoration to X, the scratches are gradually removed and new content is generated, and finally a clear and scratch-free photo is obtained, that is, the repaired photo.

[0088] S4: An image super-resolution network is established, and a convolutional neural network is used to extract shallow feature maps of the scratch-free old photos, and directly transmit them to the reconstruction module through long skip connections to retain low-frequency information.

[0089] Image super-resolution process, such as Figure 4 As shown, a further implementation method is that based on the restored scratch-free old photo LQ, Then the convolution module H is used SF (.) Extract shallow features F0∈R H,W,C

[0090] F0=H SF (I LQ ),

[0091] S5: Send the shallow feature map to the deep feature extraction module, perform deep feature modeling on the shallow features, and obtain a deep feature map for recovering the lost high-frequency information.

[0092] A further implementation method is to use the shallow feature F0 obtained to further extract features using a deep feature extraction module HDF. The deep feature extraction module is composed of K residual Swin Transformer blocks (RSTB) and a 3×3 convolution.

[0093] F DF =H DF (F0),

[0094] Each RSTB output F1, F2, F K , and the output deep features F DK , where represents the ith RSTB module, H CONV Represents the final convolutional layer. The convolutional layer can introduce the inductive bias of convolution into the Transformer-based network, laying the foundation for the subsequent fusion of shallow and deep features.

[0095]

[0096] F DF =H CONV (F K ),

[0097] S6: Through long jump connections, low-frequency information is directly transmitted to the reconstruction module to help the deep feature extraction module focus on high-frequency information. The sub-pixel convolution layer is used to upsample the low-frequency information and the high-frequency information. The scratch-free old photos are reconstructed by aggregating the upsampled low-frequency information and high-frequency information, and 4x super-resolution is performed to obtain clear scratch-free old photos.

[0098] A further implementation method is to fuse the shallow feature F0 and the deep feature F DK To reconstruct high quality images I RHQ , where H REC To rebuild the module.

[0099] I RHQ =H REC (F0+F DF ),

[0100] The shallow feature F0 mainly contains low-frequency information, while the deep feature focuses on recovering the lost high-frequency information. Specifically, using a long-distance connection to transmit low-frequency information directly to the reconstruction module can help the deep feature extraction module focus on high-frequency information and stabilize training. The features are upsampled through the sub-pixel convolution layer to achieve reconstruction.

[0101] S7: Establishing a face detection network, inputting the clear and scratch-free old photo into the face detection network to perform face detection in the photo, and obtaining a cropped image of a local face area;

[0102] For this face detection problem, we mainly focus on three aspects: scale invariance, detection accuracy and the role of contextual reasoning. That is, we train different detectors according to faces in different scale ranges, and at the same time, through S4 super-resolution of images, make facial feature points more prominent, greatly improving the accuracy of face detection. In addition, by changing the range of facial context, we can improve the accuracy of facial recognition.

[0103] S8: Establish a face restoration network, process the local area cropped image of the face separately to perform face restoration, and obtain a restored face image;

[0104] The quality of face restoration is most sensitive to human perception, so a network for face enhancement (GPEN) is used. The GPEN restoration algorithm uses GAN as an image prior design to refine the face area of ​​old photos.

[0105] Face restoration network Figure 5 As shown in the figure, the overall architecture follows the U-net architecture, adopts a CNN encoder, learns to map the input blurred image x into the latent space Z, and then performs image super-resolution through the GAN prior network. A further implementation method is to first pre-train the GAN prior network according to the training strategy of StyleGAN, and embed the pre-trained GAN model into GPEN. In order to fine-tune the GPEN model, three loss functions are used: adversarial loss L A , content loss L C And feature matching loss L F :

[0106]

[0107] Among them, X and Represents a clear image and a degraded image.

[0108]

[0109] Where T is the total number of intermediate layers used for feature extraction, D i (X) is the feature extracted at the i-th layer.

[0110] Use the FFHQ dataset to train the GAN prior network, and use the following degradation model to synthesize LQ images from the clear images in FFHQ,

[0111]

[0112] Where I is the input image, K is the blur kernel, and n σ is Gaussian noise with standard deviation σ, I d is a degraded image. A further embodiment is that, for each image, the blur kernel K is randomly selected from a set of blur models, including Gaussian blur and motion blur with different kernel sizes. Additive Gaussian noise n σ is sampled channel by channel from a normal distribution.

[0113] By using these severely degraded images to fine-tune the model, the encoder part of GPEN can learn to generate suitable latent codes and noise input to the GAN prior decoder network.

[0114] The trained GAN prior network is embedded into the U-shaped DNN as a prior decoder, and then a set of synthetic low-level neural networks is used to fine-tune the GAN prior embedded DNN. At this time, the potential code z is replaced by the fully connected layer of the DNN encoder, and the noise input of the GAN network is replaced by the output of each layer of the encoder, which will control the reconstruction of the global face structure, local face details, and face image background. At the same time, in order to leave room for the feature map extracted by the encoder, additional noise input is provided to each GAN block.

[0115] S9: Performing histogram matching on the repaired face and the face detection position to obtain a clear photo without scratches and with a complete face.

[0116] S10: An image colorization network was built using a technique called DeOldify. Because the original model had too strong a colorization effect, the photos were processed and retrained through S4-S8 with parameter adjustments. The colorization effect was more realistic visually, and a color photo with no scratches and clear faces was finally obtained.

[0117] Further implementation is to address the problem of face colorization in old photos. By fine-tuning the training parameters and retraining, the image colorization effect will be more natural. Based on the S4 super-resolution step, the contour lines of the colored pictures will be clearer and there will be no color overflow. At the same time, based on the S8 face restoration network, the facial details are greatly improved before coloring, making the color filling more detailed and the texture clearer.

[0118] According to the above steps, the overall logic and process of this solution are as follows Figure 1 As shown, after inputting the old photo, execute steps S1-S3 to obtain the scratch-free old photo, take the scratch-free old photo as input, execute S4-S6 to obtain a clear scratch-free old photo, and then execute S7 and S8 to restore the face part of the old photo separately. After performing the face restoration, execute S9 to obtain a clear scratch-free photo with a complete face, and finally execute S10 to colorize the clear scratch-free photo with a complete face.

[0119] The present invention discloses a method for comprehensive restoration of old photos with scratches. By training a deep learning model network, scratches and faces in the photos are detected by different detectors and repaired respectively by different technical solutions. Through a small target face detection algorithm and a face generation and restoration network, even portrait photos with damaged faces can have a relatively good restoration effect.

[0120] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A comprehensive method for repairing scratched old photos. It is characterized in that include: S1. Establishing a diffusion model, and adding noise to the old photo with scratches to be repaired based on the noise adding method of the diffusion model to obtain a complete Gaussian noise map; establishing a scratch denoising space according to the old photo with scratches to be repaired; S2, establishing a noise prediction network, performing noise prediction on the complete Gaussian noise map and the scratch denoising space, obtaining corresponding pure noise maps respectively, and calculating the scores of each of the pure noise maps; S3, weighted calculation of the scores of each of the pure noise images to obtain a state graph; based on the denoising method of the diffusion model, reverse iterative denoising processing is performed on the state graph to obtain an old photo without scratches; S4, establishing an image super-resolution network, using a convolutional neural network, extracting a shallow feature map of the scratch-free old photo, and directly inputting the shallow feature map into a reconstruction module through a long skip connection to retain low-frequency information; S5, inputting the shallow feature map into a deep feature extraction module, performing deep feature modeling on the shallow feature map, and obtaining a deep feature map for recovering the lost high-frequency information; S6. Through long skip connections, the low-frequency information is directly input into the reconstruction module to help the deep feature extraction module focus on the high-frequency information. The low-frequency information and the high-frequency information are upsampled using a sub-pixel convolution layer. The scratch-free old photo is reconstructed by aggregating the upsampled low-frequency information and the high-frequency information, and a 4-fold super-resolution is performed to obtain a clear scratch-free old photo. S7, establishing a face detection network, inputting the clear and scratch-free old photo into the face detection network to perform face detection in the photo, and obtaining a cropped image of a local face area; S8, establishing a face restoration network, processing the local area cropped image of the face separately to perform face restoration, and obtaining a restored face image; S9, performing histogram matching on the repaired face and the face detection position to obtain a clear photo without scratches and with a complete face; S10. Establish an image colorization network to colorize the obtained clear, scratch-free photo with a complete face, so as to obtain a color photo with no scratches and a clear face.

2. The method for comprehensive repair of scratched old photos as claimed in claim 1, It is characterized in that The method to obtain a completely Gaussian noise map is: dx=θ t (η-x)dt+ξtdw Among them, x is the state diagram at time t, θ t (η-x) and ξ t are drift and dispersion equations respectively, η is the scratched old photo, dw is Gaussian noise, θ t With ξ t is a positive parameter related to time.

3. The method for comprehensive repair of scratched old photos as claimed in claim 1, It is characterized in that The method for establishing a scratch denoising space according to the old scratched photo to be repaired comprises: Based on an odd-numbered Gaussian convolution kernel, Gaussian blur processing is performed on the old photo with scratches to be repaired, so as to obtain a brightness information feature map that retains the brightness information of the picture and eliminates the scratches; Establishing and training a scratch detection network, performing scratch detection on the old photos with scratches to be repaired, and obtaining a normal distribution of scratches; The scratch denoising space is established based on the brightness information feature map and the scratch normal distribution.

4. The method for comprehensive repair of scratched old photos as claimed in claim 3, It is characterized in that The method to build and train the scratch detection network includes: Based on real scratched old photos and synthetic scratched simulated old photos, a scratched old photo dataset is established; The scratch area of ​​each old photo in the scratch old photo data set is marked, and the scratch old photos and the marked photos to be segmented are made to correspond one to one, and an image processing network is trained to obtain scratch detection results; Based on the scratch detection result, a cross entropy loss is used to minimize the difference between the detected scratch and the actual scratch; Based on the difference, the objective function of the scratch detection network is obtained, and the establishment and training of the scratch detection network are completed.

5. The method for comprehensive repair of scratched old photos as claimed in claim 3, It is characterized in that The method for obtaining the normal distribution of scratches includes: Based on the scratch detection network, scratch detection is performed on the old photo with scratches to be repaired to obtain a binary image mask with scratch position information; Performing a cropping operation on the binary image mask and the old photo with scratches to be repaired to obtain a feature image with only scratch information; The feature map containing only scratch information is encoded to obtain a normal distribution of scratches on the old photo with scratches to be repaired.

6. The method for comprehensive repair of scratched old photos as claimed in claim 1, It is characterized in that The method for obtaining the scratch-free old photo comprises: Performing weighted calculation on the scores of each of the pure noise images to obtain a final score calculation result; Based on the final score calculation result, obtaining a state diagram for the next stage; Based on the denoising method of the diffusion model, the state diagram is subjected to reverse iterative denoising processing to exclude scratch information and generate global semantic information to obtain the scratch-free old photo.

7. The method for comprehensive repair of scratched old photos as claimed in claim 1, It is characterized in that Using a convolutional neural network, the method for extracting the shallow feature map of the scratch-free old photo is as follows: F 0 =H SF (I LQ ) Among them, F 0 is the shallow feature map; I LQ For old photos without scratches; H SF (.) is a convolutional neural network.

8. The method for comprehensive repair of scratched old photos as claimed in claim 1, It is characterized in that The method of performing deep feature modeling on the shallow feature map to obtain the deep feature map is as follows: F DF =H DF (F 0 ) Among them, F DF is the deep feature map; H DF (.) is the deep feature extraction module; F 0 It is a shallow feature map.

9. The method for comprehensive repair of scratched old photos as claimed in claim 1, It is characterized in that The method for obtaining the clear and scratch-free old photo is: I RHQ =H REC (F 0 +F DF ) Among them, I RHQ For clear old photos without scratches; H REC (.) is the reconstruction module; F DF is the deep feature map; F 0 It is a shallow feature map.

Citation Information

Patent Citations

  • Photo restoration method and system

    CN115311156A

  • Face image restoration method based on online knowledge distillation and pre-training prior

    CN116071275A

  • Old photo repairing method and system based on deep neural network

    CN116402702A

  • Diffusion model-based old photo scratch repair method and system

    CN117745593A