Dual image degradation restoration method based on adaptive multi-feature fusion

Through the dual image degradation repair method of adaptive multi-feature fusion, the self-guided module and improved U-net structure are used to solve the problem of accurate coloring and repair of old photos, and efficient repair and automated processing of grayscale images are achieved, simplifying the The training process improves image quality.

CN117237202BActive Publication Date: 2024-11-08QINGHAI NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211679235.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-26
Publication Date
2024-11-08
Estimated Expiration
2042-12-26

AI Technical Summary

Technical Problem

The existing technology is difficult to automatically realize the accurate coloring and repair of old photos, especially the inability to effectively deal with multiple degradation problems in grayscale images, such as digital noise, film particles and stains. The existing network structure is complex, the training time is long and difficult. .

Method used

Using a dual image degradation repair method based on adaptive multi-feature fusion, the self-guided module removes miscellaneous noise, extracts multi-dimensional features, and uses the neuronal mechanism that simulates the human brain visual cortex to perform feature adjustment, combined with improved U-net encoding and decoding The device structure and adaptive multi-feature fusion module carry out image color recovery and resolution restoration to avoid chessboard artifacts, and at the same time introduce an intermediate information transmission mechanism to promote information transmission.

Benefits of technology

It realizes accurate shading and repair of old photos, can automatically deal with multiple degradation problems in grayscale images, improves image quality, verifies the effectiveness and versatility of the model, and reduces training difficulty and time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117237202B_ABST
    Figure CN117237202B_ABST
Patent Text Reader

Abstract

The invention discloses a dual image degradation restoration method based on adaptive multi-feature fusion, comprising: building a restoration network, selecting an image with salt and pepper noise or Gaussian noise as an input image of the restoration network; firstly removing the noise of the input image through a self-guiding module, and extracting the multi-dimensional features of the input image; the adaptive multi-feature fusion module adaptively adjusts the multi-dimensional features from the self-guiding module and transmits them to the back end of the restoration network by simulating the working mechanism of neurons in the visual cortex of the human brain; the intermediate image is combined with the features extracted by the adaptive multi-feature fusion module to restore the image color using a U-net-based encoder-decoder structure, wherein the bottom end of the encoder-decoder uses a residual module with a hole convolution to extract deep features, and the decoder part uses a bilinear upsampling operation to restore the image resolution to avoid the chessboard artifact phenomenon caused by the traditional transposed convolution; the invention can automatically realize the accurate coloring and restoration of the image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a dual image degradation restoration method based on adaptive multi-feature fusion. Background Art

[0002] Classic denoising methods are mainly based on averaging neighborhood pixels [[1],[2]] The core of these methods is to use the domain window mean to approximate the grayscale value, thereby suppressing the local similarity noise in the image, but will lead to ignoring the detailed features in the image. [[3]] By integrating multiple collaborative filtering results to estimate pixel values, local textures are further preserved, and a cleaner and clearer image is obtained. Recently, denoising methods based on deep learning have achieved state-of-the-art results. Schuler et al. [5] A multi-layer perceptron (MLP) was trained for image denoising, and it was shown that MLP can approximate all functions and achieve the optimal level of image denoising as long as the number of training sets is sufficient. Its disadvantage is that it cannot adapt to noise of different intensities. Mao proposed a deep residual encoding and decoding framework (RED) to solve the image restoration problem. The framework uses skip connections to train deep networks. Liu et al. improved U-net and proposed a multi-level wavelet CNN (MWCNN), which applied large receptive fields to image denoising.

[0003] Traditional image colorization methods are divided into graffiti-based methods and example-based methods. The former relies on the strokes drawn by the user as a guide for coloring. The color intervals generated by this method are constrained, that is, only the colors specified by the user are propagated. The latter extracts color statistics from the reference image to color the grayscale image, which requires a strong correspondence between the content of the reference image and the input image. Some methods propose a dual-branch structure that combines local image features with global semantic predictions to obtain better semantic representation, while other methods, such as predicting the color distribution strategy of each pixel, improve the diversity of interval colors.

[0004] Solving the problem of multiple image degradation has gradually become a research hotspot in the field of image processing. [6] et al. divided the restoration of old photos into structural defects (scratches, spots) and unstructured defects (fading, low resolution). They tried to map old photos, synthetic images and real images to their own latent spaces, and learned image restoration by aligning the latent spaces of old photos and synthetic images to generate potential translation. Runsheng Xu et al. [7]We further designed the Pix-Fix network, arguing that the multiple degradations of old photos should be divided into two steps: restoration and colorization, which are implemented using three subnets respectively: the colorization subnet and the similarity subnet use the spatial color histogram calculated from the reference image to estimate the chromaticity components of the old photos; the restoration subnet restores the brightness channel from the physical and captured distortion aspects.

[0005] Ziyu Wan [6] et al. were unable to complete complex colorization of old photos due to the limitations of their network structure. [7] The referenced sample images are subjective restorations by experts, which are somewhat out of touch with reality, and also make the colors of the generated images appear unrealistic. Secondly, the selection of sample images also consumes a certain amount of manpower. Finally, in general, the above two network structures have the problem of unification. In addition to their own backbone networks, they also need to add branch sub-networks, which undoubtedly greatly increases the time and difficulty of training.

[0006] In recent years, the development of excellent photographic equipment and the growth of photographic technology have enabled images to retain the most realistic visual effects. However, in early photography, whether it is natural scenery, historical relics, portraits, or cartoon images, they are usually "black and white" grayscale images. Due to the performance defects of the equipment at the time and improper preservation methods, image degradation problems such as digital noise (miscellaneous) points, film grains, stains, and damage often occur. Manual retouching and restoration methods are not only costly and time-consuming, but also difficult to achieve ideal restoration of grayscale images. Therefore, it is of great significance to design a network model for the above double degraded images to automatically achieve accurate image coloring and restoration.

[0007] [1]Smith S.SUSAN.A new approach to low-level imageprocessing.InternationalJournal of Computer Vision,1997

[0008] [2]Tomasi C,Manduchi R.Bi lateral filtering for gray and colorimages.SixthInternational Conference on Computer Vision(IEEECat.No.98CH36271),2002

[0009] [3]Dabov K,Foi A,Katkovnik V,et al.Image Denoising by Sparse 3-DTransform-Domain Collaborative Filtering.IEEE Transactions on ImageProcessing,2007,16(8):2080–2095

[0010] [4]Mao XJ,Shen C,Yang Y B.Image restoration using very deepconvolutionalencoder-decoder networks with symmetric skip connections.NeuralInformationProcessing Systems,2016,29:2810–2818

[0011] [5]Christian J Schuler, Harold Christopher Burger, Stefan Harmeling. Amachinelearning approach for non-blind image deconvolution. In CVPR, 2013

[0012] [6]Wan Z, Zhang B, Chen D, et al.Bringing Old Photos Back to Life.IEEE / CVFConference on Computer Vision and Pattern Recognition(CVPR), 2020

[0013] [7] Xu Runsheng, Tu Zhengzhong, Du Yuanqi, et al. Pik-Fix: Restoring and Colorizing Old Photo. ArXiv: Computer Vision and Pattern Recognition, 2022 Summary of the invention

[0014] In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a dual image degradation restoration method based on adaptive multi-feature fusion, which can automatically realize accurate coloring and restoration of images.

[0015] To achieve the above object, the technical solution adopted by the present invention is: a dual image degradation restoration method based on adaptive multi-feature fusion, comprising the following steps:

[0016] Step 1: Build a restoration network and select an image with salt and pepper noise or Gaussian noise as the input image of the restoration network;

[0017] Step 2: The input image is firstly passed through the self-guiding module to remove the noise and extract the multi-dimensional features of the input image;

[0018] Step 3: The adaptive multi-feature fusion module simulates the working mechanism of neurons in the visual cortex of the human brain, adaptively adjusts the multi-dimensional features from the self-guidance module and transmits them to the back end of the repair network;

[0019] Step 4: The features extracted by the intermediate image and the adaptive multi-feature fusion module are used to restore the image color using a U-net-based encoder-decoder structure. The bottom of the encoder-decoder uses a residual module with hole convolution to extract deep features, and the decoder part uses a bilinear upsampling operation to restore the image resolution to avoid the checkerboard artifact phenomenon caused by traditional transposed convolution.

[0020] As a further improvement of the present invention, in the self-booting module of step 2:

[0021] Given an input image I of size H×W×C 0 First, a series of input images I are generated through the Shuffle / 2 operation K , Then at the third level top branch A=f K=3 (θ)Extract I K=3 The features of the second layer are introduced into the branch B = f at the beginning of the second layer. K=2 (A) guides its feature extraction; finally, through the first layer branch C = f K=1 (B) Local and global features at multiple scales are moved to the full-resolution backbone to generate the final estimate.

[0022] As a further improvement of the present invention, in the self-guiding module, all branches further include a dense link module, and the dense link module is used to promote the fusion of local and global features.

[0023] As a further improvement of the present invention, the U-net-based encoder / decoder uses downscaled convolution instead of pooling as a downsampling method based on the U-net structure. After the downscaled convolution, a hole convolution residual block is introduced. Finally, the compact deep features are restored to a high-resolution image through bilinear upsampling.

[0024] As a further improvement of the present invention, the adaptive multi-feature fusion module includes two steps: fusion and selection. The input is feature maps of three different dimensions, and the output is a feature map of a certain size after adaptive adjustment. The fusion operation generates a feature descriptor by combining multi-resolution information streams. The selection operation uses the generated feature descriptors to recalibrate the original feature maps and perform aggregation.

[0025] As a further improvement of the present invention, it also includes: introducing an intermediate information transmission mechanism between the self-booting module and the encoder-decoder, promoting the transmission of intermediate information through three parallel adaptive multi-feature fusion modules, and refining it using 1×1 convolution before aggregating it to the backbone network.

[0026] The beneficial effects of the present invention are:

[0027] This paper verifies the effectiveness of the model based on landscape and face datasets, and demonstrates the dual restoration effect of real old photos. The versatility of the proposed method is verified by testing the national intangible cultural heritage thangka dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the overall structure of the repair network in an embodiment of the present invention;

[0029] Figure 2 Schematic diagram of the structure of a self-guiding module (SGM) in an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of the structure of a codec in an embodiment of the present invention;

[0031] Figure 4 Schematic diagram of the structure of an adaptive multi-feature fusion module (AMF) in an embodiment of the present invention;

[0032] Figure 5 Schematic diagram of experimental results of the CelebA dataset in an embodiment of the present invention;

[0033] Figure 6 Schematic diagram of experimental results of Landscape dataset in an embodiment of the present invention;

[0034] Figure 7 Schematic diagram of experimental results of a real old photo dataset in an embodiment of the present invention;

[0035] Figure 8 Schematic diagram of the experimental results of the national intangible cultural heritage thangka dataset in an embodiment of the present invention;

[0036] Fig. 9This is a graph showing the experimental comparison results of AMFNet and three classic models, Pix2Pix, Deoldify, and CIC, in an embodiment of the present invention;

[0037] Fig.10 This is a graph showing the experimental results of using the model trained on CelebA to conduct experiments on real old photos in an embodiment of the present invention, and the experimental comparison results of AMFNet and three classic models: Pix2Pix, Deoldify, and CIC. DETAILED DESCRIPTION

[0038] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0039] Example

[0040] like Figure 1 As shown, a dual image degradation restoration method based on adaptive multi-feature fusion includes the following steps:

[0041] Step 1. Select an image with salt and pepper noise or Gaussian noise;

[0042] Step 2: The input image is firstly passed through the self-guiding module to remove the noise and extract the multi-dimensional features of the image;

[0043] Step 3: The adaptive multi-feature fusion module (AMF) simulates the working mechanism of neurons in the visual cortex of the human brain to adaptively adjust the multi-dimensional features from the SGM module and transmit them to the network backend;

[0044] Step 4: The features extracted by the intermediate image and the adaptive multi-feature fusion module are used to restore the image color using a U-net-based encoder-decoder structure. The bottom of the encoder-decoder uses a residual module with hole convolution to extract deep features, and the decoder part uses a bilinear upsampling operation to restore the image resolution to avoid the checkerboard artifact phenomenon caused by traditional transposed convolution.

[0045] like Figure 2 As shown, the self-guided module (SGM): The key point of the SGM module is to use the coarse semantic content to guide the recovery at a fine scale. Given an input image I of size H×W×C 0 First, a series of input images I are generated through the Shuffle / 2 operation K , Then at the third level top branch A=f K=3 (θ)Extract I K =3In addition to using convolutional blocks and residual blocks to extract shallow features, all branches also include dense link modules. Their unique internal and external connection design promotes the fusion of local and global features, making hierarchical information more compact and strengthening the transmission of deep features. Secondly, at the beginning of the second layer, the feature representation A with large-scale information is introduced into branch B = f K=2 (A) guides its feature extraction. Finally, through the first layer branch C = f K=1 (B) Move the multi-scale local and global features to the full-resolution backbone to generate the final estimate. The SGM module can use hierarchical semantic information to guide image restoration, making the generated image more delicate and smooth.

[0046] like Figure 3 As shown, the encoder-decoder module: The encoder-decoder module of this embodiment is an improved U-net structure. The original U-net model uses pooling as a downsampling method. However, the essence of pooling is to reduce the image size by discarding a large number of pixel values, which may cause the loss of local information in the image. Based on the U-net structure, this embodiment uses downscaled convolution instead of pooling as a downsampling method, thereby reducing the loss of image features. After the downscaled convolution, the dilated conv+residual blocks are also introduced. Compared with the ordinary convolution layer, under the same computing conditions, a larger receptive field can be obtained, thereby consolidating the tight transmission of deep features. Finally, the tight deep features are restored to a high-resolution image through bilinear upsampling, because bilinear upsampling can avoid the checkerboard artifact phenomenon that occurs in traditional transposed convolution.

[0047] like Figure 4 As shown in the figure, Adaptive Multi-Feature Fusion Module (AMF): AMF is a fusion multi-scale feature module. It uses the multi-feature fusion strategy of the self-attention mechanism to adaptively select effective feature maps from the feature representation of each branch. It not only combines the information of different receptive fields, but also each dimensional feature is complementary to other features. The AMF module is divided into two steps: fusion and selection. The input is feature maps of three different dimensions, and the output is a feature map of a certain size after adaptive adjustment. The fusion operation generates feature descriptors by combining multi-resolution information streams; the selection operation uses the generated feature descriptors to recalibrate the original feature maps and aggregate them. The AMF module adaptively adjusts the appropriate receptive field to the network backend by simulating the response stimulation of human brain neurons, so it is more in line with human visual perception.

[0048] Intermediate Information Transfer Mechanism (ITM): This embodiment introduces an intermediate information transfer (ITM) mechanism between the SGM module and the encoder-decoder, promotes the transmission of intermediate information through three parallel AMF modules, and refines it using 1×1 convolution before aggregating it to the backbone network. The ITM mechanism proposed in this embodiment has the following advantages: First, the multi-scale features from the SGM module help enrich the feature representation of the intermediate layer of the U-net network and assist the coloring process of the U-net network. Secondly, the mechanism connects multi-dimensional low-resolution semantic features to avoid information loss caused by the network progression process. Finally, the feature information flow is more stable due to the simplification of the feature transmission process at each level.

[0049] like Figure 5-Figure 8 As shown, this embodiment verifies the effectiveness of the model based on landscape and face datasets, and demonstrates the dual restoration effect of real old photos. The versatility of the proposed method is verified by testing the national intangible cultural heritage thangka dataset.

[0050] like Fig. 9 As shown in the figure, the experimental comparison results of AMFNet and three classic models, Pix2Pix, Deoldify, and CIC, are shown. Since these colorization methods do not have the function of correcting fading and repairing at the same time, in order to ensure the fairness of the experiment, the above three control group methods only tested the pure colorization of grayscale images without noise, while the algorithm of this embodiment tested the denoising and colorization of input images in three cases: grayscale + Gaussian noise, grayscale + salt and pepper noise, and original image + Gaussian noise. In addition, in order to verify the pure denoising ability of this algorithm, only Gaussian noise is added to the color images in the data set for training.

[0051] like Fig.10 As shown, in order to prove the generalization of the algorithm to real old photos, this embodiment uses the model trained on CelebA to experiment on real old photos, and still conducts comparative experiments with AMFNet and three classic models: Pix2Pix, Deoldify, and CIC.

[0052] The experiments use evaluation criteria such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM) and the latest learned Perceptual Image Patch Similarity (LPIPS) to assess image quality.

[0053] Table 1 Comparison of evaluation indicators of various algorithms

[0054]

[0055] The above-mentioned embodiments only express the specific implementation of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A dual image degradation restoration method based on adaptive multi-feature fusion, characterized in that: The following steps are involved: Step 1. Select an image with salt and pepper noise or Gaussian noise as the input image; Step 2: The input image is firstly passed through the self-guiding module to remove the noise and extract the multi-dimensional features of the input image; In the bootstrap module in step 2: Given an input image I of size H×W×C 0 First, a series of input images I are generated through the Shuffle / 2 operation K , Then at the third level top branch A=f K=3 (θ)Extract I K=3 The features of the second layer are introduced into the branch B = f at the beginning of the second layer. K=2 (A) guides its feature extraction; finally, through the first layer branch C = f K=1 (B) Moving local and global features to the full-resolution backbone to generate the final estimate; Step 3: The adaptive multi-feature fusion module simulates the working mechanism of neurons in the visual cortex of the human brain to adaptively adjust the multi-dimensional features from the self-guiding module and transmit them to the back end; The adaptive multi-feature fusion module includes two steps: fusion and selection. The input is feature maps of three different dimensions, and the output is a feature map of a certain size after adaptive adjustment. The fusion operation generates feature descriptors by combining multi-resolution information streams. The selection operation uses the generated feature descriptors to recalibrate the original feature maps and perform aggregation. Step 4: The features extracted by the intermediate image and the adaptive multi-feature fusion module are used to restore the image color using a U-net-based encoder-decoder structure. The bottom of the encoder-decoder uses a residual module with hole convolution to extract deep features, and the decoder part uses a bilinear upsampling operation to restore the image resolution to avoid the checkerboard artifact phenomenon caused by traditional transposed convolution.

2. The dual image degradation restoration method based on adaptive multi-feature fusion according to claim 1 is characterized in that: In the self-guiding module, all branches further include a dense link module, and the dense link module is used to promote the fusion of local and global features.

3. The dual image degradation restoration method based on adaptive multi-feature fusion according to claim 1, characterized in that: The U-net based encoder / decoder uses downscaling convolution instead of pooling as a downsampling method based on the U-net structure. After the downscaling convolution, a hole convolution residual block is introduced. Finally, the deep features are restored to a high-resolution image through bilinear upsampling.

4. The dual image degradation restoration method based on adaptive multi-feature fusion according to claim 1, characterized in that: Also includes: An intermediate information transmission mechanism is introduced between the self-bootstrapping module and the encoder-decoder, and the transmission of intermediate information is promoted through three parallel adaptive multi-feature fusion modules. Before being aggregated into the backbone network, it is refined using 1×1 convolution.