Method for blind image restoration, computer device, readable storage medium and program product

By comparing the blind image repair network across space with the loss module and the self-attention module, the accuracy problem of image repair under complex noise is solved, efficient noise cancellation and automatic repair are achieved, and the repair effect is improved.

CN114782264BActive Publication Date: 2024-11-12宁波市天一阁博物院 +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210395242.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-14
Publication Date
2024-11-12
Estimated Expiration
2042-04-14

AI Technical Summary

Technical Problem

The existing blind image repair technology is difficult to accurately repair images when facing complex noise, resulting in a large amount of noise and artifacts in the repaired image, especially when natural images are noise, the repair effect is poor.

Method used

The blind image repair network is adopted with the cross-space contrast loss module and self-attention module. By contrasting the loss function and self-attention mechanism across space, the degraded images are eliminated and the degraded images are automatically repaired, restricting feature extraction in the features of the real image space, and improving repair Accuracy.

Benefits of technology

Effectively remove noise from degraded images, improve the accuracy of blind image repair networks, generate visually reasonable and consistent repair results, reduce artifacts, and improve repair effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114782264B_ABST
    Figure CN114782264B_ABST
Patent Text Reader

Abstract

The present application discloses a method for blind image restoration, a computer device, a readable storage medium and a program product, the method comprising: inputting an image to be restored into a trained blind image restoration network to obtain a restoration result, characterized in that the blind image restoration network comprises a first encoder and a decoder: obtaining a real image and a degraded image corresponding to the real image; inputting the degraded image and reference data into the first encoder for processing, the reference data being obtained after the real image is processed by a cross-space contrast loss module; inputting the calculation result of the first encoder into the decoder for processing, and outputting a restored image; comparing the restored image with the real image and updating the blind image restoration network for cyclic training until the restored image is close to the real image, and obtaining a trained blind image restoration network. The blind image restoration method of the present application can not only eliminate noise, but also automatically restore degraded images in high-level semantics.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical fields of deep learning, image synthesis and style transfer, and in particular to a method for blind image restoration, a computer device, a readable storage medium and a program product. Background Art

[0002] The goal of traditional image restoration is to generate high-quality and visually plausible structures and textures for the missing areas at specified locations in the image. Through image restoration, users can remove or edit unwanted objects in photos. Currently, most image restoration works are based on the condition that the area to be restored (i.e., the mask) is known. However, in practical applications, it is sometimes difficult to determine the correct mask, and blind image restoration needs to be applied.

[0003] Blind image restoration is the restoration of a degraded image (or polluted image, image to be restored) without knowing the target restoration area. Blind image restoration was first defined and applied to remove text from images. Previous work on blind image restoration focused on repairing fixed-pattern noise, such as repairing specific colors or Gaussian noise.

[0004] However, if fixed pattern noise is used in training, the training model is more likely to distinguish the mask area based on the predefined noise pattern rather than the context. To solve this problem, some scholars superimpose natural images on the target image as the noise of the blind image. The existing goal of the blind image restoration task is to transform the blind image restoration task into a general restoration task. Its model usually consists of a mask prediction module and an image restoration network. Because there is complex noise on the degraded image to be restored by blind image restoration, especially when the natural image is used as the noise, it is almost impossible to accurately predict the noise area of ​​the degraded image.

[0005] Therefore, subsequent methods based on predicting noise areas (masks) will inevitably face the following problems: there is still a large amount of noise in the repaired image, and sometimes even the noise map will be used as the basis for repair, resulting in the repaired image being dominated by the noise content; in addition, due to the interference of noise, the repaired image has many artifacts in the consistency of content structure, especially when the noise area is large or the natural image content is used as noise, the situation is more serious and the repair effect is poor. Summary of the invention

[0006] Based on this, it is necessary to provide a method for blind image restoration to address the above technical issues.

[0007] The method for blind image restoration of the present application includes inputting an image to be restored into a trained blind image restoration network to obtain a restoration result, wherein the blind image restoration network includes a first encoder and a decoder, and the blind image restoration network is trained by the following steps:

[0008] Obtaining a real image and a degraded image corresponding to the real image;

[0009] Inputting the degraded image and reference data into the first encoder for processing, wherein the reference data is obtained by processing the real image through a cross-space contrast loss module;

[0010] Inputting the calculation result of the first encoder into the decoder for processing, and outputting the repaired image;

[0011] The repaired image is compared with the real image and the blind image repair network is updated for cyclic training until the repaired image is close to the real image, thereby obtaining the trained blind image repair network.

[0012] Optionally, inputting both the degraded image and the reference data into the first encoder for processing specifically includes:

[0013] The degraded image is input to the first encoder to obtain latent space features of the degraded image, and the reference data is connected to the latent space features of the degraded image.

[0014] Optionally, the real image is processed by a cross-space contrast loss module, specifically including:

[0015] The real image is input to the second encoder, and the second encoder is connected to the first encoder through a cross-spatial contrast loss module. The cross-spatial contrast loss function used by the cross-spatial contrast loss module is:

[0016] L IC =BCE(sigmoid(C(F 1 , F 2 )), I), wherein:

[0017] BCE stands for binary cross entropy loss;

[0018] Sigmoid, represents the regression function;

[0019] C, represents the cross product of two features to obtain the covariance matrix;

[0020] F 1 , represents the features corresponding to the degraded image;

[0021] F 2 , represents the features corresponding to the repaired image;

[0022] I, represents the identity matrix.

[0023] Optionally, the latent space features of the degraded image and the latent space features of the real image both include Value values, Query values ​​and Key elements;

[0024] The real image is input to a second encoder, and the second encoder is connected to the first encoder through a cross-space contrast loss module, specifically comprising:

[0025] The real image is input to the second encoder, and the second encoder outputs the latent space features of the real image. The Query value of the latent space features of the real image is connected to the Key element of the latent space features of the degraded image through the first residual block, the cross-space contrast loss module, and the second residual block in sequence.

[0026] Optionally, inputting the calculation result of the first encoder to the decoder for processing specifically includes:

[0027] The latent space features of the degraded image are input into the decoder for processing through a self-attention module.

[0028] Optionally, the degraded image has an original mask, and the first encoder outputs a generated mask of the degraded image according to the degraded image.

[0029] The blind image restoration method further includes: comparing the original mask with the generated mask, and updating the blind image restoration network for cyclic training until the restored image is close to the real image, thereby obtaining the trained blind image restoration network.

[0030] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the blind image restoration method described in the present application.

[0031] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the blind image restoration method described in the present application are implemented.

[0032] The present application also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the blind image restoration method described in the present application.

[0033] The method of blind image restoration in this application has at least one of the following effects:

[0034] This application uses a cross-space contrast loss module in the training network, which can remove noise from degraded images and realize automatic repair of degraded images in the encoding stage. Cross-space comparison can not only eliminate noise, but also automatically repair degraded images in high-level semantics, and the repaired features will be restored to pixel space by the decoder of the framework. The loss of cross-space contrast loss limits the extracted features of the degraded image to the features that are exactly the same as the features of the real image from the real image space, constrains the maximum correlation of feature values ​​at the same position, and helps to improve the accuracy of the blind image repair network. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a blind image restoration method in one embodiment of the present application;

[0036] Figure 2 This is a schematic diagram of the structure of a blind image network in one embodiment of the present application;

[0037] Figure 3 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0038] In the existing technology, blind image restoration will be disturbed or dominated by noise content, resulting in many artifacts in the content structure consistency of the restored image, and the restoration effect is poor. The essence of blind image restoration is not to find the noise area of ​​the degraded image, but to remove or intelligently filter out the noise content of the degraded image and automatically complete the visual content based on the content of the non-noise area.

[0039] See also Figure 1-2 In one embodiment of the present application, a method for blind image restoration is provided. The method for blind image restoration includes inputting an image to be restored into a trained blind image restoration network to obtain a restoration result. The blind image restoration network includes a first encoder (see Figure 2 E1) and decoder (see Figure 2 The marker D in .

[0040] The blind image restoration network is trained using the following steps:

[0041] Step S100: obtaining a real image and a degraded image corresponding to the real image.

[0042] In this step, the real image is the uncontaminated image ( Figure 2 I L gt ), the corresponding degraded image is the image after the real image is contaminated ( Figure 2 I L deg ), the repaired image is the image repaired by the blind image repair network.

[0043] Step S200: input the degraded image and reference data into a first encoder for processing, wherein the reference data is obtained by processing the real image through a cross-space contrast loss module.

[0044] In step S200, the degraded image I L deg The reference data is input to the first encoder E1 to obtain the latent space features of the degraded image, and the reference data is connected to the latent space features of the degraded image. It can be understood that the reference data is generated by real images, so the reference data can optimize the blind image restoration network and make the restoration results of the blind image restoration network more reliable.

[0045] In step S200, the real image is processed by a cross-spatial contrast loss module, specifically including: inputting the real image into the second encoder, the second encoder is connected to the first encoder through the cross-spatial contrast loss module, and the cross-spatial contrast loss function adopted by the cross-spatial contrast loss module is described in detail below.

[0046] The cross-space contrast loss function calculates the two images to be repaired F 1 、F 2 The cosine similarity of features and constraining the maximum correlation of feature values ​​at the same position helps to improve the accuracy of the blind image restoration network.

[0047] In this step, the latent space features of the degraded image and the latent space features of the real image both include Value, Query and Key elements, which are abbreviated as Figure 2 The markers V, Q, and K in .

[0048] Specifically, the real image is input to the second encoder (see Figure 2 The second encoder E2 outputs the latent space features of the real image, and the Query query value of the latent space features of the real image passes through the first residual block ( Figure 2 ResBlocks on the lower left side of the middle), cross-space contrast loss module, and the second residual block ( Figure 2 The middle part ResBlocks) is connected to the Key element of the latent space feature of the degraded image.

[0049] Step S300, input the calculation result of the first encoder to the decoder for processing, and output the repaired image. Specifically, the latent space features of the degraded image are input to the decoder for processing through the self-attention module. The self-attention module is the self-attention module in the figure, which will not be described here.

[0050] Step S400, comparing the repaired image with the real image and updating the blind image repair network for cyclic training until the repaired image is close to the real image, thereby obtaining a trained blind image repair network.

[0051] “The restored image is similar to the real image” can be understood as the discriminator D of the output image of each embodiment of the present application. I After identification, the repaired image is determined to be the same as the real image; or after the mask-aware discriminator D M After identification, the restored image is determined to be identical to the real image; or both are considered identical. Or the restored image is determined to be identical to the real image by human eyes.

[0052] Step S500: The degraded image has an original mask, and the first encoder outputs a generated mask of the degraded image according to the degraded image. The original mask is the contaminated part of the real image compared to the degraded image; the generated mask is the contaminated part generated by the blind image. The generation process of the original mask is described below. Figure 2 The mask-aware discriminator D in M Discriminatively comparing the similarity between the original mask and the generated mask can improve the restoration accuracy of the blind image network.

[0053] The blind image restoration method also includes: comparing the original mask with the generated mask, and updating the blind image restoration network for cyclic training until the restored image is close to the real image, thereby obtaining a trained blind image restoration network.

[0054] Step S600: Input the image to be repaired into the trained blind image repair network, and use an upsampling network (see Figure 2 The marker U in ( ) will increase the resolution of the image to be repaired, and the upsampling network includes the image to be repaired and the semantic features of the repair result.

[0055] Specifically, the upsampling network combines the uncontaminated area of ​​the image to be repaired and the semantic features of the repair result to improve the resolution of the image to be repaired.

[0056] The present application provides a method for blind image restoration, which uses a cross-space comparison learning mechanism to remove noise from degraded images and realizes automatic restoration of degraded images in the encoding stage. In addition, the present application also designs a mask-aware discriminator D M To further improve the restoration performance, this is the first time that comparison learning has been introduced into the blind image restoration task.

[0057] The blind image restoration network of this application adopts a two-stage blind image restoration framework from coarse to fine, which consists of a blind image restoration network and an upsampling network. In order to better restore the macro content of the degraded image, the image restoration network will be used to restore the image at a low resolution, and the upsampling network will be used to generate the detailed content of the restored image.

[0058] The blind image restoration method of this application trains the encoder in the proposed framework through the method of comparison learning, so that the encoder can intelligently and consciously filter the noise content when extracting degraded image features. Unlike self-supervised comparison learning, cross-space comparison learning can not only eliminate noise, but also automatically repair degraded images in high-level semantics. The repaired features will be restored to the pixel space by the decoder of the framework.

[0059] The structure of blind image restoration is as follows Figure 2 As shown, in the coarse stage, the network of the present application has a first encoder E1, a second encoder E2, a decoder D, a cross-space comparison learning module CSC (with a cross-space contrast loss function), a mask-aware discriminator D M , and a discriminator D acting on the output image I .

[0060] This application adds a self-attention module for long-term dependency learning between the encoder and decoder. In order to avoid over-constraint and damage to the ability of self-attention to operate long-term dependencies, this application adds some residual blocks to ResBlocks after the general query value and key element convolution layer to generate feature maps for alignment, rather than operating directly on the attention map. The input degraded image is mapped to the latent space through the encoder.

[0061] Figure 2 The actual meaning of each marker is as follows:

[0062] I L deg , is the degraded image during the training process, and is the input image to be repaired after the training is completed;

[0063] I L gt , the real image corresponding to the degraded image;

[0064] I L inp ,During the training process, it is the inpainted image;

[0065] E1, first encoder;

[0066] E2, second encoder;

[0067] V, Value;

[0068] Q, Query value;

[0069] K, Key element, VQK are latent space features

[0070] self-attention module, self-attention module;

[0071] D, decoder;

[0072] Res blocks, residual blocks;

[0073] D M ,Mask-aware Discriminator,The mask-aware discriminator converts the data in the latent space into a mask map of the input size for accurate training.

[0074] I L inp , low-resolution restoration results;

[0075] I H inp , high-resolution restoration results;

[0076] CSC, (cross-space contrastive), cross-space contrastive learning module;

[0077] D I , discriminator.

[0078] It can be understood that during training, the real image I L gt , Discriminator D I , Mask-aware Discriminator D M , and the cross-space comparison learning module CSC all participate in the training of the blind image restoration network; after the training is completed, only Figure 2 The blind image restoration network in can implement blind image restoration.

[0079] In this embodiment, the discriminator D I The main method is to use the existing data set to constrain the generated results to be consistent with the data distribution of the actual data set, that is, the discriminator can make the generated repair result image more realistic. First, the image from the real data is input into the discriminator, and the result image repaired by the repair network is also input into the discriminator. When the repair result output by the repair network makes the discriminator mistakenly believe that it comes from the real data set, the blind image repair network is considered to be trained.

[0080] This application defines the set of all real images as the real image space, and the set of all degraded images as the degraded image space. In the blind image restoration scenario, any image from the degraded image space can have some essential correlation with the corresponding real image in the real image space, which can eliminate the influence of noise. This is achieved by using a cross-space contrast loss, which limits the extracted features of the degraded image to those that are exactly the same as the features of the real image from the real image space.

[0081] In one embodiment, the present application further provides a method for blind image restoration, comprising:

[0082] Step S100, generating a degraded image for blind image restoration learning, by fusing two images together to generate a degraded image, which is the blind image to be restored in this application, or the degraded image considered to be used in the training process;

[0083] Specifically, we first randomly generate a mask, then multiply an image by the mask and add the complement of another image by the mask, so that we can generate a degraded image. The degraded data image is the image we want to repair, which is obtained by superimposing a normal image and a noise image. The superimposed area is determined by the mask. However, since the superimposed area is not known in advance, it is called blind repair.

[0084] Specifically, in step 100, the input of the blind restoration framework is an image contaminated by noise (degraded image), which is obtained by merging the noise with the image through a random mask. The expression is:

[0085] I deg =I noise ⊙M+I gt ⊙(1-M), where:

[0086] M, represents a random mask;

[0087] I noise , represents another natural image as noise;

[0088] ⊙, represents the Hadamard product operation.

[0089] Although real image noise can prevent the network from distinguishing noise by pattern, sharp edges can still help distinguish different areas. Therefore, when generating degraded images, this application performs Gaussian blur on the mask. The purpose of this modification method is to make it difficult for the network to distinguish obvious noise boundaries. Since there is no definite solution for blind restoration tasks in practical applications, the output expectation only includes visual integrity and the areas not covered by noise must be completely preserved.

[0090] In other words, first there is a data set of normal images (or real images), such as face images. The data in this data set are all normal. Any image in this data set is called I L gt Then each image in the data set is degraded, that is, the real image I is transformed into L gtThe image becomes a degraded image. In this way, a pair of images is generated, a degraded image to be repaired and its corresponding repaired result image. This pair of images is used to train the network. After the training is completed, a degraded image to be repaired I is input L deg , the output result is as close as possible to the image I corresponding to this degraded image L gt Consistent.

[0091] Step S200 to Step S400. Step S200, input the degraded image and reference data to the first encoder for processing, where the reference data is obtained by processing the real image through the cross-space contrast loss module. Step S300, input the calculation result of the first encoder to the decoder for processing, and output the repaired image. Step S400, compare the repaired image with the real image and update the blind image repair network for cyclic training until the repaired image is close to the real image, and obtain a trained blind image repair network.

[0092] In step S200 to step S400, what is actually described is: inputting the degraded image to be repaired into the blind image rough repair module, using comparison learning to repair the degraded image in the encoding stage, and the result of the repair is the latent space feature; in order to repair the degraded image, the degradation noise is eliminated by comparison learning, and then the repaired latent space feature is obtained, and the repaired latent space feature is the calculation result of the first encoder. In step S400, the blind image repair network is updated, and the content actually updated is the network parameters.

[0093] As mentioned above, this application hopes that images with different noises from the image-to-be-modified space and real images from the real image space can be mapped to the same position in the latent space. In other words, the extracted latent code should be robust to different noises.

[0094] It is assumed that the latent space features of each latent space position are quite independent in terms of semantics. For any image to be repaired that is input to the encoder, the same semantic information should not be extracted from different positions. For example, for an image of a nose with right eye noise, the repaired latent code should not contain two vectors representing the semantics of the right eye.

[0095] Therefore, cross-space comparison learning can be used for clustering. This application calculates the cosine similarity of the features of the two images to be repaired and constrains the maximum correlation of the feature values ​​at the same position. At the same time, the correlation at different positions is reduced. The initial optimization goal is to maximize the F 1 And the repaired image F 2 The semantic relevance of R(F i 1 ,F i2 ), while minimizing R(F i 1 ,F j 2 ), i! = j. In the formula, R represents the correlation between two features. i 1 ,F i 2 ) are the diagonal elements of the matrix obtained by multiplying two eigenvectors, and R(F i 1 ,F j 2 ) are off-diagonal elements.

[0096] Considering that the physical meaning of the cross-correlation matrix is ​​similar to that of the attention map, the matrix C∈R B×N1×N2 Element C ij =R(F 1 i ,F 2 j ) represents the association between the i-th element of Query and the j-th element of Key after linear mapping. In the commonly used self-attention module, Q and K come from the same input feature through different convolutional layers, N1 is the dimension of feature 1, N2 is the dimension of feature 2, and B is the result dimension of the inner product of the elements of the two dimensions. We reused some self-attention modules to calculate the similarity of the hidden codes of different images to be modified, such as Figure 1 As shown. Our cross-space alignment loss can be expressed as:

[0097] L IC =BCE(sigmoid(C(F 1 ,F 2 )),I)#Formula (1)

[0098] I represents the identity matrix, and BCE represents the binary cross entropy loss.

[0099] Sigmoid regression function;

[0100] C, represents the cross product of two features to obtain the covariance matrix;

[0101] F 1 , represents the features corresponding to the degraded image after a normal image is processed by mask;

[0102] F 2 , represents the features corresponding to the restored result image corresponding to the degraded image.

[0103] When the high-level semantics of the image to be corrected is similar to the real image, its latent code is also similar. It can be understood that cross-space comparison is actually coupling the latent space features of the degraded image and the latent space features of the real image. For ease of understanding, the reference data is expressed as the correction data formed by the real image, the second decoder, and the cross-space comparison loss module in sequence. That is, the "reference data is connected to the latent space features of the degraded image" mentioned above.

[0104] Step S500, using the mask-aware discriminator D M Perceptual Adversarial Loss (MADLoss)

[0105] Since the parameters of the attention module are updated during training, we cannot guarantee that when the distance between all possible latent space inputs is reduced, the final latent code can be close to the value representing the complete image. We have reason to speculate that the narrowed code may represent the average semantics of the training set, where masks appear frequently or with different confidence levels of correctness. However, the latent code is actually often expected to include all globally consistent information, so the approximate latent code needs to be pushed back to represent the real image. It can be understood that the push-back process is a natural mapping during the training process.

[0106] Recall the difference between the inpainted image and the complete image (normal image): the former can be visually divided into different regions, while the latter cannot. The backpropagation process is to eliminate the effect of noise on the input image. Therefore, while training the encoder to extract the hidden code from the inpainted image where the mask area cannot be predicted, we train a mask discriminator to predict the mask from the hidden code.

[0107] In order to locate the mask region more accurately, the latent code is converted to the original input size. Our mask-aware adversarial loss can be expressed as:

[0108]

[0109] in:

[0110] E, is the expected value;

[0111] is the image to be restored;

[0112] is the probability distribution function of the image to be repaired;

[0113] x, is the image without degradation;

[0114] P data(x) , is the probability distribution function of the image without degradation (perfect image);

[0115] M, is the mask image;

[0116] E(x) , is the encoder;

[0117] D M , is the predicted mask image.

[0118] Furthermore, we also include other perceptual restoration losses: we consider other traditional restoration losses including reconstruction loss, loss based on VGG (a pre-trained deep neural network model) and adversarial loss. These losses ensure that the restoration results converted by the decoder at low resolution are inp L is visually plausible. These losses can be expressed as:

[0119]

[0120]

[0121] Θ i It is the semantic feature of a certain layer extracted by the VGG network.

[0122]

[0123] D I is the discriminator network, which inputs a result, 1 for true and 0 for false.

[0124] is the image to be restored;

[0125] is the probability distribution function of the image to be repaired;

[0126] x is the image without degradation;

[0127] P data(x) is the probability distribution function of the image without degradation (perfect image);

[0128] The total coarse-stage repair loss is:

[0129]

[0130] λ adv ,λ IC ,λ mask is a hyperparameter weight value, which is used to control the weights of different loss functions. Each hyperparameter weight value can be, for example, λ adv =1.0,λ IC =2.0,λ mask =1.5

[0131] Step S600, input the restored low-resolution image into the super-resolution fine restoration module of fine restoration to obtain the final high-resolution restoration result; in order to generate a good super-sampling result, it is necessary to design an upsampling loss function.

[0132] The input of the model is a bilinear upsampled low-resolution restoration result concatenated with the original image to be repaired. Because the input of this module includes part of the original resolution image information, we prefer to call it upsampling rather than super-resolution.

[0133] For this module, the low-resolution restoration results provide visual plausibility and overall consistency; the original image provides high-resolution non-destructive area information, including color, texture, etc. We expect the upsampling results to be overall reasonable and artifact-free. We use L1 reconstruction loss and VGG basic feature loss to constrain.

[0134]

[0135] is the L1 reconstruction loss in pixel space, and L_VGG^U is the L1 reconstruction loss in feature space. is a hyperparameter that controls the weights of different loss functions. Specifically, the value of the hyperparameter can be, for example:

[0136] The total loss of our network is:

[0137] L all =L inp +L Up #Formula(8)

[0138] In step S600, the restored low-resolution image is input into a super-resolution fine restoration module of fine restoration to obtain a final high-resolution restoration result;

[0139] The feature of the latent space is a high-semantic space feature, which must be translated into the pixel space in order to finally complete the blind image restoration. Therefore, it is necessary to build a decoder to translate the features of the latent space into an image in the pixel space.

[0140] In order to better learn how to repair blind images, we first perform blind image repair in a low-resolution space, and then use the super-resolution network to fine-tune the repair results to achieve the final high-resolution repair. After the training is completed, the repaired features in the latent space are input into the decoder to obtain the final low-resolution repair result.

[0141] The embodiments of the present application provide a method for blind image restoration, a coarse-to-fine two-stage blind image restoration framework, consisting of a restoration network and an upsampling network. In order to better restore the macro content of the image to be restored, the image restoration network will be used to restore the image at a low resolution, and the upsampling network will be used to generate the detailed content of the restored image. The method uses cross-space comparison learning to remove the noise of the image to be restored, and realizes automatic restoration of the image to be restored in the encoding stage. In addition, a mask-aware discriminator is designed to improve the performance of image restoration. The mask-aware discriminator converts the data in the latent space into a mask image of the input size for accurate training.

[0142] Extensive experiments demonstrate that our blind image inpainting framework achieves better performance than the best methods on multiple datasets, and can generate visually plausible and consistent inpainting results even when the noisy regions are large and misaligned.

[0143] The latent space features are fed into the mask discriminator for mask-aware adversarial training. Since the loss can also be regarded as a pixel-level binary classification problem in the implementation, this application uses a network similar to the decoder as the mask discriminator. Finally, the decoder connects the feature map [E(I L deg ),(E(I L deg ))] as input to produce a low-resolution restoration result, where E is a neural network model for extracting image feature maps.

[0144] This application uses bilinear upsampling to repair the result I L inp It is concatenated with the original image to be modified in the channel dimension and fed into the upsampling network. Figure 2 The bilinear in the method is to convert the low-resolution restoration result into high-resolution through bilinear, but this high-resolution image is not clear enough, and then it is input into the subsequent network for refinement and improvement.

[0145] Since the upsampling network is composed of auxiliary entity information (from the low-resolution result I L inp The correct structure and the original figure to be repaired I L deg The non-degenerate part of the region) can be guided by only a network consisting of several residual blocks.

[0146] This application proposes a novel blind image restoration framework to generate visually plausible results. Cross-space comparison learning is introduced to teach the encoder how to remove noise in the image to be repaired and repair them in a high-level semantic space. Unlike traditional image restoration methods, which all repair the image to be repaired at the decoder stage, we do it at the encoding stage. In addition, a mask-aware comparison loss is used to align the latent representations of the image to be repaired and the corresponding real image, which further optimizes the performance of the restoration network.

[0147] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0148] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for blind image repair is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0149] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0150] Step S100, obtaining a real image and a degraded image corresponding to the real image;

[0151] Step S200, inputting both the degraded image and the reference data into a first encoder for processing, wherein the reference data is obtained by processing the real image through a cross-space contrast loss module;

[0152] Step S300, inputting the calculation result of the first encoder into the decoder for processing, and outputting the restored image;

[0153] Step S400, comparing the repaired image with the real image and updating the blind image repair network for cyclic training until the repaired image is close to the real image, thereby obtaining a trained blind image repair network.

[0154] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0155] Step S100, obtaining a real image and a degraded image corresponding to the real image;

[0156] Step S200, inputting both the degraded image and the reference data into a first encoder for processing, wherein the reference data is obtained by processing the real image through a cross-space contrast loss module;

[0157] Step S300, inputting the calculation result of the first encoder into the decoder for processing, and outputting the restored image;

[0158] Step S400, comparing the repaired image with the real image and updating the blind image repair network for cyclic training until the repaired image is close to the real image, thereby obtaining a trained blind image repair network.

[0159] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0160] In one embodiment, a computer program product is provided, comprising computer instructions, which when executed by a processor implement the following steps:

[0161] Step S100, obtaining a real image and a degraded image corresponding to the real image;

[0162] Step S200, inputting both the degraded image and the reference data into a first encoder for processing, wherein the reference data is obtained by processing the real image through a cross-space contrast loss module;

[0163] Step S300, inputting the calculation result of the first encoder into the decoder for processing, and outputting the restored image;

[0164] Step S400, comparing the repaired image with the real image and updating the blind image repair network for cyclic training until the repaired image is close to the real image, thereby obtaining a trained blind image repair network.

[0165] In this embodiment, the computer program product includes a program code portion for executing the steps of the method for blind image restoration in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. The computer program product may also be provided for download via a data network (e.g., via a RAN, via the Internet and / or via an RBS). Alternatively or additionally, the method may be encoded in a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), or functionality may be provided for download with the aid of a hardware description language.

[0166] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0167] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. When the technical features in different embodiments are embodied in the same figure, it can be regarded that the figure also discloses the combination examples of the various embodiments involved.

[0168] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for blind image restoration, wherein the image to be restored is input into a trained blind image restoration network to obtain a restoration result, wherein: The blind image restoration network comprises a first encoder and a decoder, and the blind image restoration network is trained by the following steps: Obtaining a real image and a degraded image corresponding to the real image; Inputting the degraded image and reference data into the first encoder for processing, wherein the reference data is obtained by processing the real image through a cross-space contrast loss module; Inputting the calculation result of the first encoder into the decoder for processing, and outputting the repaired image; Comparing the repaired image with the real image and updating the blind image repair network for cyclic training until the repaired image is close to the real image, thereby obtaining the trained blind image repair network; Processing the real image through a cross-spatial contrast loss module specifically includes: inputting the real image into a second encoder, the second encoder is connected to the first encoder through a cross-spatial contrast loss module, and the cross-spatial contrast loss function used by the cross-spatial contrast loss module is: ,in: BCE stands for binary cross entropy loss; Sigmoid, represents the regression function; C, represents the cross product of two features to obtain the covariance matrix; F 1 , represents the features corresponding to the degraded image; F 2 , represents the features corresponding to the repaired image; I, represents the identity matrix.

2. The method for blind image restoration according to claim 1, characterized in that: Inputting the degraded image and the reference data into the first encoder for processing, specifically comprising: The degraded image is input to the first encoder to obtain latent space features of the degraded image, and the reference data is connected to the latent space features of the degraded image.

3. The method for blind image restoration according to claim 1, characterized in that: The latent space features of the degraded image and the latent space features of the real image both include Value, Query and Key elements; The real image is input to a second encoder, and the second encoder is connected to the first encoder through a cross-space contrast loss module, specifically comprising: The real image is input to the second encoder, and the second encoder outputs the latent space features of the real image. The Query value of the latent space features of the real image is connected to the Key element of the latent space features of the degraded image through the first residual block, the cross-space contrast loss module, and the second residual block in sequence.

4. The method for blind image restoration according to claim 1, characterized in that: Inputting the calculation result of the first encoder into the decoder for processing, specifically comprising: The latent space features of the degraded image are input into the decoder for processing through a self-attention module.

5. The method for blind image restoration according to claim 1, characterized in that: The degraded image has an original mask, and the first encoder outputs a generated mask of the degraded image according to the degraded image. The blind image restoration method further includes: comparing the original mask with the generated mask, and updating the blind image restoration network for cyclic training until the restored image is close to the real image, thereby obtaining the trained blind image restoration network.

6. The method for blind image restoration according to claim 1, characterized in that: The image to be repaired is input into the trained blind image repair network to obtain the repair result, which specifically includes: The image to be repaired is input into the trained blind image repair network, and the resolution of the image to be repaired is improved by using an upsampling network. The upsampling network includes the image to be repaired and semantic features of the repair result.

7. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method for blind image restoration according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for blind image restoration described in any one of claims 1 to 6 are implemented.

9. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method for blind image restoration described in any one of claims 1 to 6 are implemented.