An image restoration method and system

Abstract

The application discloses an image recovery method and system, utilizes three different image recovery models of a deformation convolution attention network, a non-local enhancement network and a high-frequency enhancement double-branch network to better recover details and texture information of a damaged image, and through application of the three different image recovery models to common image recovery tasks, including synthetic noise removal, real image denoising, compression artifact removal and real image super-resolution, the advancement can be verified.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image restoration, and particularly relates to an image restoration method and system. BACKGROUND

[0002] In recent years, the popularity of various image information collection devices has brought about a large amount of digital images. However, in the process of obtaining, transmitting and storing digital images, digital images will be affected by various factors to produce different types of information loss:

[0003] (1) Information loss caused by image collection devices, noise introduced by an image signal processor (ISP) in the process of converting optical signals into electrical signals, color changes, etc.

[0004] (2) Blur caused by object or shooting device movement when shooting;

[0005] (3) Information loss caused by environmental factors, such as images obtained in rain, snow, fog, low-illumination environments;

[0006] (4) Artifacts, noise, block effects, etc. produced in image compression and reconstruction, and low-resolution images obtained under limited storage conditions.

[0007] How to efficiently restore the lost information in the damaged digital image and obtain high-quality digital images to improve visual quality and provide a good foundation for subsequent image information mining tasks (such as target detection, object segmentation, recognition, etc.) is a basic and important task in computer vision and image processing.

[0008] Image restoration contains various low-level vision tasks such as image denoising, JPEG compression artifact removal, etc. In traditional image restoration methods, researchers describe the latent patterns of images or loss factors using expert experience and theoretical reasoning, i.e., regular prior knowledge described by mathematical formulas or physical models, to predict loss information. For example, in the Block Matching 3D (BM3D) algorithm, the basic idea comes from the fact that natural images have a lot of similar repetitive information. The algorithm aggregates and transforms these similar structures using image block matching, fully utilizes sparsity and structural similarity, and performs image restoration. However, there are still some prior knowledge that cannot be expressed using existing rules. In deep learning-based image restoration algorithms, researchers train models using large amounts of data to learn and fit the patterns and prior knowledge in images or loss factors. Therefore, deep neural network-based image restoration algorithms have significantly improved performance compared to traditional algorithms. However, due to the highly ill-conditioned nature of image restoration and some problems with deep learning model methods, there are still many challenges in deep learning-based image restoration methods.

[0009] The image restoration problem is described as follows:

[0010] Image restoration contains various low-level vision tasks such as image denoising, JPEG compression artifact removal, etc. In traditional image restoration methods, researchers describe the latent patterns of images or loss factors using expert experience and theoretical reasoning, i.e., regular prior knowledge described by mathematical formulas or physical models, to predict loss information. For example, in the Block Matching 3D (BM3D) algorithm, the basic idea comes from the fact that natural images have a lot of similar repetitive information. The algorithm aggregates and transforms these similar structures using image block matching, fully utilizes sparsity and structural similarity, and performs image restoration. However, there are still some prior knowledge that cannot be expressed using existing rules. In deep learning-based image restoration algorithms, researchers train models using large amounts of data to learn and fit the patterns and prior knowledge in images or loss factors. Therefore, deep neural network-based image restoration algorithms have significantly improved performance compared to traditional algorithms. However, due to the highly ill-conditioned nature of image restoration and some problems with deep learning model methods, there are still many challenges in deep learning-based image restoration methods. The problems can be summarized as follows:

[0011] First, in recent years, various deep learning-based algorithms have emerged and been applied to various fields, bringing significant performance improvement to underlying vision tasks. Many deep learning-based methods still have some problems in obtaining larger receptive fields and learning rich features by deepening the network. With the deepening of the model, the fixed shape receptive field is expanded, but the recovered image is over-smoothed, lacking texture and edge details. To solve this problem, a deformable convolution attention network is introduced to better extract texture and edge detail features by learning focused adaptive receptive fields, improving image restoration performance.

[0012] Second, non-local information has been studied and applied in many fields, such as extracting correlation in video processing, establishing non-neighbor dependence between a series of words in natural language processing, and establishing the relationship between context keywords in text summarization extraction. In addition to extracting long-range correlation in the time domain, non-local operations can also establish correlation in the spatial domain, such as in computer vision tasks. In the field of computer vision, non-local operations that extract long-distance dependence between pixels have been used in many tasks. To better understand the effectiveness of non-local operations, it can be seen as an attention mechanism for modeling pixel-to-pixel relationships. This relationship is modeled as the dot product between the features of two pixels, and the larger the dot product value, the higher the correlation between the two pixels. The classic algorithm Non-Local Mean (NLM) uses non-local dependence in traditional image restoration methods, which takes advantage of the repetition of texture and structural information in images to improve the restoration of texture and edge details by extracting non-local correlation within a single image. However, the idea of extracting non-local correlation within a single image lacks more rich non-local information. Especially in cases where the non-local correlation within an image is limited, it is difficult to effectively obtain non-local features for image restoration. Here, a non-local enhancement network is proposed to further effectively extract non-local features in cases where the non-local similarity within an image is limited.

[0013] Third, to address the problem of texture and edge details being difficult to recover due to the loss of high-frequency information in image restoration results, researchers have proposed a class of methods that combine wavelet transform with deep learning models to improve high-frequency information recovery and generate multi-scale sparse features for better model learning. However, whether using a single or dual branch network, the image restoration method is performed in a single domain without considering multi-domain fusion to improve image restoration performance. To address the problem of texture and edge details being difficult to recover due to the loss of high-frequency information in image restoration results, a high-frequency enhancement dual-branch network is proposed to recover more texture and edge details by fusing dual-domain information. SUMMARY

[0014] The technical problems to be solved by the present application are to provide an image restoration method and system based on local non-local information and dual-domain fusion to solve the technical problem of texture edge detail loss in image restoration results.

[0015] The present application adopts the following technical solutions:

[0016] An image restoration method comprises the following steps:

[0017] S101, define a deformation convolution attention module and a cyclic attention offset module;

[0018] S102, embed the deformation convolution attention module and the cyclic attention offset module obtained in step S101 into a U-shaped network, and use discrete wavelet transform and inverse wavelet transform to apply a spatial attention module after connecting two feature maps to obtain offset features, thereby forming a deformation convolution attention network model;

[0019] S103, train the deformation convolution attention network model constructed in step S102 using different general data sets for different image restoration tasks, use mean square error as a loss function, calculate the mean square error of pixels between the image generated by the generator and the corresponding target image, calculate the gradient and return to adjust the parameters of the deformation convolution attention network model;

[0020] S104, for the gray image noise removal in the synthetic noise removal task, use different noise levels of additive white Gaussian noise to add to the gray DIV2K training set to train the deformation convolution attention network model;

[0021] S105, for color image noise removal in the synthetic noise removal task, use different noise levels of additive white Gaussian noise to add to the color DIV2K training set to train the deformation convolution attention network model;

[0022] S106, for the real noise removal task, use the general data set SIDD Medium to train the deformation convolution attention network model;

[0023] S107, for the JPEG artifact removal task, use the general data set DIV2K training set image and the training set constructed by different quality factors of JPEG compression for training;

[0024] S108, for the real image super-resolution task, use the training set image of the general data set RealSR to train the deformation convolution attention network model with different scaling factors;

[0025] S109, optimize the deformation convolution attention network model by using the model distillation method;

[0026] S110, define a teacher model of a deformation convolution attention network including two simplified parameters, in series as a learning image from a previous degradation and recovery process, the teacher model takes a clean / high-resolution image as input, the first DANet-S is used as a degradation model to generate a noise / low-resolution image, and the generated noise / low-resolution image is used as input of the second DANet-S to recover the clean / high-resolution image;

[0027] S111, define a student model, the student model shares the same structure with the recovery model in the teacher model, and the recovery model of the teacher model is used as the initialization model of the student model;

[0028] S112, train using the same data set as the previous different task, first train the teacher model, and then use the image recovery part model parameters in the trained teacher model as the initialization parameters of the student model;

[0029] S113, further train the student model using the training data, and the student model is trained to converge as the final image recovery model;

[0030] S114, apply the model distillation method for the deformation convolution attention network in steps S109-S113 to the image recovery task in steps S104-S108, so that the model in each task obtains the same image recovery ability as the original model under the condition of reducing parameters.

[0031] Specifically, in step S101, the deformation convolution attention module includes a deformation convolution layer and a double attention module, the deformation convolution attention module generates a focused adaptive receptive field in the image recovery model by combining the deformation convolution and the double attention module; the recurrent attention offset module fuses the low-resolution offset feature and the high-resolution offset feature by connecting offset feature maps of different scales to obtain spatial and texture detail information; the recurrent attention offset module transmits the offset feature of the current scale and the multi-scale offset feature between two consecutive deformation convolution attention modules.

[0032] In a second aspect, an embodiment of the present application provides an image recovery system, comprising:

[0033] A definition module defines a deformation convolution attention module and a recurrent attention offset module;

[0034] An embedding module embeds the deformation convolution attention module and the recurrent attention offset module obtained by the definition module into a U-shaped network, and uses discrete wavelet transform and inverse wavelet transform to apply a spatial attention module to obtain offset features after connecting two feature maps, to form a deformation convolution attention network model;

[0035] The training module trains the deformation convolution attention network model constructed by the embedding module for different image restoration tasks using different general data sets, takes mean square error as a loss function, calculates the mean square error of pixels between the image generated by the generator and the corresponding target image, calculates the gradient and returns to adjust the parameters of the deformation convolution attention network model;

[0036] The grayscale module trains the deformation convolution attention network model for grayscale image noise removal in the synthetic noise removal task by adding additive white Gaussian noise with different noise levels to the grayscale DIV2K training set;

[0037] The color module trains the deformation convolution attention network model for color image noise removal in the synthetic noise removal task by adding additive white Gaussian noise with different noise levels to the color DIV2K training set;

[0038] The real module trains the deformation convolution attention network model using the general data set SIDD Medium for the real noise removal task;

[0039] The artifact module trains the deformation convolution attention network model using the general data set DIV2K training set image and the training set constructed by different quality factors of JPEG compression for the JPEG artifact removal task;

[0040] The super-resolution module trains the deformation convolution attention network model using the training set images of the general data set RealSR for the real image super-resolution task;

[0041] The optimization module optimizes the deformation convolution attention network model using the model distillation method;

[0042] The series module defines a teacher model including two parameter-reduced deformation convolution attention networks, and is connected in series as a learning image from the degradation and restoration process. The teacher model takes clean / high-resolution images as input, the first DANet-S as a degradation model to generate noise / low-resolution images, and the generated noise / low-resolution images are used as input to the second DANet-S to restore clean / high-resolution images;

[0043] The restoration module defines a student model, which shares the same structure as the restoration model in the teacher model, and uses the restoration model of the teacher model as the initialization model of the student model;

[0044] The parameter module trains the same data set as the data set used in the previous different tasks, first trains the teacher model, and then uses the image restoration part model parameters in the trained teacher model as the initialization parameters of the student model;

[0045] The image module uses the training data to further train the student model, and the student model is trained to converge as a final image restoration model;

[0046] The output module applies the model distillation method of the deformation convolution attention network of the optimization module, the series module, the restoration module, the parameter module and the image module to image restoration tasks in the grayscale module, the color module, the real module, the artifact module and the super-resolution module, so that the models in each task obtain the same image restoration capability as the original model under the condition of reducing the parameters.

[0047] In a third aspect, an image restoration method is provided, comprising the following steps:

[0048] S201, define a batch non-local module and an enhanced non-local module;

[0049] S202, stack the batch non-local module and the enhanced non-local module obtained in step S201 into a U-shaped network to form a non-local enhanced network model;

[0050] S203, for different image restoration tasks, use different general data sets to train the non-local enhanced network model constructed in step S202, and use mean square error as a loss function; calculate the mean square error of the pixels between the image generated by the generator and the corresponding target image, calculate the gradient and return to adjust the parameters of the non-local enhanced network model;

[0051] S204, for the gray image noise removal in the synthetic noise removal task, use different noise levels of additive white Gaussian noise to add to the gray DIV2K training set to train the non-local enhanced network model, obtain the synthetic noise removal model under different noise levels, and apply it to the test in the synthetic noise removal task;

[0052] S205, for the real noise removal task, use the general data set SIDD Medium to train the non-local enhanced network model, obtain the real noise removal model, and apply it to the test in the real noise removal task;

[0053] S206, for the JPEG artifact removal task, use the general data set DIV2K training set image and the training set constructed by JPEG compression with different quality factors to train, obtain the JPEG artifact removal model under different quality factors, and apply it to the test in the JPEG artifact removal task;

[0054] S207, for the real image super-resolution task, use the training set images of the general data set RealSR to train the model with different scaling factors, obtain the real image super-resolution model under different magnification factors, and apply it to the test in the real image super-resolution task with different parameters.

[0055] Specifically, in step S201, the batch non-local module is used to extract non-local relationship features from a group of images to obtain non-local features.

[0056] The enhanced non-local module is constructed based on the proposed batch non-local module and different types of non-local extractors.

[0057] In a fourth aspect, an embodiment of the present application provides an image restoration system, comprising:

[0058] A definition module defines the batch non-local module and the enhanced non-local module.

[0059] An embedding module stacks and embeds the batch non-local module and the enhanced non-local module obtained by the definition module into a U-shaped network to form a non-local enhanced network model.

[0060] A calculation module uses different general data sets to train the non-local enhanced network model constructed by the embedding module for different image restoration tasks, uses mean square error as a loss function, calculates the mean square error of pixels between the image generated by the generator and the corresponding target image, calculates the gradient and returns to adjust the parameters of the non-local enhanced network model.

[0061] A grayscale module uses different noise levels of additive white Gaussian noise to add to the grayscale DIV2K training set to train the non-local enhanced network model for grayscale image noise removal in the synthetic noise removal task, obtains a synthetic noise removal model under different noise levels, and applies it to the test in the synthetic noise removal task.

[0062] A real module uses a general data set SIDD Medium to train the non-local enhanced network model for a real noise removal task, obtains a real noise removal model, and applies it to the test in the real noise removal task.

[0063] An artifact module uses a general data set DIV2K training set image and a training set constructed by JPEG compression with different quality factors to train the non-local enhanced network model for a JPEG artifact removal task, obtains a JPEG artifact removal model under different quality factors, and applies it to the test in the JPEG artifact removal task.

[0064] A super-resolution module uses a training set image of a general data set RealSR to train the non-local enhanced network model for a real image super-resolution task, obtains a real image super-resolution model under different magnification factors, and applies it to the test in the real image super-resolution task under different parameters.

[0065] In a fifth aspect, an embodiment of the present application provides an image restoration method, comprising the following steps:

[0066] S301, define a frequency domain branch and a pixel domain branch;

[0067] S302, fuse the results of the frequency domain branch and the pixel domain branch obtained in step S301 to construct a high-frequency enhancement double-branch network model;

[0068] S303, use different general data sets for different image restoration tasks to train the high-frequency enhancement double-branch network model constructed in step S302, and use mean square error as the loss function; calculate the mean square error of the pixels between the image generated by the generator and the corresponding target image, calculate the gradient and return to adjust the model parameters;

[0069] S304, for the gray image noise removal in the synthetic noise removal task, use different noise levels of additive white Gaussian noise to add to the gray DIV2K training set to train the high-frequency enhancement double-branch network model, and obtain the synthetic noise removal model under different noise levels, and apply it to the test in the synthetic noise removal task;

[0070] S305, for the real noise removal task, use the general data set SIDD Medium to train the model, obtain the real noise removal model, and apply it to the test in the real noise removal task;

[0071] S306, for the JPEG artifact removal task, use the general data set DIV2K training set image and construct a training set by performing JPEG compression with different quality factors, train the training set to obtain the JPEG artifact removal model under different quality factors, and apply it to the test in the JPEG artifact removal task.

[0072] Specifically, in step S301, the frequency domain branch uses Shearlet decomposition to extract the high-frequency subband and the low-frequency subband of the input noise image;

[0073] The pixel domain branch uses a deformation convolution Transformer module to perform local and non-local restoration on the image, stacks the deformation convolution Transformer module into a pixel branch model, and in the pixel domain, combines the noise input with the image reconstructed from the frequency domain branch as the output of the pixel domain branch in the high-frequency enhancement double-branch network model.

[0074] Specifically, in step S302, the high-frequency enhancement double-branch network model enhances the texture information and the structure information by multiplying the texture information and the low-frequency structure information weight feature map obtained by the frequency domain branch with the denoising result of the pixel domain branch, and obtains the final result through multiple convolution layers, multiplication operations and residual connection. The high-frequency enhancement information obtained by the frequency domain branch is weighted to make the denoising result recover more texture and edge detail information.

[0075] In a sixth aspect, an embodiment of the present application provides an image restoration system, comprising:

[0076] a definition module, which defines a frequency domain branch and a pixel domain branch;

[0077] a fusion module, which fuses the results of the frequency domain branch and the pixel domain branch obtained by the definition module to construct a high-frequency enhancement double-branch network model;

[0078] a calculation module, which trains the high-frequency enhancement double-branch network model constructed by the fusion module using different general data sets for different image restoration tasks, takes mean square error as a loss function, calculates the mean square error of pixels between the image generated by the generator and the corresponding target image, calculates the gradient and returns to adjust the model parameters;

[0079] a grayscale module, which uses different noise levels of additive white Gaussian noise to add to the grayscale DIV2K training set for the high-frequency enhancement double-branch network model to train the high-frequency enhancement double-branch network model for the removal of grayscale image noise in the synthetic noise removal task, obtains a synthetic noise removal model under different noise levels, and is applied to the test in the synthetic noise removal task;

[0080] a real module, which uses a general data set SIDD Medium to train the model for the real noise removal task, obtains a real noise removal model, and is applied to the test in the real noise removal task;

[0081] an artifact module, which uses a general data set DIV2K training set image and a training set constructed by JPEG compression of different quality factors to train the model for the JPEG artifact removal task, obtains a JPEG artifact removal model under different quality factors, and is applied to the test in the JPEG artifact removal task.

[0082] Compared with the prior art, the present application has at least the following beneficial effects:

[0083] The image restoration method of the present application proposes a deformation convolution attention network to solve the problem of excessive smoothing and texture loss caused by a larger receptive field in a common convolutional neural network, thereby restoring more texture and edge detail information. In order to further simplify the model parameter amount of the deformation convolution attention network and maintain advanced performance, a model distillation scheme is proposed to train a light deformation convolution attention network.

[0084] Further, the deformation convolution attention network includes two new modules, a deformation convolution attention module and a cyclic attention offset module; the model performance is improved from the aspects of extracting a more focused adaptive receptive field and fusing multi-scale offset feature.

[0085] Further, the deformed convolution attention module cooperates with various attention modules through deformed convolution to generate focused and adaptive receptive fields to better extract texture and edge detail information; the cyclic attention offset module transmits offset features generated by different layers in the encoding-decoding structure of the deformed convolution attention network to provide more focused and accurate offsets for the deformed convolution attention module.

[0086] The image restoration method of the present application uses a non-local enhancement network to extract non-local features in multiple images, so that non-local features can be effectively extracted even when the non-local similarity within an image is limited.

[0087] Further, a variety of non-local feature extraction modules are combined with the batch non-local module to form an enhanced non-local module to obtain rich non-local features. In addition to using diversified non-local information, a feature extraction module combining non-local and local information is also proposed. In the non-local and local feature extraction module, the enhanced non-local module is combined with the deformable convolution module to extract rich non-local and fine local information.

[0088] The image restoration method of the present application uses a non-local enhancement network to extract non-local features in multiple images, so that non-local features can be effectively extracted even when the non-local similarity within an image is limited.

[0089] Further, in the frequency domain branch, the low-frequency subband of the image contains structural information, and the high-frequency band contains more texture information and noise-like information. Therefore, the structural information (from the low-frequency subband) can be used to enhance the texture information (from the high-frequency subband). In the pixel domain branch, a new deformed convolution Transformer module is proposed to extract local and non-local features of the image. A pixel domain denoising model is constructed based on multiple deformed convolution Transformer modules. The final denoising result is obtained by fusing the above dual-domain restoration results.

[0090] Further, the fusion of the dual branches is to enhance the texture information and the structural information by multiplying the texture information and the low-frequency structural information weight feature map obtained by the frequency domain branch with the restoration result of the pixel domain branch, and then obtaining the final result through multiple convolution layers, multiplication operations and residual connections. The high-frequency enhancement information obtained by the frequency domain branch is weighted to make the restoration result restore more texture and edge detail information.

[0091] In summary, the present application introduces a deformation convolution attention network for the common convolution deep model to obtain a larger receptive field, which leads to an over-smoothed restored result. By learning an adaptive receptive field, the texture and edge detail features are better extracted, and the image restoration performance is improved. In view of the characteristics that there are texture and structure information in the image, the existing non-local module based on deep learning improves the restoration ability of texture and edge details by extracting the non-local correlation inside a single image. Here, a non-local enhancement network is proposed to further effectively extract non-local features in the case of limited non-local similarity inside the image. In view of the problem that the high-frequency information in the image restoration result is easy to be lost, leading to the difficulty in restoring the texture and edge details, a high-frequency enhancement double-branch network is proposed, so as to restore more texture and edge details by fusing the double-domain information.

[0092] The technical solutions of the present application are described in further detail below by means of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0093] Figure 1 It is a whole structure diagram of the deformation convolution attention network;

[0094] Figure 2 It is a structure diagram of the deformation convolution attention module;

[0095] Figure 3 It is a structure diagram of the cyclic attention offset module;

[0096] Figure 4 It is a structure diagram of the simplified deformation convolution attention network;

[0097] Figure 5 It is a flow chart of model distillation;

[0098] Figure 6 It is a structure diagram of the non-local enhancement network;

[0099] Figure 7 It is a structure diagram of the batch non-local module;

[0100] Figure 8 It is a structure diagram of the enhanced non-local module;

[0101] Figure 9 It is a structure diagram of the high-frequency enhancement double-branch network;

[0102] Figure 10 It is a structure diagram of the frequency domain branch;

[0103] Figure 11 It is a structure diagram of the pixel domain branch;

[0104] Figure 12 It is a comparison diagram of the deformation convolution attention network and the comparison method in the gray image denoising (σ=25);

[0105] Figure 13 Figure 4 is a comparison chart of the results of the deformed convolution attention network and comparative methods in color image denoising (σ = 15);

[0106] Figure 14 Figure 5 is a comparison chart of the results of the deformed convolution attention network and comparative methods in real noise removal;

[0107] Figure 15 Figure 6 is a comparison chart of the results of the deformed convolution attention network and comparative methods in JPEG artifact removal;

[0108] Figure 16 Figure 7 is a comparison chart of the results of the deformed convolution attention network and comparative methods in real-world image super-resolution;

[0109] Figure 17 Figure 8 is a comparison chart of the results of the non-local enhancement network and comparative methods in grayscale image denoising (σ = 25);

[0110] Figure 18 Figure 9 is a comparison chart of the results of the non-local enhancement network and comparative methods in real noise removal;

[0111] Figure 19 Figure 10 is a comparison chart of the results of the non-local enhancement network and comparative methods in JPEG artifact removal;

[0112] Figure 20 Figure 11 is a comparison chart of the results of the non-local enhancement network and comparative methods in real-world image super-resolution;

[0113] Figure 21 Figure 12 is a comparison chart of the results of the high-frequency enhanced double-branch network and comparative methods in grayscale image denoising (σ = 50);

[0114] Figure 22 Figure 13 is a comparison chart of the results of the high-frequency enhanced double-branch network and comparative methods in real noise removal;

[0115] Figure 23 Figure 14 is a comparison chart of the results of the high-frequency enhanced double-branch network and comparative methods in JPEG artifact removal. DETAILED DESCRIPTION

[0116] The present application has the following several research and improvement directions for texture and edge loss in image restoration:

[0117] First, in the existing convolutional neural network-based method, the receptive field is expanded by deepening the network to improve the network performance, but the image content is ignored, and simply expanding the fixed shape receptive field cannot extract effective features according to the image content, resulting in the problem of texture and edge loss in the restoration result.

[0118] Second, for the characteristics of the existence of texture and structure information in the image, the existing non-local module based on deep learning improves the recovery of texture and edge details by extracting the non-local correlation inside a single image.

[0119] Third, for the problem that the texture and edge details are difficult to recover due to the loss of high-frequency information in the image restoration result, how to preserve more high-frequency information in the image restoration process to recover more texture and edges in the damaged image needs to be studied.

[0120] For the above research direction, the present application respectively proposes a deformation convolution attention network, a non-local enhancement network and a high-frequency enhanced double-branch network, and achieves advanced effects on common image restoration tasks, including synthetic noise removal, real image denoising, compression artifact removal and real image super-resolution.

[0121] The present application provides an image restoration method based on local non-local information and dual-domain fusion, a method for combining multi-scale perception loss with a generative adversarial network, and realizing satellite image super-resolution using aerial image prior, by combining multi-scale perception loss with a generative adversarial network, realizing satellite image super-resolution under conditional constraints, wherein a network for denoising is trained using satellite images, a network for realizing image super-resolution is trained using aerial images, and a clear feature prior in the aerial image is extracted using a Gaussian mixture model, and the image after super-resolution reconstruction is further reconstructed.

[0122] The present application uses three different image restoration models, deformation convolution attention network, non-local enhancement network and high-frequency enhanced double-branch network, to better recover the details and texture information of the damaged image, and by applying the three different image restoration models to common image restoration tasks, including synthetic noise removal, real image denoising, compression artifact removal and real image super-resolution, the advancement thereof can be verified.

[0123] Deformation convolution attention network

[0124] To solve the problem that the larger receptive field in common convolutional neural network leads to over-smoothed results and texture loss, a deformation convolution attention network is proposed to recover more texture and edge details. The deformation convolution attention network contains two new modules, the deformation convolution attention module and the recurrent attention offset module. The deformation convolution attention module, through the cooperation of deformation convolution and various attention modules, produces a focused and adaptive receptive field to better extract texture and edge detail information. The recurrent attention offset module transmits the offset features generated by different layers in the encoding-decoding structure of the deformation convolution attention network, providing more focused and accurate offsets for the deformation convolution attention module. To further reduce the model parameter quantity of the deformation convolution attention network while maintaining advanced performance, a model distillation scheme is used to train a lightweight deformation convolution attention network.

[0125] Non-local enhancement network

[0126] To address the problem of repeated texture and structure information in images, existing non-local modules based on deep learning improve the recovery of texture and edge details by extracting non-local correlations within a single image. Here, a non-local enhancement network is proposed to further effectively extract non-local features in the case of limited non-local similarity within an image. The batch non-local module is proposed in the non-local enhancement network. The batch non-local module extracts non-local features from a group of images to improve the effectiveness of the correlation features.

[0127] In addition, various non-local feature extraction modules are combined with the batch non-local module to form an enhanced non-local module to obtain rich non-local features. In addition to using diverse non-local information, a feature extraction module that combines non-local and local information is proposed. In the non-local and local feature extraction module, the enhanced non-local module is combined with the deformable convolution module to extract rich non-local and fine local information. Finally, multiple non-local and local feature extraction modules are embedded in a U-shaped network structure to construct a non-local enhancement network.

[0128] High-frequency enhancement dual-branch network

[0129] To address the problem of texture and edge details being difficult to recover due to the loss of high-frequency information in image restoration results, a high-frequency enhancement dual-branch network is proposed to recover more texture and edge details by fusing dual-domain information.

[0130] The high-frequency enhanced double-branch network enhances high-frequency information by fusing double-domain (pixel domain and frequency domain) restoration results to improve the problem of missing texture and edge details. In the frequency domain branch, the low-frequency subband of the image contains structural information, and the high-frequency band contains more texture information and noise-like information. Therefore, the structural information (from the low-frequency subband) is used to enhance the texture information (from the high-frequency subband) here. In the pixel domain branch, a new deformation convolution Transformer module is proposed to extract local and non-local features of the image. A pixel domain denoising model is constructed based on multiple deformation convolution Transformer modules, and the above double-domain restoration results are fused to obtain the final denoising result.

[0131] Referring to Figure 1 The image restoration method based on the deformation convolution attention network comprises the following steps:

[0132] In the deformation convolution attention network, two new modules, namely a deformation convolution attention module and a recurrent attention offset module, are included, so that the model learns an adaptive attention field, thereby better preserving texture and edge detail information in the restored image.

[0133] Referring to Figure 2 The deformation convolution attention module includes a deformation convolution layer and a double attention module. The deformation convolution attention module combines deformation convolution with a double attention module (a channel attention module and a spatial attention module) to generate an adaptive attention field in the image restoration model, thereby better recovering texture and edge detail information.

[0134] Referring to Figure 3 The recurrent attention offset module fuses low-resolution offset feature quantities and high-resolution offset feature quantities by connecting offset feature maps of different scales to obtain more spatial and texture detail information. The recurrent attention offset module transmits offset feature quantities of the current scale and multi-scale offset feature quantities between two consecutive deformation convolution attention modules in the model.

[0135] S102, embed the deformation convolution attention module and the recurrent attention offset module obtained in step S101 in a U-shaped network, and use discrete wavelet transform (DWT) inverse wavelet transform (IWT) instead of bilinear interpolation to apply a spatial attention module after connecting two feature maps to obtain more spatially focused offset features, and finally form a deformation convolution attention network.

[0136] S103, train the above constructed deformation convolution attention model using different general data sets for different image restoration tasks, use mean square error (MSE) as the loss function, calculate the mean square error of the pixels between the image generated by the generator and the corresponding target image, calculate the gradient and return to adjust the model parameters;

[0137] S104, for the gray image noise removal in the synthetic noise removal task, use the gray DIV2K training set with different noise levels of additive white Gaussian noise to train the model, and use the general standard data set Set12, BSD68, gray Urber100 to test the performance of the network model under different noise levels, and use the peak signal-to-noise ratio (PSNR) and structural similarity parameter (SSIM) as the measurement standard to measure the image restoration effect;

[0138] S105, for the color image noise removal in the synthetic noise removal task, use the color DIV2K training set with different noise levels of additive white Gaussian noise to train the model, and use the general standard data set Koda24, CBSD68, color Unber100 to test the performance of the network model under different noise levels, and use the peak signal-to-noise ratio (PSNR) and structural similarity parameter (SSIM) as the measurement standard to measure the image restoration effect;

[0139] S106, for the real noise removal task, use the general data set SIDD Medium to train the model, and use the DND, SIDD general test data set to test the performance of the network model, and use the peak signal-to-noise ratio (PSNR) and structural similarity parameter (SSIM) as the measurement standard to measure the image restoration effect;

[0140] S107, for the JPEG artifact removal task, use the general data set DIV2K training set image, and use the training set constructed by different quality factors of JPEG compression to train, and use the LIVE1, Classic5 general test data set to test the performance of the network model, and use the peak signal-to-noise ratio (PSNR) and structural similarity parameter (SSIM) as the measurement standard to measure the image restoration effect;

[0141] S108, for the real image super-resolution task, use the training set image of the general data set RealSR to train the model with different scaling factors, and use the test data set of RealSR to test the performance of the network model, and use the peak signal-to-noise ratio (PSNR) and structural similarity parameter (SSIM) as the measurement standard to measure the image restoration effect;

[0142] S109, adopt model distillation method to optimize the deformation convolution attention network model;

[0143] Referring to Figure 4 and Figure 5 , including training a teacher model containing more parameters to guide the student model (a simplified parameter variant convolution attention network) to improve the performance of the simplified model.

[0144] S110, define the teacher model, including two simplified parameter variant convolution attention networks (DANet-S), see the specific structure of Figure 4 , in series as a learning image from the degradation and recovery process, the teacher model takes the clean / high resolution image as input, the first DANet-S as a degradation model to generate noise / low resolution image, the generated noise / low resolution image is used as the input of the second DANet-S to restore the clean / high resolution image;

[0145] Such symmetrical structure design makes the teacher model take into account the constraint loss (ConstraintLoss) in the training process. By connecting the two DANet-S, more priori is obtained by learning the degradation and recovery process, and better extraction of complex priori knowledge in different image restoration tasks.

[0146] S111, define the student model, the student model shares the same structure with the restoration model in the teacher model, so as to use the restoration model of the teacher model as the initialization model of the student model. In this way, the priori knowledge obtained in the teacher model can be effectively transferred to the student model, improving the training progress and effect of the student model;

[0147] S112, the same data set as used in the previous different task is used for training. First, train the larger teacher model, and then use the image restoration part model parameters in the trained teacher model as the initialization parameters of the student model;

[0148] S113, further fine-tuning the student model using the training data to further improve the performance. After the student model training converges, it is used as the final image restoration model, which can achieve a performance close to the original variant convolution attention network model but contains fewer parameters;

[0149] S114, apply the model distillation method of steps S109-S113 to the image restoration task in steps S104-S108, so that the DANet-S with fewer parameters has a performance close to the original variant convolution attention network model.

[0150] In still another embodiment of the present application, an image restoration system is provided, which can be used to implement the above-mentioned image restoration method, and specifically, the image restoration system comprises a definition module, an embedding module, a training module, a grayscale module, a color module, a real module, an artifact module, a super-resolution module, an optimization module, a series module, a restoration module, a parameter module, an image module and an output module.

[0151] The definition module defines the deformation convolution attention module and the cyclic attention offset module.

[0152] The embedding module embeds the deformation convolution attention module and the cyclic attention offset module obtained by the definition module into a U-shaped network, and uses discrete wavelet transform and inverse wavelet transform to apply a spatial attention module after connecting two feature maps to obtain offset features, thereby constituting a deformation convolution attention network model.

[0153] The training module trains the deformation convolution attention network model constructed by the embedding module for different image restoration tasks using different general data sets, takes mean square error as a loss function, calculates the mean square error of pixels between the image generated by the generator and the corresponding target image, calculates the gradient and returns to adjust the parameters of the deformation convolution attention network model.

[0154] The grayscale module trains the deformation convolution attention network model using different noise levels of additive white Gaussian noise added to the grayscale DIV2K training set for the grayscale image noise removal in the synthetic noise removal task.

[0155] The color module trains the deformation convolution attention network model using different noise levels of additive white Gaussian noise added to the color DIV2K training set for the color image noise removal in the synthetic noise removal task.

[0156] The real module trains the deformation convolution attention network model using the general data set SIDD Medium for the real noise removal task.

[0157] The artifact module trains the deformation convolution attention network model using the general data set DIV2K training set image and the training set constructed by different quality factors of JPEG compression for the JPEG artifact removal task.

[0158] The super-resolution module trains the deformation convolution attention network model using the training set images of the general data set RealSR for the real image super-resolution task.

[0159] The optimization module optimizes the deformation convolution attention network model by using a model distillation method.

[0160] The series module defines a teacher model of the deformation convolution attention network including two simplified parameters, and the series is used as a learning image from a degradation and recovery process, the teacher model takes a clean / high-resolution image as input, the first DANet-S is used as a degradation model to generate a noise / low-resolution image, and the generated noise / low-resolution image is used as input of the second DANet-S to recover the clean / high-resolution image;

[0161] The recovery module defines a student model, and the student model shares the same structure with the recovery model in the teacher model, and the recovery model of the teacher model is used as an initialization model of the student model;

[0162] The parameter module is trained using the same dataset as the previous different task, the teacher model is first trained, and then the image recovery part of the trained teacher model is used as the initialization parameter of the student model;

[0163] The image module further trains the student model using training data, and the student model is used as the final image recovery model after training convergence;

[0164] The output module applies the model distillation method of the deformation convolution attention network of the optimization module, the series module, the recovery module, the parameter module and the image module to the image recovery tasks in the gray module, the color module, the real module, the artifact module and the super-resolution module, so that the model in each task obtains the same image recovery ability as the original model under the condition of reducing the parameters.

[0165] Referring to Figure 6 , the image recovery method based on the non-local enhancement network comprises the following steps:

[0166] S201, the non-local enhancement network mainly includes two new modules, namely, a batch non-local module and an enhanced non-local module. By combining various non-local features to enhance local features, more texture and edge details can be retained in the image during recovery.

[0167] Referring to Figure 7 , the batch non-local module is defined, that is, non-local relationship features are extracted from a group of images to obtain richer non-local features and solve the problem that non-local correlation cannot be effectively obtained when the internal correlation of the image is limited.

[0168] Referring to Figure 8 , the enhanced non-local module is defined, which is constructed based on the proposed batch non-local module and different types of non-local extractors. The deformation convolution module learns an adaptive receptive field through deformation convolution to extract local information. Through these two modules, the non-local and local information extraction modules can combine local features and non-local features.

[0169] S202, stack the batch non-local module and the enhanced non-local module obtained in step S201 into a U-shaped network, and finally construct a non-local enhanced network model;

[0170] S203, training the non-local enhanced network model constructed above using different general data sets for different image restoration tasks, taking mean square error (MSE) as a loss function; calculating the mean square error of pixels between the image generated by the generator and the corresponding target image, calculating the gradient and returning to adjust the model parameters;

[0171] S204, for the gray image noise removal in the synthetic noise removal task, the model is trained using different noise levels of additive white Gaussian noise added to the gray DIV2K training set, the general standard data set Set12, BSD68, and gray Urber100 are used for testing, and the performance of the network model under different noise levels is tested, and the measurement standard uses peak signal-to-noise ratio (PSNR) and structural similarity parameter (SSIM) to measure the image restoration effect;

[0172] S205, for the real noise removal task, the model is trained using the general data set SIDD Medium, and the performance of the network model is tested using the general test data set DND, SIDD, the measurement standard uses peak signal-to-noise ratio (PSNR) and structural similarity parameter (SSIM) to measure the image restoration effect;

[0173] S206, for the JPEG artifact removal task, the general data set DIV2K training set image is used, and the training set constructed by JPEG compression with different quality factors is trained, the general test data set LIVE1, Classic5 is used for testing, and the performance of the network model is tested, and the measurement standard uses peak signal-to-noise ratio (PSNR) and structural similarity parameter (SSIM) to measure the image restoration effect;

[0174] S207, for the real image super-resolution task, the training set image of the general data set RealSR is used for model training with different scaling factors, the performance of the network model is tested using the test data set of RealSR, and the measurement standard uses peak signal-to-noise ratio (PSNR) and structural similarity parameter (SSIM) to measure the image restoration effect.

[0175] In another embodiment of the application, an image restoration system is provided, which can be used to implement the above-mentioned image restoration method. Specifically, the image restoration system comprises a definition module, an embedding module, a calculation module, a gray module, a real module, an artifact module and a super-resolution module.

[0176] The definition module defines the batch non-local module and the enhanced non-local module;

[0177] The embedding module stacks the batch non-local module and the enhanced non-local module obtained by the definition module into a U-shaped network to form a non-local enhanced network model;

[0178] The calculation module trains the non-local enhanced network model constructed by the embedding module using different general data sets for different image restoration tasks, takes the mean square error as a loss function, calculates the mean square error of pixels between the image generated by the generator and the target image corresponding to the image, calculates the gradient and returns to adjust the parameters of the non-local enhanced network model;

[0179] The gray module trains the non-local enhanced network model using different noise levels of additive white Gaussian noise added to the gray DIV2K training set for the gray image noise removal in the synthetic noise removal task, obtains a synthetic noise removal model under different noise levels, and is applied to the test in the synthetic noise removal task;

[0180] The real module trains the non-local enhanced network model using the general data set SIDD Medium for the real noise removal task, obtains a real noise removal model, and is applied to the test in the real noise removal task;

[0181] The artifact module trains the non-local enhanced network model using the general data set DIV2K training set image and the training set constructed by performing JPEG compression on the image with different quality factors for the JPEG artifact removal task, obtains a JPEG artifact removal model under different quality factors, and is applied to the test in the JPEG artifact removal task;

[0182] The super-resolution module trains the non-local enhanced network model using the training set images of the general data set RealSR for the real image super-resolution task, obtains a real image super-resolution model under different magnification factors, and is applied to the test in the real image super-resolution task under different parameters.

[0183] Please refer to Figure 9 The image restoration method based on the high-frequency enhanced double-branch network comprises the following steps:

[0184] S301, based on the high-frequency enhanced double-branch network, a double-branch model is included, the double-branch model restores noise input from the frequency domain and the pixel domain, and then cooperates to generate a denoising result with more texture details. The frequency domain branch and the pixel domain are fused to restore more rich texture and edge detail information. The high-frequency enhanced information and the low-frequency information obtained in the frequency domain are transformed back to the pixel domain, and are fused as the texture information and the structure information enhancement weight of the pixel domain;

[0185] Please refer toFigure 10 , define the frequency domain branch, use Shearlet decomposition to extract the high frequency subband and the low frequency subband of the input noise image. Since the low frequency subband contains more structure information, and the high frequency subband contains more noise-like information, the use of the low frequency subband helps the high frequency subband recover texture details.

[0186] Please refer to Figure 11 , define the pixel domain branch, propose a new Deformable Transformer Block (DTB) for local and non-local recovery of images, stack the DTB as a pixel branch model, and in the pixel domain, the noise input is matched with the image reconstructed from the frequency domain branch as the output of the pixel domain branch in the high frequency enhancement dual branch network.

[0187] S302, fuse the results of the frequency domain branch and the pixel domain branch obtained in step S301, enhance the texture information and the low frequency structure information weight feature map obtained by the frequency domain branch by multiplying the denoising results of the pixel domain branch, and then obtain the final result through multiple convolution layers, multiplication operations and residual connection. The high frequency enhancement information obtained by the frequency domain branch is weighted to make the denoising result recover more texture and edge detail information;

[0188] S303, use different general data sets for different image recovery tasks to train the high frequency enhancement dual branch network model constructed above, and use mean square error (MSE) as the loss function; calculate the mean square error of the pixels between the image generated by the generator and the corresponding target image, calculate the gradient and return to adjust the model parameters;

[0189] S304, for the gray image noise removal in the synthetic noise removal task, use different noise levels of additive white Gaussian noise to add to the gray DIV2K training set to train the model, and use the general standard data set Set12, BSD68, gray Urber100 to test the performance of the network model under different noise levels, and use the peak signal-to-noise ratio (PSNR) and the structural similarity parameter (SSIM) as the measurement standard to measure the image recovery effect;

[0190] S305, for the real noise removal task, use the general data set SIDD Medium to train the model, and use the DND, SIDD general test data set to test the performance of the network model, and use the peak signal-to-noise ratio (PSNR) and the structural similarity parameter (SSIM) as the measurement standard to measure the image recovery effect;

[0191] S306, for the JPEG artifact removal task, using general data set DIV2K training set image, and using different quality factors of JPEG compression to construct the training set for training, and using LIVE1, Classic5 general test data set to test the performance of the network model, and using peak signal-to-noise ratio (PSNR) and structural similarity parameter (SSIM) to measure the image restoration effect.

[0192] In another embodiment of the present application, an image restoration system is provided, which can be used to implement the above image restoration method. Specifically, the image restoration system comprises a definition module, a fusion module, a calculation module, a gray module, a real module and an artifact module.

[0193] The definition module defines the frequency domain branch and the pixel domain branch.

[0194] The fusion module fuses the results of the frequency domain branch and the pixel domain branch obtained by the definition module to construct a high-frequency enhancement double-branch network model.

[0195] The calculation module uses different general data sets to train the high-frequency enhancement double-branch network model constructed by the fusion module for different image restoration tasks, and uses mean square error as a loss function; calculates the mean square error of the pixels between the image generated by the generator and the corresponding target image, calculates the gradient and returns to adjust the model parameters.

[0196] The gray module uses different noise levels of additive white Gaussian noise to add to the gray DIV2K training set to train the high-frequency enhancement double-branch network model for gray image noise removal in the synthetic noise removal task, and obtains a synthetic noise removal model under different noise levels, which is applied to the test in the synthetic noise removal task.

[0197] The real module uses the general data set SIDD Medium to train the model for the real noise removal task to obtain a real noise removal model, which is applied to the test in the real noise removal task.

[0198] The artifact module uses the general data set DIV2K training set image to train the training set constructed by different quality factors of JPEG compression for the JPEG artifact removal task to obtain a JPEG artifact removal model under different quality factors, which is applied to the test in the JPEG artifact removal task.

[0199] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0200] The effects of the present application can be further illustrated by the following experiments:

[0201] Experimental conditions

[0202] 1. Experimental database

[0203] The experimental data of the present application is a general data set, and different data sets are used for different tasks, including DIV2K, DND, SIDD, and RealSR training sets. The trained model is tested on the standard test set, and quantitative comparison and visual comparison are made with related methods.

[0204] Synthetic noise removal using data:

[0205] The gray image denoising model in synthetic noise removal uses the gray DIV2K training set, and different levels of additive Gaussian white noise are added as noisy images. The test uses Set12, BSD68 and Urban100 test sets. The color image denoising model uses the color DIV2K training set, and different levels of additive Gaussian white noise are added as noisy images. The test uses Koda24, CBSD68, and Urban100 test sets.

[0206] Real noise removal using data:

[0207] The training set used in the real noise removal model training is the general data set SIDDMedium, and the test set uses the general test set DND and SIDD test set.

[0208] JPEG artifact removal using data:

[0209] The JPEG artifact removal model training uses the general data set DIV2K training set, and different quality factors of JPEG compression are used to generate the data set containing artifacts. The test uses the general data set LIVE1 and Classic5 data set.

[0210] RealSR training set is used for real-world image super-resolution model training, and RealSR test set is used for real-world image super-resolution model testing.

[0211] RealSR training set is used for real-world image super-resolution model training, and RealSR test set is used for real-world image super-resolution model testing.

[0212] Experiment requirements

[0213] The experiments for the three image restoration models in the proposed method include the following parts: synthetic noise removal model training, real noise removal model training, JPEG artifact removal model training, and real-world image super-resolution model training.

[0214] Synthetic noise removal model training:

[0215] In the gray image denoising task, the deformed convolution attention network in this chapter is tested for the effectiveness of removing synthetic noise from gray images damaged by Gaussian white noise. In order to fully verify the denoising ability of the model under different noise levels, this section tests different noise levels (standard deviation σ = 15, 25, 50 and 75). The experimental results are measured using PSNR and SSIM.

[0216] Real noise removal model training:

[0217] Training is performed on the SIDD medium training set, and testing is performed on the SIDD and DND test sets. The performance is measured using PSNR and SSIM.

[0218] JPEG artifact removal model training:

[0219] The DIV2K training set is used in model training (the same settings as the synthetic noise removal task), and the CLASSIC5 and LIVE1 datasets are tested for quality factors of 10, 20, 30, and 40.

[0220] Real-world image super-resolution model training:

[0221] Comparison is performed on RealSR test images with upscaling factors of ×2, ×3, and ×4. Note that all benchmark algorithms are trained on the RealSR dataset for fair comparison (comparison results are also from RealSR). RealSR contains real LR-HR image pairs of the same scene captured by adjusting the camera focal length. In RealSR, the number of training image pairs for scaling factors ×2, ×3, and ×4 is 183, 234, and 178, respectively. In addition, the dataset also provides 30 additional test images for each scaling factor. Here, the Y channel (in YCbCr color space) is used to calculate PSNR and SSIM for the super-resolution method in general.

[0222] Experimental parameter settings

[0223] Deformed Convolutional Attention Network:

[0224] In the experiments, a deformable convolutional attention network was implemented as an end-to-end model using the PyTorch library. During model training, the initial learning rate was... Then gradually decrease to The model uses the Adam optimizer ( , Training was performed for 100 epochs. The batch size was set to 12, and random horizontal and vertical flipping was performed to increase the data. The same settings were used for each image restoration task. When training the streamlined deformable convolutional attention network using the model distillation framework proposed in this chapter, the batch size was set to 5 and 12 for training the teacher and student models, respectively. The remaining settings were the same as when training the deformable convolutional attention network separately. Experiments were conducted on a server equipped with a single NVIDIA Tesla V100 GPU.

[0225] Nonlocal augmentation networks:

[0226] In settings , In this case, the Adam optimizer is used to train the model. The model is trained using an initial learning rate, and then gradually reduced to... During model training, data augmentation (including random horizontal and vertical flipping) was applied to improve performance. The training batch size was set to 6, and the training image size was 256×256. The same settings were used for all four image restoration tasks on an NVIDIA Tesla V100 using the PyTorch library.

[0227] Two-branch network:

[0228] The model was trained using the Adam optimizer, with the following settings: , Set the initial learning rate to Then gradually decrease to To improve performance, data augmentation (including random horizontal and vertical flipping) was employed. The training batch size was set to 6, and the training image size was 256×256. The same settings were used in all four image restoration tasks on an NVIDIA Tesla V100 using the PyTorch library.

[0229] Evaluation criteria for experimental results

[0230] The effects of the three models in different image restoration are quantitatively measured and compared using PSNR and SSIM, and the results are visually compared. Here, the test results are compared by tables and visual comparison to show the effectiveness of the scheme.

[0231] Comparison of experimental results

[0232] Deformed convolution attention network:

[0233] Synthetic noise removal:

[0234] Table 1 Comparison of deformed convolution attention model in synthetic noise removal grayscale image denoising task and similar method results

[0235]

[0236] See Figure 12 , the figure shows the visual comparison of the results of DANet and the latest leading synthetic denoising method, and the best result is highlighted in bold. In the visual comparison of the results, it can be observed that BM3D and DnCNN lose small texture and structure information, and MWCNN, RIDNet and SADNet have blurred edges in the results, while DANet retains the clearest lines closest to the original clean image in the comparison method. Therefore, DANet better restores the structure content and fine detail information of the damaged image, especially in images with rich texture.

[0237] Table 2 Comparison of deformed convolution attention model in synthetic noise removal color image denoising task and similar method results

[0238]

[0239] See Figure 13 , DANet is still better than other comparison methods in terms of visual quality, and DANet and DANet-S can both restore clearer image texture details and will not produce artifacts.

[0240] Real noise removal:

[0241] Table 3 Comparison of deformed convolution attention model in real image denoising task and similar method results

[0242]

[0243] See Figure 14It can be easily observed from the figure that the proposed DANet restores cleaner edges and preserves finer image details compared to other comparison methods. In the figure, the DANet preserves clearer texture details and structural content compared to several state-of-the-art methods in recent years.

[0244] JPEG artifact removal:

[0245] Table 4 Comparison of the results of the deformed convolution attention model in the JPEG artifact removal task with similar methods

[0246]

[0247] Please refer to Figure 15 As can be observed from the visual comparison in the figure, the results of MWCNN and RNAN have over-smoothing problems, while the results of DANet restore more subtle texture and edge details.

[0248] Table 5 Comparison of the results of the deformed convolution attention model in the real-world image super-resolution task with similar methods

[0249]

[0250] Please refer to Figure 16 The visual comparison further verifies the advantages of DANet in restoring structural details. DANet and DANet-S can preserve clearer edges in the restored results, while RCAN and LP-KPN restore blurred edges and edges lacking texture details.

[0251] Non-local enhancement network:

[0252] Synthetic noise removal:

[0253] Table 6 Comparison of the results of the non-local enhancement network model in the synthetic noise removal task in the grayscale image denoising task with similar methods

[0254]

[0255] Please refer to Figure 17 The figure shows a visual comparison analysis of the denoising results of the proposed method and the latest method. From the figure, it can be clearly found that the results of DnCNN lose fine texture detail information. In the visual results of RIDNet and AINDNet, some edges in the restored result images are still blurred. Compared with the latest non-local network COLA-Net, the denoising results appear to lose some lines in the enlarged area shown. While the enhanced non-local network maintains clear lines. Therefore, the enhanced non-local network can reconstruct the structural information and fine texture of the noisy image.

[0256] Realistic noise removal:

[0257] Table 7 Comparison of results of nonlocal augmentation networks with similar methods in real image denoising tasks.

[0258]

[0259] Please see Figure 18 As can be seen from the visual comparison, compared with other comparison methods, NLENet restores a clearer outline and retains more texture details.

[0260] JPEG artifact removal:

[0261] Table 8 compares the results of nonlocal augmentation networks with similar methods in JPEG artifact removal.

[0262]

[0263] Please see Figure 19 The results from DnCNN and RNAN exhibit excessive smoothing in preserving texture details. In the results from MWCNN and the recent COLA-Net network using non-local modules, blurred contours are preserved in the restored images. In contrast, NLENet can preserve finer texture details and sharper edges in the restored images, further validating its superiority.

[0264] Real-world image super-resolution:

[0265] Table 9 Comparison of results of nonlocal augmentation networks with similar methods in real-world image super-resolution tasks.

[0266]

[0267] Please see Figure 20 As can be observed from the figure, NLENet exhibits superior performance among the comparison methods. Compared to state-of-the-art methods in recent years, the PSNR is improved by 0.06. The PSNR was improved by 0.19 dB compared to the classic RCAN and LP-KPN methods, representing a 0.4 dB improvement. 0.5dB. Figures 3 to 9 Visual comparisons in the images further validate NLENet's advantages in recovering clear structural details.

[0268] Two-branch network:

[0269] Synthetic noise removal:

[0270] Table 10 compares the results of high-frequency enhanced dual-branch networks with similar methods in the task of grayscale image denoising in synthetic noise removal.

[0271]

[0272] Please refer to Figure 21 It can be clearly observed from the figure that fine details are missing in the results of DnCNN. In the visual results of SADNet and COLA-Net, the restored images obtain blurred edges and over-smoothed texture details. Therefore, HFENet can preserve more accurate structural information and subtle texture of the original image.

[0273] Real noise removal:

[0274] Table 11 Comparison of high-frequency enhanced dual-branch network in real image denoising task with similar methods

[0275]

[0276] Please refer to Figure 22 It can be found from the visual comparison with other methods that, compared with other methods, HFENet restores cleaner structural edges and preserves more texture information.

[0277] JPEG artifact removal:

[0278] Table 12 Comparison of high-frequency enhanced dual-branch network in JPEG artifact removal task with similar methods

[0279]

[0280] Please refer to Figure 23 It can be clearly observed from the figure that the results of DnCNN and RNAN have over-smoothing problems. In the results of MWCNN and COLA Net, the restored images lack texture details. While HFENet preserves more fine texture details and clear edges in the final result images, further verifying the superiority of HFENet.

[0281] The above listed test results show the effectiveness of the present application in actual situations.

[0282] The present application belongs to the field of image restoration, and relates to three image restoration algorithm models, including a deformation convolution attention network, a non-local enhancement network and a high-frequency enhanced dual-branch network, and will be applied to common image restoration tasks, including synthetic noise removal, real image denoising, compression artifact removal and real image super-resolution as examples to expand model verification and comparison. The three models proposed herein are respectively improved for different directions of the texture and edge missing problems in image restoration:

[0283] First, a deformation convolution attention network is proposed to solve the problem of over-smoothing and texture loss caused by a larger receptive field in a common convolutional neural network, thereby recovering more texture and edge detail information.

[0284] Second, the existing non-local module based on deep learning improves the recovery ability of texture and edge details by extracting the non-local correlation within a single image in view of the characteristics of repeated texture and structural information in the image.

[0285] Third, the module improves the recovery ability of texture and edge details by extracting the non-local correlation within a single image.

[0286] The experimental results comparison and analysis verify that the three proposed models achieve advanced results in multiple image restoration tasks.

[0287] In summary, the image restoration method and system based on local and non-local information and dual-domain fusion are summarized and described.

[0288] The above content only illustrates the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made according to the technical idea of the present application on the basis of the technical scheme falls within the protection scope of the claims of the present application.

Claims

1. An image restoration method characterized by, The method comprises the following steps: S301, defining a frequency domain branch and a pixel domain branch, the frequency domain branch uses Shearlet decomposition to extract high frequency subbands and low frequency subbands of an input noisy image; The pixel domain branch performs local and non-local restoration on the image based on a deformation convolution Transformer module, stacks the deformation convolution Transformer module into a pixel branch model, and in the pixel domain, matches the noisy input with the image reconstructed from the frequency domain branch as the output of the pixel domain branch in the high frequency enhancement double branch network model; S302, fusing the results of the frequency domain branch and the pixel domain branch obtained in step S301 to construct a high frequency enhancement double branch network model, the high frequency enhancement double branch network model enhances the texture information and the structure information by multiplying the texture information and the low frequency structure information weight feature map obtained by the frequency domain branch with the denoising result of the pixel domain branch, and obtains the final result through multiple convolution layers, multiplication operations and residual connection, and the high frequency enhancement information obtained by the frequency domain branch is weighted to make the denoising result restore more texture and edge detail information; S303, training the high frequency enhancement double branch network model constructed in step S302 using different general data sets for different image restoration tasks, taking mean square error as a loss function; calculating the mean square error of pixels between the image generated by the generator and the corresponding target image, calculating the gradient and returning to adjust the model parameters; S304, for the gray image noise removal in the synthetic noise removal task, using different noise levels of additive Gaussian white noise to add to the gray DIV2K training set to train the high frequency enhancement double branch network model, obtaining a synthetic noise removal model under different noise levels, and applying it to the test in the synthetic noise removal task; S305, for the real noise removal task, using the general data set SIDD Medium to train the high frequency enhancement double branch network model to obtain a real noise removal model, and applying it to the test in the real noise removal task; S306, for the JPEG artifact removal task, a training set is constructed by applying different quality factors of JPEG compression to the training set images of the general data set DIV2K, and the high frequency enhancement double branch network model is trained using the training set to obtain a JPEG artifact removal model suitable for different quality factors, and the trained JPEG artifact removal model is applied to the test in the JPEG artifact removal task.

2. An image restoration system characterized by comprising: The method comprises the following steps: defining a frequency domain branch and a pixel domain branch, the frequency domain branch uses Shearlet decomposition to extract high frequency subbands and low frequency subbands of an input noisy image; The pixel domain branch performs local and non-local restoration on the image based on a deformation convolution Transformer module, stacks the deformation convolution Transformer module into a pixel branch model, and in the pixel domain, matches the noisy input with the image reconstructed from the frequency domain branch as the output of the pixel domain branch in the high frequency enhancement double branch network model; The fusion module fuses the results of the frequency domain branch and the pixel domain branch obtained by the definition module to construct a high-frequency enhancement double-branch network model. The high-frequency enhancement double-branch network model enhances the texture information and the structure information by multiplying the texture information obtained by the frequency domain branch and the low-frequency structure information weight feature map with the denoising result of the pixel domain branch. The final result is obtained through multiple convolution layers, multiplication operations and residual connections. The high-frequency enhancement information obtained by the frequency domain branch is weighted to make the denoising result restore more texture and edge detail information. The calculation module trains the high-frequency enhancement double-branch network model constructed by the fusion module using different general data sets for different image restoration tasks, and uses mean square error as a loss function. The calculation generator calculates the mean square error of the pixels between the generated image and the corresponding target image, calculates the gradient and returns to adjust the model parameters. The gray module uses different noise levels of additive white Gaussian noise to add to the gray DIV2K training set to train the high-frequency enhancement double-branch network model for gray image noise removal in the synthetic noise removal task, and obtains a synthetic noise removal model under different noise levels. It is applied to the test in the synthetic noise removal task. The real module uses the general data set SIDD Medium to train the high-frequency enhancement double-branch network model for real noise removal tasks, and obtains a real noise removal model. It is applied to the test in the real noise removal task. The artifact module uses the general data set DIV2K training set image to apply different quality factors of JPEG compression to construct a training set, and uses the training set to train the high-frequency enhancement double-branch network model for JPEG artifact removal tasks. A JPEG artifact removal model suitable for different quality factors is obtained. The trained JPEG artifact removal model is applied to the test in the JPEG artifact removal task.

Images