Image restoration method and system based on gated convolution and fused gated attention mechanism

Abstract

The invention discloses an image restoration method and system based on gated convolution and a fused gated attention mechanism, and belongs to the field of image super-resolution restoration frameworks, and the method comprises the steps: taking a preprocessed image as the input of an image restoration model, carrying out the feature extraction and dimension raising, and obtaining a feature map; the image features are extracted by fusing the attention and self-attention mechanism of the convolutional neural network type, and iterative optimization is carried out by using the loss function, so that the local and global features of the image can be processed more effectively, the image reconstruction quality is improved, and the authenticity and detail richness of the high-resolution image are ensured at the same time.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image restoration method and system based on gated convolution and fused gated attention mechanisms. Background Art

[0002] With the rapid development of science and technology, images have become an indispensable part of people's daily lives and professional fields. Whether it is sharing pictures on social media or image applications in professional fields such as medical imaging and remote sensing detection, high-quality images are essential. However, due to various reasons, such as device limitations, transmission bandwidth, storage capacity, etc., images often suffer from quality degradation, resolution reduction, and information loss during transmission and storage. For this reason, image super-resolution technology came into being and became an important tool to solve these problems.

[0003] Image super-resolution is a technology that converts low-resolution images into high-resolution images. Its purpose is to reconstruct the high-frequency details of the image through computational methods, thereby improving the visual effect and information content of the image. Traditional image super-resolution methods mainly rely on interpolation techniques, such as bilinear interpolation and bicubic interpolation. These methods are simple and fast, but often cannot restore the high-frequency details of the image, and only smooth the image to a certain extent.

[0004] In practical applications, image super-resolution technology has shown great potential and value. In the field of medical imaging, it can help doctors obtain clearer diagnostic images and improve the accuracy of disease diagnosis. In security monitoring, super-resolution technology can improve the quality of monitoring images and help better perform facial recognition and behavior analysis. In digital entertainment and film and television production, high-resolution images can provide viewers with a more immersive and realistic viewing experience. In addition, super-resolution technology has also demonstrated its important application value in the restoration of old photos, the digitization of historical archives, and the restoration and protection of artworks.

[0005] However, despite the remarkable progress made in deep learning-based super-resolution technology, it still faces many challenges in practical applications. For example, when dealing with complex backgrounds or high-resolution images, current technologies often have difficulty in effectively reconstructing details and textures in images, and are prone to blurring, overlapping, and artifacts. In addition, ensuring the semantic consistency of the restored image so that it is close to the real scene is also a difficulty in current research. Summary of the invention

[0006] The purpose of the present invention is to provide an image restoration method and system based on gated convolution and fused gated attention mechanism, which not only improves the image reconstruction quality, but also ensures the authenticity and detail richness of high-resolution images.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions: The image restoration method based on gated convolution and fusion gated attention mechanism includes the following steps: Image preprocessing; The preprocessed image is used as the input of the image restoration model to extract features and upgrade the dimension to obtain a feature map; Use an efficient channel attention module and a convolutional window self-attention module to extract detailed features of the feature map; Perform image upsampling, reduce the dimension of the feature map extracted by the hybrid attention module and upsample it to obtain a super-resolution image; The loss function is used to calculate the loss value of the generated image, and the image restoration model is iteratively optimized; The trained image restoration model is used to extract and restore image features to complete the image super-resolution task.

[0008] Furthermore, image preprocessing includes: Rotate and mirror the input image to perform data augmentation; The image is then cut into small pieces for easier training.

[0009] Furthermore, for a given input feature , the processing process of the efficient channel attention module is: The input features are processed by a depthwise separable convolution module, and the output of the depthwise separable convolution module is residually connected to the input: Then through the GELU activation function: Then, after a channel attention module and a convolutional block, the input and output of the channel attention module are residually connected, and the input and output of the convolutional block are also residually connected: in, represents depthwise separable convolution, represents the GELU activation function, represents channel attention, represents the convolutional block, , , represents the intermediate feature, Represents the output of the efficient channel attention module.

[0010] Furthermore, the feature extraction of the convolutional window self-attention module includes the following steps: Normalize the input features: The normalized input image is divided into small windows, and self-attention operation is performed inside the local window. The output of each window is residually connected with the original input: Finally, after the convolution block, the output of the convolution block is again residually connected to the input: in, represents normalization, represents the window self-attention mechanism, represents the convolutional block, and They represent the intermediate features after layer normalization and window self-attention mechanism processing, is the output of the convolutional window self-attention module.

[0011] Furthermore, the loss function adopts the L1 loss function: in, represents the super-resolution reconstructed image, represents a high-resolution real image, Represents the norm.

[0012] Furthermore, the test steps are also included: The test is performed using a standard test set, and the test results are analyzed qualitatively and quantitatively. The quantitative analysis includes PSNR and SSIM indicators, and the qualitative analysis includes visual analysis.

[0013] Image restoration system based on gated convolution and fused gated attention mechanism, including image preprocessing module, image restoration model training module, and image restoration model testing module; The image restoration model training module includes an efficient channel attention module and a convolutional window self-attention module. The efficient channel attention module includes a depth-separable convolutional layer, a channel attention module and a convolutional block. The convolutional window self-attention module includes a normalization layer, a window self-attention module, a large-size deep convolutional layer and two small-size convolutional layers.

[0014] The advantages of the present invention are: The efficient channel attention module greatly reduces the required computing resources and model parameters by using deep separable convolution and channel attention mechanism, making the network lighter and accelerating processing speed. Its design can effectively extract image features, especially when processing information from different channels. The channel attention module can extract detailed features by assigning weights to different channels, which is particularly beneficial when applied to deep separable convolution because it isolates information between channels. Most importantly, the efficient channel attention module has only extremely low computational complexity, which can reduce the computational complexity of the model.

[0015] The self-attention mechanism usually behaves as a low-pass filter during training reasoning, while convolution usually behaves as a high-pass filter. Therefore, the use of the self-attention mechanism alone may cause the loss of high-frequency details. This is a big problem in image restoration. Therefore, the present invention adopts a convolution window self-attention mechanism to innovatively integrate local and global information processing capabilities. This hybrid structure improves the network's ability to capture details and contextual information in the image. It can capture global context information using the window self-attention mechanism, while extracting local textures and details through large convolution kernels, which is particularly important in high-resolution image reconstruction. Adding a convolution layer with a large-size convolution kernel can not only effectively extract high-frequency information, but also increase the receptive field of the model. And by using two small-size convolutions instead of multi-layer perceptrons, the Reshape of data during the forward propagation of the network can be avoided, thereby reducing IO overhead and improving the reasoning speed of the model.

[0016] In summary, the HASwin image super-resolution model proposed in this paper combines the attention and self-attention mechanisms of the convolutional neural network type. This hybrid attention mechanism can more effectively process the local and global features of the image and improve the quality and details of the reconstructed image. HASwin effectively reduces the computational complexity by using window partitioning in the self-attention module. Compared with the traditional global self-attention mechanism, it can reduce the computational burden while maintaining a large range of information processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the image restoration method based on gated convolution and fusion gated attention mechanism of the present invention; Figure 2 This is a structural diagram of the image restoration system based on gated convolution and fused gated attention mechanism of the present invention; Figure 3 Schematic diagram of an efficient self-attention module constructed in the present invention; Figure 4 A schematic diagram of a convolutional window self-attention module is constructed for the present invention. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0019] This embodiment proposes an image restoration system based on gated convolution and fusion gated attention mechanism. Please refer to Figure 2 ,include: The image preprocessing module is configured to preprocess the dataset, perform data augmentation by rotating and mirroring the input image to improve the robustness of the model, and finally cut the image into 48×48 blocks for training.

[0020] The image restoration model training module is configured to: input the image data set into the HASwin image super-resolution model, and use the HASwin network for training to obtain the trained restored image and the pre-trained model.

[0021] The image restoration model testing module is configured to: input the test image data set into the trained HASwin image super-resolution model, and obtain the super-resolution restored image in a relatively short time.

[0022] This example proposes an image restoration method based on gated convolution and fusion gated attention mechanism. Figure 1 , including the steps of: Image preprocessing,Before inputting the data into the neural network, the DF2K dataset needs to be preprocessed, such as augmentation and cropping.,Since the images in the dataset are of 2K resolution, it is difficult to train directly.,Thus, the images are first cropped into overlapping image blocks of 480×480, and then,after mirror flipping and random rotation of 90°, 180°, or 270°, 48×48 image blocks are randomly cropped from them for training.

[0023] The image restoration model training takes the preprocessed image as input. The shallow feature extraction module performs preliminary feature extraction on the input image and increases the image dimension to facilitate the subsequent extraction of deep high-dimensional features. The hybrid attention module is used to extract detailed features from the feature map obtained previously, and the channel attention mechanism and the global attention mechanism are used to mix and extract more comprehensive features: (1) Constructing an efficient self-attention module The efficient channel attention module is part of the hybrid attention HASwin proposed for efficient image super-resolution. It aims to reduce the amount of computation and the number of parameters while maintaining quantitative performance. It integrates channel attention into the model, allowing the model to focus on feature maps that contribute significantly to the reconstruction process, thereby compensating for the performance loss caused by the lightweight design.

[0024] Specifically, the efficient channel attention module consists of a 3×3 depthwise separable convolution, followed by a channel attention module and a convolution block. This design aims to reduce the computational workload and improve performance by focusing on important features. The channel attention mechanism is particularly effective when using depthwise separable convolutions because depthwise separable convolutions isolate information in different channels. Through its efficient design, the efficient channel attention module significantly reduces the computational workload by more than 50% compared to the convolutional window self-attention module, making it a lightweight and effective component in HASwin.

[0025] Given input features , the processing process of the efficient channel attention module is as follows: The input features are processed by a depthwise separable convolution module, and the output of the depthwise separable convolution module is residually connected to the input: Then through the GELU activation function: Then, after a channel attention module and a convolutional block, the input and output of the channel attention module are residually connected, and the input and output of the convolutional block are also residually connected: in, represents depthwise separable convolution, represents the GELU activation function, represents channel attention, represents the convolutional block, , , represents the intermediate feature, Represents the output of the efficient channel attention module. This module is designed to reduce computational cost and improve performance.

[0026] (2) Constructing a convolutional window self-attention module The convolutional window self-attention module is a network module specially designed for image super-resolution tasks, which combines the self-attention mechanism with the convolution operation. This structure is designed to more effectively process high-frequency details and complex spatial relationships in images, which is crucial for image super-resolution.

[0027] First, the input features are layer normalized to stabilize the training process and speed up convergence. Layer normalization is achieved by normalizing the mean and variance of the input features. Secondly, the window self-attention mechanism is an optimization of the traditional self-attention mechanism. It divides the input image into smaller windows and performs self-attention operations inside these local windows. This can reduce the computational cost while maintaining the effectiveness of self-attention. The output of each window is subsequently residually connected to the original input to retain the original feature information. Finally, the core part of the module is the convolution block, which contains a large-size deep convolution layer (such as 7x7) and two 1x1 convolution layers. Large-size convolution kernels help expand the receptive field of the network to capture richer spatial information, while the 1x1 convolution layer integrates features in the channel dimension. Finally, the output of the convolution block is residually connected to the input again. It can be expressed as: Normalize the input features: The normalized input image is divided into small windows, and self-attention operation is performed inside the local window. The output of each window is residually connected with the original input: Finally, after the convolution block, the output of the convolution block is again residually connected to the input: in, represents normalization, represents the window self-attention mechanism, represents the convolutional block, and They represent the intermediate features after layer normalization and window self-attention mechanism processing, is the output of the convolutional window self-attention module.

[0028] (3) Building an image restoration network HASwin is a deep learning model designed for image super-resolution that combines the advantages of convolutional neural networks and self-attention mechanisms. The construction process of HASwin covers the entire process from preliminary feature extraction to detail recovery. The model starts with a shallow module, usually containing convolutional layers, to capture the basic visual features of the input low-resolution image. Then, the hybrid attention stage processes these features by fusing channel attention and self-attention mechanisms, allowing the model to emphasize channel features that are more important to the reconstruction process and capture global information and local details at the same time. The features extracted and processed by the hybrid attention stage are upgraded to high resolution through an upsampling module, which usually involves transposed convolution or pixel reordering techniques. The entire network is usually trained using the L1 loss function to optimize the performance of the model and obtain clearer image edges. The design of HASwin achieves high efficiency and effectiveness in image super-resolution tasks. The specific steps for training and testing the built image restoration network are as follows: 800 high-resolution images were selected from the public dataset DIV2K to train the HASwin network. The training samples were expanded through data augmentation techniques such as random cropping, flipping, and rotation. During the training process, the batch size was set to 32. The entire network was trained 500,000 times. The training equipment configuration includes: CPU is AMD R9-5900X, GPU is NVIDIA RTX3090, memory is DDR4-3200-128G, and the code runs under the Pytorch deep learning framework. The loss function of the HASwin network includes the L1 loss function, The L1 loss function is defined as: in, represents the super-resolution reconstructed image, represents a high-resolution real image, Represents the norm. This loss function helps the model reconstruct clearer image edges in super-resolution tasks because it emphasizes pixel-level accuracy.

[0029] The trained HASwin model is tested, mainly using standard test sets such as Set5, Set14, BSD100, Urban100 and Manga109. The test results will be evaluated through qualitative analysis, such as visual comparison, and quantitative analysis, such as using indicators such as PSNR and SSIM, to evaluate the performance of the model in the image super-resolution task. Such an evaluation not only shows the effectiveness of the HASwin model, but also demonstrates its superior performance in processing high-resolution images. A person of ordinary skill in the art can implement the present invention in the form of electronic hardware or a combination of computer software and electronic hardware through a detailed description of the system modules of the present invention and in combination with the examples described in this embodiment. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0030] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image restoration method based on gated convolution and fusion gated attention mechanism, characterized in that: Includes steps: Image preprocessing; The preprocessed image is used as the input of the image restoration model to extract features and upgrade the dimension to obtain a feature map; Use an efficient channel attention module and a convolutional window self-attention module to extract detailed features of the feature map; Perform image upsampling, reduce the dimension of the feature map extracted by the hybrid attention module and upsample it to obtain a super-resolution image; The loss function is used to calculate the loss value of the generated image, and the image restoration model is iteratively optimized; The trained image restoration model is used to extract and restore image features to complete the image super-resolution task.

2. The image restoration method based on gated convolution and fusion gated attention mechanism according to claim 1, characterized in that: The image preprocessing comprises: Rotate and mirror the input image to perform data augmentation; The image is then cut into small pieces for easier training.

3. The image restoration method based on gated convolution and fusion gated attention mechanism according to claim 1, characterized in that: For a given input feature , the processing process of the efficient channel attention module is: The input features are processed by a depthwise separable convolution module, and the output of the depthwise separable convolution module is residually connected to the input: Then through the GELU activation function: Then, after a channel attention module and a convolutional block, the input and output of the channel attention module are residually connected, and the input and output of the convolutional block are also residually connected: in, represents depthwise separable convolution, represents the GELU activation function, represents channel attention, represents the convolutional block, , , represents the intermediate feature, Represents the output of the efficient channel attention module.

4. The image restoration method based on gated convolution and fusion gated attention mechanism according to claim 1, characterized in that: The convolution window self-attention module extracts features including the steps of: Normalize the input features: The normalized input image is divided into small windows, and self-attention operations are performed inside the local windows. The output of each window is residually connected to the original input: Finally, after the convolution block, the output of the convolution block is again residually connected to the input: in, represents normalization, represents the window self-attention mechanism, represents the convolutional block, and They represent the intermediate features after layer normalization and window self-attention mechanism processing, is the output of the convolutional window self-attention module.

5. The image restoration method based on gated convolution and fusion gated attention mechanism according to claim 1, characterized in that: The loss function adopts the L1 loss function: in, represents the super-resolution reconstructed image, represents a high-resolution real image, Represents the norm.

6. The image restoration method based on gated convolution and fusion gated attention mechanism according to claim 1, characterized in that: Also includes the test steps: The test is performed using a standard test set, and the test results are analyzed qualitatively and quantitatively. The quantitative analysis includes PSNR and SSIM indicators, and the qualitative analysis includes visual analysis.

7. An image restoration system based on gated convolution and fused gated attention mechanism, characterized in that: It includes image preprocessing module, image restoration model training module and image restoration model testing module; The image restoration model training module includes an efficient channel attention module and a convolutional window self-attention module. The efficient channel attention module includes a depth-separable convolutional layer, a channel attention module and a convolutional block. The convolutional window self-attention module includes a normalization layer, a window self-attention module, a large-size deep convolutional layer and two small-size convolutional layers.

Images