A single image super-resolution method based on dense multi-scale pyramid gated attention

Abstract

The present invention discloses a single image super-resolution method based on dense multi-scale pyramid gated attention. The method extracts shallow features of an image through a convolution kernel of size 3*3. Then, two dense networks are used as backbone networks to extract deep features of the image, wherein each dense network contains 12 multi-scale pyramid extraction modules and 12 gated attention distillation modules. The multi-scale pyramid extraction module is used to capture and aggregate richer features through pyramid pooling and multi-scale operations, so that the generated model can effectively restore the HR image. The gated attention distillation module is used to focus on key areas while reducing redundant information to improve the convergence of the model. Finally, the image restoration module is used to restore the features extracted by the multi-scale pyramid extraction module and the gated attention distillation module to a high-resolution image. The method of the present invention has the characteristics of simple execution, rich generated image details and wide applicability.

Description

Technical Field

[0001] The present invention belongs to the field of super-resolution, and in particular relates to a single image super-resolution method based on dense multi-scale pyramid gated attention. Background Art

[0002] With the continuous development of modern society, people's requirements for the quality of information they obtain are constantly increasing. Vision, as the main way for people to obtain information, plays an irreplaceable role in daily work and study. With the continuous advancement of science and technology, computer vision technology can basically meet people's needs for a large amount of information. As the main research object of computer vision, images can present a variety of visual information. Therefore, in various fields such as astronomy, meteorology, medicine and biology, images can be reasonably processed to make them better present the information people need, meet people's various needs, and better carry out various tasks.

[0003] Image super-resolution is a research hotspot in the field of image processing. Deep learning has been widely used in single image super-resolution, which aims to restore an accurate high-resolution image from its low-resolution image, making the image clearer and richer in details. However, single image super-resolution is a seriously ill-posed problem because multiple high-resolution images can be generated from one low-resolution image. Common single image super-resolution includes traditional interpolation methods, such as: nearest neighborhood interpolation, bilinear interpolation, and bicubic interpolation. (Yang W, Zhou F, Zhu R, et al. Deep learning for image super-resolution [J]. Neurocomputing, 2019.). These algorithms establish a continuous function based on discrete points and use the established function to calculate the pixel value of any interpolation point. It does not work well in intersection areas (such as edges) and is prone to jagged and blurry. There are also methods based on deep learning, such as: convolutional neural networks, recurrent neural networks, and Transformer.(Dong C, Loy CC, He K, et al.Image super-resolution using deep convolutional networks[J].IEEE transactions on pattern analysis and machine intelligence,2015,38(2):295-307; Kim J,Lee JK,Lee K M.Accurate image super-resolution using verydeep convolutional networks[C] / / Proceedings of the IEEE conference oncomputer vision and pattern recognition.2016:1646-1654;Kim J,Lee JK,Lee KM.Deeply-recursive convolutional network for image super-resolution[C] / / Proceedings of the IEEE conference on computer vision and patternrecognition.2016:1637-1645;Li Z,Yang J,Liu Z,et al.Feedback network for imagesuper-resolution[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2019:3867-3876;Chen X,Wang X,Zhou J,etal.Activating More Pixels in Image Super-Resolution Transformer[J].arXivpreprint arXiv:2205.04437,2022.). These algorithms ignore the interaction between deep features and shallow features, resulting in smooth texture of image details. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the background technology and propose a single image super-resolution method based on dense multi-scale pyramid gated attention. The method uses a dense network as the backbone network, and embeds 12 multi-scale pyramid extraction modules and 12 gated attention distillation modules in each dense network. It is used to overcome the problem that the shallow features and deep features do not interact closely, resulting in smooth texture of image details.

[0005] The technical solution adopted by the present invention is: a single image super-resolution method based on dense multi-scale pyramid gated attention, characterized in that: the method comprises the following steps:

[0006] Step 1: Select DIV2K as the training set of the dataset and use the bicubic interpolation method to create a low-resolution image I LR , that is, for all high-resolution images I in the training set HR Perform bicubic interpolation with downscaling factors of 2.0, 3.0, and 4.0 to achieve 2x, 3x, and 4x the corresponding low-resolution image I LR ;

[0007] Step 2: Use data enhancement technology to enhance the training set obtained in step 1, that is, to obtain the low-resolution image I LR and the corresponding high-resolution image I HR At the same time, random rotation of 90°, 180°, 270° and horizontal flipping are performed to enhance data diversity;

[0008] Step 3: Put the I made in step 2 LR After the training set is randomly cropped to a size of 64×64, it is input into the shallow feature extraction module in the dense multi-scale pyramid gated attention network for shallow feature extraction to obtain I LR The shallow features F of the image SF , where the shallow feature extraction module consists of a 3×3 convolutional layer, and its process can be expressed as:

[0009] F SF =Conv 3*3 (I LR ) (1)

[0010] Among them, Conv 3*3 (·) indicates a convolution kernel of size 3*3;

[0011] Step 4, get I LR The shallow features F SF After that, F SFThe deep feature extraction module is input to perform deep feature extraction. The deep feature extraction module is composed of two dense gated convolution modules connected in series. Each dense gated convolution module uses a dense network to connect 12 multi-scale pyramid extraction modules and 12 gated attention distillation modules. The process can be expressed as:

[0012] F DF =D2(D1(F SF )) (2)

[0013] Among them, F DF Indicates I LR The deep features of the image, D i (·) represents the i-th dense gated convolution module;

[0014] Assumption F i，j and denote the input of the j-th multi-scale pyramid extraction module and the output of the j-1-th gated attention distillation module in the i-th dense gated convolution module, respectively. Then F i，j and It can be expressed as:

[0015]

[0016] Among them, P i，j (·) and G i，j (·) represents the j-th multi-scale pyramid extraction module and gated attention distillation module in the i-th dense gated convolution module, and Cat(·) represents the channel merging operation;

[0017] Among them, the scale pyramid extraction module (PMEB: Pyramid Multi-scale Extraction Block) contains three modules: pyramid pooling, multi-scale feature extraction and multi-layer perceptron (MLP: Multilayer Perceptron). Pyramid pooling consists of two maximum pooling and average pooling with a convolution kernel size of 3*3, multi-scale feature extraction consists of multiple convolution layers with convolution kernel sizes of 1*1, n*1 and 1*n, and multi-layer perceptron (MLP) consists of two convolution layers with a convolution kernel size of 1*1, GELU activation function and LN normalization.

[0018] Assume that pg i，j Represents the output of the j-th multi-scale pyramid extraction module in the i-th dense gated convolution module and the input of the gated attention distillation module, then the j-th multi-scale pyramid extraction module in the i-th dense gated convolution module can be expressed as:

[0019]

[0020] Among them, PM (·) indicates pg i，j Enter the intermediate features obtained in the multi-scale pyramid extraction module, Conv 1*1 (·), Conv n*1 (·) and Conv 1*n (·) denotes the convolution layers with kernel sizes of 1*1, n*1, and 1*n, respectively. 3*3 (·) indicates the maximum pooling operation with a convolution kernel size of 3*3, A 3*3 (·) represents the average pooling operation with a convolution kernel size of 3*3, GELU(·) is the activation function, and LN(·) is the layer normalization operation;

[0021] The gated attention distillation block (GADB) contains two branches: channel attention mechanism extraction and spatial attention extraction mechanism. i，j The input is sent to the jth gated attention distillation module in the i-th dense gated convolution module, and the channel is doubled through a convolution layer with a convolution kernel size of 1*1, and then the channel is split into intermediate features A1 and A2 by channel:

[0022] A1,A2=Split(Conv 1*1 (p i,j )) (5)

[0023] Among them, Split(·) is a channel splitting operation, which can divide the feature into two parts equally according to the channel;

[0024] Then A1 and A2 are input into the channel attention mechanism H respectively AC and spatial attention mechanism H AS Extract channel attention and spatial attention features A C and A S , then for A C and A S Perform multiplication to obtain the output of the jth gated attention distillation module in the i-th dense gated convolution module

[0025]

[0026] Among them, H AC (·) and H AS (·) represent the channel attention mechanism and spatial attention mechanism respectively, and Mul(·) is the multiplication operation;

[0027] Step 5: I LR and its shallow features F SF and deep features FDF Input image restoration module, I LR Restore to I SR ,

[0028] I SR =H up (Conv 3*3 (F SF +F DF ))+H up (I LR ) (7)

[0029] Among them, H up (·) indicates an upsampling operation using bilinear interpolation;

[0030] The present invention effectively improves the detail texture features of the generated image, and compared with methods such as SWINIR and A2N, the PSNR is improved by 0.8 dB, and can effectively solve the problem of smooth detail texture of the generated image.

[0031] The present invention has the following advantages and positive effects:

[0032] (1) This paper proposes a pyramid multi-scale extraction block to capture and aggregate richer features to enhance the modeling ability of the network. Through pyramid pooling and multi-scale operations, the generated model can effectively restore high-resolution images.

[0033] (2) The present invention designs a gated attention distillation module to focus on key areas while reducing redundant information and improving the convergence of the model.

[0034] (3) The network model proposed in the present invention can obtain better results on various commonly used test data sets, which proves the superiority of the performance of the proposed model.

[0035] This method can be widely used in related computer vision tasks such as semantic segmentation, image classification, object detection and instance segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the architecture of DGISR (Dense-Gated Network) of an embodiment of the present invention, which mainly includes a shallow feature extraction module, a deep feature extraction module and an image reconstruction module, wherein the deep feature extraction module is composed of two dense gated convolution modules (DGCM).

[0037] Figure 2 Schematic diagram of the dense gated convolution module (DGCM), which connects 12 multi-scale pyramid extraction modules and 12 gated attention distillation modules through a dense network.

[0038] Figure 3 Schematic diagram of the Pyramid Multi-scale Extraction Block (PMEB), which includes three modules: pyramid pooling, multi-scale operation and MLP (Multilayer Perceptron).

[0039] Figure 4 Schematic diagram of the gated attention distillation block (GADB), which includes two branches: channel attention mechanism extraction and spatial attention extraction mechanism.

[0040] Figure 5 It is a general schematic diagram of the method of the present invention.

[0041] Figure 6 It is a comparison diagram of the effect images of the embodiments of the present invention. DETAILED DESCRIPTION

[0042] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0043] In order to overcome the limitations of the prior art, the present invention proposes a single image super-resolution method based on dense multi-scale pyramid gated attention. This method uses a dense network as the backbone network, and connects 12 multi-scale pyramid extraction modules and 12 gated attention distillation modules through the dense network. Due to computing power limitations, the server uses a maximum of 12 multi-scale pyramid extraction modules and 12 gated attention distillation modules. More multi-scale pyramid extraction modules and gated attention distillation modules can be added if the equipment permits, and the number of scale pyramid extraction modules and gated attention distillation modules used is proportional to the experimental results. Compared with traditional methods, this method can effectively combine shallow and deep image features. Overcome the problem of loose interaction between shallow features and deep features, so that the image details and textures of the generated image are richer.

[0044] See also Figure 1 ,The embodiment of the present invention is a single image super resolution method based on ,dense multi-scale pyramid gated attention, and the specific calculation steps are as follows.

[0045] Step 1: Use bicubic interpolation method to create low-resolution image I LR That is, for all high-resolution images I in the dataset HRPerform bicubic interpolation with downscaling factors of 2.0, 3.0, and 4.0 to achieve 2x, 3x, and 4x the corresponding low-resolution image I LR .

[0046] Step 2: Use data enhancement technology to enhance the obtained data set. That is, the obtained low-resolution image I LR and the corresponding high-resolution image I HR At the same time, random rotation of 90°, 180°, 270° and horizontal flipping are performed to enhance data diversity.

[0047] Step 3: LR After the dataset is randomly cropped to a size of 64×64, it is input into the shallow feature extraction part of the dense multi-scale pyramid gated attention network for shallow feature extraction to obtain I LR The shallow features F of the image SF The shallow feature extraction module consists of a 3×3 convolutional layer. The process can be expressed as:

[0048] F SF =Conv 3*3 (I LR ) (1)

[0049] Among them, Conv 3*3 (·) represents a convolution kernel of size 3*3.

[0050] Step 4, get I LR The shallow features F SF After that, F SF The deep feature extraction module is input to perform deep feature extraction. The deep feature extraction module consists of two dense gated convolution modules, each of which uses a dense network to connect 12 multi-scale pyramid extraction modules and 12 gated attention distillation modules. The process can be expressed as:

[0051] F DF =D2(D1(F SF )) (2)

[0052] Among them, F DF Indicates I LR The deep features of the image, D i (·) denotes the i-th dense gated convolutional module.

[0053] Assumption F i，j and denote the input of the j-th multi-scale pyramid extraction module and the output of the j-1-th gated attention distillation module in the i-th dense gated convolution module, respectively. Then F i，j and It can be expressed as:

[0054]

[0055] Among them, P i，j (·) and G i，j (·) denotes the j-th multi-scale pyramid extraction module and gated attention distillation module in the i-th dense gated convolution module. Cat(·) denotes the channel merging operation.

[0056] Assume that pg i，j Represents the output of the j-th multi-scale pyramid extraction module in the i-th dense gated convolution module and the input of the gated attention distillation module, then the j-th multi-scale pyramid extraction module in the i-th dense gated convolution module can be expressed as:

[0057]

[0058] Among them, P M (·) indicates pg i，j Enter the intermediate features obtained in the multi-scale pyramid extraction module. Conv 1*1 (·), Conv n*1 (·) and Conv 1*n (·) denotes convolutional layers with kernel sizes of 1*1, n*1, and 1*n, respectively. 3*3 (·) indicates the maximum pooling operation with a convolution kernel size of 3*3. 3*3 (·) represents the average pooling operation with a convolution kernel size of 3*3. GELU(·) is the activation function, and LN(·) is the layer normalization operation.

[0059] pg i，j The input to the jth gated attention distillation module in the i-th dense gated convolution module is first doubled in channels by a convolution layer with a convolution kernel size of 1*1, and then divided into intermediate features A1 and A2 by channel:

[0060] A1,A2=Split(Conv 1*1 (p i,j )) (5)

[0061] Among them, Split(·) is a channel splitting operation, which can divide the features into two parts equally according to the channel.

[0062] Then A1 and A2 are input into the channel attention mechanism H respectively AC and spatial attention mechanism H AS Extract channel attention and spatial attention features A C and A S Then for A C and A SPerform multiplication to obtain the output of the jth gated attention distillation module in the i-th dense gated convolution module

[0063]

[0064] Among them, H AC (·) and H AS (·) represents the channel attention mechanism and spatial attention mechanism respectively, and Mul(·) is the multiplication operation.

[0065] Step 5: I LR and its shallow features F SF and deep features F DF Input image restoration module, I LR Restore to I SR .

[0066] I SR =H up (Conv 3*3 (F SF +F DF ))+H up (I LR )(7)

[0067] Among them, H up (·) indicates an upsampling operation using bilinear interpolation.

[0068] The present invention effectively improves the detail texture features of the generated image, and compared with methods such as SWINIR and A2N, the PSNR is improved by 0.8 dB, and can effectively solve the problem of smooth detail texture of the generated image.

[0069] See also Figure 6 In order to illustrate the effectiveness of the present invention in super-resolution magnification of image detail texture, the single image super-resolution method based on dense multi-scale pyramid gated attention used in the present invention is compared with the VDSR, DRRN, SWINIR and A2N methods, as described below:

[0070] from Figure 6 It can be seen from the figure that the single image super-resolution method based on dense multi-scale pyramid gated attention used in the present invention is richer and more specific in the detail texture of the image. It can better obtain high-resolution rich texture details in areas with poor detail texture of low-resolution images.

[0071] During specific implementation, the present invention can use computer software technology to realize automatic operation process.

[0072] It should be understood that the above description of the preferred embodiment is relatively detailed, and it cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions and modifications without departing from the scope protected by the patent requirements of the present invention, all of which fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A single image super-resolution method based on dense multi-scale pyramid gated attention, Features: The method comprises the following steps: Step 1: Select DIV2K as the training set of the dataset and use the bicubic interpolation method to create a low-resolution image I LR , that is, for all high-resolution images I in the training set HR Perform bicubic interpolation with downscaling factors of 2.0, 3.0, and 4.0 to achieve 2x, 3x, and 4x the corresponding low-resolution image I LR ; Step 2: Use data enhancement technology to enhance the training set obtained in step 1, that is, to obtain the low-resolution image I LR and the corresponding high-resolution image I HR At the same time, random rotation of 90°, 180°, 270° and horizontal flipping are performed to enhance data diversity; Step 3: Put the I made in step 2 LR After the training set is randomly cropped to a size of 64×64, it is input into the shallow feature extraction module in the dense multi-scale pyramid gated attention network for shallow feature extraction to obtain I LR The shallow features F of the image SF , where the shallow feature extraction module consists of a 3×3 convolutional layer, and its process is expressed as: F SF =Conv 3*3 (I LR ) (1) Among them, Conv 3*3 (·) indicates a convolution kernel of size 3*3; Step 4, get I LR The shallow features F SF After that, F SF The deep feature extraction module is input to perform deep feature extraction. The deep feature extraction module is composed of two dense gated convolution modules connected in series. Each dense gated convolution module uses a dense network to connect 12 multi-scale pyramid extraction modules and 12 gated attention distillation modules. The process is expressed as follows: F DF =D 2 (D 1 (F SF )) (2) Among them, F DF Indicates I LR The deep features of the image, D i (·) represents the i-th dense gated convolution module; Assumption F i，j and denote the input of the j-th multi-scale pyramid extraction module and the output of the j-1-th gated attention distillation module in the i-th dense gated convolution module, respectively. Then F i，j and It is expressed as: Among them, P i，j (·) and G i，j (·) represents the j-th multi-scale pyramid extraction module and gated attention distillation module in the i-th dense gated convolution module, and Cat(·) represents the channel merging operation; Among them, the multi-scale pyramid extraction module PMEB includes three modules: pyramid pooling, multi-scale feature extraction and multi-layer perceptron MLP; pyramid pooling consists of two maximum pooling and average pooling with a convolution kernel size of 3*3, multi-scale feature extraction consists of multiple convolution layers with convolution kernel sizes of 1*1, n*1 and 1*n, and multi-layer perceptron MLP consists of two convolution layers with a convolution kernel size of 1*1, GELU activation function and LN normalization; Assume that pg i，j represents the output of the j-th multi-scale pyramid extraction module in the i-th dense gated convolution module and the input of the gated attention distillation module, then the j-th multi-scale pyramid extraction module in the i-th dense gated convolution module is expressed as: Among them, P M (·) indicates pg i，j Enter the intermediate features obtained in the multi-scale pyramid extraction module, Conv 1*1 (·), Conv n*1 (·) and Conv 1*n (·) denotes the convolution layers with kernel sizes of 1*1, n*1, and 1*n, respectively. 3*3 (·) indicates the maximum pooling operation with a convolution kernel size of 3*3, A 3*3 (·) represents the average pooling operation with a convolution kernel size of 3*3, GELU(·) is the activation function, and LN(·) is the layer normalization operation; Among them, the gated attention distillation module GADB contains two branches: channel attention mechanism extraction and spatial attention extraction mechanism; first, pg i，j The input is sent to the jth gated attention distillation module in the i-th dense gated convolution module, and the channel is doubled through a convolution layer with a convolution kernel size of 1*1, and then the channel is split into intermediate features A by channel. 1 and A 2 : A 1 ,A 2 =Split(Conv 1*1 (p i,j )) (5) Among them, Split(·) is a channel splitting operation, which divides the feature into two parts equally according to the channel; Then A 1 and A 2 Input channel attention mechanism H respectively AC and spatial attention mechanism H AS Extract channel attention and spatial attention features A C and A S , then for A C and A S Perform multiplication to obtain the output of the jth gated attention distillation module in the i-th dense gated convolution module Among them, H AC (·) and H AS (·) represent the channel attention mechanism and spatial attention mechanism respectively, and Mul(·) is the multiplication operation; Step 5: I LR and its shallow features F SF and deep features F DF Input image restoration module, I LR Restore to I SR , I SR =H up (Conv 3*3 (F SF +F DF ))+H up (I LR ) (7) Among them, H up (·) indicates an upsampling operation using bilinear interpolation.

Images