Image super-resolution method based on second-order visual attention mechanism

Abstract

The present invention relates to the field of pattern recognition technology, and in particular to an image super-resolution method based on a second-order visual attention mechanism. A low-resolution image is obtained by downsampling an input image, and the downsampled low-resolution image is encoded into a corresponding degraded representation. The non-local reproducibility of internal information of a single image is utilized to avoid dependence on a large-scale data set. A second-order visual attention mechanism is proposed. The second-order visual attention mechanism includes five second-order visual attention residual groups, each residual group includes five second-order visual attention modules, and deep features of the image are extracted through the second-order visual attention modules. The degraded representation and weight matrices of different channels are input into each second-order visual attention module, and the output features of the fifth residual group are upsampled to obtain a super-resolution image. The super-resolution image is suitable for different degradation modes in real situations and improves the image super-resolution capability.

Description

Technical Field

[0001] The present invention relates to the field of pattern recognition technology, and in particular to an image super-resolution method based on a second-order visual attention mechanism. Background Art

[0002] In real life, due to the interference of hardware and other factors, it is difficult for people to obtain satisfactory high-resolution images. With the development of deep learning, the technology of using algorithm calculation to improve the resolution of low-resolution images based on existing image sensors and hardware equipment has become the focus of research and attention. Single image super-resolution reconstruction technology is to use a single low-resolution image to reconstruct a high-resolution image through calculation. The core of image reconstruction is to increase the pixel density of the low-resolution image by adding pixels, and require that the added pixels cannot change the detail information of the original low-resolution image, enhance the resolution of the image as much as possible, and improve human visual effects and computer recognition. At present, image super-resolution technology has been widely used in many fields such as satellite remote sensing, medical imaging, criminal investigation monitoring, image compression and storage.

[0003] At present, the main image super-resolution algorithms are as follows: (1) Reconstruction-based super-resolution methods: Reconstruction-based methods introduce image prior knowledge into the reconstruction process. This type of method attempts to establish a generative model for low-resolution images, that is, a degradation model for high-resolution images, so as to restore high-quality images through the inverse processing of the model; (2) Learning-based super-resolution methods: Learning-based methods can make full use of the computing power of computers, train models through image data samples of different resolutions, establish mapping relationships between images of different resolutions, and solve image prior knowledge to replace the artificial constraints in model-based reconstruction methods; (3) Deep learning-based super-resolution algorithms: With the improvement of computer performance, deep learning has gradually played an important role in various fields. The back propagation algorithm solves the problem of training deep parameters through gradient descent. According to the relationship between images of different resolutions, researchers use multi-layer perceptrons, autoencoders, deep convolutional networks and other structures to learn better mapping relationships to reconstruct high-resolution images.

[0004] However, most current super-resolution methods rely on large datasets and are applicable to situations where the image low-resolution degradation process is fixed, but are difficult to handle situations with different degradation modes in real situations. Summary of the invention

[0005] The present invention provides an image super-resolution method based on a second-order visual attention mechanism to solve the problem that most current super-resolution methods rely on a large number of data sets, are suitable for situations where the image low-resolution degradation process is fixed, and are difficult to handle situations with different degradation modes in real situations.

[0006] In order to achieve the above object, the present invention provides an image super-resolution method based on a second-order visual attention mechanism, the method comprising:

[0007] Downsample the input image to obtain a low-resolution image;

[0008] Encode the downsampled low-resolution image into a corresponding degraded representation;

[0009] Extract the shallow features of the image and obtain the feature map F∈R of the shallow features H×W×C , where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map;

[0010] Calculate the weight matrix F of different channels in the feature map c ∈R C×1×1 ,in,

[0011] F c =Sigmoid(f 1×1 (ReLU(f 1×1 (H GPP (F)))))

[0012] Among them, H GPP ∈R C×1 Represents nonlinear channel characteristics;

[0013] Extract deep features of images through the second-order visual attention module;

[0014] Upsample the deep features of the image to obtain a super-resolution image;

[0015] The input images are randomly rotated and flipped to achieve data augmentation, and the Adam optimizer is used to optimize the training process.

[0016] In the above-mentioned image super-resolution method based on the second-order visual attention mechanism, optionally, the input image is downsampled to obtain a low-resolution image, specifically: using a low-resolution remote sensing image as input, performing an s-fold downsampling operation to obtain a lower resolution image corresponding to the input image, whose size is Construct matching image pairs for the input image.

[0017] In the above-mentioned image super-resolution method based on the second-order visual attention mechanism, optionally, the downsampled low-resolution image is encoded into a corresponding degraded representation, specifically: the downsampled low-resolution image is passed through six layers of 3×3 convolutional layers and an average pooling layer to obtain a degraded representation.

[0018] In the above-mentioned image super-resolution method based on the second-order visual attention mechanism, optionally, the shallow features of the image are extracted to obtain a feature map of the shallow features, specifically: the downsampled low-resolution image is passed through a 3×3 convolutional layer to obtain a feature map of the shallow features of the image.

[0019] In the above-mentioned image super-resolution method based on the second-order visual attention mechanism, optionally, the weight matrix of different channels in the calculated feature map is specifically:

[0020] Through the reshape operation, the feature map F∈R H×W×C Mapping to feature sequence matrix Get the covariance matrix Σ, where

[0021] Σ=XX T

[0022] in H×W represents the original feature map size, and P represents the size of each feature block;

[0023] The eigenvalue decomposition of the covariance matrix Σ is performed to obtain the eigenvalue λ. The relationship between the covariance matrix Σ, the new coordinate system W and the eigenvalue λ is: ∑w i =λ i w i , the original sample x i In the low-dimensional coordinate system z i =(z i1 ,z i2 ,...,z id' ) means (where Represents x i In the j-th dimension), we get the linear feature y∈R 3C×1 ;

[0024] The linear feature y∈R 3C×1 After the multi-layer perceptron, the nonlinear channel feature H is obtained GPP ∈R C×1 , nonlinear channel characteristics H GPP ∈R C×1 The weight matrix F of different channels in the feature map is obtained through the convolution layer, ReLU layer and Sigmoid activation function. c ∈R C×1×1 .

[0025] In the above-mentioned image super-resolution method based on the second-order visual attention mechanism, optionally, the deep features of the image are extracted by the second-order visual attention module, specifically:

[0026] The degraded representation is passed through two fully connected layers and a reshape layer to obtain the convolution kernel w, and the weight matrix is ​​passed through the visual attention convolution layer (the convolution kernel is w) and the 1×1 convolution layer to obtain the first deep feature F1, where

[0027] w=f reshape (f FC (f FC (R)))

[0028] F1=f 1×1 (f VA (F c ,w))

[0029] The degraded representation passes through two fully connected layers and a Sigmoid activation function layer to obtain the channel modulation coefficient v. Then, the channel modulation coefficient v is combined with the weight matrix F c Multiply them together to get the second depth feature F2, where

[0030] v = Sigmoid(f FC (f FC (R)))

[0031]

[0032] The feature obtained by adding the first depth feature F1 and the second depth feature F2 passes through a 3×3 convolution layer, a visual attention convolution layer, and a 3×3 convolution layer and the weight matrix F c Add together to get the output features of the first second-order visual attention module in,

[0033] F'=F1+F2

[0034]

[0035] The output features of the first second-order visual attention module As the input of the next second-order visual attention module, until the output features of the fifth second-order visual attention module are obtained, the output features of the fifth second-order visual attention module are used as the input of the next visual attention residual group, until the output features of the fifth second-order visual attention residual group, that is, the deep features of the image, are obtained.

[0036] In the above-mentioned image super-resolution method based on the second-order visual attention mechanism, it is optional that the deep features of the image are upsampled to obtain the super-resolution image, specifically: after obtaining the output features of the fifth second-order visual attention residual group, the output features are added to the input features F, and then the size (r 2C, H, W) is reshaped into a vector of size (C, rH, rW) to achieve the upsampling effect and obtain a super-resolution image.

[0037] The present invention provides an image super-resolution method based on a second-order visual attention mechanism, which obtains a low-resolution image by downsampling an input image, encodes the downsampled low-resolution image into a corresponding degraded representation, and utilizes the non-local reproducibility of the internal information of a single image to avoid dependence on a large-scale data set; a method based on a second-order visual attention mechanism is proposed, which includes five second-order visual attention residual groups, each residual group includes five second-order visual attention modules, and the deep features of the image are extracted through the second-order visual attention modules. The degraded representation and the weight matrix of different channels are input into each second-order visual attention module, and the output features of the first second-order visual attention module are used as the input of the next second-order visual attention module until the output features of the fifth second-order visual attention module are obtained. The output features of the fifth second-order visual attention module are used as the input of the next visual attention residual group until the output features of the fifth second-order visual attention residual group are obtained. Finally, the output features of the fifth residual group are upsampled to obtain a super-resolution image, which is suitable for different degradation modes in real situations and improves the image super-resolution capability.

[0038] The construction of the present invention and other objects and advantageous effects thereof will become more apparent through the description of the preferred embodiments in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A flowchart of an image super-resolution method based on a second-order visual attention mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the embodiment of the present invention will be described in more detail below in conjunction with the drawings in the preferred embodiments of the present invention. In the drawings, the same or similar reference numerals throughout represent the same or similar parts or parts with the same or similar functions. The described embodiments are part of the embodiments of the present invention, not all of them. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limitations on the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The embodiments of the present invention are described in detail below in conjunction with the drawings.

[0042] like Figure 1 As shown, the present invention provides an image super-resolution method based on a second-order visual attention mechanism, the method comprising:

[0043] Step 110: down-sample the input image to obtain a low-resolution image.

[0044] The input image is downsampled to obtain a low-resolution image, specifically: a low-resolution remote sensing image is used as input, and a downsampling operation is performed s times to obtain a lower resolution image corresponding to the input image, whose size is Construct matching image pairs for the input image.

[0045] It should be noted that, in theory, the relationship between the low-resolution image and the high-resolution image is expressed as:

[0046] I LR =(I HR *k)↓ s +n

[0047] Among them, I LR Represents a low-resolution image, I HR represents a high-resolution image, * represents a convolution operation, k represents a blur kernel, ↓ s It represents downsampling by a factor of s, and n represents noise, such as speckle noise and acoustic noise.

[0048] In the process of processing the real low-resolution image, its high-resolution image I HR The blur kernel k and noise n are unknown, and there is no reliable paired data set to train the network. Therefore, using a low-resolution image I LR As input, perform a downsampling operation s times to obtain a low-resolution image corresponding to the input image Its size is Constructs a matching image pair for the input image. Use Low-resolution images of size 1 are used as input for the training process.

[0049] Step 120: Encode the downsampled low-resolution image into a corresponding degraded representation R.

[0050] The encoding of the downsampled low-resolution image into the corresponding degraded representation is specifically: the downsampled low-resolution image is passed through six 3×3 convolutional layers and an average pooling layer to obtain the degraded representation R.

[0051] Step 130: Extract shallow features of the image and obtain a feature map F∈R of shallow features H×W×C , where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map.

[0052] The shallow features of the image are extracted to obtain a feature map of the shallow features, specifically: the low-resolution image after downsampling Through the 3×3 convolution layer, we get the feature map F∈R of the shallow features of the image. H×W×C .

[0053] Step 140: Calculate the weight matrix F of different channels in the feature map c ∈R C×1×1 ,in,

[0054] F c =Sigmoid(f 1×1 (ReLU(f 1×1 (H GPP (F)))))

[0055] Among them, H GPP ∈R C×1 Represents nonlinear channel characteristics.

[0056] It should be noted that the feature map F∈R in step 130 is reshaped by the reshape operation. H×W×C Mapping to feature sequence matrix Get the covariance matrix Σ, where

[0057] Σ=XX T

[0058] in H×W represents the original feature map size, and P represents the size of each feature block;

[0059] It should be noted that in the experiment, P=16 is set, that is, the feature map F is divided into N (P, P) feature blocks. In order to obtain the principal components of the features, the covariance matrix Σ is subjected to eigenvalue decomposition to obtain the sorted d eigenvalues: λ1≥λ2≥...≥λ d . Set W = (w1, w2, ..., wd ) is the new coordinate system, where ||w i ||2=1, Then the relationship between the covariance matrix Σ, the new coordinate system W and the eigenvalue λ is:

[0060] ∑w i =λ i w i

[0061] x i In the low-dimensional coordinate system z i =(z i1 ,z i2 ,...,z id' ) means, that is Represents x i The coordinates in the jth dimension. So x i Can be z i Reconstruction, get For the entire feature map, the original sample x i and the reconstructed sample z i The distance between them is:

[0062]

[0063] The optimization goal of principal component analysis is to make the original sample x i and the reconstructed sample z i Minimize the distance between them. i =(z i1 ,z i2 ,...,z id' ), where dimension d'<d, set d'=3, that is, take the three eigenvectors with the largest eigenvalues, and then get the feature y∈R 3C×1 , since the feature is linear at this time and the number of channels is 3C, the feature is passed through a multilayer perceptron (MLP) to convert y∈R 3C×1 becomes the nonlinear channel feature H GPP ∈R C×1 , and then the weight matrix F of different channels in the feature map is obtained through the convolution layer ReLU layer and Sigmoid activation function c ∈R C×1×1 Input to the subsequent network.

[0064] Step 150: Extract deep features of the image through a second-order visual attention module.

[0065] It should be noted that the second-order visual attention mechanism includes 5 second-order visual attention residual groups, and each residual group contains 5 second-order visual attention modules.

[0066] The degenerate representation R passes through two fully connected layers (FC layer) and a reshape layer to obtain the convolution kernel w∈R C×1×3×3 , the weight matrix F c After the visual attention convolution layer (convolution kernel is w) and the 1×1 convolution layer, the first deep feature F1 is obtained, where

[0067] w=f reshape (f FC (f FC (R)))

[0068] F1=f 1×1 (f VA (F c ,w))

[0069] It should be noted that in each second-order visual attention module, the visual attention convolution layer can adjust the size of the convolution kernel of the deep convolution layer through the degenerate representation R.

[0070] Inspired by CResMD (using control variables to scale different channels to handle multiple degraded features), the visual attention convolution layer generates the modulation coefficient of the channel feature according to the degraded representation R, so that the channel feature can adaptively adjust the channel weight according to different degraded representations R. Specifically, the degraded representation R passes through two fully connected layers and a Sigmoid activation function layer to obtain the channel modulation coefficient v, and then the channel modulation coefficient v is combined with the weight matrix F c Multiply, shrink F c The different channel components in the second depth feature F2 are obtained, where

[0071] v = Sigmoid(f FC (f FC (R)))

[0072]

[0073] The feature obtained by adding the first depth feature F1 and the second depth feature F2 passes through a 3×3 convolution layer, a visual attention convolution layer, and a 3×3 convolution layer and the weight matrix F c Add together to get the output features of the first second-order visual attention module in,

[0074] F'=F1+F2

[0075]

[0076] The output features of the first second-order visual attention module As the input of the next second-order visual attention module, until the output features of the fifth second-order visual attention module are obtained, the output features of the fifth second-order visual attention module are used as the input of the next visual attention residual group, until the output features of the fifth second-order visual attention residual group, that is, the deep features of the image, are obtained.

[0077] Step 160: Upsample the depth features of the image obtained in step 150 to obtain a super-resolution image

[0078] Specifically, after obtaining the output features of the fifth second-order visual attention residual group, the output features are added to the input features F and then passed through the PixelShuffle layer to convert the size (r 2 C,H,W) is reshaped into a vector of size (C,rH,rW) to achieve the upsampling effect and obtain a super-resolution image Where r represents the upsampling factor.

[0079] Step 170: For the input image I LR Random rotation and flipping are performed to achieve data enhancement, and the Adam optimizer is used to optimize the training of steps 130 to 160.

[0080] During the training process, the input image I LR Random rotation and flipping are performed to achieve data enhancement. The Adam optimizer is used with parameters β1 = 0.9, β2 = 0.999 and the initial learning rate is set to 1×10 -3 , and is reduced by half every 125 epochs. When the learning rate reaches 1×10 -8 The whole training process ends when .

[0081] During the training process, L = L degrad +L SR As the overall loss function, the degradation loss L degrad for:

[0082]

[0083] Among them, the temperature coefficient τ = 0.07, the total number of samples N queue =8,192, degenerate type B=32.

[0084] Query Sample Represents the image after downsampling A 48×48 image block is randomly selected from the image block, and the 48×48 image block is subjected to step 120 to obtain a corresponding degraded representation. Different image blocks in the same image are used as positive samples. By giving the image Adding additional Gaussian blur kernels can produce B types of blurred images. As negative samples The process of adding an additional Gaussian blur kernel is:

[0085]

[0086] Among them, k is the parameter of the Gaussian blur kernel.

[0087] Minimum mean square error L SR for:

[0088]

[0089] L SR is the minimum mean square error, so that the input true value image I LR and super-resolution images The error between them is minimal.

[0090] After training, the input low-resolution image I LR Perform image super-resolution reconstruction to obtain the corresponding super-resolution image I SR .

[0091] The present invention provides an image super-resolution method based on a second-order visual attention mechanism, which obtains a low-resolution image by downsampling an input image, encodes the downsampled low-resolution image into a corresponding degraded representation, and utilizes the non-local reproducibility of the internal information of a single image to avoid dependence on a large-scale data set; a method based on a second-order visual attention mechanism is proposed, which includes five second-order visual attention residual groups, each residual group includes five second-order visual attention modules, and the deep features of the image are extracted through the second-order visual attention modules. The degraded representation and the weight matrix of different channels are input into each second-order visual attention module, and the output features of the first second-order visual attention module are used as the input of the next second-order visual attention module until the output features of the fifth second-order visual attention module are obtained. The output features of the fifth second-order visual attention module are used as the input of the next visual attention residual group until the output features of the fifth second-order visual attention residual group are obtained. Finally, the output features of the fifth residual group are upsampled to obtain a super-resolution image, which is suitable for different degradation modes in real situations and improves the image super-resolution capability.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image super-resolution method based on a second-order visual attention mechanism, characterized in that: Including, the method comprises: Downsample the input image to obtain a low-resolution image; Encode the downsampled low-resolution image into the corresponding degraded representation R; Extract the shallow features of the image and obtain the feature map F∈R of the shallow features H×W×C , where H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map; Calculate the weight matrix F of different channels in the feature map c ∈R C×1×1 ,in, F c =Sigmoid(f 1×1 (ReLU(f 1×1 (H GPP (F))))) Among them, H GPP ∈R C×1 Represents nonlinear channel characteristics; Extract deep features of images through the second-order visual attention module; Upsample the deep features of the image to obtain a super-resolution image; The input image is randomly rotated and flipped to achieve data enhancement, and the Adam optimizer is used to optimize the training process; The deep features of the image are extracted by the second-order visual attention module, specifically: The degraded representation is passed through two fully connected layers and a reshape layer to obtain the convolution kernel w, and the weight matrix is ​​passed through the visual attention convolution layer and the 1×1 convolution layer to obtain the first deep feature F1, where the convolution kernel is w: w=f reshape (f FC (f FC (R))) F1=f 1×1 (f VA (F c ,w)) The degraded representation passes through two fully connected layers and a Sigmoid activation function layer to obtain the channel modulation coefficient v. Then, the channel modulation coefficient v is combined with the weight matrix F c Multiply them together to get the second depth feature F2, where v=Sigmoid(f FC (f FC (R))) The feature obtained by adding the first depth feature F1 and the second depth feature F2 passes through a 3×3 convolution layer, a visual attention convolution layer, and a 3×3 convolution layer and the weight matrix F c Add together to get the output features of the first second-order visual attention module in, F'=F1+F2 The output features of the first second-order visual attention module As the input of the next second-order visual attention module, until the output features of the fifth second-order visual attention module are obtained, the output features of the fifth second-order visual attention module are used as the input of the next visual attention residual group, until the output features of the fifth second-order visual attention residual group, that is, the deep features of the image, are obtained.

2. The image super-resolution method based on the second-order visual attention mechanism according to claim 1, characterized in that: The input image is downsampled to obtain a low-resolution image, specifically: a low-resolution remote sensing image is used as input, and a downsampling operation is performed s times to obtain a lower resolution image corresponding to the input image, whose size is Construct matching image pairs for the input image.

3. The image super-resolution method based on the second-order visual attention mechanism according to claim 1 or 2, characterized in that: The encoding of the downsampled low-resolution image into a corresponding degraded representation is specifically: the downsampled low-resolution image is passed through six 3×3 convolutional layers and an average pooling layer to obtain a degraded representation.

4. The image super-resolution method based on the second-order visual attention mechanism according to claim 3, characterized in that: The method of extracting shallow features of an image to obtain a feature map of the shallow features specifically comprises: passing the downsampled low-resolution image through a 3×3 convolutional layer to obtain a feature map of the shallow features of the image.

5. The image super-resolution method based on the second-order visual attention mechanism according to claim 4, characterized in that: The weight matrix of different channels in the calculation feature map is specifically: Through the reshape operation, the feature map F∈R H×W×C Mapping to feature sequence matrix Get the covariance matrix Σ, where S=XX T in, H×W represents the original feature map size, and P represents the size of each feature block; The eigenvalue decomposition of the covariance matrix Σ is performed to obtain the eigenvalue λ. The relationship between the covariance matrix Σ, the new coordinate system Z and the eigenvalue λ is: ∑z i =λ i w i , the original sample x i In the low-dimensional coordinate system z i =(z i1 ,z i2 ,...,z ij ) indicates that Represents x i In the j-th dimension, we get the linear feature y∈R 3C×1 ; The linear feature y∈R 3C×1 After the multi-layer perceptron, the nonlinear channel feature H is obtained GPP ∈R C×1 , nonlinear channel characteristics H GPP ∈R C×1 The weight matrix F of different channels in the feature map is obtained through the convolution layer, ReLU layer and Sigmoid activation function. c ∈R C ×1×1 .

6. The image super-resolution method based on the second-order visual attention mechanism according to claim 5, characterized in that: The deep features of the image are upsampled to obtain a super-resolution image, specifically: after obtaining the output features of the fifth second-order visual attention residual group, the output features are added to the input features F, and then the size (r 2 C, H, W) is reshaped into a vector of size (C, rH, rW) to achieve the upsampling effect and obtain a super-resolution image.

Images