Image super-resolution method based on spatial-frequency domain feature hybridization based on gradient consistent perception

Abstract

The present invention relates to the field of image super-resolution technology, and discloses a space-frequency domain feature mixed image super-resolution method based on gradient consistent perception, including: constructing a space-frequency domain feature mixed image super-resolution network, including a super-resolution branch and a regression branch, the super-resolution branch uses a space-frequency domain feature mixing module to extract mixed features of different scales from a low-resolution image, the regression branch obtains a full-resolution image based on the mixed features, and reconstructs a low-resolution gradient map based on mixed features of different scales, and the low-resolution gradient map is used to construct a gradient consistent perception learning loss function to train the space-frequency domain feature mixed image super-resolution network. The present invention globally mines available high-frequency information, and applies first-order constraints to the high-resolution gradient map and the reconstructed low-resolution gradient map, so as to explore the global dependency between hierarchical features.

Description

Technical Field

[0001] The present invention relates to the technical field of image super-resolution, and in particular to a spatial-frequency domain feature hybrid image super-resolution method based on gradient consistent perception. Background Art

[0002] Image SR is a key task in the field of computer vision. Its core goal is to restore high-resolution images from degraded low-resolution (LR) images. Image SR technology can add richer and more valuable details to images. Therefore, it has been widely used in various computer vision application scenarios, such as remote sensing image processing, medical imaging, face recognition, and video surveillance.

[0003] Recent advances in neuroscience have shown that the perception and interaction of neurons on global information is the key to performing complex perceptual tasks. However, convolution only uses sliding windows to extract local features and cannot achieve global feature interaction. Therefore, most SR models based on convolutional neural networks (CNNs) have difficulty reconstructing reliable and clear details, and may even cause pixel distortion and displacement. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the CNN-based super-resolution model in the prior art cannot achieve global feature interaction.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a spatial-frequency domain feature hybrid image super-resolution method based on gradient consistent perception, comprising the following steps:

[0006] Construct a space-frequency domain feature mixing module to perform frequency domain conversion and space-frequency domain feature mixing on the low-resolution image to obtain mixed features;

[0007] A space-frequency domain feature mixed image super-resolution network is constructed by combining the space-frequency domain feature mixing module, and a gradient consistent perception learning loss function is constructed to train the space-frequency domain feature mixed image super-resolution network; the space-frequency domain feature mixed image super-resolution network includes a super-resolution branch and a regression branch, the super-resolution branch uses the space-frequency domain feature mixing module to extract mixed features of different scales from the low-resolution image, the regression branch obtains a full-resolution image based on the mixed features, and reconstructs a low-resolution gradient map based on the mixed features of different scales, and the low-resolution gradient map is used to construct a gradient consistent perception learning loss function;

[0008] Use the trained spatial-frequency domain feature hybrid image super-resolution network to reconstruct the low-resolution image to achieve image super-resolution;

[0009] The gradient consistent perception learning loss function is constructed to train the space-frequency domain feature mixed image super-resolution network, and the space-frequency domain feature mixed image super-resolution network is trained by minimizing L. The gradient consistent perception learning loss function L is expressed as:

[0010] ;

[0011] ;

[0012] in, represents the high-resolution feature map output by the i-th level feature pyramid in the super-resolution branch, It is a standard data set with The corresponding reference image, yes The corresponding gradient map is yes The corresponding gradient map; It is the high-resolution feature map output by the i-1th level feature pyramid The gradient map of the corresponding reference image; yes The reconstructed LR gradient map output by the downsampling module in the regression branch, represents the L1 loss function; and They are all balance parameters, and s represents the image target magnification.

[0013] Preferably, the super-resolution branch comprises an R-level feature pyramid. , s represents the magnification of the image target; the feature pyramid includes N spatial-frequency domain feature mixing modules and sub-pixel convolution layers connected in sequence.

[0014] Preferably, given a low-resolution image As input to the super-resolution branch, the processing of the super-resolution branch includes the following steps:

[0015] Using convolutional layers from Extract shallow features , expressed as:

[0016] ;

[0017] in, represents the convolutional layer;

[0018] The mixed features of different scales are gradually extracted by sequentially passing through the R-level feature pyramid. The input of the r-th level feature pyramid is , the output is , expressed as:

[0019] ;

[0020] ;

[0021] Among them, the value range of r is 1,...,R; represents the function of the r-th level feature pyramid, represents the i-th spatial-frequency domain feature mixing module function in the feature pyramid, Represents the sub-pixel convolution layer function.

[0022] Preferably, given an input feature map , where C, H, and W represent the number of channels, length, and width of the feature map, respectively. The spatial-frequency domain feature mixing module function The calculation of includes the following steps:

[0023] Use fast Fourier transform to transform X from spatial domain to frequency domain to obtain frequency domain feature map , expressed as:

[0024] ;

[0025] in, Represents a set of real numbers, x and y are the vertical and horizontal indices of the frequency domain feature map, and u and v are the vertical and horizontal indices of the input feature map;

[0026] The global filter can be learned Application , thus obtaining the enhanced frequency domain feature map H, expressed as:

[0027] ;

[0028] Use inverse fast Fourier transform to transform H from frequency domain to spatial domain to obtain global features , expressed as:

[0029] ;

[0030] Use convolutional layers to extract local features of X , expressed as:

[0031] ;

[0032] Using two learnable weight parameters and right and Perform weighted addition to obtain the output of the spatial-frequency domain feature mixing module , expressed as:

[0033] .

[0034] Preferably, the regression branch includes R-level reconstruction layers corresponding to the R-level feature pyramid, and the internal operation of each level of reconstruction layer is specifically as follows:

[0035] First, the output of each level of the feature pyramid is converted into a high-resolution image using a convolutional layer. , expressed as:

[0036] ;

[0037] Among them, the value range of r is 1,...,R; represents the final output full-resolution image;

[0038] Next, the Sobel gradient extraction operator is used to generate high-resolution gradient images of different scales. :

[0039] ;

[0040] in, represents the gradient extraction operator;

[0041] Finally, R groups of downsampling units are used to perform ×2 downsampling to obtain the reconstructed low-resolution gradient map.

[0042] Preferably, the downsampling unit includes a convolutional layer, a Leaky ReLU activation layer, and a convolutional layer cascade, and the downsampling unit performs ×2 downsampling to obtain a reconstructed LR gradient map, which is expressed as:

[0043] ;

[0044] in, represents the LeakyReLU activation layer, Indicated by Reconstructed low-resolution gradient map.

[0045] Preferably, the gradient consistent perceptual learning loss function is constructed to train the space-frequency domain feature mixed image super-resolution network, and the L1 loss function is used to minimize the distance between the generated high-resolution image and the reference image. The L1 loss function is also used to impose a first-order constraint on the high-resolution gradient map and the reconstructed low-resolution gradient map to reduce the solution space and enable the model to restore reliable and sharp high-frequency details.

[0046] Preferably, the method of using a trained space-frequency domain feature mixed image super-resolution network to reconstruct the low-resolution image to be reconstructed is specifically as follows: a low-resolution image of size 3×H×W is used as the input of the space-frequency domain feature mixed image super-resolution network, and is input into the super-resolution branch after passing through a convolution layer. In the super-resolution branch, multiple feature extractions are performed by an R-level feature pyramid, and the final output mixed features enter the regression branch. After passing through a convolution layer, a high-resolution output of size 3×(H*s)×(W*s) is obtained, wherein, , s is the target magnification of the image.

[0047] The present invention has the following beneficial effects:

[0048] (1) The present invention proposes a spatial-frequency domain feature hybrid module, which uses Fourier transform to convert the image from the spatial domain to the frequency domain, and then uses a learnable global filter to filter the redundant frequencies, thereby globally mining the available high-frequency information and avoiding the local processing limitations of CNN;

[0049] (2) The present invention proposes a method for constructing a gradient-consistent perceptual learning loss function, which, in addition to minimizing the distance between the generated SR image and the reference image, also imposes a first-order constraint on the SR gradient map and the reconstructed LR gradient map, thereby exploring the global dependency between hierarchical features.

[0050] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A method step diagram of an embodiment of the present invention;

[0052] Figure 2 It is a structural diagram of a space-frequency domain feature mixing module according to an embodiment of the present invention;

[0053] Figure 3 This is a diagram of the network structure of a spatial-frequency domain feature hybrid image super-resolution according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] See also Figure 1 FIG. 1 is a diagram showing steps of a method according to an embodiment of the present invention, comprising the following steps:

[0055] S101, constructing a space-frequency domain feature mixing module, which is used to perform frequency domain conversion and space-frequency domain feature mixing on the low-resolution image to obtain mixed features;

[0056] S102, constructing a space-frequency domain feature mixed image super-resolution network in combination with a space-frequency domain feature mixed module, and constructing a gradient consistent perception learning loss function to train the space-frequency domain feature mixed image super-resolution network;

[0057] S103, using the trained space-frequency domain feature hybrid image super-resolution network to reconstruct the low-resolution image to be reconstructed, to achieve image super-resolution.

[0058] The space-frequency domain feature mixed image super-resolution network includes a super-resolution branch and a regression branch. The super-resolution branch uses a space-frequency domain feature mixing module to extract mixed features of different scales from a low-resolution image. The regression branch obtains a high-resolution image based on the mixed features and reconstructs a low-resolution gradient map based on the mixed features of different scales. The low-resolution gradient map is used to construct a gradient consistent perception learning loss function.

[0059] The implementation process of the present invention is described in detail below through specific embodiments.

[0060] 1. Construct a spatial-frequency domain feature mixing module.

[0061] Module structure see Figure 2 As shown, the calculation process inside the module is as follows:

[0062] Given an input feature map , where C, H, and W represent the number of channels, length, and width of the feature map, respectively. The overall calculation process of constructing the spatial-frequency domain feature mixing module is as follows:

[0063] First, use Fast Fourier Transform (FFT) to transform X from the spatial domain to the frequency domain to obtain the frequency domain feature map , represents the set of real numbers, and its expression is as follows:

[0064] ;

[0065] Where x and y are the vertical and horizontal indices of the frequency domain feature map, and their value ranges are 0, 1, 2, ..., H-1 and 0, 1, 2, ..., W-1, respectively. u and v are the vertical and horizontal indices of the input feature map, and their value ranges are 0, 1, 2, ..., H-1 and 0, 1, 2, ..., W-1, respectively.

[0066] For adaptive adjustment The frequency of the global filter can be learned to emphasize the high-frequency features that are beneficial to detail restoration. Application , thus obtaining the enhanced frequency domain feature map H, which is expressed as follows:

[0067] ;

[0068] Then, the Inverse Fast Fourier Transform (IFFT) is used to transform H from the frequency domain to the spatial domain to obtain the global features , which is expressed as follows:

[0069] ;

[0070] Then, the convolutional layer is used to extract the local features of X , which is expressed as follows:

[0071] ;

[0072] In order to balance local and global features, two learnable weight parameters are used and right and Weighted addition is performed to obtain the output of the spatial-frequency domain feature mixing module. :

[0073] ;

[0074] 2. Combine the spatial-frequency domain feature mixing module to construct a spatial-frequency domain feature mixing image super-resolution network.

[0075] The network structure is as follows Figure 3 As shown in Figure 1, it includes an SR branch and a regression branch. The SR branch is used to reconstruct a high-resolution SR image, while the regression branch is used to reconstruct an LR gradient map.

[0076] The SR branch aims to reconstruct the SR image in a coarse-to-fine manner using a multi-level feature pyramid. Specifically, the SR branch contains an R-level feature pyramid, where R = , s represents the image target magnification. For example, when processing a ×2 SR task, R=1; when processing a ×4 SR task, R=2; when processing a ×8 SR task, R=3.

[0077] Given an LR image As the input of the SR branch, we first use a 3×3 convolutional layer from Extract shallow features :

[0078] ;

[0079] in, represents a 3×3 convolutional layer. Next, After R-level feature pyramid, multi-scale deep features are gradually extracted. Specifically, given the output of the (r-1)-th level feature pyramid As the input of the r-th level feature pyramid, the corresponding output It can be expressed as:

[0080] ;

[0081] The value range of r is 1, ..., R. The function representing the r-th level feature pyramid consists of N spatial-frequency domain feature mixing modules and a sub-pixel convolution layer, and its expression is as follows:

[0082] ;

[0083] in, represents the spatial-frequency domain feature mixing module function, Represents the sub-pixel convolution layer function.

[0084] The regression branch aims to reconstruct the LR gradient map. Specifically, the output of each level of the feature pyramid is converted into an SR image using a convolutional layer. :

[0085] ;

[0086] The value range of r is 1, ..., R. Represents the final output full-resolution SR image.

[0087] Next, the Sobel gradient extraction operator is used to generate SR gradient maps of different scales. :

[0088] ;

[0089] in, Represents the gradient extraction operator.

[0090] Next, use R groups of downsampling units (convolutional layer, Leaky ReLU activation layer, convolutional layer cascade) to perform ×2 downsampling to obtain the reconstructed LR gradient map:

[0091] ;

[0092] in, represents the LeakyReLU activation layer, Indicated by Reconstructed LR gradient map.

[0093] 3. Construct a gradient-consistent perceptual learning loss function to train the spatial-frequency domain feature hybrid image super-resolution network.

[0094] The gradient consistent perception learning loss function L not only uses the L1 loss function to minimize the distance between the generated SR image and the reference image, but also uses the L1 loss function to impose a first-order constraint on the SR gradient map and the reconstructed LR gradient map to reduce the solution space and enable the model to restore reliable and sharp high-frequency details. Its expression is as follows:

[0095] ;

[0096] in, represents the reference image provided in the standard dataset, yes The corresponding gradient map is represents the L1 loss function. Balanced parameters and are set to and The spatial-frequency domain feature hybrid image super-resolution network based on gradient consistent perceptual learning is trained by minimizing L.

[0097] 4. Use the spatial-frequency domain feature hybrid image super-resolution network based on gradient consistent perception learning to reconstruct the low-resolution image to obtain a reconstruction result.

[0098] A low-resolution image of size 3×H×W is used as the input of the space-frequency domain feature mixed image super-resolution network. After passing through the convolution layer, it is input into the super-resolution branch. In the super-resolution minute, it undergoes multiple feature extractions of the R-level feature pyramid. The final output mixed features enter the regression branch. After passing through a convolution layer, a high-resolution output of size 3×(H*s)×(W*s) is obtained. , s is the target magnification of the image, for example, s is 2, 4 or 8.

[0099] It can be seen that the present invention provides a space-frequency domain feature mixed image super-resolution method based on gradient consistent perceptual learning. Specifically, a space-frequency domain feature mixing module is proposed, which uses Fourier transform to convert the image from the space domain to the frequency domain, and then uses a learnable global filter to filter the redundant frequencies, thereby globally mining the available high-frequency information and avoiding the limitations of CNN's local processing. Then, a space-frequency domain feature mixed image super-resolution network based on gradient consistent perceptual learning is constructed in combination with the space-frequency domain feature mixing module. Finally, a method for constructing a gradient consistent perceptual learning loss function is proposed, which, in addition to minimizing the distance between the generated SR image and the reference image, also imposes a first-order constraint on the SR gradient map and the reconstructed LR gradient map, thereby exploring the global dependency between hierarchical features.

[0100] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A spatial-frequency domain feature hybrid image super-resolution method based on gradient consistent perception, characterized in that: The following steps are involved: Construct a space-frequency domain feature mixing module to perform frequency domain conversion and space-frequency domain feature mixing on the low-resolution image to obtain mixed features; A space-frequency domain feature mixed image super-resolution network is constructed by combining the space-frequency domain feature mixing module, and a gradient consistent perception learning loss function is constructed to train the space-frequency domain feature mixed image super-resolution network; the space-frequency domain feature mixed image super-resolution network includes a super-resolution branch and a regression branch, the super-resolution branch uses the space-frequency domain feature mixing module to extract mixed features of different scales from the low-resolution image, the regression branch obtains a full-resolution image based on the mixed features, and reconstructs a low-resolution gradient map based on the mixed features of different scales, and the low-resolution gradient map is used to construct a gradient consistent perception learning loss function; Use the trained spatial-frequency domain feature hybrid image super-resolution network to reconstruct the low-resolution image to achieve image super-resolution; The gradient consistent perception learning loss function is constructed to train the space-frequency domain feature mixed image super-resolution network, and the space-frequency domain feature mixed image super-resolution network is trained by minimizing L. The gradient consistent perception learning loss function L is expressed as: R = log2 s; in, represents the high-resolution feature map output by the i-th level feature pyramid in the super-resolution branch, It is a standard data set with The corresponding reference image, yes The corresponding gradient map is yes The corresponding gradient map; It is the high-resolution feature map output by the i-1th level feature pyramid The gradient map of the corresponding reference image; yes The reconstructed LR gradient map output by the downsampling module in the regression branch, L1(·) represents the L1 loss function; λ1 and λ2 are both balance parameters, and s represents the image target magnification.

2. The method for image super-resolution based on spatial-frequency domain feature hybrid based on gradient consistent perception according to claim 1, characterized in that: The super-resolution branch includes an R-level feature pyramid, where R=log2s, and s represents the magnification of the image target; the feature pyramid includes N spatial-frequency domain feature mixing modules and sub-pixel convolution layers connected in sequence.

3. The method for image super-resolution based on spatial-frequency domain feature hybrid based on gradient consistent perception according to claim 2, characterized in that: Given a low-resolution image As input to the super-resolution branch, the processing of the super-resolution branch includes the following steps: Using convolutional layers from I LR Extract shallow features It is expressed as: Among them, Conv(·) represents the convolutional layer; The mixed features of different scales are gradually extracted by sequentially passing through the R-level feature pyramid. The input of the r-th level feature pyramid is The output is It is expressed as: Among them, the value range of r is 1,...,R; P r (·) represents the function of the r-th level feature pyramid, represents the i-th spatial-frequency domain feature mixing module function in the feature pyramid, F SP (·) represents the sub-pixel convolution layer function.

4. The method for image super-resolution based on spatial-frequency domain feature hybrid based on gradient consistent perception according to claim 3 is characterized in that: Given an input feature map Where C, H, and W represent the number of channels, length, and width of the feature map, respectively. The spatial-frequency domain feature mixing module function The calculation of includes the following steps: Use fast Fourier transform to transform X from spatial domain to frequency domain to obtain frequency domain feature map It is expressed as: in, Represents a set of real numbers, x and y are the vertical and horizontal indices of the frequency domain feature map, and u and v are the vertical and horizontal indices of the input feature map; The global filter can be learned Application Thus, the enhanced frequency domain feature map HP is obtained, which is expressed as: The HP is transformed from the frequency domain to the spatial domain using the inverse fast Fourier transform to obtain the global features. It is expressed as: Use convolutional layers to extract local features of X It is expressed as: F local =Conv(X); Using two learnable weight parameters γ1 and γ2 to adjust F local and F global Perform weighted addition to obtain the output of the spatial-frequency domain feature mixing module It is expressed as: Y=γ1·F local +γ2·F global 。 5. The spatial-frequency domain feature hybrid image super-resolution method based on gradient consistent perception according to claim 1 is characterized in that: The regression branch includes R-level reconstruction layers corresponding to the R-level feature pyramid, and the internal operation of each level of reconstruction layer is specifically as follows: First, the output of each level of the feature pyramid is converted into a high-resolution image using a convolutional layer. It is expressed as: Among them, the value range of r is 1,...,R; represents the final output full-resolution image; Next, the Sobel gradient extraction operator is used to generate high-resolution gradient images of different scales. Among them, δ(·) represents the gradient extraction operator; Finally, R groups of downsampling units are used to perform ×2 downsampling to obtain the reconstructed low-resolution gradient map.

6. The method for image super-resolution based on spatial-frequency domain feature hybrid based on gradient consistent perception according to claim 5, characterized in that: The downsampling unit includes a convolutional layer, a Leaky ReLU activation layer, and a cascade of convolutional layers. The downsampling unit performs ×2 downsampling to obtain a reconstructed LR gradient map, which is expressed as: in, represents the LeakyReLU activation layer, Indicated by Reconstructed low-resolution gradient map.

7. The method for image super-resolution based on spatial-frequency domain feature hybrid based on gradient consistent perception according to claim 1, characterized in that: The gradient consistent perceptual learning loss function is constructed to train the spatial-frequency domain feature hybrid image super-resolution network, and the L1 loss function is used to minimize the distance between the generated high-resolution image and the reference image. The L1 loss function is also used to impose a first-order constraint on the high-resolution gradient map and the reconstructed low-resolution gradient map to reduce the solution space and enable the model to restore reliable and sharp high-frequency details.

8. The method for image super-resolution based on spatial-frequency domain feature hybridization based on gradient consistent perception according to claim 1, characterized in that: The method uses a trained space-frequency domain feature mixed image super-resolution network to reconstruct the low-resolution image to be reconstructed, specifically: a low-resolution image with a size of 3×H×W is used as the input of the space-frequency domain feature mixed image super-resolution network, and is input into the super-resolution branch after passing through a convolution layer. In the super-resolution branch, multiple feature extractions are performed through an R-level feature pyramid, and the final output mixed features enter the regression branch. After passing through a convolution layer, a high-resolution output with a size of 3×(H*s)×(W*s) is obtained, where R=log2 s, and s is the image target magnification.

Images