Lightweight image super-resolution reconstruction method and device based on frequency domain separation network

Abstract

The application discloses a kind of light image super-resolution reconstruction method and device based on frequency domain separation network, belong to image processing technical field, the method includes: S1: using the convolution of pre-set size carries out shallow feature extraction to the low-resolution image to be handled and obtains shallow feature;S2: using the Fourier residual module group based on frequency domain separation network carries out deep feature extraction to shallow feature and obtains the deep feature corresponding to each module;S3: the deep feature corresponding to each module is merged again using convolution and is carried out feature fusion to obtain fusion feature;S4: shallow layer information is combined as residual information and fusion feature, to generate high-resolution image.The application designs the Fourier residual module group corresponding based on frequency domain separation network, reduces redundant structure, while maintaining good network performance greatly reduces the required parameter amount and computing power, thereby solve the technical problem of high computational complexity of existing image super-resolution reconstruction method.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and more particularly relates to a light image super-resolution reconstruction method and device based on a frequency domain separation network. BACKGROUND

[0002] Image super-resolution is a method of improving the resolution of an original image through hardware or software, that is, generating a high-resolution (HR) image from a low-resolution (LR) image. Since the image super-resolution reconstruction technology can correct damaged images caused by environmental or device factors to a certain extent, the technology is widely used in security, medicine, remote sensing, imaging and other fields, and is one of the core problems of image processing technology.

[0003] Traditional image super-resolution methods are mainly divided into three categories according to principles, namely interpolation-based, degradation-based and learning-based super-resolution algorithms. The interpolation-based method is a non-iterative spatial domain algorithm that directly estimates the value of the pixel to be inserted using known neighboring pixel points using mathematical formulas to achieve the image super-resolution reconstruction process. The degradation-based method uses an image degradation model to establish a mapping relationship between the low-resolution image and the high-resolution image to achieve constraint optimization solution. The learning-based method directly learns the mapping relationship between the low-resolution image and the high-resolution image to achieve direct reconstruction. Although the traditional method has achieved great success, the reconstruction quality is still limited, and the prior knowledge defined by humans cannot meet the demand. In recent years, the super-resolution method based on deep learning has gradually become the mainstream and has achieved the most advanced level. Thanks to powerful computing power and large data sets, the method based on deep learning can better learn feature representation and achieve better visual effects compared with traditional methods.

[0004] However, the method based on deep learning often increases the depth of the network to improve the performance of the model, which means that more parameters and computing overhead are required, and the hardware device is constrained, so it is not practical to directly deploy such a method on mobile devices and security devices. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a light image super-resolution reconstruction method and device based on a frequency domain separation network, which aims to design a Fourier residual module group corresponding to the frequency domain separation network, reduce redundant structures, obtain a light super-resolution network, greatly reduce the required parameter quantity and computing power while maintaining good performance, thereby solving the technical problem of high computational complexity of the existing image super-resolution reconstruction method.

[0006] To achieve the above object, according to one aspect of the present application, a light image super-resolution reconstruction method based on a frequency domain separation network is provided, comprising:

[0007] S1: using a preset size of convolution on a low-resolution image I to be processed to perform shallow feature extraction and obtain a shallow feature F0; LR

[0008] S2: using a Fourier residual module group based on a frequency domain separation network to perform deep feature extraction on the shallow feature F0, to obtain deep features corresponding to each Fourier residual module based on the frequency domain separation network;

[0009] S3: merging the deep features corresponding to each Fourier residual module based on the frequency domain separation network, and then using convolution to perform feature fusion to obtain a fusion feature F fused ;

[0010] S4: combining the shallow feature F0 as residual information and the fusion feature F fused to generate a high-resolution image I HR .

[0011] In one embodiment, a basic unit of each Fourier residual module based on the frequency domain separation network is a fast Fourier convolution FFC;

[0012] The fast Fourier convolution FFC can obtain a non-local receptive field by analyzing the characteristics of the spatial domain and the spectral domain, and realize cross-scale fusion within the convolution.

[0013] In one embodiment, the S2 comprises:

[0014] S21: inputting the shallow feature F0 into a first Fourier residual module based on the frequency domain separation network, and the Fourier residual module based on the frequency domain separation network performs the following operations:

[0015] using a frequency domain separation network to perform high-low frequency separation to obtain first low-frequency information F low and first high-frequency information F high ; inputting the first low-frequency information F low into a preset network to perform feature extraction to obtain second low-frequency information; inputting the first high-frequency information F high into a high-frequency distillation module to split according to channels to obtain F hl1 and F hr1 ; F hl1 passes through one layer of FFC to obtain a first layer of distillation feature F hd1 ; F hr1 passes through a fast Fourier module FFCB for feature extraction, and the FFCB comprises a FFC and a GELU activation function; wherein, F​hr1 As the residual and the feature F' generated through FFC in the FFCB hr1 The sums are fed into the GELU activation function to obtain F. hr2 ; F hr2 The first high-frequency information F is used as the feature to be processed. high Get F hd1 and F hr2 The process of obtaining F hd2 and F hr3 ; F hr3 The first high-frequency information F is used as the feature to be processed. high Get F hd1 and F hr2 The process of obtaining F hd3 and F hr4 ; For F hr4 The final distillation feature F is obtained by extracting FFCB and FFC features. hd4 ; Input the first high-frequency information F high and F hd1 F hd2 F hd3 F hd4 The features are added together and fed into a convolutional layer for feature fusion, and then fed into a spatial attention layer to obtain the second high-frequency information; the second high-frequency information and the second low-frequency information are superimposed to obtain the deep features corresponding to the first Fourier residual module based on the frequency domain separation network.

[0016] S22: Input the result obtained in S21 into the second Fourier residual module based on the frequency domain separation network to obtain the deep features corresponding to the second Fourier residual module based on the frequency domain separation network. Iterate multiple times to obtain the deep features corresponding to the third to Nth Fourier residual modules based on the frequency domain separation network.

[0017] In one embodiment, the frequency domain separation network includes a Gaussian filter;

[0018] The shallow feature F0 is passed through the Gaussian filter to obtain the low-frequency information F of the shallow feature. low Subtract the low-frequency information F from the shallow features low The high-frequency information F is obtained high .

[0019] In one embodiment, the preset network includes, in sequence, an FFC, a GELU activation layer, and an FFC.

[0020] In one embodiment, S4 includes:

[0021] The shallow information F0 is used as the residual information and the fusion feature F.fused combine and send into convolution to convert the feature to the target feature F suitable for upsampling up ;

[0022] upsample the target feature F using a sub-pixel module to obtain a high-resolution image I up ; HR .

[0023] According to another aspect of the present application, a light image super-resolution reconstruction device based on a frequency domain separation network is provided for performing the light image super-resolution reconstruction method based on the frequency domain separation network, comprising:

[0024] a shallow extraction module configured to perform shallow feature extraction on a low-resolution image I to be processed using a preset size of convolution to obtain a shallow feature F0; LR ;

[0025] a deep extraction module configured to perform deep feature extraction on the shallow feature F0 using a Fourier residual module group based on a frequency domain separation network to obtain a deep feature corresponding to each of the Fourier residual modules based on the frequency domain separation network;

[0026] a feature merging module configured to merge the deep features corresponding to each of the Fourier residual modules based on the frequency domain separation network and then perform feature fusion using convolution to obtain a fusion feature F fused ;

[0027] an image generation module configured to combine the shallow information F0 as residual information and the fusion feature F to generate a high-resolution image I fused . HR .

[0028] According to another aspect of the present application, an image processing device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0029] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0030] (1) The present application designs a Fourier residual module group based on a frequency domain separation network to perform deep feature extraction on the shallow feature F0, adopts different processing methods for high-frequency and low-frequency information of natural images, and compared with the previous super-resolution method, reduces the redundant structure to obtain a light super-resolution network, so that the network greatly reduces the required parameter amount and computing power while maintaining good performance. The present application can reduce the demand for computing resources while ensuring high-quality image reconstruction effect;

[0031] (2) For high frequency information, a residual distillation network is used to deeply extract high frequency information; for low frequency information, a two-layer network can be used to realize information extraction; high frequency information and low frequency information are processed respectively, which can further ensure the effect of super-resolution network reconstruction;

[0032] (3) The present application uses fast Fourier convolution as a basic network unit, increases the network receptive field, realizes multi-scale information fusion, and improves the network performance. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a flowchart of a lightweight image super-resolution method based on a frequency domain separation network provided by an embodiment of the present application;

[0034] Figure 2 is a schematic diagram of the overall architecture of a lightweight image super-resolution based on a frequency domain separation network provided by an embodiment of the present application;

[0035] Figure 3 is a schematic diagram of a Fourier residual module architecture based on a frequency domain separation network provided by an embodiment of the present application;

[0036] Figure 4 is a schematic diagram of a high frequency distillation module architecture provided by an embodiment of the present application;

[0037] Figure 5 is a super-resolution effect diagram after reconstruction provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0039] Figure 1 is a flowchart of a lightweight image super-resolution method based on a frequency domain separation network provided by the present application, Figure 2 is the overall architecture of a lightweight image super-resolution method based on a frequency domain separation network provided by the present application, which combines Figure 1 and Figure 2 It can be seen that the overall process of the method is as follows:

[0040] S1: input the low-resolution image I to be processed LR , a 3x3 convolution is used to perform shallow feature extraction on the low-resolution image I LR , and a shallow feature F0 is obtained.

[0041] S2: the shallow feature F0 is subjected to deep feature extraction by using the Fourier residual module group based on the frequency domain separation network to obtain deep features F1-F6.

[0042] S3: the deep features output by each layer are combined and sent to a 1x1 convolution and a 3x3 convolution for feature fusion to obtain final features F fused .

[0043] S4: the extracted shallow information F0 is taken as residual information and F fused is combined and sent to a 3x3 convolution to convert the features to features F up that are suitable for upsampling. up is upsampled to a high-resolution image I HR .

[0044] The application proposes a Fourier residual module group based on a frequency domain separation network: in the down-sampling process of a high-resolution image, the high-frequency information is severely damaged, and the low-frequency information is less lost, so if the same computing resources are used for high and low frequency information, it will cause computational redundancy. In fact, reconstructing high-frequency information requires a large amount of computing resources, while low-frequency information does not. Based on this premise, the input of each module is separated into high and low frequency information by using a Gaussian filter, and then added again after processing to the next module for processing.

[0045] Further, the super-resolution task requires a large receptive field in convolution, and previous work often adds depth with small receptive field convolution. Although the calculation amount required by the small convolution kernel is smaller than that of the large convolution kernel, the multi-layer stacking still requires more computing power. The application mainly uses fast Fourier convolution (FFC) as the basic unit of the Fourier residual module architecture based on the frequency domain separation network. Fast Fourier convolution (FFC) efficiently obtains non-local receptive field by analyzing the characteristics of the spatial domain and the frequency domain, and realizes cross-scale fusion within the convolution unit.

[0046] Further, in order to enable the network to better process the high-frequency information proposed in (1), the application proposes a high-frequency distillation module, which sends the extracted high-frequency information into it. Each layer in the module first splits the input of the previous layer into two parts according to the channel, one of which is sent to the next layer for further processing, and the other is distilled. Finally, all the distilled information is combined and sent to a 1x1 convolution fusion and then to a spatial attention layer to further help the network focus on information such as structure, texture, edge and other information that is conducive to enhancing image details.

[0047] Further, in order to enable the network to better process the low-frequency information proposed in (1), the present application proposes a low-frequency processing module, and the extracted low-frequency information is sent into the module, and each module is only composed of a simple two-layer Fourier convolution.

[0048] Figure 3 is a schematic diagram of a Fourier residual module architecture based on a frequency domain separation network provided by an embodiment of the present application, and the process of deep feature extraction in S2 is specifically described:

[0049] First, the shallow information F0 obtained in S1 is sent into the Fourier residual module architecture based on the frequency domain separation network. For the super-resolution task, a large receptive field in convolution is very needed, and previous work often adds depth to the small receptive field convolution. Although the calculation amount required by the small convolution kernel is smaller than that of the large convolution kernel, multiple layers of superposition still require more computing power. The present application mainly uses fast Fourier convolution (FFC) as the basic unit of the Fourier residual module architecture based on the frequency domain separation network. Fast Fourier convolution (FFC) efficiently obtains a non-local receptive field by analyzing the characteristics of the spatial domain and the frequency domain, and realizes cross-scale fusion within the convolution unit. In addition, the present application uses feature distillation operation to refine the features.

[0050] Specifically, the input features first pass through a frequency domain separation network to separate the high and low frequencies of the features. Specifically, the frequency domain separation network is composed of a Gaussian filter with a kernel size of 9x9 and a sigma of 25. The features will obtain the low-frequency information F low after the Gaussian filtering. low The high-frequency information F high of the feature map will be obtained by subtracting the low-frequency information F low from the original feature map. high The obtained low-frequency information F low and high-frequency information F high are processed differently. As analyzed in the foregoing, the high-frequency information of the high-resolution image is severely damaged in the downsampling process, while the low-frequency information is less lost. Therefore, using the same computing resources for high and low frequency information will cause computational redundancy. In fact, the reconstruction of high-frequency information requires a large amount of computing resources, while the low-frequency information does not. Therefore, a simple network is directly used for extracting the low-frequency information F low . This simple network includes a 3x3 FFC, a GELU activation layer and a 1x1 FFC; and the high-frequency information F high is sent into a specially designed high-frequency distillation module for processing.

[0051] Figure 4 is a high-frequency distillation module architecture schematic diagram provided by an embodiment of the present application, and the input high-frequency information F high is first split according to the channel.

[0052] Specifically, the input feature F high with 50 channels is first split into two feature maps F hl1 and F hr1 with 25 channels each, and F hl1 is distilled by a 1x1 FFC to obtain the first distilled feature F hd1 , and F hr1 is further extracted by a Fast Fourier Module (FFCB) which is composed of a 3x3 FFC and a GELU activation function. The shallow features are added to the features generated by the 3x3 FFC as a residual and then input into the GELU activation function to obtain the deep extracted features F hr2 . The above process is repeated on the extracted deep features F hr2 to obtain the distilled features F hd2 and the extracted deep features F hl3 , and the above process is repeated on the extracted deep features F hr3 to obtain the distilled features F hd3 and the extracted deep features F hr4 . The FFCB information extraction is performed on F hr4 for the last time, and then a 3x3 FFC is used to obtain the final feature F hl4 . Finally, the input feature F high is added to the distilled features F hd1 , F hd2 , F hd3 , and F hd4 , and then input into a 1x1 convolution for feature fusion, and then input into a spatial attention layer to further help the network focus on the structure, texture, edge, and other information that is conducive to enhancing the details of the image. The high-frequency information and the low-frequency information after processing are added to obtain the final output of the Fourier residual module architecture based on the frequency domain separation network, which is used as the input of the next module. The module is iterated 6 times in total to obtain the output F1-F6 of S2.

[0053] Figure 5 The left image in FIG. 1 is a low-resolution image provided by an embodiment of the present application. The image is reconstructed to high resolution by using the present application, and the final I HR is shown in the right image in FIG. 1. Figure 5

[0054] Those skilled in the art will readily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.​

Claims

1. A method for light image super-resolution reconstruction based on a frequency domain separation network, characterized in that, Comprise: S1: using a preset size of convolution on a low-resolution image to be processed performing shallow feature extraction to obtain shallow features ; S2: using a Fourier residual module group based on a frequency domain separation network to process the shallow features perform deep feature extraction to obtain deep features corresponding to each of the Fourier residual modules based on the frequency domain separation network. S3: merge the deep features corresponding to each of the Fourier residual modules based on the frequency domain separation network, and then perform feature fusion using convolution to obtain fused features ; S4: combining the shallow features as residual information and the fused features are combined to generate a high resolution image ; The S2 comprises: S21: The shallow features The first Fourier residual module based on the frequency domain separation network is input, and the Fourier residual module based on the frequency domain separation network performs the following operations: High and low frequency separation is performed using a frequency domain separation network to obtain the first low frequency information. and the first high frequency information ; the first low-frequency information Input the preset network to perform feature extraction to obtain the second low-frequency information; then input the first high-frequency information... The input high-frequency distillation module is split according to the channel to obtain and ; The first layer of distillation features is obtained after feature extraction using an FFC layer. ; Feature extraction is performed using the Fast Fourier Transform (FFCB) module, which includes the FFC and GELU activation functions; among them... As residuals and features generated by FFC in the FFCB The sums are fed into the GELU activation function to obtain ;Will The first high-frequency information is used as a feature to be processed. get and The process of obtaining and ;Will The first high-frequency information is used as a feature to be processed. get and The process of obtaining and ;right The final distillation features are obtained by extracting FFCB and FFC features. ; Input the first high-frequency information and The features are added together and fed into a convolutional layer for feature fusion, and then fed into a spatial attention layer to obtain the second high-frequency information; the second high-frequency information and the second low-frequency information are superimposed to obtain the deep features corresponding to the first Fourier residual module based on the frequency domain separation network. S22: input the result obtained by the S21 into a second Fourier residual error module based on the frequency domain separation network to obtain deep features corresponding to the second Fourier residual error module based on the frequency domain separation network, and obtain deep features corresponding to the third to Nth Fourier residual error modules based on the frequency domain separation network through multiple iterations.

2. The light image super-resolution reconstruction method based on the frequency domain separation network according to claim 1, wherein The basic unit of each Fourier residual error module based on the frequency domain separation network is a fast Fourier convolution (FFC); The FFC can obtain a non-local receptive field and realize cross-scale fusion within the convolution by analyzing the characteristics of the spatial domain and the frequency domain. 3.The method of claim 1, wherein, The frequency domain separation network comprises a Gaussian filter; subtracting the low-frequency information from the shallow features passing the shallow features through the Gaussian filter to obtain low-frequency information of the shallow features subtracting the low-frequency information from the shallow features obtaining the high-frequency information . 4.The method of claim 1, wherein, The preset network comprises a FFC, a GELU activation layer and a FFC connected in sequence.

5. The method of any one of claims 1-4, wherein, The S4 comprises: The shallow features are combined and fed into a convolution to convert the features to target features suitable for upsampling As residual information and the fused features Combined and fed into a convolution to convert the features to target features suitable for upsampling ; using a sub-pixel module on the target feature up-sampling to obtain a high-resolution image .

6. An apparatus for light-weight image super-resolution reconstruction based on a frequency domain separation network, characterized in that, A device for executing the light image super-resolution reconstruction method based on the frequency domain separation network according to any one of claims 1-5, comprising: a shallow extraction module configured to perform shallow feature extraction on the low resolution image using a preset size of convolution performing shallow feature extraction to obtain shallow features ; a deep layer extraction module configured to extract deep features from the shallow features using a set of Fourier residual modules based on a frequency domain separation network extract deep features from the shallow features using a set of Fourier residual modules based on a frequency domain separation network The feature merging module is configured to merge deep features corresponding to the Fourier residual modules of the frequency domain separation networks, and then perform feature fusion by using convolution to obtain fused features ; an image generation module configured to combine the shallow features as residual information and the fused features to generate a high resolution image . 7.An image processing device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to realize the steps of the method according to any one of claims 1-5.

Images