Image super-resolution lightweight neural network model based on frequency domain convolution

Abstract

The invention relates to the field of digital image processing, and discloses an image super-resolution lightweight neural network model based on frequency domain convolution, which comprises a multi-scale wavelet convolution module, a network module and a feature extraction module. The multi-scale wavelet convolution module firstly carries out point-by-point convolution to raise the dimension of a channel dimension, then carries out wavelet transformation to obtain frequency domain features including more image detail information, and then adopts different convolution kernels Ki for low-frequency, horizontal, vertical and diagonal frequency components to realize frequency domain multi-scale feature extraction. A novel multi-scale wavelet convolution module is designed, wavelet transform is combined on the basis of lightweight channel separable convolution, image features are finely proposed in a frequency domain, and different convolution kernels are adopted for processing according to features of different frequencies, so that lightweight multi-scale feature extraction is realized. Therefore, the effects of remarkably enhancing the network generalization performance and improving the picture resolution under the condition of increasing a small amount of parameters and calculation amount can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of digital image processing, and in particular to a lightweight neural network model for image super-resolution based on frequency domain convolution. Background Art

[0002] Since the 21st century, electronics, computer science, and microsystem technology have made great progress. Imaging systems have made great progress by integrating the above technologies. Imaging systems are widely used in various fields such as industry, medical treatment, and security.

[0003] However, current imaging systems often have problems such as blurred images and unclear texture details, which seriously restrict the application of various imaging systems in various fields, especially in medical and security applications that require high image quality. Super-resolution technology can be used to reconstruct the images output by the original imaging system in order to improve image quality, expand image resolution, and enhance the display range of image features. At present, super-resolution algorithms can be divided into traditional methods and deep learning-based methods. Traditional methods mainly include: difference-based methods, methods based on probability theory and set theory that use prior information for reconstruction, and dictionary learning methods based on sparse representation. Traditional methods have small computational complexity and high real-time performance, but poor reconstruction effects. Deep learning methods have good reconstruction effects, but large computational complexity and low real-time performance. Known technologies use lightweight modules such as deep separable convolution and small-size convolution to reduce the complexity and computational complexity of the network, making super-resolution networks suitable for edge computing devices. However, the problem faced by the prior art is that these lightweight modules usually cause the long-range dependence of the network to decrease, and the frequency domain information has not been considered, resulting in the lack of global information and insufficient multi-scale information. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a lightweight neural network model for image super-resolution based on frequency domain convolution, which solves the problem that the lightweight module usually causes a decrease in the long-range dependency of the network and does not consider the frequency domain information, resulting in the lack of global information and insufficient multi-scale information.

[0005] To achieve the above purpose, the present invention is implemented through the following technical solutions: a lightweight neural network model of image super-resolution based on frequency domain convolution, including a multi-scale wavelet convolution module, a network module, and a feature extraction module. The multi-scale wavelet convolution module first performs point-by-point convolution to increase the dimension of the channel dimension, and then performs wavelet transform to obtain frequency domain features, which contain more image detail information, and then uses different convolution kernels Ki for low-frequency, horizontal, vertical, and diagonal frequency components to achieve frequency domain multi-scale feature extraction.

[0006] Preferably, the multi-scale wavelet convolution module utilizes dilated convolution in order to enable the model to extract more abstract features while maintaining spatial resolution. In addition, we assign different dilation rates to different frequencies. Therefore, this introduces a sophisticated multi-scale analysis that enables the model to extract abstract features while also preserving localized information and being sensitive to various spatial scales of the image.

[0007] Preferably, in order to optimize the efficiency of information interaction, the multi-scale wavelet convolution module integrates a channel shuffling function to better redistribute information interaction between feature maps, so that the correlation between features of different scales can be more effectively utilized. In addition, when channel shuffling is used in combination with a specific dilated convolution, the common checkerboard effect problem in dilated convolution can be effectively alleviated. In this way, the smoothness and continuity of the generated feature map can be enhanced, thereby improving the performance of the overall model.

[0008] Preferably, the pseudo code of the multi-scale wavelet convolution module is as follows:

[0009] 1. Initialization parameters: including the number of input channels, the number of output channels and the selected wavelet type.

[0010] 2. Define different convolution operations:

[0011] A. Use a 3x3 convolution kernel for low-frequency coefficients with a dilation factor of 2.

[0012] B. Use a 3x3 convolution kernel for the horizontal high-frequency coefficients with a dilation factor of 2.

[0013] C. Use a 5x5 convolution kernel with a dilation factor of 3 for the vertical high-frequency coefficients.

[0014] D. Use a 5x5 convolution kernel with a diagonal high-frequency coefficient and a dilation factor of 3.

[0015] 3. Forward propagation:

[0016] A. Perform wavelet transform on the input features to obtain low-frequency coefficients (ll) and high-frequency coefficients (lh, hl, hh).

[0017] B. Apply corresponding convolution operations to different wavelet coefficients for processing.

[0018] C. Combine the processed wavelet coefficients into tuples.

[0019] D. Perform inverse wavelet transform on the processed wavelet coefficients to obtain output features.

[0020] E. Add a residual connection to add the input features to the output features.

[0021] 4. Return the final output features.

[0022] Preferably, the network module mainly includes three parts, shallow feature extraction, deep feature extraction and upsampling. The shallow feature extraction part is a layer of 3×3 convolution, and the upsampling part includes a layer of 1×1 convolution and a pixelshuffle function. The deep feature extraction part includes 10 consecutive residual groups, each of which contains a feature extraction module, a feature aggregation module, a layer of 3×3 convolution and a feature splicing module.

[0023] Preferably, the feature map obtained by the first layer of 3×3 convolution is copied into two copies, one is sent to the deep feature extraction module, and the other is sent to the splicing module before the upsampling layer to retain the shallow information.

[0024] Preferably, the feature extraction module designs a high-performance multi-branch structure, and its information processing process is shown in the following formula:

[0025]

[0026] The present invention provides a lightweight neural network model for image super-resolution based on frequency domain convolution. It has the following beneficial effects:

[0027] 1. The present invention designs a new multi-scale wavelet convolution module, combines wavelet transform on the basis of lightweight channel separable convolution, and finely proposes image features in the frequency domain. Different convolution kernels are used for processing according to the features of different frequencies, so as to realize lightweight multi-scale feature extraction. At the same time, different numbers of holes are set for different frequency components to realize adaptive receptive field. In addition, a channel shuffling mechanism is added at the end of the module to alleviate the chessboard effect caused by the hole convolution and enhance multi-scale information interaction. In terms of loss function, a frequency loss function is additionally added, which can effectively enhance the network's learning of high-frequency information, so as to significantly enhance the generalization performance of the network and improve the resolution of the picture while adding a small amount of parameters and calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a diagram of the multi-scale wavelet convolution architecture of the present invention;

[0029] Figure 2 It is the overall network structure diagram of the present invention;

[0030] Figure 3 This is a diagram of the feature extraction module architecture of the present invention;

[0031] Figure 4 Schematic diagram comparing the image reconstruction effects of different networks of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Embodiment 1:

[0034] Please see attached Figure 1 - Attachment Figure 4 The embodiment of the present invention provides a lightweight neural network model for image super-resolution based on frequency domain convolution. Compared with the existing lightweight convolution module, this method can significantly enhance the generalization performance of the network and improve the resolution of the image while adding a small amount of parameters and calculations. The specific architecture of this module is as follows Figure 1 As shown;

[0035] The multi-scale wavelet convolution module first performs point-by-point convolution to increase the channel dimension, and then performs wavelet transform to obtain frequency domain features, which contain more image detail information. Different convolution kernels Ki are then used for low-frequency, horizontal, vertical, and diagonal frequency components to achieve multi-scale feature extraction in the frequency domain.

[0036] In order to enable the model to extract more abstract features while maintaining spatial resolution, dilated convolutions are used. In addition, we assign different dilation rates to different frequencies. Therefore, this introduces a sophisticated multi-scale analysis that enables the model to extract abstract features while also preserving localized information, and is sensitive to various spatial scales of the image.

[0037] In order to optimize the efficiency of information interaction, the module integrates the channel shuffling function to better redistribute the interaction of information between feature maps, so that the correlation between features of different scales can be more effectively utilized. In addition, when channel shuffling is combined with a specific dilated convolution, the common checkerboard effect problem in dilated convolution can be effectively alleviated. In this way, the smoothness and continuity of the generated feature map can be enhanced, thereby improving the performance of the overall model.

[0038] The module pseudo code is as follows:

[0039] 1. Initialization parameters: including the number of input channels, the number of output channels and the selected wavelet type.

[0040] 2. Define different convolution operations:

[0041] a) Use a 3x3 convolution kernel with a dilation factor of 2 for the low-frequency coefficients (ll).

[0042] b) Use a 3x3 convolution kernel with a dilation factor of 2 for the horizontal high-frequency coefficients (lh).

[0043] c) Use a 5x5 convolution kernel with a dilation factor of 3 for the vertical high-frequency coefficients (hl).

[0044] d) Use a 5x5 convolution kernel with a dilation factor of 3 for the diagonal high-frequency coefficients (hh).

[0045] 3. Forward propagation:

[0046] a) Perform wavelet transform on the input features to obtain low-frequency coefficients (ll) and high-frequency coefficients (lh, hl, hh).

[0047] b) Apply corresponding convolution operations to different wavelet coefficients for processing.

[0048] c) Combine the processed wavelet coefficients into tuples.

[0049] d) Perform inverse wavelet transform on the processed wavelet coefficients to obtain output features.

[0050] e) Add a residual connection to add the input features to the output features.

[0051] 4. Return the final output features.

[0052] The network structure is as follows Figure 2 and Figure 3 As shown:

[0053] The network mainly consists of three parts: shallow feature extraction, deep feature extraction and upsampling. The shallow feature extraction part is a layer of 3×3 convolution, and the upsampling part includes a layer of 1×1 convolution and a pixelshuffle function. The deep feature extraction part includes 10 consecutive residual groups, each of which contains a feature extraction module, a feature aggregation module, a layer of 3×3 convolution and a feature splicing module.

[0054] The feature map obtained after the first layer of 3×3 convolution is copied into two copies, one is sent to the deep feature extraction module, and the other is sent to the splicing module before the upsampling layer to retain the shallow information.

[0055] The feature extraction module designs a high-performance multi-branch structure, and its information processing process is shown in the following formula:

[0056]

[0057] In the above formula, F in Represents the module input, (i=1,2,3,4,5;j=w,r) represents the left branch (w) and right branch (r) of the i-th layer in the module; (j=1,2,3,4,5; k=1, Conv, MKDGC-CS) represents the size and type of convolution kernel used in the jth layer. The information input to this module will be passed through the parallel 1×1 convolution on the left and the 3×3 multi-scale hole mixed group convolution on the right. The left branch is used to compress the number of feature channels, and the right branch is used to further refine the features. F3, F5 is used for splicing, and then a 1×1 convolutional layer is used to fuse the information obtained at each stage.

[0058] The feature aggregation part first passes the input through a 1×1 convolution layer, establishes two shortcut connections for this layer, and adds a 1×1 convolution layer to the left shortcut branch. Then it passes through a 7×7 maximum pooling layer. After that, these feature information is sent to two multi-scale convolution layers of different proportions, with convolution kernel sizes of 7×7, 5×5 and 5×5, 3×3, and void factors of 3 and 2. Then, it is connected to a 1×1 convolution layer. Finally, the sigmoid function is used to generate the attention matrix. The corresponding element multiplication operation is performed with the original input feature to achieve the reselection of features in space.

[0059] The last part is the upsampling reconstruction part of the network. The structure of this part includes a 1×1 convolution layer and a pixel shuffle layer.

[0060] The model is trained using the most commonly used datasets DIV2K and Flicker2K in the current SR field. The most widely used datasets Set5, Set14, BSD100, Urban100 and Manga109 are used for testing. Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used to evaluate the quality of high-resolution images generated by the network.

[0061] During the training phase of the network, we split the original input image into 64×64 size and the batch size is 64. The network contains 10 FASP modules, and the number of channels of each FASP module is 56. The AdamW optimizer is used with parameters β1=0.9, β2=0.999 and ε=10-8. The performance of the model is evaluated using the L1 loss function and the frequency loss function. The initial learning rate is 0.001 and the termination learning rate is 2e-4. We use the Pytorch framework for training and testing on the Windows 10 operating system, and the hardware platform is i7-12700 CPU and Nvidia RTX4090 GPU. Each training takes about 25 hours.

[0062] A frequency loss function is additionally added to the loss function, as shown in the following formula:

[0063] L=L1(P,T)+αL1(FFT(P),FFT(T))

[0064] Where L1(·) represents the L1 loss function, FFT(·) represents the fast Fourier transform, and α is set to 0.05 to ensure that the two loss functions are maintained at the same order of magnitude. P is the predicted value and T is the true value.

[0065] Table 1 Evaluation indicators of different networks on 5 datasets at X2, X3, and X4 resolutions. Red represents the best performance.

[0066]

[0067]

[0068] From the above table, we can find that the proposed network achieves 27 optimal indicators out of 30 indicators while maintaining the minimum number of parameters.

[0069] Figure 4 HR stands for real high-definition image. Figure 4 It can be found that the texture details of the network proposed in the present invention are clearer than those of the other six networks.

[0070] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A lightweight neural network model for image super-resolution based on frequency domain convolution, including a multi-scale wavelet convolution module, a network module, and a feature extraction module. The multi-scale wavelet convolution module first performs point-by-point convolution to increase the dimension of the channel, and then performs wavelet transform to obtain frequency domain features, which contain more image detail information, and then uses different convolution kernels Ki for low-frequency, horizontal, vertical, and diagonal frequency components to achieve multi-scale feature extraction in the frequency domain.

2. The lightweight neural network model for image super-resolution based on frequency domain convolution according to claim 1, characterized in that: The multi-scale wavelet convolution module uses dilated convolution to enable the model to extract more abstract features while maintaining spatial resolution. In addition, we assign different dilation rates to different frequencies. Therefore, this introduces a sophisticated multi-scale analysis that enables the model to extract abstract features while also preserving local information and is sensitive to various spatial scales of the image.

3. The lightweight neural network model for image super-resolution based on frequency domain convolution according to claim 1, characterized in that: In order to optimize the efficiency of information interaction, the multi-scale wavelet convolution module integrates a channel shuffling function to better redistribute information interaction between feature maps, so that the correlation between features of different scales can be more effectively utilized. In addition, when channel shuffling is combined with a specific dilated convolution, the common checkerboard effect problem in dilated convolution can be effectively alleviated. In this way, the smoothness and continuity of the generated feature map can be enhanced, thereby improving the performance of the overall model.

4. The lightweight neural network model for image super-resolution based on frequency domain convolution according to claim 1, characterized in that: The pseudo code of the multi-scale wavelet convolution module is as follows: 1). Initialization parameters: including the number of input channels, the number of output channels and the selected wavelet type; 2) Define different convolution operations: A. Use a 3x3 convolution kernel for low-frequency coefficients, with a dilation factor of 2. B. Use a 3x3 convolution kernel for the horizontal high-frequency coefficients, with a dilation factor of 2. C. Use a 5x5 convolution kernel for the vertical high-frequency coefficients, with a dilation factor of 3. D. Use a 5x5 convolution kernel for the diagonal high-frequency coefficients, with a dilation factor of 3. 3). Forward propagation: A. Perform wavelet transform on the input features to obtain low-frequency coefficients (ll) and high-frequency coefficients (lh, hl, hh), B. Apply corresponding convolution operations to different wavelet coefficients for processing, C. Combine the processed wavelet coefficients into tuples, D. Perform inverse wavelet transform on the processed wavelet coefficients to obtain the output features. E. Add residual connection to add input features to output features. 4). Return the final output features.

5. The lightweight neural network model for image super-resolution based on frequency domain convolution according to claim 1, characterized in that: The network module mainly includes three parts: shallow feature extraction, deep feature extraction and upsampling. The shallow feature extraction part is a 3×3 convolution layer, and the upsampling part includes a 1×1 convolution layer and a pixel shuffle function. The deep feature extraction part includes 10 consecutive residual groups, each of which contains a feature extraction module, a feature aggregation module, a 3×3 convolution layer and a feature splicing module.

6. The lightweight neural network model for image super-resolution based on frequency domain convolution according to claim 5, characterized in that: The feature map obtained after the first layer of 3×3 convolution is copied into two copies, one is sent to the deep feature extraction module, and the other is sent to the splicing module before the upsampling layer to retain the shallow information.

7. The lightweight neural network model for image super-resolution based on frequency domain convolution according to claim 1, characterized in that: The feature extraction module designs a high-performance multi-branch structure, and its information processing process is shown in the following formula:

Images