Image super-resolution reconstruction method and device based on multi-domain information enhancement

Abstract

The invention discloses an image super-resolution reconstruction method and device based on multi-domain information enhancement, and relates to the technical field of single image super-resolution reconstruction. A multi-path large kernel separable attention module, a multi-path large kernel attention module, a high-frequency feature enhancement module and a local information enhancement attention module are constructed; performing layer normalization integration to form a multi-domain information enhanced feature extraction block so as to form a multi-domain information enhanced lightweight image super-resolution reconstruction network model, and training parameter sets in a shallow layer feature extraction module, a deep layer feature extraction module and a reconstruction module in the model respectively; the four modules in the model focus on local information, high-frequency information, long-range information and frequency components in the transverse and longitudinal directions in the image, so that the quality of the network reconstruction image can be improved from the four aspects of overall layout, detail information, texture information and object image in the image, and the quality of the image is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of single image super-resolution reconstruction, and in particular to an image super-resolution reconstruction method and device based on multi-domain information enhancement. Background Art

[0002] Image super-resolution reconstruction technology (Super Resolution, SR) refers to the recovery of corresponding high-resolution (High Resolution, HR) images with rich details and better visual quality from one or more blurred, unclear low-resolution (Low Resolution, LR) images of the same scene; this technology is based on complex mathematical models and algorithms, and can effectively increase the spatial resolution of the image, making the image present clearer and finer details; compared with the method of improving hardware equipment to obtain high-resolution images, this software-level technology eliminates the need to use higher-end equipment to re-shoot and produce, saving a lot of costs, and has a wider range of applications and lower costs.

[0003] In recent years, convolutional neural networks have been widely used in the field of super-resolution. Unlike traditional learning algorithms that use artificially selected features, algorithms based on convolutional neural networks aim to use end-to-end learning to generate deep features in a task-oriented manner. Compared with traditional algorithms, the performance has been greatly improved, and very significant reconstruction effects have been achieved. At the same time, due to the good positive correlation between the network depth and network performance of super-resolution reconstruction algorithms based on convolutional neural networks, various deep SR models have been proposed. However, the depth of the network is also accompanied by an increase in the number of parameters and computational complexity. Specifically, the 3-layer SRCNN only requires 57K parameters and 52.7G of computational complexity, while EDSR, one of the advanced (SOTA) models with deeper layers, requires 40.7M parameters and 9384.7G of computational complexity. Although the deepening of the network improves performance, the resulting computational load limits real-world applications, such as on mobile devices or in real-time SR.

[0004] For practical applications considering limited computing resources, the demand for lightweight networks has been growing; the key issue in building lightweight networks is to balance the network performance and the cost in terms of the number of parameters (memory complexity) or floating-point operations (time complexity); since simply reducing the depth or width of the network will lead to severe performance degradation, it is necessary to carefully design the optimization of the network structure to simplify the network structure while maintaining performance; however, high-performance image super-resolution networks often have a large number of parameters. If a lightweight network with reduced parameters and computational complexity is used, the reconstructed image will usually have structural distortion and unclear texture, resulting in poor image quality, which is difficult to meet the growing performance requirements of modern consumers for mobile devices and applications. Summary of the invention

[0005] The embodiment of the present invention provides an image super-resolution reconstruction method and device based on multi-domain information enhancement, which can solve the problem in the prior art that a lightweight network with reduced parameter amount and computation amount is used, but the reconstructed image usually has structural distortion and unclear texture, resulting in poor image quality.

[0006] The embodiment of the present invention provides an image super-resolution reconstruction method based on multi-domain information enhancement, comprising the following steps:

[0007] Obtaining a low-resolution natural image to be reconstructed;

[0008] A multi-way large core separable attention module is constructed by cascading two convolutional layers, one point convolution, one GELU activation function and three sets of large core separable attention LKSA of different scales; a multi-way large core attention module is constructed by cascading two convolutional layers, one point convolution, one GELU activation function and three sets of large core attention LKA of different scales; a high-frequency feature enhancement module is constructed by cascading one depth-wise separable convolution, two point-wise convolutional layers, two GELU activation functions and one maximum pooling layer; a local information enhancement attention module is constructed by cascading four point-wise convolutions, three depth-wise separable convolutions and one sigmoid activation function; and the multi-way large core separable attention module, the multi-way large core attention module, the high-frequency feature enhancement module and the local information enhancement attention module are integrated with the layer normalization LN to form a multi-domain information enhancement feature extraction block containing different branches;

[0009] A multi-domain information enhanced lightweight image super-resolution reconstruction network model is constructed using a multi-domain information enhanced feature extraction block; the DIV2K and Flickr2K datasets are used to train the parameter sets in the shallow feature extraction module containing the convolution layer, the deep feature extraction module containing multiple multi-domain information enhanced feature extraction blocks, and the reconstruction module containing the convolution layer and the pixel shuffling layer in the multi-domain information enhanced lightweight image super-resolution reconstruction network model, respectively, to obtain the trained multi-domain information enhanced lightweight image super-resolution reconstruction network model;

[0010] The low-resolution natural image to be reconstructed is input into the trained multi-domain information enhanced lightweight image super-resolution reconstruction network model, and the image features are extracted through the convolution layer in the shallow feature extraction module to obtain a shallow feature map; the image features are extracted through the LN layer in the multiple multi-domain information enhanced feature extraction blocks cascaded in the deep feature extraction module, multiple convolution layers and point convolutions of different scales, activation functions and different attention mechanisms to obtain a deep feature map; the feature elements in the shallow feature map and the deep feature map are added through the convolution layer and the pixel shuffling layer in the reconstruction module, and pixel reconstruction is performed to obtain a high-resolution image of the image.

[0011] Preferably, the construction of the multi-channel large-core separable attention module includes:

[0012] The multi-way large kernel separable attention module consists of two 3×3 convolutional layers, 1×1 point convolution, a GELU activation function and three sets of large kernel separable attention LKSA of different scales;

[0013] The sizes of the three groups of large-core separable attentions of different scales are: 3-5-1, 5-7-1, and 7-9-1; they represent cascaded a×1 and 1×a depthwise separable convolutions, cascaded b×1 and 1×b depthwise dilated convolutions, and 1×1 point convolution PConv;

[0014] The input features of the multi-way large-core separable attention module first pass through a 3×3 convolution layer and obtain a new feature map after the GELU activation function; then, starting from the horizontal and vertical directions, three groups of large-core separable attentions of different scales are passed to obtain the corresponding three groups of feature maps; these three groups of feature maps are spliced ​​in the channel dimension, and dimensionality reduction is performed using 1×1 convolution, and then a comprehensive feature map is generated through a 3×3 convolution layer; this feature map is fused with the original input features by element-wise addition through jump connections to output the final feature map.

[0015] Preferably, the construction of the multi-channel large-core attention module includes:

[0016] The multi-channel large-core attention module consists of two 3×3 convolutional layers, 1×1 point convolution, a GELU activation function, and three sets of large-core attention LKAs of different scales;

[0017] The sizes of the three groups of large core attentions of different scales are: 3-5-1, 5-7-1, 7-9-1; ab-1 represents the cascade of a×a depth convolution DWConv, b×b depth expansion convolution DPConv and point convolution PConv;

[0018] The input features of the multi-channel large-core attention module are first passed through a 3×3 convolution layer and a GELU activation function to obtain new features, and then passed through three groups of large-core attentions of different scales to obtain three different groups of features. The three groups of feature maps are concatenated by channel and the concatenated features are reduced in dimension using a 1×1 point convolution. The long-range information features are then passed through a 3×3 convolution layer. Finally, the long-range information features are element-wise added to the original input features through a jump connection to obtain the final output features.

[0019] Preferably, the construction of the high-frequency feature enhancement module includes:

[0020] The high-frequency feature enhancement module consists of a 5×5 depthwise separable convolution, two 1×1 pointwise convolution layers, two GELU activation functions, and a maximum pooling layer;

[0021] The input features of the high-frequency feature enhancement module are first divided into two parts with the same number of channels, and then a 5×5 depthwise separable convolution layer and a GELU activation function are used to extract local high-frequency features. A maximum pooling layer is used to capture high-frequency information, and a 1×1 convolution layer combined with a GELU activation function is used to further enhance the captured high-frequency features to obtain enhanced features. Finally, the enhanced high-frequency features under the two channels are concatenated and fused, and the final output features are obtained through a 1×1 convolution layer.

[0022] Preferably, the construction of the local information enhanced attention module includes:

[0023] The local information enhanced attention module consists of 4 1×1 point convolutions, 3 5×5 depth-wise separable convolutions, and a Sigmoid activation function;

[0024] The input features of the local information enhanced attention module are first generated through a local feature extraction branch consisting of a 1×1 convolution and a 5×5 deep convolution; at the same time, the input features also pass through a local feature enhancement branch, which first generates features through a 1×1 convolution; the features pass through a pixel-level attention branch consisting of a 1×1 convolution and a Sigmoid activation function to generate an attention map, and multiply it with the features generated by the 5×5 deep convolution to obtain enhanced features; the features pass through a 5×5 deep convolution again to obtain local enhanced features, the features of the two branches are fused through a feature splicing operation, and channel dimensionality reduction is performed through a 1×1 convolution layer to obtain the final output features.

[0025] Preferably, the shallow feature extraction module comprises:

[0026] The shallow feature extraction module consists of a 3×3 convolution layer with 48 convolution kernels and a step size of 1;

[0027] The output of the shallow feature extraction module is the shallow features obtained by processing the input low-resolution image through a 3×3 convolution layer.

[0028] Preferably, the deep feature extraction module comprises:

[0029] The deep feature extraction module is composed of 6 multi-domain information enhancement feature extraction blocks MDIEB cascaded, and the number of input and output feature maps of each MDIEB is 48;

[0030] The output of the deep feature extraction module is the deep feature obtained after the shallow feature is extracted by 6 MDIEB features.

[0031] Preferably, the reconstruction module comprises:

[0032] The reconstruction module is composed of a 3×3 convolution layer, a pixel shuffle layer, and a 3×3 convolution layer cascaded together, with 48 convolution kernels and a step size of 1;

[0033] The input of the reconstruction module is the result of summing the shallow feature and the deep feature elements, and the output is a reconstructed high-resolution image.

[0034] Preferably, in the training of the multi-domain information enhanced lightweight image super-resolution reconstruction network model, the loss function used is:

[0035]

[0036] Where: θ represents the parameter set of the neural network; N is the number of training image pairs; represents the i-th real high-resolution image; Represents the i-th super-resolution image after algorithm reconstruction.

[0037] The embodiment of the present invention further provides an image super-resolution reconstruction device based on multi-domain information enhancement, comprising:

[0038] An image module, used for acquiring a low-resolution natural image to be reconstructed;

[0039] A model construction module is used to construct a multi-way large core separable attention module using two convolutional layers, one point convolution, one GELU activation function and three groups of large core separable attention LKSA cascades of different scales, to construct a multi-way large core attention module using two convolutional layers, one point convolution, one GELU activation function and three groups of large core attention LKA cascades of different scales, to construct a high-frequency feature enhancement module using a depthwise separable convolution, two point convolutional layers, two GELU activation functions and a maximum pooling layer cascade, to construct a local information enhancement attention module using four point convolutions, three depthwise separable convolutions and a sigmoid activation function cascade; and to integrate the multi-way large core separable attention module, the multi-way large core attention module, the high-frequency feature enhancement module and the local information enhancement attention module with layer normalization LN to form a multi-domain information enhancement feature extraction block containing different branches;

[0040] A lightweight image super-resolution reconstruction network model with multi-domain information enhancement is constructed using multi-domain information enhancement feature extraction blocks;

[0041] The image reconstruction module uses the DIV2K and Flickr2K datasets to train the parameter sets in the shallow feature extraction module including the convolution layer, the deep feature extraction module including multiple multi-domain information enhanced feature extraction blocks, and the reconstruction module including the convolution layer and the pixel shuffling layer in the multi-domain information enhanced lightweight image super-resolution reconstruction network model, respectively, to obtain the trained multi-domain information enhanced lightweight image super-resolution reconstruction network model;

[0042] The low-resolution natural image to be reconstructed is input into the trained multi-domain information enhanced lightweight image super-resolution reconstruction network model, and the image features are extracted through the convolution layer in the shallow feature extraction module to obtain a shallow feature map; the image features are extracted through the LN layer in the multiple multi-domain information enhanced feature extraction blocks cascaded in the deep feature extraction module, multiple convolution layers and point convolutions of different scales, activation functions and different attention mechanisms to obtain a deep feature map; the feature elements in the shallow feature map and the deep feature map are added through the convolution layer and the pixel shuffling layer in the reconstruction module, and pixel reconstruction is performed to obtain a high-resolution image of the image.

[0043] The embodiment of the present invention provides an image super-resolution reconstruction method and device based on multi-domain information enhancement. Compared with the prior art, the advantages thereof are as follows:

[0044] The present invention constructs a multi-channel large core separable attention module, a multi-channel large core attention module, a high-frequency feature enhancement module and a local information enhancement attention module from the perspectives of spatial domain, frequency domain and conversion domain, and integrates the multi-channel large core separable attention module, the multi-channel large core attention module, the high-frequency feature enhancement module and the local information enhancement attention module with layer normalization to form a multi-domain information enhancement feature extraction block containing different branches to construct a multi-domain information enhanced lightweight image super-resolution reconstruction network model, and trains the parameter sets in the shallow feature extraction module, the deep feature extraction module and the reconstruction module in the model respectively, so that the multi-channel large core attention module contained in the model focuses on the long-range information in the image, the local information enhancement attention module focuses on the local information in the image, the high-frequency feature enhancement module focuses on the high-frequency information in the image, and the multi-channel large core separable attention module focuses on the frequency components in the horizontal and vertical directions of the image, thereby improving the quality of the network reconstructed image from four aspects: the overall layout, detail information, texture information and object image in the image, so that the quality of the image is greatly improved.

[0045] In addition, the network model trained by the present invention has fewer parameters, achieves an optimal balance between model size and performance, and ensures wide applicability and practicality on various devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1A schematic diagram of the overall process of an image super-resolution reconstruction method and device based on multi-domain information enhancement provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the network structure of an image super-resolution reconstruction method and device based on multi-domain information enhancement provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram for comparing the results of an image super-resolution reconstruction method and device based on multi-domain information enhancement provided in an embodiment of the present invention; wherein: the image on the left is Img004 in the Urban100 dataset; the image on the right is the results of different lightweight image super-resolution reconstruction algorithms. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0050] See also Figure 1 to Figure 3 The embodiment of the present invention provides an image super-resolution reconstruction method based on multi-domain information enhancement, comprising the following steps:

[0051] Step 1: Design a multi-channel large core attention module, a local information enhanced attention module, a multi-channel large core, a high-frequency feature enhancement module, and a separable attention module; and use them to enhance the network's focus on long-range information, local information, high-frequency information, and frequency components in the horizontal and vertical directions in the image.

[0052] Step 2: Integrate the multi-channel large core attention module, local information enhanced attention module, multi-channel large core, high-frequency feature enhancement module, separable attention module and layer normalization together to form a multi-domain information enhanced feature extraction block.

[0053] Step 3: Use the multi-domain information enhanced feature extraction block to build a multi-domain information enhanced lightweight image super-resolution reconstruction network, which improves the quality of the network reconstructed image from four aspects: the overall layout of the image, the detail information in the image, the texture information in the image, and the image of the object in the image.

[0054] Step 4: Use the standard DIV2K or Flickr2K dataset to complete the final training of the lightweight image super-resolution reconstruction network model with multi-domain information enhancement.

[0055] Step 5: Input the low-resolution natural image to be reconstructed into the trained multi-domain information enhanced lightweight image super-resolution reconstruction network to obtain a high-resolution image of the image.

[0056] Specifically, the steps are as follows:

[0057] Step 1: Construct a multi-path large separable kernel attention module (Multi-path Large Separable Kernel Attention, MLKSA). This module uses a multi-path structure and large separable kernel attention technology (LKSA) to focus on the frequency components in the horizontal and vertical directions. The separable attention mechanism allows the model to effectively process and analyze the directional characteristics of the image in different horizontal and vertical directions, that is, different bands (different frequency ranges) while keeping the number of parameters controllable. It naturally imposes directional constraints on pixels and focuses on the image of objects in the image, thereby improving the quality of image reconstruction.

[0058] Step 2: Construct a multi-path large kernel attention module (Multi-path Large Kernel Attention, MLKA), which is responsible for processing the long-range information of the image; it adopts a multi-path structure and combines the large kernel attention (LKA) technology to expand the receptive field, so as to better capture the global dependencies in the image; the attention mechanism is used to emphasize more valuable features and improve the accuracy of reconstruction.

[0059] Step 3: Construct a high-frequency feature enhancement module (High Frequency Information Enhancement Block, HFEB), which aims to enhance the high-frequency information in the image, that is, those parts containing rich textures and sharp edges; by analyzing and enhancing high-frequency features, the details and texture information of the reconstructed image are further improved, thereby enhancing the visual effect.

[0060] Step 4: Construct a local information enhancement attention module (LIEA), which focuses on enhancing the local information of the image. Through point convolution, depth convolution and pixel attention, the network's ability to extract local information is enhanced, so that the network can focus on local key information and establish local associations in the image, so that the network can focus on and enhance the ability to reconstruct image details, such as edges and texture details.

[0061] Step 5: Combining the above modules, the MLKA, LIEA, HFEB and MLKSA modules and layer normalization (LN) are integrated together to form a Multi-Domain Information Enhancement Feature Extraction Block (MDIEB); each module is optimized for its specific information domain, and they work together to achieve high-quality image reconstruction.

[0062] Step 6: Construct a lightweight image super-resolution reconstruction network with multi-domain information enhancement, which consists of the following three cascaded parts: a shallow feature extraction module consisting of 3×3 convolutional layers, a deep feature extraction module consisting of 6 multi-domain information enhancement modules (MDIEM) cascaded, and a reconstruction module consisting of a 3×3 convolutional layer, a pixel shuffling layer, and a 3×3 convolutional layer cascaded.

[0063] Step 7: Training and testing. The proposed network is trained using the standard DIV2K or Flickr2K datasets. Then, its reconstruction performance is evaluated using the Y-channel average peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) on four standard test sets: Set5, Set14, B100, and Urban100, to ensure its versatility and robustness on a variety of natural images. The advantages of the proposed method are verified by comparing the performance of the existing technology and this technology under the same conditions.

[0064] Step 8: Input the low-resolution natural image to be reconstructed into the trained multi-domain information enhanced lightweight image super-resolution reconstruction network to obtain a high-resolution image of the image.

[0065] In step 1 of building the multi-way large kernel separable attention module (MLKSA):

[0066] The structure of MLKSA is as follows Figure 2 As shown in the figure, it consists of two 3×3 convolutional layers, 1×1 point convolution, a GELU activation function and three groups of large kernel separable attention (LKSA) of different scales; the sizes of the three groups of large kernel separable attention are: 3-5-1, 5-7-1, and 7-9-1, where ab-1 represents the cascaded a×1 and 1×a depthwise separable convolutions, the cascaded b×1 and 1×b depthwise dilated convolutions, and 1×1 point convolution (PConv); the number of convolution kernels is set to 48, and the number of depthwise convolution groups is the same as the number of convolution kernels, which is also 48.

[0067] The workflow of MLKSA is as follows: the input features first pass through a 3×3 convolution layer and undergo a nonlinear transformation through the GELU activation function to obtain a new feature map; then, starting from the horizontal and vertical directions, three sets of large-kernel separable attention mechanisms of different scales are used to learn the long-range dependencies in space to obtain the corresponding three sets of feature maps; then, these three sets of feature maps are concatenated in the channel dimension, and dimensionality reduction is performed using a 1×1 convolution, and then a comprehensive feature map is generated through a 3×3 convolution layer; finally, this feature map is fused with the original input features by element-wise addition through a jump connection to form the final output feature of MLKSA.

[0068] The output features of MLKSA focus on the image of objects in the image, thereby improving the quality of image reconstruction.

[0069] In step 2, constructing the multi-way large kernel attention module (MLKA):

[0070] The structure of MLKA is as follows Figure 2 As shown in the figure, it consists of two 3×3 convolutional layers, 1×1 point convolution, a GELU activation function and three groups of large kernel attention (LKA) of different scales; the sizes of the three groups of large kernel attention are: 3-5-1, 5-7-1, 7-9-1, where ab-1 represents the cascade of a×a depth convolution (DWConv), b×b depth expansion convolution (DPConv) and point convolution (PConv); the number of convolution kernels is set to 48, and the number of depth convolution groups is the same as the number of convolution kernels, which is also 48.

[0071] The workflow of MLKA is as follows: the input features first pass through a 3×3 convolution layer and a GELU activation function to obtain new features, and then pass through three groups of large-core attentions of different scales to learn spatial long-range relations to obtain three different sets of features. The three sets of feature maps are then concatenated by channel and the concatenated features are reduced in dimension using a 1×1 point convolution. The long-range information features are then obtained through a 3×3 convolution layer. Finally, the long-range information features are element-wise added to the original input features through a jump connection to obtain the final MLKA output features.

[0072] The output features of MLKA contain enhanced long-range information, such as the overall structure and content of the image.

[0073] In step 3, constructing the high-frequency feature enhancement module (HFEB):

[0074] The structure of HFEB is as follows Figure 2 As shown in the figure, it consists of a 5×5 depth-wise separable convolution, two 1×1 point convolution layers, two GELU activation functions, and a maximum pooling layer.

[0075] The workflow of HFEB is as follows: the input features are first split into two parts with the same number of channels, and then a 5×5 depthwise separable convolution layer and a GELU activation function are used to extract local high-frequency features; then, a maximum pooling layer is used to capture high-frequency information, and these high-frequency features are further enhanced through a 1×1 convolution layer combined with a GELU activation function to obtain enhanced features; finally, the features of the two branches are concatenated and fused, and a 1×1 convolution layer is used to obtain the final output features of HFEB.

[0076] The output features of HFEB contain enhanced high-frequency detail information, such as texture information.

[0077] In step 4, construct the local information enhanced attention module (LIEA):

[0078] The structure of LIEA is as follows Figure 2 As shown in the figure, it consists of 4 1×1 point convolutions, 3 5×5 depth-wise separable convolutions and a Sigmoid activation function; the 1×1 point convolution and the Sigmoid activation function constitute the pixel attention; the number of convolution kernels is set to 48, the step size is set to 1, and the number of depth-wise convolution groups is the same as the number of convolution kernels, which is also 48.

[0079] The workflow of LIEA is as follows: the input features first generate features through a local feature extraction branch consisting of a 1×1 convolution and a 5×5 deep convolution; at the same time, the input features also pass through a local feature enhancement branch, which first generates features through a 1×1 convolution; then, the features pass through a pixel-level attention branch consisting of a 1×1 convolution and a Sigmoid activation function to generate an attention map, and multiply it with the features generated by the 5×5 deep convolution to obtain enhanced features; then, the features pass through a 5×5 deep convolution again to obtain local enhanced features; finally, the features of the two branches are fused through a feature splicing operation, and channel dimensionality reduction is performed through a 1×1 convolution layer; in addition, in order to maintain the stability of training and prevent information loss, jump connections are introduced to obtain the final output features of LIEA.

[0080] The output features of LIEA contain enhanced local information, such as edges and texture details.

[0081] In step 5, the above modules are integrated to construct a multi-domain information enhancement feature extraction block (MDIEB):

[0082] The MLKA, LIEA, HFEB and MLKSA modules and layer normalization (LN) are integrated together to form a multi-domain information enhanced feature extraction block (MDIEB), whose structure is as follows: Figure 2 As shown in the figure, the number of input and output feature maps of MDIEB is 48.

[0083] The workflow of MDIEB is as follows: the input features are first processed by two parallel feature extraction branches: the first branch is composed of LN layer, MLKSA and MLKA cascade, and the input features are extracted to obtain new features; the second branch is composed of HFEB, and the input features are extracted to obtain high-frequency enhanced features; then, in order to make full use of multi-scale information and maintain the stability of the training process, a skip connection mechanism is introduced, that is, the features from two different branches and the original input features are added element by element; finally, the result is passed to the LIEA module to generate the final output features of MDIEB.

[0084] Each module of MDIEB is optimized for its specific information domain, and the output features enhance the overall structure of the image, detail information, texture features and object contours.

[0085] In the shallow feature extraction module of the network model of the lightweight image super-resolution reconstruction algorithm with multi-domain information enhancement in step 6:

[0086] The shallow feature extraction module consists of a 3×3 convolutional layer with 48 convolution kernels and a step size of 1.

[0087] The output of the shallow feature extraction module is the shallow features obtained by passing the input low-resolution image through only 3×3 convolution layers.

[0088] In step 6, in the deep feature extraction module of the lightweight image super-resolution reconstruction algorithm network model with multi-domain information enhancement:

[0089] The deep feature extraction module is composed of 6 MEDIB cascades, and the number of input and output feature maps of each MEDIB is 48.

[0090] The output of the deep feature extraction module is the deep features obtained after the shallow features are extracted by 6 MEDIB features.

[0091] In the reconstruction module of step 6 of constructing a lightweight image super-resolution reconstruction algorithm network model with multi-domain information enhancement:

[0092] The reconstruction module is composed of a 3×3 convolutional layer, a pixel shuffle layer, and a 3×3 convolutional layer cascaded together, with 48 convolution kernels and a step size of 1.

[0093] The input of the reconstruction module is the result of removing the shallow features and deep feature elements, and the output is a reconstructed high-resolution image.

[0094] In step 6, in constructing a lightweight image super-resolution reconstruction algorithm network model with multi-domain information enhancement:

[0095] The network model is composed of a cascade of shallow feature extraction modules, deep feature extraction modules, and reconstruction modules.

[0096] The input of the network model is a low-resolution image, and the output is a reconstructed high-resolution image.

[0097] In step 7 of model training and testing:

[0098] The proposed network is trained using the standard DIV2K or Flickr2K datasets.

[0099] L1 loss is used as the loss function of the entire model. The L1 loss function is sensitive to data and can promote rapid convergence of the model. The specific calculation is:

[0100]

[0101] Where: θ represents the parameter set of the neural network; N is the number of training image pairs; represents the i-th real high-resolution image; Represents the i-th super-resolution image after algorithm reconstruction.

[0102] Its reconstruction performance is evaluated on four standard test sets: Set5, Set14, B100, and Urban100.

[0103] The Y channel average peak signal-to-noise ratio (PSNR) and result similarity (SSIM) are used as evaluation indicators.

[0104] For the reconstructed image I SR and original high-definition images I HR , the peak signal-to-noise ratio is calculated as follows:

[0105]

[0106] Where: H and W represent I SR ,I HR The height and width of an image pair; a larger PSNR value indicates a higher quality reconstructed image.

[0107] Structural similarity evaluates the similarity between two images from three aspects: brightness, contrast, and structure. The closer the value is to 1, the better the quality of image reconstruction. The calculation formula is as follows:

[0108]

[0109] Where: u HR with u SR Indicates I HR ,I SR The grayscale average value of HR With σ SR Indicates IHR ,I SR The variance of HR-SR Indicates I SR ,I HR covariance; c1, c2, c3 are non-zero constants.

[0110] The performance of the existing technology and this technology under the same conditions are compared to verify the advantages of the proposed method.

[0111] In the reconstructed image, the low-resolution natural image to be reconstructed is input into a trained lightweight image super-resolution reconstruction network with multi-domain information enhancement to obtain a high-resolution image of the image.

[0112] The present invention is described in detail below in combination with performance tests and experimental analysis, and the evaluation indicators adopted are PSNR and SSIM which are widely used in the current super-resolution reconstruction field.

[0113] 1. The execution details of the simulation experiment of the present invention are as follows:

[0114] During the training process, the images were randomly cropped into 96×96 blocks, the training batch size was set to 16, and the feature channels were 48. Data augmentation operations were performed at the same time, such as random horizontal flipping or 90° rotation. The Adam optimizer was used for training, with parameters β1 and β2 of 0.9 and 0.999 respectively. The initial learning rate was set to 4×10 -4 , and at iteration 2×10 5 , 4×10 5 , 6×10 5 , 8×10 5 The learning rate is reduced by half at this time, and the total number of iterations is 1×10 6 ; Use L1 loss function to optimize the model; Use Pytorch1.6 deep learning framework; The programming language is python3.7, and the experiment is conducted on a GeForce RTX 3090GPU.

[0115] 2. Simulation content and results analysis:

[0116] Table 1 compares the 4x reconstruction index of the proposed algorithm and other six existing technologies (CARN, IMDN, ESRT, FMEN, DRSAN-48m, NGswin) on four standard test sets: Set5, Set14, B100, and Urban100.

[0117] Table 1 Performance comparison of different SISR models on four benchmark datasets when the scaling factor is 4

[0118]

[0119]

[0120] like Figure 3 Comparison chart of super-resolution reconstruction results of images by the proposed algorithm and six existing technologies (CARN

[66] , IMDN

[67] , ESRT

[83] , FMEN

[89] , DRSAN-48m

[90] , NGswin

[91] ) on the challenging dataset Urban100, with a magnification of 4.

[0121] It can be seen from Table 1 that the average values ​​of PSNR and SSIM of the algorithm proposed in the present invention are higher than those of the comparison methods in the above-mentioned prior art on the four data sets. Compared with the DRSAN-48m method, the algorithm of the present invention has improved by 0.15dB and 0.26dB on the Set5 and Urban100 data sets respectively with a 20% reduction in the number of parameters, and the performance is greatly improved. Figure 3 It can be seen that the image reconstructed by the algorithm proposed in the present invention is basically similar to the texture details of the original HR image, while other methods all encounter serious structural distortion and texture blurring. The algorithm of the present invention captures more contextual information from multiple dimensions by utilizing more extensive information, thus avoiding structural distortion and texture blurring of the reconstructed image. The experimental results fully demonstrate the effectiveness and innovation of the algorithm of the present invention. At the same time, the algorithm achieves a good balance between model size, computational efficiency and reconstruction performance under the premise of a relatively low number of parameters.

[0122] The present invention improves the quality of image super-resolution reconstruction by integrating multiple innovative modules; these modules include a multi-channel large kernel attention module (MLKA), a local information enhanced attention module (LIEA), a high-frequency feature enhancement module (HFEB) and a multi-channel large kernel separable attention module (MLKSA); these modules respectively optimize the long-range information, local information, high-frequency information and frequency components in the horizontal and vertical directions in the image, thereby improving the quality of the reconstructed image in four aspects: overall layout, detail information, texture information and object image; through the comprehensive utilization of such multi-domain information, the network model of the present invention can achieve high-quality image reconstruction while maintaining high efficiency, and achieve a good balance between model performance and efficiency.

[0123] In terms of comprehensiveness, the present invention comprehensively improves the image quality of super-resolution reconstruction by comprehensively considering four key dimensions of the image's overall layout, detail restoration, texture rendering, and object morphology; in terms of feature utilization, the present invention utilizes deep mining of long-range dependencies, local structures, high-frequency details, and longitudinal and lateral frequency features to significantly enhance the ability to identify and utilize important information in the image; in terms of efficiency, after extensive experimental verification, the present algorithm is superior to the current mainstream algorithms in a number of objective evaluation indicators, effectively avoiding structural distortion and texture blurring that may occur in the reconstruction process while maintaining the model size and computational efficiency; in terms of balance, on the basis of controlling the amount of parameters, the present algorithm achieves an optimal balance between model size and performance, ensuring wide applicability and practicality on various devices.

[0124] The embodiment of the present invention further provides an image super-resolution reconstruction device based on multi-domain information enhancement, comprising:

[0125] The image module is used to obtain a low-resolution natural image to be reconstructed.

[0126] A model construction module is used to construct a multi-way large core separable attention module using two convolutional layers, one point convolution, a GELU activation function and three sets of large core separable attention LKSA cascades of different scales, to construct a multi-way large core attention module using two convolutional layers, one point convolution, a GELU activation function and three sets of large core attention LKA cascades of different scales, to construct a high-frequency feature enhancement module using a depthwise separable convolution, two point convolutional layers, two GELU activation functions and a maximum pooling layer cascade, to construct a local information enhancement attention module using four point convolutions, three depthwise separable convolutions and a Sigmoid activation function cascade; and to integrate the multi-way large core separable attention module, the multi-way large core attention module, the high-frequency feature enhancement module and the local information enhancement attention module with the layer normalization LN to form a multi-domain information enhancement feature extraction block containing different branches.

[0127] A lightweight image super-resolution reconstruction network model with multi-domain information enhancement is constructed using multi-domain information enhancement feature extraction blocks.

[0128] The image reconstruction module uses the DIV2K and Flickr2K datasets to train the parameter sets of the shallow feature extraction module containing the convolution layer, the deep feature extraction module containing multiple multi-domain information enhanced feature extraction blocks, and the reconstruction module containing the convolution layer and the pixel shuffling layer in the lightweight image super-resolution reconstruction network model with multi-domain information enhancement, and obtains the trained lightweight image super-resolution reconstruction network model with multi-domain information enhancement.

[0129] The low-resolution natural image to be reconstructed is input into the trained multi-domain information enhanced lightweight image super-resolution reconstruction network model, and the image features are extracted through the convolution layer in the shallow feature extraction module to obtain a shallow feature map; the image features are extracted through the LN layer in the multiple multi-domain information enhanced feature extraction blocks cascaded in the deep feature extraction module, multiple convolution layers and point convolutions of different scales, activation functions and different attention mechanisms to obtain a deep feature map; the feature elements in the shallow feature map and the deep feature map are added through the convolution layer and the pixel shuffling layer in the reconstruction module, and pixel reconstruction is performed to obtain a high-resolution image of the image.

[0130] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An image super-resolution reconstruction method based on multi-domain information enhancement, characterized in that: The following steps are involved: Obtaining a low-resolution natural image to be reconstructed; A multi-way large-core separable attention module is constructed by cascading two convolutional layers, one point convolution, one GELU activation function and three sets of large-core separable attention LKSA of different scales. A multi-way large-core attention module is constructed by cascading two convolutional layers, one point convolution, one GELU activation function and three sets of large-core attention LKA of different scales. A high-frequency feature enhancement module is constructed by cascading one depth-separable convolution, two point convolution layers, two GELU activation functions and one maximum pooling layer. A local information enhancement attention module is constructed by cascading four point convolutions, three depth-separable convolutions and a sigmoid activation function. The multi-channel large core separable attention module, the multi-channel large core attention module, the high-frequency feature enhancement module and the local information enhancement attention module are integrated with the layer normalization LN to form a multi-domain information enhancement feature extraction block containing different branches; A multi-domain information enhanced lightweight image super-resolution reconstruction network model is constructed using a multi-domain information enhanced feature extraction block; the DIV2K and Flickr2K datasets are used to train the parameter sets in the shallow feature extraction module containing the convolution layer, the deep feature extraction module containing multiple multi-domain information enhanced feature extraction blocks, and the reconstruction module containing the convolution layer and the pixel shuffling layer in the multi-domain information enhanced lightweight image super-resolution reconstruction network model, respectively, to obtain the trained multi-domain information enhanced lightweight image super-resolution reconstruction network model; The low-resolution natural image to be reconstructed is input into the trained multi-domain information enhanced lightweight image super-resolution reconstruction network model, and the image features are extracted through the convolution layer in the shallow feature extraction module to obtain a shallow feature map; the image features are extracted through the LN layer in the multiple multi-domain information enhanced feature extraction blocks cascaded in the deep feature extraction module, multiple convolution layers and point convolutions of different scales, activation functions and different attention mechanisms to obtain a deep feature map; the feature elements in the shallow feature map and the deep feature map are added through the convolution layer and the pixel shuffling layer in the reconstruction module, and pixel reconstruction is performed to obtain a high-resolution image of the image.

2. The image super-resolution reconstruction method based on multi-domain information enhancement according to claim 1, characterized in that: The construction of the multi-channel large-core separable attention module includes: The multi-way large kernel separable attention module consists of two 3×3 convolutional layers, 1×1 point convolution, a GELU activation function and three sets of large kernel separable attention LKSA of different scales; The sizes of the three groups of large-core separable attentions of different scales are: 3-5-1, 5-7-1, and 7-9-1; they represent cascaded a×1 and 1×a depthwise separable convolutions, cascaded b×1 and 1×b depthwise dilated convolutions, and 1×1 point convolution PConv; The input features of the multi-way large-core separable attention module first pass through a 3×3 convolution layer and obtain a new feature map after the GELU activation function; then, starting from the horizontal and vertical directions, three groups of large-core separable attentions of different scales are passed to obtain the corresponding three groups of feature maps; these three groups of feature maps are spliced ​​in the channel dimension, and dimensionality reduction is performed using 1×1 convolution, and then a comprehensive feature map is generated through a 3×3 convolution layer; this feature map is fused with the original input features by element-wise addition through jump connections to output the final feature map.

3. The image super-resolution reconstruction method based on multi-domain information enhancement according to claim 1, characterized in that: The construction of the multi-channel large-core attention module includes: The multi-channel large-core attention module consists of two 3×3 convolutional layers, 1×1 point convolution, a GELU activation function, and three sets of large-core attention LKAs of different scales; The sizes of the three groups of large core attentions of different scales are: 3-5-1, 5-7-1, 7-9-1; ab-1 represents the cascade of a×a depth convolution DWConv, b×b depth expansion convolution DPConv and point convolution PConv; The input features of the multi-channel large-core attention module are first passed through a 3×3 convolution layer and a GELU activation function to obtain new features, and then passed through three groups of large-core attentions of different scales to obtain three different groups of features. The three groups of feature maps are concatenated by channel and the concatenated features are reduced in dimension using a 1×1 point convolution. The long-range information features are then passed through a 3×3 convolution layer. Finally, the long-range information features are element-wise added to the original input features through a jump connection to obtain the final output features.

4. The image super-resolution reconstruction method based on multi-domain information enhancement according to claim 1, characterized in that: The construction of the high-frequency feature enhancement module includes: The high-frequency feature enhancement module consists of a 5×5 depthwise separable convolution, two 1×1 pointwise convolution layers, two GELU activation functions, and a maximum pooling layer; The input features of the high-frequency feature enhancement module are first divided into two parts with the same number of channels, and then a 5×5 depthwise separable convolution layer and a GELU activation function are used to extract local high-frequency features. A maximum pooling layer is used to capture high-frequency information, and a 1×1 convolution layer combined with a GELU activation function is used to further enhance the captured high-frequency features to obtain enhanced features. Finally, the enhanced high-frequency features under the two channels are concatenated and fused, and the final output features are obtained through a 1×1 convolution layer.

5. The image super-resolution reconstruction method based on multi-domain information enhancement according to claim 1, characterized in that: The construction of the local information enhanced attention module includes: The local information enhanced attention module consists of 4 1×1 point convolutions, 3 5×5 depth-wise separable convolutions, and a Sigmoid activation function; The input features of the local information enhanced attention module are first generated through a local feature extraction branch consisting of a 1×1 convolution and a 5×5 deep convolution; at the same time, the input features also pass through a local feature enhancement branch, which first generates features through a 1×1 convolution; the features pass through a pixel-level attention branch consisting of a 1×1 convolution and a Sigmoid activation function to generate an attention map, and multiply it with the features generated by the 5×5 deep convolution to obtain enhanced features; the features pass through a 5×5 deep convolution again to obtain local enhanced features, the features of the two branches are fused through a feature splicing operation, and channel dimensionality reduction is performed through a 1×1 convolution layer to obtain the final output features.

6. The image super-resolution reconstruction method based on multi-domain information enhancement according to claim 1, characterized in that: The shallow feature extraction module comprises: The shallow feature extraction module consists of a 3×3 convolution layer with 48 convolution kernels and a step size of 1; The output of the shallow feature extraction module is the shallow features obtained by processing the input low-resolution image through a 3×3 convolution layer.

7. The image super-resolution reconstruction method based on multi-domain information enhancement according to claim 1, characterized in that: The deep feature extraction module comprises: The deep feature extraction module is composed of 6 multi-domain information enhancement feature extraction blocks MDIEB cascaded, and the number of input and output feature maps of each MDIEB is 48; The output of the deep feature extraction module is the deep feature obtained after the shallow feature is extracted by 6 MDIEB features.

8. The image super-resolution reconstruction method based on multi-domain information enhancement according to claim 1, characterized in that: The reconstruction module comprises: The reconstruction module is composed of a 3×3 convolution layer, a pixel shuffle layer, and a 3×3 convolution layer cascaded together, with 48 convolution kernels and a step size of 1; The input of the reconstruction module is the result of summing the shallow feature and the deep feature elements, and the output is a reconstructed high-resolution image.

9. The image super-resolution reconstruction method based on multi-domain information enhancement according to claim 1, characterized in that: In the training of the multi-domain information enhanced lightweight image super-resolution reconstruction network model, the loss function used is: Where: θ represents the parameter set of the neural network; N is the number of training image pairs; represents the i-th real high-resolution image; Represents the i-th super-resolution image after algorithm reconstruction.

10. An image super-resolution reconstruction device based on multi-domain information enhancement, characterized in that: include: An image module, used for acquiring a low-resolution natural image to be reconstructed; Model construction module, used to construct a multi-way large core separable attention module using two convolutional layers, one point convolution, one GELU activation function and three sets of large core separable attention LKSA cascades of different scales, construct a multi-way large core attention module using two convolutional layers, one point convolution, one GELU activation function and three sets of large core attention LKA cascades of different scales, construct a high-frequency feature enhancement module using a depthwise separable convolution, two pointwise convolution layers, two GELU activation functions and a maximum pooling layer cascade, and construct a local information enhancement attention module using four pointwise convolutions, three depthwise separable convolutions and a sigmoid activation function cascade; The multi-channel large core separable attention module, the multi-channel large core attention module, the high-frequency feature enhancement module and the local information enhancement attention module are integrated with the layer normalization LN to form a multi-domain information enhancement feature extraction block containing different branches; A lightweight image super-resolution reconstruction network model with multi-domain information enhancement is constructed using multi-domain information enhancement feature extraction blocks; The image reconstruction module uses the DIV2K and Flickr2K datasets to train the parameter sets in the shallow feature extraction module including the convolution layer, the deep feature extraction module including multiple multi-domain information enhanced feature extraction blocks, and the reconstruction module including the convolution layer and the pixel shuffling layer in the multi-domain information enhanced lightweight image super-resolution reconstruction network model, respectively, to obtain the trained multi-domain information enhanced lightweight image super-resolution reconstruction network model; The low-resolution natural image to be reconstructed is input into the trained multi-domain information enhanced lightweight image super-resolution reconstruction network model, and the image features are extracted through the convolution layer in the shallow feature extraction module to obtain a shallow feature map; the image features are extracted through the LN layer in the multiple multi-domain information enhanced feature extraction blocks cascaded in the deep feature extraction module, multiple convolution layers and point convolutions of different scales, activation functions and different attention mechanisms to obtain a deep feature map; the feature elements in the shallow feature map and the deep feature map are added through the convolution layer and the pixel shuffling layer in the reconstruction module, and pixel reconstruction is performed to obtain a high-resolution image of the image.

Images