An image super-resolution reconstruction method and device based on multi-domain information enhancement

Abstract

The application discloses a kind of based on multi-domain information enhancement's image super-resolution reconstruction method and device, it is related to single image super-resolution reconstruction technical field, the present application constructs multi-path large-core separable attention module, multi-path large-core attention module, high-frequency feature enhancement module and local information enhancement attention module, and with layer normalization integration forms multi-domain information enhancement feature extraction block to form multi-domain information enhancement's light image super-resolution reconstruction network model, the parameter set in shallow feature extraction module, deep feature extraction module and reconstruction module in model are trained respectively, so that the four modules contained in model respectively focus on local information, high-frequency information, long-range information, frequency component in horizontal and vertical direction in image, so as to be able to improve the quality of network reconstruction image from the overall layout, detail information, texture information and object image four aspects in image, so that the quality of image is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of single image super-resolution reconstruction, and particularly relates to an image super-resolution reconstruction method and device based on multi-domain information enhancement. BACKGROUND

[0002] Image super-resolution reconstruction technology (SR) refers to recovering a high-resolution (HR) image with rich details and better visual quality from one or more blurred and unclear low-resolution (LR) images of the same scene; this technology is based on complex mathematical models and algorithms, which can effectively increase the spatial resolution of the image, making the image present clearer and more detailed details; compared with the method of improving hardware devices to obtain high-resolution images, this software-level technology saves the link of using higher-end devices to re-shoot and make, saves a lot of cost, and has a wider application range and lower cost.

[0003] In recent years, convolutional neural networks have been widely used in the field of super-resolution; unlike traditional learning algorithms that use artificial selection of features, algorithms based on convolutional neural networks aim to use end-to-end learning to generate deep features in a task-oriented manner; compared with traditional algorithms, the performance is greatly improved, and very significant reconstruction results are achieved; at the same time, since the network depth and network performance of the super-resolution reconstruction algorithm based on convolutional neural networks have a good positive correlation, various deep SR models are proposed; however, the depth of the network also increases the number of parameters and the amount of calculation; specifically, 3-layer SRCNN only needs 57K parameters and 52.7G operation amount, while EDSR, as one of the advanced (SOTA) models with deeper layers, needs 40.7M parameters and 9384.7G operation amount; although deepening the network improves performance, the resulting computational load limits real-world applications, such as on mobile devices or real-time SR.

[0004] For practical applications that consider limited computing resources, the demand for lightweight networks has been growing; the key problem of building a lightweight network is to balance the cost between network performance and the number of parameters (memory complexity) or floating-point operations (time complexity); since simply reducing the depth or width of the network will result in a significant performance drop, careful design of network structure optimization is needed to simplify the network structure while maintaining performance; however, high-performance image super-resolution networks often have a large number of parameters, and if a lightweight network is used to reduce the number of parameters and operation amount, the reconstructed image will usually have structural distortion and unclear texture, resulting in poor image quality, which is difficult to meet the growing performance demands of modern consumers for mobile devices and applications. SUMMARY

[0005] The embodiment of the present application provides a kind of based on multi-domain information enhancement's image super-resolution reconstruction method and device, can solve the problem of poor image quality in prior art, there is the phenomenon of structural distortion and unclear texture in the image reconstructed by the lightweight network with reduced parameter quantity and operation amount.

[0006] The embodiment of the present application provides a kind of based on multi-domain information enhancement's image super-resolution reconstruction method, comprising the following steps:

[0007] Obtain the low-resolution natural image to be reconstructed;

[0008] A multi-path large-kernel separable attention module is constructed using two convolutional layers, a point convolution, a GELU activation function, and three groups of large-kernel attention LKA with different scales, a multi-path large-kernel attention module is constructed using two convolutional layers, a point convolution, a GELU activation function, and three groups of large-kernel attention LKA with different scales, a high-frequency feature enhancement module is constructed using a depth separable convolution, two point convolutional layers, two GELU activation functions, and a max-pooling layer, and a local information enhancement attention module is constructed using four point convolutions, three depth separable convolutions, and a Sigmoid activation function. The multi-path large-kernel separable attention module, the multi-path large-kernel attention module, the high-frequency feature enhancement module, and the local information enhancement attention module are integrated with layer normalization LN to form a multi-domain information enhancement feature extraction block containing different branches.

[0009] A multi-domain information enhancement lightweight image super-resolution reconstruction network model is constructed using the multi-domain information enhancement feature extraction block. The parameter set in the shallow feature extraction module containing convolutional layers, the deep feature extraction module containing multiple multi-domain information enhancement feature extraction blocks, and the reconstruction module containing convolutional layers and pixel shuffling layers in the multi-domain information enhancement lightweight image super-resolution reconstruction network model are trained using the DIV2K and Flickr2K data sets, respectively, to obtain the trained multi-domain information enhancement lightweight image super-resolution reconstruction network model.

[0010] The low-resolution natural image to be reconstructed is input into the trained multi-domain information enhancement lightweight image super-resolution reconstruction network model, the convolutional layers in the shallow feature extraction module are used to extract features from the image to obtain shallow feature maps, the LN layers, multiple convolutional layers with different scales, point convolutions, activation functions, and different attention mechanisms in the multiple multi-domain information enhancement feature extraction blocks cascaded in the deep feature extraction module are used to extract features from the image to obtain deep feature maps, and the convolutional layers and pixel shuffling layers in the reconstruction module are used to add the feature elements in the shallow feature maps and the deep feature maps and perform pixel reconstruction to obtain a high-resolution image of the image.

[0011] The three groups of large kernel separable attention of different sizes are: 3x1*1x3-5x1*1x5-1x1, 5x1*1x5-7x1*1x7-1x1, 7x1*1x7-9x1*1x9-1x1, * represents the concatenation of the decomposed horizontal 1D kernel and the vertical 1D kernel inside the convolution kernel, and - represents the concatenation between the convolution kernels;

[0012] Wherein: 3x1*1x3-5x1*1x5-1x1 represents the depth separable convolution of the concatenated 3x1 and 1x3, the depth dilated convolution of the concatenated 5x1 and 1x5, and the point convolution PConv of 1x1; 5x1*1x5-7x1*1x7-1x1 represents the depth separable convolution of the concatenated 5x1 and 1x5, the depth dilated convolution of the concatenated 7x1 and 1x7, and the point convolution PConv of 1x1; 7x1*1x7-9x1*1x9-1x1 represents the depth separable convolution of the concatenated 7x1 and 1x7, the depth dilated convolution of the concatenated 9x1 and 1x9, and the point convolution PConv of 1x1;

[0013] The three groups of large kernel separable attention of different sizes are: 3x3-5x5-1x1, 5x5-7x7-1x1, 7x7-9x9-1x1;

[0014] Wherein: 3x3-5x5-1x1 represents the concatenation of 3x3 depth convolution DWConv, 5x5 depth expansion convolution DPConv and 1x1 point convolution PConv; 5x5-7x7-1x1 represents the concatenation of 5x5 depth convolution DWConv, 7x7 depth expansion convolution DPConv and 1x1 point convolution PConv; 7x7-9x9-1x1 represents the concatenation of 7x7 depth convolution DWConv, 9x9 depth expansion convolution DPConv and 1x1 point convolution PConv.

[0015] Preferably, the construction of the multi-path large kernel separable attention module comprises:

[0016] The multi-path large kernel separable attention module is composed of two 3x3 convolution layers, one 1x1 point convolution, one GELU activation function and three groups of large kernel separable attention LSKA of different sizes;

[0017] The input features of the multi-path large kernel separable attention module are first processed by a 3x3 convolution layer and a GELU activation function to obtain new feature maps; then, three groups of features are obtained by three groups of large kernel separable attention in horizontal and vertical directions; the three groups of features are spliced in the channel dimension, and a 1x1 convolution is used for dimension reduction processing, and then a 3x3 convolution layer is used to generate a comprehensive feature map; the feature map is fused with the original input features by element addition through a skip connection, and the final feature map is output.

[0018] Preferably, the multi-path large kernel attention module is constructed, comprising:

[0019] The multi-path large kernel attention module is composed of two 3x3 convolution layers, a 1x1 point convolution, a GELU activation function, and three groups of large kernel attention LKA of different scales;

[0020] The input features of the multi-path large kernel attention module are first processed by a 3x3 convolution layer and a GELU activation function to obtain new features, and then three groups of different features are obtained by three groups of large kernel attention of different scales. The three groups of feature maps are spliced in the channel and dimensionally reduced by a 1x1 point convolution, and then a 3x3 convolution layer is used to obtain long-range information features. Finally, the long-range information features are fused with the original input features by element addition through a skip connection to obtain the final output features.

[0021] Preferably, the high-frequency feature enhancement module is constructed, comprising:

[0022] The high-frequency feature enhancement module is composed of a 5x5 depth separable convolution, two 1x1 point convolution layers, two GELU activation functions, and a maximum pooling layer;

[0023] The input features of the high-frequency feature enhancement module are first divided into two parts with the same number of channels, and then local high-frequency features are extracted by a 5x5 depth separable convolution layer and a GELU activation function. The maximum pooling layer is used to capture high-frequency information, and a 1x1 convolution layer is used to further enhance the captured high-frequency features through a GELU activation function, thereby obtaining enhanced features. Finally, the two channels of enhanced high-frequency features are spliced and fused, and a 1x1 convolution layer is used to obtain the final output features.

[0024] Preferably, the local information enhancement attention module is constructed, comprising:

[0025] The local information enhancement attention module is composed of four 1x1 point convolutions, three 5x5 depth separable convolutions, and a Sigmoid activation function;

[0026] The input feature of the local information enhancement attention module is first generated by a local feature extraction branch composed of a 1x1 convolution and a 5x5 deep convolution; meanwhile, the input feature also passes through a local feature enhancement branch, which first generates a feature through a 1x1 convolution; the feature passes through a pixel-level attention branch composed of a 1x1 convolution and a Sigmoid activation function to generate an attention map, and is multiplied by the feature generated by a 5x5 deep convolution to obtain an enhanced feature; the feature is again passed through a 5x5 deep convolution to obtain a local enhanced feature, the features of the two branches are fused through a feature splicing operation, and a 1x1 convolution layer is used for channel dimension reduction processing to obtain the final output feature.

[0027] Preferably, the shallow feature extraction module comprises:

[0028] The shallow feature extraction module is composed of a 3x3 convolution layer, the number of convolution kernels of which is 48, and the step is 1.

[0029] The output of the shallow feature extraction module is the shallow feature obtained after the low-resolution image is input and passed through a 3x3 convolution layer.

[0030] Preferably, the deep feature extraction module comprises:

[0031] The deep feature extraction module is composed of six multi-domain information enhancement feature extraction blocks (MDIEB) in cascade, and the number of input and output feature maps of each MDIEB is 48.

[0032] The output of the deep feature extraction module is the deep feature obtained after the shallow feature is passed through six MDIEB feature extraction blocks.

[0033] Preferably, the reconstruction module comprises:

[0034] The reconstruction module is composed of a 3x3 convolution layer, a pixel shuffling layer and a 3x3 convolution layer in cascade, the number of convolution kernels of which is 48, and the step is 1.

[0035] The input of the reconstruction module is the result of the element sum of the shallow feature and the deep feature, and the output is the high-resolution image after reconstruction.

[0036] Preferably, in the training of the multi-domain information enhancement lightweight image super-resolution reconstruction network model, the loss function used is:

[0037]

[0038] Wherein: θ represents the parameter set of the neural network; N is the number of training image pairs; represents the i th real high-resolution image; indicates the super-resolution image after algorithm reconstruction. i

[0039] The embodiment of the present application also provides an image super-resolution reconstruction device based on multi-domain information enhancement, comprising:

[0040] An image module is used for acquiring a low-resolution natural image to be reconstructed.

[0041] A model construction module is used for constructing a multi-path large-kernel separable attention module by using two convolution layers, one point convolution, one GELU activation function and three groups of large-kernel attention LKA cascades with different scales, constructing a multi-path large-kernel attention module by using two convolution layers, one point convolution, one GELU activation function and three groups of large-kernel attention LKA cascades with different scales, constructing a high-frequency feature enhancement module by using one deep separable convolution, two point convolution layers, two GELU activation functions and one maximum pooling layer cascade, and constructing a local information enhancement attention module by using four point convolutions, three deep separable convolutions and one Sigmoid activation function cascade; and the multi-path large-kernel separable attention module, the multi-path large-kernel attention module, the high-frequency feature enhancement module and the local information enhancement attention module are integrated with layer normalization LN to form a multi-domain information enhancement feature extraction block containing different branches.

[0042] A multi-domain information enhancement lightweight image super-resolution reconstruction network model is constructed by using the multi-domain information enhancement feature extraction block.

[0043] An image reconstruction module is used for training a parameter set in a shallow feature extraction module containing convolution layers, a deep feature extraction module containing a plurality of multi-domain information enhancement feature extraction blocks and a reconstruction module containing convolution layers and a pixel shuffling layer in the multi-domain information enhancement lightweight image super-resolution reconstruction network model by using DIV2K and Flickr2K data sets, so as to obtain a trained multi-domain information enhancement lightweight image super-resolution reconstruction network model.

[0044] The low-resolution natural image to be reconstructed is input into the trained multi-domain information enhancement lightweight image super-resolution reconstruction network model, the convolution layers in the shallow feature extraction module are used for performing feature extraction on the image, so as to obtain a shallow feature map; the LN layers, a plurality of convolution layers with different scales and point convolutions, activation functions and different attention mechanisms in the plurality of multi-domain information enhancement feature extraction blocks cascaded in the deep feature extraction module are used for performing feature extraction on the image, so as to obtain a deep feature map; and the convolution layers and the pixel shuffling layer in the reconstruction module are used for adding and summing feature elements in the shallow feature map and the deep feature map and performing pixel reconstruction, so as to obtain a high-resolution image of the image.

[0045] ​The three groups of large kernels with different scales can separate attention sizes, which are 3x1*1x3-5x1*1x5-1x1, 5x1*1x5-7x1*1x7-1x1, 7x1*1x7-9x1*1x9-1x1, * represents that the horizontal 1D kernel and the vertical 1D kernel in the decomposed kernel inside the convolution kernel are cascaded, and - represents the cascade between the convolution kernels and the convolution kernels;

[0046] 3x1*1x3-5x1*1x5-1x1 represents the depth separable convolution of the cascaded 3x1 and 1x3, the depth dilated convolution of the cascaded 5x1 and 1x5, and the point convolution PConv of 1x1; 5x1*1x5-7x1*1x7-1x1 represents the depth separable convolution of the cascaded 5x1 and 1x5, the depth dilated convolution of the cascaded 7x1 and 1x7, and the point convolution PConv of 1x1; and 7x1*1x7-9x1*1x9-1x1 represents the depth separable convolution of the cascaded 7x1 and 1x7, the depth dilated convolution of the cascaded 9x1 and 1x9, and the point convolution PConv of 1x1.

[0047] The three groups of large kernels with different scales can separate attention sizes, which are 3x1*1x3-5x1*1x5-1x1, 5x1*1x5-7x1*1x7-1x1, 7x1*1x7-9x1*1x9-1x1, * represents that the horizontal 1D kernel and the vertical 1D kernel in the decomposed kernel inside the convolution kernel are cascaded, and - represents the cascade between the convolution kernels and the convolution kernels;

[0048] 3x1*1x3-5x1*1x5-1x1 represents the depth separable convolution of the cascaded 3x1 and 1x3, the depth dilated convolution of the cascaded 5x1 and 1x5, and the point convolution PConv of 1x1; 5x1*1x5-7x1*1x7-1x1 represents the depth separable convolution of the cascaded 5x1 and 1x5, the depth dilated convolution of the cascaded 7x1 and 1x7, and the point convolution PConv of 1x1; and 7x1*1x7-9x1*1x9-1x1 represents the depth separable convolution of the cascaded 7x1 and 1x7, the depth dilated convolution of the cascaded 9x1 and 1x9, and the point convolution PConv of 1x1.

[0049] The embodiment of the present application provides a kind of based on multi-domain information enhancement's image super-resolution reconstruction method and device, compared with prior art, its beneficial effects are as follows:

[0050] This invention constructs a multi-path large-kernel separable attention module, a multi-path large-kernel attention module, a high-frequency feature enhancement module, and a local information enhancement attention module from the perspectives of spatial domain, frequency domain, and transformation domain. These modules are then integrated with layer normalization to form a multi-domain information enhancement feature extraction block containing different branches. This forms a lightweight image super-resolution reconstruction network model with multi-domain information enhancement. The parameter sets of the shallow feature extraction module, deep feature extraction module, and reconstruction module within the model are trained separately. This allows the multi-path large-kernel attention module to focus on long-range information in the image, the local information enhancement attention module to focus on local information, the high-frequency feature enhancement module to focus on high-frequency information, and the multi-path large-kernel separable attention module to focus on frequency components in the horizontal and vertical directions. This improves the quality of the reconstructed image from four aspects: overall layout, detail information, texture information, and object image, resulting in a significant improvement in image quality.

[0051] Furthermore, the network model trained by this invention has fewer parameters, achieving an optimized balance between model size and performance, and ensuring wide applicability and practicality on various devices. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the overall process of an image super-resolution reconstruction method and apparatus based on multi-domain information enhancement provided in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the network structure of an image super-resolution reconstruction method and apparatus based on multi-domain information enhancement provided in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram comparing the results of an image super-resolution reconstruction method and apparatus based on multi-domain information enhancement provided in an embodiment of the present invention; wherein: the image on the left is Img004 from the Urban100 dataset; the image on the right is the result of different lightweight image super-resolution reconstruction algorithms. Detailed Implementation

[0055] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0056] See Figures 1 to 3The embodiment of the application provides a multi-domain information enhancement-based image super-resolution reconstruction method, comprising the following steps:

[0057] Step one: design a multi-path large kernel attention module, a local information enhancement attention module, a multi-path large kernel, a high-frequency feature enhancement module and a separable attention module; and utilize the modules to enhance the network to focus on long-range information, local information, high-frequency information and frequency components in the horizontal and vertical directions of an image.

[0058] Step two: integrate the multi-path large kernel attention module, the local information enhancement attention module, the multi-path large kernel, the high-frequency feature enhancement module, the separable attention module and layer normalization together to form a multi-domain information enhancement feature extraction block.

[0059] Step three: utilize the multi-domain information enhancement feature extraction block to construct a multi-domain information enhancement lightweight image super-resolution reconstruction network, which improves the quality of a reconstructed image from four aspects of overall layout in the image, detail information in the image, texture information in the image and image of an object in the image.

[0060] Step four: utilize a standard DIV2K or Flickr2K dataset to complete final training of the multi-domain information enhancement lightweight image super-resolution reconstruction network model.

[0061] Step five: input a low-resolution natural image to be reconstructed into the trained multi-domain information enhancement lightweight image super-resolution reconstruction network to obtain a high-resolution image of the image.

[0062] Specifically, the following steps are taken:

[0063] Step one: construct a multi-path large separable kernel attention module (MLKSA), which utilizes a multi-path structure and a large separable kernel attention (LKSA) technology to focus on frequency components in the horizontal and vertical directions; the separable attention mechanism allows the model to effectively process and analyze directional features of an image in different horizontal and vertical directions, i.e., different wavebands (different frequency ranges), while keeping the number of parameters controllable; naturally imposes directional constraints on pixels, focuses on the image of an object in the image, and thus improves the image reconstruction quality.

[0064] Step two: Constructing a multi-path large kernel attention module (MLKA), which is responsible for processing long-range information of images. It adopts a multi-path structure combined with large kernel attention (LKA) technology to expand the receptive field, thereby better capturing global dependencies in images. Attention mechanisms are used to emphasize more valuable features and improve the accuracy of reconstruction.

[0065] Step three: Constructing a high frequency information enhancement block (HFEB), which aims to enhance high frequency information in images, i.e., those containing rich textures and sharp edges. By analyzing and enhancing high frequency features, the module further improves the details and texture information of the reconstructed image, enhancing visual effects.

[0066] Step four: Constructing a local information enhancement attention module (LIEA), which focuses on enhancing local information in images. Through point convolution, deep convolution, and pixel attention, the module enhances the network's ability to extract local information, allowing it to focus on local key information and establish local associations in images, thereby improving the network's ability to reconstruct image details such as edges and texture details.

[0067] Step five: Integrating the above modules, the MLKA, LIEA, HFEB, and MLKSA modules, as well as layer normalization (LN), into a multi-domain information enhancement feature extraction block (MDIEB). Each module is optimized for its specific information domain, working together to achieve high-quality image reconstruction.

[0068] Step six: Constructing a multi-domain information enhancement lightweight image super-resolution reconstruction network, which consists of the following three parts in cascade: a shallow feature extraction module composed of 3x3 convolution layers, a deep feature extraction module composed of six multi-domain information enhancement modules (MDIEM), and a reconstruction module composed of 3x3 convolution layers, pixel shuffle layers, and 3x3 convolution layers in cascade.

[0069] Step seven: training and testing, using the standard DIV2K or Flickr2K dataset to train the proposed network, and then evaluating its reconstruction performance on the Y channel average peak signal to noise ratio (PSNR) and structural similarity (SSIM) on four standard test sets of Set5, Set14, B100, and Urban100, to ensure universality and robustness on a variety of natural images; by comparing the performance of the prior art and the present technology under the same conditions, the advantages of the proposed method are verified.

[0070] Step eight: inputting the low-resolution natural image to be reconstructed into the trained multi-domain information enhancement lightweight image super-resolution reconstruction network to obtain the high-resolution image of the image.

[0071] In step one of constructing the multi-path large kernel separable attention module (MLKSA):

[0072] The structure of the MLKSA is shown in Figure 2 , which is composed of two 3x3 convolution layers, a 1x1 point convolution, a GELU activation function, and three groups of large kernel separable attention (LKSA) of different scales; the sizes of the three groups of large kernel separable attention are 3-5-1, 5-7-1, and 7-9-1, where a-b-1 represents the depth separable convolution of a x 1 and 1 x a, the depth dilated convolution of b x 1 and 1 x b, and the point convolution (PConv) of 1 x 1; the number of convolution kernels is set to 48, and the number of groups of depth convolution is consistent with the number of convolution kernels, also 48.

[0073] The working process of the MLKSA is as follows: the input feature is first passed through a 3x3 convolution layer and a GELU activation function for nonlinear transformation to obtain a new feature map; then, from the horizontal and vertical directions, three groups of large kernel separable attention mechanisms of different scales are used to learn the long-range dependencies in space to obtain three groups of corresponding feature maps; then, the three groups of feature maps are spliced in the channel dimension and processed by 1x1 convolution for dimension reduction, and a 3x3 convolution layer is used to generate a comprehensive feature map; finally, the feature map is fused with the original input feature by element-wise addition through a skip connection to form the final output feature of the MLKSA.

[0074] The output feature of the MLKSA focuses on the image of the object in the image, thereby improving the image reconstruction quality.

[0075] In step two of constructing the multi-path large kernel attention module (MLKA):

[0076] The structure of the MLKA is shown in Figure 2As shown, it consists of two 3×3 convolutional layers, a 1×1 point convolution, a GELU activation function, and three sets of large kernel attention (LKA) of different scales; the sizes of the three sets of large kernel attention are 3-5-1, 5-7-1, and 7-9-1, where ab-1 represents the concatenation of a×a depthwise convolution (DWConv), b×b depthwise expanded convolution (DPConv), and pointwise convolution (PConv); the number of convolutional kernels is set to 48, and the number of depthwise convolution groups is the same as the number of convolutional kernels, also 48.

[0077] The workflow of MLKA is as follows: The input features are first passed through a 3×3 convolutional layer and the GELU activation function to obtain new features. Then, three sets of large kernel attention at different scales are used to learn spatial long-range relationships to obtain three different sets of features. The three sets of feature maps are then concatenated by channel and dimensionality reduction is performed on the concatenated features using a 1×1 point convolution. After passing through a 3×3 convolutional layer, long-range information features are obtained. Finally, the long-range information features are added element-wise to the original input features through skip connections to obtain the final output features of MLKA.

[0078] The output features of MLKA include enhanced long-range information, such as the overall structure and content of the image.

[0079] In step three, during the construction of the High Frequency Feature Enhancement Module (HFEB):

[0080] The structure of HFEB is as follows Figure 2 As shown, it consists of a 5×5 depthwise separable convolution, two 1×1 point convolutional layers, two GELU activation functions, and a max pooling layer.

[0081] The workflow of HFEB is as follows: The input features are first segmented into two parts with the same number of channels, and then local high-frequency features are extracted through a 5×5 depthwise separable convolutional layer and the GELU activation function. Next, a max pooling layer is used to capture high-frequency information, and these high-frequency features are further enhanced through a 1×1 convolutional layer combined with the GELU activation function to obtain the enhanced features. Finally, the features of the two branches are concatenated and fused, and the final output features of HFEB are obtained through a 1×1 convolutional layer.

[0082] The output features of HFEB include enhanced high-frequency detail information, such as texture information.

[0083] In step four, during the construction of the Local Information Enhancement Attention Module (LIEA):

[0084] The structure of LIEA is as follows Figure 2As shown, it is composed of 4 1x1 point convolutions, 3 5x5 depth separable convolutions and a Sigmoid activation function; wherein the 1x1 point convolution and the Sigmoid activation function constitute the pixel attention; the number of convolution kernels is set to 48, and the step is set to 1; the number of groups of depth convolution is consistent with the number of convolution kernels, which is also 48.

[0085] The workflow of LIEA is as follows: the input feature is first generated by a local feature extraction branch composed of 1x1 convolution and 5x5 depth convolution; at the same time, the input feature also passes through a local feature enhancement branch, which first generates features through 1x1 convolution; then, the features pass through a pixel-level attention branch composed of 1x1 convolution and Sigmoid activation function to generate an attention map, and multiply it with the features generated by 5x5 depth convolution to obtain enhanced features; then, the features pass through 5x5 depth convolution again to obtain local enhanced features; finally, the features of the two branches are fused through feature concatenation operation, and channel dimension reduction processing is performed through 1x1 convolution layer; in addition, in order to maintain the stability of training and prevent information loss, a skip connection is introduced to obtain the final output feature of LIEA.

[0086] The output feature of LIEA contains enhanced local information, such as edge and texture details.

[0087] In step five, the above modules are integrated to build a multi-domain information enhancement feature extraction block (MDIEB):

[0088] The MLKA, LIEA, HFEB and MLKSA modules and layer normalization (LN) are integrated together to form a multi-domain information enhancement feature extraction block (MDIEB), and its structure is as shown in Figure 2 The number of input and output feature maps of MDIEB is 48.

[0089] The workflow of MDIEB is as follows: the input feature is first processed by two parallel feature extraction branches: the first branch is composed of LN layer, MLKSA and MLKA in cascade, and the input feature is processed by the first branch to obtain new features; the second branch is composed of HFEB, and the input feature is processed by the second branch to obtain high-frequency enhanced features; then, in order to make full use of multi-scale information and maintain the stability of the training process, a skip connection mechanism is introduced, that is, the features from the two different branches and the original input feature are added element by element; finally, the result is transmitted to the LIEA module to produce the output feature of the final MDIEB.

[0090] Each module of MDIEB is optimized for its specific information domain, and the output feature enhances the overall structure, detailed information, texture features and object outline of the image.

[0091] In the shallow feature extraction module of the multi-domain information enhanced lightweight image super-resolution reconstruction algorithm network model constructed in step six:

[0092] The shallow feature extraction module is composed of a 3x3 convolution layer, and the number of convolution kernels is 48 and the step is 1.

[0093] The output of the shallow feature extraction module is the shallow feature obtained after the low-resolution image input is only 3x3 convolution layer.

[0094] In the deep feature extraction module of the multi-domain information enhanced lightweight image super-resolution reconstruction algorithm network model constructed in step six:

[0095] The deep feature extraction module is composed of 6 MEDIB cascades, and the number of input and output feature maps of each MEDIB is 48.

[0096] The output of the deep feature extraction module is the deep feature obtained after the shallow feature is extracted by 6 MEDIBs.

[0097] In the reconstruction module of the multi-domain information enhanced lightweight image super-resolution reconstruction algorithm network model constructed in step six:

[0098] The reconstruction module is composed of a 3x3 convolution layer, a pixel shuffle layer, and a 3x3 convolution layer, and the number of convolution kernels is 48 and the step is 1.

[0099] The input of the reconstruction module is the result of the shallow feature and the deep feature element sum, and the output is the reconstructed high-resolution image.

[0100] In the multi-domain information enhanced lightweight image super-resolution reconstruction algorithm network model constructed in step six:

[0101] The network model is composed of a shallow feature extraction module, a deep feature extraction module, and a reconstruction module.

[0102] The input of the network model is a low-resolution image, and the output is a reconstructed high-resolution image.

[0103] In the model training and testing of step seven:

[0104] The proposed network is trained using the standard DIV2K or Flickr2K dataset.

[0105] Adopt L 1 loss as the loss function of the whole model, L 1 loss function is sensitive to data, which can promote the model to converge quickly, and the specific calculation is:

[0106]

[0107] wherein: θ represents a parameter set of a neural network; N is the number of training image pairs; represents the i-th real high-resolution image; i represents the i-th super-resolution image after algorithm reconstruction. i

[0108] The reconstruction performance was evaluated on four standard test sets, Set5, Set14, B100, and Urban100.

[0109] The Y channel average peak signal-to-noise ratio (PSNR) and the result similarity (SSIM) were used as evaluation indexes.

[0110] For the reconstructed image I SR and the original high-definition image I HR , the peak signal-to-noise ratio was calculated according to the following formula:

[0111]

[0112]

[0113] wherein: H and W represent I SR , I HR the height and width of the image pair; the larger the PSNR value, the higher the quality of the reconstructed image.

[0114] The structural similarity evaluates the similarity between two images from three aspects of brightness, contrast, and structure, and the closer the value is to 1, the better the quality of the image reconstruction, and the calculation formula is as follows:

[0115]

[0116] wherein: and represent I HR , I SR the average gray value of and represent I HR , I SR the variance of represents I SR , I HR the covariance of​​c 1、 c 2、 c 3 is a non-zero constant.

[0117] The performance of the prior art and the present technology under the same conditions is compared to verify the advantages of the proposed method.

[0118] In the reconstructed image, the low-resolution natural image to be reconstructed is input into the trained multi-domain information enhanced lightweight image super-resolution reconstruction network to obtain a high-resolution image of the image.

[0119] The present application will be described in detail below in combination with performance tests and experimental analysis, and the evaluation index used is the PSNR and SSIM widely used in the current super-resolution reconstruction field.

[0120] 1, the execution details of the simulation experiment of the present application are:

[0121] In the training process, the image is randomly cropped into a 96x96 size image block, the training batch size is set to 16, and the feature channel is 48; At the same time, data enhancement operations are performed, such as random horizontal flipping or 90° rotation; trained by Adam optimizer, parameters β1 and β2 are 0.9 and 0.999 respectively; the initial learning rate is set to 4x10 −4 , and the learning rate is reduced by half at iteration 2x10 5 , 4x10 5 , 6x10 5 , 8x10 5 times, and the total number of iterations is 1x10 6 ; the L 1 loss function is used to optimize the model; the Pytorch1.6 deep learning framework is used; the programming language is python3.7, and the experiment is performed on a GeForce RTX 3090 GPU.

[0122] 2, simulation content and result analysis:

[0123] Table 1 is a comparison of the 4x reconstruction indicators of the algorithm proposed in the present application and other six prior arts (CARN, IMDN, ESRT, FMEN, DRSAN-48m, NGswin) on four standard test sets Set5, Set14, B100, Urban100.

[0124] Table 1 Performance comparison of different SSIR models on four benchmark data sets with a scaling factor of 4

[0125]

[0126] For example Figure 3The comparison chart of the image super-resolution reconstruction results of the algorithm and six prior arts (CARN, IMDN, ESRT, FMEN, DRSAN-48m and NGswin) on the challenging data set Urban100 is shown in the figure, and the magnification is 4.

[0127] As can be seen from Table 1, the average values of PSNR and SSIM of the algorithm are higher than those of the above-mentioned prior art comparison methods on four data sets, and compared with the DRSAN-48m method, the algorithm improves by 0.15 dB and 0.26 dB on the Set5 and Urban100 data sets respectively under the condition of reducing the parameter amount by 20%, and the performance is greatly improved. Figure 3 It can be seen that the reconstructed image of the algorithm is basically similar to the original HR image in texture details, while other methods all suffer from serious structure distortion and texture blur; the algorithm captures more context information from multiple dimensions by using more extensive information, and avoids the situation of structure distortion and texture blur of the reconstructed image; the experimental results fully prove the effectiveness and innovation of the algorithm; at the same time, the algorithm realizes a good balance between model size, calculation efficiency and reconstruction performance under the premise of low parameter amount.

[0128] The present application improves the quality of image super-resolution reconstruction by integrating multiple innovative modules; these modules include multi-path large kernel attention module (MLKA), local information enhancement attention module (LIEA), high-frequency feature enhancement module (HFEB) and multi-path large kernel separable attention module (MLKSA); these modules respectively optimize the long-range information, local information, high-frequency information and frequency components in the horizontal and vertical directions in the image, so as to improve the quality of the reconstructed image in four aspects of overall layout, detail information, texture information and object image; through the comprehensive utilization of multi-domain information, the network model of the present application can realize high-quality image reconstruction while maintaining high efficiency, and a good balance between model performance and efficiency is achieved.

[0129] Overall, the present application comprehensively considers the overall layout of the image, detail recovery, texture rendering and object shape four key dimensions, and comprehensively improves the image quality of super-resolution reconstruction; in feature utilization, the present application uses long-range dependence, local structure, high-frequency details and the deep mining of longitudinal and transverse frequency characteristics to significantly enhance the recognition and utilization of important information in the image; in efficiency, through extensive experimental verification, the present application is superior to the current mainstream algorithm in multiple objective evaluation indexes, effectively avoids the possible structure distortion and texture blur in the reconstruction process, and maintains the model size and computing efficiency; in balance, on the basis of controlling the parameter amount, the present application realizes the optimization balance between the model size and performance, and ensures the wide applicability and practicability on various devices.

[0130] The embodiment of the present application also provides an image super-resolution reconstruction device based on multi-domain information enhancement, comprising:

[0131] An image module is configured to acquire a low-resolution natural image to be reconstructed.

[0132] A model construction module is configured to construct a multi-path large-kernel separable attention module by using two convolutional layers, one point convolution, one GELU activation function and three groups of large-kernel attention LKA of different scales in cascade, construct a multi-path large-kernel attention module by using two convolutional layers, one point convolution, one GELU activation function and three groups of large-kernel attention LKA of different scales in cascade, construct a high-frequency feature enhancement module by using one depth separable convolution, two point convolutional layers, two GELU activation functions and one maximum pooling layer in cascade, and construct a local information enhancement attention module by using four point convolutions, three depth separable convolutions and one Sigmoid activation function in cascade; and the multi-path large-kernel separable attention module, the multi-path large-kernel attention module, the high-frequency feature enhancement module and the local information enhancement attention module are integrated with layer normalization LN to form a multi-domain information enhancement feature extraction block containing different branches.

[0133] A multi-domain information enhancement lightweight image super-resolution reconstruction network model is constructed by using the multi-domain information enhancement feature extraction block.

[0134] An image reconstruction module is configured to train a parameter set in a shallow feature extraction module containing convolutional layers, a deep feature extraction module containing a plurality of multi-domain information enhancement feature extraction blocks and a reconstruction module containing convolutional layers and a pixel shuffling layer in the multi-domain information enhancement lightweight image super-resolution reconstruction network model by using DIV2K and Flickr2K data sets, respectively, to obtain a trained multi-domain information enhancement lightweight image super-resolution reconstruction network model.

[0135] The low-resolution natural image to be reconstructed is input into the trained multi-domain information enhancement lightweight image super-resolution reconstruction network model, the image is subjected to feature extraction through the convolution layer in the shallow feature extraction module to obtain a shallow feature map; the image is subjected to feature extraction through the LN layer, multiple different scale convolution layers and point convolution, an activation function and different attention mechanisms in the multiple multi-domain information enhancement feature extraction blocks cascaded in the deep feature extraction module to obtain a deep feature map; the feature elements in the shallow feature map and the deep feature map are added and pixel reconstruction is performed through the convolution layer and the pixel shuffling layer in the reconstruction module to obtain a high-resolution image of the image.

[0136] The above-described embodiments only express several embodiments of the present application, which are described in a more specific and detailed manner, but should not be understood as a limitation on the scope of the patent. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, which are all within the scope of protection of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A multi-domain information enhancement based image super-resolution reconstruction method, characterized in that, The method comprises the following steps: acquiring a low-resolution natural image to be reconstructed; a multi-path large-kernel separable attention module is constructed by cascading two convolution layers, a point convolution, a GELU activation function and three groups of large-kernel attention with different scales, a multi-path large-kernel attention module is constructed by cascading two convolution layers, a point convolution, a GELU activation function and three groups of large-kernel attention with different scales, a high-frequency feature enhancement module is constructed by cascading a depth separable convolution, two point convolution layers, two GELU activation functions and a maximum pooling layer, and a local information enhancement attention module is constructed by cascading four point convolutions, three depth separable convolutions and a Sigmoid activation function; the multi-path large-kernel separable attention module, the multi-path large-kernel attention module, the high-frequency feature enhancement module and the local information enhancement attention module are integrated with layer normalization LN to form a multi-domain information enhancement feature extraction block containing different branches; a multi-domain information enhancement lightweight image super-resolution reconstruction network model is constructed by using the multi-domain information enhancement feature extraction block; the parameter set in the shallow feature extraction module containing convolution layers, the deep feature extraction module containing multiple multi-domain information enhancement feature extraction blocks and the reconstruction module containing convolution layers and pixel shuffling layers in the multi-domain information enhancement lightweight image super-resolution reconstruction network model are trained respectively using the DIV2K and Flickr2K data sets, and a trained multi-domain information enhancement lightweight image super-resolution reconstruction network model is obtained; a low-resolution natural image to be reconstructed is input into the trained multi-domain information enhancement lightweight image super-resolution reconstruction network model, the convolution layers in the shallow feature extraction module are used to extract features of the image, and a shallow feature map is obtained; the LN layers, multiple convolution layers with different scales and point convolutions, activation functions and different attention mechanisms in the multiple multi-domain information enhancement feature extraction blocks cascaded in the deep feature extraction module are used to extract features of the image, and a deep feature map is obtained; the convolution layers and the pixel shuffling layers in the reconstruction module are used to add features in the shallow feature map and the deep feature map and perform pixel reconstruction, and a high-resolution image of the image is obtained; the three groups of large-kernel separable attention with different scales have the following sizes: 3x1*1x3-5x1*1x5-1x1, 5x1*1x5-7x1*1x7-1x1 and 7x1*1x7-9x1*1x9-1x1, * represents the horizontal 1D kernel and the vertical 1D kernel inside the convolution kernel, and - represents the cascade between the convolution kernels; Wherein: 3x1*1x3-5x1*1x5-1x1 represents a depth separable convolution of 3x1 and 1x3 in cascade, a depth dilated convolution of 5x1 and 1x5 in cascade, and a point convolution PConv of 1x1; 5x1*1x5-7x1*1x7-1x1 represents a depth separable convolution of 5x1 and 1x5 in cascade, a depth dilated convolution of 7x1 and 1x7 in cascade, and a point convolution PConv of 1x1; 7x1*1x7-9x1*1x9-1x1 represents a depth separable convolution of 7x1 and 1x7 in cascade, a depth dilated convolution of 9x1 and 1x9 in cascade, and a point convolution PConv of 1x1; The sizes of the three groups of large kernel attention of different scales are: 3x3-5x5-1x1, 5x5-7x7-1x1, and 7x7-9x9-1x1; Wherein: 3x3-5x5-1x1 represents a 3x3 depth convolution DWConv, a 5x5 depth expansion convolution DPConv, and a 1x1 point convolution PConv; 5x5-7x7-1x1 represents a 5x5 depth convolution DWConv, a 7x7 depth expansion convolution DPConv, and a 1x1 point convolution PConv; 7x7-9x9-1x1 represents a 7x7 depth convolution DWConv, a 9x9 depth expansion convolution DPConv, and a 1x1 point convolution PConv.

2. The method of claim 1, wherein the method is based on multi-domain information enhancement. The construction of the multi-path large kernel separable attention module comprises: The multi-path large kernel separable attention module is composed of two 3x3 convolution layers, a 1x1 point convolution, a GELU activation function, and three groups of large kernel separable attention LSKA of different scales; The input feature of the multi-path large kernel separable attention module is first processed by a 3x3 convolution layer and a GELU activation function to obtain a new feature map; then, from the horizontal and vertical directions, three groups of features are obtained through three groups of large kernel separable attention of different scales; the three groups of features are spliced in the channel dimension, and a 1x1 convolution is used for dimension reduction processing; finally, a 3x3 convolution layer is used to generate a comprehensive feature map, which is fused with the original input feature by element addition through a skip connection, and an output feature map is obtained.

3. The method of claim 1, wherein the method is based on multi-domain information enhancement. The construction of the multi-path large kernel attention module comprises: The multi-path large kernel attention module is composed of two 3x3 convolution layers, a 1x1 point convolution, a GELU activation function, and three groups of large kernel attention LKA of different scales; The input feature of the multi-path large kernel attention module is first processed by a 3x3 convolution layer and a GELU activation function to obtain a new feature, and then three groups of different features are obtained through three groups of large kernel attention of different scales; the three groups of features are spliced in the channel dimension, and a 1x1 point convolution is used for dimension reduction processing; then, a 3x3 convolution layer is used to obtain a long-range information feature; finally, the long-range information feature is fused with the original input feature by element addition through a skip connection, and an output feature is obtained.

4. The method of claim 1, wherein, The construction of the high-frequency feature enhancement module comprises: The high-frequency feature enhancement module is composed of one 5*5 deep separable convolution, two 1*1 point convolution layers, two GELU activation functions, and one max-pooling layer. The input feature of the high-frequency feature enhancement module is first divided into two parts with the same number of channels, and then local high-frequency features are extracted through a 5*5 deep separable convolution layer and a GELU activation function; the max-pooling layer is used to capture high-frequency information, and a 1*1 convolution layer is used to further enhance the captured high-frequency features through a GELU activation function, so as to obtain enhanced features; finally, the two enhanced high-frequency features in the channel are spliced and fused, and the final output feature is obtained through a 1*1 convolution layer.

5. The method of claim 1, wherein the method is based on multi-domain information enhancement. The construction of the local information enhancement attention module includes: The local information enhancement attention module is composed of four 1*1 point convolutions, three 5*5 deep separable convolutions, and one Sigmoid activation function. The input feature of the local information enhancement attention module is first generated through a local feature extraction branch composed of a 1*1 convolution and a 5*5 deep convolution; meanwhile, the input feature also passes through a local feature enhancement branch, which first generates a feature through a 1*1 convolution; the feature passes through a pixel-level attention branch composed of a 1*1 convolution and a Sigmoid activation function to generate an attention map, and is multiplied by the feature generated by the 5*5 deep convolution to obtain an enhanced feature; the feature is again processed by a 5*5 deep convolution to obtain a locally enhanced feature, the features of the two branches are fused through a feature splicing operation, and the final output feature is obtained through a 1*1 convolution layer for channel dimension reduction.

6. The method of claim 1, wherein, The shallow feature extraction module includes: The shallow feature extraction module is composed of a 3*3 convolution layer with 48 convolution kernel numbers and a step length of 1. The output of the shallow feature extraction module is a shallow feature obtained by processing a low-resolution image through a 3*3 convolution layer.

7. The method of claim 1, wherein the method is based on multi-domain information enhancement. The deep feature extraction module includes: The deep feature extraction module is composed of six cascaded multi-domain information enhancement feature extraction blocks (MDIEBs), and the number of input and output feature maps of each MDIEB is 48. The output of the deep feature extraction module is a deep feature obtained by processing a shallow feature through six MDIEBs.

8. The method of claim 1, wherein, The reconstruction module includes: The reconstruction module is composed of a 3*3 convolution layer, a pixel shuffling layer, and a 3*3 convolution layer in cascade, with 48 convolution kernel numbers and a step length of 1. The input of the reconstruction module is the result of taking the sum of the elements of the shallow feature and the deep feature, and the output is a reconstructed high-resolution image.

9. The method of claim 1, wherein, In the training of the multi-domain information enhancement lightweight image super-resolution reconstruction network model, the loss function used is: in: θ Represents the parameter set of a neural network; N The number of training image pairs; Indicates the first i A true high-resolution image; The algorithm reconstructs the first... i A super-resolution image.

10. An image super-resolution reconstruction device based on multi-domain information enhancement, characterized by, includes: An image module for obtaining a low-resolution natural image to be reconstructed; The model construction module is configured to construct a multi-path large-kernel separable attention module by cascading two convolutional layers, a point convolution, a GELU activation function, and three groups of large-kernel attention with different scales, construct a multi-path large-kernel attention module by cascading two convolutional layers, a point convolution, a GELU activation function, and three groups of large-kernel attention with different scales, construct a high-frequency feature enhancement module by cascading a depth separable convolution, two point convolutional layers, two GELU activation functions, and a max-pooling layer, and construct a local information enhancement attention module by cascading four point convolutions, three depth separable convolutions, and a Sigmoid activation function; The multi-path large-kernel separable attention module, the multi-path large-kernel attention module, the high-frequency feature enhancement module, and the local information enhancement attention module are integrated with layer normalization LN to form a multi-domain information enhancement feature extraction block containing different branches. The multi-domain information enhancement feature extraction block is used to construct a multi-domain information enhancement lightweight image super-resolution reconstruction network model. The image reconstruction module is configured to train a set of parameters in a shallow feature extraction module containing convolutional layers, a deep feature extraction module containing a plurality of multi-domain information enhancement feature extraction blocks, and a reconstruction module containing convolutional layers and a pixel shuffle layer in the multi-domain information enhancement lightweight image super-resolution reconstruction network model using DIV2K and Flickr2K datasets, to obtain a trained multi-domain information enhancement lightweight image super-resolution reconstruction network model. A low-resolution natural image to be reconstructed is input into the trained multi-domain information enhancement lightweight image super-resolution reconstruction network model, the convolutional layers in the shallow feature extraction module are used to extract features of the image to obtain a shallow feature map, the LN layers, a plurality of convolutional layers with different scales, point convolutions, activation functions, and different attention mechanisms in the plurality of multi-domain information enhancement feature extraction blocks cascaded in the deep feature extraction module are used to extract features of the image to obtain a deep feature map, and the convolutional layers and the pixel shuffle layer in the reconstruction module are used to add and reconstruct pixels in the shallow feature map and the deep feature map to obtain a high-resolution image of the image. The three groups of large-kernel separable attention with different scales have sizes of 3x1*1x3-5x1*1x5-1x1, 5x1*1x5-7x1*1x7-1x1, and 7x1*1x7-9x1*1x9-1x1, * represents a horizontal 1D kernel and a vertical 1D kernel inside a convolution kernel that are cascaded, and - represents a cascade between convolution kernels. wherein: 3x1*1x3-5x1*1x5-1x1 represents depth separable convolution of 3x1 and 1x3 cascaded, depth dilated convolution of 5x1 and 1x5 cascaded, and point convolution PConv of 1x1; 5x1*1x5-7x1*1x7-1x1 represents depth separable convolution of 5x1 and 1x5 cascaded, depth dilated convolution of 7x1 and 1x7 cascaded, and point convolution PConv of 1x1; 7x1*1x7-9x1*1x9-1x1 represents depth separable convolution of 7x1 and 1x7 cascaded, depth dilated convolution of 9x1 and 1x9 cascaded, and point convolution PConv of 1x1; The sizes of the three groups of different scale large kernel attentions are: 3x3-5x5-1x1, 5x5-7x7-1x1, 7x7-9x9-1x1; wherein: 3x3-5x5-1x1 represents 3x3 depth convolution DWConv, 5x5 depth dilated convolution DPConv and 1x1 point convolution PConv cascaded; 5x5-7x7-1x1 represents 5x5 depth convolution DWConv, 7x7 depth dilated convolution DPConv and 1x1 point convolution PConv cascaded; 7x7-9x9-1x1 represents 7x7 depth convolution DWConv, 9x9 depth dilated convolution DPConv and 1x1 point convolution PConv cascaded.

Images