A light-weight image super-resolution method and device based on spatial and frequency domains

Abstract

The application discloses a kind of light-weighted image super-resolution method and device based on space and frequency domain, which comprises the following steps: S1: low-resolution image is collected to carry out dynamic convolution, and shallow feature is obtained;S2: shallow feature is input into the space domain feature calculation module of multi-scale perception, and middle layer feature is obtained;S3: the last feature is input into the frequency domain feature calculation module of high-frequency enhancement, and deep feature is obtained;S4: deep feature is convolved, and enhanced feature is obtained;S5: enhanced feature and original input feature are fused, and finally super-resolution image is output. By using the application, the image after super-resolution contains more high-frequency information and structural information texture details, and the calculation amount is less, and the running speed is fast.

Description

Technical Field

[0001] The present invention relates to the technical field of image denoising, and in particular to a lightweight image super-resolution method and device. Background Art

[0002] With the rapid development of digital image processing technology, image super-resolution (SR) technology is playing an increasingly important role in multimedia, medical imaging, satellite image processing, and other fields. Image super-resolution technology aims to enhance image resolution and restore detailed information through algorithms to meet the requirements of high-resolution display devices.

[0003] However, with the increasing prevalence of mobile devices and embedded systems, the demand for lightweight image processing models is growing. Traditional image super-resolution models often have numerous parameters and high computational complexity, making them difficult to run in real time on resource-constrained devices. Therefore, the research on lightweight models has become particularly important. Lightweight models aim to reduce the number of model parameters and computational complexity while maintaining or improving image processing performance. Such models can run efficiently on mobile devices and embedded systems, meeting the needs of real-time processing.

[0004] Research has shown that existing deep learning-based research has achieved good results on quantitative metrics such as PSNR and the structural similarity index (SSIM). However, these models contain a large number of parameters and are computationally inefficient, making real-time image super-resolution difficult to achieve on mobile devices and embedded systems. Some existing lightweight super-resolution networks need to be improved in terms of model structure, and more effective attention mechanism modules could be incorporated. In recent years, many studies on image super-resolution have demonstrated that neural networks trained using attention mechanisms can enhance tissue edge texture, but they typically only use a single attention mechanism, without significant improvement in capturing attention information.

[0005] In summary, although the super-resolution images reconstructed by neural networks in the existing technology have high quantitative indicators such as PSNR and SSIM, there are problems such as the network obtaining single image features, insufficient attention information and time-consuming model calculation. Summary of the Invention

[0006] In order to solve the above technical problems, the present invention provides a lightweight image super-resolution method and device based on spatial domain and frequency domain, which can provide mobile devices with better quality image super-resolution services.

[0007] Specifically, the method comprises the following steps:

[0008] S1: Collect images, perform dynamic convolution, and obtain shallow features;

[0009] S2: Input shallow features into the multi-scale perception spatial feature calculation module to obtain mid-level features; S2 includes the following steps:

[0010] S2.1: Perform two downsampling operations on shallow features:

[0011]

[0012]

[0013] In the above formula, Represents multiples of n Image downsampling operation.

[0014] S2.2: Perform channel attention calculation on features at three different scales:

[0015]

[0016]

[0017]

[0018] In the above formula, CAB It is the channel attention module, and its specific calculation method is:

[0019]

[0020] In the above formula, represents maximum pooling, Represents a matrix multiplication calculation.

[0021] S2.3: Perform image upsampling on two of the three calculation results in the previous step:

[0022]

[0023]

[0024] In the above formula, Represents multiples of n The image upsampling operation.

[0025] S2.4: Concatenate the three image features to obtain the middle-level features:

[0026]

[0027] In the above formula, Represents a splicing operation.

[0028] S3: Input the middle-level features into the high-frequency enhanced frequency domain feature calculation module to obtain deep-level features;

[0029] S4: Convolve the deep features to obtain enhanced features;

[0030] S5: Fuse the enhanced features with the original input features and finally output a super-resolution image.

[0031] Preferably, S1 comprises the following steps:

[0032] S1.1: Acquire three-channel images in RGB color space ;

[0033] S1.2: Perform dynamic convolution calculation to obtain shallow features:

[0034] First, perform convolution:

[0035]

[0036] In the above formula, The representative size is n The convolution kernel of

[0037] Then perform global feature extraction to obtain the global feature vector G :

[0038]

[0039] In the above formula, stands for adaptive average pooling;

[0040] Then dynamic weight K generate:

[0041]

[0042] In the above formula, W 1 and W 2 represents the learnable weight matrix, ReLU represents the activation function, softmax The formula is: , N is the number of feature channels;

[0043] Finally, dynamic convolution operation is performed to obtain shallow features f 1:

[0044]

[0045] In the above formula, substitute The convolution weights are K The convolution kernel.

[0046] Preferably, S3 includes the following steps:

[0047] S3.1: Calculation The mean feature map of :

[0048]

[0049] In the above formula, Represents the mean of each channel in the calculation graph;

[0050] S3.2: Obtain high-frequency feature maps by difference calculation, combined with Output deep features:

[0051]

[0052] Preferably, S4 includes the following steps:

[0053] S4.1: Yes Perform convolution calculation to obtain enhanced features:

[0054]

[0055] Preferably, S5 includes the following steps:

[0056] S5.1: and Perform feature fusion and finally output super-resolution image :

[0057]

[0058] S5.2: Obtain a lightweight image super-resolution model based on spatial and frequency domains;

[0059] S5.3: Evaluate the model image super-resolution effect by calculating the quantitative indicators PSNR and SSIM.

[0060] The second technical solution adopted by the present invention is: a lightweight image super-resolution method and device based on spatial domain and frequency domain, comprising:

[0061] Multi-scale perception spatial domain module: used to capture texture information on larger spatial structures;

[0062] Frequency domain module for high-frequency enhancement: used to emphasize the texture details of high-frequency information in the image;

[0063] Feature fusion module: used to merge the enhanced features with the original image to generate a reconstructed super-resolution image.

[0064] The beneficial effects of the method and device of the present invention are as follows: the present invention first obtains shallow features through convolution calculation; inputs these into a multi-scale perception spatial domain feature calculation module to obtain mid-level features; then inputs them into a high-frequency enhanced frequency domain feature calculation module to obtain deep features; and finally, performs convolution calculation to obtain enhanced features, which are fused with the original input features to ultimately output a super-resolution image. The present invention can effectively address the common challenges of existing neural network models in image super-resolution. On the one hand, it utilizes a multi-scale perception spatial domain module to effectively capture texture information on larger spatial structures; on the other hand, it utilizes a high-frequency enhanced frequency domain module to emphasize texture details in the high-frequency information of the image. Simultaneously, the current modules and processes utilize lightweight calculations, ultimately achieving a lightweight image super-resolution effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the prior art and the drawings required for use in the embodiments. The following drawings are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0066] FIG1 is a flow chart of a lightweight image super-resolution method based on spatial and frequency domains according to the present invention;

[0067] FIG2 is a schematic diagram of a specific embodiment of a lightweight image super-resolution method based on spatial domain and frequency domain according to the present invention;

[0068] Figure 3 This is a schematic diagram of the structure of a spatial feature calculation module of a lightweight image super-resolution method based on spatial domain and frequency domain in the present invention;

[0069] Figure 4 This is a schematic structural diagram of a frequency domain feature calculation module of a lightweight image super-resolution method based on spatial domain and frequency domain of the present invention;

[0070] Figure 5 This is a super-resolution effect diagram after the present invention is implemented. Specific implementation plan

[0071] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making creative work are within the scope of protection of the present invention.

[0072] The embodiments of the present invention provide a lightweight image super-resolution method and apparatus based on the spatial domain and the frequency domain, which are used to achieve high-quality image super-resolution effects by using a lightweight computing method at the algorithm level.

[0073] A typical embodiment of the present invention, referring to Figure 1 、 Figure 2 , the method comprises the following steps:

[0074] S1: Collect images, perform dynamic convolution, and obtain shallow features;

[0075] S2: Input shallow features into the multi-scale perception spatial feature calculation module to obtain mid-level features; S2 includes the following steps:

[0076] S2.1: Perform two downsampling operations on shallow features:

[0077]

[0078]

[0079] In the above formula, Represents multiples of n Image downsampling operation.

[0080] S2.2: Perform channel attention calculation on features at three different scales:

[0081]

[0082]

[0083]

[0084] In the above formula, CAB It is the channel attention module, and its specific calculation method is:

[0085]

[0086] In the above formula, represents maximum pooling, Represents a matrix multiplication calculation.

[0087] S2.3: Perform image upsampling on two of the three calculation results in the previous step:

[0088]

[0089]

[0090] In the above formula, Represents multiples of nThe image upsampling operation.

[0091] S2.4: Concatenate the three image features to obtain the middle-level features:

[0092]

[0093] In the above formula, Represents a splicing operation.

[0094] S3: Input the middle-level features into the high-frequency enhanced frequency domain feature calculation module to obtain deep-level features;

[0095] S4: Convolve the deep features to obtain enhanced features;

[0096] S5: Fuse the enhanced features with the original input features and finally output a super-resolution image.

[0097] Furthermore, as a preferred embodiment of the present method, S1 includes the following steps:

[0098] S1.1: Acquire three-channel images in RGB color space ;

[0099] Furthermore, the three-channel image of the RGB color space is collected , which specifically include:

[0100] Collect a dataset of original clear images at high resolution and label them as "hr". Then perform image downsampling. The downsampling operation can use image interpolation operations such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation to obtain a three-channel image in RGB color space. , marked as the "lr" tag.

[0101] S1.2: Perform dynamic convolution calculation to obtain shallow features:

[0102] First, perform convolution:

[0103]

[0104] In the above formula, The representative size is n The convolution kernel.

[0105] Then perform global feature extraction to obtain the global feature vector G :

[0106]

[0107] In the above formula, stands for Adaptive Average Pooling.

[0108] Then dynamic weightK generate:

[0109]

[0110] In the above formula, W 1 and W 2 represents the learnable weight matrix, ReLU represents the activation function, softmax The formula is: , N is the number of feature channels.

[0111] Finally, dynamic convolution operation is performed to obtain shallow features f 1:

[0112]

[0113] In the above formula, substitute The convolution weights are K The convolution kernel.

[0114] Furthermore, S3 includes the following steps:

[0115] S3.1: Calculation The mean feature map of :

[0116]

[0117] In the above formula, Represents the mean of each channel in the computation graph.

[0118] S3.2: Obtain high-frequency feature maps by difference calculation, combined with Output deep features:

[0119]

[0120] Further, S4 includes the following steps:

[0121] right Perform convolution calculation to obtain enhanced features:

[0122]

[0123] Further, S5 includes the following steps:

[0124] S5.1: and Perform feature fusion and finally output super-resolution image :

[0125]

[0126] S5.2: Obtain a lightweight image super-resolution model based on spatial and frequency domains;

[0127] S5.3: Evaluate the model image super-resolution effect by calculating the quantitative indicators PSNR and SSIM.

[0128] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make other equivalent modifications or substitutions without violating the spirit of the invention, and these equivalent modifications or substitutions are included in the scope defined by the claims.

Claims

1. A lightweight image super-resolution method based on spatial and frequency domains, characterized in that: The method comprises the following steps: S1: Collect images, perform dynamic convolution, and obtain shallow features; S2: Input shallow features into the multi-scale perception spatial feature calculation module to obtain mid-level features; S2 includes the following steps: S2.1: Perform two downsampling operations on shallow features: In the above formula, Represents multiples of n Image downsampling operation; S2.2: Perform channel attention calculation on features at three different scales: In the above formula, CAB It is the channel attention module, and its specific calculation method is: In the above formula, represents maximum pooling, Represents matrix multiplication calculation; S2.3: Perform image upsampling on two of the three calculation results in the previous step: In the above formula, Represents multiples of n Image upsampling operation; S2.4: Concatenate the three image features to obtain the middle-level features: In the above formula, Represents a splicing operation; S3: Input the middle-level features into the high-frequency enhanced frequency domain feature calculation module to obtain deep-level features; S4: Convolve the deep features to obtain enhanced features; S5: Fuse the enhanced features with the original input features and finally output a super-resolution image.

2. The lightweight image super-resolution method based on spatial domain and frequency domain according to claim 1, characterized in that: Said S1 comprises the following steps: S1.1: Acquire three-channel images in RGB color space ; S1.2: Perform dynamic convolution calculation to obtain shallow features: First, perform convolution: In the above formula, The representative size is n The convolution kernel of Then perform global feature extraction to obtain the global feature vector G : In the above formula, stands for adaptive average pooling; Then dynamic weight K generate: In the above formula, W 1 and W 2 represents the learnable weight matrix, ReLU represents the activation function, softmax The formula is: , N is the number of feature channels; Finally, dynamic convolution operation is performed to obtain shallow features f 1: In the above formula, substitute The convolution weights are K The convolution kernel.

3. The lightweight image super-resolution method based on spatial domain and frequency domain according to claim 1, characterized in that: S3 includes the following steps: S3.1: Calculation The mean feature map of : In the above formula, Represents the mean of each channel in the calculation graph; S3.2: Obtain high-frequency feature maps by difference calculation, combined with Output deep features: 。 4. The lightweight image super-resolution method based on spatial domain and frequency domain according to claim 1, characterized in that: S4 includes the following steps: S4.1: Yes Perform convolution calculation to obtain enhanced features: 。 5. The lightweight image super-resolution method based on spatial domain and frequency domain according to claim 1, characterized in that: S5 includes the following steps: S5.1: and Perform feature fusion and finally output super-resolution image : S5.2: Obtain a lightweight image super-resolution model based on spatial and frequency domains; S5.3: Evaluate the model image super-resolution effect by calculating the quantitative indicators PSNR and SSIM.

6. A device using the lightweight image super-resolution method based on spatial domain and frequency domain according to any one of claims 1 to 5, characterized in that: include: Multi-scale perception spatial domain module: used to capture texture information on larger spatial structures; Frequency domain module for high-frequency enhancement: used to emphasize the texture details of high-frequency information in the image; Feature fusion module: used to merge the enhanced features with the original image to generate a reconstructed super-resolution image.

Images