Lightweight image super-resolution network based on local self-attention

Abstract

The invention relates to the technical field of image processing, in particular to a lightweight image super-resolution network based on local self-attention, which comprises a shallow feature extraction module, a plurality of multi-scale local feature extraction modules, a global self-attention module and an up-sampling module, the multi-scale local feature extraction module is used for extracting local features according to gradually increased feature scales, and the multi-scale local feature extraction module comprises a micro-scale feature extraction module, a mesoscale local self-attention module, a deformable local self-attention module and a cavity local self-attention module which are cascaded. According to the method provided by the invention, the pixels with high correlation degree can be accurately utilized in a relatively large space range, and the reconstruction of a high-quality SR image is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of graphics processing technology, and in particular to a lightweight image super-resolution network based on local self-attention. Background Art

[0002] Image super-resolution (SR) can restore detail-rich high-resolution (HR) images from existing low-resolution images (LR), and is therefore widely used in remote sensing imaging, infrared imaging, medical imaging, video surveillance and other fields.

[0003] There are three mainstream methods for implementing SR tasks in the existing technology.

[0004] The interpolation-based SR method is a traditional image enhancement technology, which mainly includes bilinear interpolation, bicubic interpolation, etc. Although these methods are simple and easy to implement, their shortcomings are also obvious. They are usually unable to restore fine textures and edge information in the image. Processing complex images may cause slight blurring in the image. They cannot restore the details or textures that are originally lacking, have no learning ability, and require relatively high computational costs.

[0005] The other type is to use the powerful feature expression ability of Convolutional Neural Network (CNN) to form CNN-based SR methods, such as IMDN and VapSR. In order to improve the quality of SR images, this type of CNN-based SR method expands the receptive field by stacking deeper networks. However, deepening the network will significantly increase the computational complexity, which is not conducive to application in practical scenarios.

[0006] Advanced SR methods introduce Transformers with strong global information modeling capabilities into SR tasks, such as ESRT, LBNet, and Omni-SR, to form lightweight SR algorithms. An attribution analysis method called LAM can be used to visually check the spatial range of informative pixels used by advanced lightweight SR methods. Among them, LAM can emphasize the pixels used by the SR method in the reconstruction process. DI can reflect the spatial range of the pixels used, and the higher the DI, the more pixels the SR method uses. Figure 1 The LAM images obtained by different advanced lightweight SR methods are shown. Figure 1In the figure, the image blocks marked by the red and green hollow boxes represent the image blocks to be reconstructed. Since the image blocks marked by the red hollow box and the red solid box have similar texture details, the pixels in the red solid box have a higher reference value during the SR process of the image blocks marked by the red hollow box. Similarly, the pixels in the green solid box have a higher reference value during the SR process of the image blocks marked by the green hollow box.

[0007] like Figure 1 As shown in the figure, compared with CNN-based IMDN and VapSR, Transformer-based ESRT and LBNet use a narrower spatial range of pixels, and the resulting SR image quality is better (for example, higher PSNR objective indicators, better subjective visual effects). Although Omni-SR uses a larger spatial range of pixels, due to excessive reference to areas with low texture detail similarity, the quality of its SR image is poor and even spatial distortion occurs.

[0008] The reason for the above phenomenon is that Figure 1 In a class of images represented by , there are few long-range structural similarities, so it is difficult for Transformer to effectively model long-range feature dependencies. Different from the way ESRT and LBNet model feature dependencies, Omni-SR divides the entire image into several non-overlapping 8×8 blocks through the proposed Meso-OSA module, and uses the self-attention mechanism to model the dependencies between the 64 pixels in these blocks. However, due to the presence of distracting image blocks and the overly sparse pixel sampling of the Global-OSA module, Omni-SR has difficulty accurately modeling the dependencies between pixels, resulting in a decrease in SR performance. Summary of the invention

[0009] The present invention provides a lightweight image super-resolution network based on local self-attention, which can accurately utilize pixels with high correlation in a large spatial range, and is helpful to reconstruct high-quality SR images.

[0010] This application provides the following technical solutions:

[0011] The lightweight image super-resolution network LSANet based on local self-attention includes a cascaded shallow feature extraction module SFE, several multi-scale local feature extraction modules MSLE, a global self-attention module GSA and an upsampling module, wherein the shallow feature extraction module SFE is used to extract shallow features from the input low-resolution image, the multi-scale local feature extraction module MSLE is used to extract local features according to gradually increasing feature scales, the global self-attention module GSA is used to extract global features, and the upsampling module is used to restore the spatial resolution of the feature image. The LSANet can be expressed as:

[0012]

[0013] Among them, I SR represents the reconstructed SR image, I LR represents the LR image to be reconstructed; f SFE (·) indicates the function corresponding to the SFE module; Represents the functions corresponding to n MSLE modules; f GSA (·) represents the function corresponding to the GSA module; ↑(·) represents the function corresponding to the upsampling module.

[0014] Furthermore, the MSLE module includes a cascaded micro-scale feature extraction module FSFE, a medium-scale local self-attention module MsLSA, a deformable local self-attention module DeLSA, and a hollow local self-attention module DiLSA, wherein:

[0015] The micro-scale feature extraction module FSFE is used to extract important features around the center point at a micro-scale;

[0016] The mesoscale local self-attention module MsLSA is used to efficiently model the dependencies between neighboring pixels in a sliding window at a mesoscale and extract high-quality local features;

[0017] The deformable local self-attention module DeLSA is used to break through the limitation of sliding window on the receptive field. It finds feature points and models the dependencies between them according to the generated offset matrix to enhance the network's adaptability to unknown LR images.

[0018] The dilated local self-attention module DiLSA is used to further expand the receptive field and extract local features at a larger scale.

[0019] Further, the MSLE module can be expressed as:

[0020]

[0021] in Represents the input features of the MSLE module; f FSFE (·) represents the function corresponding to the FSFE module; f MSLSA (·) represents the function corresponding to the MsLSA module; f DeLSA (·) represents the function corresponding to the DeLSA module; f DiLSA (·) represents the function corresponding to the DiLSA module.

[0022] Furthermore, the mesoscale local self-attention module MsLSA includes a cascaded moving sliding window unit, a patch extraction unit, a multi-head self-attention unit and a feedforward neural network unit, wherein:

[0023] Moving sliding window unit, used to move the sliding window to extract the set W of local feature blocks ms

[0024]

[0025] in, Represents the function corresponding to the operation of extracting local feature blocks; Represents the input features of the local feature block; α ms and β ms Represent the size and step size of the sliding window respectively; Represents the i-th local feature block; Representative in W ms The number of local feature blocks in ; Represents the operation of rounding a value upwards; Respectively represent the length and width of the input feature map;

[0026] Patch extraction unit, used to extract To extract the set of patches that participate in dependency modeling, the process can be defined as:

[0027]

[0028] in, Represents the i-th local feature block in MsLSA A collection of patches in the Representative from Extract the function corresponding to the patch operation; Represents the i-th local feature block The jth patch of , (x0, y0) represents the patch in The absolute position on Representatives in The number of patches in δ ms Represents the size of the patch;

[0029] Multi-head self-attention unit for modeling using self-attention mechanism The dependencies between patches in the model are calculated and the set of modeled patches is output. The process can be defined as:

[0030]

[0031] Among them, Attenion(·) represents the function corresponding to Multi-Head Self-Attention (MHSA); Norm(·) represents the function corresponding to layer normalization; FFN(·) represents the function corresponding to Feed-Forward Network (FFN); Γ(·) represents the operation of expanding a patch into a token; ∧(·) represents the operation of folding a token into a patch;

[0032] Feedforward neural network unit, used to Perform the feature shape transformation operation opposite to the above process to obtain the output feature

[0033] Furthermore, the deformable local self-attention module DeLSA includes a cascaded moving sliding window unit, an offset matrix extraction unit, a patch extraction unit, a multi-head self-attention unit and a feedforward neural network unit, wherein:

[0034] Moving sliding window unit, used to move the sliding window to extract the set W of local feature blocks ms , which is the same as the moving sliding window unit of MsLSA;

[0035] The offset matrix extraction unit OEB is used to dynamically adjust the position of the convolution kernel to adapt to local deformation in the image and can be expressed as:

[0036]

[0037] in, Represents the input features of OEB; CMP(·) represents the function corresponding to channel average (maximum) pooling; Reshape(·) represents the function corresponding to the vector shape reconstruction operation; Sigmoid(·) represents the Sigmoid activation function; Linear(·) represents the function corresponding to the linear mapping layer; Update(·) represents the function corresponding to the rounding and offset limiting operation of the offset matrix; Offset represents the output offset matrix;

[0038] Patch extraction unit, used to extract To extract the set of patches that participate in dependency modeling, the process can be defined as:

[0039]

[0040] in, represents the set of patches in the ith local feature block in DeLSA; (Δx, Δy) represents the offset, Δx, Δy∈[-MAX,MAX], and Represents an integer; Represents the patch after the offset is introduced; Represents the number of local feature blocks in DeLSA; Represents the number of patches in the local feature block; α de Represents the size of the local feature block in DeLSA;

[0041] Multi-head self-attention unit for modeling using self-attention mechanism The dependencies between patches in the model are calculated and the set of modeled patches is output. Same as the multi-head self-attention unit of MsLSA;

[0042] Feedforward neural network unit, used to Perform the feature shape transformation operation opposite to the above process to obtain the output feature Same feedforward neural network unit as MsLSA.

[0043] Furthermore, the dilated local self-attention module DiLSA is used to capture richer contextual information by further expanding the receptive field, including a cascaded moving sliding window unit, a patch extraction unit, a multi-head self-attention unit and a feedforward neural network unit, wherein the patch extraction unit, the multi-head self-attention unit and the feedforward neural network unit are the same as MsLSA, and the process of moving the sliding window and extracting the local feature block can be defined as:

[0044]

[0045] Among them, W di Represents a collection of extracted local feature blocks; Represents the input features of DiLSA; DiLSA uses a sliding window to The function corresponding to the operation of extracting local feature blocks; α di , β di and τ di Represent the size, step size and hole size of the sliding window with holes respectively; Representative in W di The i-th local feature block in; Represent the length and width of the input feature map respectively.

[0046] Furthermore, the number of MSLEs is set to 5, and the number of channels of the network is set to 64; each MSLE module includes 1 MsLSA, 1 DeLSA and 1 DiLSA; the sliding window size of MsLSA is 5x5, and the step size is 3; the sliding window size of DeLSA is 7x7, and the step size is 7, and the minimum and maximum values ​​of the offset are set to -7 and 7 respectively; the sliding window size of DiLSA is 11x11, and the step size is 11; the hole of DiLSA is set to 1, that is, the receptive field of DiLSA is 22×22.

[0047] Furthermore, the lightweight image super-resolution network uses the training set As the LR-HR image pair, the mean absolute error (MAE) is used to optimize the parameters, and the error function is defined as follows:

[0048]

[0049] Where L(Θ) represents the error function; Θ represents the learnable parameter; N represents the number of images in the training set; ‖·‖ MAE Represents the function corresponding to MAE; the initial learning rate is 5×10 -4 And it decays by half every 200 epochs, patch_size is set to 64, and batch_size is set to 64.

[0050] The principles and advantages of the present invention are:

[0051] The present invention proposes a lightweight image super-resolution network LSANet based on local self-attention, which consists of a shallow feature extraction block (SFE), a multi-scale local feature extraction block (MSLE), a global self-attention module (GSA) and an upsampling module (UpSample). In the MSLE module, local feature extraction modules of various scales are constructed, and the extracted feature scales gradually increase from shallow to deep. First, the FSFE module can extract important local features around the center point at a microscale; secondly, the present invention designs a medium-scale local self-attention module (MsLSA) to model the dependency between neighboring pixels in the sliding window, and to model the dependency between pixels on the entire image by moving the sliding window, thereby improving the feature expression capability; then, the present invention designs a deformable local self-attention module (DeLSA), which adaptively adjusts the scale and receptive field by generating an offset matrix to break through the restriction of the rectangle on the receptive field, thereby enhancing the network's adaptability to unknown low-resolution images; finally, a dilated local self-attention module (DiLSA) is proposed, which further expands the receptive field by introducing holes to extract large-scale local features, which helps the proposed LSANet to obtain stronger local information modeling capabilities and obtain high-quality SR images. By combining MsLSA, DeLSA and DiLSA, a multi-scale local feature extraction module (MSLE) is constructed, which extracts local features of different scales through the self-attention mechanism and complements the global features to improve the SR performance.

[0052] The LSANet proposed in this paper can accurately utilize pixels with high correlation in a large spatial range, which is helpful to reconstruct high-quality SR images. It is superior to other advanced SR methods in both quantitative indicators and visual quality, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 LAM graphs obtained by different current advanced lightweight SR methods in the prior art;

[0054] Figure 2 The diagram is an architecture diagram of a lightweight image super-resolution network based on local self-attention of the present invention;

[0055] Figure 3 It is the structural diagram of each unit of the MSLE module;

[0056] Figure 4 Comparison of the calculation process of MsLSA, ViT and standard convolution;

[0057] Figure 5 Schematic diagram for extracting the offset matrix;

[0058] Figure 6 Comparison between DiLSA and LAM with dilated convolution of the same receptive field;

[0059] Figure 7 The impact of the sliding window size and step size on performance and efficiency in MsLSA;

[0060] Figure 8 The impact of DiLSA with different holes on SR performance;

[0061] Fig. 9 Comparison of visual perception quality between LSANet and advanced SR methods. DETAILED DESCRIPTION

[0062] Image resolution is an important parameter of an image and can be used to characterize the fineness of the image. The higher the resolution of an image, the more pixels it contains, the more details it can provide, and therefore the clearer the image will be. In the fields of image processing and computer vision, "low resolution" and "super resolution" are often used to describe image quality or image processing techniques. Low resolution refers to a low pixel density of an image, that is, a small number of pixels in the image. This results in insufficient image detail and may appear blurry. Super resolution is an image enhancement technique that refers to the process of restoring a high-resolution image from one or more low-resolution images. Its purpose is to improve the clarity and details of an image and make the image look sharper and more realistic.

[0063] The present invention proposes a lightweight image super-resolution network based on local self-attention, which can accurately utilize pixels with high correlation in a large spatial range to reconstruct low-resolution images and obtain high-quality high-resolution images.

[0064] The following is further described in detail through specific implementation methods:

[0065] Figure 2 Shown is an architecture diagram of a lightweight image super-resolution network based on local self-attention provided by an embodiment of the present invention.

[0066] In one embodiment, the lightweight image super-resolution network LSANet based on local self-attention provided by an embodiment of the present invention includes a cascaded shallow feature extraction module SFE, several multi-scale local feature extraction modules MSLE, a global self-attention module GSA and an upsampling module, wherein the shallow feature extraction module SFE is used to extract shallow features from the input low-resolution image, the multi-scale local feature extraction module MSLE is used to extract local features according to gradually increasing feature scales, the global self-attention module GSA is used to extract global features, and the upsampling module is used to restore the spatial resolution of the feature image.

[0067] The LSANet can be expressed as:

[0068]

[0069] Among them, I SR represents the reconstructed SR image, I LR represents the LR image to be reconstructed; f SFE (·) indicates the function corresponding to the SFE module; Represents the functions corresponding to n MSLE modules; f GSA (·) represents the function corresponding to the GSA module; ↑(·) represents the function corresponding to the upsampling module.

[0070] In the implementation process, the shallow feature extraction module includes a 3x3 convolution layer; the multi-scale local feature extraction modules are connected in a cascade manner, the GSA consists of a multi-head self-attention module and a feedforward neural network, and the upsampling module is implemented using sub-pixel convolution.

[0071] In one embodiment, the MSLE module includes a cascaded micro-scale feature extraction module FSFE, a plurality of medium-scale local self-attention modules MsLSA, a plurality of deformable local self-attention modules DeLSA, and a plurality of void local self-attention modules DiLSA, wherein:

[0072] The micro-scale feature extraction module FSFE is used to extract important features around the center point at a micro-scale. Its structure is as follows Figure 3 (a)

[0073] The medium-scale local self-attention module MsLSA is used to efficiently model the dependencies between neighboring pixels in the sliding window at a medium scale, thereby extracting high-quality local features. Its structure is as follows Figure 3 (b)

[0074] The deformable local self-attention module DeLSA is used to break through the limitation of the sliding window on the receptive field. It finds feature points and models the dependencies between them according to the generated offset matrix to enhance the network's adaptability to unknown LR images. Its structure is as follows: Figure 3 (c)

[0075] The dilated local self-attention module DiLSA is used to further expand the receptive field and extract local features at a larger scale. Its structure is as follows: Figure 3 (d) as shown.

[0076] In one embodiment, the MSLE module may be represented as:

[0077]

[0078] in Represents the input features of the MSLE module; f FSFE (·) represents the function corresponding to the FSFE module; f MSLSA (·) represents the function corresponding to the MsLSA module; f DeLSA (·) represents the function corresponding to the DeLSA module; f DiLSA (·) represents the function corresponding to the DiLSA module.

[0079] In one embodiment, in order to enhance the local feature extraction capability, the inventors proposed a medium-scale local self-attention module MsLSA. Different from the visual Transformer (ViT) modeling the dependencies between large-scale (e.g., 16×16) patches, the proposed MsLSA can not only model the dependencies between 1×1 pixels within the sliding window, but also model the dependencies between pixels on the entire image by moving the sliding window, by using smaller-scale patches. This implements a local feature extraction operation similar to the standard convolution. At the same time, the proposed MsLSA significantly reduces the computational complexity relative to ViT. Figure 4 Shows the computational process comparison of MsLSA, ViT, and standard convolution.

[0080] The mesoscale local self-attention module includes a cascaded moving window unit, a patch extraction unit, a multi-head self-attention unit, and a feedforward neural network unit. Specifically, the proposed MsLSA can be divided into four steps:

[0081] 1) Given an input feature map in and Represent the length, width and number of channels of the input feature map respectively. The proposed MsLSA extracts the set W of local feature blocks by moving the sliding window. ms ,

[0082]

[0083] in, Represents the function corresponding to the operation of extracting local feature blocks; Represents the input features of the local feature block; α ms and β ms Represent the size and step size of the sliding window respectively. Represents the i-th local feature block; Representative in W ms The number of local feature blocks in ; Represents the operation of rounding a value upwards; Represent the length and width of the input feature map respectively.

[0084] 2) Then, MsLSA To extract the set of patches that participate in dependency modeling. The process can be defined as:

[0085]

[0086] in, Represents the i-th local feature block in MsLSA A collection of patches in . Representative from Extract the function corresponding to the patch operation. Represents the i-th local feature block The jth patch of , (x0, y0) represents the patch in The absolute position on. Representatives in The number of patches in δ ms represents the size of the patch. In particular, when α ms =β ms And δ ms When =1, the extraction patch of MsLSA is equivalent to the Meso-OSA module in Omni-SR.

[0087] 3) Next, MsLSA uses the self-attention mechanism to model The dependencies between patches in the model are calculated and the set of modeled patches is output. The process can be defined as:

[0088]

[0089] Among them, Attention(·) represents the function corresponding to Multi-Head Self-Attention (MHSA). Norm(·) represents the function corresponding to layer normalization. FFN(·) represents the function corresponding to Feed-Forward Network (FFN). Γ(·) represents the operation of expanding a patch into a token. Λ(·) represents the operation of folding a token into a patch.

[0090] 4) Finally Perform the feature shape transformation operation opposite to the above process to obtain the output feature

[0091] In one embodiment, in order to break through the limitation of sliding window on receptive field, the inventors proposed a deformable local self-attention module (DeLSA). By adaptively adjusting the pixels involved in dependency modeling, the proposed DeLSA can not only make the model emphasize important features, but also alleviate the model's erroneous feature dependencies. Unlike MsLSA, DeLSA adds a learnable offset matrix extraction before extracting the Patch operation, so that the absolute position of the patch involved in the operation on the feature map is offset.

[0092] like Figure 3 As shown in (c), the deformable local self-attention module DeLSA includes a moving sliding window unit, an offset matrix extraction unit, a patch extraction unit, a multi-head self-attention unit, and a feedforward neural network unit. Specifically, the proposed DeLSA can be divided into five steps, including moving the sliding window and extracting local feature blocks, offset matrix extraction, extracting patches, multi-head self-attention, and FFN, where:

[0093] Move the sliding window and extract the local feature block W ms , this step is the same as MsLSA;

[0094] Offset matrix extraction is a key step in the DeLSA module. To solve this problem, the inventors proposed an efficient offset matrix extraction block (OEB). Since the time and space complexity of OEB is independent of the number of channels and only related to the size of the sliding window, it can efficiently extract the offset matrix.

[0095] The structure of OEB is as follows Figure 5The input of this step is a feature map with a length of H, a width of W, and a number of channels W. First, a 2k*2k pooling window is used to perform a channel maximum pooling operation on the input feature map to obtain a C*2k*2k feature map; then the pooled feature map is reshaped into a 1*4K 2 The one-dimensional vector is transformed into 1*4K through the linear layer. 2 The vector is then constrained to a value between 0 and 1 through the Sigmoid function. The vector activated by Sigmoid is reshaped into a 2K*K two-dimensional matrix. This two-dimensional matrix is ​​updated to an offset matrix after superimposing the offset. The offset matrix is ​​used in the subsequent Patch extraction step to dynamically adjust the position of the convolution kernel. In this way, the model can process complex images more flexibly and improve the accuracy of prediction.

[0096] The OEB module can be represented as:

[0097]

[0098] in, Represents the input features of OEB; CMP(·) represents the function corresponding to Channel MaxPooling (CMP); Reshape(·) represents the function corresponding to the vector shape reconstruction operation; Sigmoid(·) represents the Sigmoid activation function; Linear(·) represents the function corresponding to the linear mapping layer; Update(·) represents the function corresponding to the operation of rounding the value of the offset matrix and limiting the offset to [-MAX, MAX]; Offset represents the output offset matrix.

[0099] The steps to extract a patch can be defined as:

[0100]

[0101] in, represents the set of patches in the ith local feature block in DeLSA; (Δx, Δy) represents the offset, Δx, Δy∈[-MAX,MAX], and Represents an integer; Represents the patch after the offset is introduced; Represents the number of local feature blocks in DeLSA; Represents the number of patches in the local feature block; α de Represents the size of the local feature block in DeLSA;

[0102] The steps of multi-head self-attention and FFN are the same as MsLSA.

[0103] In one embodiment, in order to enhance the details of the SR image, the inventors proposed a dilated local self-attention module (DiLSA) to capture richer contextual information by further expanding the receptive field. Unlike Global-OSA in Omni-SR, where the pixels involved in dependency modeling are sparse, DiLSA involves more dense pixels in dependency modeling.

[0104] DiLSA includes a moving sliding window unit, a patch extraction unit, a multi-head self-attention unit, and a feedforward neural network unit. The patch extraction unit, the multi-head self-attention unit, and the feedforward neural network unit are the same as MsLSA. The process of moving the sliding window and extracting local feature blocks can be defined as:

[0105]

[0106] Among them, W di Represents a collection of extracted local feature blocks; Represents the input features of DiLSA; DiLSA uses a sliding window to The function corresponding to the operation of extracting local feature blocks; α di , β di and τ di Represent the size, step size and hole size of the sliding window with holes respectively; Representative in W di The i-th local feature block in; Represent the length and width of the input feature map respectively.

[0107] In actual operation, the size of the sliding window is adjusted by setting different dilations to obtain different dilation multiples. The design of the proposed DiLSA module is similar to that of the dilated convolution, but has better large-scale local feature extraction capabilities. Figure 6 The proposed DiLSA is compared with the LAM of Dilated Convolution (DConv). Figure 6 It can be seen that although the receptive fields of DiLSA and DConv are consistent, DiLSA actually utilizes pixels in a larger spatial range, which indicates that DiLSA helps the proposed LSANet to obtain stronger local information modeling capabilities and obtain high-quality SR images.

[0108] In one implementation, the final parameters of the lightweight image super-resolution network are determined by comparing the performance of MSLE modules with different parameters:

[0109] 1) In the MSLE module, the feature extraction scales corresponding to the proposed MsLSA, DeLSA and DiLSA gradually increase. In order to verify the effectiveness of the proposed MsLSA, DeLSA and DiLSA, the inventors constructed MSLE modules containing different numbers of MsLSA, DeLSA and DiLSA, and used Set5 (×4) as the test set. The performance comparison results are shown in Table 1.

[0110] Table 1

[0111]

[0112] As can be seen from Table 1, when MSLE contains 3 MsLSAs, the corresponding network can only extract features of a single scale and the receptive field is limited to the sliding window. When MSLE contains 2 MsLSAs and 1 DeLSA, the corresponding network obtains a 0.18dB increase in PSNR on Set5(×4). This is because the added DeLSA can not only break through the fixed receptive field of the sliding window, but also adaptively adjust the pixels involved in dependency modeling. When MSLE contains 1 MsLSA, 1 DeLSA and 1 DiLSA, the corresponding network obtains a 0.30dB increase in PSNR. This is because the added DiLSA can use a larger receptive field to extract multi-scale local features and achieve scale complementarity by jointly using MsLSA, DeLSA and DiLSA. The above results verify the effectiveness of the proposed MsLSA, DeLSA and DiLSA.

[0113] 2) The scale complementarity of MsLSA, DeLSA and DiLSA enables the proposed LSANet to extract high-quality local features, thereby improving the quality of SR images. However, the influence of hyperparameters on the performance of the above three modules is extremely critical. Figure 7 The effect of sliding windows of different sizes and step lengths on performance and efficiency in MsLSA is shown. Figure 7 It can be clearly seen that when the window size is equal to 3, the PSNR obtained by the corresponding model is not good because the window is too small and the receptive field is limited. When the step size is constant, the increase in window size will improve PSNR, but it will also significantly increase FLOPs. When the window size is constant, the increase in step size will reduce PSNR, but it will also significantly reduce FLOPs. Therefore, in order to better balance performance and efficiency, the sliding window size of MsLSA is selected as 5×5 and the step size is 3.

[0114] Table 2

[0115]

[0116] Table 2 shows the effect of different maximum offset values ​​MAX on the performance of DeLSA. As can be seen from Table 2, the PSNR value does not increase with the increase of the maximum offset value MAX. This is because too high an offset value may cause excessive distortion of the receptive field, resulting in the establishment of incorrect feature dependencies between unrelated pixels, thereby reducing PSNR. Compared with the images in Set5, the images in Urban100 have a larger resolution and more similar image blocks. Therefore, too high an offset value has little effect on Urban100. Finally, DeLSA with a maximum offset value of 7 is selected, that is, the pixel offset is between [-7,7].

[0117] Figure 8 The effect of using different holes in DiLSA on SR performance is shown. Figure 8 It can be seen that for the proposed DiLSA, increasing Dilation can expand the network receptive field without increasing the number of network parameters. When Dilation is equal to 0, DiLSA is equivalent to MsLSA. In this case, its receptive field is limited, resulting in SR performance degradation. When Dilation is equal to 2, 3, or 4, although the receptive field increases, sparse sampling leads to a lack of correlation between distant pixels, which also leads to SR performance degradation. Therefore, DiLSA with Dilation equal to 1 is selected as the final version.

[0118] In the proposed LSANet, the number of MSLEs is set to 5, and the number of channels of the network is set to 64. The corresponding network parameters are around 700k, which can achieve a good balance between performance and efficiency. Each MSLE module includes 1 MsLSA, 1 DeLSA and 1 DiLSA; the sliding window size of MsLSA is 5x5, and the step size is 3; the sliding window size of DeLSA is 7×7, the step size is 7, and the minimum and maximum values ​​of the offset are set to -7 and 7 respectively; the sliding window size of DiLSA is 11×11, and the step size is 11; the hole of DiLSA is set to 1, that is, the receptive field of DiLSA is 22×22.

[0119] In one embodiment, the lightweight image super-resolution network uses a training set As the LR-HR image pair, the mean absolute error (MAE) is used to optimize the parameters, and the error function is defined as follows:

[0120]

[0121] Where L(Θ) represents the error function; Θ represents the learnable parameter; N represents the number of images in the training set; ‖·‖ MAE Represents the function corresponding to MAE.

[0122] During the training phase, the initial learning rate is 5×10-4 And the decay is halved every 200 epochs, patch_size is set to 64, and batch_size is set to 64. The model is implemented on the PyTorch 1.11.0 platform, and all training is done on an NVIDIA V100 GPU.

[0123] Furthermore, the inventors designed a variety of experiments to verify the indicators of the lightweight image super-resolution network based on local self-attention and other advanced SR methods.

[0124] 1) Compare the objective indicators of LSANet and advanced SR methods using benchmark test sets

[0125] Table 3 shows the FLOPs comparison between LSANet and advanced Transformer-based SR methods. The experiment is conducted on the Manga109 (×4) test dataset. Although the FLOPs of LSANet is slightly higher than LBNet and Omni-SR, LSANet is still highly competitive due to its excellent performance.

[0126] Table 3

[0127]

[0128] Table 4 shows the comparison results of objective indicators of LSANet and other advanced SR methods. As can be seen from Table 4, LSANet has achieved the best results in all magnification factors; especially on Urban100, LSANet is far ahead of other advanced methods.

[0129] Table 4

[0130]

[0131]

[0132] 2) Visual quality comparison

[0133] Fig. 9 The visual perception quality comparison of LSANet and advanced SR methods, including VDSR, CARN, IMDN, ESRT, LBNet, and Omni-SR, is shown. Fig. 9 It can be seen that the proposed LSANet can restore more delicate details and reduce blur artifacts, resulting in sharper edges and more natural textures. In contrast, the SR images of other methods are blurred or even distorted. The above results once again verify the superiority of the proposed LSANet.

[0134] 3) Ablation experiment of MSLE module

[0135] In order to verify the effectiveness of the MSLE module, a relevant ablation experiment is used for comparison. In this experiment, RG, HPB and LFFM modules are used to replace the MSLE module in the proposed LSANet, where RG, HPB and LFFM are CNN-based feature extraction modules, and the receptive field is expanded by stacking 3×3 convolutions. In this experiment, RG, HPB and LFFM modules are used to replace the MSLE module in LSANet. Table 5 lists the performance comparisons of the above three CNN-based feature extraction modules and MSLE. As can be seen from Table 3, the proposed MSLE leads other modules by a large margin. Compared with LFFM, MSLE leads by as much as 0.49dB on Urban100(×4). Compared with RG, the lead of MSLE on Urban100(×4) is further expanded to 0.62dB. The above results fully demonstrate the superiority of the proposed MSLE and the potential of the self-attention module in extracting local features.

[0136] Table 5

[0137]

[0138] 4) Real image super-resolution comparison

[0139] In order to verify the effectiveness of the proposed LSANet in the real scene image SR task, the inventors used the RealSR dataset containing real-world images to train and test LSANet, and compared it with other advanced lightweight SR methods. The results are shown in Table 6. As can be seen from Table 6, compared with SRResNet, IMDN, and ESRT, the proposed LSANet achieves the best PSNR at all scaling factors. Compared with LP-KPN designed specifically for RealSR, although the PSNR of LSANet is slightly lower at the ×4 scaling factor, it leads by 0.11dB and 0.12dB at the ×2 and ×3 scaling factors, respectively. The above results prove that the proposed LSANet can restore images of real scenes.

[0140] Table 6

[0141]

[0142] The above are only embodiments of the present invention. The invention is not limited to the field involved in this implementation case. The common knowledge such as the known specific structure and characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A lightweight image super-resolution network based on local self-attention, characterized by: It includes a cascaded shallow feature extraction module SFE, several multi-scale local feature extraction modules MSLE, a global self-attention module GSA and an upsampling module, wherein the shallow feature extraction module SFE is used to extract shallow features from the input low-resolution image, the multi-scale local feature extraction module MSLE is used to extract local features according to gradually increasing feature scales, the global self-attention module GSA is used to extract global features, and the upsampling module is used to restore the spatial resolution of the feature image. The lightweight image super-resolution network based on local self-attention can be expressed as: Among them, I SR represents the reconstructed SR image, I LR represents the LR image to be reconstructed; f SFE (·) indicates the function corresponding to the SFE module; Represents the functions corresponding to n MSLE modules; f GSA (·) represents the function corresponding to the GSA module; ↑(·) represents the function corresponding to the upsampling module.

2. The lightweight image super-resolution network based on local self-attention according to claim 1, characterized in that: The multi-scale local feature extraction module MSLE includes a cascaded micro-scale feature extraction module FSFE, a plurality of medium-scale local self-attention modules MsLSA, a plurality of deformable local self-attention modules DeLSA, and a plurality of hollow local self-attention modules DiLSA, wherein: The micro-scale feature extraction module FSFE is used to extract important features around the center point at a micro-scale; The mesoscale local self-attention module MsLSA is used to efficiently model the dependencies between neighboring pixels in a sliding window at a mesoscale and extract high-quality local features; The deformable local self-attention module DeLSA is used to break through the limitation of sliding window on the receptive field. It finds feature points and models the dependencies between them according to the generated offset matrix to enhance the network's adaptability to unknown LR images. The dilated local self-attention module DiLSA is used to further expand the receptive field and extract local features at a larger scale.

3. The lightweight image super-resolution network based on local self-attention according to claim 2, characterized in that: The multi-scale local feature extraction module MSLE can be expressed as: in Represents the input features of the MSLE module; f FSFE (·) represents the function corresponding to the FSFE module; f MsLSA (·) represents the function corresponding to the MsLSA module; f DeLSA (·) represents the function corresponding to the DeLSA module; f DiLSA (·) represents the function corresponding to the DiLSA module.

4. The lightweight image super-resolution network based on local self-attention according to claim 3, characterized in that: The mesoscale local self-attention module MsLSA comprises a cascaded moving sliding window unit, a patch extraction unit, a multi-head self-attention unit and a feedforward neural network unit, wherein: Moving sliding window unit, used to move the sliding window to extract the set W of local feature blocks ms in, Represents the function corresponding to the operation of extracting local feature blocks; Represents the input features of the local feature block; α ms and β ms Represent the size and step size of the sliding window respectively; Represents the i-th local feature block; Representative in W ms The number of local feature blocks in ; Represents the operation of rounding a value upwards; Respectively represent the length and width of the input feature map; Patch extraction unit, used to extract To extract the set of patches that participate in dependency modeling, the process can be defined as: in, Represents the i-th local feature block in MsLSA A collection of patches in the Representative from Extract the function corresponding to the patch operation; Represents the i-th local feature block The jth patch of , (x0, y0) represents the patch in The absolute position on Representatives in The number of patches in δ ms Represents the size of the patch; Multi-head self-attention unit for modeling using self-attention mechanism The dependencies between patches in the model are calculated and the set of modeled patches is output. The process can be defined as: Among them, Attenion(·) represents the function corresponding to Multi-Head Self-Attention (MHSA); Norm(·) represents the function corresponding to layer normalization; FFN(·) represents the function corresponding to Feed-Forward Network (FFN); Γ(·) represents the operation of expanding a patch into a token; ∧(·) represents the operation of folding a token into a patch; Feedforward neural network unit, used to Perform the feature shape transformation operation opposite to the above process to obtain the output feature 5. The lightweight image super-resolution network based on local self-attention according to claim 4, characterized in that: The deformable local self-attention module DeLSA comprises a cascaded moving sliding window unit, an offset matrix extraction unit, a patch extraction unit, a multi-head self-attention unit and a feedforward neural network unit, wherein: Moving sliding window unit, used to move the sliding window to extract the set W of local feature blocks ms , which is the same as the moving sliding window unit of MsLSA; The offset matrix extraction unit OEB is used to dynamically adjust the position of the convolution kernel to adapt to local deformation in the image and can be expressed as: in, Represents the input features of OEB; CMP(·) represents the function corresponding to channel average (maximum) pooling; Reshape(·) represents the function corresponding to the vector shape reconstruction operation; Sigmoid(·) represents the Sigmoid activation function; Linear(·) represents the function corresponding to the linear mapping layer; Update(·) represents the function corresponding to the rounding and offset limiting operation of the offset matrix; Offset represents the output offset matrix; Patch extraction unit, used to extract To extract the set of patches that participate in dependency modeling, the process can be defined as: in, represents the set of patches in the ith local feature block in DeLSA; (Δx, Δy) represents the offset, Δx, Δy∈[-MAX,MAX], and ￠ represents an integer; Represents the patch after the offset is introduced; Represents the number of local feature blocks in DeLSA; Represents the number of patches in the local feature block; α de Represents the size of the local feature block in DeLSA; Multi-head self-attention unit for modeling using self-attention mechanism The dependencies between patches in the model are calculated and the set of modeled patches is output. Same as the multi-head self-attention unit of MsLSA; Feedforward neural network unit, used to Perform the feature shape transformation operation opposite to the above process to obtain the output feature Same as the feedforward neural network unit of MsLSA.

6. The lightweight image super-resolution network based on local self-attention according to claim 5, characterized in that: The hollow local self-attention module DiLSA includes a cascaded moving sliding window unit, a patch extraction unit, a multi-head self-attention unit and a feedforward neural network unit, wherein the patch extraction unit, the multi-head self-attention unit and the feedforward neural network unit are the same as MsLSA. The process of moving the sliding window and extracting the local feature block can be defined as: Among them, W di Represents a collection of extracted local feature blocks; Represents the input features of DiLSA; DiLSA uses a sliding window to The function corresponding to the operation of extracting local feature blocks; α di , β di and τ di Represent the size, step size and hole size of the sliding window with holes respectively; Representative in W di The i-th local feature block in; Represent the length and width of the input feature map respectively.

7. The lightweight image super-resolution network based on local self-attention according to claim 6, characterized in that: The number of MSLE modules is set to 5, and the number of channels of the network is set to 64; each MSLE module includes 1 MsLSA, 1 DeLSA and 1 DiLSA; the sliding window size of MsLSA is 5×5, and the step size is 3; the sliding window size of DeLSA is 7×7, and the step size is 7, and the minimum and maximum values ​​of the offset are set to -7 and 7 respectively; the sliding window size of DiLSA is 11×11, and the step size is 11; the hole of DiLSA is set to 1, and the receptive field of DiLSA is 22×22.

8. The lightweight image super-resolution network based on local self-attention according to claim 7, characterized in that: The lightweight image super-resolution network uses the training set As the LR-HR image pair, the mean absolute error (MAE) is used to optimize the parameters, and the error function is defined as follows: Where L(Θ) represents the error function; Θ represents the learnable parameter; N represents the number of images in the training set; ‖·‖ MAE Represents the function corresponding to MAE; the initial learning rate is 5×10 -4 And it decays by half every 200 epochs, patch_size is set to 64, and batch_size is set to 64.

Images