A lightweight super-resolution reconstruction method

Abstract

This invention discloses a lightweight super-resolution reconstruction method. The method involves inputting the image to be processed into a depth feature extraction network to obtain image depth features; and then, based on these depth features and the image to be processed, obtaining a super-resolution reconstructed image. The depth feature extraction network includes n depth feature transformation blocks, each comprising a progressive thinning module, a linear transformation module, and a multi-scale feature fusion module, used to extract multi-scale feature information from the image to be processed through U-shaped feature distillation. The super-resolution reconstruction method provided by this invention significantly reduces the number of model parameters while improving model performance by utilizing depthwise separable convolution and linear transformation modules.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and computer vision technology, and more specifically to a lightweight super-resolution reconstruction method. Background Technology

[0002] The fundamental task of image super-resolution reconstruction (SR) is to restore a low-resolution image to the desired high-resolution image.

[0003] With the rise of convolutional neural networks and the unprecedented improvement of graphics processing units, more and more deep learning-based SR models have been proposed. This approach can improve the mapping capability of feature extraction modules by combining multiple functional blocks, and can improve the reconstruction performance of the network by combining more feature extraction modules. However, the stacking of modules will bring high computational complexity and a large number of parameters, resulting in insufficient real-time performance and applicability of the network.

[0004] Meanwhile, since the fundamental purpose of image super-resolution reconstruction is to enlarge the image size, image upsampling is extremely important in the reconstruction process. Currently, commonly used upsampling methods include interpolation-based upsampling, transposed convolution, and subpixel convolution. Because subpixel convolution uses reconstruction information obtained entirely from the original image, it is beneficial for restoring image details and is currently the most commonly used method in SR tasks. However, this upsampling method still has an unresolved problem: the reconstruction result exhibits significant unevenness. Therefore, solving the current upsampling problem is also extremely important for improving the reconstruction effect of SR.

[0005] Therefore, in view of the above-mentioned defects in the prior art, how to provide a lightweight super-resolution reconstruction method to at least partially solve the above problems is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a lightweight super-resolution reconstruction method, which aims to improve model performance while reducing the number of model parameters by utilizing depthwise separable convolution and linear transformation modules.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A lightweight super-resolution reconstruction method includes,

[0009] The image to be processed is input into a depth feature extraction network to obtain image depth features. The depth feature extraction network includes n depth feature transformation blocks. Each depth feature transformation block includes a progressive thinning module, a linear transformation module, and a multi-scale feature fusion module, which are used to extract multi-scale feature information, i.e. image depth features, of the image to be processed through U-shaped feature distillation.

[0010] A super-resolution reconstructed image is obtained based on the image depth features and the image to be processed.

[0011] Preferably, before the image to be processed is input into the deep feature extraction network, it first enters the shallow feature extraction network to extract the shallow features of the image to be processed.

[0012] Preferably, the shallow feature extraction network includes a 3×3 convolutional layer and a 1×1 convolutional layer.

[0013] Preferably, the depth image feature extraction step includes:

[0014] After the image to be processed enters the progressive thinning module, it is separated into fine features and coarse features through channel separation. The fine features are then entered into the linear transformation module to obtain the transformed fine features. The coarse features are then subjected to convolutional downsampling to obtain multi-scale depth features.

[0015] The transformed fine features and the corresponding multi-scale depth features are then combined and fused together in the multi-scale feature fusion module to obtain the fused image depth features.

[0016] Preferably, the fusion process starts with the smallest depth feature, sequentially upsamples it, and then splices and fuses it with the corresponding transformed fine features. This process is passed up layer by layer until all features of each branch are spliced ​​together, and then channel shuffling is performed to output the fused image depth features.

[0017] Preferably, the number of linear transformation modules depends on the number of branches of fine features at different scales; that is, the smaller the image size in the branch, the more linear transformations are used, so as to make full use of the time difference between the progressive thinning module and the multi-scale feature fusion module; performing corresponding linear transformations on the fine features of different branches can effectively improve the feature mapping capability of the network.

[0018] Preferably, the image depth features are sequentially processed through multiple basic transformer blocks for attention calculation before super-resolution reconstruction. Channel-level attention calculation based on transformer blocks allows for the acquisition of more relevant feature maps through channel-level global information sharing, thereby reducing unsmoothness in sub-pixel reconstructed images, while also reducing the size of the attention map and lowering computational complexity.

[0019] Preferably, a nonlinear transformation layer is provided between every two basic transformer blocks to share parameters for attention calculation. This reduces the amount of computation while ensuring that the parameters are not completely consistent each time, thereby ensuring the expressive power of the model.

[0020] Preferably, the process of attention calculation in the basic transformer block includes:

[0021] After increasing the number of channels in the image depth features, channel separation is used to calculate the attention components Q, K, and V.

[0022] The transposed component K is then multiplied sequentially by the components Q and V to obtain the feature information calculated by attention.

[0023] Preferably, the process of obtaining a super-resolution reconstructed image based on the depth image features and the image to be processed includes:

[0024] The depth image features are amplified by a sub-pixel convolutional upsampling layer to obtain deep semantic features; the image to be processed is upsampled by bilinear interpolation to obtain the original structural information; the deep semantic features and the original structural information are added together to obtain the super-resolution reconstructed image of the image to be processed. This application uses a concatenation operation instead of an addition operation, which can reduce the amount of computation; at the same time, the application of pointwise convolution can fully mix feature information.

[0025] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a lightweight super-resolution reconstruction method, which improves the feature mapping capability of the network by combining different linear transformation modules, extracts multi-scale feature information by using U-shaped feature distillation, and significantly reduces the number of parameters of the super-resolution reconstruction network by utilizing depthwise separable convolution and linear transformation modules.

[0026] On the other hand, this invention utilizes lightweight recursive transformer blocks for attention computation at the channel level, which can obtain more relevant feature maps based on channel-level global information sharing, thereby ensuring the efficient use of extracted features. Furthermore, by capturing long-distance information between different image blocks, it compensates for the lack of smoothness between different image blocks in subpixel upsampling, reduces the size of the attention map, lowers computational complexity, and effectively ensures the reconstruction performance of the network. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0028] Figure 1 A schematic diagram of the lightweight super-resolution reconstruction network structure provided by the present invention;

[0029] Figure 2A schematic diagram of the deep feature conversion block structure provided by the present invention;

[0030] Figure 3 A schematic diagram of the recursive transformer module structure provided by the present invention;

[0031] Figure 4 The overall flowchart of the lightweight super-resolution reconstruction method provided by the present invention is shown. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] This invention provides a lightweight super-resolution reconstruction method, which mainly includes the following steps:

[0034] The image to be processed is input into the deep feature extraction network to obtain the image depth features. The deep feature extraction network includes n deep feature transformation blocks. Each deep feature transformation block includes a progressive thinning module, a linear transformation module, and a multi-scale feature fusion module, which are used to extract multi-scale feature information of the image to be processed, i.e., image depth features, through U-shaped feature distillation.

[0035] Based on the image depth features and the image to be processed, a super-resolution reconstructed image is obtained.

[0036] The following describes the steps for extracting image depth features:

[0037] First, after the image to be processed enters the progressive thinning module, fine features and coarse features are obtained through channel separation. The fine features enter the linear transformation module to obtain the transformed fine features; while the coarse features are subjected to convolutional downsampling to obtain multi-scale depth features.

[0038] Subsequently, the transformed fine features and the corresponding multi-scale depth features are combined and fused together in the multi-scale feature fusion module to obtain the fused image depth features.

[0039] It should be noted that during the fusion process, we start with the smallest depth features, then upsample them sequentially and stitch them together with the corresponding transformed fine features. This process is passed up layer by layer until all features of each branch are stitched together and then channel shuffling is performed. Finally, the fused image depth features are output.

[0040] The number of linear transformation modules depends on the number of branches of fine features at different scales. That is, the smaller the image size in the branch, the more linear transformations are used. By making full use of the time difference between the progressive thinning module and the multi-scale feature fusion module, and performing corresponding linear transformations on the fine features of different branches, the feature mapping capability of the network can be effectively improved.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] In one embodiment, such as Figure 1 Before entering the deep feature extraction network, the image to be processed first enters the shallow feature extraction network for shallow feature extraction. The shallow feature network consists of one 3×3 convolutional layer and one 1×1 convolutional layer. Specifically, shallow feature extraction of the image to be processed is performed using the following formula:

[0043] F0 = f PW (f DW (I LR ))

[0044] In the formula, I LR This represents the input image to be processed; f DW This indicates a convolution calculation that uses a 3×3 convolution to perform a non-linear mapping on the image to be processed; f PW This indicates a pointwise convolution calculation using 1×1 convolution to change the number of feature channels; F0 is the extracted shallow features of the image to be processed.

[0045] Then, the extracted shallow features are fed into a deep feature extraction network for further deep feature extraction. In this application, the deep feature extraction network includes n deep feature transformation blocks, and each deep feature transformation block includes a progressive thinning module, a linear transformation module, and a multi-scale feature fusion module. The three modules extract multi-scale feature information of the image to be processed through U-shaped feature distillation.

[0046] For ease of understanding, the extraction process can be represented by the following formula:

[0047]

[0048] F out =F n +F n-1 +F n-2 +…+F0

[0049] Among them, F n This represents the deep feature output extracted by the nth deep feature transformation block; F represents the processing function for the nth deep feature transformation block; outThis represents the output of a deep feature extraction network consisting of n deep feature transformation blocks.

[0050] Furthermore, the deep feature extraction network extracts multi-scale feature information of the image to be processed through U-shaped feature distillation, including:

[0051] The extracted shallow features are processed by a progressive refinement module, where they are separated into fine and coarse features. The fine features are then processed by a linear transformation module to obtain the transformed fine features. Fine features from different branches undergo corresponding linear transformations to improve the network’s feature mapping capability. Meanwhile, coarse features are downsampled using convolution operations to obtain multi-scale depth features.

[0052] Subsequently, the transformed fine features and the corresponding multi-scale depth features enter the multi-scale feature fusion module. Starting from the smallest depth feature, it is upsampled sequentially and spliced ​​with the corresponding transformed fine features. This process is passed up layer by layer until all features of each branch are spliced ​​and then channel shuffling is performed to output the fused depth image features. These features are then input to the next depth feature transformation block to continue depth feature extraction.

[0053] For details, please refer to [the relevant documentation]. Figure 2 The number of feature information channels in the input and output of the deep feature transformation block is uniformly set to 64, and the size of the input and output feature maps is C*H*W (number of channels*height*width). Taking the separation of three-layer feature channels as an example.

[0054] The size of the first fine feature obtained after the first channel separation is 32*H*W, and the size of the first coarse feature is 32*H*W. Further, the size of the first coarse feature after being downsampled by the Conv2d(3,3,3) convolutional layer is 32*H / 2*W / 2. This process not only performs scale transformation, but also extracts features at the corresponding scale. After that, a batch normalization layer is passed to make the network training process more stable.

[0055] Furthermore, the first layer of coarse features is separated by the second layer of channels to obtain the second layer of fine features and coarse features, both of which are 16*H / 2*W / 2 in size. Then, the coarse features of the second layer continue to pass through the Conv2d(3,3,3) convolutional layer, and the downsampled size is 16*H / 4*W / 4. Similarly, they are passed through the batch normalization layer to make the network training process more stable.

[0056] Next, the second layer of coarse features is separated by the third layer of channels to obtain the third layer of fine features and coarse features, both with a size of 8*H / 4*W / 4. The third layer of coarse features is then passed through a Conv2d(3,3,3) convolutional layer, and the downsampled feature map has a size of 8*H / 8*W / 8. It is then passed through a batch normalization layer for data normalization.

[0057] In the progressive refinement module of the i-th deep feature transformation block, channel separation and scaling of the input features are performed using the following formula:

[0058]

[0059]

[0060]

[0061]

[0062] In the formula, Indicates input features; f DW This indicates a depthwise convolution computation that downsamples the feature map; Split indicates a channel splitting operation. Represents the j-th fine feature; The j-th coarse feature to be further processed; f relu and f bn These represent the activation function and the batch normalization operation, respectively.

[0063] While extracting features at the corresponding scales, the fine features retained in the three branches are subjected to corresponding linear transformations to enrich the feature mapping capability of the module, resulting in fine features at three different scales, which facilitates the fusion of coarse and fine features across multiple branches and scales.

[0064] Specifically, in the i-th deep feature transformation block, the fine feature transformation can be performed according to the following formula:

[0065]

[0066]

[0067]

[0068]

[0069] in, This represents the j-th linear transformation block; This represents the fine features of the input to the j-th linear transform block; This represents the fine features of the output of the j-th linear transform block.

[0070] Finally, the multi-branch, multi-scale coarse and fine features are fused into the multi-scale feature fusion module. The specific process is as follows: calculate the local context information of the third-layer coarse features, including pointwise convolution of Conv2d(8,8,1) --> batch normalization --> ReLU activation.

[0071] Then, subpixel convolution is used to upsample the feature map, and the size of the obtained feature map is 8*H / 4*W / 4; it is then concatenated with the fine features of the third branch and the local context is calculated. The calculation process is: pointwise convolution of Conv2d(16,16,1) --> batch normalization --> ReLU activation. Thus, the information fusion of coarse and fine features at the third scale is achieved, and the size of the fused feature information is 16*H / 4*W / 4.

[0072] Furthermore, subpixel convolution is used to upsample the feature map, resulting in a feature map size of 16*H / 2*W / 2. This feature map is then concatenated with the fine features of the second branch and the local context is calculated. The calculation process includes pointwise convolution of Conv2d(32,32,1) --> batch normalization --> ReLU activation, thereby achieving information fusion at the second scale. The size of the fused feature information is 32*H / 2*W / 2.

[0073] Finally, subpixel convolution is used to upsample the feature map to obtain feature information of size 32*H*W. This information is then concatenated with the fine features of the first branch and subjected to channel shuffling. This yields deep feature information fused from features of three scales.

[0074] This application uses subpixel rearrangement to enlarge the size of feature maps in coarse feature branches, uses concatenation operations instead of addition operations to reduce computation, and then applies pointwise convolution to fully mix feature information while ensuring that the number of input and output feature maps for each branch is consistent.

[0075] The above fusion process can be represented by the following formula:

[0076]

[0077]

[0078]

[0079]

[0080] In the formula, This represents the j-th subpixel rearrangement operation; f represents the output after the j-th upsampling; PW _ represents pointwise convolution computation; Concat represents concatenation operation along the channel dimension; CS represents channel shuffling operation.

[0081] At this point, the image depth feature extraction is complete. The image can then proceed to the feature reconstruction network, where a super-resolution reconstructed image is obtained based on the depth image features and the image to be processed. The specific process includes:

[0082] The deep image features are amplified by a subpixel convolutional upsampling layer to obtain deep semantic features; the image to be processed is upsampled by bilinear interpolation to obtain the original structural information; the deep semantic features and the original structural information are added together to obtain the super-resolution reconstructed image of the image to be processed.

[0083] In one embodiment, such as Figure 1 As shown, the feature reconstruction part uses a subpixel upsampling layer to amplify the semantic feature map, and then passes it through a depthwise separable convolutional layer consisting of a 3×3 convolution and a 1×1 convolution to obtain deep semantic features; while the original image is upsampled by bilinear interpolation and then passes through a depthwise separable convolutional layer to obtain the original structural information; further, the original structural information and the deep semantic information are added together to obtain the reconstructed high-resolution image.

[0084] Specifically, its expression is as follows:

[0085] I SR =f PW (f DW (f SPC (F k )))+f PW (f DW (R bic (I LR )))

[0086] Among them, I LR Indicates the input low-resolution image; I SR Represents the reconstructed high-resolution image; f SPC Indicates subpixel upsampling operation; R bic This indicates a double interpolation upsampling operation.

[0087] On the other hand, this application considers that the non-smoothness problem in sub-pixel upsampling is partly due to insufficient information sharing between channel-level features, and therefore proposes to use a transformer to solve this problem. Since the introduction of the transformer, many representative transformer structures have been proposed in the field of computer vision due to its self-attention mechanism, which can obtain long-range feature information. Among them, SwinIR is the most representative work applying the transformer to the SR task, and its performance is significantly better than other state-of-the-art methods, but it suffers from high computational complexity. ESRT designed an efficient transformer that uses a grouping strategy to reduce computational complexity and GPU resource usage, and applied it to the SR task, but this method reduces the transformer's ability to acquire global information.

[0088] In response, this invention discloses a novel recursive transformer network to optimize existing subpixel upsampling methods.

[0089] Specifically, the depth image features output by the deep feature extraction network are first processed by a recursive transformer network to perform attention calculations on the obtained deep features before super-resolution reconstruction. This further transformation can make more effective use of the extracted features, thereby solving the problem of unevenness in sub-pixel upsampling results to some extent.

[0090] In one embodiment, the recursive transformer network includes multiple cascaded lightweight recursive transformer modules, which can improve network performance while maintaining lightweight design. Simultaneously, attention calculations are performed on the acquired deep features, and further transformations can more effectively utilize the extracted features, thus addressing the issue of unevenness in sub-pixel upsampling results to some extent. Furthermore, each recursive transformer module includes multiple lightweight basic transformer blocks, such as... Figure 3 As shown, a non-linear transformation layer is provided between every two basic transformer blocks to share parameters for attention calculation; that is, a non-linear transformation layer is introduced between every two recursive operations to ensure that the parameters of each feature mapping are not completely consistent with less computation, thereby ensuring the expressive power of the model.

[0091] The basic transformer block performs channel-level attention computation, which can obtain more relevant feature maps based on channel-level global information sharing, thereby reducing the unsmoothness in subpixel reconstructed images, reducing the size of attention maps, and lowering computational complexity.

[0092] Furthermore, in one embodiment, the specific process of each recursive transformer module is as follows: Figure 3 As shown, the number of input and output feature maps is also 64. The deep features entering the recursive transformer module first enter the first basic transformer block. After normalization, the number of channels is increased by Conv2d(64,64*3,1) convolutional layer. Channel separation is used to obtain the three components Q, K, and V for calculating global attention. The three components are then processed by Conv3d(64,64,3) convolutional layer to extract the local context of the feature map, and the shape of the three components is transformed into HW*C.

[0093] Since the non-smoothing problem of subpixel upsampling results is related to the feature differences at the channel level, using transposed attention at the channel level in multi-head attention computation can enable global information sharing between feature maps.

[0094] Specifically, the process of transposed attention calculation is as follows: Figure 3 As shown, the shape transformation of component K includes transposing it, and the transposed component K T ∈R C*HW Furthermore, K T With the shape-transformed component Q∈R HW*C Matrix multiplication is performed to obtain an attention map of size C*C, which effectively reduces the computational complexity of self-attention compared to an attention map of size HW*HW. Then, the components V∈R... HW*C Perform matrix multiplication with the attention map of size C*C to obtain feature information of size HW*C.

[0095] Furthermore, the feature information of size HW*C is transformed back to the original shape H*W*C. This transformation process is implemented by calling code: `from einops import rearrange`; the implementation process includes:

[0096] rearrange(out,'b head c(hw)->b(head c)h w',

[0097] head=self.num_heads,h=h,w=w),

[0098] The feature information is added to the feature information before attention calculation, which preserves the original information to some extent. Then, after layer regularization and fully connected layers, the feature transformation capability of this module is further enriched, and the output features of the basic transformer are obtained.

[0099] In one embodiment, before outputting the features of the basic transformer, the features before and after the layer regularization and the fully connected layer are added together again to prevent gradient vanishing, which helps in training deep networks, while fully preserving the original information during feature transformation.

[0100] The output of the previous basic transformer block continues to be processed in the same structure. At this time, the parameters of the module are the results of the nonlinear transformation layer calculation with the existing parameters, until the last basic transformer block is executed and the output of a recursive transformer module is obtained.

[0101] To facilitate understanding, the computation process of the lightweight recursive transformer network will be further explained below.

[0102] In the basic transformer block, self-attention based on the cross-covariance matrix is ​​calculated using the following formula:

[0103]

[0104] Attention(Q,K,V)=V·Softmax(K T ·Q / τ)

[0105] In the formula, Re represents the input of the k-th recursive transformer module; Re represents the shape transformation operation; Q, K, V ∈ R HW×C H, W, and C represent the length, width, and number of channels of the feature map, respectively; τ is a learnable scaling parameter used to control the dot product of K and Q.

[0106] Furthermore, the computation process of the basic transformer block and the recursive transformer module is as follows:

[0107]

[0108]

[0109]

[0110] in, It is the output of the s-th basic transformer block; Define the execution function of the s-th basic transformer block; NLL s FFN represents the s-th nonlinear transformation layer. s This represents the fully connected layer that performs non-linear feature mapping in the s-th basic transformer block; This represents the output of the k-th recursive transformer module.

[0111] Furthermore, multiple cascaded recursive transformer modules can be calculated using the following formula:

[0112]

[0113] In the formula, F k This represents the output feature of the k-th recursive transformer module; F represents the function executed by the k-th recursive transformer module; out This represents the deep feature information output by the deep feature extraction network.

[0114] The input feature maps are sequentially passed through multiple recursive transformer modules to obtain deep semantic information from the output of the recursive transformer network, so as to make full and effective use of the extracted features and finally reconstruct the details and textures in the high-resolution image.

[0115] In one embodiment, the overall flow of the lightweight super-resolution reconstruction method disclosed in this invention is as follows: Figure 4 As shown, that is:

[0116] S1. Obtain the image block to be processed.

[0117] S2. Use shallow convolution to extract shallow features from the image.

[0118] S3. Extract image depth features using a deep feature extraction network.

[0119] S4. The output of the deep feature extraction network enters the feature reconstruction part, and attention is calculated through a recursive transformer network to ensure efficient use of the extracted features.

[0120] S5. After feature reconstruction, a reconstructed high-resolution image is obtained.

[0121] In another embodiment, after constructing the super-resolution reconstruction network as described above, this application requires training before use. Furthermore, this invention uses Charbonnier loss as the loss function during training.

[0122] Furthermore, during training, the low-resolution image to be processed is divided into image blocks of fixed size, while the size of the reference high-resolution image is the low-resolution image size multiplied by the corresponding magnification scale; it should be noted that during testing, the image to be processed does not need to be resized.

[0123] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0124] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A lightweight super-resolution reconstruction method, characterized in that, Comprising, input the image to be processed into a deep feature extraction network to obtain image deep features; the deep feature extraction network comprises n deep feature conversion blocks, each of which comprises a progressive refinement module, a linear transformation module and a multi-scale feature fusion module, for extracting image deep features of the image to be processed through U-shaped feature distillation; the extraction step of the image deep features comprises: After the image to be processed enters the progressive refinement module, fine features and coarse features are obtained through channel separation, the fine features enter the linear transformation module to obtain transformed fine features; the coarse features are subjected to convolution down-sampling to obtain multi-scale deep features; The transformed fine features and the corresponding multi-scale deep features jointly enter the multi-scale feature fusion module for splicing and fusion, and the image deep features are obtained after fusion; The fusion starts from the smallest scale deep features, sequentially performs up-sampling and splicing fusion with the corresponding transformed fine features, and transmits layer by layer upwards until the features of each branch are spliced and the channel is mixed, and the fused image deep features are output; According to the image deep features and the image to be processed, a super-resolution reconstructed image is obtained.

2. The method of claim 1, wherein, Before the image to be processed is input into the deep feature extraction network, it is first input into a shallow feature extraction network to extract shallow features of the image to be processed, and then the shallow features are input into the deep feature extraction network.

3. The method of claim 2, wherein, The shallow feature extraction network comprises one layer of 3x3 convolution and one layer of 1x1 convolution.

4. The method of claim 1, wherein, The number of linear transformation modules depends on the number of branches of fine features of different scales.

5. The method of claim 1, wherein, The image deep features sequentially pass through a plurality of basic transformer blocks for attention calculation before super-resolution reconstruction.

6. The lightweight super-resolution reconstruction method of claim 5, wherein, A nonlinear conversion layer is provided between every two basic transformer blocks for sharing parameters for attention calculation.

7. The method of claim 5, wherein the method further comprises: The process of attention calculation by the basic transformer block comprises: After the image deep features increase the number of channels, the channel separation is used to calculate the attention components Q, K and V, After transposing the component K, it is multiplied with the components Q and V in sequence to obtain feature information after attention calculation.

8. The method of claim 1, wherein, According to the image deep features and the image to be processed, a super-resolution reconstructed image is obtained, comprising: The image deep features are enlarged through a sub-pixel convolution up-sampling layer to obtain deep semantic features; the image to be processed is up-sampled through bilinear interpolation to obtain original structure information, and the deep semantic features and the original structure information are added to obtain a super-resolution reconstructed image of the image to be processed.

Images