An arbitrary scale image super-resolution reconstruction method based on local features

Abstract

The application discloses a local feature-based arbitrary scale image super-resolution reconstruction method. Two efficient plug-and-play modules are designed, which can be combined with existing fixed scale super-resolution networks to realize the arbitrary scale super-resolution task. Among them, the scale-aware feature adaptive module (SLFAM) solves the problem of lack of adaptability to scale and local structure of image in the feature extraction process by generating a scale-aware feature adaptive dynamic convolution kernel; the local feature adaptive up-sampling module (LFAUM) fuses various information such as scale information, projection position coordinates, local features and receptive field offsets to generate a deformable up-sampling convolution kernel in the up-sampling reconstruction process, and realizes the function of arbitrary scale up-sampling reconstruction. The application is mainly applied to the technical field of image super-resolution reconstruction, and has a wide application prospect in the fields of remote sensing satellite images, medical images, security monitoring and the like.

Description

TECHNICAL FIELD

[0001] The application relates to a local feature-based arbitrary scale image super-resolution reconstruction method and belongs to the computer vision and intelligent information technology field. BACKGROUND

[0002] Image super-resolution reconstruction is an important research hotspot in the computer vision field, which aims to restore a low-resolution image into a high-resolution image. As a long-standing and challenging ill-posed problem in the computer vision field, it is often used in security monitoring, medical images and remote sensing satellite images, and can also be combined with computer vision downstream tasks such as image segmentation, detection and recognition and action tracking to obtain more extensive applications.

[0003] In recent years, single image super-resolution (SISR) has attracted widespread attention from researchers, and the development of deep learning has enabled SISR to make a series of important progress and promote many related technologies to be applied to actual life and production. In practical applications, the magnification multiple of an image is not fixed, but most SISR methods can only magnify the image to a fixed integer multiple, such as x2, x3, x4, etc., and the same model can only reconstruct one scale. If multiple scales need to be reconstructed, multiple models need to be trained, which obviously wastes computing resources and restricts the development of super-resolution technology. In order to solve this problem, an arbitrary scale image super-resolution reconstruction method under a single model is proposed, and representative algorithms include MetaSR, ArbSR, EQSR, etc., which to some extent alleviate this problem. However, the existing algorithms still have problems such as insufficient use of local features, inability to perform long and wide scale reconstruction, and fixed up-sampling receptive field range, which leads to unsatisfactory reconstruction results. In view of the above problems, the application takes single image super-resolution technology as the research object and proposes a local feature-based arbitrary scale image super-resolution reconstruction method, aiming to enhance the reconstruction performance of the network and improve the visual effect of the reconstructed image. SUMMARY

[0004] The application proposes a local feature-based arbitrary scale image super-resolution reconstruction method, which mainly consists of a scale-aware local feature adaptation module (SLFAM) and a local feature adaptation up-sampling module (LFAUM).

[0005] Specifically, the SLFAM can generate a dynamic scale-aware local feature adaptive convolution kernel, so that the network can extract image features of different scales and regions; and the LFAUM based on deformable convolution enables the reconstruction up-sampling receptive field to adaptively change its range and direction in the image reconstruction process, enhances the network's geometric transformation modeling capability, and generates a super-resolution image more adaptive to the texture of the image itself. Both modules can be combined with existing conventional super-resolution reconstruction networks in a plug-and-play manner, so that the network produces super-resolution results of any scale. A local feature-based arbitrary scale image super-resolution reconstruction method comprises the following steps:

[0006] (1) For the SLFAM, scale information and image features are connected as inputs to generate several scale-aware weights through dynamic convolution;

[0007] (2) The original scale-aware modulation convolution kernel is decomposed into a point-by-point convolution kernel and a channel-by-channel convolution kernel, and the scale-aware weights in (1) are used to weight and add several point-by-point convolution kernels and channel-by-channel convolution kernels to generate modulated point-by-point convolution kernels and channel-by-channel convolution kernels, which are then respectively convolved with the features to obtain scale-aware local adaptive features;

[0008] (3) For the LFAUM, the length-width scale information, the projection relative position and the image features are connected as inputs to generate a scale-aware local adaptive up-sampling convolution kernel after passing through an attention module;

[0009] (4) The offset and modulation coefficient of the up-sampling convolution kernel receptive field in (3) are learned according to the features and scale information, and the up-sampling receptive field is adaptively changed in range and direction according to the image content and local texture, and then convolved with the interpolated high-resolution features to generate the final super-resolution image;

[0010] (5) Both modules can be combined with existing fixed-scale super-resolution networks in a plug-and-play manner, wherein the SLFAM is inserted in series after each backbone module to extract scale-aware features, and the LFAUM replaces the original up-sampling module to realize the function of arbitrary scale up-sampling.

[0011] Compared with the prior art, the present application has the following advantages:

[0012] 1. The present application uses scale information and features to generate dynamic perception weights, so that the network can perceive different scales while being more adaptive to local structures, increasing the complexity of the network and having stronger fitting and adaptability than static convolution;

[0013] 2、The application decomposes the modulation convolution kernel into point-by-point convolution kernel and channel-by-channel convolution kernel, reduces the operation amount during modulation by using the separable convolution method, and improves the processing efficiency; in addition, different attention weights are given to the point-by-point convolution kernel and the channel-by-channel convolution kernel, so that the information at the channel level and the pixel level can be used respectively, and the representation ability of the network is enhanced;

[0014] 3、The application generates adaptive up-sampling convolution kernels by combining long and wide scale information, position information and local features, compared with the prior art which usually generates up-sampling convolution kernels by using scale and position information only, the application can adapt the up-sampling network to image content and local structure, generate different convolution kernels for regions with the same scale but large differences in content, such as smooth regions and edge regions, to perform reconstruction; in addition, the application can complete asymmetric reconstruction tasks of different long and wide scales, and has strong flexibility;

[0015] 4、The application can be easily combined with a fixed scale super-resolution network, is convenient and efficient, is beneficial to actual deployment, and has high operation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The figure is a whole network model block diagram of the application;

[0017] Figure 2 The figure is a structure diagram of the scale-aware local feature adaptive module (SLFAM) of the application;

[0018] Figure 3 The figure is a structure diagram of the local feature adaptive up-sampling module (LFAUM) of the application;

[0019] Figure 4 The figure is a visual effect comparison diagram of the application in a symmetric scale super-resolution reconstruction task;

[0020] Figure 5 The figure is a visual effect comparison diagram of the application in an asymmetric scale super-resolution reconstruction task. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the application clearer and more apparent, the application will be further described in detail below with reference to the drawings. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0022] As Figure 1As shown, a local feature-based arbitrary scale image super-resolution reconstruction method mainly includes two parts, a scale-aware local feature adaptation module (SLFAM) and a local feature adaptation upsampling module (LFAUM). Among them, the SLFAM is inserted into the backbone module as a feature extraction module, and the LFAUM is placed at the end of the network to realize the function of arbitrary scale super-resolution reconstruction.

[0023] 1. Overall network framework and processing process:

[0024] The entire network is divided into three parts: shallow feature extraction, deep feature extraction, and arbitrary scale upsampling reconstruction. Given a low-resolution input image I LR and a paired high-resolution image I HR , the purpose of the super-resolution reconstruction task is to generate a super-resolution image I HR with the same resolution as I SR . The low-resolution input image I LR is first extracted by shallow feature extraction to obtain shallow features F0, which is represented as:

[0025] F0=f0(I LR ) (1)

[0026] Where f0(·) represents a 3x3 convolution used to extract shallow features.

[0027] The deep feature extraction process takes the shallow features F0 and the reconstruction scale factors r h , r w in the vertical and horizontal directions as inputs. The deep feature extraction process includes multiple residual groups (RGs) to generate deep features F1. The SLFAM proposed in the present application is inserted into each RG as a feature extraction module. The process of obtaining deep features F1 can be represented as:

[0028] F1=f n (…(f2(f1(F0,r h ,r w )))) (2)

[0029] Where f i (·) (i from 1 to n) represents the i-th module composed of RG and SLFAM.

[0030] The arbitrary scale upsampling reconstruction process takes the deep features F1 and the reconstruction scale factors rh w As input, the final super-resolution image I SR obtained after processing by the LFAUM proposed in the application

[0031] I SR = f u (F1,r h ,r w ) (3)

[0032] Wherein f u (·) represents the processing procedure of the up-sampling module SLFAM.

[0033] The application adopts L1 loss as the loss function to optimize the model, and the L1 loss can be represented as:

[0034] L1(θ)=||I SR -I HR ||1 (4)

[0035] Wherein θ represents the parameters of the super-resolution network, I SR represents the super-resolution reconstructed image generated by the network, and I HR represents the corresponding high-resolution reference image.

[0036] 2. Scale-aware Local Feature Adaptation Module (SLFAM)

[0037] The structure of SLFAM is as follows: Figure 2 ​As shown, for different scale factors, the traditional super-resolution method usually needs to train a separate network to extract low-resolution features specific to each scale factor, that is, each scale factor has its own corresponding network weight. However, under the condition of image super-resolution at an arbitrary scale, especially for image super-resolution at a non-integer scale, it is obviously a waste of resources to train network parameters for each scale factor separately and store the model. Therefore, it is necessary to train a single model to solve the problem of super-resolution reconstruction at an arbitrary scale. Although the super-resolution tasks of different scale factors have similarities, there are also fundamental differences, especially when there is a significant difference in scale factors. Extracting features specific to a scale is the key to realizing single-model super-resolution at an arbitrary scale. If only the backbone network (such as RG) is used to extract features, the weights of the feature extraction network are fixed for different scale factors, resulting in the same features being extracted under different scale factors. Obviously, for different scale factors, the extracted features should be different. Because as the scale factor increases, more and more unknown pixels need to be reconstructed, more attention should be paid to high-frequency detail features when reconstructing the image, otherwise the reconstructed image may appear blurred. Therefore, the application introduces scale and local feature information in the feature extraction process, extracts scale-aware local structure adaptive features, and the specific process is as follows:

[0038] Given the low-resolution feature F LR , it is input into the SLFAM together with the scale factors r h , r w in the vertical and horizontal directions. The scale factor vector [r h , r w ] is first enlarged to the same size as F LR , and then connected in the channel dimension with the feature F LR . The connected feature needs to pass through an attention module. Specifically, it first passes through adaptive mean pooling, then passes through two fully connected layers, and finally is normalized by the softmax function to obtain K dynamic attention weights π k . The weights π k are processed by a dynamic weight generator to obtain scale-aware feature adaptive dynamic weights. Specifically, the dynamic weight generator includes K parallel convolution kernels, and after weighting with the learned dynamic attention weights π k , an aggregated convolution kernel is obtained, that is, the scale-aware feature adaptive dynamic weight, and the process can be represented as:

[0039]

[0040]

[0041] where W k and b kdenote the weights and bias of the K-th convolution kernel in the dynamic weight generator, π k is its corresponding dynamic attention weight. π k satisfy the following range:

[0042] 0≤π k (F,r h ,r w )≤1 (7)

[0043]

[0044] To further reduce the amount of parameters, the invention decomposes the aggregated single dynamic convolution kernel into a channel-wise convolution kernel dwand a point-wise convolution kernel pw. Channel-wise convolution only operates on a two-dimensional plane and does not change the number of channels. Since channel-wise convolution cannot utilize the feature information of different channels at the same spatial position, point-wise convolution is adopted to combine the feature information of different channels. The two operations are performed by different dynamic weight generators, which can be represented as:

[0045] dw=f d1 (Concat(F LR ,r h ,r w )) (9)

[0046] pw=f d2 (Concat(F LR ,r h ,r w )) (10)

[0047] where F LR denotes the input feature, r h , r w denote the scale factors in the vertical and horizontal directions, and f d1 (·) and f d2 (·) denote the processing process of the dynamic weight generator.

[0048] After obtaining the channel-wise convolution kernel dwand the point-wise convolution kernel pw, the input feature is respectively convolved with them, and the output feature F L ' R of SLFAM can be represented as:

[0049] F’ LR =F LR *dw*pw (11)

[0050] Considering the instability of the dynamic weight generator during training, the invention adopts an annealing strategy, that is, a larger temperature coefficient is added to the softmax function in the attention module at the beginning of training, so that the attention weight tends to be uniformly distributed. The specific process is defined by the following formula:

[0051]

[0052] where ζ k is the input feature of the softmax function. In the initial softmax function, τ = 1. As τ increases, the softmax output result is more sparse. As the number of iterations increases, the training gradually stabilizes, and the temperature coefficient continuously decreases, eventually reducing to 1. This ensures stable training while accelerating network convergence.

[0053] 3. Local Feature Adaptation Upsampling Module (LFAUM)

[0054] The LFAUM structure is shown in Figure 3 Existing arbitrary scale super-resolution methods only use scale factors and coordinate information to predict the weights of the up-sampling kernel, i.e., the up-sampling weights are position-dependent, but do not consider that the up-sampling weights should also be content-dependent. The LFAUM proposed in the present application combines scale factors and local features of low-resolution images to jointly predict the weights of the up-sampling kernel. The predicted up-sampling kernel determines the value of each reconstructed pixel according to its nearest low-resolution pixel. However, since the number of reconstructed pixels is much larger than the number of low-resolution pixels, it is inevitable that multiple reconstructed pixels will map to the same low-resolution pixel. In this case, it is necessary to introduce scale information and coordinate information to predict the weights of the up-sampling kernel to distinguish between reconstructed pixels that map to the same low-resolution pixel. In addition, in order to make the learned up-sampling kernel adapt to different regions of the image, the local feature information of the image also needs to be considered.

[0055] Low-resolution input feature High-resolution reconstructed feature generated by LFAUM where C in and C out represent the input and output channels, respectively. Finally, the high-resolution reconstructed feature F HR is mapped to the high-resolution image space through convolution to obtain the output image, the specific process being as follows:

[0056] The position matrix P is composed of four parts: the vertical direction scale factor r h , the horizontal direction scale factor r w , the horizontal direction relative position R(x), and the vertical direction relative position R(y). r h and r wThe purpose of taking the reciprocal is to map the data to [0, 1] consistent with the size of the input image data. The position matrix P contains the coordinate offset relationship when the reconstructed pixel is projected to the low-resolution space. For each pixel (x, y) in the high-resolution space, after directly projecting to the low-resolution space, the corresponding position coordinates L(x) and L(y) are calculated as follows:

[0057]

[0058]

[0059] The coordinates obtained by the above formula are in the continuous space of the low-resolution image, however, the actual image is composed of discrete points, therefore, the relative position coordinates R(x) and R(y) are introduced to replace the absolute position coordinates L(x) and L(y) for position representation, the process is as follows:

[0060]

[0061]

[0062] The relative position coordinates R(x) and R(y) are obtained by calculating the relative position offset of L(x) and L(y) to the nearest low-resolution image coordinates, and their values are limited to [0, 1].

[0063] In order to facilitate the convolution operation, the unfold expansion function is used to expand the low-resolution feature F LR according to the k x k convolution kernel, to obtain Then, according to the mapping relationship of the reconstructed pixel to the corresponding low-resolution pixel, the expanded feature F' LR is converted to F HR is connected with the position matrix P, and the following method is used to predict the up-sampling convolution kernel weight:

[0064] W = f (Concat (F HR , P)) (17)

[0065] Where f(·) represents the weight prediction module, specifically two 1 x 1 convolution and an activation function ReLU.

[0066] However, due to the difference between the up-sampling scale and the reconstruction position, the fixed shape of the up-sampling filter may cause a certain deformation of the object. Therefore, for visual tasks that require fine positioning, such as super-resolution reconstruction, it is necessary to adaptively determine the size and range of the receptive field. In order to achieve better up-sampling, the present application learns the additional offset of the receptive field from the low-resolution features, and then uses these offsets to adaptively change the spatial sampling position during the reconstruction process, thereby enhancing the attention to the relevant regions of the image. In addition, for the up-sampling convolution kernel of each pixel, the present application not only learns the offset of the receptive field, but also learns a feature modulation amplitude, which enables the up-sampling module to change the spatial distribution and relative influence of the features, further enhancing the geometric transformation modeling capability of the entire network, and the specific process is as follows:

[0067] R = [(-1, -1), (-1, 0),..., (1, 1)] (18)

[0068] The regular 3x3 square grid R defines the size and range of the receptive field, i.e. corresponding to a 3x3 convolution kernel. For each position p0 on the output feature F SR , there is:

[0069]

[0070] where p n has traversed all positions in the square grid R.

[0071] In the deformable up-sampling process of the present application, the regular square grid R is adaptively offset according to the learned offset δp n |n = 1, 2,..., N, and the feature distribution is adaptively corrected according to the learned modulation scalar Δm p . Where N is the number of points in the square grid R, δp n includes the horizontal and vertical offsets of each sampling point, and the modulation scalar Δm p ranges from [0, 1]. The feature value of the reconstructed feature F SR at position p0 can be corrected as:

[0072]

[0073] At this time, the receptive field sampling points are sampled at irregular offset positions (p0 + δp n + δp n ), rather than the previous regular square positions. Usually δp n is not an integer and cannot directly correspond to the feature value of the pixel. Therefore, the feature value at the sampling point is obtained by bilinear interpolation with the nearest four adjacent pixels.

[0074] The reconstructed image F SRThe feature value at the position (x, y) is obtained from the high-resolution feature F HR with the deformed dynamic up-sampling convolution kernel W dconv are jointly determined:

[0075] F SR (x, y) = F HR (x, y) * W donv (21)

[0076] By combining the vertical and horizontal scale information, the coordinate projection relative position, the image local feature, and the up-sampling receptive field offset in the deformed dynamic up-sampling convolution kernel, a clearer and more detailed image can be reconstructed.

[0077] In order to verify the effectiveness of the arbitrary scale image super-resolution reconstruction method based on local features described in the present application, the following will be compared in detail through experiments.

[0078] Experimental environment: operating system Ubuntu 20.04, deep learning framework Pytorch 1.1.0, Python version 3.7. The present application compares the PSNR and SSIM results (the larger the PSNR and SSIM, the better the network performance) of the model on the mainstream image super-resolution test sets Set5, Set14, BSD100 and Urban100. The present application selects five mainstream arbitrary scale image super-resolution reconstruction methods based on deep learning in recent years for comparison, specifically:

[0079] MetaSR: the method proposed by Hu et al., reference document "Hu X, Mu H, Zhang X, et al. Meta-SR: A magnification-arbitrary network for super-resolution [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 1575-1584."

[0080] ArbSR: the method proposed by Wang et al., reference document "Wang L, Wang Y, Lin Z, et al. Learning a single network for scale-arbitrary super-resolution [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 4801-4810."

[0081] LIIF: The method proposed by Chen et al., reference "Chen Y, Liu S, Wang X. Learning continuous image representation with local implicit image function [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 8628-8638."

[0082] LTE: The method proposed by Lee et al., reference "Lee J, Jin K H. Local texture estimator for implicit representation function [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 1929-1938."

[0083] CLIT: The method proposed by Chen et al., reference "Chen H W, Xu Y S, Hong M F, et al. Cascaded Local Implicit Transformer for Arbitrary-Scale Super-Resolution [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 18257-18267."

[0084] The test results are shown in Tables 1 and 2, where Table 1 is the PSNR and SSIM results on symmetric scales (the same vertical and horizontal scale factors), and Table 2 is the PSNR and SSIM results on asymmetric scales. The best results are marked in bold. Compared with other methods, it can be found that the method of the present application has better reconstruction performance. At the same time, Figure 4 and Figure 5 The objective visual effect comparison of the present application is also shown, Figure 4 for the visual effect comparison under symmetric scale, Figure 5 for the visual effect comparison under asymmetric scale, it can be found that the method of the present application can reconstruct better image edge details and other information, and has better texture.

[0085] Table 1 Comparison of results of various arbitrary scale super-resolution reconstruction methods under symmetric scale

[0086]

[0087] Table 2 Comparison of results of various arbitrary scale super-resolution reconstruction methods under asymmetric scale

[0088]

[0089] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art will understand that they can still modify the technical solutions described in the foregoing examples, or make equivalent substitutions for some or all of the technical features therein; and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A local feature based arbitrary scale image super-resolution reconstruction method, characterized in that The method comprises the following steps: (1) For the scale-aware local feature adaptive module SLFAM, the scale factor r h in the vertical direction, the scale factor r w in the horizontal direction, and the low-resolution feature F LR are connected as inputs, and after processing by the attention module, K dynamic attention weights π k are obtained; the dynamic attention weights π k are processed by the dynamic weight generator to obtain scale-aware feature adaptive dynamic weights; The dynamic weight generator includes K parallel convolution kernels, and the learned dynamic attention weight π k After weighting, an aggregated convolution kernel, i.e., a scale-aware feature adaptive dynamic weight, is obtained, and the process can be represented as: wherein F LR represents a low-resolution feature, W k and b k represent the weight and bias of the Kth convolution kernel in the dynamic weight generator, π k is its corresponding dynamic attention weight; π k satisfies the following range: 0≤π k (F LR ,r h ,r w )≤1 (3) (2) the aggregated single convolution kernel is decomposed into a channel-wise convolution kernel dw and a point-wise convolution kernel pw, which can be expressed as: dw = f d1 (Concat(F LR ,r h ,r w )) (5) pw = f d2 (Concat(F LR , r h , r w )) (6) where r h , r w denote the scale factors in the vertical and horizontal directions, f d1 (·) and f d2 (·) denote the processing procedure of the dynamic weight generator; after obtaining the per-channel convolution kernel dw and the per-point convolution kernel pw, the low-resolution feature is respectively convolved with them, and the output feature F' LR of the SLFAM can be represented as: F' LR = F LR *dw*pw (7) (3) For the local feature adaptive up-sampling module LFAUM, the scale factor r h in the vertical direction, the scale factor r w in the horizontal direction, the relative position coordinates R(x), R(y), and the high-resolution feature F HR connected as input, after the weight prediction module, the scale-aware local adaptive up-sampling convolution kernel weight W is generated; the process can be represented as: W = f (Concat (F HR , P)) (8) where f(·) denotes the weight prediction module, which is composed of two 1 x 1 convolutions and one activation function ReLU; P is the position matrix, which is composed of four parts: the scale factor r h in the vertical direction, the scale factor r w in the horizontal direction, the relative position in the vertical direction R(y), and the relative position in the horizontal direction R(x). The process of obtaining the relative position coordinates can be represented as: Wherein, L(x) and L(y) represent the absolute position coordinates, and the calculation method is: The relative position coordinates R(x) and R(y) are obtained by computing the relative position offset of L(x) and L(y) to the nearest low resolution image coordinates, whose values are limited in [0, 1]; the scale factor r h and the scale factor in the horizontal direction r w are taken as inverses, with the purpose of mapping the data to [0, 1], consistent with the input image data size; (4) high-resolution feature F after the unfolding and coordinate mapping operations HR and the scale factor r in the vertical direction h , the scale factor r in the horizontal direction w , the receptive field shift δp of the up-sampling convolution kernel weight W in the learning step (3) n and the modulation coefficient Δm p , wherein n = 1, 2, …, N; N is the number of points in the receptive field square grid R; and the up-sampling receptive field is adapted to change the range and direction according to the image content and local texture, to obtain a deformable up-sampling convolution kernel, which is then convolved with the interpolated high-resolution feature F HR to generate the final super-resolution image; (5) the scale-aware local feature adaptive module SLFAM and the local feature adaptive up-sampling module LFAUM can be combined with the existing fixed scale super-resolution network in a plug-and-play manner, wherein the scale-aware local feature adaptive module SLFAM is inserted in series after each backbone module to extract scale-aware features, and the local feature adaptive up-sampling module LFAUM replaces the original up-sampling module to realize the function of arbitrary scale up-sampling reconstruction.

2. The method of claim 1, wherein, The step (2) decomposes the aggregated single dynamic convolution kernel into a channel-wise convolution kernel dw and a point-wise convolution kernel pw, the channel-wise convolution only operates on a two-dimensional plane and does not change the number of channels; since the channel-wise convolution cannot utilize the feature information of different channels at the same spatial position, the point-wise convolution is adopted to combine the feature information of different channels; the two operations are performed by different dynamic weight generators.

3. The method of claim 1, wherein, The step (4) makes the up-sampling receptive field adaptively change the range and direction according to the image content and local texture, and obtains the original reconstruction feature F SR on each position p0 has: where p n All positions in the square grid R = [(-1,-1),(-1,0),...,(1,1)] are traversed; according to the learned receptive field shift δp n with modulation coefficient Δm p , the reconstructed feature F SR The feature value at position p0may be corrected as: At this time, the receptive field samples at irregular shifted positions (p0+ p n + δp n ) instead of the regular square positions; usually δp n is not an integer, and cannot directly correspond to the feature value of a pixel; therefore, the feature value at the sampling point is obtained by bilinear interpolation with the nearest four neighboring pixels; reconstructed image F SR the feature value at position (x, y) is determined from the high-resolution features F HR with the deformed dynamic up-sampling convolution kernel W dconv together with F SR (x,y) = F HR (x,y)*W dconv (15).

Images