Local feature-based super-resolution reconstruction method for images of any scale

Abstract

The invention discloses an arbitrary-scale image super-resolution reconstruction method based on local features. According to the invention, two efficient plug-and-play modules are designed and can be combined with an existing fixed-scale super-resolution network so as to realize a super-resolution task of any scale. Wherein the scale perception feature adaptive module (SLFAM) solves the problem of lack of adaptability to the scale and the local structure of an image in the feature extraction process by generating a scale perception feature adaptive dynamic convolution kernel; the local feature adaptive up-sampling module (LFAUM) fuses scale information, projection position coordinates, local features, receptive field offset and other information in the up-sampling reconstruction process to generate a deformable up-sampling convolution kernel, and the function of up-sampling reconstruction of any scale is achieved. The method is mainly applied to the technical field of image super-resolution reconstruction, and has wide application prospects in the fields of remote sensing satellite images, medical images, security monitoring and the like.

Description

Technical Field

[0001] The invention relates to a method for super-resolution reconstruction of an arbitrary-scale image based on local features, and belongs to the field of computer vision and intelligent information technology. Background Art

[0002] Image super-resolution reconstruction is an important research hotspot in the field of computer vision, which aims to restore low-resolution images to high-resolution images. As a long-standing challenging pathological problem in the field of computer vision, it is often used in security monitoring, medical images, remote sensing satellite images and other fields. It can also be combined with downstream computer vision tasks such as image segmentation, detection and recognition, and motion tracking to obtain wider applications.

[0003] In recent years, single image super-resolution (SISR) has received extensive attention from researchers. In particular, the development of deep learning has led to a series of important advances in SISR and promoted the application of many related technologies in real life production. In practical applications, the magnification of images is not fixed, but most SISR methods can only magnify images to fixed integer multiples, such as ×2, ×3, ×4, etc., and the same model can only reconstruct one scale. If reconstruction of multiple scales is required, multiple models need to be trained, which obviously leads to a waste of computing resources and restricts the development of super-resolution technology. In order to solve this problem, a super-resolution reconstruction method for arbitrary scale images under a single model has been proposed. Representative algorithms include MetaSR, ArbSR, EQSR, etc., which alleviate this problem to a certain extent. However, existing algorithms still have problems such as insufficient utilization of local features, inability to reconstruct different scales of length and width, and fixed upsampling receptive field range, resulting in unsatisfactory reconstruction effects. In view of the above problems, the present invention takes single image super-resolution technology as the research object and proposes a super-resolution reconstruction method for images of any scale based on local features, aiming to enhance the reconstruction performance of the network and improve the visual effect of the reconstructed image. Summary of the invention

[0004] The present invention proposes a super-resolution reconstruction method for images of arbitrary scales based on local features, which mainly consists of two parts: a scale-aware Local Feature Adaptation Module (SLFAM) and a local feature adaptive upsampling module (LFAUM).

[0005] Specifically, SLFAM can generate dynamic scale-aware local feature adaptive convolution kernels, enabling the network to extract image features of different scales and regions; while LFAUM based on deformable convolution enables the reconstruction upsampling receptive field to adaptively change its range and direction during the image reconstruction process, enhancing the network's geometric transformation modeling capabilities and generating super-resolution images that are more adapted to the image's own texture. Both modules can be combined with existing conventional super-resolution reconstruction networks in a plug-and-play manner to produce super-resolution results of any scale. A method for super-resolution reconstruction of images of any scale based on local features, comprising the following steps:

[0006] (1) For SLFAM, the scale information and image features are connected as input, and several scale-aware weights are generated through dynamic convolution;

[0007] (2) Decompose the original scale-aware modulated convolution kernel into point-by-point convolution kernel and channel-by-channel convolution kernel, perform weighted addition on several point-by-point convolution kernels and channel-by-channel convolution kernels by using the scale-aware weights in (1), generate modulated point-by-point convolution kernel and channel-by-channel convolution kernel, and then perform convolution operations with the features respectively to obtain scale-aware local adaptive features;

[0008] (3) For LFAUM, the length and width scale information, the relative position of the projection, and the image features are connected as input, and after passing through the attention module, a scale-aware local adaptive upsampling convolution kernel is generated;

[0009] (4) learning the offset and modulation coefficient of the receptive field of the up-sampled convolution kernel in (3) based on the feature and scale information, and making the up-sampled receptive field adaptively change its range and direction according to the image content and local texture, and then performing a convolution operation with the interpolated high-resolution features to generate the final super-resolution image;

[0010] (5) Both modules can be combined with the existing fixed-scale super-resolution network in a plug-and-play manner. SLFAM is inserted in series into each backbone module to perform scale-aware feature extraction, and LFAUM replaces the original upsampling module to achieve the function of upsampling at any scale.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] 1. The present invention uses scale information and features to jointly generate dynamic perception weights, so that the network can perceive different scales and adapt to local structures more, increasing the complexity of the network and having stronger fitting and adaptability than static convolution;

[0013] 2. The present invention decomposes the modulated convolution kernel into a point-by-point convolution kernel and a channel-by-channel convolution kernel, and uses a separable convolution method to reduce the amount of computation during modulation and improve processing efficiency; in addition, assigning different attention weights to the point-by-point convolution kernel and the channel-by-channel convolution kernel can respectively utilize information at the channel level and the pixel level, thereby enhancing the representation ability of the network;

[0014] 3. The present invention combines the length and width scale information, position information and local features to generate an adaptive upsampling convolution kernel. Compared with the prior art which usually only uses scale and position information to generate upsampling convolution kernels, the present invention can make the upsampling network adapt to the image content and local structure, and generate different convolution kernels for reconstruction of areas with the same scale but large content differences, such as smooth areas and edge areas; in addition, the present invention can complete asymmetric reconstruction tasks of different length and width scales, and has strong flexibility;

[0015] 4. The present invention can be easily combined with a fixed-scale super-resolution network, which is convenient and efficient, conducive to practical deployment, and has high computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall network model block diagram of the present invention;

[0017] Figure 2 Schematic diagram of the scale-aware local feature adaptation module (SLFAM) of the present invention;

[0018] Figure 3 4 is a structural diagram of a local feature adaptive upsampling module (LFAUM) of the present invention;

[0019] Figure 4 This is a comparison chart of the visual effects of the present invention on the symmetric scale super-resolution reconstruction task;

[0020] Figure 5 This is a comparison chart of the visual effects of the present invention on the asymmetric scale super-resolution reconstruction task. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] like Figure 1As shown in the figure, a super-resolution reconstruction method for arbitrary-scale images based on local features mainly includes two parts, the scale-aware local feature adaptation module (SLFAM) and the local feature adaptive upsampling module (LFAUM). Among them, SLFAM is inserted into the backbone module in series as a feature extraction module, and LFAUM is placed at the end of the network as an upsampling module to realize the function of arbitrary-scale super-resolution reconstruction.

[0023] 1. Overall network framework and processing process:

[0024] The entire network is divided into three parts: shallow feature extraction, deep feature extraction and arbitrary scale upsampling reconstruction. Given a low-resolution input image I LR and paired high-resolution images I HR , the purpose of the super-resolution reconstruction task is to generate HR Super-resolution image I with the same resolution SR . Low-resolution input image I LR First, the shallow feature F0 is obtained by shallow feature extraction, which is expressed as:

[0025] F0=f0(I LR ) (1)

[0026] Among them, f0(·) represents a 3×3 convolution, which is used to extract shallow features.

[0027] The deep feature extraction process combines the shallow feature F0 and the reconstruction scale factor r in the vertical and horizontal directions h 、r w As input. The deep feature extraction process includes multiple residual groups (RG) to generate deep features F1. The SLFAM proposed in the present invention is inserted in series after each RG as a feature extraction module. The process of obtaining the deep feature F1 can be expressed as:

[0028] F1=f n (…(f2(f1(F0,r h ,r w )))) (2)

[0029] Among them, f i (·)(i ranges from 1 to n) represents the i-th module composed of RG and SLFAM.

[0030] The upsampling reconstruction process at any scale combines the deep feature F1 and the reconstruction scale factor r in the vertical and horizontal directionsh 、r w As input, the final super-resolution image I is obtained after LFAUM processing proposed by the present invention. SR , the process can be expressed as:

[0031] I SR =f u (F1,r h ,r w ) (3)

[0032] Among them, f u (·) represents the processing of the upsampling module SLFAM.

[0033] The present invention uses L1 loss as the loss function to optimize the model. L1 loss can be expressed as:

[0034] L1(θ)=||I SR -I HR ||1 (4)

[0035] Among them, θ represents the parameters of the super-resolution network, I SR Represents the super-resolution reconstructed image generated by the network, I HR Represents the corresponding high-resolution reference image.

[0036] 2. Scale-aware Local Feature Adaptation Module (SLFAM)

[0037] The structure of SLFAM is as follows Figure 2As shown in Figure 1, for different scale factors, traditional super-resolution methods usually need to train separate networks to extract low-resolution features specific to each scale factor, that is, each scale factor has its own corresponding network weights. However, under the condition of image super-resolution at any scale, especially for image super-resolution at non-integer scales, it is obviously a waste of resources to train network parameters and store models for each scale factor separately. Therefore, it is necessary to train a single model to solve the problem of super-resolution reconstruction at any scale. Although the super-resolution tasks of different scale factors have similarities, there are also fundamental differences, especially when there are significant differences in scale factors. Extracting features of a specific scale is the key to achieving single-model super-resolution at any scale. If only a backbone network (such as RG) is used to extract features, the weights of the feature extraction network are fixed for different scale factors, resulting in the same extracted features at different scale factors. Obviously, for different scale factors, the extracted features should be different. Because as the scale factor increases, more and more unknown pixels need to be reconstructed. When reconstructing an image, more attention should be paid to high-frequency detail features, otherwise the reconstructed image may appear blurred. Therefore, the present invention introduces scale and local feature information in the feature extraction process to extract scale-aware local structure adaptive features. The specific process is as follows:

[0038] Given a low-resolution feature F LR , and the vertical and horizontal scale factors r h 、r w Together as the input of SLFAM. The vector composed of scale factors [r h ,r w ] First zoom in to the same size as F LR The same size, then in the channel dimension with the feature F LR The connected features need to go through an attention module. Specifically, they first go through adaptive mean pooling, then through two fully connected layers, and finally normalized by the softmax function to obtain K dynamic attention weights π k . Weight π k After being processed by the dynamic weight generator, the scale-aware feature adaptive dynamic weight is obtained. Specifically, the dynamic weight generator includes K parallel convolution kernels and the learned dynamic attention weight π k After weighting, we get an aggregated convolution kernel, which is the adaptive dynamic weight of scale-aware features. The process can be expressed as:

[0039]

[0040]

[0041] Among them, W k and b krepresents the weight and bias of the Kth convolution kernel in the dynamic weight generator, π k is its corresponding dynamic attention weight. π k Meet the following ranges:

[0042] 0≤π k (F,r h ,r w )≤1 (7)

[0043]

[0044] In order to further reduce the number of parameters, the present invention decomposes the aggregated single dynamic convolution kernel into a channel-by-channel convolution kernel dw and a point-by-point convolution kernel pw. Channel-by-channel convolution only operates on a two-dimensional plane and does not change the number of channels. Since channel-by-channel convolution cannot utilize the feature information of different channels at the same spatial position, point-by-point convolution is used to combine the feature information of different channels. These two operations are performed by different dynamic weight generators and can be expressed as:

[0045] dw=f d1 (Concat(F LR ,r h ,r w )) (9)

[0046] pw=f d2 (Concat(F LR ,r h ,r w )) (10)

[0047] Among them, F LR represents the input feature, r h 、r w Represents the scale factor in both vertical and horizontal directions, f d1 (·) and f d2 (·) represents the processing of the dynamic weight generator.

[0048] After obtaining the channel-by-channel convolution kernel dw and the point-by-point convolution kernel pw, the input features are convolved with them respectively, and the output feature F of SLFAM is L ' R It can be expressed as:

[0049] F' LR =F LR *dw*pw (11)

[0050] Considering the instability of training caused by the dynamic weight generator, the present invention adopts an annealing strategy. At the beginning of training, a large temperature coefficient is added to the softmax function in the attention module to make the attention weights tend to be evenly distributed. The specific process is defined by the following formula:

[0051]

[0052] Among them, k is the input feature of the softmax function. In the initial softmax function, τ=1. As τ increases, the softmax output becomes sparser. As the number of iterations increases, the training gradually stabilizes, the temperature coefficient decreases, and finally decreases to 1. This ensures the stability of the training while accelerating the convergence of the network.

[0053] 3. Local Feature Adaptation Upsampling Module (LFAUM)

[0054] LFAUM structure Figure 3 As shown. Existing arbitrary-scale super-resolution methods only use scale factors and coordinate information to predict the weights of upsampling kernels, that is, the upsampling weights are position-related, but do not consider that the upsampling weights should also be content-related. The LFAUM proposed in the present invention combines the scale factor and the local features of the low-resolution image to jointly predict the weights of the upsampling kernel. The predicted upsampling kernel determines the value of each reconstructed pixel based on its nearest low-resolution pixel. However, since the number of reconstructed pixels is much larger than the number of low-resolution pixels, it is inevitable that multiple reconstructed pixels will be mapped to the same low-resolution pixel. In this case, it is necessary to introduce scale information and coordinate information to predict the weights of the upsampling kernel to distinguish the reconstructed pixels mapped to the same low-resolution pixel. In addition, in order to make the learned upsampling kernel adapt to different areas of the image, the local feature information of the image also needs to be considered.

[0055] Low-resolution input features Generate high-resolution reconstruction features through LFAUM Among them C in and C out Represent the input and output channels respectively. Finally, the high-resolution reconstruction feature F HR It is mapped to the high-resolution image space through convolution to obtain the output image. The specific process is as follows:

[0056] The position matrix P consists of four parts: vertical scale factor r h , horizontal scale factor r w , horizontal relative position R(x) and vertical relative position R(y). h and r wThe purpose of taking the inverse is to map the data to [0,1], which is consistent with the size of the input image data. The position matrix P contains the coordinate offset relationship when the reconstructed pixel is projected into the low-resolution space. For each pixel (x, y) in the high-resolution space, after directly projecting it into the low-resolution space, the corresponding position coordinates L(x) and L(y) are calculated as follows:

[0057]

[0058]

[0059] The coordinates obtained by the above formula are in the continuous space of the low-resolution image. However, the actual image is composed of discrete points. Therefore, the relative position coordinates R(x) and R(y) are introduced to replace the absolute position coordinates L(x) and L(y) for position representation. The process is:

[0060]

[0061]

[0062] The relative position coordinates R(x) and R(y) are obtained by calculating the relative position offsets of L(x) and L(y) to the nearest low-resolution image coordinates, and their values ​​are restricted to [0,1].

[0063] In order to facilitate the convolution operation, the unfold function is used to expand the low-resolution feature F according to the k×k convolution kernel. LR ,get Then, according to the mapping relationship between the reconstructed pixels and the corresponding low-resolution pixels, the expanded feature F' LR Convert to Then F HR Concatenate with the position matrix P and predict the upsampling convolution kernel weights using:

[0064] W = f(Concat(F HR ,P)) (17)

[0065] Among them, f(·) represents the weight prediction module, which is specifically two 1×1 convolutions and an activation function ReLU.

[0066] However, due to the differences in upsampling scale and reconstruction position, fixed-shape upsampling filters may cause certain deformation of objects. Therefore, for visual tasks that require fine positioning, such as super-resolution reconstruction, it is necessary to adaptively determine the size and range of the receptive field. In order to achieve better upsampling, the present invention learns additional offsets of the receptive field from low-resolution features, and then uses these offsets to adaptively change the spatial sampling position during the reconstruction process, thereby enhancing the focus on relevant areas of the image. In addition, for the upsampling convolution kernel of each pixel, the present invention not only learns the offset of the receptive field, but also learns a feature modulation amplitude, which enables the upsampling module to change the spatial distribution and relative influence of the features, further enhancing the geometric transformation modeling capability of the entire network. The specific process is as follows:

[0067] R=[(-1,-1),(-1,0),...,(1,1)] (18)

[0068] The regular 3×3 square grid R defines the size and range of the receptive field, which corresponds to a 3×3 convolution kernel. For the output feature F SR Every position p0 on has:

[0069]

[0070] Among them, p n All positions in the square grid R are traversed.

[0071] In the deformable upsampling process of the present invention, the regular square grid R is based on the learned offset δp n |n=1,2,...,N for adaptive offset, the feature distribution is based on the learned modulation scalar Δm p Perform adaptive correction. Where N is the number of points in the square grid R, δp n Includes the horizontal and vertical offsets of each sampling point, the modulation scalar Δm p The range of is [0,1]. Reconstructed feature F SR The eigenvalue at position p0 can be corrected to:

[0072]

[0073] At this time, the receptive field sampling points are at irregular offset positions (p0+p n +δp n ) for sampling, rather than the regular square positions as before. Usually δp n is not an integer and cannot directly correspond to the eigenvalue of the pixel. Therefore, the eigenvalue at the sampling point is obtained by bilinear interpolation with the nearest four adjacent pixels.

[0074] Reconstructed image F SRThe feature value at position (x, y) is determined by the high-resolution feature F HR And the deformed dynamic upsampling convolution kernel W dconv Jointly determine:

[0075] F SR (x,y)=F HR (x,y)*W donv (twenty one)

[0076] By combining vertical and horizontal scale information, relative position of coordinate projection, local image features, and upsampling receptive field offset in the deformed dynamic upsampling convolution kernel, a clearer and more detailed image can be reconstructed.

[0077] In order to verify the effectiveness of the local feature-based arbitrary-scale image super-resolution reconstruction method described in the present invention, a detailed comparison will be conducted through experiments below.

[0078] Experimental environment: operating system Ubuntu20.04, deep learning framework Pytorch 1.1.0, Python version 3.7. This paper compares the PSNR and SSIM results of the model on the mainstream image super-resolution test sets Set5, Set14, BSD100 and Urban100 (the larger the PSNR and SSIM, the better the network performance). This paper selects five mainstream deep learning-based arbitrary scale image super-resolution reconstruction methods for comparison, specifically:

[0079] MetaSR: The method proposed by Hu et al., reference "Hu X, Mu H, Zhang X, et al. Meta-SR: Amagnification-arbitrary network for super-resolution [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 1575-1584."

[0080] ArbSR: Wang et al., reference “Wang L, Wang Y, Lin Z, et al. Learning a single network for scale-arbitrary super-resolution [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 4801-4810.”

[0081] LIIF: Chen et al., reference “Chen Y, Liu S, Wang X. Learning continuous image representation with local implicit image function [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 8628-8638.”

[0082] LTE: The method proposed by Lee et al., reference "Lee J, Jin K H. Local texture estimator for implicit representation function [C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2022: 1929-1938."

[0083] CLIT: Chen et al., Chen HW, Xu Y S, Hong MF, et al. Cascaded Local Implicit Transformer for Arbitrary-Scale Super-Resolution[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 18257-18267.

[0084] The test results are shown in Tables 1 and 2, where Table 1 shows the PSNR and SSIM results on a symmetric scale (same vertical and horizontal scale factors), and Table 2 shows the PSNR and SSIM results on an asymmetric scale. The best results are marked in bold. Compared with other methods, it can be found that the method of the present invention has better reconstruction performance. At the same time, Figure 4 and Figure 5 It also shows the objective visual effect comparison of the present invention. Figure 4 For the visual effect comparison under symmetrical scale, Figure 5 By comparing the visual effects under asymmetric scales, it can be found that the method of the present invention can reconstruct better information such as image edge details and has better texture.

[0085] Table 1 Comparison of results of various arbitrary-scale super-resolution reconstruction methods at symmetric scales

[0086]

[0087] Table 2 Comparison of results of various arbitrary-scale super-resolution reconstruction methods at asymmetric scales

[0088]

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for super-resolution reconstruction of images of any scale based on local features, characterized in that The following steps are involved: (1) For the scale-aware local feature adaptation module (SLFAM), the scale information r in the vertical and horizontal directions is h 、r w Connected with the image features as input, several scale-aware feature adaptive weights π are obtained through a dynamic weight generator k ; (2) Decompose the scale-aware dynamic convolution kernel into channel-by-channel convolution kernel dw and point-by-point convolution kernel pw, and use the scale-aware feature adaptive weight π in step (1) k Perform weighted addition on several dw and pw to generate modulated dw and pw, and then perform convolution operation with the input features to obtain scale-aware local adaptive features; (3) For the local feature adaptive upsampling module (LFAUM), the scale information r in the vertical and horizontal directions is h and r w , the relative position coordinates R(x) and R(y) are connected to the image features as input, and after passing through the weight prediction module, a scale-aware local adaptive upsampling convolution kernel is generated; (4) According to the expanded features and scale information r h and r w The receptive field offset δp of the upsampling convolution kernel W in step (3) is learned n and modulation coefficient Δm p The up-sampled receptive field is adaptively changed in range and direction according to the image content and local texture to obtain a deformable up-sampled convolution kernel, which is then convolved with the interpolated high-resolution features to generate the final super-resolution image. (5) Both modules, SLFAM and LFAUM, can be combined with the existing fixed-scale super-resolution network in a plug-and-play manner. SLFAM is inserted in series into each backbone module to perform scale-aware feature extraction, and LFAUM replaces the original upsampling module to achieve the function of upsampling and reconstruction at any scale.

2. The method according to claim 1, characterized in that The step (1) converts the low-resolution feature F LR and the scale information in the vertical and horizontal directions r h 、r w Together as the input of the dynamic weight generator, after being processed by the attention module, several scale-aware feature adaptive weights π are generated. k ; In the dynamic weight generator, the adaptive weight π k The weighted addition of several parallel convolution kernels is aggregated into a scale-aware feature adaptive convolution kernel. The process can be expressed as: Among them, W k and b k represents the weight and bias of the Kth convolution kernel in the dynamic weight generator, π k is its corresponding dynamic attention weight; π k Meet the following ranges: 0≤π k (F,r h ,r w )≤1 (3) 3. The method according to claim 1, characterized in that: The step (2) decomposes the aggregated single dynamic convolution kernel into a channel-by-channel convolution kernel dw and a point-by-point convolution kernel pw. The channel-by-channel convolution only operates on a two-dimensional plane and does not change the number of channels. Since the channel-by-channel convolution cannot utilize the feature information of different channels at the same spatial position, the point-by-point convolution is used to combine the feature information of different channels. These two operations are performed by different dynamic weight generators and can be expressed as: dw=f d1 (Concat(F LR ,r h ,r w )) (5) pw=f d2 (Concat(F LR ,r h ,r w )) (6) Among them, F LR represents the input feature, r h 、r w Represents the scale factor in both vertical and horizontal directions, f d1 (·) and f d2 (·) represents the processing of the dynamic weight generator; After obtaining the channel-by-channel convolution kernel dw and the point-by-point convolution kernel pw, the input features are convolved with them respectively, and the output feature F′ of SLFAM is LR It can be expressed as: F′ LR =F LR *dw*pw (7)。 4. The method according to claim 1, characterized in that In step (3), the vertical and horizontal scale factors, relative position coordinates and the expanded image features are connected as inputs, and a scale-aware local adaptive upsampling convolution kernel W is generated after passing through the weight prediction module. The process can be expressed as: W=f(Concat(F HR ,P)) (8) Among them, f(·) represents the weight prediction module, which is specifically two 1×1 convolutions and an activation function ReLU; P is the position matrix, which consists of four parts: the vertical scale r h , horizontal scale r w , the vertical relative position R(y) and the horizontal relative position R(x), the process of obtaining the relative position coordinates can be expressed as: Among them, L(x) and L(y) represent the absolute position coordinates, which are calculated as follows: The relative position coordinates R(x) and R(y) are obtained by calculating the relative position offsets of L(x) and L(y) to the nearest low-resolution image coordinates, and their values ​​are restricted to [0,1]. h and r w The purpose of taking the inverse is to map the data to [0,1], which is consistent with the size of the input image data.

5. The method according to claim 1, characterized in that The step (4) is based on the feature and the scale information r in the vertical and horizontal directions. h and r w Learning the receptive field offset δp of the upsampling convolution kernel W n |n=1,2,...,N and modulation coefficient Δm p , and the up-sampled receptive field adaptively changes its range and direction according to the image content and local texture. The original output feature F SR Every position p0 on has: Among them, p n Traversing all positions in the square grid R = [(-1,-1), (-1,0), ..., (1,1)]; according to the learned receptive field offset δp n and modulation coefficient Δm p , reconstruct feature F SR The eigenvalue at position p0 can be corrected to: At this time, the receptive field is at an irregular offset position (p0+p n +δp n ) for sampling, rather than the previous regular square positions; usually δp n It is not an integer and cannot directly correspond to the eigenvalue of the pixel; therefore, the eigenvalue at the sampling point is obtained by bilinear interpolation with the nearest four adjacent pixels; Reconstructed image F SR The feature value at position (x, y) is determined by the high-resolution feature F HR And the deformed dynamic upsampling convolution kernel W dconv Jointly determine: F SR (x,y)=F HR (x,y)*W donv (15)。

Images