Lightweight remote sensing image super-resolution reconstruction method based on dynamic snakelike convolution

Abstract

The invention provides a lightweight remote sensing image super-resolution reconstruction method based on dynamic snakelike convolution, relates to the technical field of image reconstruction, and provides a bar-shaped feature super-pixel interaction network SFSPIN, which aggregates bar-shaped tubular super-pixel features by stacking bar-shaped feature super-pixel interaction modules SFSPI, and performs super-resolution reconstruction on the bar-shaped tubular super-pixel features by stacking the bar-shaped feature super-pixel interaction modules SFSPI. And deep feature extraction from local and global perspectives is realized. In addition, different from most direct up-sampling reconstruction methods, the method adopts a CBAMUpConv module, deep features are further integrated from the angles of space and channels, and thus the reconstruction performance is improved. A comprehensive experiment result shows that the bar-shaped characteristic superpixel interaction network model SFSPIN can effectively extract the bar-shaped tubular characteristics in the remote sensing image, and shows excellent performance in the aspects of quantitative indexes and visual quality.

Description

Technical Field

[0001] The present invention relates to the technical field of image reconstruction, and in particular to a lightweight remote sensing image super-resolution reconstruction method based on dynamic snake convolution. Background Art

[0002] Remote sensing images are an important tool for recording and transmitting earth information, and have become an indispensable part of modern practical applications. By providing a variety of observation data, it has a wide range of applications in environmental monitoring and protection, urban planning and management, agricultural management, water resources management, disaster monitoring and rescue, etc. However, in the process of acquiring remote sensing images, factors such as atmospheric turbulence, lighting conditions, noise, motion blur, and sensor characteristics inevitably affect the image quality. These objective conditions and hardware performance limitations have led to the need to improve the resolution and image quality of the obtained remote sensing images. Upgrading physical imaging equipment not only costs extra money, but also prolongs the deployment cycle of remote sensing imaging systems. Improving the resolution of remote sensing images can not only improve the efficiency of data utilization, but also provide more accurate information support for scientific research and practical applications in various fields. Therefore, researching and developing remote sensing image super-resolution reconstruction technology has important theoretical and practical significance.

[0003] The single-image remote sensing image super-resolution reconstruction technology aims to reconstruct a high-resolution remote sensing image from the corresponding degraded low-resolution remote sensing image, thereby improving the clarity and detail of the image. To this end, researchers have proposed a variety of representative super-resolution reconstruction methods for remote sensing images, which can be roughly divided into three categories: interpolation-based methods, reconstruction-based methods, and learning-based methods. Interpolation-based methods are widely used in the early stage of single-image super-resolution reconstruction. Bilinear interpolation, bicubic interpolation, and nearest interpolation methods are usually used to estimate unknown pixel values. Although these methods are simple, direct, and widely used, it is difficult to effectively restore the details in remote sensing images. Reconstruction-based methods use prior knowledge to restore high-frequency details of images. Although they can achieve good reconstruction results, they are greatly limited in practical applications due to the complex calculation and optimization process. Learning-based algorithms learn end-to-end mappings between high-resolution and low-resolution image pairs from a large amount of training data, thereby improving the resolution of remote sensing images. With the development of computer hardware and machine learning technology, algorithms based on deep learning have become a research hotspot for single remote sensing image super-resolution.

[0004] Since Dong et al. proposed the pioneering work SRCNN, deep learning has made remarkable progress in the field of image super-resolution reconstruction. In particular, super-resolution reconstruction algorithms based on convolutional neural networks have emerged continuously and achieved impressive reconstruction results. Many literatures use the idea of ​​residual connection to build deeper and more complex networks to improve model performance. However, these methods require high computing resources and costs, and resource-constrained scenarios limit their application effects. In order to solve the problems of large model parameters, complex calculations and slow training speed, Shi et al., Dong et al. proposed efficient reconstruction models. In addition, the attention mechanism has been widely demonstrated to have significant effects in high-level visual tasks and low-level visual tasks including super-resolution (SR). The attention mechanism enables the network to selectively focus on the relevant areas of the input features in space and channels, thereby achieving better reconstruction effects.

[0005] Although deep learning-based super-resolution (SR) algorithms have performed well in reconstructing natural images, remote sensing images pose unique challenges due to their different scales, shapes, and complex textures. Therefore, these algorithms cannot be directly applied to remote sensing image reconstruction tasks. Remote sensing image super-resolution (RSISR) is based on natural image super-resolution and makes specific improvements to the characteristics of remote sensing images (RSIs). Figure 1 As shown in the figure, there are a large number of strip-shaped tubular structures in remote sensing images, such as roads, rivers, parking space markings, and zebra crossings. Some methods try to change the way of convolution to extract features, but these convolution operations still cannot extract narrow and irregular structural features well. In addition, although Transformer-based methods can learn long-range features, many methods divide images into fixed image blocks, which not only increases the computational complexity and memory consumption, but also cuts off the original continuous structure in the image, produces boundary artifacts, and thus affects the details of reconstruction. To solve these problems, we propose a hybrid network of CNN and Transformer with strip feature superpixel interaction. The strip feature superpixel interaction module (SFSPI) in this network uses dynamic snake convolution and superpixel aggregation to effectively extract strip tubular features while reducing the amount of calculation. In addition, some super-resolution reconstruction models use standard convolution and pixel rearrangement in the final reconstruction part to improve the resolution, but using only standard convolution cannot fully utilize the spatial and channel information of deep features.

[0006] Dong et al. proposed the SRCNN model, which applied deep learning to the field of image super-resolution for the first time, and realized end-to-end mapping between low-resolution images and high-resolution images. Compared with traditional reconstruction methods, SRCNN has improved in both speed and quality. However, the pre-upsampling module in SRCNN makes the model calculation complex and converges slowly, and the simple three-layer structure cannot fully utilize the context information of the image. In order to further improve the reconstruction quality, some models in recent years have adopted more complex and effective structures. With the residual network proposed by He et al. to solve the gradient vanishing and degradation problems in deep networks, Kim et al. proposed the VDSR model using the idea of ​​residual learning, which alleviated the gradient vanishing problem and reconstructed images with richer details and clearer images; Lim et al. removed the BN layer in the SRResNet structure and constructed an enhanced deep super-resolution network model EDSR, which not only improved the image quality, but also saved about 40% of memory in training, thereby realizing a deeper network structure under the same computing resource conditions. In addition, the introduction of the attention mechanism enables the reconstruction model to focus on important information, thereby reconstructing high-quality images. Zhang et al. first applied the attention mechanism to the field of super-resolution and proposed the residual channel attention network RCAN, which significantly improved the network representation ability by learning more useful channel features.

[0007] As an important research direction for low-level visual tasks, the quality of super-resolution reconstruction of remote sensing images has a crucial impact on downstream high-level visual tasks such as remote sensing image target detection, instance segmentation, and classification. Therefore, improving the reconstruction effect of remote sensing images has attracted the attention of more and more researchers. Liebel et al. first proposed the use of SRCNN for super-resolution reconstruction of single remote sensing images, which achieved better results than traditional interpolation methods. However, this method does not consider the scale diversity of remote sensing images. To address the problem of large scale spans in remote sensing images, Lei et al. proposed a local-global combined network LGCNet based on VDSR, which cascades features of different levels through a cross structure to achieve higher quality reconstruction effects. Xu et al. proposed a deep memory network DMCN with a symmetric hourglass structure with multiple residual connections to address the problem of multi-scale feature extraction. This network achieves good results while greatly reducing memory usage. In addition, Ma et al. proposed a CNN super-resolution reconstruction network structure WTCRR that combines wavelet transform with local and global residual connections. This method reduces the artifacts of the reconstructed image and enhances edge details through two-dimensional discrete wavelet transform and its inverse transform.

[0008] In recent years, thanks to the excellent performance of the Transformer self-attention mechanism in long-distance learning and inspired by the success of VIT in high-level visual tasks, Transformer-based image super-resolution reconstruction methods have been widely studied. Chen et al. proposed a pre-trained network model IPT, which can effectively handle a variety of low-level visual tasks such as super-resolution, rain removal, and denoising, and achieved significant performance improvement. Based on the structure of Swin Transformer, Liang et al. proposed a hybrid network model SwinIR for image restoration, which showed good performance in low-level computer vision tasks such as image super-resolution reconstruction, image denoising, and JPEG compression artifact reduction. Fang et al. proposed a hybrid network HNCT combining CNN and Transformer for lightweight image super-resolution, and achieved state-of-the-art levels in reconstruction effect and model complexity. In addition, Li et al. proposed an effective lightweight dynamic local and global self-attention network DLGSANet, which significantly reduced the number of parameters and computational cost, while achieving accuracy comparable to advanced methods.

[0009] Recently, more and more researchers have begun to explore Transformer-based super-resolution reconstruction methods for remote sensing images. Lei et al. proposed a Transformer-based multi-level enhancement network TransENet, which integrates high- and low-dimensional features and effectively improves the high-level feature representation ability after upsampling. He et al. proposed a dense spectral network DsTer that combines Transformer and ResNet, which meets the needs of 3D data processing and learning long-distance relationships for remote sensing images. At the same time, dense connections help the model to mine features from multi-level Transformers. Tu et al. proposed a generative adversarial network SWCGAN, which combines the advantages of CNN and Swin Transformer, and considers the characteristics of remote sensing images with rich information, large size, and strong correlation between pixels to achieve more effective super-resolution reconstruction. Tang et al. proposed a hybrid hierarchical Transformer network HSTNet. The network uses the internal recursive information of single scale and cross-scale in the image to effectively calculate the correlation between high-dimensional and low-dimensional features, thereby reconstructing images with rich contours and edges.

[0010] Although many studies have improved the model reconstruction effect from different angles, the depth and computational complexity of the model have also increased dramatically. To solve this problem, researchers have proposed a variety of lightweight reconstruction models. RFDN reduces the computational complexity through the residual feature distillation mechanism. This method removes redundant information in the feature extraction process and retains only the most important features, thereby achieving efficient image super-resolution reconstruction; CARN adopts a cascaded residual network structure, introduces cascade modules and efficient convolution operations, and achieves fast, accurate and lightweight image super-resolution reconstruction. ESRT achieves better performance at a lower computational cost by enhancing the feature expression ability and long-term dependency of similar blocks in the image. ELAN uses shifted convolution and grouped multi-scale self-attention to effectively extract local and global features, and further improves the speed through the attention mechanism.

[0011] In the field of remote sensing images, considering the requirements of some application scenarios for the deployment flexibility of reconstruction models, as well as the limited resources of embedded devices and mobile platforms, researchers have explored lightweight remote sensing image super-resolution reconstruction models in recent years. Wang et al. proposed an attention-based multi-layer feature fusion network AMFFN, which efficiently extracts key features through partial shallow residual blocks (PSRBs) and dynamic feature distillation modules (DFDBs) to ensure the robustness of features, and its effect is better than other classical methods. Wang et al. proposed a contextual transformation network (CTN) for lightweight remote sensing image super-resolution, which effectively integrates contextual features at different levels through contextual transformation layers and contextual feature aggregation modules while reducing the amount of model parameters. Although these models have achieved certain results while being lightweight, they do not specifically process the strip-shaped tubular features in remote sensing images. In contrast, our model uses the deformation flexibility of convolution and the long-distance dependency of Transformer, combined with superpixel aggregation, to effectively reconstruct the narrow strip-shaped tubular features in remote sensing images, and achieves a good balance between the amount of parameters and performance. Summary of the invention

[0012] In view of the shortcomings of the prior art, the present invention provides a lightweight remote sensing image super-resolution reconstruction method based on dynamic snake convolution; a convolution module CBAMUpConv with scalable channel number including channel attention and spatial attention is proposed. Under the premise of almost no increase in parameters and computational overhead, this module combines up-convolution and CBAM attention mechanisms, and innovatively performs channel and spatial attention operations on the local, global and tubular deep features extracted from the front part of the framework, so that the model can better identify and learn the key information and positions in the deep features, thereby improving the reconstruction effect.

[0013] A lightweight remote sensing image super-resolution reconstruction method based on dynamic snake convolution includes the following steps:

[0014] Step 1: Given a low-resolution remote sensing image RSI LR ∈R H×W×3 , processed by a convolutional layer to generate shallow features F S ∈R H×W×C ; Where H and W represent the height and width of the remote sensing image, and C represents the number of feature channels.

[0015] Step 2: A deep feature extraction module is constructed by stacking N strip feature superpixel interaction modules SFSPI; the deep feature extraction module extracts the deep feature from the shallow feature F S Extract deep features F D ∈R H×W×C ; Each strip feature superpixel interaction module SFSPI consists of a strip feature superpixel internal and external interaction module SFSPCI and a convolution operation module CBAMConv containing channel attention and spatial attention. Each strip feature superpixel interaction module SFSPI uses residual connection, specifically:

[0016]

[0017] Among them, d i represents the feature of the i-th strip feature superpixel interaction module, x sfspci(i) represents the output feature of the internal and external interaction module of the i-th strip feature superpixel, f 1 represents the internal and external interaction of the strip feature superpixel, f 2 Represents the convolution module CBAMConv operation that includes channel attention and spatial attention;

[0018] Step 3: The deep feature extraction module outputs the deep feature F D , after reconstruction module, high-resolution remote sensing images are obtained Among them, H out Represents the height of the reconstructed output image, W out Indicates the width of the reconstructed output image; specifically:

[0019] RSI HR =f lastconv (f ps (f cbamup (F D ))) (2)

[0020] Among them, f cbamup represents the convolutional module CBAMUpConv with scalable channel number including channel attention and spatial attention, f psrepresents the pixel rearrangement module, f lastconv represents the last convolutional layer; the reconstruction module first uses the spatial and channel attention learning F of the convolutional computation module CBAMUpConv with scalable channel numbers including channel attention and spatial attention. D The deep features in the network are then reshaped to the corresponding scale using pixel rearrangement. Finally, a convolution is used to aggregate the features and output a remote sensing image with three channels.

[0021] The strip feature superpixel internal and external interaction module SFSPCI is used to extract strip tubular features in remote sensing images and generate superpixels; then feature aggregation is performed from the local and global perspectives of superpixels to learn a more comprehensive feature representation; specifically, it includes dynamic snake convolution DSC, superpixel aggregation SPA, superpixel cross attention SPCA and superpixel intra-attention SPIA;

[0022] The dynamic snake convolution DSC is specifically as follows: given a standard convolution kernel of 9×9 2D convolution K, the center point coordinates are K i =(x i ,y i ); By introducing the offset Δ, the convolution can focus on the complex bar structure more flexibly and straighten the standard convolution kernel on the x-axis and y-axis; the execution process on the y-axis is as follows, and the specific position of each grid in K is represented as: k i±c =(x i±c ,y i±c ), where c = {0, 1, 2, 3, 4} represents the horizontal distance from the center grid; each grid position k in the convolution kernel K i±c The selection of is an accumulation process. i Initially, the position of the grid away from the center depends on the position of the previous grid: i In comparison, K i+1 The offset Δ={δ|δ∈[-1,-1]} is added. Therefore, the offsets need to be accumulated, specifically:

[0023]

[0024] Similarly, the execution process formula on the y-axis is as follows:

[0025]

[0026] The superpixel aggregation SPA is specifically as follows: SSN is used to segment the image in a superpixel manner. Specifically, the feature map is first divided into regular grids and the superpixels s∈R are initialized. M×C , M represents the number of superpixels; assuming that a pixel p∈R N×C, N = H × W is the number of pixels, which belongs to one of the M superpixels; then, the Soft Pixel-Superpixel Association is iteratively used to establish the relationship graph between pixels and superpixels; for the tth iteration, the relationship graph is calculated as follows:

[0027]

[0028] in represents the soft connection between pixel p and the i-th superpixel at the t-th iteration; D represents the distance between the calculated pixel and the superpixel center; I p represents the pth pixel of the original image, represents the center of the i-th superpixel at the t-1th iteration; in order to reduce the amount of calculation and ensure the locality of superpixels, only the association between a certain pixel and its surrounding superpixels is calculated; then, the superpixel center is updated, and the formula is:

[0029]

[0030] in, represents the i-th superpixel center at the t-th iteration; Represents the normalization constant along the column; after T iterations, a final relationship matrix Q is obtained T .

[0031] The superpixel cross attention SPCA is specifically: using superpixels as a proxy for pixels, using the self-attention mechanism to learn the relationship between long-range features, given a flattened pixel p∈R HW×C and superpixel s∈R M×C , where M is the number of superpixels; first, a linear mapping is used to calculate the query: Q s ∈R M×D , key: K x ∈R HW×D , value: V x ∈R HW ×C , the formula is as follows:

[0032]

[0033] Among them, s represents super pixel, x represents ordinary pixel, is the weight matrix of query, key and value;

[0034] Then calculate the updated superpixel features, the formula is as follows:

[0035]

[0036] in, is the scaling factor that transfers the aggregated pixels to the pixels. Specifically, use To get the pixel query, Q s As key, s u As the value value, the updated superpixel features are finally mapped back to pixel-level features;

[0037] The superpixel intra-attention SPIA is specifically as follows: The top-N selection module uses the relationship matrix between pixels and superpixels Select the first N pixels that are most similar to the superpixel, calculate the relationship between the pixels inside the superpixel according to the self-attention mechanism of formula (7) (8), and then disperse the calculated pixels back to their corresponding positions in the image. Use the value key to obtain all the features before the pixel attention operation and integrate them with the pixels dispersed back to the original image.

[0038] The convolution module CBAMUpConv with scalable channel number including channel attention and spatial attention uses the convolution block attention module CBAM when processing features, but does not change the number of channels of the features; the feature F with rich information obtained by the deep feature extraction module d ∈R H×W×C First, the convolutional block attention module CBAM is used to integrate spatially and channel-wise, and then the convolution operation is used to expand the feature channel to H×W×C up , where C up =C×upscale 2 , C represents the number of channels, and upscale represents the model parameters.

[0039] The beneficial effects of adopting the above technical solution are:

[0040] The present invention provides a lightweight remote sensing image super-resolution reconstruction method based on dynamic snake convolution, and proposes a lightweight remote sensing image super-resolution reconstruction model-strip feature superpixel interaction network (SFSPIN). The network combines the deformation flexibility of convolutional neural network (CNN) and the long-distance learning ability of Transformer, and can more effectively restore the strip tubular features in remote sensing images. Specifically, the continuous strip feature superpixel internal and external interaction module (SFSPCI) module uses dynamic snake convolution to extract the strip tubular features in remote sensing images, and uses the advantages of superpixel aggregation to process continuous interpretable local areas, and then performs local and global deep feature extraction. In addition, the convolution module CBAMConv containing channel attention and spatial attention and the convolution module CBAMUpConv containing channel attention and spatial attention with scalable channel number further enhance the model's learning ability for deep features from the perspective of space and channels. A series of comparative experiments show that the performance of the strip feature superpixel interaction network (SFSPIN) is better than that of previous algorithms. At the same time, we introduce an attention mechanism in the reconstruction module, which provides an effective solution for improving the reconstruction performance of the super-resolution framework. Combining an attention mechanism adapted to the characteristics of remote sensing images may further improve the reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a remote sensing example image with strip-shaped tubular features;

[0042] Figure 2 Overall image reconstruction for lightweight remote sensing image super-resolution;

[0043] Figure 3 Schematic diagram of the superpixel cross attention SPCA structure;

[0044] Figure 4 Schematic diagram of the SPIA structure within superpixel attention;

[0045] Figure 5 Schematic diagram of a variant of the CBAM attention mechanism;

[0046] Where (a) - spatial and channel attention mechanism in deep feature extraction module; (b) - spatial and channel attention mechanism in reconstruction module;

[0047] Figure 6 Graph showing the effect of the number of DSCs on model performance (×4);

[0048] Figure 7 The curve comparison of PSNR of different methods (×4);

[0049] Figure 8This is a visual comparison of the super-resolution effects of remote sensing images on (×2, ×3, ×4); DETAILED DESCRIPTION

[0050] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0051] A lightweight remote sensing image super-resolution reconstruction method based on dynamic snake convolution includes the following steps:

[0052] Step 1: Given a low-resolution remote sensing image RSI LR ∈R H×W×3 , processed by a convolutional layer to generate shallow features F S ∈R H×W×C ; Where H and W represent the height and width of the remote sensing image, and C represents the number of feature channels.

[0053] Step 2: A deep feature extraction module is constructed by stacking N strip feature superpixel interaction modules SFSPI; the deep feature extraction module extracts the deep feature from the shallow feature F S Extract the deep features F with rich information D ∈R H×W×C ; Each strip feature superpixel interaction module SFSPI consists of a strip feature superpixel internal and external interaction module SFSPCI and a convolutional operation module CBAMConv (Convolutional Block Attention Module with Convolution) containing channel attention and spatial attention; In addition, in order to ensure training stability, each strip feature superpixel interaction module SFSPI uses residual connection, specifically:

[0054]

[0055] Among them, d i represents the feature of the i-th strip feature superpixel interaction module, x sfspci(i) represents the output feature of the internal and external interaction module of the i-th strip feature superpixel, f 1 represents the internal and external interaction of the strip feature superpixel, f 2 Represents the convolution module CBAMConv operation that includes channel attention and spatial attention;

[0056] Step 3: The deep feature extraction module outputs the deep feature F D , after reconstruction module, high-resolution remote sensing images are obtained Among them, H out Represents the height of the reconstructed output image, W outIndicates the width of the reconstructed output image; specifically:

[0057] RSI HR =f lastconv (f ps (f cbamup (F D ))) (10)

[0058] Among them, f cbamup represents the convolutional module CBAMUpConv with scalable channel number including channel attention and spatial attention, f ps represents the pixel rearrangement module, f lastconv represents the last convolutional layer; the reconstruction module first uses the spatial and channel attention learning F of the convolutional computation module CBAMUpConv with scalable channel numbers including channel attention and spatial attention. D The deep features in the network are then reshaped to the corresponding scale using pixel rearrangement. Finally, a convolution is used to aggregate the features and output a remote sensing image with three channels.

[0059] The strip feature superpixel internal and external interaction module SFSPCI is used to extract strip tubular features in remote sensing images and generate superpixels; then feature aggregation is performed from the local and global perspectives of superpixels to learn a more comprehensive feature representation; specifically, it includes dynamic snake convolution DSC, superpixel aggregation SPA, superpixel cross attention SPCA and superpixel intra-attention SPIA;

[0060] The dynamic snake convolution DSC: remote sensing images capture the electromagnetic radiation information of the earth's surface and its coverage, which is recorded in the form of images through different remote sensing sensors (such as optical, infrared, microwave, etc.). Remote sensing images contain rich shape features, among which rivers, roads, etc. mainly present curved and slender strip-shaped tubular feature structures. These structures are randomly distributed and complex and changeable, which brings challenges to the super-resolution reconstruction of remote sensing images. However, standard convolution can only extract local features of fixed size, and the hole convolution only expands the receptive field. Although deformable convolution has achieved good results in adapting to the geometric shapes of different targets, it is difficult to play a better role in the above challenges. Therefore, we use dynamic snake convolution to extract strip-shaped tubular structures. Specifically, given a standard 2D convolution K with a convolution kernel of 9×9 and a center point coordinate of K i =(x i ,y i ); By introducing the offset Δ, the convolution can focus on the complex bar structure more flexibly and straighten the standard convolution kernel on the x-axis and y-axis; the execution process on the y-axis is as follows, and the specific position of each grid in K is represented as: k i±c =(x i±c ,yi±c ), where c = {0, 1, 2, 3, 4} represents the horizontal distance from the center grid; each grid position k in the convolution kernel K i±c The selection of is an accumulation process. i Initially, the position of the grid away from the center depends on the position of the previous grid: i In comparison, K i+1 The offset Δ={δ|δ∈[-1,-1]} is added. Therefore, the offsets need to be accumulated to ensure that the shape of the convolution kernel conforms to the linear structure. Specifically:

[0061]

[0062] Similarly, the execution process formula on the y-axis is as follows:

[0063]

[0064] The superpixel aggregation SPA is specifically: using SSN to segment the image in a superpixel manner, rather than simply dividing the image into fixed-size image blocks. This method can obtain more accurate boundaries by aggregating similar pixels, while reducing the risk of boundary artifacts. Specifically, the feature map is first divided into a regular grid and the superpixels s∈R are initialized. M×C , M represents the number of superpixels; assuming that a pixel p∈R N×C , N = H × W is the number of pixels, which belongs to one of the M superpixels; then, the Soft Pixel-Superpixel Association is iteratively used to establish the relationship graph between pixels and superpixels; for the tth iteration, the relationship graph is calculated as follows:

[0065]

[0066] in represents the soft connection between pixel p and the i-th superpixel at the t-th iteration; D represents the distance between the calculated pixel and the superpixel center; I p represents the pth pixel of the original image, represents the center of the i-th superpixel at the t-1th iteration; in order to reduce the amount of calculation and ensure the locality of superpixels, only the association between a certain pixel and its surrounding superpixels is calculated; then, the superpixel center is updated, and the formula is:

[0067]

[0068] in, represents the i-th superpixel center at the t-th iteration; Represents the normalization constant along the column; after T iterations, a final relationship matrix Q is obtainedT .

[0069] The superpixel cross attention SPCA is specifically: using superpixels as a proxy for pixels and using the self-attention mechanism to learn the relationship between long-distance features, which is conducive to the complementarity between features and improves the reconstruction quality of remote sensing images. Figure 3 As shown: Given a flattened pixel p∈R HW×C and superpixel s∈R M×C , where M is the number of superpixels; first, a linear mapping is used to calculate the query: Q s ∈R M×D , key: K x ∈R HW×D , value: V x ∈R HW×C , the formula is as follows:

[0070]

[0071] Among them, s represents super pixel, x represents ordinary pixel, is the weight matrix of query, key and value;

[0072] Then calculate the updated superpixel features, the formula is as follows:

[0073]

[0074] in, is the scaling factor used to prevent gradient vanishing. Then we pass the aggregated pixels to the pixels, using To get the pixel query, Q s As key, s u As the value value, the updated superpixel features are finally mapped back to pixel-level features. After the above process, the model can learn the global features of remote sensing images.

[0075] The superpixel intra-attention SPIA is specifically: Figure 4 As shown in Figure 1, an intuitive way to improve the resolution of super-resolved images is to use the complementarity of similar pixels within superpixels. However, different superpixels have different numbers of pixels, which inevitably increases the complexity of parallel computing and increases memory consumption due to the large number of pixels. To solve this problem, the top-N selection module uses the relationship matrix between pixels and superpixels to Select the top N pixels that are most similar to the superpixel, calculate the relationship between the pixels inside the superpixel according to the self-attention mechanism of formula (7) (8), and then disperse the calculated pixels back to their corresponding positions in the image. Although the top-N selection module solves the corresponding problem, there are still some similar pixels in the superpixel that have not been calculated. Therefore, use the value key to obtain all the features before the pixel attention operation and integrate them with the pixels dispersed back to the original image.

[0076] The reconstruction part of many models only uses upconvolution and pixel rearrangement to achieve image pixel enlargement. Although this method is simple and effective, it fails to fully utilize the rich spatial and channel information between deep features, which is crucial for reconstruction. Therefore, we propose spatial and channel attention upconvolution to solve this problem. With almost negligible computation and memory consumption, the model performance is significantly improved.

[0077] Different from the convolution module CBAMConv including channel attention and spatial attention in the strip feature superpixel interaction module SFSPI, the convolution module CBAMUpConv including channel attention and spatial attention with expandable channel number uses the convolution block attention module CBAM when processing features, but does not change the number of channels of the features; specifically, Figure 5 As shown, Figure 5 (a) Spatial and channel attention mechanisms in the deep feature extraction module; Figure 5 (b) Spatial and channel attention mechanisms in the reconstruction module;

[0078] The feature F with rich information obtained by the deep feature extraction module d ∈R H×W×C First, the convolutional block attention module CBAM is used to integrate spatially and channel-wise, and then the convolution operation is used to expand the feature channel to H×W×C up , where C up =C×upscale 2 , C represents the number of channels, and upscale represents the model parameter scale 2 3 4.

[0079] This section describes the specific details of the ablation experiments and the performance of each module in super-resolution reconstruction of remote sensing images at different scales.

[0080] In order to fairly and reliably verify the effect of the proposed method, we used a public remote sensing image dataset AID. The dataset contains 10,000 images of 30 scene categories such as cities, forests, and waters. The dimension of each image is 600×600 pixels. We randomly selected 80%, 10%, and 10% of the total number as training datasets, validation datasets, and test datasets, respectively, and used the bicubic interpolation method to degrade the images into corresponding low-resolution images of ×2, ×3, and ×4 scales.

[0081] During the evaluation process, two commonly used evaluation indicators, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), were used on the YCbCr channel. In order to improve efficiency, 15% of the validation set was selected to evaluate the reconstruction effect of the model. The lightweight remote sensing image super-resolution reconstruction of the overall image Figure 2 shown.

[0082] Implementation details:

[0083] We designed two variants of the strip feature superpixel interaction network with different complexity, named Small Scale Strip Feature Superpixel Interaction Network (SFSPIN-S) and Strip Feature Superpixel Interaction Network (SFSPIN). For the Strip Feature Superpixel Interaction Network (SFSPIN), the model contains 8 Strip Feature Superpixel In-Out Interaction (SFSCI) residual groups, dynamic snake convolution (DSConv) is used in each residual group, and a convolutional block attention module (CBAM) is introduced before pixel rearrangement. In the Small Scale Strip Feature Superpixel Interaction Network (SFSPIN-S), we only add dynamic snake convolutions to the first four residual groups to further reduce the number of parameters while maintaining performance. The rest of the settings are consistent with the Strip Feature Superpixel Interaction Network (SFSPIN).

[0084] Training settings

[0085] The experiment used a 64×64 image block size and a batch size of 40 to train the model. During the training process, the initial learning rate was set to 2e-4, and as the training progressed, it was halved at [250, 400, 450, 475, 500] rounds to speed up convergence and improve the robustness of the model. Adam was selected as the optimizer to optimize the model by minimizing the L1 loss (β1=0.9 and β2=0.99). The training process was carried out for a total of 600 rounds. In order to enhance the diversity of training data and improve the generalization ability of the model, image enhancement techniques such as randomly rotating images by 90 degrees, 180 degrees, and 270 degrees and horizontally inverting were used. In this experiment, the PyTorch framework was used and the model was deployed on a device equipped with a 4090 GPU.

[0086] Ablation experiment:

[0087] Ablation experiments are conducted on the AID dataset to verify the role of dynamic snake convolution (DSConv) and convolutional block attention module (CBAM) in remote sensing image super-resolution. For fairness, ×2, ×3, and ×4 strip feature superpixel interaction networks (SFSPIN) are trained under the same settings.

[0088] Table 1 The impact of different modules on model performance

[0089]

[0090]

[0091] Dynamic Snake Convolution (DSConv)

[0092] In the strip feature superpixel interaction network, dynamic snake convolution is used as the first module, which helps the subsequent modules to fully learn the strip tubular features in remote sensing images from the perspective of superpixel aggregation and local-global. The results in Table 1 show that after adding dynamic snake convolution, the performance of the model shows different degrees of performance improvement at each scale compared with no dynamic snake convolution. This shows that dynamic snake convolution can effectively extract narrow and long tubular features in remote sensing images, thereby improving the reconstruction quality of remote sensing images.

[0093] Convolutional module CBAMConv with channel attention and spatial attention and convolutional module CBAMUpConv with scalable channel number with channel attention and spatial attention.

[0094] In order to verify the effectiveness of each module, ablation experiments were conducted, and the specific results are shown in Table 1. It should be noted that in the strip feature superpixel internal and external interaction module (SFSPCI) and the reconstruction module, the residual convolution module CBAMConv with channel attention and spatial attention and the convolution module CBAMUpConv with scalable channel number containing channel attention and spatial attention were introduced respectively. The experimental results show that the spatial and channel attention mechanism of the convolution block attention module CBAM in the deep feature extraction module and the reconstruction module significantly improves the feature learning and expression ability of the model. In addition, the results in Table 1 also show that the combination of dynamic snake convolution DSC and the convolution module CBAMConv with channel attention and spatial attention is better than using dynamic snake convolution (DSC) alone, which shows that the introduction of attention mechanism in the deep feature extraction module can better integrate the strip features of remote sensing images. When the dynamic snake convolution (DSC), the convolution module CBAMConv containing channel attention and spatial attention, and the convolution module CBAMUpConv with scalable channel number containing channel attention and spatial attention are combined, the model performance is further improved, indicating that the attention mechanism introduced by the reconstruction module can more effectively learn deep features in space and channels, which is beneficial to the reconstruction effect of the model.

[0095] The number of dynamic snake convolutions

[0096] The above experiments verify that the strip feature superpixel interaction network (SFSPIN) model outperforms any single module, confirms the integration effect of the convolution module CBAMConv containing channel attention and spatial attention and the convolution module CBAMUpConv containing channel attention and spatial attention with scalable channel number on the strip tubular features extracted by the dynamic snake convolution module, and verifies the importance of the attention mechanism in the reconstruction module. We further studied the impact of the number of dynamic snake convolutions on the performance of the model to strike a balance between parameters and performance. We set the number of dynamic snake convolutions in the residual group of the strip feature superpixel internal and external interaction module from front to back, from 1 block to 8 blocks. Figure 6 As shown in the figure, when the number of dynamic snake convolutions is 6, the peak signal-to-noise ratio and the structural similarity index reach the maximum value; when the number of dynamic snake convolutions is 4, the peak signal-to-noise ratio reaches the maximum value and the structural similarity index reaches the second largest value. In order to further reduce the number of parameters, the number of dynamic snake convolutions is finally set to 4, thus obtaining a strip feature superpixel interaction network (SFSPIN-S) with fewer parameters.

[0097] Comparison between this embodiment and other lightweight methods:

[0098] In this example, the two proposed models SFSPIN-S and SFSPIN are compared with 10 current advanced lightweight models. These models include AWSRN-M, RFDN, LatticeNet, MAFFSRN-L based on convolutional neural network (CNN), and ESRT, SPIN, ELAN-light natural image super-resolution algorithms based on Transformer. In addition, we also compare the algorithms LGCNet, CTNet, and AMFFN for remote sensing images. The specific comparison results are shown in Table 2. Figure 7 and Figure 8 Among them, Table 2 provides a quantitative comparison. Figure 7 shows a visual comparison of PSNR curves, Figure 8 A comparison of visual results is shown.

[0099] The quantitative results are as follows:

[0100] Table 2 shows the quantitative comparison results of different methods at the scales of ×2, ×3, and ×4 for the number of dynamic snake convolutions. The SFSPIN model we proposed outperforms other methods at all scales. Specifically, on the AID dataset (the number of dynamic snake convolutions ×2), our model SFSPIN achieves significant improvements of 0.43dB and 0.2dB compared to LGCNet and CTNet, respectively. In addition, SFSPIN-S also shows better performance than other methods. All these quantitative results show that our method extracts and reconstructs strip-shaped tubular features in remote sensing images more effectively.

[0101] Table 2 provides quantitative comparisons with state-of-the-art methods.

[0102]

[0103]

[0104] The visual effect is as follows:

[0105] Figure 8Visual comparisons are shown (×4, ×3, ×2); in some challenging scenes, previous methods may suffer from blurring artifacts, distortions, or inaccurate texture restoration. In contrast, our method significantly reduces blurring artifacts and preserves more strip-shaped tubular structures and finer texture features. For example, when we compare the remote sensing image "church_110" at ×2, most of the compared methods have difficulty in recovering the strip-shaped tubular structures and texture details in the remote sensing image, and often produce blurring artifacts. However, the results generated by our model are closest to the high-resolution image, recovering the correct structure and texture features with less blur. Similar results are also observed in the reconstruction process of the images "stadium_283" and "parking_226". This is mainly attributed to the superior extraction and representation capabilities of our model in strip-shaped tubular features. In addition, the spatial and channel attention mechanisms of deep features also help to more fully recover the details of remote sensing images.

[0106] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A lightweight remote sensing image super-resolution reconstruction method based on dynamic snake convolution, characterized in that: The following steps are involved: Step 1: Given a low-resolution remote sensing image RSI LR ∈R H×W×3 , processed by a convolutional layer to generate shallow features F S ∈R H×W×C ; Where H and W represent the height and width of the remote sensing image, and C represents the number of feature channels; Step 2: A deep feature extraction module is formed by stacking N strip feature superpixel interaction modules SFSPI; The deep feature extraction module extracts the shallow features F S Extract the deep features F D ∈R H×W×C ; Each strip feature superpixel interaction module SFSPI consists of a strip feature superpixel internal and external interaction module SFSPCI and a convolution operation module CBAMConv containing channel attention and spatial attention. Each strip feature superpixel interaction module SFSPI uses residual connection, specifically: Among them, d i represents the feature of the i-th strip feature superpixel interaction module, x sfspci(i) represents the output feature of the inside-outside interaction module of the i-th strip feature superpixel, f1 represents the inside-outside interaction operation of the strip feature superpixel, and f2 represents the convolution module CBAMConv operation including channel attention and spatial attention; Step 3: The deep feature extraction module outputs the deep feature F D , after reconstruction module, high-resolution remote sensing images are obtained Among them, H out Represents the height of the reconstructed output image, W out Indicates the width of the reconstructed output image; specifically: RSI HR =f lastconv (in ps (in cbamup (F D ))) (2) Among them, f cbamup represents the convolutional module CBAMUpConv with scalable channel number including channel attention and spatial attention, f ps represents the pixel rearrangement module, f lastconv represents the last convolutional layer; the reconstruction module first uses the spatial and channel attention learning F of the convolutional computation module CBAMUpConv with scalable channel numbers including channel attention and spatial attention. D The deep features in are then reshaped to the corresponding scale using pixel rearrangement. Finally, a convolution is used to aggregate the features and output a remote sensing image with three channels.

2. According to claim 1, a lightweight remote sensing image super-resolution reconstruction method based on dynamic snake convolution is characterized in that: The strip feature superpixel inside-outside interaction module SFSPCI is used to extract strip tubular features in remote sensing images and generate superpixels, and perform feature aggregation from the local and global perspectives of superpixels to learn a more comprehensive feature representation; specifically, it includes dynamic snake convolution DSC, superpixel aggregation SPA, superpixel cross attention SPCA and superpixel intra-attention SPIA.

3. According to claim 1, a lightweight remote sensing image super-resolution reconstruction method based on dynamic snake convolution is characterized in that: The dynamic snake convolution DSC is specifically as follows: given a standard convolution kernel of 9×9 2D convolution K, the center point coordinates are K i =(x i ,y i ); By introducing the offset Δ, the convolution can focus on the complex bar structure more flexibly and straighten the standard convolution kernel on the x-axis and y-axis; the execution process on the y-axis is as follows, and the specific position of each grid in K is represented as: k i±c =(x i±c ,y i±c ), where c = {0, 1, 2, 3, 4} represents the horizontal distance from the center grid; each grid position k in the convolution kernel K i±c The selection of is an accumulation process; from the center position K i Initially, the position of the grid away from the center depends on the position of the previous grid: i In comparison, K i+1 The offset Δ={δ|δ∈[-1,-1]} is added; therefore, the offsets need to be accumulated, specifically: Similarly, the execution process formula on the y-axis is as follows: The superpixel aggregation SPA is specifically as follows: SSN is used to segment the image in a superpixel manner. Specifically, the feature map is first divided into regular grids and the superpixels s∈R are initialized. M×C , M represents the number of superpixels; assuming that a pixel p∈R N×C , N = H × W is the number of pixels, which belongs to one of the M superpixels; then, the soft pixel-superpixel association is used to iteratively establish the relationship graph between pixels and superpixels; for the tth iteration, the relationship graph is calculated as follows: in represents the soft connection between pixel p and the i-th superpixel at the t-th iteration; D represents the distance between the calculated pixel and the superpixel center; I p represents the pth pixel of the original image, represents the center of the i-th superpixel at the t-1th iteration; in order to reduce the amount of calculation and ensure the locality of superpixels, only the association between a certain pixel and its surrounding superpixels is calculated; then, the superpixel center is updated, and the formula is: in, represents the i-th superpixel center at the t-th iteration; Represents the normalization constant along the column; after T iterations, a final relationship matrix Q is obtained T .

4. According to claim 1, a lightweight remote sensing image super-resolution reconstruction method based on dynamic snake convolution is characterized in that: The superpixel cross attention SPCA is specifically: using superpixels as a proxy for pixels, using the self-attention mechanism to learn the relationship between long-range features, given a flattened pixel p∈R HW×C and superpixel s∈R M×C , where M is the number of superpixels; first, a linear mapping is used to calculate the query: Q s ∈R M×D , key: K x ∈R HW×D , value: V x ∈R HW×C , the formula is as follows: Among them, s represents super pixel, x represents normal pixel, is the weight matrix of query, key and value; Then calculate the updated superpixel features, the formula is as follows: in, is the scaling factor that transfers the aggregated pixels to the pixels. Specifically, use To get the pixel query, Q s As key, s u As the value value, the updated superpixel features are finally mapped back to pixel-level features.

5. The lightweight remote sensing image super-resolution reconstruction method based on dynamic snake convolution according to claim 1, characterized in that: The superpixel intra-attention SPIA is specifically as follows: The top-N selection module uses the relationship matrix between pixels and superpixels Select the first N pixels that are most similar to the superpixel, calculate the relationship between the pixels inside the superpixel according to the self-attention mechanism of formula (7) (8), and then disperse the calculated pixels back to their corresponding positions in the image; Use the value key to get all the features before the pixel attention operation and integrate them with the pixels scattered back to the original image.

6. The lightweight remote sensing image super-resolution reconstruction method based on dynamic snake convolution according to claim 1, characterized in that: The convolution module CBAMUpConv with scalable channel number including channel attention and spatial attention uses the convolution block attention module CBAM when processing features, but does not change the number of channels of the features; the feature F with rich information obtained by the deep feature extraction module d ∈R H×W×C First, the convolutional block attention module CBAM is used to integrate spatially and channel-wise, and then the convolution operation is used to expand the feature channel to H×W×C up , where C up =C×upscale 2 , C represents the number of channels, and upscale represents the model parameters.

Images