Image super-resolution network construction method based on dynamic sparse self-attention

Abstract

The present invention discloses a method for constructing an image super-resolution network based on dynamic sparse self-attention, which belongs to the field of remote sensing image processing technology. The present invention is based on the Transformer super-resolution architecture, designs dynamic sparse self-attention, highlights high-frequency information and reduces noise, and effectively improves the quality of coastal aquaculture pond image restoration; develops a scale-aware interaction layer to enhance the consistency of ponds of different scales during super-resolution; finally, the spatial resolution of the Sentinel-2 image is super-resolutioned from 10m to 2.5m, and an edge-supervised U-Net network is used to achieve fine-grained extraction of coastal aquaculture ponds in the Sentinel-2 image; the present invention improves the reconstruction effect of the Sentinel-2 image, reduces reconstruction artifacts, obtains Sentinel-2 super-resolution images, achieves fine-grained extraction of coastal aquaculture ponds in medium-resolution images, and reduces costs.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, mainly to the field of remote sensing image data processing technology, and specifically to a method for constructing an image super-resolution network based on dynamic sparse self-attention. Background Art

[0002] Intelligent extraction of coastal aquaculture ponds not only provides important data support for marine resource management and environmental protection, but also provides a scientific basis for aquaculture planning and management. The combination of remote sensing technology and deep learning has greatly promoted the automation of large-scale monitoring of coastal aquaculture ponds. This is especially true for high-resolution remote sensing imagery, where models can often achieve good prediction accuracy through effective training.

[0003] While high-resolution remote sensing imagery (such as 2-meter and sub-meter resolutions) provides excellent image quality and robust model predictions, it is expensive and long-term data is difficult to obtain. The Sentinel-2 satellite provides global observation data with a maximum spatial resolution of 10 meters and a temporal resolution of 5 days, making it a promising data source for intelligent monitoring of coastal aquaculture ponds. However, its spatial resolution often fails to meet the sample accuracy required for supervised training and prediction using deep learning. Image super-resolution technology uses image processing to upscale low-resolution images to high resolution. With the rapid development of deep learning technology, image super-resolution has shown promise in addressing image resolution limitations in coastal aquaculture pond monitoring.

[0004] The current mainstream super-resolution algorithms include those based on convolutional neural networks, Transformer, diffusion models, and generative adversarial networks. Among them, the Transformer-based super-resolution algorithm has outstanding performance in image super-resolution tasks due to its powerful long-range and global aggregation capabilities. However, when the existing Transformer model is directly applied to the image super-resolution task of coastal aquaculture ponds, it faces two key challenges. First, the Transformer takes all features into consideration when performing self-attention calculations, resulting in a large number of redundant feature units. A large number of redundant feature units will weaken the attention allocation to high-frequency details and texture information, resulting in unsatisfactory reconstruction effects. Second, single-scale feature representation cannot effectively capture the correlation of coastal aquaculture ponds at different scales.

[0005] CN113408467B proposes an intelligent extraction method for coastal aquaculture ponds based on Sentinel-2 satellite imagery and a cloud platform. First, using the Google Earrh Engine (GEE) cloud platform, coastal research was gridded at 0.1° x 0.1°. Grid areas of potential marine aquaculture ponds were then determined based on the JRC surface water dataset and a DEM dataset. Next, cloud-free median image synthesis was performed pixel-by-pixel for Sentinel-2 MSI images during the rainy season with low cloud cover. This generated an image set, and the NDWI, NDVI, NDBI, texture features, and edge features were calculated for each image in the set. Finally, based on the constructed features, the Kmeans algorithm was used for image segmentation, and coastal aquaculture ponds were screened using decision tree classification. This method has a low degree of automation and is more research-oriented, making it unsuitable for large-scale statistical analysis and extraction in engineering applications.

[0006] CN1113378679B proposes a coastal aquaculture pond extraction method based on improved geometric features and feature-preserving sampling. First, the coastal zone is extracted from the pre-processed Lansat-8 image data. Then, based on the improved geometric features, river seed areas are selected, including obtaining long river areas based on the shape features of the minimum circumscribed matrix ratio, removing the aquaculture pond areas that are connected due to the dam's unclear shape features based on the average width shape features, and finally removing the river image data from the coastal water image data to obtain the aquaculture pond area image information. This method effectively extracts coastal aquaculture ponds containing rivers from multispectral images at medium resolution. However, when processing images with lower resolution, the aquaculture pond is only one or two pixels wide, easily adhering to adjacent aquaculture ponds and difficult to remove. In addition, the method is designed with multiple steps and complex calculations, which is not efficient when processing large-scale data.

[0007] Therefore, there is an urgent need to construct an image super-resolution network construction method for coastal aquaculture pond image restoration that can extract detail information more accurately, effectively solve the boundary adhesion problem, and have a higher degree of automation and adaptability. Summary of the Invention

[0008] In response to the problems of high data acquisition cost, insufficient data temporal resolution, and low degree of automation and adaptability of network models in existing methods, the present invention proposes an improved scheme suitable for coastal aquaculture pond image restoration. Specifically, it designs a dynamic sparse self-attention and scale-aware interaction layer, which can effectively reduce artifacts in image restoration and improve the restoration quality of small ponds. The edge-supervised U-Net network is used to successfully extract coastal aquaculture ponds and their edges in super-resolution Sentinel-2 images, providing a complete solution from super-resolution to segmentation, and providing a method for constructing an image super-resolution network based on dynamic sparse self-attention.

[0009] The image super-resolution network construction method based on dynamic sparse self-attention includes:

[0010] S1. Construct the dynamic sparse window interaction block DSwin Block, which consists of dynamic sparse self-attention DSA, window self-attention WSA and scale-aware interaction module SAIM;

[0011] S2. Build an image super-resolution network model based on dynamic sparse window interaction blocks;

[0012] S3. Obtain Sentinel-2 satellite images of coastal aquaculture ponds to construct a training dataset;

[0013] S4. Train the image super-resolution network model based on the training dataset to obtain training weights, and then perform restoration and reconstruction of the Sentinel-2 image based on the training weights to obtain a Sentinel-2 super-resolution image.

[0014] S5. Extraction of coastal aquaculture ponds based on Sentinel-2 super-resolution images.

[0015] Furthermore, S1 includes:

[0016] S1.1, construct dynamic sparse self-attention (DSA) to capture high-frequency detail and texture information required for image restoration;

[0017] Input a set of features , through convolution, the features Mapping to query vector , key vector Sum value vector :

[0018] ;

[0019] in, Representation characteristics The size is , 、 and Represents characteristics The feature dimension, feature map height and feature map width of Represents Chunk, represents 1×1 convolution, Indicates that the convolution kernel size is Depthwise separable convolution;

[0020] After performing Euclidean L2 normalization on the query vector and the key vector, a dot product operation is performed on the transpose of the query vector and the key vector to generate a similarity dense matrix :

[0021] ;

[0022] in, Represents the transpose of the key vector.

[0023] Furthermore, a learnable interval partition parameter is introduced , Top-K feature selection is performed in three different intervals, The similarity dense matrix of a feature with other features is , used to divide the values ​​of different sparse stages for:

[0024] ;

[0025] in, Indicates that Restricted to a specified range Inside;

[0026] Define sparse mask based on Perform interval Top-K selection to obtain the sparse similarity matrix of each stage , and then through the activation function softmax Normalize and get the output of dynamic sparse self-attention :

[0027] ;

[0028] ;

[0029] in, Indicates The features are similar and the similarity ranking is top The index collection of Indicates the remaining locations except the index collection.

[0030] Furthermore, S1 also includes:

[0031] S1.2, constructing a scale-aware interaction module (SAIM) to perform multi-scale feature interaction processing;

[0032] Input a set of features First, the input features are mapped to the hidden feature space through 1×1 convolution to obtain the hidden space features :

[0033] ;

[0034] in, Represents characteristics The feature dimension, feature map height and feature map width of

[0035] Depthwise separable convolutions are used to extract features from different scales. Extract features of different scales and use block operations to divide features of different scales into three parts:

[0036] ;

[0037] in, Indicates the number of channels, Represents the activation function GELU, Respectively represent the features of the first part, the second part, and the third part obtained by the block operation, Indicates the convolution kernel size of depth-wise separable convolution;

[0038] Part of the features of the feature maps of different scales are spliced ​​together to obtain multi-scale features, and depth-wise separable convolution is used to interactively process the multi-scale features:

[0039] ;

[0040] in, Indicates the concatenation of different features in the channel dimension. represents regularized Dropout, Represents the interactive feature map obtained by interactively processing multi-scale features, Indicates the convolution kernel size of depth-wise separable convolution;

[0041] Fuse the three sets of interaction feature maps, use 1×1 convolution to restore the feature dimension of the fused feature maps, and obtain the output features of the scale-aware interaction module :

[0042] ;

[0043] The output features contain feature information at different scales.

[0044] Furthermore, S1 also includes:

[0045] S1.3, build the dynamic sparse window interaction block DSwin Block, and transform the features Enter the dynamic sparse window interaction block, first Perform layer normalization to obtain normalized features , and then through dynamic sparse self-attention and window self-attention Perform feature extraction and compare the feature extraction results with the input features Add up to get the self-attention extraction feature :

[0046] ;

[0047] in, , and They represent the feature extraction results of dynamic sparse self-attention and window self-attention respectively;

[0048] Use layer normalization LayerNorm and SAIM to Processing is performed to obtain multi-scale features through residual splicing :

[0049] ;

[0050] in, , Representation layer normalization, Represents the scale-aware interaction layer.

[0051] Furthermore, in S2, the low-resolution image Input the built image super-resolution network model, and perform feature extraction and image reconstruction on low-resolution images based on the image super-resolution network model, including:

[0052] S2.1, shallow feature extraction;

[0053] The input low-resolution image is transformed into Map to high-dimensional feature space and extract shallow features :

[0054] ;

[0055] S2.2, deep feature extraction;

[0056] Deep feature extraction consists of N layers of residual feature selection groups RASG, and the features extracted by RASG in each layer for:

[0057] ;

[0058] in, Represents the input features of RASG at each layer, and extracts shallow features As the initial input feature for deep feature extraction, it is equivalent to the input feature of RASG in the first layer. , the input features of the RASG from the second to the Nth layer are the output features of the RASG in the previous layer, Indicates continuity Use dynamic sparse window interaction blocks for the second time;

[0059] S2.3, image restoration and reconstruction;

[0060] A layer of 3×3 convolution is used to reduce the dimension of deep features from 180 to 64, and then a group of 3×3 convolutions is used to map the number of channels back to 3 channels. Finally, pixel shuffle is used to restore the image resolution to 4 times the original, realizing the restoration and reconstruction of Sentinel-2 images.

[0061] Furthermore, S3 includes:

[0062] S3.1: Acquire Sentinel-2 images of coastal aquaculture ponds, resample the bands of the Sentinel-2 images to 10-meter spatial resolution, and then perform band synthesis to obtain a 10-meter resolution RGB image.

[0063] S3.2, create training sample pairs and build a training dataset;

[0064] High-resolution images with similar shapes to the coastal aquaculture ponds in the target area were selected, and the selected images were downsampled to adjust the spatial resolution of the images to 2.5 meters and 10 meters. The image size was adjusted, and the image size of the 2.5-meter spatial resolution was set to 256, and the image size of the 10-meter spatial resolution was set to 64. Images with the same image spatial resolution of 2.5 meters and 10 meters were selected to form a training sample pair. Several training sample pairs were made based on the selected high-resolution images to construct a training dataset.

[0065] Furthermore, S4 includes:

[0066] S4.1, model training;

[0067] The training samples in the training data set are input into the model, the initial learning rate of the model training is set to 0.0001, the loss function is L1 loss, and the optimizer AdamW is used to train the model for 100 rounds to obtain the model weights;

[0068] S4.2, image stitching and reconstruction;

[0069] Acquire Sentinel-2 images of coastal aquaculture ponds in the target area, use the acquired images as input samples, and perform super-resolution processing on the images based on the model weights.

[0070] The input sample pair is divided into several small blocks, each of which is a patch. Super-resolution processing is performed on each small block, and then the complete high-resolution image is reconstructed by splicing. For each extracted patch , apply the super-resolution network for separate reconstruction:

[0071] ;

[0072] in, represents the super-resolution model, is the network parameter, all patches are input into the network independently for processing to generate corresponding high-resolution patches , obtain the reconstructed patches and splice the reconstructed patches to complete the image reconstruction.

[0073] Furthermore, the reconstructed patches are adaptively weighted fused to achieve smooth patch splicing and complete image reconstruction. First, a weight attenuation matrix is ​​constructed in the overlapping area of ​​the high-resolution patches. , and then perform pixel-by-pixel weighted accumulation of all patches according to their spatial positions:

[0074] ;

[0075] ;

[0076] in, Represents a pixel, represents the Euclidean distance from the border of the overlapping region, Indicates the overlap size, represents the pixel weight of the patch, represents the reconstructed patch at the corresponding position.

[0077] Furthermore, S5 includes:

[0078] S5.1, create image segmentation dataset;

[0079] Obtain the vector file of the Sentinel-2 super-resolution image and convert the vector file into a mask label. Based on the mask label, obtain the edge label corresponding to the Sentinel-2 super-resolution image to create a Sentinel-2 super-resolution image dataset.

[0080] S5.2, Construction and training of segmentation network for coastal aquaculture ponds;

[0081] Add an edge output branch to the end of the fully convolutional symmetric network U-Net to introduce edge supervision. Set the initial learning rate of network training to 0.01 and the loss function to NLL loss. Use the AdamW optimizer and the CosineAnnealing learning rate scheduler to train the network on the Sentinel-2 super-resolution image dataset for 100 rounds to obtain image segmentation weights.

[0082] S5.3, extraction and restoration of coastal aquaculture ponds;

[0083] Based on the trained image segmentation weights, coastal aquaculture ponds are extracted from super-resolution Sentinel-2 images. The initial extraction results of the ponds are restored by combining opening and closing operations, and finally converted into engineering files.

[0084] Compared with the prior art, the present invention has the following beneficial effects:

[0085] This invention improves the super-resolution effect of the existing Transformer architecture in coastal aquaculture pond images, including reducing reconstruction artifacts and improving edge smoothness;

[0086] The present invention realizes the restoration and reconstruction of existing Sentinel-2 images, improves the resolution of Sentinel-2 images, and reduces the imaging cost of applying deep learning technology to the refined extraction task of coastal aquaculture ponds. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 A diagram of the dynamic sparse attention structure provided by an embodiment of the present invention.

[0088] Figure 2 This is a structural diagram of the scale-aware interaction module provided in an embodiment of the present invention.

[0089] Figure 3 This is an architectural diagram of the image reconstruction network model provided by an embodiment of the present invention.

[0090] Figure 4 This is a diagram showing the extraction results of a coastal aquaculture pond based on a Sentinel-2 super-resolution image provided by an embodiment of the present invention.

[0091] Figure 5 for Figure 4 Mask extraction result diagram.

[0092] Figure 6 for Figure 4 Edge extraction result diagram. DETAILED DESCRIPTION

[0093] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention are described clearly and completely below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0094] The image super-resolution network construction method based on dynamic sparse self-attention includes:

[0095] S1. Construct the dynamic sparse window interaction block DSwin Block, which consists of dynamic sparse self-attention DSA, window self-attention WSA and scale-aware interaction module SAIM;

[0096] S2. Build an image super-resolution network model based on dynamic sparse window interaction blocks;

[0097] S3. Obtain Sentinel-2 satellite images of coastal aquaculture ponds to construct a training dataset;

[0098] S4. Train the image super-resolution network model based on the training dataset to obtain training weights, and then perform restoration and reconstruction of the Sentinel-2 image based on the training weights to obtain a Sentinel-2 super-resolution image.

[0099] S5. Extraction of coastal aquaculture ponds based on Sentinel-2 super-resolution images.

[0100] Specifically, S1 includes:

[0101] S1.1, construct dynamic sparse self-attention (DSA) to capture high-frequency detail and texture information required for image restoration;

[0102] like Figure 1 As shown, a set of features is input to the dynamic sparse self-attention module The dynamic sparse self-attention module DSA first processes the input features through 1×1 convolution and 3×3 depth-separable convolution, and then reshapes the features Mapping to query vector , key vector Sum value vector , specifically expressed as:

[0103] ;

[0104] in, Representation characteristics The size is , 、 and Represents characteristics The feature dimension, feature map height and feature map width of Chunk represents a chunk, used to separate query vectors , key vector Sum value vector , Represents 1×1 convolution, which is a point convolution, used to transform the input features Convert to query vector , key vector Sum value vector , providing a basis for self-attention calculation, Indicates that the convolution kernel size is The depth-wise separable convolution is equivalent to Depthwise separable convolution is used to extract deeper feature information;

[0105] Specifically, the features can also be transformed into Mapping to query vector , key vector Sum value vector .

[0106] After performing Euclidean L2 normalization on the query vector and the key vector, a dot product operation is performed on the transpose of the query vector and the key vector to generate a similarity dense matrix :

[0107] ;

[0108] in, represents the transpose of the key vector, Represents the feature dimension of the feature vector group.

[0109] Relative to calculating the shape in the spatial dimension The attention matrix and channel similarity strategy are more helpful to reduce memory consumption and achieve efficient reasoning.

[0110] The key to dynamic sparse self-attention (DSA) is to remove the interference of noise features when calculating self-attention, which is done by selecting The most relevant k feature vectors capture the high-frequency detail information and texture information required for image restoration, such as the dams and water bodies of coastal aquaculture ponds. The channel similarity strategy helps reduce memory consumption and achieve efficient reasoning. By introducing learnable parameters , the attention of different sparse ranges is divided and modeled in different intervals, and the dimension of the query feature vector is .

[0111] Specifically, three learnable interval partitioning parameters are introduced , Top-K feature selection is performed in three different intervals, The similarity dense matrix of a feature with other features is , used to divide the values ​​of different sparse stages for:

[0112] ;

[0113] in, Indicates that Restricted to a specified range Inside;

[0114] Specifically, , , ;

[0115] For each stage, a sparse mask is defined based on Perform interval Top-K selection to obtain the sparse similarity matrix of each stage , and then through the activation function softmax Normalize and get the output of dynamic sparse self-attention :

[0116] ;

[0117] ;

[0118] in, Indicates The features are similar and the similarity ranking is top The index collection of Indicates the remaining positions except the index collection, by setting the remaining positions to Achieve sparsification, sparse similarity matrix for each stage Perform softmax normalization and compare it with the corresponding value vector Multiply and get the attention output , will be resized (shaped) to , by integrating the information of different attention heads through a layer of point convolution to enhance the image reconstruction quality; this mechanism effectively retains the high response correlation between each feature and other key contexts, and improves the model's modeling ability in complex backgrounds or unstructured areas through the fusion of multiple groups of sparse attention subspaces. Figure 1 The three similarity dense matrices in represent the similarity dense matrices after screening in different intervals, where the white blocks represent the filtered elements and the gray blocks represent the retained elements.

[0119] Specifically, S1 also includes:

[0120] S1.2, constructing a scale-aware interaction module (SAIM) to perform multi-scale feature interaction processing and enhance the scale adaptability of the model through multi-scale feature extraction and adaptive weighting mechanism;

[0121] like Figure 2 As shown, a set of features is input to the scale-aware interaction module The scale-aware interaction module first transforms the input features into Map to a hidden feature space of different dimensions to obtain latent space features :

[0122] ;

[0123] in, The size is , Represents characteristics The feature dimension, feature map height and feature map width of

[0124] Then, three layer-by-layer convolutions are used to extract features from different scales, and depth-wise separable convolutions with kernel sizes of 3, 5, and 7 are used to extract features from different scales. Extract features of different scales and divide them into three parts using block operation for interactive processing of features of different scales. The specific expression is:

[0125] ;

[0126] in, Indicates the number of channels, Represents the activation function GELU, Respectively represent the features of the first part, the second part, and the third part obtained by the block operation, and The size is , Indicates the convolution kernel size of depth-wise separable convolution;

[0127] Part of the features of the feature maps of different scales are spliced ​​together to obtain multi-scale features, and depth-wise separable convolution is used to interactively process the multi-scale features:

[0128] ;

[0129] in, It means splicing different features in the channel dimension, which is equivalent to fusing feature information of different scales. Represents regularized Dropout, which is used to prevent the network from overfitting. Represents the interactive feature map obtained by interactively processing multi-scale features, Indicates the convolution kernel size of depthwise separable convolution; depthwise separable convolution with kernel sizes of 3, 5, and 7 is used to further extract the detailed features of the image and enhance its expressiveness through nonlinear activation functions;

[0130] Fuse the three sets of interactive feature maps, use 1×1 convolution to restore the feature dimension of the fused feature map, map the number of channels of the fused feature map to the original channel, and obtain the output features of the scale-aware interaction module. :

[0131] ;

[0132] Different features are spliced ​​together, and the feature size after splicing is , restore the feature dimension of the fused feature map through 1×1 convolution, map the number of channels of the fused feature map to the original channel, and obtain the output feature The size is , the obtained output features contain feature information at different scales.

[0133] Coastal aquaculture ponds have different scales at different locations in remote sensing images. Due to the fixed channel mapping method, traditional feedforward networks (MLPs) cannot dynamically adjust the influence weights of information at different scales, which limits the model's adaptability in complex scale environments. The present invention constructs a scale-aware interaction module (SAIM) to perform multi-scale feature interaction processing. Compared with the traditional multilayer perceptron (MLP) structure, the scale-aware interaction module (SAIM) effectively solves the adaptability problem of pond images of different scales through multi-scale feature extraction and spatial adaptive scale attention mechanism. This module can not only enhance detailed information at the local scale, but also ensure the consistency of ponds of different scales in the overall reconstruction.

[0134] Specifically, S1 also includes:

[0135] S1.3, build the dynamic sparse window interaction block DSwin Block, and transform the features Enter the dynamic sparse window interaction block, first Perform layer normalization to improve the training stability of deep learning network models and obtain normalized features , , and then the features Input into dynamic sparse self-attention and window self-attention, through dynamic sparse self-attention and window self-attention Perform feature extraction and compare the feature extraction results with the input features Add up to get the self-attention extraction feature :

[0136] ;

[0137] in, Representation characteristics The size is , and They represent the feature extraction results of dynamic sparse self-attention and window self-attention respectively;

[0138] DSA improves attention allocation to key areas by screening key information, but its global screening method may lead to insufficient expression of local features, especially in the structural recovery and detailed texture modeling of small-scale ponds, where there is still potential information loss. Therefore, in the self-attention calculation process, WSA makes feature calculation more localized by dividing the local window, thereby enhancing the feature interaction ability of the local area.

[0139] Use layer normalization LayerNorm and SAIM to Processing, stabilizing the training process, and obtaining multi-scale features through residual splicing :

[0140] ;

[0141] in, Representation characteristics The size is , Representation layer normalization, Represents the scale-aware interaction layer.

[0142] Specifically, in S2, Figure 3 As shown, the low-resolution image Input the built image super-resolution network model, and perform feature extraction and image reconstruction on low-resolution images based on the image super-resolution network model, including:

[0143] S2.1, shallow feature extraction;

[0144] Assume the input low-resolution image is First, the input low-resolution image is transformed through a layer of 3×3 convolution Map to high-dimensional feature space and extract shallow features , expressed as:

[0145] ;

[0146] Among them, 180 and Represents characteristics The feature dimension, feature map height and feature map width of

[0147] S2.2, deep feature extraction;

[0148] The shallow features extracted As the initial input of deep features, deep feature extraction consists of N layers of residual feature selection groups RASG, and the features extracted by RASG in each layer for:

[0149] ;

[0150] in, Represents the input features of RASG at each layer, and extracts shallow features As the initial input feature for deep feature extraction, it is equivalent to the input feature of RASG in the first layer. , the input features of the RASG from the second to the Nth layer are the output features of the RASG in the previous layer, Indicates continuity The dynamic sparse window interaction block is used for the first time; the number of RASG layers and the number of DSwin Blocks in each layer of RASG set in the present invention during the experiment are both 6; finally, LayerNorm is used for normalization, and the residuals of the deep feature extraction results and the shallow feature extraction results are added to prevent feature collapse.

[0151] S2.3, image restoration and reconstruction;

[0152] A layer of 3×3 convolution is used to reduce the dimension of deep features from 180 to 64 to reduce computational complexity and retain important feature information. Then a set of 3×3 convolutions is used to map the number of channels back to 3 channels. Finally, pixel shuffling is used to restore the image resolution to 4 times the original, realizing the restoration and reconstruction of Sentinel-2 images.

[0153] Specifically, S3 includes:

[0154] S3.1. Download and process Sentinel-2 imagery. Obtain Sentinel-2 imagery of coastal aquaculture ponds. Use the Sentinel Application Platform (SNAP) software to resample the Sentinel-2 imagery to a 10-meter spatial resolution. Use the remote sensing image processing software ENVI (The Environment for Visualizing Images) to synthesize the bands and obtain a 10-meter-resolution RGB image.

[0155] S3.2, create training sample pairs and build a training dataset;

[0156] High-resolution images with similar shapes to the coastal aquaculture ponds in the target area were selected, and the selected images were downsampled. The spatial resolution of the images was adjusted to 2.5 meters and 10 meters. The image size was adjusted, and the image size of the 2.5-meter spatial resolution was set to 256, and the image size of the 10-meter spatial resolution was set to 64. Images with the same image spatial resolution of 2.5 meters and 10 meters were selected to form a training sample pair. Based on the selected high-resolution images, the image quantity was enhanced through geometric rotation to produce several training sample pairs and construct a training dataset.

[0157] Specifically, S4 includes:

[0158] S4.1, model training;

[0159] The training samples in the training data set are input into the model. The initial learning rate of the model training is set to 0.0001. The learning rate is halved when the training cycle is halfway through. The loss function is L1 loss. The model is trained using the optimizer AdamW for 100 rounds to obtain the model weights.

[0160] S4.2, image stitching and reconstruction;

[0161] Acquire Sentinel-2 images of coastal aquaculture ponds in the target area, use the acquired images as input samples, and perform super-resolution processing on the images based on the model weights.

[0162] The present invention adopts the strategy of "patch segmentation-local super-resolution-global reconstruction", dividing the input sample pair into several small blocks, each of which is a patch. Super-resolution processing is performed on each small block, and then a complete high-resolution image is reconstructed by stitching. This method not only improves computational efficiency, but also ensures a smooth transition of the stitching area and effectively reduces boundary artifacts.

[0163] For each extracted patch , apply the super-resolution network for separate reconstruction:

[0164] ;

[0165] in, represents the super-resolution model, is the network parameter, all patches are input into the network independently for processing to generate corresponding high-resolution patches , obtain the reconstructed patches and splice the reconstructed patches to complete the image reconstruction.

[0166] Specifically, due to the overlapping areas between patches, direct splicing may produce boundary transition mutations or splicing error accumulation. To alleviate this problem, the present invention performs adaptive weighted fusion on the reconstructed patches to achieve smooth splicing of patches and complete image reconstruction. First, a weight attenuation matrix is ​​constructed in the overlapping area of ​​the high-resolution patches. , the weight matrix makes the overlapping center area have a higher weight, retaining more information; the edge area gradually decays to reduce the stitching error:

[0167] ;

[0168] Finally, all patches are weighted and accumulated pixel by pixel according to their spatial positions:

[0169] ;

[0170] in, Represents a pixel, represents the Euclidean distance from the border of the overlapping region, Indicates the overlap size, Represents the pixel weights from different patches, ensuring seamless transition across the entire reconstructed image. Represents the reconstructed patch at the corresponding position. The image reconstruction network model of the present invention is based on the super-resolution algorithm and adopts the strategy of "patch segmentation-local super-resolution-global reconstruction" to perform super-resolution reconstruction of the Sentinel-2 image. Compared with the original Sentinel-2 image, the detailed information of the coastal aquaculture pond in the super-resolution Sentinel-2 image is well restored, and smooth edges and clear pond textures are obtained. At the same time, the present invention performs adaptive weighted fusion on the reconstructed patches, so that there are no obvious splicing marks in the super-resolution image, and the structure of the coastal aquaculture pond is clearly visible.

[0171] Specifically, S5 includes:

[0172] S5.1, create image segmentation dataset;

[0173] Vector files of Sentinel-2 super-resolution images were obtained and converted into mask labels. Based on the mask labels, edge labels corresponding to the Sentinel-2 super-resolution images were obtained to create a Sentinel-2 super-resolution image dataset. Vector files of reconstructed coastal aquaculture ponds were obtained using the geographic information system (GIS) data processing software Arcgis and converted into mask labels. The edge labels were then obtained using the Canny edge detection algorithm based on the mask labels.

[0174] S5.2, Construction and training of segmentation network for coastal aquaculture ponds;

[0175] Add an edge output branch to the end of the fully convolutional symmetric network U-Net to introduce edge supervision. Set the initial learning rate of network training to 0.01 and the loss function to NLL loss. Use the AdamW optimizer and the CosineAnnealing learning rate scheduler to train the network on the Sentinel-2 super-resolution image dataset for 100 rounds to obtain image segmentation weights.

[0176] S5.3, extraction and restoration of coastal aquaculture ponds;

[0177] Based on the trained image segmentation weights, coastal aquaculture ponds are extracted from the Sentinel-2 super-resolution image. The initial extraction results of the ponds are restored by combining opening and closing operations, and finally converted into engineering files.

[0178] The embodiment of the present invention uses a dynamic sparse self-attention image super-resolution network model to extract image features and train the model on the Sentinel-2 image, obtain training weights, perform image restoration and image reconstruction on the Sentinel-2 image based on the training weights, improve the resolution of the Sentinel-2 image, and obtain a Sentinel-2 super-resolution image. Then, an image segmentation and extraction model is constructed, and based on the image segmentation algorithm, coastal aquaculture ponds are extracted from the Sentinel-2 super-resolution image to obtain the extraction results of the coastal beauty ponds. Figure 4 This is a diagram showing the extraction results of a coastal aquaculture pond based on a Sentinel-2 super-resolution image provided by an embodiment of the present invention. Figure 5 for Figure 4 The mask extraction result map of Figure 6 for Figure 4 The edge extraction result graph is obtained by Figure 4 、 5 ,6 It can be observed that the extracted aquaculture ponds and edges are very complete and have high accuracy.

[0179] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for constructing an image super-resolution network based on dynamic sparse self-attention, characterized by: include: S1. Construct the dynamic sparse window interaction block DSwin Block, which consists of dynamic sparse self-attention DSA, window self-attention WSA and scale-aware interaction module SAIM; S2. Build an image super-resolution network model based on dynamic sparse window interaction blocks; S3. Obtain Sentinel-2 satellite images of coastal aquaculture ponds to construct a training dataset; S4. Train the image super-resolution network model based on the training dataset to obtain training weights, and then perform restoration and reconstruction of the Sentinel-2 image based on the training weights to obtain a Sentinel-2 super-resolution image. S5. Extraction of coastal aquaculture ponds based on Sentinel-2 super-resolution images; S1 includes: S1.1, construct dynamic sparse self-attention (DSA) to capture high-frequency detail and texture information required for image restoration; S1.2, constructing a scale-aware interaction module (SAIM) to perform multi-scale feature interaction processing; Input a set of features First, the input features are mapped to the hidden feature space through 1×1 convolution to obtain the hidden space features : ； in, Represents characteristics The feature dimension, feature map height and feature map width of Depthwise separable convolutions are used to extract features from different scales. Extract features of different scales and use block operations to divide features of different scales into three parts: ； in, Indicates the number of channels, Represents the activation function GELU, Respectively represent the features of the first part, the second part, and the third part obtained by the block operation, Indicates the convolution kernel size of depth-wise separable convolution; Part of the features of the feature maps of different scales are spliced ​​together to obtain multi-scale features, and depth-wise separable convolution is used to interactively process the multi-scale features: ； in, Indicates the concatenation of different features in the channel dimension. represents regularized Dropout, Represents the interactive feature map obtained by interactively processing multi-scale features, Indicates the convolution kernel size of depth-wise separable convolution; Fuse the three sets of interaction feature maps, use 1×1 convolution to restore the feature dimension of the fused feature maps, and obtain the output features of the scale-aware interaction module : ； The output features contain feature information at different scales; S1.3, build the dynamic sparse window interaction block DSwin Block, and transform the features Enter the dynamic sparse window interaction block, first Perform layer normalization to obtain normalized features , and then through dynamic sparse self-attention and window self-attention Perform feature extraction and compare the feature extraction results with the input features Add up to get the self-attention extraction feature : ； in, , and They represent the feature extraction results of dynamic sparse self-attention and window self-attention respectively; Use layer normalization LayerNorm and SAIM to Processing is performed to obtain multi-scale features through residual splicing : ； in, , Representation layer normalization, Represents the scale-aware interaction layer.

2. The method for constructing an image super-resolution network based on dynamic sparse self-attention according to claim 1, characterized in that: S1.1 includes: Input a set of features , through convolution, the features Mapping to query vector , key vector Sum value vector : ； in, Representation characteristics The size is , 、 and Represents characteristics The feature dimension, feature map height and feature map width of Represents Chunk, represents 1×1 convolution, Indicates that the convolution kernel size is Depthwise separable convolution; After performing Euclidean L2 normalization on the query vector and the key vector, a dot product operation is performed on the transpose of the query vector and the key vector to generate a similarity dense matrix : ； in, Represents the transpose of the key vector.

3. The method for constructing an image super-resolution network based on dynamic sparse self-attention according to claim 2, characterized in that: Introducing learnable interval partitioning parameters , Top-K feature selection is performed in three different intervals, The similarity dense matrix of a feature with other features is , used to divide the values ​​of different sparse stages for: ； in, Indicates that Restricted to a specified range Inside; Define sparse mask based on Perform interval Top-K selection to obtain the sparse similarity matrix of each stage , and then through the activation function softmax Normalize and get the output of dynamic sparse self-attention : ； ； in, Indicates The features are similar and the similarity ranking is top The index collection of Indicates the remaining locations except the index collection.

4. The method for constructing an image super-resolution network based on dynamic sparse self-attention according to claim 3, characterized in that: In S2, the low-resolution image Input the built image super-resolution network model, and perform feature extraction and image reconstruction on low-resolution images based on the image super-resolution network model, including: S2.1, shallow feature extraction; The input low-resolution image is transformed into Map to high-dimensional feature space and extract shallow features : ； S2.2, deep feature extraction; Deep feature extraction consists of N layers of residual feature selection groups RASG, and the features extracted by RASG in each layer for: ； in, Represents the input features of the first layer of RASG, and extracts the shallow features As the initial input feature for deep feature extraction, it is equivalent to the input feature of RASG in the first layer. , the input features of the RASG from the second to the Nth layer are the output features of the RASG in the previous layer, Indicates continuity Use dynamic sparse window interaction blocks twice; S2.3, image restoration and reconstruction; A layer of 3×3 convolution is used to reduce the dimension of deep features from 180 to 64, and then a group of 3×3 convolutions is used to map the number of channels back to 3 channels. Finally, pixel shuffle is used to restore the image resolution to 4 times the original, realizing the restoration and reconstruction of Sentinel-2 images.

5. The method for constructing an image super-resolution network based on dynamic sparse self-attention according to claim 4, wherein S3 include: S3.1: Acquire Sentinel-2 images of coastal aquaculture ponds, resample the bands of the Sentinel-2 images to 10-meter spatial resolution, and then perform band synthesis to obtain a 10-meter resolution RGB image. S3.2, create training sample pairs and build a training dataset; High-resolution images with similar shapes to the coastal aquaculture ponds in the target area were selected, and the selected images were downsampled to adjust the spatial resolution of the images to 2.5 meters and 10 meters. The image size was adjusted, and the image size of the 2.5-meter spatial resolution was set to 256, and the image size of the 10-meter spatial resolution was set to 64. Images with the same image spatial resolution of 2.5 meters and 10 meters were selected to form a training sample pair. Several training sample pairs were made based on the selected high-resolution images to construct a training dataset.

6. The method for constructing an image super-resolution network based on dynamic sparse self-attention according to claim 5, wherein S4 include: S4.1, model training; The training samples in the training data set are input into the model, the initial learning rate of the model training is set to 0.0001, the loss function is L1 loss, and the optimizer AdamW is used to train the model for 100 rounds to obtain the model weights; S4.2, image stitching and reconstruction; Acquire Sentinel-2 images of coastal aquaculture ponds in the target area, use the acquired images as input samples, and perform super-resolution processing on the images based on the model weights. The input sample pair is divided into several small blocks, each of which is a patch. Super-resolution processing is performed on each small block, and then the complete high-resolution image is reconstructed by splicing. For each extracted patch , apply the super-resolution network for separate reconstruction: ； in, represents the super-resolution model, is the network parameter, all patches are input into the network independently for processing to generate corresponding high-resolution patches , obtain the reconstructed patches and splice the reconstructed patches to complete the image reconstruction.

7. The method for constructing an image super-resolution network based on dynamic sparse self-attention according to claim 6, characterized in that: The reconstructed patches are adaptively weighted and fused to achieve smooth patch splicing and complete image reconstruction. First, a weight attenuation matrix is ​​constructed in the overlapping area of ​​the high-resolution patches. , and then perform pixel-by-pixel weighted accumulation of all patches according to their spatial positions: ； ； in, Represents a pixel, represents the Euclidean distance from the border of the overlapping region, Indicates the overlap size, represents the pixel weight of the patch, represents the reconstructed patch at the corresponding position.

8. The method for constructing an image super-resolution network based on dynamic sparse self-attention according to claim 7, wherein S5 include: S5.1, create image segmentation dataset; Obtain the vector file of the Sentinel-2 super-resolution image and convert the vector file into a mask label. Based on the mask label, obtain the edge label corresponding to the Sentinel-2 super-resolution image to create a Sentinel-2 super-resolution image dataset. S5.2, Construction and training of segmentation network for coastal aquaculture ponds; Add an edge output branch to the end of the fully convolutional symmetric network U-Net to introduce edge supervision. Set the initial learning rate of network training to 0.01 and the loss function to NLL loss. Use the AdamW optimizer and the CosineAnnealing learning rate scheduler to train the network on the Sentinel-2 super-resolution image dataset for 100 rounds to obtain image segmentation weights. S5.3, extraction and restoration of coastal aquaculture ponds; Based on the trained image segmentation weights, coastal aquaculture ponds are extracted from the Sentinel-2 super-resolution image. The initial extraction results of the ponds are restored by combining opening and closing operations, and finally converted into engineering files.

Images