A remote sensing image panchromatic sharpening method based on deep learning attention mechanism

Abstract

The application belongs to the technical field of remote sensing image processing, and discloses a remote sensing image panchromatic sharpening method based on a deep learning attention mechanism, aiming to solve the problems of spectral distortion, spatial blur and difficulty in cross-modal feature alignment of traditional panchromatic sharpening methods, and provide high-precision remote sensing data support for the fields of city planning, environmental monitoring and the like. The method comprises the following steps: constructing a training data set (cutting high-resolution PAN and low-resolution MS image blocks, filtering through satellite-specific MTF, downsampling to obtain input data, and taking the non-downsampled MS block as a label); extracting PAN and LR-MS basic features through a double-branch structure and splicing; collaboratively fusing features through a PanAB module (window, sliding window, longitude and latitude attention); restoring the size through deconvolution upsampling, adjusting the channel number through convolution, and outputting an HR-MS image. The application takes into account the spatial resolution and spectral fidelity, reduces the model overhead, and realizes efficient processing.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a method for panchromatic sharpening of remote sensing images based on a deep learning attention mechanism. Background Technology

[0002] With the development of aerospace remote sensing technology, high-resolution remote sensing imagery has become an important data source for acquiring information about the Earth's surface, and it has significant application value in fields such as urban planning, environmental monitoring, and disaster emergency response. However, the acquisition of high-resolution multispectral (HR-MS) imagery is limited by factors such as sensor R&D costs, imaging conditions, and data storage. In practical applications, it is often necessary to fuse high spatial resolution panchromatic (PAN) imagery with low spatial resolution multispectral (LR-MS) imagery using panchromatic sharpening technology to obtain HR-MS imagery that combines the advantages of both. Existing panchromatic sharpening techniques have limitations: traditional component substitution methods are prone to spectral distortion, multi-resolution analysis methods suffer from local spatial distortion, and model-based methods have high computational complexity; existing deep learning methods face problems such as insufficient feature extraction, cross-modal semantic gaps, and difficulties in multi-scale feature alignment, making it difficult for the generated HR-MS imagery to simultaneously capture spatial detail and spectral consistency, thus limiting the improvement of the application value of remote sensing data. Summary of the Invention

[0003] To address the problems in existing panchromatic sharpening techniques, such as insufficient fusion of spatial details in panchromatic (PAN) images with spectral information from low-resolution multispectral (LR-MS) images, resulting in spectral distortion or spatial blurring, and difficulty in effectively capturing long-distance ground features, this invention discloses a remote sensing image panchromatic sharpening method based on a deep learning attention mechanism. The specific implementation is as follows: A method for panchromatic sharpening of remote sensing images based on a deep learning attention mechanism includes the following steps: (1) Constructing the training dataset: Obtain the original high-resolution panchromatic PAN image and low-resolution multispectral (MS) image, cut them into image blocks of a specified size, and filter and downsample them using the modulation transfer function (MTF) specific to the corresponding satellite to obtain the input data and label data GT; (2) Feature extraction: The PAN image to be processed and the low-resolution multispectral LR-MS image are respectively input into the dual-branch structure, and the basic features are extracted by the convolutional layer and stitched by channel to obtain the initial cross-modal feature map; (3) Feature fusion optimization: The initial cross-modal feature map is input into the PanAB attention module, and the spatial and spectral features are deeply fused through the collaborative processing of window attention (WA), sliding window attention (SWA) and latitude and longitude attention (LLA); (4) Output generation: The feature map size is restored by upsampling through the deconvolution layer, and the number of channels is adjusted by convolution to output high-resolution multispectral HR-MS image.

[0004] Preferably, in step (1), the PAN image is cut into N image blocks of size 256×256, and the MS image is cut into N image blocks of size 4×64×64, where 4 is the number of multispectral bands.

[0005] Preferably, in step (1), the label data GT is the MS image block that has not been downsampled after being cut.

[0006] Preferably, in step (2), the convolutional layer of the dual-branch structure uses a 3×3 convolutional kernel.

[0007] Preferably, in step (3), the window attention WA is implemented by dividing the feature map into multiple non-overlapping windows and performing self-attention calculation independently in each window.

[0008] Preferably, in step (3), the sliding window attention (SWA) breaks the isolation of windows through the sliding window strategy, and realizes feature interaction and fusion between adjacent windows.

[0009] Preferably, in step (3), latitude and longitude attention (LLA) captures long-distance spatial dependencies in the global image by simulating the transmission of latitude and longitude direction information.

[0010] Preferably, in step (2), the convolutional layer with the dual-branch structure is followed by a GroupNorm normalization layer and a SiLU activation function.

[0011] Preferably, in step (4), the upsampling objective of the deconvolution layer is to make the feature map size consistent with the spatial resolution of the input PAN image.

[0012] Preferably, in step (4), the goal of adjusting the number of channels in the convolution is to make the number of channels in the feature map consistent with the number of bands in the input LR-MS image.

[0013] Compared with the prior art, the beneficial effects of the present invention are: This invention provides a panchromatic sharpening method for remote sensing images that balances spatial resolution enhancement with spectral fidelity and sufficient feature fusion. It solves the problems of spectral distortion, spatial blurring, and difficulty in cross-modal feature alignment in traditional methods, and provides high-precision remote sensing data support for fields such as urban planning, environmental monitoring, and disaster emergency response.

[0014] This invention effectively solves the problems of spectral distortion and spatial blurring in existing panchromatic sharpening techniques through multi-attention collaboration; at the same time, the local computational design of window attention reduces model overhead, achieving low-energy and high-efficiency processing, providing high-quality image support for scenarios such as urban planning and disaster monitoring, and is an ideal panchromatic sharpening method for remote sensing images. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the method route of the present invention; Figure 2 This is a schematic diagram of the PanAB module structure; Figure 3 This is a diagram illustrating how attention is calculated. Detailed Implementation

[0016] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. Unless otherwise specified, the raw materials and equipment used can be purchased from the market or are commonly used in the art. The methods in the embodiments, unless otherwise specified, are conventional methods in the art. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0017] First, training data simulation is conducted to construct the dataset required for model training. Taking remote sensing data from satellites such as Gaofen-2 as an example, this process is divided into two steps: data segmentation and data simulation. First, the original high-resolution panchromatic (PAN) image and low-resolution multispectral (MS) image are acquired. The PAN image is segmented into N image blocks of size 256×256, and the MS image is segmented into N image blocks of size 4×64×64 (4 being the number of multispectral bands). Then, the segmented image blocks are filtered using the modulation transfer function (MTF) specific to the corresponding satellite to simulate the blurring effect of real imaging. Subsequently, low-resolution panchromatic image blocks and low-resolution multispectral image blocks are obtained through downsampling. At the same time, the segmented, unsampled MS image blocks are used as label data (GT). This yields the input data and label data required for network training.

[0018] In the model processing stage, the high-resolution panchromatic image and the low-resolution multispectral image to be processed are first input into a dual-branch structure. Basic features of the two image types are extracted through convolutional layers. Then, the basic features output from the dual branches are concatenated by channel to obtain an initial cross-modal feature map, achieving preliminary association between the features of the two image types. The core of this method, the PanAB attention module, performs deep optimization and fusion of the initial cross-modal feature map. It is composed of three parts: Windows Attention (WA), Shift Windows Attention (SWA), and Latitude-Longitude Attention (LLA). This design aims to fully capture and fuse the spatial and spectral information of the images to achieve high-precision panchromatic sharpening.

[0019] The core idea of ​​the window attention module is to divide the feature map into multiple non-overlapping windows. By performing self-attention computation independently within each window, the computational cost is effectively reduced. Specifically, in traditional multi-head self-attention mechanisms, the computational cost increases dramatically with the size of the feature map. The calculation formula is as follows, while window attention optimizes this by dividing the map into windows and then performing self-attention operations within each window. This approach makes the model more efficient in processing local features, enabling it to quickly capture spatial-spectral correlation information within the window, while avoiding the huge computational overhead of global computation of the entire feature map.

[0020] In the window attention formula, q represents the feature map to be processed; k is the feature transformation for generating the attention "query", k is the feature transformation for generating the attention "key", and v is the feature transformation for generating the "value". Representative feature map Up to target location The local window to which (corresponding specific coordinates) belong. This refers to the traversal position within that window. and The transpose of the dot product is then used to generate attention weights via Softmax. Finally, these weights are used to... We obtain the window attention (WA) by performing a weighted summation.

[0021] Sliding window attention is a mechanism that further extends window attention. In window attention, there is a lack of direct information interaction between windows. Sliding window attention, however, introduces a sliding window strategy, enabling feature interaction and fusion between windows. Specifically, in different computational steps, windows slide according to certain rules, breaking down the isolation between windows. This sliding operation allows features from adjacent windows to influence each other, helping the model learn richer contextual information. Especially when dealing with continuous and correlated ground features, it can better capture cross-window spatial-spectral dependencies, thereby improving the model's overall understanding and representation of image features.

[0022] First, the feature map Perform a "Shift" operation to shift the originally non-adjacent windows, allowing the features of different windows to become related; then perform window attention (WA) calculation on the shifted feature map; finally, restore the original position of the feature map by "UnShift", and finally obtain sliding window attention (SWA).

[0023] Latitude and longitude attention takes a global perspective, considering long-range spatial dependencies in imagery. In remote sensing imagery, long-range semantic relationships may exist between different regions. Latitude and longitude attention, through a special calculation method, simulates information transmission along latitude and longitude directions, enabling it to capture these long-range spatial dependencies globally. The specific calculation formula is as follows. It can overcome the limitations of windows and local regions, effectively integrating relevant information from different locations in the image. This allows the model to better maintain spectral consistency and spatial structure integrity when processing scenes with long-range feature dependencies, thus providing a more accurate feature representation for panchromatic sharpening. In WA and SWA, self-attention is densely computed between the current pixel and other pixels within the same window. Although moving the window can expand the receptive field to the entire image, we can use LLA to enhance the network's ability to capture information more effectively from the entire image.

[0024] Feature map Through "Meridian Reshaping" After reorganizing (by vertical columns), focus on calculations within the same "meridian (column)"; this is achieved through "latitudinal reshaping". After (reorganizing by horizontal row), attention is focused only on calculations within the same "latitude line (row)". First, go through... Complete the meridional attention calculation to obtain the intermediate result M; then... After completing the latitudinal attention calculation, the latitude and longitude attention (LLA) is finally obtained.

[0025] After completing the collaborative optimization of the above three attention mechanisms, the feature map after deep fusion is gradually upsampled through deconvolution layers to restore the feature map size, so that it is consistent with the spatial resolution of the input panchromatic image; finally, the number of channels of the feature map is adjusted through convolution to be consistent with the number of bands of the input multispectral image, and finally the high-resolution multispectral (HR-MS) image is output.

[0026] Although embodiments of the present invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to the above embodiments without departing from the principles and spirit of the present invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for panchromatic sharpening of remote sensing images based on deep learning attention mechanism, characterized in that, Includes the following steps: (1) Constructing the training dataset: Obtain the original high-resolution panchromatic PAN image and low-resolution multispectral (MS) image, cut them into image blocks of a specified size, and filter and downsample them using the modulation transfer function (MTF) specific to the corresponding satellite to obtain the input data and label data GT; (2) Feature extraction: The PAN image to be processed and the low-resolution multispectral LR-MS image are respectively input into the dual-branch structure, and the basic features are extracted by the convolutional layer and stitched by channel to obtain the initial cross-modal feature map; (3) Feature fusion optimization: The initial cross-modal feature map is input into the PanAB attention module, and the spatial and spectral features are deeply fused through the collaborative processing of window attention (WA), sliding window attention (SWA) and latitude and longitude attention (LLA); (4) Output generation: The feature map size is restored by upsampling through the deconvolution layer, and the number of channels is adjusted by convolution to output high-resolution multispectral HR-MS image.

2. The method of claim 1, wherein, In step (1), the PAN image is cut into N image blocks of size 256×256, and the MS image is cut into N image blocks of size 4×64×64, where 4 is the number of multispectral bands.

3. The method of claim 1, wherein, In step (1), the label data GT is the MS image block that has not been downsampled after being cut.

4. The method according to claim 1, characterized in that, In step (2), the convolutional layer of the dual-branch structure uses a 3×3 convolutional kernel.

5. The method according to claim 1, characterized in that, In step (3), the window attention WA is implemented by dividing the feature map into multiple non-overlapping windows and performing self-attention calculation independently in each window.

6. The method according to claim 1, characterized in that, In step (3), the sliding window attention (SWA) breaks the isolation of windows through the sliding window strategy, and realizes feature interaction and fusion between adjacent windows.

7. The method according to claim 1, characterized in that, In step (3), the latitude and longitude attention (LLA) captures long-distance spatial dependencies in the global image by simulating the transmission of latitude and longitude information.

8. The method according to claim 1, characterized in that, In step (2), the GroupNorm normalization layer and the SiLU activation function are connected sequentially after the convolutional layer of the dual-branch structure.

9. The method according to claim 1, characterized in that, In step (4), the upsampling objective of the deconvolution layer is to make the feature map size consistent with the spatial resolution of the input PAN image.

10. The method according to claim 1, characterized in that, In step (4), the goal of adjusting the number of channels in the convolution is to make the number of channels in the feature map consistent with the number of bands in the input LR-MS image.

Images