A Panchromatic Sharpening Method Based on Complementary Fusion Network
By building a complementary fusion network model, the problems of high-frequency edges and details loss and insufficient spectral maintenance in the full-color sharpening method are solved, and efficient spatial detail enhancement and spectral fidelity improvement are achieved.
Patent Information
- Application Number
- CN202310186246.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-01
- Publication Date
- 2025-08-01
- Estimated Expiration
- 2043-03-01
AI Technical Summary
There are problems in existing full-color sharpening methods such as high-frequency edge and detail loss and insufficient spectral retention capabilities, especially in deep learning-based methods, which are difficult to effectively fuse complementary information of multi-spectral and full-color images in the resulting images.
A complementary fusion network model is built, including local branches and global branches, and local information is extracted using convolutional layers and deformable cross attention modules, and complementary information is enhanced through the fusion enhancement module, weakening redundant information, and finally generating full-color sharpening results through the reconstruction module.
While enhancing spatial detail information, it reduces spectral distortion, improves the quality of full-color sharpened images, and improves spatial resolution and spectral fidelity.
Smart Images
Figure CN116228577B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and particularly relates to a pan-sharpening method. Background Art
[0002] At present, most optical Earth observation satellites, such as GaoFen-2, QuickBird, WorldView-2, etc., can simultaneously capture remote sensing images with different characteristics through two sensors, namely, multi-spectral (MS) images with high spectral resolution and low spatial resolution, and panchromatic (PAN) images with high spatial resolution and low spectral resolution, providing sufficient data sources for pan-sharpening technology. Pansharpening is a technology for fusing multi-spectral and panchromatic images. Its essence is to use the fine spatial detail information in the panchromatic image to sharpen the multi-spectral image, so as to obtain a multi-spectral image with high spatial resolution. This technology has a wide range of applications, such as being used in tasks such as change detection, target detection, and land classification. And in the software ENVI, this technology is used to enhance the image to improve the image readability. Therefore, it not only attracts a great deal of attention in the scientific community, but also is sought after by the business community and plays an important role in real life.
[0003] After more than forty years of development, the performance of pan-sharpening methods has been continuously improved. In particular, deep learning-based methods perform better than traditional methods. Among them, methods based on convolutional neural networks have attracted more attention. However, since neural networks inherently tend to fit low-frequency components with high priority, it inevitably leads to the loss of some high-frequency edges and details in the resulting image. In addition, many methods do not take targeted measures to solve the problem of the lack of network spectral preservation ability. Summary of the Invention
[0004] In order to overcome the deficiencies of the prior art, the present invention provides a pan-sharpening method based on a complementary fusion network. The pan-sharpening problem is processed by constructing a complementary fusion network model, which includes two branches of local and global, and two modules of fusion enhancement and reconstruction; the local branch is mainly composed of convolutional layers to fully utilize its local receptive field to extract local information; the global branch uses deformable cross-attention for global-guided fusion in the wavelet domain of the spatial-spectral combination; three initial fusion features can be obtained through the two branches, and the fusion enhancement module takes them as inputs, enhances the complementary parts and weakens the redundant parts through spatial and channel attention, and forms the final fusion feature after integration; finally, the pan-sharpening result image is obtained through the reconstruction module. The present invention can make full use of the rich complementary and redundant information contained in the multi-spectral image and the panchromatic image, so as to reduce spectral distortion while enhancing spatial detail information.
[0005] The technical solutions adopted by the present invention to solve its technical problems include the following steps:
[0006] Step 1: Dataset Preparation;
[0007] Cut image patches from paired and registered large-scale remote sensing MS images and PAN images in the order from left to right and from top to bottom, and divide these image patches into a training set, a validation set, and a test set; first perform normalization processing on the training set, the validation set, and the test set; then process the image patches in the training set, the validation set, and the test set according to the Wald protocol, and then use the processed image patches as the input of the model; the original MS image patches that have not been processed by the Wald protocol are used as reference images;
[0008] Step 2: Construction of Complementary Fusion Network Model;
[0009] The complementary fusion network model includes four parts: a local branch, a global branch, a fusion enhancement module, and a reconstruction module;
[0010] The construction process of each specific part is as follows:
[0011] Step 2-1: Construction of the local branch;
[0012] The local branch consists of a convolutional layer, a skip connection, and a fusion module;
[0013] The window size of the convolutional layer is 3×3, the padding is 1, and the stride is 1; the convolutional layer and other layers are used to form CB1 convolutional blocks and CB2 convolutional blocks; the CB1 convolutional block is a structure in which a convolutional layer, a batch normalization BN layer, an LReLU activation function, a convolutional layer, a BN layer, and an LReLU activation function are cascaded in sequence; the CB2 convolutional block refers to a residual structure in which the initial input of the CB1 convolutional block and the output of the second BN layer are added and sent to the last LReLU activation function; the skip connection refers to the connection operation in which the outputs of two adjacent modules are simultaneously input into another module; the fusion module includes an operation of splicing the inputs according to the channel dimension and a 1×1 convolutional layer;
[0014] The panchromatic image P and the upsampled multispectral image After being spliced in the channel dimension, they are used as the input of the local branch. The input first passes through a 3×3 convolutional layer to obtain the initial shallow feature F1, and then passes through 3 cascaded CB2 convolutional blocks to obtain three features F2, F3, and F4 from shallow to deep respectively; next, the obtained features are sent into 3 skip connection fusion modules, F1 and F2 are sent into a fusion module to obtain F r1 , F3 and F4 are sent into a fusion module to obtain F r2 , the obtained F r1 and F r2 are sent into a fusion module to obtain the final output F local ;
[0015] Step 2-2: Construct the global branch network;
[0016] The global branch includes two branches, namely the MS branch and the PAN branch, and their inputs are respectively and P;
[0017] The MS branch and the PAN branch have the same structure, and both include their respective one frequency-aware block FAB and a shared guiding fusion module GFM; the frequency-aware block is coupled by a DWT layer and two parallel convolutional branches; the convolutional branch is formed by cascading a CB1 convolutional block and a CB2 convolutional block; the guiding fusion module consists of two deformable cross-attention DCA modules and two corresponding IDWT layers;
[0018] and P pass through their respective frequency-aware blocks to obtain their respective high- and low-frequency perception features and Then, respectively, and are transformed to obtain Q1, V1 and Q2, V2; the transformation refers to the operation of merging the last two dimensions of the fourth-order tensor feature into one dimension and exchanging it with the original second dimension. In particular it is also first processed by a 1×1 convolutional layer to make its channel dimension consistent with ; Q1, V2 and Q2, V1 obtained are respectively fed into a deformable cross-attention module to obtain the required spectral feature F spec and the required spatial feature F spat ; finally, and F spec are fed into an IDWT layer to obtain the fused feature F global1 , and F passed through the 1×1 convolutional layer spat are fed into another IDWT layer to obtain the fused feature F global2 ;
[0019] Step 2-3: Fusion enhancement module;
[0020] The fusion enhancement module includes three channel spatial attention CSA modules and a fusion module;
[0021] The channel spatial attention module includes a cascaded channel attention module and a spatial attention module; the channel attention module means that the input is respectively processed by a channel average pooling, a channel max pooling, then processed by a CB3 convolution block, the resulting matrices are added together, passed through a sigmoid function to obtain a weight coefficient, and finally the obtained weight coefficient is multiplied by the input matrix; the CB3 convolution block refers to a structure in which a 1×1 convolutional layer, a ReLU activation function, and a 1×1 convolutional layer are cascaded in sequence; the result of the channel attention module is fed into the spatial attention module; the spatial attention module means that the input is respectively processed by a spatial average pooling, a spatial max pooling, the resulting results are concatenated in the channel dimension and then fed into a 7×7 convolution, passed through a sigmoid function to obtain a weight coefficient, and finally the obtained weight coefficient is multiplied by the input matrix; the fusion module includes an operation of concatenating the input in the channel dimension and a 1×1 convolutional layer;
[0022] The preliminary fusion feature F obtained by the local and global branch networks local and F global1 , G global2 respectively pass through a spatial + channel attention module of each, and the obtained outputs are directly fed into a fusion module for integration to obtain the final fusion feature F fusion ;
[0023] Step 2 - 4: Reconstruction module;
[0024] The reconstruction module is composed of a cascaded CB2 convolution block and a CB1 convolution block;
[0025] The final fusion feature F fusion passes through the reconstruction module to obtain the panchromatic sharpening result;
[0026] Step 3: Loss function;
[0027] The loss function includes KL divergence loss and mean absolute error MAE loss;
[0028] The KL divergence loss is defined as follows:
[0029]
[0030] Among them, for two input features F in1 and F in2 are processed using the softmax function to ensure that the sum of all values of the two probability distributions is 1; this loss function restricts the distribution of F in1 to be close to the distribution of F in2 ; F in1 (i, j) and F in2 (i, j) respectively represent the pixel values of the features F in1 and F in2 ;
[0031] The MAE loss is defined as follows:
[0032]
[0033] where B represents the number of image channels, and R b represents the channels of the b-th reference image, represents the channels of the b-th result image, and ||*|| represents the L1 norm calculation operation;
[0034] The overall loss function is expressed as:
[0035]
[0036] where R(*) represents the operation of merging the last two dimensions of the fourth-order tensor feature, and α represents the balance parameter;
[0037] Step 4: Adopt the loss function Use the training set to train the network model. During the training process, use the validation set to obtain the trained network model, and finally use the test set to test the effectiveness of the model.
[0038] Preferably, the normalization process means dividing the pixel values of the processed image by 2047.0.
[0039] Preferably, the Wald protocol processing means first filtering the original MS and PAN images using a Gaussian smoothing kernel of size 5×5, and then downsampling to 1 / 4 of the original spatial resolution.
[0040] Preferably, the upsampled multispectral image means bicubic interpolation upsampling the MS image patches to the spatial resolution of the PAN image patches.
[0041] Preferably, α is set to 1×10 5 .
[0042] Preferably, the sampling factor during downsampling in Step 1 is 4.
[0043] Preferably, the PAN image patch sizes of the training set and the validation set are 256×256, and the MS image patch sizes are 64×64×4; the PAN image patch size of the test set is 1024×1024, and the MS image patch size is 256×256×4; the data volume ratio of the training set, the validation set, and the test set is 8:1:1.
[0044] The beneficial effects of the present invention are as follows:
[0045] 1) The model of the present invention is carefully designed to include two branches, namely, a local branch and a global branch, which complement each other. Among them, the global branch breaks the limitation of only considering the spatial domain in the past. In the wavelet domain, a cross-attention module is used to model long-range dependencies, and the high-frequency and low-frequency information required for the low-frequency and high-frequency parts is found respectively. The local branch adopts a coupled structure of pure convolution and extracts features containing local information by using the local receptive field. In addition, spatial and channel attention can enhance important information in the features and suppress unimportant information. This module integrates the preliminary fusion features obtained from the local branch and the global branch to enhance complementary information and reduce redundant information.
[0046] 2) To reduce the spectral distortion degree of the fused image, the KL divergence loss function is introduced to narrow the correlation between the MS and PAN features, thereby providing support for guiding the fusion module.
[0047] 3) For enhancing spatial detail information, not only skip connections are introduced in the design of the local branch to make full use of the feature information at each stage, but also in the global branch, frequency-aware features are adaptively learned through the DWT and IDWT layers, and high-frequency and low-frequency information is used in an explicit manner to promote the learning of high-frequency details. Brief Description of the Drawings
[0048] Figure 1 is the flowchart of the method of the present invention.
[0049] Figure 2 is the structural diagram of the network model constructed by the method of the present invention.
[0050] Figure 3 is the structural diagram of the channel spatial attention module of the present invention.
[0051] Figure 4 is the comparison chart of the fusion results of the method of the present invention and different classical methods in the embodiment. Detailed Embodiment
[0052] The present invention will be further described below in conjunction with the drawings and embodiments.
[0053] The present invention provides a pan-sharpening method based on a complementary fusion network. By constructing a complementary fusion network model to extract global and local information respectively and then reasonably fuse them, the pan-sharpening problem is processed from a new perspective. The model includes a local branch, a global branch, a fusion enhancement module, and a reconstruction module. Among them, the local branch is mainly composed of convolutional layers to make full use of its local receptive field to extract local information; the global branch uses deformable cross-attention for global-guided fusion in the wavelet domain of the spatial-spectral combination; three initial fusion features can be obtained through the two branches, and the fusion enhancement module takes them as inputs, enhances the complementary parts and weakens the redundant parts through spatial and channel attention, and forms the final fusion feature after integrating the results; finally, the pan-sharpened result image is obtained through the reconstruction module. To make the guided fusion module in the global branch play a greater role, the KL divergence loss function is introduced to limit the difference degree of the corresponding frequency domain features. The design of the complementary branch and the enhancement module combines the advantages of multiple attention mechanisms and discrete wavelet transform (DWT) layers and their inverse transform (IDWT) layers, which can further improve the performance of the fusion method.
[0054] It includes the following steps:
[0055] Step 1: Dataset preparation;
[0056] The data comes from QuickBird (QB) and WorldView-2 (WV-2) satellite sensors. For QB data, the spatial resolution of PAN is 0.6 meters, the spatial resolution of MS is 2.4 meters, and MS includes 4 spectral bands: blue, green, red, and near-infrared bands. For WV-2 data, the spatial resolution of PAN is 0.5 meters, the spatial resolution of MS is 2 meters, and MS includes 8 spectral bands: cyan, yellow, infrared, blue, green, red, near-infrared 1 band, and near-infrared 2 band. The spatial resolution ratio between MS and PAN images in both datasets is 4.
[0057] The large-scale paired and registered MS images and PAN images are intercepted at corresponding positions without overlap in the order from left to right and from top to bottom. The intercepted image patches are divided into three parts, which are used as the training set, validation set, and test set respectively; the size of the image patches in the training set and validation set is 256×256 (PAN) / 64×64×4 (MS), and the size of the image patches in the test set is 1024×1024 (PAN) / 256×256×4 (MS). The ratio of the training, validation, and test data volumes is 8∶1∶1. Since there is no reference image, the MS image patches and PAN image patches in the training set, validation set, and test set are processed according to the Wald protocol, and then the processed images are used as the inputs of the network, and the original MS images are used as the reference images.
[0058] Step 2: Construction of the complementary fusion network model;
[0059] The network model is divided into four parts: two branches, namely the local branch and the global branch, and two modules, namely the fusion enhancement module and the reconstruction module.
[0060] The construction process of each specific part is as follows:
[0061] Step 2-1: Construct the local branch network;
[0062] The local branch mainly consists of a convolutional layer, skip connections, and a fusion module. The window size of the convolutional layer is 3×3, the padding is 1, and the stride is 1; the convolutional layer is mostly used in a convolutional block; the convolutional block is divided into two types: CB1 convolutional block and CB2 convolutional block; the CB1 convolutional block refers to a structure in which a convolutional layer, a batch normalization (BN) layer, a Leaky Rectified Linear Unit (LReLU) activation function, a convolutional layer, a BN layer, and an LReLU activation function are cascaded in sequence; the CB2 convolutional block refers to a residual structure in which the initial input and the output of the second BN layer are added and sent to the last LReLU activation function in the structure of the CB1 convolutional block. The skip connection means that the outputs of two adjacent modules are simultaneously input into another module; the fusion module includes an operation of concatenating the inputs along the channel dimension and a 1×1 convolutional layer.
[0063] The panchromatic image P and the upsampled multispectral image After being concatenated along the channel dimension, they are used as the input of the local branch. The input first passes through a 3×3 convolutional layer to obtain the initial shallow feature F1, and then passes through 3 cascaded CB2 convolutional blocks to obtain three features F2, F3, and F4 from shallow to deep respectively; next, the obtained features are sent into 3 skip connection fusion modules, F1 and F2 are sent into a fusion module to obtain F r1 , F3 and F4 are sent into a fusion module to obtain F r2 , the obtained F r1 and F r2 are sent into a fusion module to obtain the final output F local .
[0064] The upsampled multispectral image refers to upsampling the MS image block by bicubic interpolation.
[0065] Step 2-2: Construct the global branch network;
[0066] The global branch also includes two branches, namely the MS branch and the PAN branch, and their inputs are respectively With P; the MS branch and the PAN branch have the same structure, both containing a respective Frequency Awareness Block (FAB) and a shared Guided Fusion Module (GFM); the Frequency Awareness Block is coupled by a DWT layer and two parallel convolutional branches; the convolutional branch contains a cascaded CB1 convolutional block and a CB2 convolutional block; the Guided Fusion Module consists of two Deformable Cross-Attention (DCA) modules and two corresponding IDWT layers.
[0067] P and the other pass through their respective Frequency Awareness Blocks to obtain their respective high- and low-frequency perception features and Then, respectively, and are transformed to obtain Q1, V2 and Q2, V1; the transformation refers to the operation of merging the last two dimensions of the fourth-order tensor feature into one dimension and exchanging it with the original second dimension; in particular, should also be processed first through a 1×1 convolutional layer to make its channel dimension consistent with ; the obtained Q1, V1 and Q2, V2 are respectively fed into a Deformable Cross-Attention module to obtain the required spectral feature F spec and the required spatial feature F spat ; finally, and F spec are fed into an IDWT layer to obtain the fused feature F global1 , [[ID=3i]] and F passed through the 1×1 convolutional layer spat are fed into another IDWT layer to obtain the fused feature F global2 .
[0068] Step 2-3: Fusion Enhancement Module;
[0069] The Fusion Enhancement Module is also composed of two types of modules, namely three Channel Spatial Attention (CSA) modules and one fusion module.
[0070] The channel-spatial attention module is composed of a cascaded channel attention module and a spatial attention module. The channel attention module means that the input is respectively passed through channel average pooling and channel max pooling, then respectively processed by their own CB3 convolutional blocks, the resulting matrices are added, and then passed through the sigmoid function to obtain the weight coefficient. Finally, the obtained weight coefficient is multiplied by the input matrix. The CB3 convolutional block is composed of a 1×1 convolutional layer, a ReLU activation function, and a 1×1 convolutional layer in cascade. The result of the channel attention module is fed into the spatial attention module. The spatial attention module means that the input is respectively passed through spatial average pooling and spatial max pooling, the resulting results are concatenated in the channel dimension and then fed into a 7×7 convolution, and then passed through the sigmoid function to obtain the weight coefficient. Finally, the obtained weight coefficient is multiplied by the input matrix. The fusion module includes an operation of concatenating the input along the channel dimension and a 1×1 convolutional layer.
[0071] The preliminary fusion feature F obtained by the local and global branch networks local and F global1 , F global2 respectively pass through their own one channel-spatial attention module, and the obtained outputs are directly fed into a fusion module for integration to obtain the final fusion feature F fusion .
[0072] Step 2-4: Reconstruction module;
[0073] The reconstruction module is composed of a cascaded CB2 convolutional block and a CB1 convolutional block.
[0074] The final fusion feature F fusion passes through the reconstruction module to obtain the panchromatic sharpening result.
[0075] Step 3: Loss function
[0076] The loss function includes KL divergence loss and mean absolute error (MAE) loss;
[0077] The KL divergence loss is defined as follows:
[0078]
[0079] Among them, for the two input features F in1 and F in2 first use the softmax function to process to ensure that the sum of all values of the two probability distributions is 1. This loss function restricts the distribution of F in1 to approach the distribution of F in2 ; F in1 (i, j) and F in2 respectively represent the pixel values of the features F in1 and F in2 ;
[0080] The MAE loss is defined as follows:
[0081]
[0082] where B represents the total number of image channels, R b represents the b-th channel of the reference image, represents the b-th channel of the result image, and ||*|| represents the L1 norm calculation operation.
[0083] The overall loss function is expressed as:
[0084]
[0085] where R(*) represents the operation of merging the last two dimensions of the fourth-order tensor feature, α represents the balance parameter, and is set to 1×10 according to the magnitudes of the MAE and KL loss values 5 .
[0086] Step 4: Adopt the loss function Use the training set to train the network model. During the training process, use the validation set to observe the model performance and obtain the best-performing network model. Finally, use the test set to test the effectiveness of the model. Specific embodiments:
[0088] (1) Dataset preparation:
[0089] Use panchromatic images and multispectral images with a width-to-height ratio of 4:1 and registered with each other, and perform the following processing in PyCharm:
[0090] ① Read the images in mat data format, divide the original images into two parts, which are used as training data images and test data images respectively. The division principle is that the widths of the two parts are the same and the height ratio is 9:1. Do this for both PAN and MS;
[0091] ② For the training data part, intercept the corresponding image patches of the matching PAN and MS training images from left to right and top to bottom. The size of the PAN image patch is 256×256, and the size of the MS image patch is 64×64×4 (4 is the number of channels, when the number of MS channels is 8, it can be changed accordingly to 8). The test data part is constructed in a similar way, where the size of the PAN image patch is 1024×1024, and the size of the MS image patch is 256×256×4.
[0092] ③ Randomly divide 1 / 9 of the training data part as the validation set data.
[0093] So far, the training set, validation set, and test set data have been obtained and saved in the.mat format. For QB, the training set contains 6,943 pairs of images, the validation set contains 743 pairs of images, and the test set contains 156 pairs of images; for WV-2, the training set contains 9,641 pairs of images, the validation set contains 945 pairs of images, and the test set contains 136 pairs of images.
[0094] ④ When processing according to the Wald protocol, after Gaussian blurring the PAN image and the MS image using the cv2.GaussianBlur() function with a Gaussian kernel of 5×5 and a standard deviation of 2, the images are then downsampled by a factor of 4 in the "cv2.INTER_AREA" manner using the cv2.resize() function to form new images for the training set. The same operation is performed on the validation set and the test set.
[0095] So far, the data set preparation steps are completed.
[0096] (2) Construction of the complementary fusion network model
[0097] The network structure diagram is shown in Figure 2 , and the important parameters for constructing the network include:
[0098] ① The convolutional layers used in the entire network: a 1×1 convolutional layer with padding 0 and stride 1; a 3×3 convolutional layer with padding 1 and stride 1; a 7×7 convolutional layer with padding 3 and stride 1.
[0099] The output features of all convolutional layers in the local branch contain 48 channels. In the global branch, without special transformation and explanation, the output features of all convolutional layers related to the low-frequency perception features contain 48 channels; the output features of all convolutional layers related to the high-frequency perception features contain 3×48 channels. The output features of the convolutional layer in the fusion module contain 48 channels. In the convolutional layer of the reconstruction module, except that the output channels of the last layer are the number of bands of the multispectral image, other output features contain 48 channels.
[0100] The structure of the deformable cross-attention module uses the structure in the literature "Deformable detr: Deformable transformers for end-to-end object detection". Preferably, the number of heads is 8 and the number of points is 4.
[0101] The structure of the channel spatial attention module is shown in Figure 3 .
[0102] Unless otherwise specified, the activation function of other layers uses torch.nn.leakyReLU(0.2), and the activation function of the last layer uses torch.nn.Tanh(); the BN layer uses torch.nn.BatchNorm2d().
[0103] ②Processing of high frequencies by DWT layer and IDWT layer: After an image or feature passes through the DWT layer, a low-frequency part and three high-frequency parts are obtained. For convenience of processing, the three high-frequency parts are concatenated in the channel dimension in order; in the input part of the IDWT layer, a low-frequency part and three high-frequency parts are still required. At this time, the fused high frequency will be decomposed into three parts in the channel dimension in order as the three high-frequency inputs of the inverse wavelet layer.
[0104] (3) Network training
[0105] (4) ① Input image: Panchromatic image P, with a size of 64×64 (height × width) and the upsampled multispectral image
[0106] with a size of 64×64×4 (height × width × number of channels). Here, the upsampled multispectral is obtained by upsampling the multispectral image with a size of 16×16×4 by 4 times using the torch.nn.functional.interpolate() function in the "bicubic"
[0107] manner. The input image to the network is normalized, which means dividing the pixel values of the input image by 2047.0.
[0108] ② Other relevant settings: Use the Adam optimizer to update the parameters under the PyTorch framework. The number of training epochs is set to 2000, the batch size is set to 32, and the initial learning rate is set to 0.0001. The "haar" wavelet is used. The network effect is tested using the validation set at the end of each epoch, and the network parameters with the best effect are saved.
[0109] ③ Stopping condition for training: The loss function of the network reaches the convergence state.
[0110] (5) Network testing
[0111] ① Input image: Panchromatic image P, with a size of 256×256 (height × width) and the upsampled multispectral image with a size of 256×256×4 (height × width × number of channels). Here, the upsampled multispectral is obtained by upsampling the multispectral image with a size of 64×64×4 by 4 times using the torch.nn.functional.interpolate() function in the "bicubic" manner.
[0112] ②Load the network parameters with the best performance saved during the training phase or the network parameters saved in the last epoch, and test the input images.
[0113] ③Network output result: The final output result is visualized in MATLAB and displayed after data normalization. For multi-channel images, only the visualization results of the red, green, and blue bands are shown. As Figure 3 shown, the visualization results of a pair of images in the QB dataset are presented. From left to right and top to bottom, they are the PAN image, the upsampled MS image, the reference image, the visualization result of the comparison method TANI (Pansharpening via Triplet Attention Network With Information Interaction in the literature), the visualization result of the comparison method NLRNet (NLRNet: An efficient nonlocal attention ResNet for pansharpening in the literature), and the visualization result image of CFNet (the method of this embodiment). It can be seen that the method of this embodiment is superior to the comparison methods in terms of spatial detail enhancement and spectral information preservation.
Claims
1. A panchromatic sharpening method based on a complementary fusion network, characterized in that It includes the following steps: Step 1: Dataset preparation; Intercept image patches from paired and registered large-scale remote sensing images, namely MS images and PAN images, in the order from left to right and from top to bottom, and divide these image patches into a training set, a validation set, and a test set; first, perform normalization processing on the training set, the validation set, and the test set; then process the image patches in the training set, the validation set, and the test set according to the Wald protocol, and then use the processed image patches as the input of the model; the original MS image patches without being processed by the Wald protocol are used as reference images; Step 2: Construction of the complementary fusion network model; The complementary fusion network model includes four parts: a local branch, a global branch, a fusion enhancement module, and a reconstruction module; The specific construction process of each part is as follows: Step 2-1: Construction of the local branch; The local branch consists of a convolutional layer, skip connections, and a fusion module; The window size of the convolutional layer is 3×3, the padding is 1, and the stride is 1; the CB1 convolutional block and the CB2 convolutional block are composed of a convolutional layer and other layers; the CB1 convolutional block is a sequential cascade structure of a convolutional layer, a batch normalization (BN) layer, a Leaky ReLU (LReLU) activation function, a convolutional layer, a BN layer, and an LReLU activation function; the CB2 convolutional block refers to a residual structure in which the initial input of the CB1 convolutional block and the output of the second BN layer are added and then fed into the last LReLU activation function; the skip connection refers to the connection operation where the outputs of two adjacent modules are simultaneously input into another module; the fusion module includes an operation of concatenating the inputs along the channel dimension and a 1×1 convolutional layer; The panchromatic image P and the upsampled multispectral image are concatenated in the channel dimension and used as the input of the local branch. The input first passes through a 3×3 convolutional layer to obtain the initial shallow feature F1, and then passes through 3 cascaded CB2 convolutional blocks to obtain three features F2, F3, and F4 from shallow to deep respectively. Next, the obtained features are fed into 3 skip connection fusion modules. F1 and F2 are fed into a fusion module to obtain F r1 , F3 and F4 are fed into a fusion module to obtain F r2 . The obtained F r1 and F r2 are fed into a fusion module to obtain the final output F local ; Step 2-2: Construction of the global branch network; The global branch contains two branches, namely the MS branch and the PAN branch, and their inputs are respectively and P; The MS branch and the PAN branch have the same structure, both including one frequency-aware block (FAB) of their own and one shared guided fusion module (GFM); the frequency-aware block is coupled by a discrete wavelet transform (DWT) layer and two parallel convolutional branches; the convolutional branch is composed of a CB1 convolutional block and a CB2 convolutional block in cascade; the guided fusion module consists of two deformable cross-attention (DCA) modules and two corresponding inverse discrete wavelet transform (IDWT) layers; After passing through their respective frequency perception blocks, P and [object] respectively obtain their high- and low-frequency perception features and Then, respectively and are transformed to obtain Q1, V1 and Q2, V2; the transformation refers to the operation of combining the last two dimensions of the fourth-order tensor feature into one dimension and exchanging it with the original second dimension. In particular is also first processed through a 1×1 convolutional layer to make its channel dimension consistent with ; the obtained Q1, V2 and Q2, V1 are respectively fed into a deformable cross-attention module to obtain the required spectral feature F spec and the required spatial feature F spat ; finally, and F spec are fed into an IDWT layer to obtain the fused feature F global1 , and F passed through the 1×1 convolutional layer spat are fed into another IDWT layer to obtain the fused feature F global2 ; Step 2-3: Fusion enhancement module; The fusion enhancement module includes three channel-spatial attention (CSA) modules and one fusion module; The channel-spatial attention module includes a cascaded channel attention module and a spatial attention module; the channel attention module means that the input is respectively passed through channel average pooling and channel max pooling, then processed by the CB3 convolutional block respectively, the resulting matrices are added, and then passed through the sigmoid function to obtain the weight coefficient, and finally the obtained weight coefficient is multiplied by the input matrix; the CB3 convolutional block is a sequential cascade structure of a 1×1 convolutional layer, a ReLU activation function, and a 1×1 convolutional layer; the result of the channel attention module is fed into the spatial attention module; the spatial attention module means that the input is respectively passed through spatial average pooling and spatial max pooling, the resulting results are concatenated in the channel dimension and then fed into a 7×7 convolution, and then passed through the sigmoid function to obtain the weight coefficient, and finally the obtained weight coefficient is multiplied by the input matrix; The fusion module includes an operation of concatenating the inputs along the channel dimension and a 1×1 convolutional layer; The initially fused feature F obtained from the local and global branch networks local and F global1 , F global2 respectively pass through a spatial + channel attention module of each, and the obtained outputs are directly fed into a fusion module for integration to obtain the final fused feature F fusion ; Step 2-4: Reconstruction module; The reconstruction module is formed by cascading a CB2 convolutional block and a CB1 convolutional block; The final fused feature F fusion is passed through a reconstruction module to obtain the pan-sharpened result; Step 3: Loss function; The loss function includes KL divergence loss and mean absolute error (MAE) loss; The KL divergence loss is defined as follows: Among them, for the two input features F in1 and F in2 are processed using the softmax function to ensure that the sum of all values of the two probability distributions is 1; this loss function restricts the distribution of F in1 to approach the distribution of F in2 ; F in1 (i, j) and F in2 (i, j) respectively represent the pixel values of the features F in1 and F in2 ; The MAE loss is defined as follows: where B represents the number of image channels, and R b represents the channel of the b-th reference image, represents the channel of the b-th result image, and ||*|| represents the calculation operation of the first norm; The overall loss function is expressed as: where, R(*) represents an operation of combining the last two dimensions of the fourth-order tensor feature, and α represents a balance parameter; Step 4: Adopt a loss function Use the training set to train the network model. During the training process, use the validation set to obtain the trained network model. Finally, use the test set to test the effectiveness of the model.
2. The panchromatic sharpening method based on a complementary fusion network according to claim 1, wherein The normalization process means that the pixel values of the processed image are divided by 2047.
0.
3. A panchromatic sharpening method based on a complementary fusion network according to claim 1, characterized in that The Wald protocol processing means that first, the original MS and PAN images are filtered using a Gaussian smoothing kernel of size 5×5, and then downsampled to 1 / 4 of the original spatial resolution.
4. A panchromatic sharpening method based on a complementary fusion network according to claim 1, characterized in that The upsampled multi-spectral image refers to upsampling the MS image patches to the spatial resolution of the PAN image patches by bicubic interpolation.
5. A panchromatic sharpening method based on a complementary fusion network according to claim 1, characterized in that, Let α be set to 1×10 5 .
6. The panchromatic sharpening method based on a complementary fusion network according to claim 1, wherein In Step 1, the downsampling factor is 4 during downsampling.
7. A panchromatic sharpening method based on a complementary fusion network according to claim 1, characterized in that The size of the PAN image patches in the training set and the validation set is 256×256, and the size of the MS image patches is 64×64×4; the size of the PAN image patches in the test set is 1024×1024, and the size of the MS image patches is 256×256×4; the ratio of the data volumes of the training set, the validation set, and the test set is 8∶1∶1.