A remote sensing image fusion method based on window cross attention

Abstract

The application provides a remote sensing image fusion method based on window cross attention, a novel remote sensing image fusion network based on window cross attention is used to fuse panchromatic and multispectral images into high-resolution multispectral images. We combine high-pass filtering with deep feature extraction to mine more texture information, overcoming the problem that shallow extraction is insufficient for high-frequency information extraction, and the relationship between multispectral and panchromatic images obtained according to feature similarity is more accurate. Then, we establish the cross-modal relationship between panchromatic and multispectral images through a pixel-level window cross attention mechanism between local windows of multispectral and panchromatic images. Compared with patch-level attention, pixel-level attention is more helpful in preserving fine-grained features. Therefore, more spatial details from the panchromatic image are transferred to the multispectral image, and the fused multispectral image is clearer.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of remote sensing image fusion, and relates to a remote sensing image fusion based on window cross attention, which is suitable for various multispectral and panchromatic image fusion application scenarios. BACKGROUND

[0002] With the rapid development of satellite sensor technology, multispectral images have been widely used in military systems and environmental analysis. However, due to the limitations of satellite sensor technology, only high spatial resolution, low spectral resolution panchromatic (PAN) images or spectral information-rich, low spatial resolution multispectral (MS) images can be captured. In order to generate multispectral images with high spatial resolution, remote sensing image fusion technology for fusing multispectral and panchromatic images has been widely studied.

[0003] Existing remote sensing image fusion techniques can be mainly divided into four categories: component substitution (CS), multi-resolution analysis (MRA), model-based methods and deep learning (DL) based methods. Component substitution is to decompose multispectral images into multiple components, and then replace spatial components with panchromatic images. However, due to incomplete component separation, some spectral information in multispectral images may be lost. Multi-resolution analysis is to inject high-frequency information of panchromatic images into multispectral images in the transform domain. Multi-resolution analysis can better preserve spectral information, but sometimes spatial distortion occurs. Model-based methods establish an optimization model by constructing prior constraints, but the large amount of calculation cost and the difficulty of selecting optimal manual parameters limit their application in practical applications. The mainstream deep learning network at present is still based on convolutional neural network, and this kind of method directly connects multispectral and panchromatic images before inputting them into the network. This strategy cannot fully utilize the cross-modal correlation between multispectral and panchromatic images. In addition, the operation of convolution kernel on all pixel points is the same, which cannot focus on effective features and suppress redundant information. Therefore, it is easy to cause blurring in high-texture remote sensing images. Therefore, how to design an end-to-end deep learning network to explore the cross-modal correlation between panchromatic images and multispectral images, better transfer the spatial texture details of panchromatic images to multispectral images, and obtain multispectral images with rich texture information and as small spectral distortion as possible is an important problem in the field of remote sensing image fusion. SUMMARY

[0004] In order to solve the problems existing in the prior art, the present application provides a remote sensing image fusion method based on window cross attention.

[0005] The present application provides a remote sensing image fusion method based on window cross attention, comprising the following steps:

[0006] Step 1, construct a deep texture feature extraction module based on multispectral image and panchromatic image characteristics to convert the input image to the feature domain;

[0007] Step 2, construct a window cross attention module to obtain the cross-modal fine-grained relationship between the multispectral image and the panchromatic image, and output a feature image;

[0008] Step 3, construct an image decoding module to transmit the generated feature image back to the image domain to obtain the final fused image;

[0009] Step 4, construct a target function to drive the training of the image fusion model, which includes the deep texture feature extraction module, the window cross attention module and the image decoding module;

[0010] Step 5, train the above image fusion model using simulation data, and test the trained model on simulation test set and real test set.

[0011] Further, the specific implementation of step 1 is as follows;

[0012] Step 1.1, construct a high-pass filter to extract the high-frequency information of the input image, which includes multispectral image M, blurred panchromatic image P, panchromatic image P, multispectral image M, blurred panchromatic image P and panchromatic image P after high-pass filter processing respectively G(M), G(P) and G(P);

[0013] Step 1.2, construct a single-channel texture extraction module to extract the high-frequency features of G(P) and G(P), and obtain K and V, the single-channel texture extraction module includes three convolutional layers, the number of convolutional kernels increases layer by layer, and the receptive field of convolutional kernels decreases layer by layer to extract multi-scale detailed information;

[0014] Step 1.3, construct a multi-channel texture extraction module to extract the high-frequency features of G(M) to obtain Q, the multi-channel texture extraction module also includes three convolutional layers, the number of convolutional kernels increases layer by layer, and the convolutional kernels are all 1x1 size.

[0015] Further, in step 1.1, the blurred panchromatic image is obtained by downsampling and then upsampling the original panchromatic image, and the high-pass filter is realized by subtracting the low-frequency content obtained by averaging filtering the original image from the original image. The average filtering is realized by a global pooling layer.

[0016] Further, in step 1.2, the number of convolutional kernels in the three convolutional layers changes from 32, 64 to 128, and the receptive field of the convolutional kernels changes from 7x7, 5x5 to 3x3.

[0017] Further, the specific implementation of step 2 is as follows;

[0018] Step 2.1, input high-frequency features Q / K / V∈R H，W，C Divide into n windows:

[0019] Q = [q] 1 q 2 , ..., q n ]

[0020] K = [k 1 k 2 , ..., k n ]

[0021] V = [v 1 v 2 , ..., v n ]

[0022] Where, q i / k i / v i ∈R h，w，C , C is the number of feature channels, H and w are the image size, and h and w are the window size;

[0023] Step 2.2, in order to extract fine-grained features, each window q is transformed through dimensionality transformation. i v i v i Expand into a pixel sequence, for the m-th pixel in the sequence and the nth pixel Calculate the feature similarity between them:

[0024]

[0025] in, This represents the pixel-level cross-modal correlation within window i;

[0026] Step 2.3: Normalize the correlation between pixels obtained in Step 2.2 using the softmax function.

[0027]

[0028] in, Represents the first color from the panchromatic image The pixel point to the multispectral image Injection gain per pixel;

[0029] Step 2.4, based on the injection gain To extract texture information from the panchromatic image, the m-th pixel of the i-th window in the output feature image is calculated as follows:

[0030]

[0031] Step 2.5, the unfolded pixel sequence is folded into the original pixel window through dimension transformation, to obtain the i-th window of the output image:

[0032]

[0033] Step 2.6, the output feature image of each window is obtained respectively through window cross attention, and finally all the feature images of the windows are spliced to obtain the final output feature image:

[0034] O=[O 1 , O 2 , …, O n ].

[0035] Further, the specific implementation manner of step 3 is as follows:

[0036] Step 3.1, in order to preserve the high-frequency feature information in the multi-spectral image, the output feature image obtained through the window cross attention is added to the multi-spectral feature image Q through a skip connection, to obtain a high-frequency feature image;

[0037] Step 3.2, the obtained high-frequency feature image is fused through a convolution layer to obtain a higher-dimensional multi-channel feature image;

[0038] Step 3.3, four 1x1 convolution layers are used to remap the multi-channel feature image to a four-channel image, to obtain a reconstructed high-frequency image, and then the obtained reconstructed high-frequency image is added to the low-frequency multi-spectral image to obtain the final fusion image.

[0039] Further, the convolution layer in step 3.2 has 256 channels and a kernel size of 3x3.

[0040] Further, in step 3.3, the number of convolution kernels of the four convolution layers is 128, 64, 32 and 4 respectively.

[0041] Further, the loss function constructed in step 4 is as follows:

[0042]

[0043] Wherein, F n and G n represent the fusion image and the reference image respectively, and b is the batch size.

[0044] Further, step 5 also includes comparing the test results with existing algorithms through objective evaluation indexes, and the objective evaluation indexes include the peak signal-to-noise ratio and the no-reference index.

[0045] Compared with the prior art, the application has the advantages and beneficial effects of:

[0046] Firstly, the up-sampled multi-spectral image and the blurred panchromatic image and the original panchromatic image are sent into a high-pass filter, and then are sent into a multi-channel and a single-channel deep feature extraction module respectively, so as to convert the images into a feature domain and extract deep high-frequency features; then the extracted high-frequency features are represented as a query vector Q, a key vector K and a value vector V, and a window cross attention is used to obtain the cross-modal correlation between the multi-spectral image and the panchromatic image; finally, image reconstruction is performed, and the fused high-frequency feature image is converted back to the image domain. Since the high-pass filter is combined with deep feature extraction, more texture information can be mined, so that the relationship between the multi-spectral and panchromatic images obtained according to the feature similarity is more accurate. In addition, a cross-modal relationship is established between the local windows of the multi-spectral and panchromatic images through a pixel-level window cross attention mechanism. Compared with patch-level attention, pixel-level attention helps to preserve fine-grained features and can transfer more spatial details from the panchromatic image to the multi-spectral image. Therefore, the fused multi-spectral image is clearer and the spectral information is well preserved. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is the overall framework diagram of the window cross attention-based remote sensing image fusion network of the embodiment.

[0048] Figure 2 is the network architecture diagram of the window cross attention module of the embodiment. Wherein SM is a Softmax normalization function, and RS is a dimension transformation module.

[0049] Figure 3 is the test result diagram of the simulation data of the embodiment, wherein (a) is a low-resolution multi-spectral image, (b) is the result of IHS, (c) is the result of PNN, (d) is the result of FusionNet, (e) is the result of the method proposed in the application, and (f) is a reference image.

[0050] Figure 4 is the test result diagram of the real data of the embodiment, wherein (a) is a low-resolution multi-spectral image, (b) is the result of IHS, (c) is the result of PNN, (d) is the result of FusionNet, (e) is the result of the method proposed in the application, and (f) is a panchromatic image. DETAILED DESCRIPTION

[0051] In order to facilitate those skilled in the art to understand and implement the present application, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the present application and do not limit the present application.

[0052] This invention primarily addresses the application requirement of acquiring high-resolution multispectral images. We combine high-pass filtering with deep feature extraction to uncover more texture information. Then, we establish a cross-modal relationship between the panchromatic and multispectral images through a pixel-level window cross-attention mechanism between local windows in the multispectral and panchromatic images. This allows us to transfer more texture details from the panchromatic image to the multispectral image, resulting in a multispectral fused image with rich spatial detail and low spectral distortion.

[0053] Appendix Figure 1 This is an overall framework diagram of the remote sensing image fusion network based on window cross-attention in an embodiment, with attached... Figure 2 This is a network architecture diagram of the window cross-attention module in this embodiment. This embodiment provides a remote sensing image fusion method based on window cross-attention to achieve the fusion of multispectral and panchromatic images, specifically including the following steps:

[0054] Step 1: Construct a deep texture feature extraction module based on the characteristics of multispectral and panchromatic images to transform the input image into the feature domain. The specific implementation includes the following sub-steps:

[0055] Step 1.1: Construct a high-pass filter to extract high-frequency information from the input image. The input image includes a multispectral image M and a blurred panchromatic image. The panchromatic image P is obtained by downsampling and then upsampling the original panchromatic image. The high-pass filter is designed by subtracting the low-frequency content obtained by averaging the original image from the original image. The averaging filter is implemented through a global pooling layer. The multispectral image M and the blurred panchromatic image are also considered. After processing the panchromatic image P with a high-pass filter, we obtain G(M) and G(M), respectively. And G(P).

[0056] Step 1.2: Construct a single-channel texture extraction module. The high-frequency features of G(P) are used to obtain K and V. The single-channel texture extraction module consists of three convolutional layers with progressively increasing kernel size to extract contour and high-dimensional features of the image. The image resolution remains constant throughout this process, while the number of kernels varies from 32 and 64 to 128. The receptive field of the kernels decreases progressively from 7×7 and 5×5 to 3×3 to extract multi-scale details. The kernels first cover larger areas of the image to extract more information, then gradually decrease in size to learn deeper details in smaller areas.

[0057] Step 1.3: Construct a multi-channel texture extraction module to extract high-frequency features of G(M) to obtain Q. Similar to Step 1.2, the number of convolutional kernels in the multi-channel texture extraction module increases layer by layer, changing from 32, 64 to 128. All convolutional kernels use a 1×1 size to maintain spatial fidelity and maximize the utilization of spatial information in the multispectral image.

[0058] Step 2: Construct a Window Cross-Attention (WCA) module to obtain fine-grained cross-modal relationships between multispectral and panchromatic images. The specific implementation includes the following sub-steps:

[0059] Step 2.1: Convert the input high-frequency features Q / K / V∈R H，W，C Divide into n windows:

[0060] Q = [q] 1 q 2 , ..., q n ]

[0061] K = [k 1 k 2 , ..., k n ]

[0062] V = [v 1 v 2 , ..., v n ]

[0063] Where, q i / k i / v i ∈R h，w，C , C is the number of feature channels. In this embodiment, H = 256, W = 256, h = 2, w = 2, C = 128, n = 16384. H and W are the image sizes, and h and w are the window sizes, representing dividing the image into 16384 2x2 image blocks.

[0064] Step 2.2: To extract fine-grained features, each window q is transformed using dimensionality transformation (RS). i v i v i Expand into a pixel sequence. For the m-th pixel in the sequence... and the nth pixel In similarity relation calculation (CRM), the feature similarity between them is calculated using inner product operations:

[0065]

[0066] in, This represents the pixel-level cross-modal correlation within window i.

[0067] Step 2.3: Normalize the correlation between pixels obtained in Step 2.2 using the softmax function (SM):

[0068]

[0069] in, Represents the first color from the panchromatic image The pixel point to the multispectral image Injection gain per pixel.

[0070] Step 2.4: Based on the injection gain Texture information can be extracted from a panchromatic image. Therefore, the m-th pixel of the i-th window in the output feature image is calculated as follows:

[0071]

[0072] Step 2.5: Fold the expanded pixel sequence into a pixel window using dimensionality transformation (RS) to obtain the i-th window of the output image:

[0073]

[0074] Step 2.6: Through window cross-attention, obtain the output feature image of each window separately, and finally concatenate the feature images of all windows to obtain the final output feature image:

[0075] O = [O 1 O 2 , ..., O n ]

[0076] Step 3: Construct an image decoding module to transmit the generated feature image back to the image domain. The specific implementation includes the following sub-steps:

[0077] Step 3.1: In order to preserve the high-frequency feature information in the multispectral image, the output feature image obtained by window cross attention is added to the multispectral feature image Q through a skip connection to obtain the high-frequency feature image.

[0078] Step 3.2: The high-frequency feature image obtained by fusion is first passed through a convolution kernel with 256 channels and a size of 3×3 to obtain a higher-dimensional multi-channel feature image.

[0079] Step 3.3: Using four 1×1 convolutional kernels, with the number of kernels in the four layers being 128, 64, 32, and 4 respectively, the multi-channel feature image is remapped into a four-channel image to obtain the reconstructed high-frequency image. Then, the reconstructed high-frequency image is added to the low-frequency multispectral image to obtain the final fused image.

[0080] Step 4: Construct the image fusion model objective function to drive model training. The specific implementation includes the following sub-steps:

[0081] Step 4.1: Construct the loss function. Construct an L2-based loss function:

[0082]

[0083] Among them, F n and G n These represent the fused image and the reference image, respectively, and b is the batch size. In this embodiment, b = 8.

[0084] Step 4.2: Randomly select b data points from the training set and input them into the network to complete one iteration, and adjust the network parameters.

[0085] Step 5: Train the network using simulation data, and test the trained model on both simulation and real test sets, comparing it with other algorithms. The specific implementation includes the following sub-steps:

[0086] Step 5.1: Train the network using simulation data and compare the test results with various comparison methods in terms of visual and objective evaluation metrics. In this embodiment, we mainly use images from the Gaofen-2 satellite. 4000 image pairs are divided into 90% for training and 10% for validation. The reference image is a raw MS image with a resolution of 256×256. Downsampled multispectral and panchromatic images with a factor of 4 are used as input, with an input image size of 256×256. The simulation test image size is 512×512. To verify the effectiveness of the proposed method, we compare it with traditional methods and deep learning-based methods. The traditional method is IHS, and the deep learning methods include PNN and FusionNet. During the training phase, the initial learning rate is 0.001, and the batch size is set to 8. When the peak signal-to-noise ratio (PSNR) of the validation set degrades to 20 epochs, the initial learning rate is decayed by multiplying by 0.5. We use 450 epochs to train the proposed network and optimize it using the Adam optimizer. The visual comparison results are attached. Figure 3 As shown in Table 1, the objective evaluation metric is the peak signal-to-noise ratio (PSNR), and the average results on the simulation test set are shown in Table 1.

[0087] Step 5.2: Test network performance using real data, and compare the test results with various comparison methods using visual and objective evaluation metrics. 210 real images of size 512×512 were selected for testing to verify the performance of the proposed method in the real world. The comparison methods were IHS, PNN, and FusionNet. The visual comparison results are attached. Figure 4As shown in Table 2, the objective evaluation metric is the QNR (Queries No Reference) metric, and the average results on the actual test set are shown in Table 2.

[0088] Table 1 Comparison of average PSNR (dB) for different methods in simulation data (ideal value: +∞)

[0089]

[0090] Table 2 Comparison of average PSNR (dB) for different methods using real data (ideal value: +∞)

[0091]

[0092] As can be seen, our proposed method first extracts high-frequency features, then obtains pixel-level cross-modal correlations between panchromatic and multispectral images through window cross-attention, and finally transfers the texture details of the panchromatic image to the multispectral image, achieving the best effect of remote sensing image fusion.

[0093] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0094] It should be understood that the above description of the embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art can make substitutions or modifications under the guidance of this invention without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A remote sensing image fusion method based on window cross-attention, characterized in that, Includes the following steps: Step 1: Construct a deep texture feature extraction module based on the characteristics of multispectral and panchromatic images to transform the input image into the feature domain; The specific implementation method of step 1 is as follows; Step 1.1: Construct a high-pass filter to extract high-frequency information from the input image, which includes a multispectral image M and a blurred panchromatic image. Panchromatic image P, multispectral image M, and blurred panchromatic image. After processing the panchromatic image P with a high-pass filter, we obtain G(M) and G(M) respectively. ) and G(P); Step 1.2: Construct a single-channel texture extraction module to extract G( The high-frequency features of G(P) are used to obtain K and V. In the single-channel texture extraction module, there are three convolutional layers with the number of convolutional kernels increasing layer by layer and the receptive field of the convolutional kernels decreasing layer by layer, in order to extract multi-scale detail information. Step 1.3: Construct a multi-channel texture extraction module to extract high-frequency features of G(M) to obtain Q. The multi-channel texture extraction module also includes three convolutional layers, with the number of convolutional kernels increasing layer by layer. All convolutional kernels employ... size; Step 2: Construct a window cross-attention module to obtain cross-modal fine-grained relationships between multispectral and panchromatic images, and output feature images; The specific implementation method of step 2 is as follows; Step 2.1, input high-frequency features Divided into One window: in, , , H represents the number of feature channels, H and W represent the image size, and h and w represent the window size. Step 2.2, in order to extract fine-grained features, each window is transformed through dimensionality transformation. , , Expanded into a pixel sequence, for the th pixel in the sequence... 1 pixel and the 1 pixel Calculate the feature similarity between them: in, Represents the window Pixel-level cross-modal correlation within the region; Step 2.3, the correlation between pixels obtained in Step 2.2 is processed through... softmax Normalize the function: in, Represents the first color from the panchromatic image The pixel point to the multispectral image Injection gain per pixel; Step 2.4, based on the injection gain Extracting texture information from the panchromatic image, thus outputting the first feature image. The first window The calculation for each pixel is as follows: Step 2.5: Fold the expanded pixel sequence back into the original pixel window through dimensional transformation to obtain the first pixel window of the output image. One window: Step 2.6: Through window cross-attention, the output feature image of each window is obtained separately. Finally, the feature images of all windows are concatenated to obtain the final output feature image. ； Step 3: Construct an image decoding module to transmit the generated feature image back to the image domain to obtain the final fused image; Step 4: Construct the training of the objective function-driven image fusion model, which includes a deep texture feature extraction module, a window cross attention module, and an image decoding module; Step 5: Train the above image fusion model using simulation data, and test the trained model on both the simulation test set and the real test set.

2. The remote sensing image fusion method based on window cross-attention as described in claim 1, characterized in that: In step 1.1, the blurred panchromatic image is obtained by downsampling and then upsampling the original panchromatic image. The high-pass filter is implemented by subtracting the low-frequency content obtained by averaging the original image from the original image. The averaging filter is implemented through a global pooling layer.

3. The remote sensing image fusion method based on window cross-attention as described in claim 1, characterized in that: In step 1.2, the number of convolutional kernels in the three convolutional layers changes from 32 and 64 to 128, and the receptive field of the convolutional kernels changes from 7×7 and 5×5 to 3×3.

4. The remote sensing image fusion method based on window cross-attention as described in claim 1, characterized in that: The specific implementation method of step 3 is as follows; Step 3.1: To preserve high-frequency feature information in the multispectral image, a skip connection is used to combine the output feature image obtained from window cross-attention with the multispectral feature image. Add them together to obtain a high-frequency feature image; Step 3.2: The high-frequency feature image obtained by fusion is passed through a convolutional layer to obtain a higher-dimensional multi-channel feature image; Step 3.3, using 4 sizes of The convolutional layer remaps the multi-channel feature image to a four-channel image, resulting in a reconstructed high-frequency image. Then, the reconstructed high-frequency image is added to the low-frequency multispectral image to obtain the final fused image.

5. The remote sensing image fusion method based on window cross-attention as described in claim 4, characterized in that: The channels of the convolutional layer in step 3.2 are The kernel size is .

6. The remote sensing image fusion method based on window cross-attention as described in claim 4, characterized in that: In step 3.3, the number of convolutional kernels in the four convolutional layers are 128, 64, 32, and 4, respectively.

7. The remote sensing image fusion method based on window cross-attention as described in claim 1, characterized in that: The loss function constructed in step 4 is as follows; in, and These represent the fused image and the reference image, respectively. It refers to the batch size.

8. The remote sensing image fusion method based on window cross-attention as described in claim 1, characterized in that: Step 5 also includes comparing the test results with existing algorithms using objective evaluation metrics, including peak signal-to-noise ratio and no-reference metrics.

Images