A remote sensing image fusion method and system based on fusion correction

Abstract

The application provides a remote sensing image fusion method and system based on fusion correction. In the fusion stage, multiscale network is used to fuse multispectral images and panchromatic images to utilize the complementarity of different scale features, provide more comprehensive and rich information, then a feature preservation module constructed by a channel multi-head attention selectively focuses on features more useful for the fusion task, reduces the influence of irrelevant features, and connects the extracted features to the image reconstruction stage through a jump connection, thereby retaining more feature information, reducing the loss of spatial details and spectral information in network forward propagation. In the correction stage, the obtained fusion image is subjected to spatial and spectral degradation to obtain spatial residuals and spectral residuals, the spatial residuals and the spectral residuals are fused through a network to obtain fusion residuals, and the fusion residuals are used to correct the fusion image. Therefore, the multispectral image obtained by fusion has clear spatial details and small spectral distortion.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of remote sensing image fusion, and relates to a remote sensing image fusion method and system based on fusion correction, which is suitable for solving the application scenarios where the fused image needs to be further corrected due to unclearness. BACKGROUND

[0002] Remote sensing image fusion technology aims to comprehensively utilize the high spatial resolution and low spectral resolution of panchromatic images, and the high spectral resolution and low spatial resolution of multispectral images, so as to generate multispectral images with high spatial resolution. This technology fuses the complementary information of panchromatic and multispectral images to provide better remote sensing images in the fields of agriculture, military, geographic exploration, etc.

[0003] At present, there are various remote sensing image fusion technologies, including component substitution (CS), multi-resolution analysis (MRA), model-based methods, and deep learning (DL) based methods. The component substitution method fuses by replacing the spatial components in multispectral images with panchromatic images, but this method may cause spectral distortion. The multi-resolution analysis method only fuses the high-frequency domain information of panchromatic images into multispectral images to better preserve spectral information, but may cause loss of spatial information. The model-based method fuses by establishing an optimization model containing spatial and spectral constraints, but usually requires high computational cost.

[0004] In recent years, deep learning based remote sensing image fusion methods have been widely studied, taking advantage of the excellent feature extraction capability of neural networks. However, traditional deep learning models have a problem that the network depth is too deep, causing the loss of shallow features, which limits the quality of the fused image, and the fused image often still has spatial and spectral distortion. Therefore, how to design an end-to-end deep learning framework to improve the preservation of shallow features and further correct the fused image to obtain multispectral images with rich spatial details and accurate spectral information has become an important challenge in the field of remote sensing image fusion. SUMMARY

[0005] In order to solve the problems existing in the prior art, the present application provides a remote sensing image fusion method based on image space and spectral degradation learning to correct the fused image, which further improves the quality of the fused image through fusion and correction, so as to solve the problem that the image fused through the network still has spatial and spectral distortion in the field of remote sensing image fusion, resulting in unclear fused image. In the fusion stage, we fuse the multispectral image and the panchromatic image by using the multi-scale network to utilize the complementarity of different scale features and provide more comprehensive and rich information; then the feature preservation module constructed by the channel multi-head attention selectively focuses on the features more useful for the fusion task, reduces the influence of irrelevant features, and connects the extracted features to the image reconstruction stage through the jump connection, so as to retain more feature information and reduce the loss of spatial details and spectral information in the network forward propagation. In the correction stage, the spatial residual and the spectral residual are obtained by degrading the obtained fused image in space and spectrum, the spatial residual and the spectral residual are fused through the network to obtain the fusion residual, and the fused image is corrected by using the fusion residual. Therefore, the multispectral image obtained by fusion has clear spatial details and small spectral distortion.

[0006] The remote sensing image fusion method based on fusion correction provided by the present application comprises the following steps:

[0007] Step 1, interpolating the multispectral image and concatenating it along the channel dimension with the panchromatic image;

[0008] Step 2, performing feature fusion on the multispectral image and the panchromatic image in the feature domain and multi-scale to obtain an initial fused image through a fusion module;

[0009] Step 3, further correcting the initial fused image through a residual correction network to obtain a corrected fused image, and finally combining the results of coarse fusion and correction through a cascade and convolution operation to prevent the loss of useful information in coarse fusion, wherein the input of the cascade operation is the output of the fusion module and the output of the residual correction network, the output of the fusion module is coarse fusion, and the residual correction is a further correction on the output of the fusion module;

[0010] Step 4, constructing a target function to drive the training of an image fusion model, wherein the image fusion model comprises a fusion module and a residual correction network;

[0011] Step 5, training the image fusion model by using simulation data, and testing the trained model on a test set.

[0012] Further, the specific implementation mode of step 1 is as follows:

[0013] Step 1.1, upsampling the multispectral image to match the size of the panchromatic image;

[0014] Step 1.2: Concatenate the multispectral image and the panchromatic image along the channel dimension.

[0015] Furthermore, the fusion module in step 2 includes an encoder, a decoder, and a feature preservation module. Its input is the cascaded multispectral and panchromatic images in step 1. The encoder includes multiple convolutional layers, the decoder includes multiple convolutional layers, and the encoder and decoder are connected through the feature preservation module, which includes a channel multi-head attention module, a normalization layer, and a convolutional layer.

[0016] Furthermore, the specific implementation method of step 2 is as follows;

[0017] Step 2.1: The encoder employs m convolutional layers, with the number of kernels increasing progressively in each layer. This aims to fuse multidimensional features from multispectral and panchromatic images. Average pooling is performed after the first three convolutional layers to gradually reduce the image size. This allows the convolutional kernels to cover a wider area of ​​the feature map, further expanding the receptive field and capturing more contextual information, thus fusing features at multiple scales. The output of the i-th layer... The final encoder output Z∈R C×H×W H and W represent the height and width of the feature map, and C represents the number of channels;

[0018] Step 2.2, the feature preservation module first processes the input Z... i Perform a dimensional transformation to obtain

[0019] Step 2.3, then X in The input is fed into the channel multi-head attention, and first a dimensional transformation is performed to obtain... Then use three matrices W Q W K W V Multiplying the input X yields the query vector Q, the key vector K, and the value vector V. The number of feature channels C in Q, K, and V. i Split into multiple attention heads:

[0020] Q = [Q1, Q2, ..., Q h ]

[0021] K = [K1, K2, ..., K h ]

[0022] V = [V1, V2, ..., V h ]

[0023] Where C i = h × d, where h is the number of attention heads, d is the number of feature channels per attention head, and W Q WK , W V are three learnable parameter matrices, which are first initialized randomly and then adjusted through network learning;

[0024] Step 2.4, on each attention head d features, calculate the spatial similarity between the features of Q i and K i :

[0025]

[0026] Wherein, S i ∈R d,d , S i represents the self-similarity between the two features of the feature channel on the i-th attention head, σ i represents a learnable parameter to adjust the size of the result;

[0027] Step 2.5, normalize the correlation between the feature sequences obtained in step 2.4:

[0028] G i =softmax(S i )

[0029] Step 2.6, multiply the self-similarity G i by the transposed value vector V i , and obtain the output vector of the i-th attention head as follows:

[0030]

[0031] Step 2.7, through self-attention, the output vectors of each attention head are obtained, and finally the output vectors of all attention heads are spliced through concatenation to obtain the output vector:

[0032]

[0033] Wherein,

[0034] Step 2.8, perform linear transformation on the output vector again, and fold it into a feature image

[0035]

[0036] Step 2.9, construct position encoding through convolution layer to mark the relative position information between feature channels:

[0037] PE=f p (V)

[0038] Wherein, f p(·) is a position encoding function, which consists of two convolutions, one activation function, position encoding

[0039]

[0040] Step 2.10, the output of the channel attention plus the position encoding gets the final output vector of the channel multi-head attention:

[0041]

[0042] Step 2.11, in order to keep the original features, the output of the channel multi-head attention plus the input of the feature preservation module gets:

[0043]

[0044] Step 2.12, the output of step 2.11 first passes through layer normalization, and then passes through a feedforward network consisting of two convolutional layers and one convolutional operation to get the final output of the feature preservation module.

[0045] Step 2.13, in the decoder part, m convolutional layers are used. In the first m-1 convolutional layers, the feature map first undergoes an inverse convolution operation to increase the size of the image, and then is concatenated with the preserved features obtained by the corresponding encoder through the feature preservation module. Next, further feature extraction and conversion are performed through the convolutional layers. Finally, in the mth convolutional layer, a convolution operation with a convolution kernel number of k is used to map the feature map back to the image domain to obtain the fusion image F1.

[0046] Further, in step 2.1, the convolutional layer consists of convolution + LeakyReLu activation function, and the number of channels increases layer by layer.

[0047] In step 2.9, the number of channels of the convolutional layer is equal to the number of channels C i The activation function is the GELU function.

[0048] Further, the specific implementation of step 3 is as follows:

[0049] Step 3.1, simulate the degradation function by convolution and ReLu function, respectively, for spectral degradation and spatial degradation of the fusion image, to obtain the pseudo panchromatic image P' and the pseudo multi-spectral image M'.

[0050] P' = f spe (F1)

[0051] M' = f spa (F1)

[0052] Where, f spe (·) represents the spectral degradation function, f spa(·) represents a spatial degradation function.

[0053] Step 3.2, subtract the original panchromatic image and multispectral image from the pseudo panchromatic image and pseudo multispectral image to obtain spatial residual and spectral residual:

[0054] dP = P' - P

[0055] dM = M' - M

[0056] Wherein, P represents the panchromatic image, M represents the multispectral image, dP, dM respectively represents the spatial and spectral residual.

[0057] Step 3.3, cascade the spatial residual and spectral residual, and perform feature extraction through a convolution layer;

[0058] Step 3.4, fuse the spatial and spectral residuals through 4 residual blocks;

[0059] Step 3.5, convert the residual fusion result to image domain through a convolution to obtain the fusion residual:

[0060] dF = Conv(Res4((dP, dM)))

[0061] Wherein, Res4 represents the 4 residual blocks in step 3.4, and Conv represents convolution.

[0062] Step 3.6, add the fusion image and the fusion residual to obtain the output image F2 of the residual correction network:

[0063] F2 = F1 + dF

[0064] Step 3.7, cascade the output image of the residual correction network and the fusion image obtained by the fusion module and then convolve to obtain the final fusion image F:

[0065] F = Conv(Cat(F1, F2))

[0066] Wherein, Cat represents the concatenation operation.

[0067] Further, in step 3.4, all residual blocks contain two convolution operations and a ReLu function.

[0068] Further, the loss function constructed in step 4 is as follows:

[0069]

[0070] Wherein, F and G respectively represent the fusion image and the reference image.

[0071] Further, the parameters of the trained fusion model are loaded in step 5, the test image is loaded again, the test image is fused by the fusion model, and the test result is compared with the existing remote sensing fusion method through subjective vision and objective evaluation index, and the objective evaluation index includes peak signal-to-noise ratio and spectral angle matching.

[0072] The application also provides a remote sensing image fusion system based on fusion correction, characterized by comprising the following modules:

[0073] The cascade module is used for interpolating the multispectral image and cascading the multispectral image along the channel dimension with the panchromatic image;

[0074] The initial fusion module is used for performing feature fusion on the multispectral image and the panchromatic image in a feature domain and multiple scales by the fusion module to obtain an initial fusion image;

[0075] The residual correction module is used for further correcting the initial fusion image by the residual correction network to obtain a corrected fusion image, and finally combining the coarse fusion and the correction result through a cascade and convolution operation to prevent the loss of useful information in the coarse fusion, and the input of the cascade operation is the output of the fusion module and the output of the residual correction network;

[0076] The model construction and training module is used for constructing a target function to drive the training of the image fusion model, and the image fusion model comprises the fusion module and the residual correction network;

[0077] The test module is used for training the image fusion model by using simulation data and testing the trained model on a test set.

[0078] Compared with the prior art, the application has the following advantages and beneficial effects:

[0079] The application firstly cascades the panchromatic image and the multispectral image, and then sends them into a fusion module composed of a decoder, an encoder and a feature preservation module. The encoder and the decoder adopt a multi-scale network to perform multi-scale and multi-dimensional fusion on the panchromatic image and the multispectral image. The feature preservation module preserves important features through channel multi-head attention and uses the features for image reconstruction in the decoder to obtain an initial fusion image. In the residual correction network, spatial and spectral degradation is used to learn spatial and spectral residuals. The spatial and spectral residuals are fused by a residual block to obtain a fusion residual, and the fusion residual is used to correct the fusion image to obtain a final fusion image. In the fusion module, the feature preservation module constructed by the channel multi-head attention selectively focuses on the features more useful for the fusion task, reduces the influence of irrelevant features, and connects the extracted features to the image reconstruction stage through a skip connection, thereby preserving more feature information and reducing the loss of spatial details and spectral information in the forward propagation of the network. In the correction stage, the fusion image is further corrected through spatial and spectral degradation. Therefore, the multispectral image obtained by fusion has clear spatial details and small spectral distortion. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 is the overall framework diagram of the remote sensing image fusion network based on fusion correction of an embodiment of the application.

[0081] Figure 2 is the network architecture diagram of the fusion module and the feature preservation module in the fusion module of an embodiment of the application.

[0082] Figure 3 is a schematic diagram of the channel multi-head attention algorithm of an embodiment of the application.

[0083] Figure 4 is the network architecture diagram of the residual correction network of an embodiment of the application.

[0084] Figure 5 is a test result diagram of simulation data of an embodiment of the application, wherein (a) is a low-resolution multispectral image, (b) is the result of PNN, (c) is the result of FusionNet, (d) is the result of ADKNet, (e) is the result of the method proposed in the application, and (f) is a reference image. DETAILED DESCRIPTION

[0085] In order to facilitate those skilled in the art to understand and implement the application, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the application and do not limit the application.

[0086] This invention primarily addresses the application need for further correction of unclear fused images in the field of remote sensing image fusion. We utilize a multi-scale network to fuse panchromatic and multispectral images, and employ a feature preservation module constructed using multi-channel attention to retain important features for reconstructing the fused image. After obtaining the fused image, spatial and spectral residuals are obtained through spatial and spectral degradation in the residual correction network. These residuals are then fused using residual blocks, and finally, the resulting fused residuals are used to correct the fused image. This results in a multispectral fused image with rich spatial detail and minimal spectral distortion.

[0087] Appendix Figure 1 This is an overall framework diagram of the remote sensing image fusion network based on fusion correction in the embodiment, with attached... Figure 2 This is a network architecture diagram of the fusion module and the feature preservation module within the fusion module in the embodiment, attached. Figure 3 This is a schematic diagram of the channel multi-head attention algorithm, attached. Figure 4 This is a network architecture diagram of the residual correction network. This embodiment provides a remote sensing image fusion method based on fusion correction to address the problem of unclear fused images requiring further correction in the field of remote sensing image fusion. Specifically, it includes the following steps:

[0088] Step 1: Interpolate the multispectral image and concatenate it with the panchromatic image along the channel dimension.

[0089] The specific implementation includes the following sub-steps:

[0090] Step 1.1: Upsample the multispectral image to match the size of the panchromatic image.

[0091] Step 1.2: Concatenate the multispectral image and the panchromatic image along the channel dimension.

[0092] Step 2: The initial fused image is obtained by fusing features of the multispectral image and the panchromatic image at multiple scales in the feature domain through the fusion module. The fusion module includes an encoder, a decoder, and a feature preservation module. The encoder includes four convolutional layers, and the decoder includes four convolutional layers. The encoder and decoder are connected through the feature preservation module, which includes a channel multi-head attention module, a normalization layer, and three convolutional layers.

[0093] The specific implementation includes the following sub-steps:

[0094] Step 2.1: The encoder adopts 4 layers of convolutional layers composed of convolution + LeakyReLu activation function, where the number of convolution kernels increases layer by layer, aiming to fuse the multi-dimensional features of multispectral and panchromatic images. After the first three convolutional layers, an average pooling operation is performed, so that the image size gradually decreases. In this way, the convolution kernel can cover a wider area of the feature map, further expand the range of the receptive field, and capture more context information, thereby fusing features at multiple scales. The output of the i-th layer is The final encoder output Z ∈ R C×H×W , C represents the number of channels. In this embodiment, C = 256, H = 64, W = 64, and the C i output by the 4 convolutional layers of the encoder is 32, 64, 128, and 256, respectively, and the convolution kernel size is 3x3.

[0095] Step 2.2, the feature preservation module first performs dimension transformation on the input Z i to obtain

[0096] X = [x1, x2, …, x n ]

[0097] Step 2.3, then X in is input into the channel multi-head attention, which first performs dimension transformation to obtain Then, three matrices W Q , W K , and W V are multiplied with the input X to obtain query vector Q, key vector K, and value vector V, The feature channel number C i in Q, K, and V is split into multiple attention heads:

[0098] Q = [Q1, Q2, …, Q h ]

[0099] K = [K1, K2, …, K h ]

[0100] V = [V1, V2, …, V h ]

[0101] Where C i = hxd, h is the number of attention heads, d is the number of feature channels of each attention head, and W Q , W K , and W V are three learnable parameter matrices. In this embodiment, d = 4.

[0102] Step 2.4, on the d features of each attention head, Q i and Ki Spatial similarity between features:

[0103]

[0104] Among them, S i ∈R d,d S i σ represents the self-similarity between pairwise features of the feature channels at the i-th attention head. i This indicates that learnable parameters can be used to adjust the size of the result;

[0105] Step 2.5: Normalize the correlation between the feature sequences obtained in Step 2.4:

[0106] G i =softmax(S i )

[0107] Step 2.6, calculate the self-similarity G. i Multiply by the transpose of the value vector V i The output vector of the i-th attention head is obtained as follows:

[0108]

[0109] Step 2.7: Through self-attention, the output vectors of each attention head are obtained simultaneously. Finally, the output vectors of all attention heads are concatenated and concatenated to obtain the output vector:

[0110]

[0111] in,

[0112] Step 2.8 involves performing a linear transformation on the output vector and then folding it into a feature image through dimensionality transformation.

[0113]

[0114] Step 2.9: Construct positional encoding through convolutional layers to mark the relative positional information between feature channels:

[0115] PE = f p (V)

[0116] Among them, f p (·) is the positional encoding function, consisting of two convolutions and one activation function. The kernel size for each convolution is 3×3, and the number of channels corresponds to the number of channels C output by the corresponding encoder layer. i The same applies; the activation function is the GELU function positional encoding.

[0117] Step 2.10, the output of the channel attention is added to the position encoding to obtain the final output vector of the channel multi-head attention:

[0118]

[0119] Step 2.11, in order to maintain the original features, the output of the channel multi-head attention is added to the input of the feature preservation module to obtain:

[0120]

[0121] Step 2.12, the output of step 2.11 is first passed through layer normalization, and then passed through a feedforward network composed of two convolutional layers and a convolutional operation to obtain the final output of the feature preservation module, where the three convolutional kernel sizes are 1x1, 3x3, and 1x1, respectively, and the number of channels is 4C i ,4C i ,C i ;

[0122] Step 2.13, in the decoder part, we use four convolutional layers. In the first three convolutional layers, the number of convolutional kernels changes from 128, 64, to 32, and the first three convolutional layers are composed of two two-dimensional convolutions + LeakyReLu activation functions. The feature map is first passed through a deconvolution operation to increase the size of the image, and then concatenated with the preserved features obtained by the encoder through the feature preservation module. Next, further feature extraction and conversion are performed through the convolutional layer. Finally, in the fourth convolutional layer, we use a convolutional operation with a convolutional kernel size of 4 to map the feature map back to the image domain to obtain the fused image F1.

[0123] Step 3: further correct the initial fused image through the residual correction network to obtain the corrected fused image, and finally combine the coarse fusion and the corrected result through a concatenation and convolution operation to prevent the loss of useful information in the coarse fusion. The input of the concatenation operation is the output of the fusion module and the output of the residual correction network, and the output of the fusion module is the coarse fusion, and the residual correction is a further correction of the output of the fusion module. The specific implementation includes the following sub-steps:

[0124] Step 3.1, simulate the degradation function through convolution and ReLu function, respectively, to perform spectral degradation and spatial degradation on the fused image to obtain pseudo-panchromatic image P' and pseudo-multispectral image M'.

[0125] P' = f spe (F1)

[0126] M' = f spa (F1)

[0127] where f spe (·) represents the spectral degradation function, fspa (·) represents a spatial degradation function, the number of convolution channels in the spectral degradation function is 1, and the size is 3x3. The number of convolution channels in the spatial degradation function is 4, and the size is 3x3.

[0128] Step 3.2, subtract the original panchromatic image and multispectral image from the pseudo panchromatic image and pseudo multispectral image to obtain spatial residual and spectral residual:

[0129] dP = P' - P

[0130] dM = M' - M

[0131] Where P represents the panchromatic image, M represents the multispectral image, dP and dM represent the spatial and spectral residuals, respectively.

[0132] Step 3.3, cascade the spatial residual and spectral residual, and perform feature extraction through a convolution layer, wherein the number of channels of the convolution layer is 64 and the size is 3x3;

[0133] Step 3.4, fuse the spatial and spectral residuals through 4 residual blocks, all of which contain two convolution operations and a ReLu function, and the number of channels of the convolution layer is 64 and the size is 3x3;

[0134] Step 3.5, convert the residual fusion result to the image domain through a convolution to obtain the fusion residual:

[0135] dF = Conv(Res4((dP, dM)))

[0136] Where Res4 represents the 4 residual blocks in step 3.4, Conv represents convolution, and the number of convolution channels is 4 and the size is 1x1.

[0137] Step 3.6, add the fusion image and the fusion residual to obtain the output image F2 of the residual correction network:

[0138] F2 = F1 + dF

[0139] Step 3.7, cascade the output image of the residual correction network and the fusion image obtained by the fusion module and then perform convolution, wherein the number of convolution channels is 4 and the size is 1x1, to obtain the final fusion image F:

[0140] F = Conv(Cat(F1, F2))

[0141] Where Cat represents the concatenation operation.

[0142] Step 4: Construct the target function to drive the training of the image fusion model. The specific implementation includes the following sub-steps:

[0143] Step 4.1: Constructing loss function. Constructing L2-based loss function:

[0144]

[0145] Where F and G represent the fused image and the reference image, respectively.

[0146] Step 4.2: Randomly select 32 data inputs from the training set to input the network, complete one iteration, and adjust the network parameters.

[0147] Step 5: Train the above image fusion model using simulation data and test the trained model on the test set. The specific implementation includes the following sub-steps:

[0148] Step 5.1: Train the above fusion correction model using image data from the Gaofen-2 satellite. In this embodiment, the training data set is divided into a training set and a validation set, with a data set size ratio of 9:1. The resolution of the original multispectral image and panchromatic image is 64x64, 256x256. The image after downsampling the original multispectral and panchromatic images by 4 times is used as the input of the network, and the original multispectral image is used as the reference image. In network training, the initial learning rate is 0.0001, the batch size is 32, the network is optimized by the Adam optimizer, and the training rounds are 160 rounds.

[0149] Step 5.2: Load the fusion model trained on the Gaofen-2 satellite and test it on the test set. Compare the test results with each comparison method in terms of visual and objective evaluation indicators. The test image size is 512x512. To verify the effectiveness of the proposed method, we compare it with supervised learning methods. The supervised learning methods include PNN, FusionNet, and ADKNet. The visual comparison results are shown in FIG. 5. Figure 5 The objective evaluation indicators are peak signal-to-noise ratio (PSNR) and spectral angle matching (SAM). PSNR measures the closeness of the fused image to the true value, and SAM measures the spectral distortion of the fused image. The average results on the test set are shown in Table 1.

[0150] Table 1: Comparison of indicators of different methods for simulation data (ideal value: PSNR: +∞, SAM: 0)

[0151]

[0152] As can be seen, the proposed method first fuses the panchromatic and multispectral images through an end-to-end fusion correction framework, then preserves important spatial and spectral features through the feature preservation module, and finally performs spatial and spectral correction to obtain multispectral fusion images with rich spatial details and spectral distortion.

[0153] The application also provides a remote sensing image fusion system based on fusion correction, characterized by comprising the following modules.

[0154] A cascading module is configured to interpolate the multispectral image and cascade the interpolated multispectral image with the panchromatic image along the channel dimension.

[0155] An initial fusion module is configured to perform feature fusion on the multispectral image and the panchromatic image in a feature domain and at multiple scales by using the fusion module to obtain an initial fusion image.

[0156] A residual correction module is configured to further correct the initial fusion image by using a residual correction network to obtain a corrected fusion image.

[0157] A model construction and training module is configured to construct a training of an image fusion model driven by a target function, wherein the image fusion model comprises the fusion module and the residual correction network.

[0158] A test module is configured to train the image fusion model by using simulation data and test the trained model on a test set.

[0159] The specific implementation manners of the modules are the same as those of the steps, and the application will not be described.

[0160] It should be understood that the parts not described in detail in the specification are all prior art.

[0161] It should be understood that the above description of the embodiments is detailed, and therefore should not be considered as a limitation on the scope of patent protection of the application. Those skilled in the art can make substitutions or modifications without departing from the scope of the claims, and all such substitutions or modifications fall within the scope of the application. The scope of patent protection of the application should be subject to the appended claims.

Claims

1. A remote sensing image fusion method based on fusion correction, characterized in that, The method comprises the following steps: Step 1, interpolating the multispectral image and concatenating along the channel dimension with the panchromatic image; Step 2, performing feature fusion on the multispectral image and the panchromatic image in the feature domain and at multiple scales through a fusion module to obtain an initial fusion image; Step 3, further correcting the initial fusion image through a residual correction network to obtain a corrected fusion image, and finally combining the coarse fusion and the correction result through a concatenation and convolution operation to prevent the loss of useful information in the coarse fusion, the input of the concatenation operation being the output of the fusion module and the output of the residual correction network; The specific implementation of the residual correction network in step 3 is as follows: Step 3.1, simulating a degradation function through convolution and ReLu function to perform spectral degradation and spatial degradation on the fusion image respectively to obtain a pseudo-panchromatic image P' and a pseudo-multispectral image M'; P' = f spe (F1) M' = f spe (F1) where f spe (·) denotes a spectral degradation function, f spa (·) denotes a spatial degradation function; Step 3.2, subtracting the original panchromatic image and multispectral image from the pseudo-panchromatic image and pseudo-multispectral image to obtain spatial and spectral residuals: dP = P' - P dM = M' - M Wherein, P represents the panchromatic image, M represents the multispectral image, dP and dM represent the spatial and spectral residuals respectively; Step 3.3, concatenating the spatial and spectral residuals and performing feature extraction through a convolution layer; Step 3.4, fusing the spatial and spectral residuals through four residual blocks; Step 3.5, converting the residual fusion result to the image domain through a convolution to obtain a fusion residual: dF = Conv(Res4((dP, dM))) Wherein, Res4 represents the four residual blocks in step 3.4, and Conv represents convolution; Step 3.6, adding the fusion image and the fusion residual to obtain the output image F2 of the residual correction network: F2 = F1 + dF Step 3.7, concatenating and convolving the output image of the residual correction network and the fusion image obtained by the fusion module to obtain the final fusion image F: F = Conv(Cat(F1, F2)) Wherein, Cat represents the concatenation operation; Step 4, constructing a target function to drive the training of the image fusion model, wherein the image fusion model comprises the fusion module and the residual correction network; Step 5, training the above image fusion model using simulation data and testing the trained model on a test set.

2. The method according to claim 1, wherein the method is characterized by: The specific implementation of step 1 is as follows: Step 1.1, upsampling the multispectral image to match the size of the panchromatic image; Step 1.2, concatenating the multispectral image and the panchromatic image in the channel dimension.

3. The method according to claim 1, wherein the method is characterized by: The fusion module in step 2 comprises an encoder, a decoder and a feature preservation module, and the input of the fusion module is the concatenated multispectral and panchromatic image in step 1, wherein the encoder comprises a plurality of convolution layers, the decoder comprises a plurality of convolution layers, and the encoder and the decoder are connected through the feature preservation module, and the feature preservation module comprises a channel multi-head attention module, a normalization layer and a convolution layer.

4. The method according to claim 1, wherein the method is characterized by: The specific implementation of step 2 is as follows: Step 2.1, the encoder adopts m layers of convolutional layers, the number of convolutional kernels of each layer increases layer by layer, aiming to fuse the multi-dimensional features of the multispectral and panchromatic images, and average pooling operations are performed after the first m-1 layers of convolutional layers, so that the image size gradually decreases; the output of the i-th layer is set as Then the final encoder output Z ∈ R C×H×W H and W represent the height and width of the feature map, and C represents the number of channels. Step 2.2, the feature preserving module first applies a dimension transformation to the input Z i to obtain Step 2.3, then X in Input into channel multi-head attention, first dimension transformation to get Then use three matrices W Q , W K , W V Multiply with input Q to get query vector Q, key vector K, value vector V, Split the feature channel number C i In Q, K, V into multiple attention heads: Q = [Q1, Q2,..., Q h ] K = [K1, K2,..., K h ] V = [V1, V2,..., V h ] where C i = h x d, h is the number of attention heads, d is the number of feature channels per attention head, W Q , W K , W V are three learnable parameter matrices; Step 2.

4. On each attention head d features, compute Q i and the spatial similarity between the features of K i and K wherein S i ∈ R d,d , S i represents the self-similarity between the pairwise features of the i-th attention head on the feature channel, σ i represents a learnable parameter to adjust the result size; Step 2.5, normalizing the correlation between the feature sequences obtained in step 2.4: G i = softmax(S i ) Step 2.

6. Multiply the self-similarity G i with the transposed value vector V i The output vector of the i-th attention head is given by Step 2.7, the output vector of each attention head is obtained by self-attention, and finally the output vectors of all attention heads are spliced by concatenation to obtain the output vector: wherein Step 2.

8. The output vector is again linearly transformed and folded into a feature image by a dimension transformation Step 2.9, the relative position information between feature channels is marked by constructing position encoding through convolutional layer: PE = f p (V) where f p (·) is a position encoding function consisting of two convolutions, a position encoding Step 2.10, the output of channel attention is added to the position encoding to obtain the final output vector of channel multi-head attention: Step 2.11, in order to keep the original features, the output of channel multi-head attention is added to the input of feature preservation module to obtain: Step 2.12, the output of step 2.11 is first normalized by layer normalization, and then the final output of the feature preservation module is obtained by passing through two convolutional layers and a convolutional operation composed of a feedforward network: Step 2.13, in the decoder part, m convolutional layers are used, in the first m-1 convolutional layers, the feature map is first subjected to deconvolution operation to increase the size of the image, and then concatenated with the reserved features obtained by the feature preservation module of the corresponding layer encoder; next, further feature extraction and conversion are carried out through convolutional layer, finally, in the mth convolutional layer, a convolution operation with a convolution kernel number of k is used to map the feature map back to the image domain to obtain the fusion image F1.

5. The method according to claim 4, wherein the method is based on fusion correction. 5 In step 2.1, the convolutional layer is composed of convolution + LeakyReLu activation function, and the number of channels increases layer by layer; The number of convolution layer channels in step 2.9 and the channel C of the corresponding encoder layer output i The same, the activation function is the GELU function.

6. The fusion correction-based remote sensing image fusion method of claim 1, wherein: In step 3.4, all residual blocks contain two convolutional operations and a ReLu function.

7. The fusion correction-based remote sensing image fusion method of claim 1, wherein: The constructed loss function in step 4 is as follows: Where F and G represent the fusion image and the reference image respectively.

8. The fusion correction-based remote sensing image fusion method of claim 1, wherein: In step 5, the parameters of the trained fusion model are loaded, and then the test image is loaded. The test image is fused by the fusion model, and the test results are compared with the existing remote sensing fusion methods through subjective vision and objective evaluation indexes. The objective evaluation indexes include peak signal-to-noise ratio, spectral angle matching.

9. A remote sensing image fusion system based on fusion correction, characterized in that, The modules include: The concatenation module is used to concatenate the interpolated multi-spectral image and the panchromatic image along the channel dimension; The initial fusion module is used to fuse the multi-spectral image and the panchromatic image in the feature domain through the fusion module to obtain an initial fusion image; The residual correction module further corrects the initial fusion image through the residual correction network to obtain a corrected fusion image. Finally, a concatenation and convolution operation is used to further combine the coarse fusion and the corrected result to prevent the loss of useful information in the coarse fusion. The input of the concatenation operation is the output of the fusion module and the output of the residual correction network. The specific implementation of the residual correction network is as follows: Step 3.1, the spectral degradation and spatial degradation of the fusion image are simulated by convolution and ReLu function to obtain pseudo-panchromatic image P' and pseudo-multi-spectral image M'; P' = f spe (F1) M' = f spa (F1) where f spe (·) denotes a spectral degradation function, f spa (·) denotes a spatial degradation function; Step 3.2, the pseudo-panchromatic image and the pseudo-multi-spectral image are subtracted from the original panchromatic image and multi-spectral image to obtain the spatial residual and spectral residual: dP = P' - P dM = M' - M Where P represents the panchromatic image, M represents the multi-spectral image, and dP, dM represent the spatial and spectral residuals respectively; Step 3.3, the spatial residual and the spectral residual are concatenated and subjected to feature extraction through a convolutional layer. Step 3.4, fusing the spatial and spectral residuals through four residual blocks; Step 3.5, converting the residual fusion result to the image domain through a convolution to obtain a fused residual: dF = Conv(Res4((dP, dM))) Wherein, Res4 represents the four residual blocks in step 3.4, and Conv represents convolution. Step 3.6, adding the fused image and the fused residual to obtain the output image F2 of the residual correction network: F2 = F1 + dF Step 3.7, concatenating the output image of the residual correction network and the fused image obtained by the fusion module and then convolving to obtain the final fused image F: F = Conv(Cat(F1, F2)) Wherein, Cat represents the concatenation operation. A model construction and training module is configured to construct a training of a target function driven image fusion model, wherein the image fusion model comprises a fusion module, a residual correction network. A test module is configured to train the image fusion model using simulation data and test the trained model on a test set.

Images