Remote sensing image fusion method, device, electronic device and storage medium

Abstract

The present invention discloses a remote sensing image fusion method, device, electronic device, and storage medium, relating to the field of image fusion technology. The method comprises: inputting a panchromatic image and a low-resolution multispectral image into a remote sensing image fusion model to obtain a high-resolution multispectral image; the remote sensing image fusion model comprises a multi-level feature extraction module, a multi-level feature fusion module, and a CMI-UNet++ image reconstruction module, each of which includes an attention mechanism layer with a multi-purpose adaptive receptive field. The attention mechanism layer is determined based on the SA attention mechanism and the inception architecture. The present invention can fully utilize the panchromatic and multispectral images in the target scene, effectively fusing the spatial information of the panchromatic image with the spectral information of the multispectral image, obtaining a fused image rich in detail information and improving the resolution of the fused image.

Description

Technical Field

[0001] The present invention relates to the field of image fusion technology, and in particular to a remote sensing image fusion method, device, electronic device and storage medium based on a multi-purpose adaptive receptive field attention mechanism and a composite multi-input reconstruction neural network. Background Art

[0002] Satellites carry a variety of optical sensors to obtain different remote sensing images. Key remote sensing images include multispectral (MS) images, panchromatic (PAN) images, and hyperspectral images. The remote sensing images received by satellites vary depending on the optical sensor used and the reflectance spectrum. Multispectral sensors can obtain more spectral information than panchromatic sensors. However, the spatial resolution of MS images is typically lower than that of PAN images. PAN and MS images are fused after they have been registered. Ideally, all spectral information of the MS image should be retained while maintaining the spatial resolution of the PAN image.

[0003] Due to their excellent performance, deep learning algorithms have also been a research hotspot in recent years, demonstrating superior performance and wide applicability across various fields. In 2018, Yuan et al. introduced multi-scale feature extraction and residual learning algorithms into convolutional neural networks, developing a multi-scale and multi-depth convolutional neural network for generalization and sharpening of remote sensing images, achieving excellent fusion results. Liu et al. also developed a generative adversarial network for generalization and sharpening of remote sensing images, which can effectively fuse MS and PAN images.

[0004] Currently, remote sensing image fusion methods have been extensively studied, but some issues remain unresolved. For example, while component replacement-based remote sensing image fusion methods are computationally efficient, they ignore the inherent spectral connections between PAN and MS images, making the fused image susceptible to spectral distortion. Multi-scale analysis-based remote sensing image fusion methods suffer from long simulation times, information redundancy, and a single gradient direction, and the fused image may contain ghosting. Model-based remote sensing image fusion methods also suffer from the difficulty of accurately setting hyperparameters and the lack of expressive power of the model itself. Summary of the Invention

[0005] The present invention aims to address, at least to some extent, one of the technical problems in the related art. To this end, the present invention provides a remote sensing image fusion method, apparatus, electronic device, and storage medium that fully utilize the panchromatic and multispectral images of a target scene, effectively fusing the spatial information of the panchromatic image with the spectral information of the multispectral image, resulting in a fused image rich in detail and improving the resolution of the fused image.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] In a first aspect, the present invention provides a remote sensing image fusion method, comprising:

[0008] Acquire panchromatic and low-resolution multispectral images of the target scene;

[0009] Inputting the panchromatic image and the low-resolution multispectral image into a remote sensing image fusion model to obtain a remote sensing reconstructed image; the remote sensing reconstructed image is a high-resolution multispectral image;

[0010] The remote sensing image fusion model includes:

[0011] A multi-level feature extraction module is used to perform multi-scale decomposition on the panchromatic image and the low-resolution multispectral image, respectively, to obtain multi-level feature maps of spectral information and multi-level feature maps of spatial information;

[0012] A multi-level feature fusion module is used to fuse the multi-level feature maps of spectral information and the multi-level feature maps of spatial information to obtain a multi-level fused feature map;

[0013] CMI-UNet++ image reconstruction module, used to obtain remote sensing reconstructed images based on multi-level fusion feature maps;

[0014] Among them, the multi-level feature extraction module, the multi-level feature fusion module and the CMI-UNet++ image reconstruction module all contain a multi-purpose adaptive receptive field attention mechanism layer; the attention mechanism layer is determined based on the SA attention mechanism and the inception structure.

[0015] In a second aspect, the present invention provides a remote sensing image fusion device, comprising:

[0016] A data acquisition module, used to acquire panchromatic images and low-resolution multispectral images of the target scene;

[0017] a remote sensing reconstructed image determination module, configured to input the panchromatic image and the low-resolution multispectral image into a remote sensing image fusion model to obtain a remote sensing reconstructed image; the remote sensing reconstructed image is a high-resolution multispectral image;

[0018] The remote sensing image fusion model includes:

[0019] A multi-level feature extraction module is used to perform multi-scale decomposition on the panchromatic image and the low-resolution multispectral image, respectively, to obtain multi-level feature maps of spectral information and multi-level feature maps of spatial information;

[0020] A multi-level feature fusion module is used to fuse the multi-level feature maps of spectral information and the multi-level feature maps of spatial information to obtain a multi-level fused feature map;

[0021] CMI-UNet++ image reconstruction module, used to obtain remote sensing reconstructed images based on multi-level fusion feature maps;

[0022] Among them, the multi-level feature extraction module, the multi-level feature fusion module and the CMI-UNet++ image reconstruction module all contain a multi-purpose adaptive receptive field attention mechanism layer; the attention mechanism layer is determined based on the SA attention mechanism and the inception structure.

[0023] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the remote sensing image fusion method according to the first aspect.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the remote sensing image fusion method described in the first aspect.

[0025] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0026] In response to the problems involved in the background technology, the present invention designs a remote sensing image fusion model with an AE structure. PAN images and LRMS images (i.e., low-resolution MS images) are used as the input of the remote sensing image fusion model, and HRMS images (i.e., high-resolution MS images) are used as the output of the remote sensing image fusion model. The remote sensing image fusion model can be represented as three modules: a multi-level feature extraction module, a multi-level feature fusion module, and a CMI-UNet++ image reconstruction module. The multi-level feature extraction module is used to extract the spatial information and spectral information of the PAN image and the LRMS image. The multi-level feature fusion module can be regarded as superimposing the information output from the previous layer twice to fully fuse the extracted spatial information and spectral information. Finally, the performance of the CMI-UNet++ image reconstruction module benefits from the multi-level jump connection of the internal nodes, which enables the model to effectively utilize the differences and complementarities of the multi-layer features of the image to reconstruct high-resolution MS images. Among them, the multi-scale feature extraction module and the multi-scale feature fusion module can be regarded as encoders in the AE structure. The encoder can extract feature maps and also realize the fusion of feature information. The CMI-UNet++ image reconstruction module can be regarded as the decoder in the AE structure. The decoder is targetedly optimized according to the UNet++ network to achieve the generation of high-resolution MS images; in addition, the present invention also constructs a multi-purpose adaptive receptive field attention mechanism layer to reduce feature information loss and multi-scale focusing of core information. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 Schematic diagram of the process of remote sensing image fusion method of the present invention;

[0029] Figure 2 Schematic diagram of the structure of the remote sensing image fusion model of the present invention;

[0030] Figure 3 Schematic diagram of the structure of the multi-level feature extraction module of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of the target attention mechanism layer of the present invention;

[0032] Figure 5 Schematic diagram of the structure of the multi-level feature fusion module of the present invention;

[0033] Figure 6This is a schematic diagram of the structure of the CMI-UNet++ image reconstruction module of the present invention;

[0034] Figure 7 This is a comparison diagram of the present invention and the prior art;

[0035] Figure 8 It is a structural schematic diagram of the remote sensing image fusion device of the present invention. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Example 1

[0039] like Figure 1 As shown, an embodiment of the present invention provides a remote sensing image fusion method, including:

[0040] Step 100: Acquire a panchromatic image and a low-resolution multispectral image of the target scene.

[0041] Step 200: Input the panchromatic image and the low-resolution multispectral image into a remote sensing image fusion model to obtain a remote sensing reconstructed image; the remote sensing reconstructed image is a high-resolution multispectral image.

[0042] The remote sensing image fusion model includes:

[0043] The multi-level feature extraction module is used to perform multi-scale decomposition on the panchromatic image and the low-resolution multispectral image respectively, and obtain multi-level feature maps of spectral information and multi-level feature maps of spatial information.

[0044] The multi-level feature fusion module is used to fuse the multi-level feature maps of spectral information and the multi-level feature maps of spatial information to obtain a multi-level fused feature map.

[0045] The CMI-UNet++ image reconstruction module is used to obtain remote sensing reconstructed images based on multi-level fusion feature maps.

[0046] Among them, the multi-level feature extraction module, the multi-level feature fusion module and the CMI-UNet++ image reconstruction module all contain a multi-purpose adaptive receptive field attention mechanism layer; the attention mechanism layer is determined based on the SA attention mechanism and the inception structure.

[0047] The target attention mechanism layer is a multi-purpose adaptive receptive field attention mechanism layer; the target attention mechanism layer includes a fusion neuron and multiple attention mechanism neurons connected to the fusion neuron. The attention mechanism neurons include convolution neurons, channel attention mechanism neurons, and spatial attention mechanism neurons. The convolution neurons are used to process the input features using the convolution kernel to obtain the first sub-feature and the second sub-feature; the channel attention mechanism neurons are used to process the first sub-feature; the spatial attention mechanism neurons are used to process the second sub-feature; and the fusion neurons are used to fuse the processed first and second sub-features.

[0048] As a preferred implementation, the multi-stage feature extraction module described in the embodiment of the present invention includes: a multispectral image feature extraction unit and a panchromatic image feature extraction unit.

[0049] The multispectral image feature extraction unit includes a first feature map extraction subunit, a second feature map extraction subunit, a third feature map extraction subunit, a fourth feature map extraction subunit and a fifth feature map extraction subunit; the first feature map extraction subunit includes a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the target attention mechanism layer is an attention mechanism layer with a multi-purpose adaptive receptive field; the second feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the third feature map extraction subunit includes a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the fourth feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the fifth feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence.

[0050] The full-color image feature extraction unit includes a sixth feature map extraction subunit, a seventh feature map extraction subunit, an eighth feature map extraction subunit, a ninth feature map extraction subunit and a tenth feature map extraction subunit; the sixth feature map extraction subunit includes a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the seventh feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the eighth feature map extraction subunit includes a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the ninth feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the tenth feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence.

[0051] Furthermore, the multi-level feature fusion module includes a first-layer feature fusion unit and a second-layer feature fusion unit.

[0052] The first-layer feature fusion unit includes a feature map extraction subunit, a first feature fusion subunit, a second feature fusion subunit, a third feature fusion subunit, a fourth feature fusion subunit and a fifth feature fusion subunit; the first feature fusion subunit includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the second feature fusion subunit includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the third feature fusion subunit includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the fourth feature fusion subunit includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the fifth feature fusion subunit includes a feature map splicing layer.

[0053] The second-layer feature fusion unit includes a sixth feature fusion subunit, a seventh feature fusion subunit, an eighth feature fusion subunit, a ninth feature fusion subunit and a tenth feature fusion subunit; the sixth feature fusion subunit includes a convolutional layer, a feature map splicing layer and a target attention mechanism layer connected in sequence; the seventh feature fusion subunit includes a feature map splicing layer; the eighth feature fusion subunit includes a feature map splicing layer and a target attention mechanism layer connected in sequence; the ninth feature fusion subunit includes a feature map splicing layer; the tenth feature fusion subunit includes a feature map splicing layer and a target attention mechanism layer.

[0054] The feature map extraction subunit is used to extract features from the full-color image and the low-resolution multispectral image to obtain a feature map related to spectral information and a feature map related to spatial information.

[0055] The first feature fusion subunit is used to fuse the feature map output by the first feature map extraction subunit and the feature map output by the sixth feature map extraction subunit.

[0056] The second feature fusion subunit is used to fuse the feature map output by the second feature map extraction subunit and the feature map output by the seventh feature map extraction subunit.

[0057] The third feature fusion subunit is used to fuse the feature map output by the third feature map extraction subunit and the feature map output by the eighth feature map extraction subunit.

[0058] The fourth feature fusion subunit is used to fuse the feature map output by the fourth feature map extraction subunit and the feature map output by the ninth feature map extraction subunit.

[0059] The fifth feature fusion subunit is used to splice the feature map output by the fifth feature map extraction subunit and the feature map output by the tenth feature map extraction subunit.

[0060] The sixth feature fusion subunit is used to fuse the feature map about spectral information and the feature map about spatial information output by the feature map extraction subunit to obtain a first fused feature map.

[0061] The seventh feature fusion subunit is used to fuse the feature map output by the activation function layer in the first feature fusion subunit and the feature map output by the feature map splicing layer in the second feature fusion subunit to obtain a second fused feature map.

[0062] The eighth feature fusion subunit is used to fuse the feature map output by the activation function layer in the second feature fusion subunit and the feature map output by the feature map splicing layer in the third feature fusion subunit to obtain a third fused feature map.

[0063] The ninth feature fusion subunit is used to fuse the feature map output by the activation function layer in the third feature fusion subunit and the feature map output by the feature map splicing layer in the fourth feature fusion subunit to obtain a fourth fused feature map.

[0064] The tenth feature fusion subunit is used to fuse the feature map output by the activation function layer in the fourth feature fusion subunit and the feature map output by the feature map splicing layer in the fifth feature fusion subunit to obtain a fifth fused feature map.

[0065] As a preferred embodiment, the CMI-UNet++ image reconstruction module described in the embodiment of the present invention is used to use the CMI-UNet++ neural network to transform the first fusion feature map, the second fusion feature map, the third fusion feature map, the fourth fusion feature map and the fifth fusion feature map output by the multi-level feature fusion module; wherein, the connection method between neurons in the CMI-UNet++ neural network is a multi-level jump connection method.

[0066] Furthermore, the CMI-UNet++ image reconstruction module includes a first image reconstruction unit, a second image reconstruction unit, a third image reconstruction unit, a fourth image reconstruction unit and a fifth image reconstruction unit.

[0067] The first image reconstruction unit includes a first image reconstruction neuron, a second image reconstruction neuron, a third image reconstruction neuron, a fourth image reconstruction neuron and a fifth image reconstruction neuron; the first image reconstruction neuron includes a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the second image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the third image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the fourth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the fifth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence.

[0068] The second image reconstruction unit includes a sixth image reconstruction neuron, a seventh image reconstruction neuron, an eighth image reconstruction neuron and a ninth image reconstruction neuron; the sixth image reconstruction neuron includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the seventh image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the eighth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the ninth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence.

[0069] The third image reconstruction unit includes a tenth image reconstruction neuron, an eleventh image reconstruction neuron and a twelfth image reconstruction neuron; the tenth image reconstruction neuron includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the eleventh image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the twelfth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence.

[0070] The fourth image reconstruction unit includes a thirteenth image reconstruction neuron and a fourteenth image reconstruction neuron; the thirteenth image reconstruction neuron includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the fourteenth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence.

[0071] The fifth image reconstruction unit includes a fifteenth image reconstruction neuron; the fifteenth image reconstruction neuron includes a convolution layer, a normalization layer and an activation function layer connected in sequence.

[0072] Among them, the output end of the first image reconstruction neuron is respectively connected to the input end of the second image reconstruction neuron, the input end of the third image reconstruction neuron, the input end of the fourth image reconstruction neuron and the input end of the fifth image reconstruction neuron; the output end of the second image reconstruction neuron is respectively connected to the input end of the third image reconstruction neuron, the input end of the fourth image reconstruction neuron and the input end of the fifth image reconstruction neuron; the output end of the third image reconstruction neuron is respectively connected to the input end of the fourth image reconstruction neuron and the input end of the fifth image reconstruction neuron; the output end of the fourth image reconstruction neuron is connected to the input end of the fifth image reconstruction neuron.

[0073] The output end of the sixth image reconstruction neuron is respectively connected to the input end of the second image reconstruction neuron, the input end of the seventh image reconstruction neuron, the input end of the eighth image reconstruction neuron and the input end of the ninth image reconstruction neuron; the output end of the seventh image reconstruction neuron is respectively connected to the input end of the third image reconstruction neuron, the input end of the eighth image reconstruction neuron and the input end of the ninth image reconstruction neuron; the output end of the eighth image reconstruction neuron is respectively connected to the input end of the fourth image reconstruction neuron and the input end of the ninth image reconstruction neuron; the output end of the ninth image reconstruction neuron is connected to the input end of the fifth image reconstruction neuron.

[0074] The output end of the tenth image reconstruction neuron is respectively connected to the input end of the seventh image reconstruction neuron, the input end of the eleventh image reconstruction neuron and the input end of the twelfth image reconstruction neuron; the output end of the eleventh image reconstruction neuron is respectively connected to the input end of the eighth image reconstruction neuron and the input end of the twelfth image reconstruction neuron; the output end of the twelfth image reconstruction neuron is connected to the input end of the ninth image reconstruction neuron.

[0075] The output end of the thirteenth image reconstruction neuron is connected to the input end of the eleventh image reconstruction neuron and the input end of the fourteenth image reconstruction neuron respectively; the output end of the fourteenth image reconstruction neuron is connected to the input end of the twelfth image reconstruction neuron.

[0076] The output end of the fifteenth image reconstruction neuron is connected to the input end of the fourteenth image reconstruction neuron.

[0077] Example 2

[0078] The present invention provides a remote sensing image fusion method, which is as follows:

[0079] S1: Input source image A and source image B; source image A is LRMSi=[LRMS1, LRMS2, LRMS3], and source image B is PAN.

[0080] S2: Use the multi-level feature extraction module to perform multi-scale decomposition on the source image A and the source image B respectively to obtain multi-level feature maps of spectral information and spatial information.

[0081] like Figure 2 As shown in Part A of Figure 1, the multi-level feature extraction module serves as the head of the remote sensing image fusion model. It consists of two parts. The two parts have similar structures but different parameters. Table 1 shows the details of the multi-level feature extraction module.

[0082] Table 1 Multi-level feature extraction module parameters

[0083]

[0084] The feature map output by the multi-level feature extraction module is extremely important for the subsequent modules. The multi-level feature extraction module can be expressed as:

[0085]

[0086]

[0087] Where P∈R H×W Represents PAN image, M∈R (H / 4)×(W / 4)×C represents the LRMS image, Represents multi-level feature maps about spectral information, Represents a multi-level feature map of spatial information, C represents the number of channels, i represents the number of layers, and the value range is [0, n], H represents the length, W represents the width, and ↓ represents a two-fold downsampling operation.

[0088] like Figure 3As shown in the figure, the LRMS image and the PAN image are input to the multi-level feature extraction module. However, due to the different specifications of the two images, the multi-level feature extraction module is designed to be divided into two parts: a multispectral image feature extraction unit and a panchromatic image feature extraction unit. The multi-level feature extraction module is also structurally divided into two parts, each consisting of five downsampling layers. Each downsampling layer contains a convolutional layer (Conv), a normalization layer (BN), and an activation function layer (ReLU). Some downsampling layers also include a multi-purpose adaptive receptive field attention mechanism layer (referred to as the target attention mechanism layer or ASA attention module layer). Using ReLU as the activation function makes its derivation easier and the training speed faster.

[0089] The definition of the activation function layer is shown in formula (2):

[0090]

[0091] Among them, c represents different channels.

[0092] Based on the SA attention mechanism and the Inception architecture, this paper proposes an ASA attention module layer. Specifically, the ASA attention module first uses several different convolutional neurons to differentiate and segment the input features and resize them into two sub-features. Then, for each sub-feature group, a permutation unit is used to characterize the dependencies of the features in the spatial and channel dimensions. Finally, the processed features are numerically superimposed and reduced in the first dimension, and finally fused and output. The code flow of the ASA attention module layer is shown in Table 2.

[0093] Table 2 Code flow chart of ASA attention module layer

[0094]

[0095]

[0096] Usually, an attention module is added after the convolutional neurons to focus on the feature information, which inevitably increases the number of training parameters. Therefore, pooling and linear upsampling are integrated into the attention module to achieve the adjustability of feature scale. At this point, the ASA attention module layer can be used as a convolutional neuron with adaptive receptive field and hybrid attention mechanism. The structure diagram of the ASA attention module layer is as follows: Figure 4 shown.

[0097] S3: The feature map extracted by the multi-level feature extraction module is input into the multi-level feature fusion module, and is fused through a multi-level fusion method so that the spectral information and spatial information of the MS image and the PAN image are fully fused.

[0098] Multi-level feature fusion modules such as Figure 2 As shown in Part B, its detailed structure is as follows Figure 5 As shown in Figure 3, the feature fusion module can be described by equation (3).

[0099]

[0100]

[0101] Among them, the PAN image is represented as P∈R H×W , the low-resolution MS image is represented as M∈R (H / 4)×(W / 4)×C ,and and Represented as the output of the multi-level feature extraction module, Y f1 ∈R (H / j)×(W / j)×(C×j) Represents the output of the first layer feature fusion unit, Y f2 ∈R (H / j)×(W / j)×(C×j) Represents the output of the second-layer feature fusion unit. Where C represents the number of channels, i, j represent the number of layers, and C, i, j = 0, 1, 2, ... n. Conv represents a convolutional layer, BN represents a batch normalization layer, ReLU represents a ReLU activation function layer, and ASA represents an ASA attention module layer.

[0102] The multi-level feature fusion module is divided into two layers. The first-level feature fusion unit has six subunits. The feature maps output by the multi-level feature extraction module are used as input, and then pass through a convolutional layer, a batch normalization layer, and a ReLU activation function layer to the second-level feature fusion unit. The second-level feature fusion unit has five subunits. The output of the first-level feature fusion unit is used as input, and then passes through a convolutional layer, a batch normalization layer, and a ReLU activation function layer to output. To minimize feature loss, the extracted five-layer feature maps are fused twice with the PAN image and the MS image. Details of the multi-level feature fusion module are shown in Table 3.

[0103] Table 3 Multi-level feature fusion module parameters

[0104]

[0105] Among them, Concat represents the feature map concatenation layer.

[0106] S4: The fused feature map is input into the CMI-UNet++ neural network in the CMI-UNet++ image reconstruction module to obtain a high-resolution MS target image.

[0107] CMI-UNet++ image reconstruction module Figure 2 As shown in Part C of Figure 6The CMI-UNet++ image reconstruction module is embedded as a tail in the overall model. The input to the CMI-UNet++ image reconstruction module consists of five fused feature maps of different sizes, with the following parameters: 16×16×2048, 32×32×1024, 64×64×512, 128×128×256, and 256×256×128. The CMI-UNet++ image reconstruction module can obtain rich spectral and spatial information, making the model more robust and improving the reconstruction performance. Details are shown in Table 4.

[0108] Table 4 CMI-UNet++ image reconstruction module parameters

[0109]

[0110] S5: Calculate the LOSS difference between the high-resolution MS target image and the label image and back-propagate the optimized network.

[0111] During the training phase, the epoch is set to 550, the batch size is 1, the optimizer is Adam, and the learning rate is set to 0.0001. The loss function is extremely important in any model. The loss function affects the training process of the model and even affects the generated results. The present invention uses L1 and L2 as loss functions, and their equations can be expressed as formula (4). L1 and L2 represent the pixel difference and mean square error, respectively, where Ψ represents the output of the network and Y is the actual ground image, i.e., the label image.

[0112]

[0113] S6: Input MS and PAN images into the model in fixed batches and cycle through S0-S5 in sequence.

[0114] S7: Repeat S5 according to the hyperparameters until the specified number of times.

[0115] Example 3

[0116] The embodiment of the present invention selects multiple groups of remote sensing images for fusion and compares them with various existing fusion algorithms to verify the advantages of the method of the present invention.

[0117] The QuickBird satellite remote sensing image dataset, provided by the University of Maryland, contains original high-resolution MS images and registered PAN images. Following the Wald rule, the present invention uses the downsampled low-resolution MS images and PAN images as network inputs, while the original high-resolution MS images serve as reference images. The entire high-resolution MS, low-resolution MS, and PAN images are segmented and organized into 256×256, 64×64, and 256×256 image pairs. Images containing black edges are then cleaned to yield 795 pairs of usable images. The training dataset contains 700 pairs of images, and the test dataset contains 95 pairs. Images in the datasets were randomly selected.

[0118] In order to verify the advancement and effectiveness of the method of the present invention in remote sensing image fusion. Eight methods, Brovey, CNMF, MTF-GLP-HPM, PCA, SFIM, GSA, PNN, and PanNet, were used for comparative analysis. The codes of the above comparison methods are all public and the parameters have not changed. Moreover, the tested images all follow the diversification of landform information. Quantitative and qualitative evaluations were also performed on the entire experiment, in which the quantitative evaluation method was divided into reference evaluation indicators and non-reference evaluation indicators. The reference evaluation indicators include Peak Signal to Noise Ratio (PSNR), Structural Similarity (SSIM), Spectral Angle Mapper (SAM), and relative dimensionless global integrated error (ERGAS), spatial correlation coefficient (SCC), and quality index (Q). The non-reference evaluation indicators include spectral distortion index (D λ ), spatial distortion index (D s ) and QNR. A brief introduction to these indicators is given here.

[0119] According to the Wald protocol, the present invention simulates high-resolution MS images as ground truth data and applies them to qualitative and quantitative evaluation. First, qualitative analysis is performed, such as Figure 7As shown, overall, all methods can reconstruct a fused image, but all suffer from spatial loss and spectral distortion. The images generated by the Brovey and PCA methods exhibit severe spectral distortion, but spatial detail is restored relatively finely. The performance of Brovey and PCA is most striking in the last two images. The Brovey method restores spectral information toward a green tint. Compared to Brovey, PCA's spectral restoration is weaker, resulting in an almost black-and-white image. In the first column of images, Brovey, PCA, GSA, and CNMF methods reproduce the water surface spectral information in a gray-black color. This result is closer to the PAN image, but indicates significant spectral distortion. In the second column of images, the Brovey and PCA methods generate images that tend toward a green tint, while the other four traditional methods achieve a higher degree of spectral restoration. These methods effectively preserve the spectral information of the MS image. However, compared to the reference image, the image appears oversharpened. While the PNN and PanNet methods retain spectral information well, the images are somewhat blurred. Overall, the methods in this chapter perform best. In the third column of images, the SFIM, GSA, CNMF, and MTF_GLP_HPM methods all oversharpen, giving the entire image a relief-like appearance, especially in the case of MTF_GLP_HPM. The PNN and PanNet methods, on the other hand, achieve significantly less sharpness, rendering details unclear throughout the image, with only blurred outlines visible. In the fourth column of images, the fused images produced by the PNN and PanNet methods appear blurry compared to the ground truth. Road lines are distorted, and details of buildings are unrecognizable. The fused images produced by the six traditional methods are oversharpened, clearly displaying details that were absent in the reference image. This suggests that these methods favor the PAN image when restoring spatial details, ignoring the spatial information of the MS image. This results in the fused images appearing oversharpened compared to the ground truth. The fusion results from the methods in this chapter are all close to the ground truth, with virtually no visible spectral distortion or loss of detail restoration.

[0120] In the fifth column, all methods except the CNMF method successfully restore the red house in the center of the image. The CNMF method generates an overall yellowish image compared to the reference image, while the other methods restore spectral information more accurately. In the sixth column, due to the presence of dense vegetation and houses on the ground, our method also suffers from some loss of spatial detail, resulting in a slightly blurry image compared to the reference image. All of the methods except the PNN and PanNet methods also generate images with ghosting and spectral distortion. In the seventh column, with the exception of F-UNet++, PNN, and PanNet, the remaining six methods exhibit superior spatial information preservation, but also suffer from varying degrees of spectral distortion. Furthermore, our method preserves more spatial detail than the PNN and PanNet methods. In the last two columns, the river color in both the MS and reference images tends to be light yellow. However, with the exception of our method, none of the other methods achieves a good restoration. The Brovey and PCA methods primarily extract spectral information from the PAN image and fuse it into the generated image. Through the above analysis, the method in this chapter has obtained the best results in qualitative analysis, and is the best in both spatial details and spectral information reconstruction.

[0121] In addition, the above test images were quantitatively compared, as shown in Table 5. The experiment used nine evaluation indicators, including reference evaluation indicators including PSNR, SSIM, SAM, ERGAS, SCC, Q; non-reference evaluation indicators including D λ 、D s , and QNR. Among the reference-based evaluation metrics, our method achieved the best performance. In the non-reference evaluation metrics, both the spectral distortion index and the non-reference quality index achieved first place. Although it did not take first place in the spatial distortion index, our method was only 0.008266 points behind the first place. In summary, through qualitative and quantitative comparisons with various comparison methods, our method has proven its effectiveness and advancement.

[0122] Table 5 Comparative test table

[0123]

[0124]

[0125] Example 4

[0126] In order to execute the method corresponding to the above-mentioned embodiment 1 and achieve corresponding functions and technical effects, a remote sensing image fusion device is provided below.

[0127] like Figure 8 As shown, the remote sensing image fusion device includes:

[0128] The data acquisition module 1 is used to acquire a panchromatic image and a low-resolution multispectral image of the target scene.

[0129] The remote sensing reconstructed image determination module 2 is used to input the panchromatic image and the low-resolution multispectral image into a remote sensing image fusion model to obtain a remote sensing reconstructed image; the remote sensing reconstructed image is a high-resolution multispectral image.

[0130] The remote sensing image fusion model includes:

[0131] The multi-level feature extraction module is used to perform multi-scale decomposition on the panchromatic image and the low-resolution multispectral image respectively, and obtain multi-level feature maps of spectral information and multi-level feature maps of spatial information.

[0132] The multi-level feature fusion module is used to fuse the multi-level feature maps of spectral information and the multi-level feature maps of spatial information to obtain a multi-level fused feature map.

[0133] The CMI-UNet++ image reconstruction module is used to obtain remote sensing reconstructed images based on multi-level fusion feature maps.

[0134] Among them, the multi-level feature extraction module, the multi-level feature fusion module and the CMI-UNet++ image reconstruction module all contain a multi-purpose adaptive receptive field attention mechanism layer; the attention mechanism layer is determined based on the SA attention mechanism and the inception structure.

[0135] Example 3

[0136] An embodiment of the present invention provides an electronic device including a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the remote sensing image fusion method of embodiment 1.

[0137] Optionally, the above-mentioned electronic device may be a server.

[0138] In addition, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the remote sensing image fusion evaluation method of the first embodiment is implemented.

[0139] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0140] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A remote sensing image fusion method, characterized in that: include: Acquire panchromatic and low-resolution multispectral images of the target scene; Inputting the panchromatic image and the low-resolution multispectral image into a remote sensing image fusion model to obtain a remote sensing reconstructed image; The remote sensing reconstructed image is a high-resolution multispectral image; The remote sensing image fusion model includes: A multi-level feature extraction module is used to perform multi-scale decomposition on the panchromatic image and the low-resolution multispectral image, respectively, to obtain multi-level feature maps of spectral information and multi-level feature maps of spatial information; A multi-level feature fusion module is used to fuse the multi-level feature maps of spectral information and the multi-level feature maps of spatial information to obtain a multi-level fused feature map; The composite multi-input CMI-UNet++ image reconstruction module is used to obtain remote sensing reconstructed images based on multi-level fusion feature maps; Among them, the multi-level feature extraction module, the multi-level feature fusion module and the composite multi-input CMI-UNet++ image reconstruction module all contain a multi-purpose adaptive receptive field attention mechanism layer; the attention mechanism layer is determined based on the SA attention mechanism and the inception structure.

2. The remote sensing image fusion method according to claim 1, characterized in that: The target attention mechanism layer is a multi-purpose adaptive receptive field attention mechanism layer; the target attention mechanism layer includes a fusion neuron and a plurality of attention mechanism neurons connected to the fusion neuron; Among them, the attention mechanism neurons include convolution neurons, channel attention mechanism neurons and spatial attention mechanism neurons; the convolution neurons are used to process the input features through the convolution kernel to obtain the first sub-feature and the second sub-feature; the channel attention mechanism neurons are used to process the first sub-feature; the spatial attention mechanism neurons are used to process the second sub-feature; the fusion neurons are used to fuse the processed first sub-feature and the second sub-feature.

3. The remote sensing image fusion method according to claim 2, characterized in that: The multi-level feature extraction module includes: a multispectral image feature extraction unit and a panchromatic image feature extraction unit; The multispectral image feature extraction unit includes a first feature map extraction subunit, a second feature map extraction subunit, a third feature map extraction subunit, a fourth feature map extraction subunit and a fifth feature map extraction subunit; the first feature map extraction subunit includes a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the target attention mechanism layer is an attention mechanism layer with a multi-purpose adaptive receptive field; the second feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the third feature map extraction subunit includes a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the fourth feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the fifth feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence; The full-color image feature extraction unit includes a sixth feature map extraction subunit, a seventh feature map extraction subunit, an eighth feature map extraction subunit, a ninth feature map extraction subunit and a tenth feature map extraction subunit; the sixth feature map extraction subunit includes a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the seventh feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the eighth feature map extraction subunit includes a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the ninth feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the tenth feature map extraction subunit includes a convolution layer, a normalization layer and an activation function layer connected in sequence.

4. The remote sensing image fusion method according to claim 3, characterized in that: The multi-level feature fusion module includes a first-layer feature fusion unit and a second-layer feature fusion unit; The first-layer feature fusion unit includes a feature map extraction subunit, a first feature fusion subunit, a second feature fusion subunit, a third feature fusion subunit, a fourth feature fusion subunit and a fifth feature fusion subunit; the first feature fusion subunit includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the second feature fusion subunit includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the third feature fusion subunit includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the fourth feature fusion subunit includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the fifth feature fusion subunit includes a feature map splicing layer; The second-layer feature fusion unit includes a sixth feature fusion subunit, a seventh feature fusion subunit, an eighth feature fusion subunit, a ninth feature fusion subunit and a tenth feature fusion subunit; The sixth feature fusion subunit includes a convolution layer, a feature map splicing layer and a target attention mechanism layer connected in sequence; the seventh feature fusion subunit includes a feature map splicing layer; the eighth feature fusion subunit includes a feature map splicing layer and a target attention mechanism layer connected in sequence; the ninth feature fusion subunit includes a feature map splicing layer; the tenth feature fusion subunit includes a feature map splicing layer and a target attention mechanism layer.

5. The remote sensing image fusion method according to claim 4, characterized in that: The feature map extraction subunit is used to extract features from the panchromatic image and the low-resolution multispectral image to obtain a feature map of spectral information and a feature map of spatial information; The first feature fusion subunit is used to fuse the feature map output by the first feature map extraction subunit and the feature map output by the sixth feature map extraction subunit; The second feature fusion subunit is used to fuse the feature map output by the second feature map extraction subunit and the feature map output by the seventh feature map extraction subunit; The third feature fusion subunit is used to fuse the feature map output by the third feature map extraction subunit and the feature map output by the eighth feature map extraction subunit; The fourth feature fusion subunit is used to fuse the feature map output by the fourth feature map extraction subunit and the feature map output by the ninth feature map extraction subunit; The fifth feature fusion subunit is used to splice the feature map output by the fifth feature map extraction subunit and the feature map output by the tenth feature map extraction subunit; The sixth feature fusion subunit is used to fuse the feature map about spectral information and the feature map about spatial information output by the feature map extraction subunit to obtain a first fused feature map; The seventh feature fusion subunit is used to fuse the feature map output by the activation function layer in the first feature fusion subunit and the feature map output by the feature map splicing layer in the second feature fusion subunit to obtain a second fused feature map; The eighth feature fusion subunit is used to fuse the feature map output by the activation function layer in the second feature fusion subunit and the feature map output by the feature map splicing layer in the third feature fusion subunit to obtain a third fused feature map; The ninth feature fusion subunit is used to fuse the feature map output by the activation function layer in the third feature fusion subunit and the feature map output by the feature map splicing layer in the fourth feature fusion subunit to obtain a fourth fused feature map; The tenth feature fusion subunit is used to fuse the feature map output by the activation function layer in the fourth feature fusion subunit and the feature map output by the feature map splicing layer in the fifth feature fusion subunit to obtain a fifth fused feature map.

6. The remote sensing image fusion method according to claim 1, characterized in that: The CMI-UNet++ image reconstruction module includes a first image reconstruction unit, a second image reconstruction unit, a third image reconstruction unit, a fourth image reconstruction unit and a fifth image reconstruction unit; The first image reconstruction unit includes a first image reconstruction neuron, a second image reconstruction neuron, a third image reconstruction neuron, a fourth image reconstruction neuron and a fifth image reconstruction neuron; the first image reconstruction neuron includes a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the second image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the third image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the fourth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; the fifth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer, an activation function layer and a target attention mechanism layer connected in sequence; The second image reconstruction unit includes a sixth image reconstruction neuron, a seventh image reconstruction neuron, an eighth image reconstruction neuron and a ninth image reconstruction neuron; the sixth image reconstruction neuron includes a convolution layer, a normalization layer and an activation function layer connected in sequence; the seventh image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the eighth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; the ninth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer and an activation function layer connected in sequence; The third image reconstruction unit includes a tenth image reconstruction neuron, an eleventh image reconstruction neuron, and a twelfth image reconstruction neuron; the tenth image reconstruction neuron includes a convolution layer, a normalization layer, and an activation function layer connected in sequence; the eleventh image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer, and an activation function layer connected in sequence; the twelfth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer, and an activation function layer connected in sequence; The fourth image reconstruction unit includes a thirteenth image reconstruction neuron and a fourteenth image reconstruction neuron; the thirteenth image reconstruction neuron includes a convolution layer, a normalization layer, and an activation function layer connected in sequence; the fourteenth image reconstruction neuron includes a feature map splicing layer, a convolution layer, a normalization layer, and an activation function layer connected in sequence; The fifth image reconstruction unit includes a fifteenth image reconstruction neuron; the fifteenth image reconstruction neuron includes a convolution layer, a normalization layer and an activation function layer connected in sequence.

7. The remote sensing image fusion method according to claim 6, characterized in that: The output end of the first image reconstruction neuron is connected to the input end of the second image reconstruction neuron, the input end of the third image reconstruction neuron, the input end of the fourth image reconstruction neuron, and the input end of the fifth image reconstruction neuron respectively; the output end of the second image reconstruction neuron is connected to the input end of the third image reconstruction neuron, the input end of the fourth image reconstruction neuron, and the input end of the fifth image reconstruction neuron respectively; the output end of the third image reconstruction neuron is connected to the input end of the fourth image reconstruction neuron and the input end of the fifth image reconstruction neuron respectively; the output end of the fourth image reconstruction neuron is connected to the input end of the fifth image reconstruction neuron; The output end of the sixth image reconstruction neuron is connected to the input end of the second image reconstruction neuron, the input end of the seventh image reconstruction neuron, the input end of the eighth image reconstruction neuron, and the input end of the ninth image reconstruction neuron respectively; the output end of the seventh image reconstruction neuron is connected to the input end of the third image reconstruction neuron, the input end of the eighth image reconstruction neuron, and the input end of the ninth image reconstruction neuron respectively; the output end of the eighth image reconstruction neuron is connected to the input end of the fourth image reconstruction neuron and the input end of the ninth image reconstruction neuron respectively; the output end of the ninth image reconstruction neuron is connected to the input end of the fifth image reconstruction neuron; The output end of the tenth image reconstruction neuron is connected to the input end of the seventh image reconstruction neuron, the input end of the eleventh image reconstruction neuron, and the input end of the twelfth image reconstruction neuron respectively; the output end of the eleventh image reconstruction neuron is connected to the input end of the eighth image reconstruction neuron and the input end of the twelfth image reconstruction neuron respectively; the output end of the twelfth image reconstruction neuron is connected to the input end of the ninth image reconstruction neuron; The output end of the thirteenth image reconstruction neuron is connected to the input end of the eleventh image reconstruction neuron and the input end of the fourteenth image reconstruction neuron respectively; the output end of the fourteenth image reconstruction neuron is connected to the input end of the twelfth image reconstruction neuron; The output end of the fifteenth image reconstruction neuron is connected to the input end of the fourteenth image reconstruction neuron.

8. A remote sensing image fusion device, characterized in that: include: A data acquisition module, used to acquire panchromatic images and low-resolution multispectral images of the target scene; a remote sensing reconstructed image determination module, configured to input the panchromatic image and the low-resolution multispectral image into a remote sensing image fusion model to obtain a remote sensing reconstructed image; The remote sensing reconstructed image is a high-resolution multispectral image; The remote sensing image fusion model includes: A multi-level feature extraction module is used to perform multi-scale decomposition on the panchromatic image and the low-resolution multispectral image, respectively, to obtain multi-level feature maps of spectral information and multi-level feature maps of spatial information; A multi-level feature fusion module is used to fuse the multi-level feature maps of spectral information and the multi-level feature maps of spatial information to obtain a multi-level fused feature map; The composite multi-input CMI-UNet++ image reconstruction module is used to obtain remote sensing reconstructed images based on multi-level fusion feature maps; Among them, the multi-level feature extraction module, the multi-level feature fusion module and the composite multi-input CMI-UNet++ image reconstruction module all contain a multi-purpose adaptive receptive field attention mechanism layer; the attention mechanism layer is determined based on the SA attention mechanism and the inception structure.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the remote sensing image fusion method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The device stores a computer program, which, when executed by a processor, implements the remote sensing image fusion method according to any one of claims 1 to 7.

Images