Remote sensing image fusion method and system based on deep neural network

Abstract

The present disclosure provides a remote sensing image fusion method and system based on a deep neural network, the method comprising: obtaining a panchromatic image and a multispectral image, preprocessing to obtain a set of images to be fused; based on the set of images to be fused and the panchromatic image, obtaining a spatial feature image and a spectral feature image; inputting the set of images to be fused, the spatial feature image and the spectral feature image into a first fusion module, a spatial information extraction module and a spectral information extraction module respectively to obtain first fusion features, spatial features and spectral features; inputting the first fusion features, the spatial features and the spectral features into a decoder to reconstruct a high-resolution multispectral image. By constructing the spectral information extraction module and the spatial information extraction module to extract rich spatial information and spectral information respectively, and considering dimension transformation and multiplication operation in the feature fusion stage, the information between different features is interacted and enhanced, and the quality of the high-resolution multispectral image is further improved.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, and particularly relates to a remote sensing image fusion method and system based on deep neural networks. Background Technology

[0002] Due to technological and physical limitations, the spatial resolution of images provided by various imaging sensors is often limited. Many orbital satellites, such as QuickBird, GeoEye-1, WorldView-4, and GaoFen-2, can simultaneously provide panchromatic and multispectral images. Generally, multispectral images contain multiple bands and have rich spectral information, but less spatial information. Compared to multispectral images, single-band panchromatic images have richer spatial details but less spectral information. Panchromatic image sharpening aims to fuse panchromatic (PAN) images and low spatial resolution multispectral (LRMS) images to obtain high spatial and spectral resolution (HRMS) images. HRMS images, rich in information and spatial detail, are of great significance in various applications such as military systems, environmental monitoring, and mapping services.

[0003] Classical panchromatic sharpening methods can be broadly categorized into three types: component substitution (CS) methods, multi-resolution analysis (MRA) methods, and variational optimization (VO) methods. In recent years, deep learning (DL)-based panchromatic sharpening methods have demonstrated significant advantages over traditional methods. However, the inventors have discovered that existing deep learning-based panchromatic sharpening methods can only accept datasets from a single satellite at a time, and cannot handle multiple datasets simultaneously. Furthermore, they fail to fully capture rich spatial location and spectral information, still exhibiting problems of spatial information blurring and spectral distortion. Summary of the Invention

[0004] In order to solve the above problems, the present invention aims to provide a remote sensing image fusion method and system based on deep neural networks. By constructing a spectral information extraction module and a spatial information extraction module to extract rich spatial and spectral information respectively, and considering dimensional transformation and multiplication operations in the feature fusion stage, the information between different features can interact and be enhanced, thereby further improving the quality of high-resolution multispectral images.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a remote sensing image fusion method based on a deep neural network, comprising:

[0007] Obtain panchromatic and multispectral images, perform preprocessing, and obtain the image set to be fused;

[0008] Based on the image set to be fused and the panchromatic image, spatial feature images and spectral feature images are obtained;

[0009] The image set to be fused, the spatial feature image, and the spectral feature image are respectively input into the first fusion module, the spatial information extraction module, and the spectral information extraction module to obtain the first fused feature, the spatial feature, and the spectral feature.

[0010] The first fusion feature, spatial feature, and spectral feature are input into the decoder to reconstruct a high-resolution multispectral image.

[0011] Preferably, acquiring panchromatic and multispectral images, performing preprocessing, and obtaining the image set to be fused specifically includes:

[0012] The multispectral image is upsampled four times to obtain the first multispectral image;

[0013] The first panchromatic image is obtained by downsampling the panchromatic image by four times and upsampling it by four times.

[0014] The first multispectral image and the first panchromatic image constitute the image set to be fused.

[0015] Preferably, obtaining spatial feature images and spectral feature images based on the image set to be fused and the panchromatic image specifically includes:

[0016] Subtracting the first panchromatic image from the corresponding panchromatic image yields the spatial feature image;

[0017] The first panchromatic image is stacked four times in the channel dimension to form a four-channel image, which is then subtracted from the first multispectral image to obtain the spectral feature information image.

[0018] Preferably, the image set to be fused is stacked in the channel dimension and input into the first fusion module to obtain the first fusion feature; the first fusion module includes a first neural network, an encoder and a CD module.

[0019] Preferably, the first neural network includes a convolutional layer with a 3×3 kernel and a stride of 1, a max pooling layer with a 2×2 kernel and a stride of 2, a convolutional layer with a 3×3 kernel and a stride of 1, a max pooling layer with a 2×2 kernel and a stride of 2, and a convolutional layer with a 3×3 kernel and a stride of 1.

[0020] Preferably, the CD module includes a max pooling layer with a 2×2 kernel and a stride of 2, a convolutional layer with a 3×3 kernel, a stride of 1, and padding of 1, a convolutional layer with a kernel size of 3x3, a stride of 1, and padding of 1, a max pooling layer with a 2x2 kernel and a stride of 2, and three hidden fully connected layers.

[0021] Preferably, the spatial feature image is input to the spatial information extraction module to obtain spatial features; specifically including:

[0022] Perform FFT transformation on the spatial feature image to obtain amplitude and phase information;

[0023] After convolving the amplitude information, an inverse FFT transformation is performed with the phase information to obtain the spatial features.

[0024] Preferably, the spectral feature image is input to the spectral information extraction module to obtain spectral features; specifically including:

[0025] The first spectral feature is obtained by performing a convolution operation on the spectral feature image;

[0026] The first spectral feature is subjected to two dimensional transformations, and the results of the two dimensional transformations are multiplied to obtain the spectral feature.

[0027] Preferably, the first fused feature, spatial feature, and spectral feature are input into the decoder to reconstruct the high-resolution multispectral image, specifically including:

[0028] The first fusion feature is transformed in dimension and multiplied with the spectral feature matrix to obtain the first result;

[0029] After performing dimensionality transformation and convolution operations on the first result, it is multiplied element-wise with the spatial features and then added to obtain the second result;

[0030] The second result is processed through three convolutional layers with a kernel size of 3×3 and a stride of 1 to obtain a high-resolution multispectral image.

[0031] Secondly, the present invention provides a remote sensing image fusion system based on a deep neural network, comprising:

[0032] Data acquisition module: used to acquire panchromatic and multispectral images, perform preprocessing, and obtain the image set to be fused;

[0033] Data processing module: used to obtain spatial feature images and spectral feature images based on the image set to be fused and the panchromatic image;

[0034] Feature acquisition module: used to input the image set to be fused, spatial feature image and spectral feature image into the first fusion module, spatial information extraction module and spectral information extraction module respectively to obtain the first fused feature, spatial feature and spectral feature;

[0035] Image reconstruction module: used to input the first fused features, spatial features and spectral features into the decoder to reconstruct a high-resolution multispectral image.

[0036] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0037] (1) The present invention places panchromatic images and multispectral images in the information extraction module for processing, extracting spatial information and spectral information respectively, so that the present invention can obtain accurate spatial and spectral information at the same time, which is beneficial to the use of subsequent image processing technology and improves the utilization rate of spatial and spectral information.

[0038] (2) This invention inputs images from four datasets together into the network for training, training a general framework network. A contrastive loss function is used to make the image styles of each dataset more consistent. KL Loss is used to ensure feature alignment of the four satellite data, so that the reconstructed high-resolution multispectral images contain more detailed information.

[0039] (3) In the feature fusion stage, this invention enables the information between different features to interact and be enhanced through dimensional transformation and multiplication operations. This interaction allows the originally independent features to generate new and meaningful representations during the fusion process, thereby improving the expressive power of the features. By alternating between dimensional transformation and multiplication operations, the fused features become more compact and effective.

[0040] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0041] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute a limitation thereof.

[0042] Figure 1 The main flowchart of a remote sensing image fusion method based on a deep neural network provided by the present invention;

[0043] Figure 2 A detailed flowchart of a remote sensing image fusion method based on a deep neural network provided by the present invention;

[0044] Figure 3 This is a comparison chart showing the results of fusing four datasets using the present invention: low spatial resolution multispectral images and high spatial resolution panchromatic images. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] Example 1

[0047] like Figure 1 As shown, this embodiment discloses a remote sensing image fusion method based on deep neural networks, including the following steps:

[0048] S1: Acquire panchromatic and multispectral images, perform preprocessing, and obtain the image set to be fused;

[0049] S2: Based on the image set to be fused and the panchromatic image, obtain the spatial feature image and the spectral feature image;

[0050] S3: Input the image set to be fused, the spatial feature image, and the spectral feature image into the first fusion module, the spatial information extraction module, and the spectral information extraction module, respectively, to obtain the first fused feature, the spatial feature, and the spectral feature;

[0051] S4: Input the first fused feature, spatial feature and spectral feature into the decoder to reconstruct the high-resolution multispectral image.

[0052] In this embodiment, spatial information is input into the spatial feature extraction module to obtain spatial information; spectral information is input into the spectral feature extraction module to obtain spectral information. Full-size four-satellite panchromatic and multispectral images are input into the encoder to obtain a full-size fused feature map. The full-size fused feature map is input into the decoder, undergoing convolution, upsampling, convolution, upsampling, and convolution operations. This, along with the spatial information obtained from the spatial feature extraction module and the spectral information obtained from the spectral feature extraction module, is input twice into the SCM block to achieve feature interaction and enhancement. Finally, after three convolutional blocks, the number of channels is reduced to 4, resulting in a high spatial resolution multispectral image. Specific steps are as follows: Figure 2 As shown, the detailed process is as follows:

[0053] (1) Input image: Input a full-size panchromatic image. F ∈R 256×256×1 and multispectral images Full-scale LRMS image L F ∈R 64×64×4 .

[0054] (2) Image preprocessing: The full-size multispectral image L... F ∈R 64×64×4 A four-fold upsampling was performed to obtain the first full-size multispectral image L. F ∈R 256×256×4 ; to convert the full-size, full-color image P F ∈R 256×256×1 Perform a 4x downsampling, followed by a 4x upsampling, to obtain the first full-size panchromatic image P. F ∈R 256×256×1 ;

[0055] The first full-size panchromatic image P F ∈R 256×256×1 Compared with the input full-size panchromatic image P F ∈R 256×256×1Subtracting the corresponding values ​​yields spatial feature information;

[0056] The first full-size panchromatic image P F ∈R 256×256×1 A four-channel image P is formed by stacking four times in the channel dimension. F ∈R 256×256×4 , with the first full-size multispectral image L F ∈R 256×256×4 Subtracting the corresponding values ​​yields the spectral feature information.

[0057] (3) Building the core module: The core module consists of four parts: encoder, decoder, spatial information extraction module, and spectral information extraction module.

[0058] (3.1) Encoder: In order to make full use of the spatial information of panchromatic image (PAN) and the spectral information of multispectral image (LRMS), this embodiment uses two branch networks to extract the features of PAN and LRMS respectively to obtain their complementary information.

[0059] The spectral branch has 4 input channels, and the spatial branch has 1 input channel. The input panchromatic image (PAN) and the input multispectral image (LRMS) are stacked in the channel dimension and fused through a neural network. The fusion is performed through a convolutional layer with a kernel of 3×3 and a stride of 1, a max pooling layer with a kernel of 2×2 and a stride of 2, a convolutional layer with a kernel of 3×3 and a stride of 1, a max pooling layer with a kernel of 2×2 and a stride of 2, and a convolutional layer with a kernel of 3×3 and a stride of 1 to obtain the first fused feature.

[0060] The first fused feature input is passed through the encoder and then fed into the CD module, which outputs the KL divergence (KL Loss) to ensure feature alignment across the four channels. The CD module consists of a 2×2 kernel max-pooling layer with a stride of 2, a 3×3 kernel convolutional layer with a stride of 1 and padding of 1, performing batch normalization and ReLU activation on the 64-channel feature map, a 3×3 kernel convolutional layer with a stride of 1 and padding of 1, performing batch normalization and ReLU activation on the 64-channel feature map, performing max-pooling with a 2×2 kernel and a stride of 2, three hidden fully connected layers, and an output of a 10-dimensional tensor, which outputs the second fused feature.

[0061] (3.2) Decoder: The decoder is used to reconstruct the features extracted by the encoder, thereby obtaining the high spatial resolution multispectral image HRMS that we want.

[0062] The second fusion feature generated by the encoder is sequentially passed through a 3×3 convolutional layer with a stride of 1, a 4×4 convolutional layer with a stride of 2 and padding of 1 transposed convolution to upsample the image, a 3×3 convolutional layer with a stride of 1, a 4×4 convolutional layer with a stride of 2 and padding of 1 transposed convolution to upsample the image, and a 3×3 convolutional layer with a stride of 1 to output the information.

[0063] (3.3) Spectral Information Extraction Module: The spectral feature information obtained in step (2) is used as input and passed through a convolutional layer, a LeakyReLU activation function layer, another convolutional layer, another LeakyReLU activation function layer, and another convolutional layer. The convolutional layer consists of a 3×3 kernel and a stride of 1. By further extracting the input spectral information, a sequence containing rich spectral information and features is output.

[0064] Next, the output B×C×H×W is transformed into the format B×C×(H*W), and then the format B×C×(H*W) is changed to B×(H*W)×C. Finally, the matrix in B×C×(H*W) format is multiplied by the matrix in B×(H*W)×C format to obtain the Gram Matrix, which is then used as the output. This output sequence is used to calculate the Gram Matrix and act as attention for the input features. By calculating the correlations between sequences, the model can focus its attention on important parts at different positions in the sequence, thereby achieving better semantic understanding and modeling.

[0065] After the Gram Matrix is ​​passed through three fully connected layers, the output is a tensor of size 10, which is used to calculate the contrastive loss.

[0066] (3.4) Spatial Information Extraction Module: The spatial feature information obtained in step (2) is used as input. After FFT (Fast Fourier Transform), it forms amplitude and phase branches. The phase information remains unchanged. The amplitude information is processed by a 3×3 convolution kernel with a stride of 1, followed by a LeakyReLU activation function layer. After the convolution operation with a 3×3 kernel and a stride of 1, it is combined with the phase information and processed by inverse FFT to generate the final output. The amplitude branch, after passing through the CD module, forms a contrastive loss function.

[0067] The panchromatic image's PAN branch is input into the spatial information extraction module. After FFT transformation, considering that the phase component describes the relative position or offset between different frequency components, the phase component of the PAN is not changed and is directly input into the inverse FFT transformation, which can more fully preserve the relative position of different frequency components. The amplitude component of the PAN contains rich structural information such as edges and textures. After convolution operation, more high-frequency information in the PAN is further extracted, thus ensuring that more sufficient spatial information is extracted.

[0068] (3.5) High spatial resolution multispectral images

[0069] Then, it goes through two SCM blocks, which include the results generated by the spatial information extraction module, the results generated by the spectral information extraction module, and the results output by the encoder.

[0070] First, the spectral information extraction module extracts a Gram Matrix, which contains the spectral information of the image. To embed the extracted spectral information into the encoder output, we employ a dense attention operation to transform and multiply the resulting matrix. Specifically, in the first SCM block, the encoder output B×C×H×W is transformed into a B×C×(H*W) format, multiplied with the matrix generated by the spectral information extraction module, and the first result is output.

[0071] Next, in the second SCM block, the output of the first result, B×C×(H*W), is transformed into the format B×C×H×W. After passing through a convolutional layer with a kernel of 3×3 and a stride of 1, it is multiplied element-wise with the result generated by the spatial information extraction module (also passed through a convolutional layer with a kernel of 3×3 and a stride of 1) and then added together to output the second result. This addition avoids information loss and alleviates the gradient vanishing problem.

[0072] After passing through three more convolutional layers with a kernel size of 3×3 and a stride of 1, where the number of output channels is 32, 16, and 4 respectively, the desired high spatial resolution multispectral image HRMS∈R is obtained. 256×256×4 .

[0073] (4) Loss functions: The loss functions used in this model are Contrastive Loss, KL Loss and MSE Loss.

[0074] (4.1) Contrastive Loss is analogous to the contrastive loss in simCLR, and includes positive and negative sample pairs. Contrastive Loss is used twice in this invention: Loss1 = Contrastive Loss1, and Loss2 = Contrastive Loss2. KLLoss refers to KL divergence, and the purpose of the loss is to align the features of the four satellites. Loss3 = KL Loss. Loss4 represents the L2 loss between the output HRMS and the actual HRMS labels.

[0075] (4.2) The total loss function is

[0076] Loss=Loss1+Loss2+Loss3+Loss4.

[0077] (5) Model training process

[0078] (5.1) Input Image

[0079] This embodiment uses data from the GeoEye-1, WorldView-4, QuickBird, and GaoFen-2 satellites as datasets. The GeoEye-1 dataset, taken on February 24, 2009, in Hobart, Australia, has an LR-MS spatial resolution of 1.84m and a PAN image spatial resolution of 0.46m. The GeoEye-1 dataset contains images of hills, trees, rural areas, and urban areas; the proposed method uses images from simulated data to construct the training dataset. The QuickBird dataset, taken on September 30, 2008, in Xi'an, China, has an LR-MS spatial resolution of 2.56m and a PAN image spatial resolution of 0.64m. The QuickBird dataset contains urban areas, including numerous buildings, roads, green belts, and other urban elements. The WorldView-4 image data, taken on April 5, 2017, in Acapulco, Mexico, has LR-MS and PAN image spatial resolutions of 1.24m and 0.31m, respectively. The World View-4 dataset depicts a city nestled between mountains and the sea, encompassing the urban area, densely forested hills, and sandy coastline. The city itself contains numerous buildings, roads, greenbelts, and other urban elements. The GaoFen-2 satellite, launched in August 2014, is China's first large low-orbit remote sensing satellite with its lifespan requirement increased from 3 years to 5-8 years. Its LR-MS spatial resolution is 3.24m, and its PAN image spatial resolution is 0.81m.

[0080] This embodiment inputs a full-scale LRMS image. F ∈R64×64×4 and full-scale PAN imageP F ∈R 256×256×1 .

[0081] The multispectral image is upsampled to the size of a panchromatic image (256×256×4), forming paired training data with the corresponding panchromatic image. The panchromatic image is then downsampled and then upsampled, keeping its size unchanged, to blur it. The difference between the blurred panchromatic image and the input panchromatic image is recorded as spatial information, which is then input into the spatial feature extraction module to obtain spatial information.

[0082] The blurred panchromatic image is stacked four times and the difference is calculated with a low-resolution multispectral image that has been upsampled four times to obtain spectral information. The spectral information is then input into the spectral feature extraction module to obtain spectral information.

[0083] (5.2) Input the full-size and small-size four-satellite images into the encoder for fusion.

[0084] Full-size four-satellite panchromatic and multispectral images are input into the encoder to obtain a full-size fused feature map.

[0085] (5.3) Reconstructing high-resolution multispectral images

[0086] F full-size fused feature map F Full2 ∈R 64×64×256 The input to the decoder undergoes convolution, upsampling, convolution, upsampling, and convolution operations. This, along with the spatial information obtained from the spatial feature extraction module and the spectral information obtained from the spectral feature extraction module, is input twice into the SCM block. Finally, it passes through three convolution blocks to reduce the number of channels to 4, yielding the target image HRMS∈R. 256×256×4 This description describes the process of image reconstruction across the entire size domain.

[0087] (5.4) Simulation Analysis

[0088] 1. Simulation environment:

[0089] PyCharm Community Edition 2022.1.2x64, NVIDIA GEFORCE RTX3090, Ubuntu18.04.

[0090] 2. Simulation content:

[0091] Simulation 1: This invention is used to fuse low spatial resolution multispectral images and high spatial resolution panchromatic images from the GeoEye-1, WorldView-4, QuickBird, and GaoFen-2 datasets. The results are as follows: Figure 3 As shown, where:

[0092] Figure 3 In the diagram, (1a), (2a), (3a), and (4a) are low spatial resolution multispectral images, each 64×64×4 in size; (1b), (2b), (3b), and (4b) are high spatial resolution panchromatic images, each 256×256×1 in size; (1c), (2c), (3c), and (4c) are Ground-Truth images, i.e., reference images of the fusion result, each 256×256×4 in size; (1d), (2d), (3d), and (4d) are images obtained by applying the present invention to... Figure 3 (a) and Figure 3 (b) The high spatial resolution multispectral image obtained after fusion, with a size of 256×256×4.

[0093] (1a)(1b)(1c)(1d) are images of GeoEye-1, (2a)(2b)(2c)(2d) are images of GaoFen-2, (3a)(3b)(3c)(3d) are images of QuickBird, and (4a)(4b)(4c)(4d) are images of WorldView-4.

[0094] Depend on Figure 3 visible, Figure 3 (d) Spatial detail information compared to Figure 3 (a) There is a significant improvement; the edges of roads and buildings are clearer, and... Figure 3 (b) In comparison, Figure 3 (d) has richer color information, thus demonstrating that the present invention can... Figure 3 (a) and Figure 3 (b) Achieve better integration.

[0095] Simulation 2: To demonstrate the effectiveness of the present invention, the method of the present invention and the existing BDSD transformation method were compared respectively.

[0096] The GS transform method and MTF-GLP-CBD transform method are used to fuse the images to be fused, and the fusion results are evaluated using objective metrics, as follows:

[0097] 1) The correlation coefficient CC represents the degree of preservation of spectral information. The result is in the range [0,1]. The closer the correlation coefficient is to 1, the more similar the fusion result is to the reference image.

[0098] 2) Root Mean Square Error (RMSE) represents the square root of the ratio of the square of the deviation between the predicted value and the true value to the number of observations n. The smaller the value, the better the fusion result.

[0099] 3) Global comprehensive error index ERG, based on RMSE, takes into account the scale relationship between the fused image and the observed image, with a range of [0,1]. The closer the index is to 1, the better.

[0100] 4) Spectral radian (SAM) indicates the degree of spectral distortion. The closer it is to 0, the better the fusion result.

[0101] 5) Global quality assessment indexQ represents the overall similarity of the images in space and spectrum. The result ranges from [0,1]. The larger the global quality assessment index, the more similar the fused image is to the reference image.

[0102] 6) Overall Image Quality Index (UIQI) indicates how close the fused image is to the reference image. The closer it is to 1, the better the fusion result.

[0103] The fusion results of the present invention and the prior art are evaluated from an objective perspective based on the above evaluation indicators. The results on the GeoEye-1 dataset are shown in Table 1 below.

[0104] Table 1. Objective evaluation of the fusion results of various methods on the GeoEye-1 dataset.

[0105]

[0106] Table 2. Objective evaluation of the fusion results of various methods on the QuickBird dataset.

[0107]

[0108] Table 3. Objective evaluation of the fusion results of various methods on the WorldView-4 dataset.

[0109]

[0110]

[0111] Table 4. Objective evaluation of the fusion results of various methods on the GaoFen-2 dataset.

[0112]

[0113] As can be seen from the table above, the correlation coefficient CC, global quality assessment indexQ, and overall image quality index UIQI of the present invention are all greater than the evaluation values ​​of the prior art. The root mean square error RMSE, global error score ERG, and spectral radian SAM are mostly smaller than the evaluation values ​​of the prior art. The above evaluation values ​​are better than the evaluation values ​​of the prior art. It can be seen that most of the objective evaluation indicators of the present invention are better than the objective evaluation indicators of the prior art.

[0114] This invention discloses a remote sensing image fusion method based on a deep neural network to improve spatial and spectral resolution, primarily addressing the problems of spatial information blurring and spectral distortion in existing technologies. Data from four channels are trained simultaneously to form a unified deep neural network, which can accurately fuse spatial and spectral information and can be used in remote sensing fields such as target recognition, land cover classification, and environmental monitoring.

[0115] Example 2

[0116] This embodiment provides a remote sensing image fusion system based on a deep neural network, including:

[0117] Data acquisition module: used to acquire panchromatic and multispectral images, perform preprocessing, and obtain the image set to be fused;

[0118] Data processing module: used to obtain spatial feature images and spectral feature images based on the image set to be fused and the panchromatic image;

[0119] Feature acquisition module: used to input the image set to be fused, spatial feature image and spectral feature image into the first fusion module, spatial information extraction module and spectral information extraction module respectively to obtain the first fused feature, spatial feature and spectral feature;

[0120] Image reconstruction module: used to input the first fused features, spatial features and spectral features into the decoder to reconstruct a high-resolution multispectral image.

[0121] The modules involved in the above embodiment 2 correspond to those in embodiment 1. For specific implementation details, please refer to the relevant description section of embodiment 1.

[0122] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A remote sensing image fusion method based on deep neural networks, characterized in that, include: The process involves acquiring panchromatic and multispectral images, performing preprocessing to obtain a set of images to be fused, specifically including: upsampling the multispectral image by four times to obtain a first multispectral image; downsampling and upsampling the panchromatic image by four times to obtain a first panchromatic image; and combining the first multispectral image and the first panchromatic image to form the set of images to be fused. Based on the image set to be fused and the panchromatic image, spatial feature images and spectral feature images are obtained, specifically including: subtracting the first panchromatic image from the corresponding panchromatic image to obtain the spatial feature image; stacking the first panchromatic image four times in the channel dimension to form a four-channel image, subtracting it from the corresponding first multispectral image to obtain the spectral feature image; The image set to be fused, the spatial feature image, and the spectral feature image are respectively input into the first fusion module, the spatial information extraction module, and the spectral information extraction module to obtain the first fused feature, the spatial feature, and the spectral feature. The image set to be fused is stacked along the channel dimension and input into the first fusion module to obtain the first fusion feature; the first fusion module includes a first neural network, an encoder and a CD module; The first neural network includes a convolutional layer with a kernel size of 3×3 and a stride of 1, a max pooling layer with a window size of 2×2 and a stride of 2, a convolutional layer with a kernel size of 3×3 and a stride of 1, a max pooling layer with a window size of 2×2 and a stride of 2, and a convolutional layer with a kernel size of 3×3 and a stride of 1. The CD module includes a max pooling layer with a window size of 2×2 and a stride of 2, a convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1, a convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1, a max pooling layer with a window size of 2×2 and a stride of 2, and three hidden fully connected layers. The first fusion feature, spatial feature, and spectral feature are input into the decoder to reconstruct a high-resolution multispectral image.

2. The remote sensing image fusion method based on deep neural networks as described in claim 1, characterized in that, The spatial feature image is input into the spatial information extraction module to obtain spatial features; specifically, this includes: Perform FFT transformation on the spatial feature image to obtain amplitude and phase information; After convolving the amplitude information, an inverse FFT transformation is performed with the phase information to obtain the spatial features.

3. The remote sensing image fusion method based on deep neural networks as described in claim 1, characterized in that, The spectral feature image is input into the spectral information extraction module to obtain spectral features; specifically including: The first spectral feature is obtained by performing a convolution operation on the spectral feature image; The first spectral feature is subjected to two dimensional transformations, and the results of the two dimensional transformations are multiplied to obtain the spectral feature.

4. The remote sensing image fusion method based on deep neural networks as described in claim 1, characterized in that, The first fused feature, spatial feature, and spectral feature are input into the decoder to reconstruct a high-resolution multispectral image, specifically including: The first fusion feature is transformed in dimension and multiplied with the spectral feature matrix to obtain the first result; After performing dimensionality transformation and convolution operations on the first result, it is multiplied element-wise with the spatial features and then added to obtain the second result; The second result is processed through three convolutional layers with a kernel size of 3×3 and a stride of 1 to obtain a high-resolution multispectral image.

5. A remote sensing image fusion system based on a deep neural network, employing the remote sensing image fusion method based on a deep neural network as described in any one of claims 1-4, characterized in that, include: Data acquisition module: used to acquire panchromatic and multispectral images, perform preprocessing, and obtain the image set to be fused; Data processing module: used to obtain spatial feature images and spectral feature images based on the image set to be fused and the panchromatic image; Feature acquisition module: used to input the image set to be fused, spatial feature image and spectral feature image into the first fusion module, spatial information extraction module and spectral information extraction module respectively to obtain the first fused feature, spatial feature and spectral feature; Image reconstruction module: used to input the first fused features, spatial features and spectral features into the decoder to reconstruct a high-resolution multispectral image.

Images