Remote sensing image fusion method and system based on semi-supervised deep neural network

Abstract

The present disclosure provides a kind of remote sensing image fusion method and system based on semi-supervised deep neural network, including obtaining high spatial resolution panchromatic image and low spatial resolution multispectral image to be fused, and it is preprocessed;The panchromatic image and multispectral image are input into double branch network, respectively extract the spatial information of panchromatic image and the spectral information feature of multispectral image;The feature map of the extracted spatial information and spectral information is stacked first after resolution perception, then the stacked feature map is fused and reconstructed, and the resolution perception result is injected, to obtain fusion image.

Description

Technical Field

[0001] This disclosure relates to the field of remote sensing image fusion technology, specifically to a remote sensing image fusion method and system based on a semi-supervised deep neural network. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] In recent years, numerous high-resolution optical Earth observation satellites, such as QuickBird, GeoEye, WorldView-2, and GaoFen-2, have been launched, providing researchers in the remote sensing field with a wealth of data applicable to various research areas, such as agriculture, land surveying, and environmental monitoring. In remote sensing systems, satellites can acquire two types of images in entirely different ways: multispectral images (MS) and panchromatic images (PAN). Multispectral images have high spectral resolution but low spatial resolution, limited by onboard storage and broadband transmission. Conversely, panchromatic images have low spectral resolution but high spatial resolution due to their large instantaneous field of view. Panchromatic sharpening (i.e., panchromatic and multispectral image fusion) aims to generate high spatial resolution multispectral images by combining the spatial and spectral information of panchromatic and multispectral images, providing a good solution to alleviate the aforementioned problems.

[0004] Traditional fusion methods can generally be divided into two categories: 1) Component Substitution (CS) based methods: The basic assumption of the CS method is that the geometric details of the MS image exist in its structural components, which can be obtained by transforming it into a new space. Then, the structural components are replaced or partially replaced using a histogram-matched version of the PAN to inject spatial information. Finally, panchromatic sharpening is achieved through inverse transform. 2) Multi-resolution Analysis (MRA) based methods: The MRA-based method assumes that the spatial information missing in the MS can be inferred from the high frequencies of the corresponding PAN image. To sharpen the multispectral image, multi-resolution analysis algorithms, such as Discrete Wavelet Transform (DWT) and Curvelet Transform, are applied to the multispectral image to extract high-frequency information, which is then injected into the corresponding MS image.

[0005] Recently, deep learning has achieved great success in various fields, such as computer vision, pattern recognition, and image processing. Many researchers have also introduced deep learning methods based on convolutional neural networks (CNNs) into pancolor sharpening tasks, such as PNN, PanNet, and PSGAN, but there is still no good method for pancolor sharpening. Summary of the Invention

[0006] To address the aforementioned issues, this disclosure proposes a remote sensing image fusion method and system based on semi-supervised deep neural networks. In the process of reconstructing high-resolution multispectral images, a resolution sensing module is used to capture cross-resolution information, thereby further improving image quality.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions:

[0008] Remote sensing image fusion methods based on semi-supervised deep neural networks include:

[0009] Acquire the high spatial resolution panchromatic image and the low spatial resolution multispectral image to be fused, and preprocess them;

[0010] The panchromatic image and the multispectral image are input into a dual-branch network to extract the spatial information of the panchromatic image and the spectral information features of the multispectral image, respectively.

[0011] After stacking the feature maps of extracted spatial and spectral information, resolution sensing is first performed. Then, when fusing and reconstructing the stacked feature maps, the resolution sensing result is injected to obtain the fused image.

[0012] According to other embodiments, the present disclosure adopts the following technical solutions:

[0013] A remote sensing image fusion system based on semi-supervised deep neural networks includes:

[0014] The image acquisition module is used to acquire the high spatial resolution panchromatic image and the low spatial resolution multispectral image to be fused, and to preprocess them.

[0015] The feature extraction module is used to input the panchromatic image and the multispectral image into a dual-branch network to extract the spatial information of the panchromatic image and the spectral information features of the multispectral image, respectively.

[0016] The resolution sensing module is used to stack the feature maps of the extracted spatial and spectral information and then perform resolution sensing.

[0017] The image fusion and reconstruction module is used to inject resolution-aware results when fusing and reconstructing stacked feature maps to obtain a fused image.

[0018] Furthermore, it also includes a domain adversarial module;

[0019] The feature extraction module includes an encoder and a decoder, which utilize the spatial information of the panchromatic image and the spectral information of the multispectral image to extract spatial features and spectral features respectively using two branch networks to obtain complementary information.

[0020] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0021] First, this disclosure places panchromatic images and multispectral images in a dual-branch network for step-by-step processing, extracting spatial and spectral information respectively. This allows the disclosure to obtain accurate spatial and spectral information, which is beneficial for the use of subsequent image processing techniques and improves the utilization rate of spatial and spectral information.

[0022] Second, since there should be interaction or communication between images of different resolutions during the learning or training process, the learned features become more complementary. By injecting cross-resolution information into the decoder through a resolution-aware module, the reconstructed high-resolution multispectral image contains more detailed information. Attached Figure Description

[0023] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0024] Figure 1 This is a flowchart illustrating the method implementation in the embodiments of this disclosure;

[0025] Figure 2 This is a detailed diagram showing the configuration of each network module in the embodiments of this disclosure;

[0026] Figure 3 This is a comparison diagram showing the results of fusing low spatial resolution multispectral images and high spatial resolution panchromatic images using the embodiments of this disclosure;

[0027] Figure 3 (a) is a low spatial resolution multispectral image;

[0028] Figure 3 (b) is a high spatial resolution panchromatic image;

[0029] Figure 3 (c) is the Ground-Truth, i.e., the reference image of the fusion result;

[0030] Figure 3 (d) For the purpose of adopting this disclosure Figure 3 (a) and Figure 3 (b) The high spatial resolution multispectral image obtained after fusion; Detailed implementation method:

[0031] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0033] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0034] Example 1

[0035] One embodiment of this disclosure provides a remote sensing image fusion method based on a semi-supervised deep neural network, comprising the following steps:

[0036] S101: Obtain the high spatial resolution panchromatic image and the low spatial resolution multispectral image to be fused, and preprocess them;

[0037] Specifically, Bicubic interpolation upsampling is used to upsample the multispectral image to the size of the panchromatic image, and a low-resolution multispectral image and panchromatic image are obtained by performing a four-fold upsampling.

[0038] S102: Input the panchromatic image and the multispectral image into the dual-branch network to extract the spatial information of the panchromatic image and the spectral information features of the multispectral image, respectively;

[0039] Specifically, bicubic interpolation upsampling is used to upsample the multispectral image to the size of a panchromatic image. The low-resolution multispectral image and the panchromatic image, which are upsampled four times, are then input into the dual-branch network.

[0040] S103: After stacking the feature maps of the extracted spatial and spectral information, resolution perception is first performed. Then, when fusing and reconstructing the stacked feature maps, the resolution perception result is injected to obtain the fused image.

[0041] Specifically, the stacking results are input into the resolution extractor for resolution sensing, and the resolution sensing results are output.

[0042] The stacked results of full-size and small-size are input into the encoder for fusion. After passing through two convolutional blocks, the full-size fused feature map and the small-size fused feature map are obtained.

[0043] When reconstructing a high-resolution multispectral image, the fused feature map is input into the decoder. It first passes through the first convolutional module to obtain the feature map. Then, the results of the first and second convolutions are stacked and input into the third convolutional block. The output then enters the fourth convolutional block to increase the number of channels. The feature map output from the dual-branch network and the result output from the fourth convolutional block are stacked and input into the fifth convolutional block. Finally, it passes through the sixth convolutional block to reduce the number of channels, thus obtaining the reconstructed target image.

[0044] For example, the specific implementation process of the technical solution in this application is as follows:

[0045] As one implementation method, in order to obtain high spatial resolution multispectral images, the specific model construction of this disclosure is as follows:

[0046] (1) Input image

[0047] Input a full-size panchromatic image (Full-scale PANimageP) F ∈R 256×256×1 and multispectral images Full-scale LRMS image L F ∈R 64×64×4 Input a small-sized panchromatic image. (Reduced-scale PAN image P) R ∈R 64×64×1 Reduced-scale LRMS image L (multispectral image) R ∈R 16×16×4 .

[0048] The preprocessing process before input is as follows: The full-size multispectral image L... F ∈R 64×64×4 and small-sized multispectral images L R ∈R 16×16×4 Image L was obtained by performing upsampling by four times. F ∈R 256×256×4 and L R ∈R 64×64×4 .

[0049] (2) Construct a semi-supervised deep learning network core module, which consists of four parts: encoder, decoder, resolution perception module and domain adversarial module.

[0050] To fully utilize the spatial information of the panchromatic image (PAN) and the spectral information of the multispectral image (LRMS), an encoder module, consisting of two branch networks, extracts features from PAN and LRMS respectively to obtain complementary information. Each branch consists of two convolutional layers with a 3×3 kernel and a stride of 1. The spectral branch has 4 input channels, and the spatial branch has 1. Following this is a LeakyReLU activation function and a downsampling layer, which uses a 2×2 kernel and a stride of 2 for downsampling. After passing through the two branch networks, the feature maps are first stacked and then fused using two subsequent convolutional blocks. The first convolutional block consists of two 3×3 kernel layers with a stride of 1, and the second convolutional block uses a 2×2 kernel layer with a stride of 2 to downsample the image again.

[0051] The features extracted by the encoder are reconstructed in the decoder to obtain a high spatial resolution multispectral image (HRMS). The decoder contains six modules. The first two modules perform convolution operations with a 3×3 kernel and a stride of 1. The outputs of the first and second modules are then stacked and input into the third convolutional block. The fourth convolutional block performs an upsampling operation using a transposed convolution with a 2×2 kernel and a stride of 2. To obtain richer spatial and spectral information, the upsampled result and the downsampled result from the dual-branch network are stacked to obtain a 256×256×192 image, which is then input into the fifth module to reduce the number of channels to 64. Finally, the image is input into the sixth module to obtain the desired high spatial resolution multispectral image HRMS∈R. 256×256×4 .

[0052] After stacking the feature maps of the extracted spatial and spectral information, resolution sensing is performed using a resolution sensing module. This module takes the stacked results of the encoder's two branches as input and its main structure consists of two layers of two-dimensional convolutions and two layers of LeakyReLU activation functions. The first convolution module consists of two layers of convolution operations with a kernel size of 3×3 and a stride of 1, using LeakyReLU activation function. The second convolution module consists of one layer of convolution operations with a kernel size of 2×2 and a stride of 2, reducing the resolution to half of its original value.

[0053] Specifically, the method for obtaining the results of the sensory discrimination is as follows:

[0054] The output of the resolution-aware module and the output of the encoder are multiplied element-wise, and then added to the output of the resolution-aware module and injected into the downstream decoder, so that the HRMS reconstruction process includes more resolution information.

[0055] Since both full-size and small-size images were acquired, a domain adversarial module was constructed to distinguish between the full-size and small-size domains. This module comprises two linear layers, a batch normalization layer, and a softmax activation function. The domain adversarial module takes the encoder output as input, passes through the first linear layer to output 100 samples, performs batch normalization on the samples, and then passes through another linear layer to set the output samples to 2, resulting in a tensor of shape (batchsize, 2). Finally, this tensor is input to the softmax function, which fixes the result between (0, 1) for use with the NLLLoss loss function during training.

[0056] As one implementation method, in order to obtain high spatial resolution multispectral images, the cross-resolution model training is as follows:

[0057] Input a full-size, full-color image P F ∈R 256×256×1 and multispectral images L F ∈R 64×64×4 First, the multispectral image is upsampled by 4 times using Bicubic interpolation upsampling to obtain image L. F ∈R 256×256×4 Next, the panchromatic image P F ∈R 256 ×256×1 and upsampled multispectral image L F ∈R 256×256×4 The common input encoder, after passing through a two-branch network, yields B. PAN ∈R 128 ×128×64 and B MS ∈R 128×128×64 Then stack the two to get C Full1 ∈R 128×128×128 Finally, the stacked result C Full1 ∈R 128×128×128 The input is fed into the next two fusion blocks to obtain F. Full2 ∈R 64×64×256 The resolution-aware module is stacked to produce C. Full1 ∈R 128×128×128 As input, it is used to extract resolution information (β and γ), and then the resolution information and the fused block F are used. Full2 ∈R 64×64×256 The images are input together into the decoder, and finally the reconstructed full-size fused image F is obtained. F ∈R 256×256×4 .

[0058] Specifically, the process in the decoder is as follows:

[0059] F input =β⊙F Full2 +γ

[0060] ⊙ represents element-wise multiplication, β and γ are the extracted resolution information, and F Full2 It is the output of the encoder, F input It is the input to the decoder.

[0061] The cross-resolution full-scale fused image is C. F ∈R 256×256×4 :

[0062] The acquisition process is as follows: The generation process of the cross-resolution full-size fused image is similar to that of the full-size fused image, except that the cross-resolution fused image involves upsampling the resolution information of the small-size image extracted by the resolution sensing module and injecting it into the reconstruction process of the full-size image, so that the cross-resolution full-size fused image C... F ∈R 256×256×4 It contains low-resolution information.

[0063] The specific process is as follows:

[0064] F input =β↑⊙F Full2 +γ↑

[0065] ↑ indicates an upsampling operation. The low-resolution information β and γ extracted from the small-sized image by the resolution sensing module needs to be upsampled before being combined with the decoder output F. Full2 ∈R 64×64×256 Combine them, and then input them into the decoder.

[0066] The reduced-scale fused image is F. R ∈R 64×64×4 The acquisition process is as follows:

[0067] Input a small-sized panchromatic image P R ∈R 64×64×1 and multispectral images L R ∈R 16×16×4 First, the multispectral image is upsampled by 4 times using Bicubic interpolation upsampling to obtain image L. R ∈R 64×64×4 Secondly, the panchromatic image P R ∈R 64×64×1 and upsampled multispectral image L R ∈R 64×64×4 The common input encoder, after passing through a two-branch network, yields B. PAN∈R 32×32×64 and B MS ∈R 32×32×64 Then, after stacking, we get C. Redu1 ∈R 32×32×128 Finally, the stacked result C Redu1 ∈R 32×32×128 The input is fed into the next two fusion blocks to obtain F. Redu2 ∈R 16×16×256 The resolution-aware module is stacked to produce C. Redu1 ∈R 32×32×128 As input, it is used to extract resolution information, and then the resolution information and the fused block F are... Redu2 ∈R 16×16×256 The two images are input together into the decoder to obtain a small-sized fused image F. R ∈R 64×64×4 The reconstruction process for small-sized fused images is similar to that for full-size images, except that the input and output image sizes differ.

[0068] F input =β⊙F Redu2 +γ

[0069] In the small-size fused image reconstruction process, the resolution information β and γ are derived from the small-size image, combined with the encoder output in the same manner, and then input into the decoder to obtain the small-size fused image F. R ∈R 64×64×4 .

[0070] Then obtain the cross-resolution small-size fused image C R ∈R 64×64×4 The resolution information in this process comes from the full-scale image, which is then downsampled and injected into the decoder, so that the small-sized fused image contains high-resolution information.

[0071] F input =β↓⊙F Redu2 +γ↓

[0072] ↓ indicates a downsampling operation. The high-resolution information β and γ extracted from the full-size image by the resolution sensing module need to be downsampled before being combined with the decoder output F. Redu2 ∈R 16×16×256 The images are combined and then input into the decoder to obtain a small-sized fused image C with cross-resolution. R ∈R 64×64×4 .

[0073] The domain adversarial module is based on the fusion block F Redu2 ∈R 16×16×256 and fusion block F Full2 ∈R 64×64×256The four-fold downsampling result is used as input to separate the full-size domain and the small-size domain, making their features more distinct. Furthermore, to reduce the number of parameters and alleviate computational burden, the encoder, decoder, and resolution sensing module in this model share parameters.

[0074] Furthermore, the loss function is constructed. This model uses two loss functions: MSELOSS and NLLLOSS. The constructed loss function is as follows:

[0075]

[0076]

[0077]

[0078]

[0079] Where H is the input full-size multispectral image, and H is used as the reduced-scale reference image for the small-size fusion result. H∈R 64×64×4 ;F R It is a reduced-scale fused image, i.e., a small-sized fused image; C R It is a Cross-resolution Reduced-scale Fused image, i.e., a cross-resolution small-size fused image; F F It is a full-scale fused image; C F It is a Cross-resolution Full-scale Fused image; L F It is a full-scale LRMS, i.e., a full-size low-resolution multispectral image. D(·) is the bicubic interpolation space downsampling operator.

[0080] NLLLoss is similar to the cross-entropy loss function and is often used in classification problems. However, NLLLoss needs to be combined with operations such as softmax and log as a loss function, resulting in:

[0081] Loss5=NLLLoss(full_domain, label0)

[0082] Loss6=NLLLoss(redu_domain, label1)

[0083] Where full_domain and redu_domain are the outputs of the model (the result after softmax), label0 is a tensor with all zeros, and label1 is a tensor with all 1s.

[0084] The total loss function is then:

[0085] Loss=Loss1+Loss2+Loss3+Loss4+Loss5+Loss6

[0086] Example 2

[0087] One embodiment of this disclosure provides a remote sensing image fusion system based on a semi-supervised deep neural network, including: an image acquisition module for acquiring a high spatial resolution panchromatic image and a low spatial resolution multispectral image to be fused, and preprocessing them;

[0088] The feature extraction module is used to input the panchromatic image and the multispectral image into a dual-branch network to extract the spatial information of the panchromatic image and the spectral information features of the multispectral image, respectively.

[0089] The resolution sensing module is used to stack the feature maps of the extracted spatial and spectral information and then perform resolution sensing.

[0090] The image fusion and reconstruction module is used to inject resolution-aware results when fusing and reconstructing stacked feature maps to obtain a fused image.

[0091] Furthermore, it also includes a domain adversarial module;

[0092] The feature extraction module includes an encoder and a decoder, which are used to extract spatial features and spectral features respectively using two branch networks to obtain complementary information by utilizing the spatial information of the panchromatic image and the spectral information of the multispectral image.

[0093] As one embodiment, the following method steps are specifically implemented using the above system modules:

[0094] The dataset disclosed herein uses low spatial resolution multispectral images and high spatial resolution panchromatic images taken by the GeoEye-1 satellite in Hobart, Australia in February 2009.

[0095] Step 1: Input full-scale LRMS images respectively. F ∈R 64×64×4 And full-scale PAN image P F ∈R 256×256×1 Reduced-scale LRMS imageL (small-sized multispectral image)R ∈R 16×16×4 And panchromatic image Reduced-scale PAN image P R ∈R 64×64×1 The positions of the full-size and small-size images correspond, with the only difference being high and low resolution. The full-size multispectral image serves as the reference image for the small-size fusion result HRMS.

[0096] In this process, Bicubic interpolation upsampling is used to upsample the multispectral image to the size of a panchromatic image, i.e., the full size is 256×256×4 and the small size is 64×64×4, forming paired training data with the corresponding panchromatic image.

[0097] The low-resolution multispectral image and panchromatic image, upsampled by four times, are input into the dual-branch network to obtain B. PAN ∈R 128 ×128×64 and B MS ∈R 128×128×64 Then stack the two to get C Full1 ∈R 128×128×128 Similarly, there are also small-sized stacking results C. Redu1 ∈R 32×32×128 .

[0098] Step 2: Input the full-size and small-size stacking results into the encoder for fusion.

[0099] The full-size stacking result C Full1 ∈R 128×128×128 The result of stacking small sizes C Redu1 ∈R 32×32×128 The input is fed into the encoder and passes through two convolutional blocks to obtain the full-size fused feature map F. Full2 ∈R 64×64×256 and small-sized fusion feature map F Redu2 ∈R 16×16×256 .

[0100] Step 3: Conduct domain confrontation

[0101] Function full-size feature map F Full2 ∈R 64×64×256 A bicubic interpolation with a 4x downsampling was performed to obtain a result of size 16×16×256, which was then fused with the smaller feature map F. Redu2 ∈R 16×16×256 Implement domain adversarial mechanisms. The domain adversarial module consists of two linear layers, a batch normalization layer, and a softmax activation function. The softmax activation function maps domain information to the range of 0 to 1. Then, combined with NLLOSs and 0 and 1 labels, it separates the full-size domain from the small-size domain.

[0102] Step 4: Extract resolution factor

[0103] The stacked result C Full1 ∈R 128×128×128 The input is fed into the resolution extractor, and the output is F. β ∈R 64×64×256 and F γ ∈R 64×64×256 Similarly, there is also the small resolution factor R. β ∈R 16×16×256 and R γ ∈R 16×16×256 It is worth noting that β and γ here are not simple Arabic numerals, but four-dimensional tensors that contain resolution information.

[0104] Step 5: Reconstruct high-resolution multispectral images

[0105] Function full-size feature map F Full2 ∈R 64×64×256 The input to the decoder first passes through the first convolutional module, resulting in a 128×128×128 feature map. Then, the results of the first and second convolutions are stacked and input into the third convolutional block, where the image resolution remains 128×128. The image then enters the fourth convolutional block, increasing the number of channels to 128. Finally, the feature map B, containing rich spatial and spectral information from the dual-branch network, is input. PAN ∈R 128×128×64 and B MS ∈R 128×128×64 The result of convolution block 4 is stacked and fed into the fifth convolution block, which is a transposed convolution responsible for upscaling the image to 256×256. Finally, it passes through the sixth convolution block to reduce the number of channels to 4, resulting in the target image HRMS∈R. 256×256×4 This description describes the image reconstruction process in the full-size domain. The reconstruction process in the small-size domain is similar, except that the resolution is 1 / 4 of the full-size domain.

[0106] The effects of this disclosure can be further illustrated by the following simulation.

[0107] 1. Simulation environment:

[0108] PyCharm Community Edition 2022.1.2x64, NVIDIA GEFORCE RTX3090, Ubuntu18.04.

[0109] 2. Simulation content:

[0110] This disclosure is used to fuse a low spatial resolution multispectral image and a high spatial resolution panchromatic image taken by the GeoEye-1 satellite in Hobart, Australia in February 2009. The result is as follows: Figure 3 As shown, where:

[0111] Figure 3 (a) is a low spatial resolution multispectral image, with a size of 64×64×4.

[0112] Figure 3 (b) is a high spatial resolution panchromatic image with a size of 256×256×1.

[0113] Figure 3 (c) is the Ground-Truth, i.e., the reference image of the fusion result, with a size of 256×256×4.

[0114] Figure 3 (d) To use the present invention for Figure 3 (a) and Figure 3 (b) The high spatial resolution multispectral image obtained after fusion, with a size of 256×256×4.

[0115] Depend on Figure 3 visible, Figure 3 (d) Spatial detail information compared to Figure 3 (a) There is a significant improvement; the edges of roads and buildings are clearer, and... Figure 3 (b) In comparison, Figure 3 (d) has richer color information, thus demonstrating that the present invention can... Figure 3 (a) and Figure 3 (b) Achieve better integration.

[0116] Simulation 2: To demonstrate the effectiveness of this invention, the method of this invention and existing technologies such as BDSD transform, AWLP transform, Indusion transform, SVT transform, VPLGC transform, and other deep neural network methods such as PNN and PanNet were compared. Figure 2 (a) and Figure 2 (b) The images to be fused are fused, and the fusion results are evaluated using objective metrics, as follows:

[0117] 1) The correlation coefficient CC represents the degree of preservation of spectral information. The result is in the range [0,1]. The closer the correlation coefficient is to 1, the more similar the fusion result is to the reference image.

[0118] 2) Root Mean Square Error (RMSE) represents the square root of the ratio of the square of the deviation between the predicted value and the true value to the number of observations n. The smaller the value, the better the fusion result.

[0119] 3) Global comprehensive error index ERG, based on RMSE, takes into account the scale relationship between the fused image and the observed image, with an interval of [0,1]. The closer the index is to 1, the better.

[0120] 4) Spectral radian (SAM) indicates the degree of spectral distortion. The closer it is to 0, the better the fusion result.

[0121] 5) Global quality assessment indexQ represents the overall similarity of the images in space and spectrum. The result ranges from [0,1]. The larger the global quality assessment index, the more similar the fused image is to the reference image.

[0122] 6) Overall Image Quality Index (UIQI) indicates how close the fused image is to the reference image. The closer it is to 1, the better the fusion result.

[0123] The integration results of the present invention and the prior art were evaluated from an objective evaluation perspective based on the above evaluation indicators, and the results are shown in Table 1.

[0124] Table 1. Objective evaluation of the fusion results of various methods

[0125]

[0126] As shown in Table 1, the correlation coefficient CC, global quality assessment indexQ, and overall image quality index UIQI of this disclosure are all greater than the evaluation values ​​of the prior art, while the root mean square error RMSE, global error score ERG, and spectral radian SAM are all less than the evaluation values ​​of the prior art. All of the above evaluation values ​​are better than the evaluation values ​​of the prior art. It can be seen that most of the objective evaluation indicators of this disclosure are better than the objective evaluation indicators of the prior art.

[0127] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A method for remote sensing image fusion based on semi-supervised deep neural network, characterized in that, The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. 2.The method of claim 1, wherein, The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. 3.The method of claim 1, wherein, The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. 4.The method of claim 3, wherein, The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. 5.The method of claim 1, wherein, The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. 6.The method of claim 1, wherein, The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. 7.The method of claim 1, wherein, The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. 8.The method of claim 1, wherein, The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. 9.The method of claim 7, wherein, The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network.

10. A remote sensing image fusion system based on semi-supervised deep neural networks, characterized in that, The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote sensing image fusion method based on a dual-branch network. The application relates to a high-resolution remote An image acquisition module is configured to acquire high spatial resolution panchromatic images and low spatial resolution multispectral images to be fused and pre-process the images; A feature extraction module is configured to input the panchromatic images and the multispectral images into a dual-branch network to extract spatial information of the panchromatic images and spectral information of the multispectral images, respectively. A resolution perception module is configured to stack the extracted spatial information and spectral information feature maps and then perform resolution perception. An image fusion reconstruction module is configured to inject a resolution perception result when performing fusion reconstruction on the stacked feature maps to obtain a fused image. Specifically, the resolution perception module takes the stack of the dual-branch encoder results as input, and its main structure includes two two-dimensional convolution layers and two LeakyRelu activation functions. The first convolution module includes two convolution layers with a 3*3 kernel and a step of 1, and the activation function is LeakyRelu. The second convolution module includes one convolution layer with a 2*2 kernel and a step of 2, so that the resolution is reduced to 1 / 2 of the original resolution. The output result of the resolution perception module is multiplied element by element with the output result of the encoder, and then added to the output result of the resolution perception module, and then injected into the downstream decoder, so that the HRMS reconstruction process contains more resolution information.

11. The semi-supervised deep neural network-based remote sensing image fusion system of claim 10, wherein, Further comprising a domain adversarial module. The feature extraction module includes an encoder and a decoder, which are configured to extract spatial features and spectral features by using the spatial information of the panchromatic images and the spectral information of the multispectral images, and obtain complementary information.

Images