Multi-branch multi-scale laplacian progressive remote sensing image fusion method and system

Abstract

The application discloses a kind of multi-branch multiscale laplace progressive remote sensing image fusion method and system, input multi-branch multiscale laplace progressive remote sensing image fusion network to be fused and processed remote sensing image, carry out image fusion;The multi-branch multiscale laplace progressive remote sensing image fusion network includes the panchromatic branch f pb for extracting panchromatic image information, mb The multispectral branch f For extracting multispectral image and the fusion branch for fusing and reconstructing both information;The method proposed in the application can effectively alleviate the distortion and unreal phenomenon of fusion image, compared with other mainstream methods, the method proposed in the application is in the small order of magnitude range while obtaining good fusion effect, the parameter and running speed of model, has certain practical significance;Meanwhile, the method proposed in the application only uses basic convolution layer and ReLU activation function layer, with strong scalability and potential.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image fusion, and relates to a remote sensing image fusion method and system, in particular to a multi-branch multi-scale Laplacian progressive remote sensing image fusion method and system. BACKGROUND

[0002] Due to the limitation of factors such as radiation energy and data capacity of a remote sensing satellite imaging system, a trade-off between spatial and spectral resolution needs to be made when an image is acquired. For example, most remote sensing satellites can only provide a panchromatic image with high spatial resolution and a multispectral image with high spectral resolution at the same time, and cannot directly provide a multispectral image with high spatial resolution. Therefore, in order to overcome this hardware limitation to obtain a multispectral image with high spatial resolution and promote the development of subsequent visual tasks, a panchromatic sharpening technique emerges as the times require, that is, complementary information of a panchromatic image and a multispectral image is fused to obtain a multispectral image with high spatial resolution.

[0003] In recent years, the rapid development of satellite sensors, mainly reflected in the number of spectral bands and the difference of spectral range of panchromatic and multispectral images. Specifically, this change has developed from multispectral images with only 3 bands and panchromatic images covering only the visible spectrum (SPOT-1 satellite, SPOT-2 satellite, etc.) to multispectral images with 4 bands and panchromatic images covering visible and near-infrared spectrum (QuickBird satellite, Gaofen-2 satellite, etc.) and multispectral images with more than 8 bands (WorldView-2, WordView-3 satellite, etc.). These hardware advances have driven the development of panchromatic sharpening techniques to some extent. In addition, as the demand for high spatial resolution multispectral images gradually increases in practical engineering and remote sensing applications, a large number of researchers have turned to the study of panchromatic sharpening techniques, introducing some related emerging theories and mathematical research hotspots into the field of panchromatic sharpening, such as the emergence of a large number of panchromatic sharpening methods based on sparse representation and deep learning. As of now, panchromatic sharpening methods can be roughly divided into four categories: component substitution (CS) based methods, multi-resolution analysis (MRA) based methods, variational optimization (VO) based methods, and deep learning (DL) based methods. In addition, the first three methods can be unified as traditional methods, which have been widely applied due to their efficiency and simplicity, such as in the software ENVI, Earth Resource Data Analysis System Imagine, and PCIGeomatica. In recent years, due to the powerful fitting ability of convolutional neural networks for large data, deep learning-based methods have received a lot of attention. Deep learning-based methods can be divided into single-branch network structures based on super-resolution ideas, double-branch structures that extract panchromatic images and multispectral images separately, and progressive structures based on the above two structures. It is worth mentioning that the fusion results of these deep learning-based methods are better than traditional methods in some objective indicators and human subjective perception.

[0004] However, the prior art has the following disadvantages:

[0005] 1. Most traditional methods cause spatial distortion and spectral distortion in the fused image due to their relative simplicity;

[0006] 2. Some deep learning methods also cause spectral distortion;

[0007] 3. Some deep learning methods based on super-resolution ideas cause local regions of the fused image to be inconsistent with the true situation. SUMMARY

[0008] The present application provides a multi-branch multi-scale Laplacian progressive remote sensing image fusion method and system to solve the image distortion phenomenon of the existing traditional method and the local non-true image of the image fused by the partial deep learning method in the field of remote sensing image fusion.

[0009] The technical scheme of the method is: a multi-branch multi-scale Laplacian progressive remote sensing image fusion method, inputting the remote sensing image to be fused into a multi-branch multi-scale Laplacian progressive remote sensing image fusion network for image fusion.

[0010] The multi-branch multi-scale Laplacian progressive remote sensing image fusion network comprises a panchromatic branch for extracting panchromatic image information , a multispectral branch for extracting multispectral images , and a fusion branch for fusing and reconstructing the information of both.

[0011] The panchromatic branch is used to sequentially downsample the panchromatic image to 2 times and 4 times of the original panchromatic image P , respectively denoted as , .

[0012] The multispectral branch is used to sequentially upsample the multispectral image to 2 times and 4 times of the original multispectral image M , respectively denoted as , .

[0013] After the and M are spliced in the channel dimension and injected into the fusion branch, the fusion branch extracts the input features through a recursive convolution module and up-samples the feature map by 2 times, and then injects the and , and then fuses the spliced features through a recursive convolution module and up-samples the feature map by 2 times again, and then injects the P and , and finally obtains a high spatial resolution multispectral image after passing through a convolution layer.

[0014] As a preferred, the panchromatic branch comprises a first recursive convolution module layer, a first ReLU activation function layer, a first MaxPooling layer, a first convolution layer, a second recursive convolution module layer, a second ReLU activation function layer, a second MaxPooling layer, and a second convolution layer connected in sequence; the first MaxPooling layer outputs after passing through the first convolution layer, denoted as The output of the second MaxPooling layer is then passed through the second convolutional layer and output as follows: .

[0015] Preferably, the multispectral branch It includes a third recursive convolutional module layer, a third ReLU activation function layer, a first upsample layer, a fourth ReLU activation function layer, a third convolutional layer, a fourth recursive convolutional module layer, a fifth ReLU activation function layer, a second upsample layer, a sixth ReLU activation function layer, and a fourth convolutional layer, all connected in sequence. The output of the fourth ReLU activation function layer is then passed through the third convolutional layer and output as follows: The output of the sixth ReLU activation function layer is output after passing through the fourth convolutional layer, denoted as... .

[0016] Preferably, the fusion branch includes a first Concat layer, a fifth convolutional layer, a seventh ReLU activation function layer, a fifth recursive convolutional module, a sixth convolutional layer, a third Upsample layer, a second Concat layer, a sixth recursive convolutional module, a fourth Upsample layer, an eighth ReLU activation function layer, a third Concat layer, and a seventh convolutional layer connected in sequence.

[0017] The and M After being concatenated by the first Concat layer, the and After being concatenated by the second Concat layer, the... P and After being stitched together by the third Concat layer, and finally passed through the seventh convolutional layer, a high spatial resolution multispectral image is obtained. .

[0018] Preferably, the first, second, third, fourth, fifth, and sixth recursive convolution modules are each composed of 10 sets of convolutional layers and ReLU activation function layers stacked sequentially, wherein the convolutional kernel size of the convolutional layer is 3, the stride is 1, and the padding is 1.

[0019] Preferably, the multi-branch, multi-scale Laplacian progressive remote sensing image fusion network is a pre-trained network; the loss function used during training is... for:

[0020] ;

[0021] ;

[0022] ;

[0023] ;

[0024] ;

[0025] in, It is the Frobenius paradigm. , , , There are 4 scaling factors; , , These represent the reference image, the reference image scaled down by a factor of 2 using bicubic interpolation, and the reference image scaled down by a factor of 4 using bicubic interpolation, respectively. , , All of these are outputs of the fusion branch, where, , These are the outputs of the fifth convolutional layer after passing through the fifth recursive convolutional module and the outputs of the sixth convolutional layer after passing through the sixth recursive convolutional module. This is the final output; LMS is the original multispectral image. M The result of upsampling by 4 times; N represents the total number of images in the training set; , , , Four proportional coefficients were set based on subjective experience.

[0026] The technical solution adopted by the system of this invention is: a multi-branch, multi-scale Laplacian progressive remote sensing image fusion system, comprising:

[0027] One or more processors;

[0028] A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the multi-branch, multi-scale Laplacian progressive remote sensing image fusion method.

[0029] The beneficial effects of this invention are as follows:

[0030] 1. A simple and effective basic fusion framework was constructed. It can achieve good fusion results using only a simple recursive convolution module and has good potential for scalability.

[0031] 2. A novel multi-branch weighted loss function is proposed;

[0032] 3. Compared with other methods, this invention achieves good fusion results while keeping the model parameters and running speed within a very small order of magnitude, which has great practical significance. Attached Figure Description

[0033] The technical solutions described herein are further illustrated below using examples and specific implementation methods. Additionally, accompanying drawings are used in the description of the technical solutions. Those skilled in the art can, without any creative effort, obtain other drawings and the intent of the present invention based on these drawings.

[0034] Figure 1 This is a schematic diagram of a multi-branch, multi-scale Laplacian progressive remote sensing image fusion network according to an embodiment of the present invention.

[0035] Figure 2 This is a qualitative comparison chart of the resolution reduction test results of the embodiments of the present invention and other methods.

[0036] Figure 3 This is a qualitative comparison chart of the full-resolution test results of the embodiments of the present invention and other methods. Detailed Implementation

[0037] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0038] The present invention provides a multi-branch, multi-scale Laplacian progressive remote sensing image fusion method, which inputs the remote sensing images to be fused into a multi-branch, multi-scale Laplacian progressive remote sensing image fusion network for image fusion.

[0039] Please see Figure 1 The multi-branch, multi-scale Laplacian progressive remote sensing image fusion network includes a panchromatic branch for extracting panchromatic image information. Multispectral branching for extracting multispectral images And a fusion branch used to integrate and reconstruct information from both;

[0040] The full-color branch Used to downsample a panchromatic image sequentially to the original panchromatic image. P 2 times and 4 times are respectively denoted as , ;

[0041] The multispectral branch Used to sequentially upsample a multispectral image to the original multispectral image. M 2 times and 4 times are respectively denoted as , ;

[0042] The and M After concatenation along the channel dimension, a fusion branch is injected. This fusion branch extracts the input features using a recursive convolution module and upsamples the feature map by a factor of 2 before injection. and The concatenated features are then fused again using a recursive convolution module, and the feature map is upsampled by a factor of 2 before being injected. P and Finally, after passing through a convolutional layer, a high spatial resolution multispectral image is obtained. .

[0043] In this embodiment, the panchromatic image is used as the sole input to the panchromatic branch, and the texture details at different scales of the panchromatic image are fully extracted through the panchromatic branch. Similarly, the multispectral image is used as the sole input to the multispectral branch, and the spectral information of the multispectral image is fully extracted through the multispectral branch. Finally, the rich spatial and spectral information extracted by the two branches is injected into the fusion branch, and the image information is fused and reconstructed through the fusion branch to obtain the desired multispectral image with high spatial resolution.

[0044] This embodiment uses pancolor branching to sequentially downsample the pancolor image to the original pancolor image. P 2 times ( ) and 4 times ( This process can be described as follows:

[0045]

[0046] Simultaneously, the multispectral image is sequentially upsampled to the original multispectral image through multispectral branching. M 2 times ( ) and 4 times ( This can be expressed as:

[0047]

[0048] Then and M After concatenation along the channel dimension, a fusion branch is injected. This fusion branch extracts the input features using a recursive convolution module and upsamples the feature map by a factor of 2. Then, the feature map is injected. and The concatenated features are then fused again using a recursive convolution module, and the feature map is upsampled by a factor of 2 and injected. P and Finally, after passing through a convolutional layer, a high spatial resolution multispectral image is obtained. The entire process can be described as follows:

[0049]

[0050]

[0051]

[0052]

[0053] in To fuse high spatial resolution multispectral images of different scales in the branch. To perform a splicing operation on the channel dimension of the features.

[0054] In one implementation, the panchromatic branch It includes a first recursive convolutional module layer, a first ReLU activation function layer, a first MaxPooling layer, a first convolutional layer, a second recursive convolutional module layer, a second ReLU activation function layer, a second MaxPooling layer, and a second convolutional layer, connected sequentially; the output of the first MaxPooling layer is output after passing through the first convolutional layer, denoted as . The output of the second MaxPooling layer is then passed through the second convolutional layer and output as follows: .

[0055] In one embodiment, the multispectral branch It includes a third recursive convolutional module layer, a third ReLU activation function layer, a first upsample layer, a fourth ReLU activation function layer, a third convolutional layer, a fourth recursive convolutional module layer, a fifth ReLU activation function layer, a second upsample layer, a sixth ReLU activation function layer, and a fourth convolutional layer, all connected in sequence. The output of the fourth ReLU activation function layer is then passed through the third convolutional layer and output as follows: The output of the sixth ReLU activation function layer is output after passing through the fourth convolutional layer, denoted as... .

[0056] In one implementation, the fusion branch includes a first Concat layer, a fifth convolutional layer, a seventh ReLU activation function layer, a fifth recursive convolutional module, a sixth convolutional layer, a third Upsample layer, a second Concat layer, a sixth recursive convolutional module, a fourth Upsample layer, an eighth ReLU activation function layer, a third Concat layer, and a seventh convolutional layer connected in sequence.

[0057] The and MAfter being concatenated by the first Concat layer, the and After being concatenated by the second Concat layer, the... P and After being stitched together by the third Concat layer, and finally passed through the seventh convolutional layer, a high spatial resolution multispectral image is obtained. .

[0058] In one implementation, the first, second, third, fourth, fifth, and sixth recursive convolution modules are each composed of 10 sets of convolutional layers and ReLU activation function layers stacked sequentially, wherein the convolutional kernel size of the convolutional layer is 3, the stride is 1, and the padding is 1.

[0059] In one implementation, the multi-branch, multi-scale Laplacian progressive remote sensing image fusion network is a trained network; the loss function used during training is... for:

[0060] ;

[0061] ;

[0062] ;

[0063] ;

[0064] ;

[0065] in, It is the Frobenius paradigm. , , , There are 4 scaling factors; , , These represent the reference image, the reference image scaled down by a factor of 2 using bicubic interpolation, and the reference image scaled down by a factor of 4 using bicubic interpolation, respectively. , , All of these are outputs of the fusion branch, where, , These are the outputs of the fifth convolutional layer after passing through the fifth recursive convolutional module and the outputs of the sixth convolutional layer after passing through the sixth recursive convolutional module. This is the final output; LMS is the original multispectral image.M The result of upsampling by 4 times; N represents the total number of images in the training set; , , , The four proportional coefficients are set based on subjective experience, and in this embodiment, the values ​​are 0.2, 0.3, 0.3, and 0.5, respectively.

[0066] In one implementation, the Adam optimizer is used, where betas and weight decay are set to (0.9, 0.999) and 0, respectively. A loss function is used, and the model is trained for 600 epochs with a batch size of 32. To achieve better performance, the initial learning rate is set to 0.0003, and the learning rate is decayed to 0.9 times the initial value every 100 epochs.

[0067] The following section describes how to train the network using a dataset and verify the fusion effect.

[0068] The experiment selected publicly available datasets in the remote sensing field (Gaofen-2 dataset, QuickBird dataset, and WorldView-3 dataset). The multispectral images from Gaofen-2 and QuickBird are 4-band (red, green, blue, and near-infrared); the multispectral images from WorldView-3 are 8-band (coastal blue, blue, green, yellow, red, red edge, near-infrared 1, and near-infrared 2). The spatial resolutions of the corresponding multispectral images in these three datasets are 3.2m, 2.44m, and 1.2m, respectively, while the spatial resolutions of the corresponding panchromatic images are 0.8m, 0.61m, and 0.3m, respectively. In the experiment, panchromatic and multispectral image pairs from these datasets were selected as input to the network, and a training strategy was used to train them.

[0069] Table 1 shows the basic information of the public dataset used in this embodiment. During training, this embodiment uses Wald's protocol to downsample the original panchromatic and multispectral images. The downsampled image pairs are used as input to the network for training, and the original image pairs are used as reference images to compare with the model output and calculate the loss function for backpropagation. The testing is divided into down-resolution testing and full-resolution testing. In the down-resolution testing, we still use the degraded images for testing, and the original image pairs are used as reference images to compare with the output results to verify the fusion effect of the model. In the full-resolution testing, this embodiment uses the original images as input to the model. Since there are no reference images in this case, this embodiment uses a parameterless reference index to quantitatively evaluate the fusion results.

[0070] Table 1. Basic information about the dataset

[0071]

[0072] Table 2 presents a quantitative comparison of the embodiments of the present invention with other methods. In the comparative experiments, 10 state-of-the-art methods were selected for comparison with the method proposed in this embodiment. These include eight traditional pan-color sharpening methods: the GramSchmidt sharpening approach (GS), Brovey, the GLP with MTF-matched filter (MTF-GLP), the High Pass Filtering (HPF), the Intensity-Hue-Saturation (IHS), the Principal Component Analysis (PCA), the partial replacement adaptive component substitution approach (PRACS), and the smoothing filter-based intensity modulation (SFIM). Additionally, two deep learning methods were selected: PNN and PanNet. Furthermore, this embodiment selected eight metrics to evaluate the down-resolution performance and three metrics to evaluate the full-resolution performance. The eight metrics used to evaluate low-resolution images are Structural Similarity Metric (SSIM), Peak Signal-to-Noise Ratio (PSNR), Spectral Angle Mapping (SAM), Correlation Coefficient (CC), Spatial Correlation Coefficient (sCC), Root Mean Square Error (RMSE), Relative Dimensionless Global Error (ERGAS), and Universal Image Quality Index (UIQI). These eight metrics primarily assess the similarity and error between the fused image and the ground truth. The other three parameter-free evaluation metrics are used to evaluate the objective performance at full resolution, including the Spectral Distortion Index (D...). λ Spatial distortion index (D) s ) and parameterless reference quality (QNR).

[0073] Table 2 shows the quantitative results from the WorldView-3 satellite.

[0074]

[0075] *↑ indicates that the larger the value, the better the performance; ↓ indicates that the smaller the value, the better the performance.

[0076] Please see Figure 2The image shows a qualitative comparison of the resolution reduction test results of this invention with other methods. It can be seen that the results of GS, IHS, PCA, and PanNet exhibit significant spectral distortion compared to LRMS and the reference image, while traditional methods such as Brovey, HPF, GLP, PRACS, and SFIM also lose a large amount of spatial detail. In contrast, our method produces the best visually sharpened image, even if it is not perfectly consistent with the reference image in some local details.

[0077] Please see Figure 3 The image shows a qualitative comparison of the full-resolution test results of the embodiments of the present invention with other methods. It can be seen that the results of GS, Brovey, IHS, PCA, and PanNet show severe global spectral distortion, while HPF and SFIM show local spectral distortion and are generally relatively blurry. PNN does not do well in preserving the spectral distribution of some local features. For example, the color of the building roof is different from the color in the LRMS image. In contrast, our method achieves a good balance between spectral fidelity and texture preservation.

[0078] The remote sensing image fusion method of this invention mainly consists of different scales, different branches, a special residual connection method, and a multi-scale loss function designed for the entire network. Experiments show that this method, using only basic convolutional modules and ReLU activation functions, outperforms most mainstream traditional methods and deep learning-based methods in terms of fusion performance. It alleviates the distortion and unrealistic phenomena of fused images to a certain extent. In addition, the designed loss function effectively improves the generalization ability of the model.

[0079] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A multi-branch, multi-scale Laplacian progressive remote sensing image fusion method, characterized in that: The remote sensing images to be fused are input into a multi-branch, multi-scale Laplacian progressive remote sensing image fusion network for image fusion. The multi-branch, multi-scale Laplacian progressive remote sensing image fusion network includes a panchromatic branch for extracting panchromatic image information. Multispectral branching for extracting multispectral images And a fusion branch used to integrate and reconstruct information from both; The full-color branch It is used to downsample a panchromatic image twice to obtain the original panchromatic image. P The preset multiples are denoted as follows: , It includes a first recursive convolutional module layer, a first ReLU activation function layer, a first MaxPooling layer, a first convolutional layer, a second recursive convolutional module layer, a second ReLU activation function layer, a second MaxPooling layer, and a second convolutional layer, connected sequentially. The output of the first MaxPooling layer is output after passing through the first convolutional layer, denoted as... The output of the second MaxPooling layer is then passed through the second convolutional layer and output as follows: ; The multispectral branch This is used to upsample a multispectral image twice to the original multispectral image. M The preset multiples are denoted as follows: , It includes a third recursive convolutional module layer, a third ReLU activation function layer, a first upsample layer, a fourth ReLU activation function layer, a third convolutional layer, a fourth recursive convolutional module layer, a fifth ReLU activation function layer, a second upsample layer, a sixth ReLU activation function layer, and a fourth convolutional layer, all connected sequentially. The output of the fourth ReLU activation function layer is then passed through the third convolutional layer and output as follows: The output of the sixth ReLU activation function layer is output after passing through the fourth convolutional layer, denoted as... ; The fusion branch includes a first Concat layer, a fifth convolutional layer, a seventh ReLU activation function layer, a fifth recursive convolutional module, a sixth convolutional layer, a third Upsample layer, a second Concat layer, a sixth recursive convolutional module, a fourth Upsample layer, an eighth ReLU activation function layer, a third Concat layer, and a seventh convolutional layer, connected sequentially. The and M After being concatenated by the first Concat layer, the and After being concatenated by the second Concat layer, the... P and After being stitched together by the third Concat layer, and finally passed through the seventh convolutional layer, a high spatial resolution multispectral image is obtained. .

2. The multi-branch, multi-scale Laplacian progressive remote sensing image fusion method according to claim 1, characterized in that: The first, second, third, fourth, fifth, and sixth recursive convolution modules are each composed of 10 sets of convolutional layers and ReLU activation function layers stacked sequentially.

3. The multi-branch, multi-scale Laplacian progressive remote sensing image fusion method according to any one of claims 1-2, characterized in that: The multi-branch, multi-scale Laplacian progressive remote sensing image fusion network is a pre-trained network; Loss function used during training for: ； ； ； ； ； in, It is the Frobenius paradigm. , , , There are 4 scaling factors; , , These represent the reference image, the result of the reference image being scaled down by a preset factor through bicubic interpolation, and the result of the reference image being scaled down by a preset factor through bicubic interpolation, respectively. , , All of these are outputs of the fusion branch, where, , These are the outputs of the fifth convolutional layer after passing through the fifth recursive convolutional module and the outputs of the sixth convolutional layer after passing through the sixth recursive convolutional module. This is the final output; LMS is the original multispectral image. M The result of upsampling by a preset factor; N represents the total number of images in the training set; , , , Four proportional coefficients were set based on subjective experience.

4. A multi-branch, multi-scale Laplacian progressive remote sensing image fusion system, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the multi-branch, multi-scale Laplacian progressive remote sensing image fusion method as described in any one of claims 1 to 3.

Images