A multi-spectral image fusion method based on interpretable neural networks

Abstract

The application provides a multispectral image fusion method based on an interpretable neural network, first, multispectral images are obtained through a multispectral sensor, a panchromatic image of the same area is obtained through the multispectral sensor, and then a data fidelity term is constructed; a residual relationship between a fusion image and the panchromatic image is established through a deep network, and used as a prior constraint term; a fusion model is constructed through the data fidelity term and the deep learning prior term; based on the fusion model, an optimization model is iteratively solved through a proximal gradient descent method; the iterative solving step is mapped into a deep network architecture, a required multispectral panchromatic image deep network model is generated through training of a training data set; and based on the generated multispectral panchromatic image deep network model, multispectral and panchromatic images of a given input are fused to generate a final fusion image. The application generates optimal parameter settings through data driving, thereby improving fusion performance.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image fusion and relates to a multispectral image fusion method based on interpretable neural networks. It is suitable for applications that improve the spatial resolution of multispectral images through image fusion. Background Technology

[0002] Due to limitations in sensor optical system design, there is a mutually constraining relationship between spatial and spectral resolution in remote sensing images. Multispectral images possess high spectral resolution but relatively low spatial resolution, while panchromatic images offer high spatial resolution but limited spectral information. Fusing these two types of images from the same location yields images with both high spectral and spatial resolution. This allows for applications in disaster monitoring, military reconnaissance, land surveying, and agricultural analysis, enabling more precise applications and improving detection accuracy.

[0003] Currently, representative multispectral image fusion methods mainly include: component substitution (CS), multi-resolution analysis (MRA), variational model optimization (VO), and deep learning (DL). The CS method first separates the spatial and spectral information of the multispectral image through matrix factorization, and then superimposes the spatial information of the panchromatic and multispectral images before performing an inverse transformation to obtain the fused image. While the CS method can effectively enhance spatial details, it often suffers from significant spectral distortion. The MRA method decomposes the multispectral and panchromatic images at multiple scales and fuses the corresponding decomposition coefficients at different scales to obtain the fused image. Compared to the CS method, the MRA method can obtain better spectral information, but its spatial detail clarity is relatively poor. The VO method first establishes a functional relationship between the fused image and the source image, and then constructs a fusion model by combining different prior constraints, thus transforming the image fusion problem into a mathematical optimization problem, and obtaining the optimal fusion result through iterative optimization. Compared to the CS and MRA methods, the VO method can obtain better high-resolution spatial and spectral information. However, this method often constructs prior models based on linear features (such as gradients), making it difficult to describe the complex nonlinear relationships in natural scenes, resulting in limited accuracy of the imaging model. Furthermore, the model parameters require manual adjustment, which is often time-consuming and difficult to optimize. Thanks to the powerful nonlinear feature representation capabilities of deep learning methods, deep learning-based multispectral image fusion has gradually become a research hotspot in recent years. While both methods address optimization problems, deep learning methods introduce adaptive learning mechanisms and nonlinear functions, using neural networks to iteratively optimize parameters, avoiding the blindness of manual parameter adjustments and reducing complexity. However, traditional deep learning methods often treat the fusion process as a black box, ignoring the real physical imaging significance, and the fusion performance still needs improvement. Summary of the Invention

[0004] This invention addresses the shortcomings of existing remote sensing image fusion methods by proposing a multispectral image fusion method based on an interpretable neural network. The method includes the following steps:

[0005] Step 1: Obtain multispectral images using a multispectral sensor, and obtain panchromatic images of the same region using a multispectral sensor;

[0006] Step 2: Construct a data fidelity item using the multispectral image described in Step 1;

[0007] Step 3: Establish the residual relationship between the fused image and the panchromatic image using a deep network as a priori constraint.

[0008] Step 4: Construct a fusion model using the data fidelity terms described in Step 2 and the deep learning prior terms described in Step 3;

[0009] Step 5: Based on the fusion model described in Step 4, iteratively solve the optimal model using the proximal gradient descent method;

[0010] Step 6: Map the iterative solution steps in Step 5 to a deep network architecture, and train the required multispectral panchromatic image deep network model using the training dataset;

[0011] Step 7: Based on the multispectral panchromatic image depth network model generated in Step 6, fuse the given input multispectral and panchromatic images to generate the final fused image.

[0012] Preferably, the multispectral image in step 1 is denoted as A panchromatic image is denoted as Where m and n represent the spatial resolution dimensions of the multispectral image, its spatial resolution is 1 / c of ​​that of the panchromatic image, and it has b bands; a high-resolution multispectral image... and panchromatic image After MTF filtering and spatial downsampling, low-resolution multispectral images M and panchromatic images P are generated for subsequent deep network model training.

[0013] Preferably, the data fidelity item in step 2 consists of a multispectral image fidelity item; the fused image is denoted as... This indicates that the image has b bands and a resolution of m×n.

[0014] A multispectral image M can be considered as the result of a fused image F after blurring and downsampling, that is:

[0015] M = ΨF + N

[0016] Where Ψ represents the fuzzing and spatial downsampling operators, and N represents Gaussian noise with a mean of 0. The constructed data fidelity term is...

[0017]

[0018] in This represents the Frobenius norm.

[0019] Preferably, the constraint term in step 3 is the residual relationship between the fused image F and the panchromatic image P. However, since the characteristics of natural scenes are often complex, it is difficult to accurately describe them using the L1 norm, which is only suitable for representing sparse characteristics. To solve the above problem, the residual relationship is described by a data-driven deep learning function φ, which is generated by training a deep network. In this case, the constructed prior constraint term is...

[0020]

[0021] Where R p This is a spectral mapping matrix used to describe the correspondence between different spectral bands of the multispectral array and the panchromatic image P. This represents the result of copying the panchromatic image P along the spectral direction as b copies.

[0022] Preferably, the fusion model described in step 4 is constructed using the data fidelity term described in step 2 and the deep learning constraint term described in step 3 as follows:

[0023]

[0024] Where λ is the parameter that balances the data fidelity term and the prior constraint term.

[0025] Preferably, step 5 includes the following sub-steps:

[0026] Step 5.1: Introduce intermediate variables The near-end gradient descent iterative method is transformed into two iterative formulas:

[0027] Z t+1 =F t -μΨ T (ΨF t -M) (1)

[0028]

[0029] Where t represents the number of iterations, and μ represents the gradient step size parameter. Ψ T This represents the transpose of Ψ.

[0030] Step 5.2: Solve for the (t+1)th iteration value Z according to formula (1). t+1 .

[0031] Step 5.3: First, we can approximate the result.

[0032]

[0033] α is a parameter to be solved. Therefore, formula (2) can also be simplified to

[0034]

[0035] in make The final formula (2) can be equivalent to:

[0036]

[0037] Step 5.4: Solve the W subproblem. This subproblem can be directly solved using the SwinResUnet network module, which combines the Transformer and CNN architectures. [1] The solution is to be performed. The network should contain three sets of encoders (E1, E2, E3) and decoders (D1, D2, D3) with different dimensions, with embedded feature extraction units. Simultaneously, the corresponding encoders (E1, E2, E3)... i ) and decoder (D i Establish residual short connection structures between them to enhance feature transfer at different scales.

[0038] [1]Zhang K, Li Y, Liang J, Cao J, Zhang Y, Tang H, Timofte R and GoolL. 2022. Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis[EB / OL].

[0039] Step 5.5: Based on the obtained W, approximate the solution for F:

[0040]

[0041] in

[0042] Step 5.6: Iterate repeatedly until the difference between two iterations is small enough to meet the termination condition; or until the number of iterations reaches the maximum value, and the output F is the fused image.

[0043] Step 5.7: Map the above steps to a deep network architecture, embedding each model iteration process into the corresponding deep network module. Figure 1 The network structure for single-iteration optimization is presented. The solution to iterative formula (1) is a closed-loop analytical form, requiring no neural network parameter learning. The solution to iterative formula (2) is transformed into solving the intermediate variable W, and a network module combining Transformer and CNN architectures is used as the solver. Based on the strong mapping relationship learning capability of deep learning, accurate solution results are obtained. Finally, convolutional layer units are designed to represent the spectral response parameter matrix in step 5.5. Then, the optimized fusion result for this iteration is obtained by inverse solving.

[0044] Step 5.8: Combine high-resolution multispectral images The L1 loss function is used to constrain the network and drive parameter optimization.

[0045]

[0046] The L1 norm can ensure the spectral consistency between the fusion result and the high-resolution multispectral image while also achieving good structural sparsity between the two images. In this invention, the interpretable fusion network is trained end-to-end using the L1 norm. On the one hand, it learns the accurate mapping relationship between the data, improving the quality of the fusion model; on the other hand, the physical mechanism in the model optimization also drives the optimization of network parameters, reducing the parameter size and improving the model's generalization ability.

[0047] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0048] This invention proposes a new remote sensing image fusion method based on deep learning networks. (1) A nonlinear deep learning prior is used to construct the remote sensing image fusion model, which effectively improves the accuracy of the fusion model. (2) The model solving process is further mapped to an interpretable deep network architecture, and the optimal parameter settings are generated through data-driven methods, thereby improving the fusion performance. Attached Figure Description

[0049] Figure 1 This is a flowchart of the iteration process using a neural network.

[0050] Figure 2 This is a diagram of the SwinResUnet network structure. Detailed Implementation

[0051] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0052] This invention provides a multispectral image fusion method based on interpretable neural networks, the specific implementation steps of which include:

[0053] Step 1: Acquire multispectral and panchromatic images of the same location from the Gaofen-2 remote sensing satellite.

[0054] Step 1: This remote sensing satellite can provide panchromatic images with a spatial resolution of 0.8m and multispectral images with a spatial resolution of 3.2m. The spectral bands include four bands: red (600-670nm), green (510-590nm), blue (440-510nm), and near-infrared (760-910nm). The multispectral image is denoted as... A panchromatic image is denoted as High-resolution multispectral images and panchromatic image Low-resolution multispectral images are generated by MTF filtering and spatial downsampling respectively. and panchromatic image Used for subsequent deep network model training.

[0055] Step 2: Construct a data fidelity item using the multispectral images described in Step 1.

[0056] The data fidelity term in step 2 consists of a multispectral image fidelity term; the fused image is denoted as... This indicates that the image has 4 bands and a resolution of m×n=300×300;

[0057] A multispectral image M can be considered as the result of a fused image F after blurring and downsampling, that is:

[0058] M = ΨF + N

[0059] Where Ψ represents the fuzzing and spatial downsampling operators, and N represents Gaussian noise with a mean of 0. The constructed data fidelity term is...

[0060]

[0061] in This represents the Frobenius norm.

[0062] Step 3: Establish the residual relationship between the fused image and the panchromatic image using an interpretable neural network as a priori constraint.

[0063] The constraint term mentioned in step 3 is the residual relationship between the fused image F and the panchromatic image P. This residual relationship is described using a data-driven deep learning function φ, and the prior constraint term is constructed as follows:

[0064]

[0065] Where R p This is a spectral mapping matrix used to describe the correspondence between different spectral bands of the multispectral array and the panchromatic image P. This represents the result of copying the panchromatic image P four times along the spectral direction.

[0066] Step 4: Construct a fusion model using the data fidelity terms described in Step 2 and the deep learning prior terms described in Step 3.

[0067] The fusion model described in step 4 is as follows:

[0068]

[0069] Where λ is the parameter that balances the data fidelity term and the prior constraint term.

[0070] Step 5: Based on the fusion model described in Step 4, the optimal model is solved iteratively using the proximal gradient descent method.

[0071] Step 5 includes the following sub-steps:

[0072] Step 5.1: Introduce intermediate variables The near-end gradient descent iterative method is transformed into two iterative formulas:

[0073] Z t+1 =F t -μΨ T (ΨF t -M) (1)

[0074]

[0075] Where t represents the number of iterations, and μ represents the gradient step size parameter. Ψ T This represents the transpose of Ψ.

[0076] Step 5.2: Substitute the data to solve for the Z value at the (t+1)th iteration.

[0077] Step 5.3: By simplification, we can obtain:

[0078]

[0079] Therefore, formula (2) can also be simplified to

[0080]

[0081] in make The final formula (2) can be equivalent to:

[0082]

[0083] Step 5.4: Solve the W subproblem. This is done using a deep learning neural network. The implementation uses the SwinResUnet network module, which combines the Transformer and CNN architectures.

[0084] Step 5.6: Based on the obtained W, approximate the solution for F:

[0085]

[0086] in,

[0087] Step 5.7: Iterate repeatedly until the difference between two iterations is small enough to meet the termination condition; or the number of iterations reaches its maximum value. The output F is the fused image.

[0088] Step 5.8: The neural network training samples are from the Gaofen-2 remote sensing satellite. 16,000 sets of data are selected for deep learning training to generate the required multispectral fusion deep model.

[0089] Step 6: Based on the generated multispectral fusion depth model, the given input multispectral and panchromatic images are fused to generate the final fused image. 28 images from other regions (different from the training data regions) are used as test data. For quantitative evaluation, statistical analysis is performed on the 28 test images, and the average values ​​of various evaluation metrics are shown in Table 1.

[0090] Table 1. Quantitative evaluation based on 28 test images from the Gaofen-2 simulation dataset.

[0091]

[0092] The superior performance of this invention is verified through simulation experiments.

[0093] The simulation experiments are mainly based on the Wald simulation protocol: high-resolution multispectral and panchromatic images are processed by MTF filtering and spatial downsampling to generate low-resolution multispectral and panchromatic images for multispectral image fusion experiments. The high-resolution multispectral image is then used as the ground truth image for qualitative and quantitative analysis of the fusion results. The MTF filter is similar to an ideal low-pass filter, having a cutoff amplitude at the Nyquist frequency. By comparing the differences between the ground truth image and the fused image, a qualitative analysis of the characteristics of different methods can be conducted from a subjective visual perspective. Furthermore, the effectiveness of the algorithm can be quantitatively evaluated using full-reference image quality evaluation metrics, including: Relative Dimensionless Global Error in Synthesis (ERGAS), Spectral Angle Mapping (SAM), and Global Comprehensive Score (Q). 2n Structural Similarity Index (SSIM), Root Mean Square Error (RMSE), Relative Average Spectral Error (RASE), Universal Image Quality Index (UIQI), and Peak Signal-to-Noise Ratio (PSNR). ERGAS and Q 2nThese are comprehensive performance evaluation metrics for multispectral image fusion. SAM measures the spectral distortion of the fused image by calculating the absolute angle between two vectors corresponding to the ground truth image. SSIM measures the structural similarity between images. RASE evaluates the global spectral quality of the fused image. PSNR and RMSE primarily measure the deviation between the fused result and the ground truth from the perspective of pixel differences. UIQI evaluates the degree of structural distortion between the fused image and the reference image.

[0094] The simulation experiment primarily used a dataset provided by the Gaofen-2 remote sensing satellite to conduct multispectral image fusion experiments for verification. It should be understood that any parts not detailed in this document belong to existing technology.

[0095] It should be understood that the above description of the embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art can make substitutions or modifications under the guidance of this invention without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for multi-spectral image fusion based on interpretable neural networks, characterized in that, The method comprises the following steps: Step 1: obtaining a multispectral image through a multispectral sensor, and obtaining a panchromatic image of the same region through the multispectral sensor; Step 2: constructing a data fidelity term through the multispectral image in step 1; Step 3: establishing a residual relationship between the fused image and the panchromatic image through a deep network, as a prior constraint term; The constraint term described in step 3 is the residual relationship of the fused image F and the panchromatic image P, and the residual relationship is driven by a deep learning function The deep learning function is generated by training a deep network, and at this time, the constructed prior constraint term is: Wherein, the fused image is denoted as , indicating that the image has b bands and the resolution size is ; is a spectral mapping matrix, used to describe the correspondence between different spectral bands of the multispectral image and the panchromatic image P, represents the result of copying the panchromatic image P along the spectral direction b times. Step 4: constructing a fusion model through the data fidelity term in step 2 and the deep learning prior term in step 3; The fusion model is as follows: wherein the fused image is denoted as , indicating that the image has b bands and the resolution size is ; the multispectral image M can be regarded as the result of blurring and downsampling of the fused image F, , indicating the blurring and spatial downsampling operator, is a spectral mapping matrix, used to describe the correspondence between different spectral bands of the multispectral image and the panchromatic image P, represents the result of copying the panchromatic image P along the spectral direction b times; is a parameter balancing the data fidelity term and the prior constraint term; Step 5: iteratively solving an optimization model through a proximal gradient descent method based on the fusion model in step 4; Step 6: mapping the iterative solving step in step 5 into a deep network architecture, and training a multispectral panchromatic image deep network model through a training data set; Step 7: based on the multispectral panchromatic image deep network model generated in step 6, fusing multispectral and panchromatic images for a given input to generate a final fused image.

2. The multi-spectral image fusion method based on an interpretable neural network according to claim 1, characterized in that: The multispectral image described in step 1 is denoted as The panchromatic image is denoted as wherein and n represents the spatial resolution size of the multispectral image, and the spatial resolution of the multispectral image is 1 / c of the panchromatic image, and b is the number of wavebands; the multispectral image with high resolution and the panchromatic image are respectively subjected to MTF filtering and spatial downsampling to generate a multispectral image M with low resolution and a panchromatic image P with low resolution for subsequent deep network model training.

3. The multi-spectral image fusion method based on an interpretable neural network according to claim 1, characterized in that: The data fidelity term in step 2 is composed of a multispectral image fidelity term; the fused image is denoted as , indicating that the image has b bands and the resolution size is ; The multispectral image M can be regarded as the result of blurring and downsampling of the fused image F, that is: where denotes the blur and spatial down-sampling operator, N denotes a Gaussian noise with mean 0, and the data fidelity term is constructed as wherein denotes the Frobenius norm.

4. The multi-spectral image fusion method based on an interpretable neural network according to claim 1, characterized in that: The specific implementation of step 5 comprises the following sub-steps: Step 5.1, Introduce intermediate variable The proximal gradient descent iteration is transformed into two iteration formulas: (1) (2) where t denotes the iteration number, denotes a gradient step parameter, , denotes the transpose of Step 5.2, solve for the (t+1)th iteration value Z according to equation (1) t+1 ; Step 5.3: first, the approximation can be obtained is a parameter to be solved; therefore, equation (2) can also be simplified as wherein , let Finally, equation (2) can be equivalent to: Step 5.4: solve the W sub-problem, which can be directly solved by using a network module SwinResUnet combining the Transformer and CNN architectures; Step 5.5: according to the obtained W, the F is approximately solved: wherein ; Step 5.6: repeatedly iterate until the difference between two iterations is small enough to meet the termination condition; or the iteration number reaches the maximum value, and the F is output as the fused image. Step 5.7, map the above steps to a deep network architecture, embed each model iteration process in the corresponding deep network module, wherein the solution of iteration formula (1) is a closed-loop analytical form without neural network parameter learning; the solution of iteration formula (2) is transformed into the solution problem of the intermediate variable W, and a network module combining the Transformer and CNN architecture is used as a solver, and based on the strong mapping relationship learning ability of deep learning, an accurate solution result is obtained, and finally a convolutional layer unit is designed to represent the spectral response parameter matrix in step 5.5 , and the optimized fusion result under this iteration is solved. Step 5.8, combining high-resolution multispectral images , using the network is constrained and driven to optimize parameters as a loss function, i.e. The norm can better realize the structural sparsity between the two images while ensuring the spectral consistency of the fusion result with the high-resolution multispectral image spectrum.

5. The interpretable neural network-based multispectral image fusion method of claim 4, wherein: The SwinResUnet comprises three groups of encoders (E1, E2, E3) and decoders (D1, D2, D3) at different dimensions, and a feature extraction unit is embedded in the middle, and a residual short connection structure is established between the corresponding encoders (E i ) and decoders (D i ), thereby enhancing the feature transmission effect at different scales.

6. The interpretable neural network-based multispectral image fusion method of claim 1, wherein: Also included is step 8, employing global relative dimensionless error ERGAS (Relative Dimensionless Global Error in Synthesis), spectral angle mapping SAM (Spectral Angle Mapping), global synthesis score Q 2n , structural similarity SSIM (Structural Similarity Index), root mean square error RMSE (Root Mean Square Error), relative average spectral error RASE (Relative Average Spectral Error), universal image quality index UIQI (Universal Image Quality Index) and peak signal-to-noise ratio PSNR (Peak Signal-to-Noise Ratio) to evaluate the fusion effect of the fused image.

Images