A multi-source satellite remote sensing image fusion method and system based on a multi-level framework

Abstract

The present invention relates to the field of aerospace information, and in particular to a multi-source satellite remote sensing image fusion method and system based on a multi-level framework. The method comprises the following steps: acquiring multi-source satellite remote sensing image data, and pre-processing the multi-source satellite remote sensing image data to obtain a fused image dataset; constructing a progressive multi-level fusion strategy for the multi-source satellite remote sensing image data based on the multi-level framework; obtaining multispectral high spatial resolution data of the multi-source satellite remote sensing image data using the progressive multi-level fusion strategy based on the fused image dataset; establishing a multi-fusion method collaborative architecture, combining the multi-spectral high spatial resolution data with the multi-fusion method collaborative architecture to obtain multi-source satellite remote sensing fused image data, thereby realizing the fusion of multi-source satellite remote sensing images. The present invention improves the fusion accuracy of multi-source satellite images, obtains multi-source satellite remote sensing fused image data, and provides technical support for the development of high-resolution remote sensing image applications.

Description

Technical Field

[0001] The present invention relates to the field of aerospace information, and in particular to a multi-source satellite remote sensing image fusion method and system based on a multi-level framework. Background Art

[0002] Spectral and spatial resolution have always been core parameters of interest in remote sensing image processing. However, due to sensor technology limitations, there is often a trade-off between spatial and spectral resolution. These limitations pose challenges in performing fine-grained object classification, extracting complex information, and performing parameter inversion using remote sensing images. Acquiring higher-resolution satellite remote sensing imagery for high-precision Earth observation has been a hot topic and a challenging issue for research both domestically and internationally.

[0003] The goal of remote sensing image fusion is to intuitively and effectively improve the spatial resolution of multispectral or hyperspectral imagery, thereby enhancing and enriching the information and detail of the images. This approach can be widely applied in fields such as ground object classification and global change monitoring. Pixel-level fusion is one of the three fundamental approaches in satellite remote sensing image fusion. However, existing pixel-level fusion methods are often only applicable to a single satellite and are ineffective for fusing multi-source satellite imagery. Furthermore, when the ground resolution of panchromatic and multispectral imagery differs significantly, the fusion effect is poor. Therefore, there is an urgent need to develop multi-source satellite remote sensing image fusion technology to meet the needs of a wider range of applications.

[0004] To overcome the challenges of existing multi-source remote sensing image fusion technologies, this paper provides a multi-source satellite remote sensing image fusion method and system based on a multi-level framework. This method is suitable for fusing multi-source satellite imagery with significant differences in spatial resolution. The fused remote sensing images can provide enhanced information and richer details, providing data support for downstream applications such as remote sensing image-based object recognition and classification, information extraction, and parameter inversion. Summary of the Invention

[0005] In view of the defects in the prior art, the present invention provides a multi-source satellite remote sensing image fusion method and system based on a multi-level framework.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a multi-source satellite remote sensing image fusion method based on a multi-level framework, the method comprising the following steps: acquiring multi-source satellite remote sensing image data, and pre-processing the multi-source satellite remote sensing image data to obtain a fused image dataset; constructing a progressive multi-level fusion strategy for the multi-source satellite remote sensing image data based on the multi-level framework; obtaining multispectral high spatial resolution data of the multi-source satellite remote sensing image data using the progressive multi-level fusion strategy based on the fused image dataset; establishing a multi-fusion method collaborative architecture, combining the multi-spectral high spatial resolution data with the multi-fusion method collaborative architecture to obtain multi-source satellite remote sensing fused image data, thereby realizing the fusion of multi-source satellite remote sensing images. The present invention achieves efficient integration and quality improvement of image data through the multi-level framework and collaborative architecture design. First, the preprocessing stage effectively eliminates the radiation differences and geometric distortions among multi-source data, providing standardized input for subsequent fusion; second, the progressive multi-level fusion strategy enhances spatial resolution while preserving spectral details through staged feature extraction and progressive information fusion; the multispectral image generation link compares multiple categories of fusion methods through a multi-method collaborative architecture, significantly improving the spectral fidelity and geometric accuracy of the fused image; it effectively solves the fusion problem of multi-source data with large scale differences, providing higher-precision temporally and spatially consistent image data for applications such as remote sensing monitoring and land object classification.

[0007] Optionally, the method of acquiring multi-source satellite remote sensing image data and pre-processing the multi-source satellite remote sensing image data to obtain a fused image dataset includes: acquiring first satellite image data and second satellite image data, using the first satellite image data and the second satellite image data as the multi-source satellite remote sensing image data; performing radiometric correction and atmospheric correction on the first satellite image data to obtain first surface reflectance data; performing radiometric correction, atmospheric correction, and geometric correction on the second satellite image data to obtain second surface reflectance data; and combining the first surface reflectance data and the second surface reflectance data to construct the fused image dataset. The present invention significantly improves the quality of remote sensing images through multi-source data integration and standardization processing; provides richer original information for subsequent fusion; performs radiometric correction and atmospheric correction on the first satellite image to effectively eliminate the influence of atmospheric scattering and obtain the true surface reflectance; performs multi-level pre-processing of radiometric correction, atmospheric correction, and geometric correction on the second satellite image, simultaneously solving the problems of sensor response differences, atmospheric interference, and geometric distortion, ensuring the consistency of multi-source data, and providing data support for constructing a high-precision fused image dataset.

[0008] Optionally, the construction of the fused image dataset by combining the first surface reflectance data and the second surface reflectance data includes: registering the first surface reflectance data and the second surface reflectance data; resampling the registered second surface reflectance data to obtain a low-resolution-high-resolution image pair; and constructing the fused image dataset based on the low-resolution-high-resolution image pair. The present invention significantly improves the geometric accuracy and information compatibility of image fusion through fine registration and scale alignment before multi-source data fusion; ensures the strict correspondence of multi-source data in the same geographic coordinate system through registration processing, providing a spatial benchmark for subsequent fusion; resampling the second surface reflectance data achieves scale matching of spectral information and spatial details, and then constructs a fused image dataset with multi-scale feature associations, providing structured input for subsequent multi-level fusion strategies, and effectively improving the spatial continuity and spectral authenticity of the fused image.

[0009] Optionally, the multi-level framework-based progressive multi-stage fusion strategy for constructing the multi-source satellite remote sensing image data includes constructing the progressive multi-stage fusion strategy based on homologous satellite image fusion, heterologous satellite image fusion, and multispectral-panchromatic image fusion. By constructing a progressive multi-stage fusion strategy, the present invention achieves deep integration and optimization of remote sensing image information. By performing homologous, heterologous, and multispectral-panchromatic image fusion in stages, image features are gradually refined, effectively balancing the contradiction between spectral fidelity and spatial resolution, and significantly improving the overall quality of the fused image.

[0010] Optionally, the progressive multi-level fusion strategy includes: a fusion process of the progressive multi-level fusion strategy satisfies the following relationship:

[0011] ;

[0012] in, For the The high-resolution multispectral fusion image output after level fusion is For the The spatial resolution of the fused input is For the Level fusion model, For pre-processed low-resolution multispectral images at all levels, For high-resolution auxiliary images at all levels, is the index variable of the fusion model, The total number of fusions. The present invention achieves the coordinated optimization of the spatial resolution and spectral quality of remote sensing images through a step-by-step fusion model. Each level of the fusion model fuses the features of low-resolution multispectral images with high-resolution auxiliary images, gradually improving the spatial detail expression of the output image. It effectively balances the contradiction between spectral fidelity and spatial resolution, and avoids the information loss of direct fusion of a single model through progressive optimization of the multi-level model, so as to obtain high-quality multispectral high-spatial resolution data.

[0013] Optionally, based on the fused image dataset, the multispectral high spatial resolution data of the multi-source satellite remote sensing image data is obtained using the progressive multi-level fusion strategy, including: based on the same-source satellite image fusion, the first satellite image data is subjected to multi-resolution data fusion to obtain multispectral enhancement data; based on the different-source satellite image fusion, the multispectral enhancement data is fused with the second satellite image data to obtain high spatial resolution data; based on the multispectral-panchromatic image fusion, the high spatial resolution data is fused with the panchromatic image of the second satellite image data to obtain the multispectral high spatial resolution data. The present invention significantly improves the overall quality of multi-source satellite remote sensing images through a progressive multi-level fusion strategy; the same-source satellite fusion stage enhances the multispectral detail expression of the first satellite image through multi-resolution fusion; the multi-source satellite fusion stage integrates the enhanced multispectral data with the second satellite image, achieving a coordinated improvement in spectral information and spatial resolution; the multispectral-panchromatic image fusion stage obtains multispectral high spatial resolution data by fusing high-resolution panchromatic bands with multispectral data, thereby improving data quality.

[0014] Optionally, the establishment of a collaborative architecture for multiple fusion methods includes: combining a component replacement method, a multi-scale decomposition method, and a deep learning method to construct the collaborative architecture for multiple fusion methods. The present invention integrates the component replacement method, the multi-scale decomposition method, and the deep learning method to construct a collaborative architecture for multiple fusion methods with complementary advantages. The component replacement method ensures the fidelity of spectral information, the multi-scale decomposition method enhances the expression of spatial details, and the deep learning method optimizes the fusion of nonlinear features. By comparing multiple categories of fusion methods, the overall quality of the fused image is improved.

[0015] Optionally, the multi-source satellite remote sensing fusion image data is obtained by combining the multi-spectral high spatial resolution data with the multi-fusion method collaborative architecture to achieve the fusion of multi-source satellite remote sensing images, including: performing component replacement on the multi-spectral high spatial resolution data based on the component replacement method to obtain first multi-source satellite remote sensing fusion image data; performing multi-scale decomposition analysis on the multi-spectral high spatial resolution data according to the multi-scale decomposition method to obtain second multi-source satellite remote sensing fusion image data; obtaining third multi-source satellite remote sensing fusion image data based on the deep learning method and combining the multi-spectral high spatial resolution data; using the first multi-source satellite remote sensing fusion image data, the second multi-source satellite remote sensing fusion image data and the third multi-source satellite remote sensing fusion image data as the multi-source satellite remote sensing fusion image data. The present invention improves the comprehensive quality of multi-source satellite remote sensing images through different fusion methods; the collaboratively generated multi-source fusion image data has high spectral resolution, high spatial resolution and strong environmental adaptability, providing an accurate data foundation for applications such as remote sensing monitoring and land object classification.

[0016] Optionally, the multi-fusion method collaborative architecture includes: the component replacement method satisfies the following relationship:

[0017] ;

[0018] in, For the output high-resolution image, represents the inverse transformation of the color space, is the panchromatic image after histogram matching, represents the retained components that are not replaced; the multi-scale decomposition method satisfies the following relationship:

[0019] ;

[0020] in, After fusion High-resolution multispectral imagery of the bands, For the Low-resolution multispectral images after band upsampling, is the gain coefficient, is the high-frequency component of the panchromatic image, is the original full-color image, represents low-pass filtering; the deep learning method satisfies the following relationship:

[0021] ;

[0022] in, For the output high-resolution image, is a parameterized neural network, represents the model parameters, is the original full-color image, This is an upsampled low-resolution multispectral image. This invention achieves the fusion of multispectral, high-spatial-resolution data through a collaborative architecture of multiple fusion methods, optimizing the fusion performance of complex land features and significantly improving the spectral fidelity, spatial resolution, and adaptability of the fused image to complex scenes.

[0023] In a second aspect, the present invention provides a multi-source satellite remote sensing image fusion system based on a multi-level framework. The system implements the multi-source satellite remote sensing image fusion method based on a multi-level framework provided by the present invention. The system is characterized in that the system includes an input device, an output device, a processor, and a memory, wherein the input device, output device, processor, and memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions. The present invention constructs an efficient multi-source satellite remote sensing image fusion processing platform by integrating high-performance hardware, realizes the fusion of remote sensing images, and provides stable and efficient hardware support for large-scale multi-source image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of a multi-source satellite remote sensing image fusion method based on a multi-level framework according to an embodiment of the present invention;

[0025] Figure 2 This is a structural diagram of a multi-source satellite remote sensing image fusion method based on a multi-level framework according to an embodiment of the present invention;

[0026] Figure 3 This is a framework diagram of a multi-source satellite remote sensing image fusion system based on a multi-level framework according to an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of an execution flow of a multi-source satellite remote sensing image fusion system based on a multi-level framework according to an embodiment of the present invention;

[0028] Figure 5 Schematic diagram of a fused image output by a multi-source satellite remote sensing image fusion method based on a multi-level framework according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] Specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described herein are for illustrative purposes only and are not intended to limit the present invention. In the following description, numerous specific details are set forth to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that these specific details are not necessarily required to practice the present invention. In other instances, well-known circuits, software, or methods are not specifically described to avoid obscuring the present invention.

[0030] Throughout this specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present invention. Therefore, appearances of the phrases "in one embodiment," "in an embodiment," "an example," or "an example" in various places throughout this specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, or characteristics may be combined in any suitable combinations and / or subcombinations in one or more embodiments or examples. Furthermore, those of ordinary skill in the art will appreciate that the figures provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0031] See Figure 1 An embodiment of the present invention provides a multi-source satellite remote sensing image fusion method based on a multi-level framework, the method comprising the following steps:

[0032] S1. Acquire multi-source satellite remote sensing image data, and pre-process the multi-source satellite remote sensing image data to obtain a fused image dataset.

[0033] In this embodiment, first satellite image data and second satellite image data are obtained, and the first satellite image data and the second satellite image data are used as multi-source satellite remote sensing image data; radiation correction and atmospheric correction are performed on the first satellite image data to obtain first surface reflectance data; radiation correction, atmospheric correction and geometric correction are performed on the second satellite image data to obtain second surface reflectance data; the first surface reflectance data and the second surface reflectance data are combined to construct the fused image dataset, including: aligning the first surface reflectance data and the second surface reflectance data; resampling the aligned second surface reflectance data to obtain a low-resolution-high-resolution image pair; and constructing the fused image dataset based on the low-resolution-high-resolution image pair.

[0034] In this example, Sentinel-2B (Sentinel-2B) and Gaofen-2 (GF-2) satellite remote sensing images are used as demonstrations. Sentinel-2B satellite remote sensing images serve as the first satellite image data, while GF-2 satellite remote sensing images serve as the second satellite image data. The Sentinel-2B satellite remote sensing images are radiometrically and atmospherically corrected to generate the first surface reflectance data; the GF-2 satellite remote sensing images are radiometrically, atmospherically, and geometrically corrected to generate the second surface reflectance data. Preprocessing reduces sensor differences between the two types of satellite remote sensing images and ensures sufficient registration accuracy, with a registration error of ≤1 pixel.

[0035] The two types of images that have completed preprocessing are used as the fusion input data of the progressive multi-level fusion strategy. Three fused image datasets are created according to the hierarchical fusion of 20m-10m, 10m-5m, and 5m-1m. Among them, there are 8645 20m-10m fused image pairs, 5984 10m-5m fused image pairs, and 3216 5m-1m fused image pairs.

[0036] Specifically, preprocessing was performed on Sentinel-2B (20m and 10m multispectral bands) and GF-2 (5m multispectral and 1m panchromatic band) satellite remote sensing images. The 10m band of Sentinel-2B satellite remote sensing images includes four optical bands: blue (B2: 458nm–523nm), green (B3: 543nm–578nm), red (B4: 650nm–680nm), and near-infrared (B8: 785nm–900nm). The 20m band of Sentinel-2B satellite remote sensing images includes six bands: red edge (B5–B7: 698nm–793nm), narrow near-infrared (B8A: 855nm–875nm), and shortwave infrared (B11–B12: 1565nm–2280nm). The multispectral bands of GF-2 satellite remote sensing images include blue (450nm–520nm), green (520nm–590nm), red (630nm–690nm), and near-infrared (770nm–890nm). During data preprocessing, they are resampled to 5m. The panchromatic band (450nm–900nm) of GF-2 satellite remote sensing images has a spatial resolution of 1m.

[0037] Specifically, the Sentinel-2B satellite remote sensing images are radiometrically corrected using the absolute radiometric calibration coefficients provided by the China Resources Satellite Application Center, satisfying the following relationship:

[0038]

[0039] in, is the radiance value, is the digital quantization value of the original image pixel, is the calibration gain coefficient, is the calibration offset.

[0040] It should be noted that the calibration offset is usually close to the radiance (unit: ).

[0041] Furthermore, the Python interface to the 6S Radiative Transfer Code (Py6S) module is used to perform atmospheric correction on the Sentinel-2B satellite remote sensing image to generate the first surface reflectance data.

[0042] Specifically, the GF-2 satellite remote sensing image is radiometrically corrected using the same radiometric correction method as that used for the Sentinel-2B satellite remote sensing image.

[0043] Furthermore, the fast atmospheric correction method (Fast Line-of-sight Atmospheric Analysis of Spectral Hypercubes, referred to as Flash) was used to perform atmospheric correction on the multispectral data of the GF-2 satellite remote sensing imagery. Then, the rational polynomial coefficients were combined with the ground control points to perform geometric correction on the GF-2 satellite remote sensing imagery to generate the second surface reflectance data.

[0044] In this embodiment, the second surface reflectance data of the GF-2 satellite remote sensing image is aligned with the first surface reflectance data of the Sentinel-2B satellite remote sensing image through the Geospatial Data Abstraction Library (GDAL) (error ≤ 1 pixel), and the multispectral band image of the aligned GF-2 satellite remote sensing image is resampled to a resolution of 5m.

[0045] See Figure 2 The figure shows a structural block diagram of a multi-source satellite remote sensing image fusion method based on a multi-level framework. Sentinel-2B satellite remote sensing images and GF-2 satellite remote sensing images are fused based on the multi-level framework. During implementation, a fusion dataset needs to be constructed. A training set is generated based on the Wald's Degradation Protocol (Wald). A Gaussian blur kernel is applied to high-resolution images (such as the panchromatic band of the GF-2 satellite remote sensing image), and downsampling is performed to generate low-resolution-high-resolution image pairs. The construction of the fusion dataset is completed by combining hierarchical fusion.

[0046] The above Gaussian blur kernel satisfies the following relationship:

[0047]

[0048] in, is the standard deviation of the Gaussian kernel, is the super-resolution factor.

[0049] S2. Constructing a progressive multi-level fusion strategy for the multi-source satellite remote sensing image data based on a multi-level framework.

[0050] In this example, a progressive multi-level fusion strategy is constructed based on the fusion of homologous satellite images, heterologous satellite images, and multispectral-panchromatic image fusion. This progressive multi-level fusion strategy reduces information loss during training data generation and fusion by incorporating the multi-level features of remote sensing images.

[0051] In an optional embodiment, the traditional fusion process satisfies the following relationship:

[0052]

[0053] in, The output high-resolution multispectral fusion image is represents the fusion function, is a pre-processed low-resolution multispectral image. High-resolution auxiliary images.

[0054] In this embodiment, the fusion process of the progressive multi-level fusion strategy satisfies the following relationship:

[0055]

[0056] in, For the The high-resolution multispectral fusion image output after level fusion is For the The spatial resolution of the fused input is For the Level fusion model, For pre-processed low-resolution multispectral images at all levels, For high-resolution auxiliary images at all levels, is the index variable of the fusion model, is the total number of fusions.

[0057] S3. Based on the fused image data set, the progressive multi-level fusion strategy is used to obtain multispectral high spatial resolution data of the multi-source satellite remote sensing image data.

[0058] In this embodiment, based on homologous satellite image fusion, the first satellite image data is subjected to multi-resolution data fusion to obtain multispectral enhanced data; based on heterologous satellite image fusion, the multispectral enhanced data is fused with the second satellite image data to obtain high spatial resolution data; based on multispectral-panchromatic image fusion, the high spatial resolution data and the panchromatic image of the second satellite image data are fused to obtain multispectral high spatial resolution data.

[0059] First, a 20m-resolution Sentinel-2B satellite remote sensing image was fused with a 10m-resolution image to increase the spatial detail of the 20m-resolution band to 10m resolution. The 10m Sentinel-2B satellite remote sensing image was then fused with a 5m GF-2 satellite remote sensing image to increase the spatial resolution of the Sentinel-2B satellite remote sensing image to 5m resolution. Finally, the 5m-resolution fused image was fused with a 1m panchromatic image from the GF-2 satellite remote sensing image to obtain 1m-resolution multispectral high-spatial-resolution data.

[0060] Specifically, the progressive multi-level fusion strategy gradually improves resolution through three stages: Stage 1, fusion of same-source satellite images (20m to 10m): 20m multispectral data from Sentinel-2B satellite remote sensing imagery is fused with 10m multispectral data to enhance spectral detail; Stage 2, fusion of heterogeneous satellite images (10m to 5m): 10m multispectral images from Sentinel-2B satellite remote sensing imagery fused in Stage 1 are fused with 5m multispectral images from GF-2 satellite remote sensing imagery to improve spatial resolution; Stage 3, 5m to 1m: 5m multispectral images from GF-2 satellite remote sensing imagery fused in Stage 2 are fused with 1m multispectral-panchromatic images from GF-2 satellite remote sensing imagery. Ultimately, the fused multispectral high-spatial-resolution data achieves a ground resolution of 1m. Each stage utilizes multi-resolution input collaborative optimization, using low-level fusion results as high-level prior knowledge to avoid information loss associated with direct fusion of cross-scale multi-source remote sensing imagery.

[0061] S4. Establish a multi-fusion method collaborative architecture, and obtain multi-source satellite remote sensing fusion image data based on the multi-fusion method collaborative architecture and the multi-spectral high spatial resolution data to achieve the fusion of multi-source satellite remote sensing images.

[0062] In this embodiment, a multi-fusion method collaborative architecture is constructed by combining a component replacement method, a multi-scale decomposition method, and a deep learning method; based on the component replacement method, component replacement is performed on the multispectral high spatial resolution data to obtain first multi-source satellite remote sensing fusion image data; according to the multi-scale decomposition method, multi-scale decomposition analysis is performed on the multispectral high spatial resolution data to obtain second multi-source satellite remote sensing fusion image data; based on the deep learning method, the multispectral high spatial resolution data is combined to obtain third multi-source satellite remote sensing fusion image data; the multi-source satellite remote sensing fusion image data is described as the first multi-source satellite remote sensing fusion image data, the second multi-source satellite remote sensing fusion image data, and the third multi-source satellite remote sensing fusion image data.

[0063] In this embodiment, the specific technical implementation of the multi-fusion method collaborative architecture is as follows:

[0064] (1) Component replacement method: Brovey transform (Brovey) directly replaces the brightness component by linearly weighting the normalized multispectral band and the panchromatic band; principal component analysis (PCA) performs principal component analysis on the multispectral image, selects the first principal component and replaces it after matching it with the panchromatic image histogram; Gram-Schmidt transform (GS) / Gram-Schmidt Adaptive Transform (GSA) algorithm uses Gram-Schmidt orthogonalization to construct a simulated panchromatic band and realizes component replacement through least squares regression.

[0065] Specifically, the Brovey transform method, based on color normalization and product operations, proportionally fuses each band of the low-resolution multispectral image with the high-resolution panchromatic image; the principal component analysis method projects the multispectral image into the principal component space, replaces the first principal component with the panchromatic image, and then reconstructs it by inverse transformation; the GS method generates a new component space by orthogonal projection, replaces the first component with the panchromatic image, and generates a fused image by inverse projection. The GSA method introduces adaptive weights to adjust the spectral injection amount and optimizes the spectral consistency of the GS transform. The above methods are classified as component replacement methods, and the calculation process of the component replacement method satisfies the following relationship:

[0066]

[0067] in, For the output high-resolution image, represents the inverse transformation of the color space, is the panchromatic image after histogram matching, Indicates the retained components that have not been replaced.

[0068] Furthermore, based on the component replacement method, the multispectral high spatial resolution data is subjected to component replacement to obtain the first multi-source satellite remote sensing fusion image data.

[0069] (2) Multi-scale decomposition method: The Modulation Transfer Function based Generalized Laplacian Pyramid (MTF_GLP) method designs a Gaussian low-pass filter based on the sensor modulation transfer function, performs multi-level pyramid decomposition on the full-color image, extracts high-frequency detail components at the N-level decomposition layer, and injects the details into the multispectral image after intensity-chrominance-saturation transformation by injecting gain coefficients; the Smoothing Filter-based Intensity Modulation (SFIM) method generates a synthetic full-color image through local spatial filtering, and uses ratio operations to inject high-frequency information into each multispectral band.

[0070] Specifically, SFIM uses spatial filtering to separate high-frequency details from panchromatic images and then injects them into multispectral images using the ratio method. Coupled Nonnegative Matrix Factorization (CNMF) jointly optimizes the nonnegative matrix factorization of multispectral and panchromatic images, reconstructing high-resolution images using an endmember-abundance model. These methods are categorized as multiscale decomposition methods, and the computational process of multiscale decomposition satisfies the following relationship:

[0071]

[0072] in, After fusion High-resolution multispectral imagery of the bands, For the Low-resolution multispectral images after band upsampling, is the gain coefficient, is the high-frequency component of the panchromatic image, is the original full-color image, Indicates low-pass filtering.

[0073] Furthermore, the multi-scale decomposition method is used to perform multi-scale decomposition analysis on the multi-spectral high spatial resolution data to obtain the second multi-source satellite remote sensing fusion image data.

[0074] (3) Deep learning: The Multi-Scale Deep Convolutional Neural Network (MSDCNN) adopts a dual-branch encoding and decoding architecture: the upper branch extracts the spatial texture features of the full-color image through 5-layer convolution, and the lower branch uses dilated convolution to expand the receptive field of the multispectral image, and finally realizes feature fusion and upsampling through the sub-pixel convolution layer; the Pan-Spectral Reconstruction Transformer (PSRT) method constructs a cascaded Transformer module, first establishes the long-range dependency relationship between the multispectral and full-color images through the multi-head self-attention mechanism, performs cross-modal interaction in the feature space, and finally uses a deformable convolutional network to complete the detail reconstruction.

[0075] Specifically, MSDCNN uses cascaded convolutional layers to extract multi-scale features and fuses spatial-spectral information through residual learning. PSRT, based on the Transformer's cross-modal attention mechanism, models the long-range dependencies between panchromatic and multispectral images. The Multi-task Spectral-Spatial Transformer (MSST) jointly optimizes spectral fidelity and spatial detail tasks, extracting features in parallel through spectral and spatial branches. The above methods are classified as deep learning-based methods. The computational process of deep learning methods includes:

[0076] First, the image is fused through network calculation, and the calculation process satisfies the following relationship:

[0077]

[0078] in, For the output high-resolution image, is a parameterized neural network, represents the model parameters, is the original full-color image, It is a low-resolution multispectral image after upsampling.

[0079] Second, the neural network parameters are optimized by calculating the mean square error loss to satisfy the following relationship:

[0080]

[0081] in, is the mean square error loss value, is the number of batch training samples, is the number of multispectral bands, is the sample index variable, is the band index variable, For the Band No. The predicted high-resolution image of samples, For the Band No. Real high-resolution images of samples, represents the square of the 2-norm.

[0082] Furthermore, based on the deep learning method, the third multi-source satellite remote sensing fusion image data was obtained by combining multispectral high spatial resolution data.

[0083] In this embodiment, the multi-source satellite remote sensing fusion image data is represented by first multi-source satellite remote sensing fusion image data, second multi-source satellite remote sensing fusion image data and third multi-source satellite remote sensing fusion image data.

[0084] S5. Construct a multi-dimensional quantitative evaluation system, and perform quality evaluation on the multi-source satellite remote sensing fusion image data based on the multi-dimensional quantitative evaluation system to obtain a fusion evaluation result.

[0085] In this embodiment, six evaluation indicators are used to evaluate the fusion results: (1) Compare the difference between the fused image and the reference image pixel by pixel to calculate the Peak Signal-to-Noise Ratio (PSNR); (2) Calculate the local brightness, contrast, and structural similarity through a sliding window (such as 11×11), take the global mean, and calculate the Structural Similarity Index (SSIM); (3) Calculate the spectral vector angle pixel by pixel, calculate the global average (in degrees), and calculate the Spectral Angle Mapper (SAM); (4) Calculate the normalized root mean square error for each band, perform weighted averaging and scaling, and calculate the Relative Global Dimensionless Synthesis Error (ERGAS); (5) Directly calculate the mean square of the pixel-level difference between the fused image and the reference image to obtain the Mean Squared Error (MSE); (6) Calculate the ratio of the covariance to the standard deviation of the fused image and the reference image in the range [-1,1] to obtain the Correlation Coefficient (C Coefficient, referred to as CC).

[0086] Peak signal-to-noise ratio (PSNR) is used to quantify pixel-level reconstruction accuracy. PSNR evaluates pixel-level reconstruction quality by comparing the maximum possible signal strength and error strength of the image. A larger value indicates less noise interference in the fused image and higher accuracy. The calculation steps satisfy the following relationship:

[0087]

[0088] in, is the peak signal-to-noise ratio, is the theoretical maximum value of the pixel value, is the mean square error.

[0089] The Structural Similarity Index (SSIM) measures the fidelity of texture and structure. SSIM measures the similarity of two images by integrating brightness, contrast, and structural information. The closer the value is to 1, the better the structural fidelity. The calculation steps satisfy the following relationship:

[0090]

[0091] in, is the structural similarity index, and is the mean of the two images, is the covariance, and is the variance, and is a constant used to stabilize the calculation.

[0092] Spectral fidelity assessment is based on the calculation of pixel spectral vector deviation based on spectral angle (SAM). SAM evaluates the degree of spectral distortion by calculating the radian value of the angle between spectral vectors. The smaller the value (closer to 0), the higher the spectral fidelity. The calculation steps satisfy the following relationship:

[0093]

[0094] in, is the spectral angle, is the inverse cosine function, is the total number of bands of the image, is the band index variable, For the reference image The spectral vector value of the band, To fuse the images The spectral vector value of the band.

[0095] The global relative error (ERGAS) is calculated to evaluate the degree of multi-band joint spectral distortion. ERGAS integrates multi-band errors and resolution changes. The smaller the value, the lower the spectral distortion. The calculation steps satisfy the following relationship:

[0096]

[0097] in, is the global relative error, is the spatial resolution ratio of the fused image to the original image, is the total number of bands of the image, is the band index variable, For the The root mean square error of the band, The reference image The pixel mean of the band.

[0098] It should be noted that the spatial resolution ratio of the fused image to the original image is: if the resolution is increased from 10m to 5m .

[0099] The mean square error (MSE) is calculated to represent the overall error level. MSE quantifies the overall error level of the image by calculating the square mean of the difference pixel by pixel. The smaller the value, the higher the fusion accuracy. The calculation steps satisfy the following relationship:

[0100]

[0101] in, is the mean square error, is the total number of pixels in a single band, is the total number of bands of the image, is the band index variable, is the pixel index variable, To fuse the images Band No. The value of pixels, For the reference image Band No. The value of pixels.

[0102] Spatial quality assessment uses the correlation coefficient (CC) to calculate the spatial correlation between the fused image and the reference data by covariance and variance analysis. CC quantifies the spatial correlation by calculating the ratio of the covariance of the two images to the standard deviation. The closer the value is to 1, the higher the spatial consistency. The calculation steps satisfy the following relationship:

[0103]

[0104] in, is the correlation coefficient, is the total number of pixels in a single band, is the pixel index variable, The first The gray value of a pixel, is the pixel mean of the fused image, The reference image The gray value of a pixel, is the pixel mean of the reference image.

[0105] See Figure 3 In an optional embodiment, the present invention provides a multi-source satellite remote sensing image fusion system based on a multi-level framework. The system includes an input device, an output device, a processor, and a memory, wherein the hardware components are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to call the program instructions and execute the specific steps of the embodiments of the multi-source satellite remote sensing image fusion method based on a multi-level framework provided by the present invention. The multi-source satellite remote sensing image fusion system based on a multi-level framework provided by the present invention has a complete structure, objective stability, and enhances the overall applicability and practical application capabilities of the present invention.

[0106] See Figure 4 , the figure is a schematic diagram of the execution process of a multi-source satellite remote sensing image fusion system based on a multi-level framework; in an optional embodiment, the multi-source satellite remote sensing image fusion system based on a multi-level framework includes a data preprocessing module, a hierarchical fusion module, a multi-level output module and a fusion result evaluation module.

[0107] The data preprocessing module is used to preprocess the multi-source satellite remote sensing image data to obtain a fused image data set.

[0108] The hierarchical fusion module is used to construct the progressive multi-level fusion strategy and the multi-fusion method collaborative architecture, and fuse the multi-source satellite remote sensing image data to obtain multi-source satellite remote sensing fusion image data.

[0109] The multi-level output module outputs intermediate and final resolutions (10m, 5m, and 1m) after each fusion level. High-level fusion reuses lower-level features, avoiding the error accumulation associated with direct fusion across a wide range of resolutions. Users can select the output level as needed, balancing computational efficiency and accuracy.

[0110] The fusion result evaluation module is used to construct a multi-dimensional quantitative evaluation system to perform quality evaluation on the multi-source satellite remote sensing fusion image data to obtain a fusion evaluation result.

[0111] In an optional embodiment, Sentinel-2B Level-1C data, including 10m (blue, green, red, and near-infrared) and 20m (red-edge and narrow near-infrared) resolution bands, was acquired from the European Space Agency (ESA). GF-2 Level-1A data, including 1m panchromatic and 4m multispectral imagery, was acquired from the China Resources Satellite Center. The satellite remote sensing imagery covered the Hushan Reservoir and its surrounding areas in Wenchang City, Hainan Province.

[0112] The reservoir's surrounding landscape is rich and diverse, including eutrophic waters, surrounding small lakes and reservoirs, rivers, the main dam, auxiliary dams, winding roads (both solidified artificial roads and natural trails), drawdown zones, grasslands, aquatic plants (such as water hyacinth and reeds), cash crops (such as pineapples), and other natural vegetation. The area also contains artificial structures such as residential buildings, industrial buildings, fish ponds, and turtle ponds. High-resolution multispectral / hyperspectral satellite remote sensing imagery is urgently needed to effectively manage the region's ecosystems and water resources, particularly to address the complex land cover landscape and monitor water quality.

[0113] A total of six GF-2 panchromatic and multispectral high-spatial-resolution data sets and two Sentinel-2B multispectral high-spatial-resolution data sets were used. First, a fusion system for GF-2 and Sentinel-2B images of the Hushan Reservoir and surrounding area was constructed, consisting of a data preprocessing module, a hierarchical fusion module, and a fusion quality assessment module. By inputting the GF-2 high-resolution images and Sentinel-2B multispectral images, high-quality, hierarchical fusion images were generated, laying the data foundation for remote sensing monitoring of the Hushan Reservoir and surrounding area.

[0114] The remote sensing images generated by step-by-step fusion were quantitatively evaluated using the following parameters: correlation coefficient (CC), structural similarity index (SSIM), spectral angle (SAM), global relative error (ERGAS), mean square error (MSE), and peak signal-to-noise ratio (PSNR). The evaluation results are shown in Table 1:

[0115] Table 1

[0116]

[0117] Among them, bicubic interpolation (Bicubic) calculates new pixel values ​​based on surrounding pixels and is used for image scaling and rotation, balancing smoothness and detail retention.

[0118] Fusion results of Sentinel-2B data using the 20m and 10m resolution bands as a reference revealed that deep learning methods (MSST, MSDCNN, and PSRT) outperformed traditional methods in terms of PSNR, with margins of at least 13. MSST achieved the highest PSNR of 48.05, demonstrating strong noise suppression capabilities. For SAM, MSDCNN achieved the lowest value of 1.223°, 14% higher than PSRT, while Brovey achieved 2.005°, highlighting the effectiveness of component replacement methods in preserving spectral features. In terms of the ERGAS metric, PSRT achieved the highest score of 0.3282, a 62% improvement over GS, while SFIM (1.4283) and coupled non-negative matrix factorization (CNMF) (1.343) demonstrated the limitations of multi-resolution decomposition methods. SSIM showed that MSDCNN achieved a similarity of 0.991, highly consistent with human visual perception, while Brovey and GS, with a similarity of 0.8984, still faced challenges with local texture degradation. This study systematically evaluated the fusion performance of Sentinel-2B (a multispectral band with 10m resolution) and GF-2 multispectral-panchromatic imagery downscaled to 5m spatial resolution using different fusion methods. The results showed that the different fusion methods exhibited similar performance differences across datasets with varying resolutions (20m to 10m) and (10m to 5m). In experiments with homologous satellite remote sensing imagery fusion, the MSDCNN method demonstrated the best overall performance (PSNR=47.68dB, ERGAS=0.3413), particularly in spectral angle mapping (SAM=1.223) and structural similarity (SSIM=0.991). For heterogeneous sensor data fusion, the MSDCNN maintained its leading position (PSNR=46.96dB, CC=0.9983), while the PSRT algorithm maintained a constraint parameter (CC) above 0.98 for both fusion tasks, highlighting the advantages of the Transformer architecture in its cross-modal attention mechanism. It is noteworthy that the average ERGAS value for homologous fusion (0.815±0.371) is significantly lower than that for heterologous fusion (1.193±0.893), indicating that homologous image fusion is more effective than heterologous image fusion. Furthermore, deep learning methods significantly outperform traditional methods on real data, particularly when spectra do not overlap. Deep learning methods better restore image details and maintain the spectral characteristics of the original data.

[0119] See Figure 5The figure shows a schematic diagram of the fused image output by the multi-source satellite remote sensing image fusion method based on a multi-level framework; in terms of image restoration, the sharpening effect at 5 times the resolution is more obvious than the melting effect at 2 times, and small ground objects, roads and scattered plots appear clearer.

[0120] The hierarchical fusion framework effectively overcomes the information loss common in traditional image fusion methods by progressively fusing images of varying spatial resolutions (from 20m and 10m resolution fusion of homologous images, to 10m and 5m resolution fusion of heterogeneous images, and finally, from multi-source 5m resolution images to 1m resolution). At each fusion stage, the framework ensures a gradual enhancement of spatial detail and outputs the fusion results at multiple resolution levels, thereby improving overall image quality. This progressive fusion strategy is particularly well-suited for remote sensing data with multiple spectral bands, effectively balancing the preservation of spectral and spatial information and avoiding the detail loss that occurs during image sharpening in traditional methods. Furthermore, the hierarchical fusion framework demonstrates excellent scalability and generalization capabilities, maintaining high accuracy and stability when fusing remote sensing data from different sensors or at different resolutions.

[0121] The size of spatial resolution has a significant impact on the fusion effect. Fusion tasks with large spatial resolution differences are more difficult to achieve than those with small spatial resolution differences. This was analyzed by comparing indicator data with 5x and 2x resolution differences. Smaller resolution differences (such as 2x) significantly improve fusion quality, including significant increases in PSNR, SSIM, and CC, and a significant decrease in MSE, indicating better fusion results. In terms of spectral fidelity, spectral errors generally increase as the resolution difference increases. In addition, deep learning performs better in fusion of images with small resolution differences. Larger resolution differences make it difficult to recover spatial details, reduce spectral fidelity, and weaken the correlation with the reference image. However, when the resolution difference decreases, the fusion effect of deep learning methods is significantly improved, while the fusion performance of traditional methods improves but only to a limited extent.

[0122] This paper proposes a progressive fusion architecture for multi-source satellites, addressing the information loss problem in traditional methods of fusion with large resolution differences, and achieving high-quality fusion of multi-source satellite imagery. This also demonstrates the advantages of deep learning in remote sensing image fusion. The PSNR of MSDCNN and PSRT is significantly improved compared to traditional methods (such as GS), providing a superior data foundation for high-resolution remote sensing missions and supporting improved accuracy in downstream applications such as water extraction and land feature classification.

[0123] In summary, the method of the present invention provides a multi-source satellite remote sensing image fusion method and system based on a multi-level framework, which significantly improves the accuracy of multi-source satellite remote sensing image fusion through a closed loop of data preprocessing-hierarchical fusion-multi-method collaboration-fusion result evaluation, provides multispectral images with high-resolution characteristics, and provides important technical support for the development of high-resolution remote sensing image applications. The method of the present invention is easy to understand, simple to calculate, has a small workload, is convenient for engineering application, and provides a theoretical basis and technical support for the further development of the aerospace information field.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

Claims

1. A multi-source satellite remote sensing image fusion method based on a multi-level framework, characterized in that: The steps include: Acquiring multi-source satellite remote sensing image data, and preprocessing the multi-source satellite remote sensing image data to obtain a fused image dataset; Constructing a progressive multi-level fusion strategy for the multi-source satellite remote sensing image data based on a multi-level framework; Based on the fused image data set, the progressive multi-level fusion strategy is used to obtain multispectral high spatial resolution data of the multi-source satellite remote sensing image data; Establishing a multi-fusion method collaborative framework, obtaining multi-source satellite remote sensing fusion image data based on the multi-fusion method collaborative framework and combining the multi-spectral high spatial resolution data to achieve fusion of multi-source satellite remote sensing images; The fusion process of the progressive multi-level fusion strategy satisfies the following relationship: ； in, For the The high-resolution multispectral fusion image output after level fusion is For the The spatial resolution of the fused input is For the Level fusion model, For pre-processed low-resolution multispectral images at all levels, For high-resolution auxiliary images at all levels, is the index variable of the fusion model, is the total number of fusions.

2. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 1 is characterized in that: The acquiring of multi-source satellite remote sensing image data and preprocessing the multi-source satellite remote sensing image data to obtain a fused image dataset includes: Acquire first satellite image data and second satellite image data, and use the first satellite image data and the second satellite image data as the multi-source satellite remote sensing image data; Performing radiation correction and atmospheric correction on the first satellite image data to obtain first surface reflectance data; performing radiometric correction, atmospheric correction, and geometric correction on the second satellite image data to obtain second surface reflectance data; The fused image dataset is constructed by combining the first surface reflectance data and the second surface reflectance data.

3. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 2 is characterized in that: The step of combining the first surface reflectance data and the second surface reflectance data to construct the fused image dataset includes: registering the first surface reflectivity data and the second surface reflectivity data; Resampling the registered second surface reflectance data to obtain a low-resolution-high-resolution image pair; The fused image dataset is constructed based on the low-resolution-high-resolution image pairs.

4. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 3 is characterized in that: The progressive multi-level fusion strategy for constructing the multi-source satellite remote sensing image data based on the multi-level framework includes: The progressive multi-level fusion strategy is constructed based on the fusion of homologous satellite images, heterologous satellite images and multispectral-panchromatic image fusion.

5. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 4 is characterized in that: The step of obtaining multispectral high spatial resolution data of the multi-source satellite remote sensing image data using the progressive multi-level fusion strategy based on the fused image dataset includes: Based on the fusion of the same-source satellite images, performing multi-resolution data fusion on the first satellite image data to obtain multispectral enhanced data; According to the heterogeneous satellite image fusion, the multispectral enhanced data is fused with the second satellite image data to obtain high spatial resolution data; According to the multispectral-panchromatic image fusion, the high spatial resolution data and the panchromatic image of the second satellite image data are fused to obtain the multispectral high spatial resolution data.

6. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 1, characterized in that: The establishment of a multi-fusion method collaborative architecture includes: The multi-fusion method collaborative architecture is constructed by combining component replacement method, multi-scale decomposition method and deep learning method.

7. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 6, characterized in that: The method of combining the multi-spectral high spatial resolution data with the multi-fusion method collaborative architecture to obtain multi-source satellite remote sensing fusion image data and realize the fusion of multi-source satellite remote sensing images includes: Performing component replacement on the multispectral high spatial resolution data based on the component replacement method to obtain first multi-source satellite remote sensing fusion image data; Performing multi-scale decomposition analysis on the multispectral high spatial resolution data according to the multi-scale decomposition method to obtain second multi-source satellite remote sensing fusion image data; According to the deep learning method, the multispectral high spatial resolution data is combined to obtain third multi-source satellite remote sensing fusion image data; The first multi-source satellite remote sensing fusion image data, the second multi-source satellite remote sensing fusion image data and the third multi-source satellite remote sensing fusion image data are used as the multi-source satellite remote sensing fusion image data.

8. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 6, characterized in that: The multi-fusion method collaborative architecture includes: The component replacement method satisfies the following relationship: ； in, For the output high-resolution image, represents the inverse transformation of the color space, is the panchromatic image after histogram matching, Indicates the retained components that have not been replaced; The multi-scale decomposition method satisfies the following relationship: ； in, After fusion High-resolution multispectral imagery of the bands, For the Low-resolution multispectral images after band upsampling, is the gain coefficient, is the high-frequency component of the panchromatic image, is the original full-color image, Represents low-pass filtering; The deep learning method satisfies the following relationship: ； in, For the output high-resolution image, is a parameterized neural network, represents the model parameters, is the original full-color image, It is a low-resolution multispectral image after upsampling.

9. A multi-source satellite remote sensing image fusion system based on a multi-level framework, characterized in that: The system includes an input device, an output device, a processor and a memory, wherein the input device, the output device, the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the multi-source satellite remote sensing image fusion method based on a multi-level framework according to any one of claims 1 to 8.

Images