Multi-source satellite remote sensing image fusion method and system based on multistage framework

Abstract

The invention relates to the field of aerospace information, in particular to a multi-source satellite remote sensing image fusion method and system based on a multilevel framework, and the method comprises the following steps: obtaining multi-source satellite remote sensing image data, and preprocessing the multi-source satellite remote sensing image data to obtain a fused image data set; constructing a progressive multi-level fusion strategy of the multi-source satellite remote sensing image data based on a multi-level framework; according to the fused image data set, obtaining multispectral high-spatial-resolution data of the multisource satellite remote sensing image data by using the progressive multi-level fusion strategy; and establishing a multi-fusion method collaborative architecture, and obtaining multi-source satellite remote sensing fusion image data according to the multi-fusion method collaborative architecture in combination with the multispectral high-spatial resolution data, thereby realizing fusion of the multi-source satellite remote sensing images. According to the method, the fusion precision of the multi-source satellite image is improved, the multi-source satellite remote sensing fusion image data is obtained, and technical support is provided for development of high-resolution remote sensing image application.

Description

Technical Field

[0001] The present invention relates to the field of aerospace information, and in particular, to a multi-source satellite remote sensing image fusion method and system based on a multi-level framework. Background Art

[0002] Spectral resolution and spatial resolution have always been the core parameters concerned in the field of remote sensing image processing. However, due to the technical limitations of sensors, there are often trade-offs between spatial resolution and spectral resolution of sensors. These limitations make it challenging for remote sensing images to perform fine ground object classification, complex information extraction, and parameter inversion. How to obtain higher-resolution satellite remote sensing images for high-precision earth observation has always been a hot and difficult issue in domestic and foreign research.

[0003] The purpose of remote sensing image fusion is to intuitively and effectively improve the spatial resolution of multi-spectral images or hyperspectral images, so as to enhance and enrich the information and details of the images, and can be more widely applied to fields such as ground object target classification and global change monitoring. The pixel-level fusion method is one of the three basic methods in the field of satellite remote sensing image fusion. However, the existing pixel-level fusion methods are often only applicable to a single satellite, and the fusion effect for multi-source satellite images is poor. Moreover, when the ground resolution difference between panchromatic and multi-spectral images is large, the fusion effect is poor. Therefore, it is urgent to develop multi-source satellite remote sensing image fusion technology to meet more application requirements.

[0004] In order to overcome the problems existing in the existing multi-source remote sensing image fusion technology, the present invention provides a multi-source satellite remote sensing image fusion method and system based on a multi-level framework. The present invention is applicable to the fusion of multi-source satellite images with large spatial resolution differences. The remote sensing images fused by using the present invention can provide enhanced information and more abundant details, and provide data guarantee for the downstream task applications such as ground object target recognition and classification, information extraction, and parameter inversion based on remote sensing images. Summary of the Invention

[0005] Aiming at the defects in the prior art, the present invention provides a multi-source satellite remote sensing image fusion method and system based on a multi-level framework.

[0006] To achieve the above object, in a first aspect, the present invention provides a multi-source satellite remote sensing image fusion method based on a multi-level framework. The method includes the following steps: obtaining multi-source satellite remote sensing image data, and preprocessing the multi-source satellite remote sensing image data to obtain a fused image dataset; constructing a progressive multi-level fusion strategy for the multi-source satellite remote sensing image data based on the multi-level framework; according to the fused image dataset, using the progressive multi-level fusion strategy to obtain multi-spectral high-spatial-resolution data of the multi-source satellite remote sensing image data; establishing a multi-fusion method collaborative architecture, and combining the multi-spectral high-spatial-resolution data according to the multi-fusion method collaborative architecture to obtain multi-source satellite remote sensing fused image data, so as to realize the fusion of multi-source satellite remote sensing images. Through the design of the multi-level framework and the collaborative architecture, the present invention realizes the efficient integration and quality improvement of image data. First, in the preprocessing stage, the radiation difference and geometric distortion between multi-source data are effectively eliminated, providing a standardized input for subsequent fusion; secondly, the progressive multi-level fusion strategy enhances the spatial resolution while retaining spectral details through staged feature extraction and progressive information fusion; in the multi-spectral image generation link, the multi-category fusion methods are compared through the multi-method collaborative architecture, significantly improving the spectral fidelity and geometric accuracy of the fused image; effectively solving the fusion problem of large scale differences in multi-source data, and providing higher-precision spatio-temporally consistent image data for applications such as remote sensing monitoring and ground object classification.

[0007] Optionally, the step of obtaining multi-source satellite remote sensing image data and preprocessing the multi-source satellite remote sensing image data to obtain a fused image dataset includes: obtaining first satellite image data and second satellite image data, and using the first satellite image data and the second satellite image data as the multi-source satellite remote sensing image data; performing radiation correction and atmospheric correction on the first satellite image data to obtain first surface reflectance data; performing radiation correction, atmospheric correction, and geometric correction on the second satellite image data to obtain second surface reflectance data; combining the first surface reflectance data and the second surface reflectance data to construct the fused image dataset. Through the integration and standardization processing of multi-source data, the present invention significantly improves the quality of remote sensing images; provides richer original information for subsequent fusion; performing radiation correction and atmospheric correction on the first satellite image effectively eliminates the influence of atmospheric scattering and obtains the true surface reflectance; performing multi-stage preprocessing of radiation correction, atmospheric correction, and geometric correction on the second satellite image synchronously solves the problems of sensor response difference, atmospheric interference, and geometric distortion, ensuring the consistency of multi-source data and providing data support for constructing a high-precision fused image dataset.

[0008] Optionally, constructing the fused image dataset by combining the first surface reflectance data and the second surface reflectance data includes: registering the first surface reflectance data and the second surface reflectance data; resampling the registered second surface reflectance data to obtain a low-resolution - high-resolution image pair; constructing the fused image dataset based on the low-resolution - high-resolution image pair. Through the fine registration and scale alignment before multi-source data fusion, the present invention significantly improves the geometric accuracy and information compatibility of image fusion; through the registration process, it ensures the strict correspondence of multi-source data in the same geographic coordinate system, providing a spatial reference for subsequent fusion; resampling the second surface reflectance data realizes the scale matching of spectral information and spatial details, and then constructs a fused image dataset with multi-scale feature associations, providing a structured input for subsequent multi-level fusion strategies, effectively improving the spatial continuity and spectral authenticity of the fused image.

[0009] Optionally, the progressive multi-level fusion strategy for constructing the multi-source satellite remote sensing image data based on a multi-level framework includes: constructing the progressive multi-level fusion strategy based on homologous satellite image fusion, heterologous satellite image fusion, and multi-spectral - panchromatic image fusion. By constructing the progressive multi-level fusion strategy, the present invention realizes the deep integration and optimization of remote sensing image information; by performing homologous, heterologous, and multi-spectral - panchromatic image fusion in stages, gradually refining the image features, effectively balancing the contradiction between spectral fidelity and spatial resolution, and significantly improving the comprehensive quality of the fused image.

[0010] Optionally, the progressive multi-level fusion strategy includes: the fusion process of the progressive multi-level fusion strategy satisfies the following relationship: ; where is the high-resolution multi-spectral fused image output after the -th level of fusion, is the spatial resolution input for the -th level of fusion, is the -th level of fusion model, are the preprocessed low-resolution multi-spectral images at each level, are the high-resolution auxiliary images at each level, is the index variable of the fusion model, is the total number of fusions. Through a step-by-step progressive fusion model, the present invention realizes the collaborative optimization of the spatial resolution and spectral quality of remote sensing images; each level of the fusion model performs feature fusion on the low-resolution multispectral image and the high-resolution auxiliary image, gradually enhancing the spatial detail expressiveness of the output image; effectively balancing the contradiction between spectral fidelity and spatial resolution, through progressive optimization of multiple models, avoiding information loss caused by direct fusion of a single model, so as to obtain high-quality multispectral high-spatial-resolution data.

[0011] Optionally, obtaining the multispectral high-spatial-resolution data of the multi-source satellite remote sensing images by using the progressive multi-level fusion strategy based on the fused image dataset includes: based on the homologous satellite image fusion, performing multi-resolution data fusion on the first satellite image data to obtain multispectral enhanced data; according to the heterogeneous satellite image fusion, fusing the multispectral enhanced data with the second satellite image data to obtain high-spatial-resolution data; based on the multispectral-panchromatic image fusion, fusing the high-spatial-resolution data and the panchromatic image of the second satellite image to obtain the multispectral high-spatial-resolution data. Through the progressive multi-level fusion strategy, the present invention significantly improves the comprehensive quality of multi-source satellite remote sensing images; in the homologous satellite fusion stage, the multi-spectral detail expressiveness of the first satellite image is enhanced through multi-resolution fusion; in the multi-source satellite fusion stage, the enhanced multi-spectral data and the second satellite image are integrated to achieve the collaborative improvement of spectral information and spatial resolution; in the multispectral-panchromatic image fusion stage, the multispectral high-spatial-resolution data is obtained by fusing the high-resolution panchromatic band and the multi-spectral data, improving the data quality.

[0012] Optionally, establishing the collaborative architecture of the multi-fusion method includes: constructing the collaborative architecture of the multi-fusion method by combining the component substitution method, the multi-scale decomposition method, and the deep learning method. The present invention constructs a collaborative architecture of multi-fusion methods with complementary advantages by integrating the component substitution method, the multi-scale decomposition method, and the deep learning method. The component substitution method ensures spectral information fidelity, the multi-scale decomposition method strengthens spatial detail expression, and the deep learning method optimizes non-linear feature fusion. By comparing multiple types of fusion methods, the comprehensive quality of the fused image is improved.

[0013] Optionally, the multi-fusion method collaborative architecture combines the multi-spectral high-spatial-resolution data to obtain multi-source satellite remote sensing fusion image data, realizing the fusion of multi-source satellite remote sensing images, including: performing component substitution on the multi-spectral high-spatial-resolution data based on the component substitution method to obtain the first multi-source satellite remote sensing fusion image data; performing multi-scale decomposition analysis on the multi-spectral high-spatial-resolution data according to the multi-scale decomposition method to obtain the second multi-source satellite remote sensing fusion image data; based on the deep learning method, combining the multi-spectral high-spatial-resolution data to obtain the third multi-source satellite remote sensing fusion image data; using the first multi-source satellite remote sensing fusion image data, the second multi-source satellite remote sensing fusion image data, and the third multi-source satellite remote sensing fusion image data as the multi-source satellite remote sensing fusion image data. The present invention improves the comprehensive quality of multi-source satellite remote sensing images through different fusion methods; the collaboratively generated multi-source fusion image data has both high spectral resolution, high spatial resolution, and strong environmental adaptability, providing an accurate data basis for applications such as remote sensing monitoring and ground object classification.

[0014] Optionally, the multi-fusion method collaborative architecture includes: The component substitution method satisfies the following relationship: ; where is the output high-resolution image, represents the inverse transformation of the color space, is the panchromatic image after histogram matching, represents the remaining components that are not replaced; The multi-scale decomposition method satisfies the following relationship: ; where is the high-resolution multi-spectral image of the th band after fusion, is the low-resolution multi-spectral image of the th band after upsampling, is the gain coefficient, is the high-frequency component of the panchromatic image, is the original panchromatic image, represents low-pass filtering; The deep learning method satisfies the following relationship: ; where is the output high-resolution image, is the parameterized neural network, represents the model parameters, is the original panchromatic image, is the low-resolution multispectral image after upsampling. The present invention completes the fusion of multispectral high-spatial-resolution data through a multi-fusion method collaborative architecture, optimizes the fusion performance of complex ground object features, and significantly improves the spectral fidelity, spatial resolution of the fused image, and the adaptability to complex scenes.

[0015] In a second aspect, the present invention provides a multi-source satellite remote sensing image fusion system based on a multi-level framework. The system executes the multi-source satellite remote sensing image fusion method provided by the present invention. The system includes an input device, an output device, a processor, and a memory. The input device, the output device, the processor, and the memory are interconnected. Among them, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions. The present invention constructs an efficient multi-source satellite remote sensing image fusion processing platform by integrating high-performance hardware, realizes the fusion of remote sensing images, and provides stable and efficient hardware support for large-scale multi-source image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of a multi-source satellite remote sensing image fusion method based on a multi-level framework according to an embodiment of the present invention;

[0017] Figure 2 is a structural block diagram of a multi-source satellite remote sensing image fusion method based on a multi-level framework according to an embodiment of the present invention;

[0018] Figure 3 is a framework diagram of a multi-source satellite remote sensing image fusion system based on a multi-level framework according to an embodiment of the present invention;

[0019] Figure 4 is a schematic diagram of the execution process of a multi-source satellite remote sensing image fusion system based on a multi-level framework according to an embodiment of the present invention;

[0020] Figure 5 is a schematic diagram of the fused image output by a multi-source satellite remote sensing image fusion method based on a multi-level framework according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] Specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described here are only for illustrative purposes and are not intended to limit the present invention. In the following description, in order to provide a thorough understanding of the present invention, a large number of specific details are set forth. However, it is obvious to those of ordinary skill in the art that the present invention does not have to employ these specific details. In other instances, well-known circuits, software, or methods have not been specifically described to avoid obscuring the present invention.

[0022] Throughout the specification, references to "one embodiment", "an embodiment", "an example", or "an example" mean that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present invention. Thus, the phrases "in one embodiment", "in an embodiment", "an example", or "an example" that appear throughout the specification do not necessarily all refer to the same embodiment or example. Additionally, the particular features, structures, or characteristics may be combined in any suitable combination and / or sub - combination in one or more embodiments or examples. Further, those of ordinary skill in the art should understand that the diagrams provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0023] Please refer to Figure 1 , an embodiment of the present invention provides a multi - source satellite remote sensing image fusion method based on a multi - level framework, and the method includes the following steps:

[0024] S1. Obtain multi - source satellite remote sensing image data, and pre - process the multi - source satellite remote sensing image data to obtain a fused image dataset.

[0025] In this embodiment, first satellite image data and second satellite image data are obtained, and the first satellite image data and the second satellite image data are used as the multi - source satellite remote sensing image data; the first satellite image data is subjected to radiometric correction and atmospheric correction to obtain first surface reflectance data; the second satellite image data is subjected to radiometric correction, atmospheric correction, and geometric correction to obtain second surface reflectance data; the fused image dataset is constructed by combining the first surface reflectance data and the second surface reflectance data, including: registering the first surface reflectance data and the second surface reflectance data; resampling the registered second surface reflectance data to obtain a low - resolution - high - resolution image pair; constructing the fused image dataset based on the low - resolution - high - resolution image pair.

[0026] In this embodiment, taking the Sentinel - 2B satellite remote sensing image and the GF - 2 satellite remote sensing image as an example, the Sentinel - 2B satellite remote sensing image is used as the first satellite image data, and the GF - 2 satellite remote sensing image is used as the second satellite image data. Radiometric correction and atmospheric correction are performed on the Sentinel - 2B satellite remote sensing image to generate first surface reflectance data; radiometric correction, atmospheric correction, and geometric correction are performed on the GF - 2 satellite remote sensing image to generate second surface reflectance data. The sensor differences between the two types of satellite remote sensing images are reduced through pre - processing, and sufficient registration accuracy is ensured, with the registration error ≤ 1 pixel.

[0027] The two types of images that have completed preprocessing are used as the fusion input data for the progressive multi-level fusion strategy, and three fusion image datasets are created respectively according to the hierarchical fusion of 20m-10m, 10m-5m, and 5m-1m; among them, there are 8,645 pairs of 20m-10m fusion images, 5,984 pairs of 10m-5m fusion images, and 3,216 pairs of 5m-1m fusion images.

[0028] Specifically, preprocess the Sentinel-2B (20m, 10m multi-spectral bands) satellite remote sensing images and GF-2 (5m multi-spectral, 1m panchromatic band) satellite remote sensing images. The 10m band of the Sentinel-2B satellite remote sensing images includes four optical bands: blue (B2: 458nm–523nm), green (B3: 543nm–578nm), red (B4: 650nm–680nm), and near-infrared (B8: 785 nm–900nm). The 20m band of the Sentinel-2B satellite remote sensing images covers six bands: red edge (B5–B7: 698nm–793nm), narrow near-infrared (B8A: 855nm–875nm), and short-wave infrared (B11–B12: 1565nm–2280nm). The multi-spectral bands of the GF-2 satellite remote sensing images include four bands: blue (450nm–520nm), green (520nm–590nm), red (630nm–690nm), and near-infrared (770nm–890nm), and are resampled to 5m during the data preprocessing. The panchromatic band (450nm–900nm) of the GF-2 satellite remote sensing images has a 1m spatial resolution.

[0029] Specifically, perform radiometric calibration on the Sentinel-2B satellite remote sensing images using the absolute radiometric calibration coefficients provided by the China National Space Administration Center for Resources Satellite Data and Application, satisfying the following relationship:

[0030]

[0031] Among them, is the radiance value, is the digital quantization value of the original image pixel, is the calibration gain coefficient, is the calibration offset.

[0032] It should be noted that the calibration offset is usually close to the radiance, (unit: )

[0033] Further, the Python interface to the 6S Radiative Transfer Code (Py6S) module is used to perform atmospheric correction on the Sentinel-2B satellite remote sensing image to generate the first surface reflectance data.

[0034] Specifically, the GF-2 satellite remote sensing image is radiometrically corrected using the same radiometric correction method as the Sentinel-2B satellite remote sensing image.

[0035] Further, the Fast Line-of-sight Atmospheric Analysis of Spectral Hypercubes (Flash) is used to perform atmospheric correction on the multispectral data of the GF-2 satellite remote sensing image, and then the rational polynomial coefficients are combined with ground control points to perform geometric correction on the GF-2 satellite remote sensing image to generate the second surface reflectance data.

[0036] In this embodiment, the second surface reflectance data of the GF-2 satellite remote sensing image is registered with the first surface reflectance data of the Sentinel-2B satellite remote sensing image through the Geospatial Data Abstraction Library (GDAL) (error ≤ 1 pixel), and the multispectral band image of the registered GF-2 satellite remote sensing image is resampled to a resolution of 5 m.

[0037] Please refer to Figure 2 , which shows the structural block diagram of the multi-source satellite remote sensing image fusion method based on a multi-level framework; the Sentinel-2B satellite remote sensing image and the GF-2 satellite remote sensing image are fused based on the multi-level framework. When implemented, it is necessary to construct a fusion data set. Based on the Wald's Degradation Protocol (Wald), a training set is generated. A Gaussian blur kernel is applied to the high-resolution image (such as the panchromatic band of the GF-2 satellite remote sensing image) and downsampled to generate a low-resolution - high-resolution image pair. The fusion data set is constructed by combining hierarchical fusion.

[0038] The above Gaussian blur kernel satisfies the following relationship:

[0039]

[0040] Among them, is the standard deviation of the Gaussian kernel, is the super-resolution factor.

[0041] S2. Construct a progressive multi-level fusion strategy for the multi-source satellite remote sensing image data based on a multi-level framework.

[0042] In this embodiment, a progressive multi-level fusion strategy is constructed based on homologous satellite image fusion, heterologous satellite image fusion, and multi-spectral panchromatic image fusion. The progressive multi-level fusion strategy reduces information loss during the generation and fusion of training data by introducing the multi-level features of the remote sensing images themselves.

[0043] In an alternative embodiment, the traditional fusion process satisfies the following relationship:

[0044]

[0045] where, is the output high-resolution multi-spectral fused image, represents the fusion function, is the pre-processed low-resolution multi-spectral image, is the high-resolution auxiliary image.

[0046] In this embodiment, the fusion process of the progressive multi-level fusion strategy satisfies the following relationship:

[0047]

[0048] where, is the high-resolution multi-spectral fused image output after the -level fusion, is the spatial resolution input for the -level fusion, is the -level fusion model, are the pre-processed low-resolution multi-spectral images at each level, are the high-resolution auxiliary images at each level, is the index variable of the fusion model, is the total number of fusions.

[0049] S3. Based on the fused image dataset, use the progressive multi-level fusion strategy to obtain the multi-spectral high spatial resolution data of the multi-source satellite remote sensing image data.

[0050] In this embodiment, based on homologous satellite image fusion, multi-resolution data fusion is performed on the first satellite image data to obtain multi-spectral enhanced data; according to heterologous satellite image fusion, the multi-spectral enhanced data is fused with the second satellite image data to obtain high spatial resolution data; based on multi-spectral panchromatic image fusion, the high spatial resolution data and the panchromatic image of the second satellite image data are fused to obtain multi-spectral high spatial resolution data.

[0051] First, fuse the 20m-resolution image and the 10m-resolution image of the Sentinel-2B satellite remote sensing image to enhance the spatial details of the 20m-resolution band to 10m resolution. Then, fuse the 10m Sentinel-2B satellite remote sensing image with the 5m GF-2 satellite remote sensing image to increase the spatial resolution of the Sentinel-2B satellite remote sensing image to 5m resolution. Finally, fuse the 5m-resolution fused image with the 1m panchromatic image of the GF-2 satellite remote sensing image to obtain 1m-resolution multispectral high-spatial-resolution data.

[0052] Specifically, the progressive multi-level fusion strategy improves the resolution layer by layer through three stages, including: Stage 1, homologous satellite image fusion, 20m→10m: fuse the 20m multispectral data and the 10m multispectral data of the Sentinel-2B satellite remote sensing image to enhance spectral details; Stage 2, heterologous satellite image fusion, 10m→5m: fuse the 10m multispectral image of the Sentinel-2B satellite remote sensing image after the fusion in Stage 1 with the 5m multispectral image of the GF-2 satellite remote sensing image to improve the spatial resolution; Stage 3, 5m→1m: fuse the 5m multispectral image of the GF-2 satellite remote sensing image after the fusion in Stage 2 with the 1m multispectral-panchromatic image of the GF-2 satellite remote sensing image. Finally, the fused multispectral high-spatial-resolution data obtains a 1m-level ground resolution. Each stage uses multi-resolution input collaborative optimization, using the low-level fusion result as the high-level prior knowledge, avoiding information loss caused by directly fusing cross-scale multi-source remote sensing images.

[0053] S4. Establish a multi-fusion method collaborative architecture, and combine the multi-source satellite remote sensing fusion image data according to the multi-fusion method collaborative architecture with the multispectral high-spatial-resolution data to achieve the fusion of multi-source satellite remote sensing images.

[0054] In this embodiment, a multi-fusion method collaborative architecture is constructed by combining the component replacement method, the multi-scale decomposition method, and the deep learning method; the component replacement of the multispectral high-spatial-resolution data is performed based on the component replacement method to obtain the first multi-source satellite remote sensing fusion image data; the multi-scale decomposition analysis of the multispectral high-spatial-resolution data is performed according to the multi-scale decomposition method to obtain the second multi-source satellite remote sensing fusion image data; based on the deep learning method, the third multi-source satellite remote sensing fusion image data is obtained by combining the multispectral high-spatial-resolution data; the multi-source satellite remote sensing fusion image data is obtained from the first multi-source satellite remote sensing fusion image data, the second multi-source satellite remote sensing fusion image data, and the third multi-source satellite remote sensing fusion image data.

[0055] In this embodiment, the specific technical implementation of the multi-fusion method collaborative architecture is as follows:

[0056] (1) Component substitution method: The Brovey Transform (abbreviated as Brovey) linearly weights the normalized multispectral bands and the panchromatic band, and directly replaces the brightness component; the Principal Component Analysis (abbreviated as PCA) method performs principal component analysis on the multispectral image, selects the first principal component to match the histogram of the panchromatic image and then replaces it; the Gram-Schmidt Transform (abbreviated as GS) / Gram-Schmidt Adaptive Transform (abbreviated as GSA) algorithm uses Gram-Schmidt orthogonalization to construct a simulated panchromatic band, and realizes component substitution through least squares regression.

[0057] Specifically, the Brovey transform method, based on color normalization and product operation, fuses each band of the low-resolution multispectral image with the high-resolution panchromatic image in proportion; the principal component analysis method projects the multispectral image into the principal component space, replaces the first principal component with the panchromatic image and then performs inverse transformation to reconstruct; the GS method generates a new component space through orthogonal projection, replaces the first component with the panchromatic image, and inverse projects to generate a fused image. The GSA method introduces an adaptive weight to adjust the spectral injection amount and optimizes the spectral consistency of the GS transform. The above methods are classified as component substitution methods, and the calculation process of the component substitution method satisfies the following relationship:

[0058]

[0059] Among them, is the output high-resolution image, represents the inverse transformation of the color space, is the panchromatic image after histogram matching, represents the remaining components that are not replaced.

[0060] Further, based on the component substitution method, component substitution is performed on the multispectral high-spatial-resolution data to obtain the first multi-source satellite remote sensing fusion image data.

[0061] (2) Multi-scale decomposition method: The Modulation Transfer Function based Generalized Laplacian Pyramid (MTF_GLP) method designs a Gaussian low-pass filter based on the sensor modulation transfer function, performs multi-level pyramid decomposition on the panchromatic image, extracts high-frequency detail components at the N-level decomposition layer, and injects the details into the multi-spectral image transformed by intensity-chrominance-saturation through injecting gain coefficients; the Smoothing Filter-based Intensity Modulation (SFIM) generates a synthetic panchromatic image through local spatial filtering and injects high-frequency information into each band of the multi-spectral image using ratio operation.

[0062] Specifically, SFIM separates the high-frequency details of the panchromatic image through spatial filtering and injects them into the multi-spectral image using the ratio method. Coupled Nonnegative Matrix Factorization (CNMF) jointly optimizes the non-negative matrix factorization of the multi-spectral and panchromatic images and reconstructs the high-resolution image through the endmember-abundance model. The above methods are classified as multi-scale decomposition methods, and the calculation process of the multi-scale decomposition method satisfies the following relationship:

[0063]

[0064] Among them, is the high-resolution multi-spectral image of the th band after fusion, is the low-resolution multi-spectral image of the th band after upsampling, is the gain coefficient, is the high-frequency component of the panchromatic image, is the original panchromatic image, represents low-pass filtering.

[0065] Further, the multi-scale decomposition analysis of the multi-spectral high spatial resolution data is performed according to the multi-scale decomposition method to obtain the second multi-source satellite remote sensing fusion image data.

[0066] (3) Deep learning: The Multi-Scale Deep Convolutional Neural Network (MSDCNN) adopts a dual-branch encoding and decoding architecture. The upper branch extracts the spatial texture features of the panchromatic image through 5 layers of convolution. The lower branch uses dilated convolution to expand the receptive field of the multi-spectral image. Finally, feature fusion and upsampling are achieved through a sub-pixel convolutional layer. The Pan-Spectral Reconstruction Transformer (PSRT) method constructs a cascaded Transformer module. First, the long-range dependence relationship between the multi-spectral and panchromatic images is established through the multi-head self-attention mechanism for cross-modal interaction in the feature space. Finally, a deformable convolutional network is used to complete the detail reconstruction.

[0067] Specifically, MSDCNN uses cascaded convolutional layers to extract multi-scale features and fuses spatial-spectral information through residual learning. PSRT is based on the cross-modal attention mechanism of Transformer to model the long-range dependence relationship between panchromatic and multi-spectral images. The Multi-task Spectral-Spatial Transformer (MSST) jointly optimizes the spectral fidelity and spatial detail tasks and extracts features through the spectral branch and the spatial branch in parallel. The above methods are classified as deep learning-based methods. The calculation process of the deep learning method includes:

[0068] First, calculate the fused image through the network, and the calculation process satisfies the following relationship:

[0069]

[0070] Among them, is the output high-resolution image, is the parameterized neural network, represents the model parameters, is the original panchromatic image, is the upsampled low-resolution multi-spectral image.

[0071] Second, optimize the neural network parameters by calculating the mean square error loss, which satisfies the following relationship:

[0072]

[0073] Among them, is the mean square error loss value, is the number of batch training samples, is the number of multi-spectral bands, is the sample index variable, is the band index variable, For the predicted high-resolution image of the th sample in the band, For the true high-resolution image of the th sample in the band, represents the square of the two-norm.

[0074] Furthermore, according to the deep learning method, the third multi-source satellite remote sensing fusion image data is obtained by combining multi-spectral high-spatial resolution data.

[0075] In this embodiment, the multi-source satellite remote sensing fusion image data includes the first multi-source satellite remote sensing fusion image data, the second multi-source satellite remote sensing fusion image data, and the third multi-source satellite remote sensing fusion image data.

[0076] S5. Construct a multi-dimensional quantitative evaluation system, and perform quality evaluation on the multi-source satellite remote sensing fusion image data based on the multi-dimensional quantitative evaluation system to obtain a fusion evaluation result.

[0077] In this embodiment, six evaluation indicators are used to evaluate the fusion result: (1) Compare the difference between the fused image and the reference image pixel by pixel, and calculate the Peak Signal-to-Noise Ratio (PSNR for short); (2) Calculate the local brightness, contrast, and structural similarity through a sliding window (such as 11×11), take the global mean, and calculate the Structural Similarity Index (SSIM for short); (3) Calculate the spectral vector angle between pixels, and statistically calculate the global average value (in degrees) to calculate the Spectral Angle Mapper (SAM for short); (4) Calculate the normalized root mean square error for each band, scale it after weighted averaging, and calculate the Relative Global Dimensionless Synthesis Error (ERGAS for short); (5) Directly calculate the mean square of the pixel-level differences between the fused image and the reference image to obtain the Mean Squared Error (MSE for short); (6) Calculate the ratio of the covariance to the standard deviation between the fused image and the reference image, in the range [-1, 1], to obtain the Correlation Coefficient (CC for short).

[0078] The peak signal-to-noise ratio (unit: dB) is used to quantify the pixel-level reconstruction accuracy. PSNR evaluates the pixel-level reconstruction quality by comparing the maximum possible signal intensity and the error intensity of the image. The larger the value, the smaller the noise interference and the higher the accuracy of the fused image. The calculation steps satisfy the following relationship:

[0079]

[0080] Among them, is the peak signal-to-noise ratio, is the theoretical maximum value of the pixel value, is the mean square error.

[0081] The structural similarity index (SSIM) measures texture and structural fidelity. SSIM measures the similarity between two images by comprehensively considering luminance, contrast, and structural information. The closer the value is to 1, the better the structural fidelity; the calculation steps satisfy the following relationship:

[0082]

[0083] Among them, is the structural similarity index, and are the means of the two images, is the covariance, and are the variances, and are constants used to stabilize the calculation.

[0084] Spectral fidelity assessment is based on the spectral angle (SAM) to calculate the deviation of the pixel spectral vector. SAM evaluates the spectral distortion degree by calculating the radian value of the angle between spectral vectors. The smaller the value (close to 0), the higher the spectral fidelity. The calculation steps satisfy the following relationship:

[0085]

[0086] Among them, is the spectral angle, is the inverse cosine function, is the total number of bands of the image, is the band index variable, is the spectral vector value of the reference image in the th band, is the spectral vector value of the fused image in the th band.

[0087] Calculate the global relative error (ERGAS) to evaluate the multi-band joint spectral distortion degree. ERGAS comprehensively considers the multi-band error and resolution change. The smaller the value, the lower the spectral distortion. The calculation steps satisfy the following relationship:

[0088]

[0089] Among them, is the global relative error, is the spatial resolution ratio of the fused image to the original image, is the total number of bands of the image, is the band index variable, for the root mean square error of the band, is the pixel mean of the band of the reference image.

[0090] It should be noted that the spatial resolution ratio of the fused image to the original image: when the resolution is increased from 10m to 5m .

[0091] Calculate the mean square error (MSE) to characterize the overall error level. MSE quantifies the overall error level of the image by calculating the mean of the squared differences pixel by pixel. The smaller the value, the higher the fusion accuracy. The calculation steps satisfy the following relationship:

[0092]

[0093] where, is the mean square error, is the total number of pixels in a single band, is the total number of bands of the image, is the band index variable, is the pixel index variable, is the value of the fused image at the band and the th pixel, is the value of the reference image at the band and the th pixel.

[0094] Spatial quality assessment calculates the covariance and variance analysis of the spatial correlation between the fused image and the reference data through the correlation coefficient (CC). CC quantifies its spatial correlation by calculating the ratio of the covariance and standard deviation of the two images. The closer the value is to 1, the higher the spatial consistency. The calculation steps satisfy the following relationship:

[0095]

[0096] where, is the correlation coefficient, is the total number of pixels in a single band, is the pixel index variable, is the gray value of the th pixel in the fused image, >is the pixel mean of the fused image, is the gray value of the th pixel in the reference image, is the pixel mean of the reference image.

[0097] Please refer toFigure 3 , in an optional embodiment, the present invention provides a multi-source satellite remote sensing image fusion system based on a multi-level framework. The system includes an input device, an output device, a processor, and a memory, which are interconnected with each other. Among them, the memory is used to store a computer program, and the computer program includes program instructions. The processor is configured to call the program instructions to execute the specific steps of the related embodiments of the multi-source satellite remote sensing image fusion method provided by the present invention. The multi-source satellite remote sensing image fusion system provided by the present invention has a complete structure, is objective and stable, and improves the overall applicability and practical application ability of the present invention.

[0098] Please refer to Figure 4 , which is a schematic diagram of the execution process of the multi-source satellite remote sensing image fusion system based on a multi-level framework; in an optional embodiment, the multi-source satellite remote sensing image fusion system based on a multi-level framework includes a data preprocessing module, a hierarchical fusion module, a multi-level output module, and a fusion result evaluation module.

[0099] The data preprocessing module is used to preprocess the multi-source satellite remote sensing image data to obtain a fused image dataset.

[0100] The hierarchical fusion module is used to construct the progressive multi-level fusion strategy and the collaborative architecture of the multi-fusion method, and fuse the multi-source satellite remote sensing image data to obtain multi-source satellite remote sensing fused image data.

[0101] The multi-level output module is used to output intermediate and final resolution results (10m, 5m, 1m) after each level of fusion. High-level fusion reuses low-level features to avoid error accumulation caused by direct fusion of large-span resolutions. Users can select the output level as needed to balance computational efficiency and accuracy requirements.

[0102] The fusion result evaluation module is used to construct a multi-dimensional quantitative evaluation system to evaluate the quality of the multi-source satellite remote sensing fused image data to obtain a fusion evaluation result.

[0103] In an optional embodiment, Sentinel-2B Level-1C data is obtained from the European Space Agency platform, including 10m (blue, green, red, near-infrared) and 20m (red edge, narrow near-infrared) resolution bands; GF-2 Level-1A data is obtained from the China National Space Administration Center, including 1m panchromatic and 4m multispectral images. The research area covered by the satellite remote sensing images is the Hushan Reservoir and its surrounding areas in Wenchang City, Hainan Province.

[0104] The land cover types around the reservoir are rich and diverse, including eutrophic water bodies in the reservoir area, small lakes and reservoirs in the surrounding areas, rivers, main dams, auxiliary dams, winding roads (including solidified artificial roads and natural trails), drawdown zones, grasslands, aquatic plants (such as water hyacinths, reeds, etc.), cash crops (such as pineapples), and other natural vegetation. There are also artificial facilities such as residential buildings, industrial buildings, fish ponds, and turtle ponds in the area. To effectively manage the ecosystem and water resources in this area, especially to deal with complex land cover situations and conduct water environment quality monitoring, high-resolution multispectral / hyperspectral satellite remote sensing images are urgently needed.

[0105] A total of six scenes of GF-2 panchromatic and multispectral high-spatial-resolution data and two scenes of Sentinel-2B multispectral high-spatial-resolution data were used. First, a fusion system for GF-2 images and Sentinel-2B images of the Lake Mountain Reservoir and its surrounding areas was constructed, including a data preprocessing module, a hierarchical fusion module, and a fusion quality evaluation module. By inputting GF-2 high-resolution images and Sentinel-2B multispectral images, high-quality gradually fused images were obtained, laying a data foundation for remote sensing monitoring of the Lake Mountain Reservoir and its surrounding areas.

[0106] The remotely sensed images generated by gradual fusion were quantitatively evaluated. The evaluation parameters included correlation coefficient (CC), structural similarity index (SSIM), spectral angle (SAM), global relative error (ERGAS), mean square error (MSE), and peak signal-to-noise ratio (PSNR). The evaluation results are shown in Table 1:

[0107] Table 1

[0108]

[0109] Among them, bicubic interpolation (abbreviated as Bicubic), which calculates new pixel values based on surrounding pixels, is used for image scaling, rotation, and balancing smoothness and detail retention.

[0110] In the Sentinel-2B data fusion results with 20m and 10m resolution bands as references, it was found that in terms of the PSNR metric, deep learning methods (MSST, MSDCNN, PSRT) outperformed traditional methods, with a marginal value of at least 13. The highest PSNR of MSST was 48.05, showing strong noise suppression ability. For SAM, MSDCNN reached the lowest value of 1.223°, 14% higher than PSRT, while the result of Brovey was 2.005°, highlighting the effectiveness of the component replacement method in retaining spectral features. In terms of the ERGAS metric, PSRT reached the highest score of 0.3282, a 62% improvement over GS, while SFIM (1.4283) and coupled non-negative matrix factorization (CNMF) (1.343) showed the limitations of the multi-resolution decomposition method. SSIM indicated that MSDCNN reached 0.991, showing a high consistency with human visual perception, while the similarity of Brovey and GS was 0.8984, still facing the challenge of local texture degradation. The research systematically evaluated the fusion performance of Sentinel-2B (multispectral bands with 10m resolution) and GF-2 multispectral-panchromatic images downscaled to 5m spatial resolution using different fusion methods. The results showed that different fusion methods presented basically consistent performance difference characteristics on the datasets with resolution differences (20m level → 10m level) and (10m level → 5m level). In the homologous satellite remote sensing image fusion experiment, the MSDCNN method demonstrated the optimal comprehensive performance (PSNR = 47.68dB, ERGAS = 0.3413), especially outstanding in the spectral angle mapping (SAM = 1.223) and structural similarity (SSIM = 0.991) metrics. For the heterologous sensor data fusion scenario, MSDCNN continued to maintain the leading advantage (PSNR = 46.96dB, CC = 0.9983), while the constraint parameter (CC value) of the PSRT algorithm in both types of fusion tasks remained above 0.98, highlighting the advantages of the Transformer architecture in the cross-modal attention mechanism. It is worth noting that the average ERGAS value of homologous fusion (0.815 ± 0.371) was significantly lower than that of heterologous fusion (1.193 ± 0.893), indicating that the fusion effect of homologous images was better than that of heterologous images. Additionally, in the performance on real data, the performance of deep learning methods was significantly better than that of traditional methods, especially in the case of non-overlapping spectra, where deep learning methods better restored image details and maintained the spectral characteristics of the original data.

[0111] Please refer to Figure 5, which is a schematic diagram of the fused image output by the multi-source satellite remote sensing image fusion method based on a multi-level framework; in terms of image restoration, the sharpening effect at 5 times the resolution is more obvious than the melting effect at 2 times, and small ground objects, roads, and scattered plots show a clearer effect.

[0112] The hierarchical fusion framework effectively overcomes the common information loss problem in traditional image fusion methods by gradually fusing images with different spatial resolutions (from the fusion of 20m and 10m resolutions of homologous images, the fusion of 10m resolution and 5m resolution of heterogeneous images, and the fusion of multi-source 5m resolution images to 1m resolution images). At each fusion stage, the framework ensures the gradual enhancement of spatial details and outputs the fusion result through multi-level resolutions, thus improving the overall image quality. This progressive fusion strategy is particularly suitable for remote sensing data with multiple spectral bands, effectively balancing the preservation of spectral and spatial information and avoiding the loss of details that occurs in the traditional method during image sharpening. In addition, the hierarchical fusion framework exhibits good scalability and generalization ability, and can maintain high accuracy and stability when facing different sensor data or different resolution remote sensing data fusion tasks.

[0113] The size of the spatial resolution has an important impact on the fusion effect. The fusion task with a large spatial resolution difference is more difficult to achieve than the fusion task with a small spatial resolution difference. Analyze by comparing the index data under 5 times and 2 times the resolution difference. A smaller resolution difference (such as 2 times) will significantly improve the fusion quality, including a significant increase in PSNR, SSIM, and CC, and a significant decrease in MSE, indicating a better fusion effect. In terms of spectral fidelity, the spectral error generally increases when the resolution difference increases. And deep learning performs better in the fusion of images with small resolution differences. A large resolution difference will lead to difficulties in restoring spatial details, a decrease in spectral fidelity, and a weakening of the correlation with the reference image. When the resolution difference decreases, the fusion effect of the deep learning method improves significantly, and the performance of the traditional method improves but is limited.

[0114] The present invention proposes a progressive fusion architecture for multi-source satellites, which solves the information loss problem in the fusion of large resolution differences by traditional methods and realizes the high-quality fusion of multi-source satellite images. And it verifies the advantages of deep learning in remote sensing image fusion. The PSNR of MSDCNN and PSRT is significantly improved compared with traditional methods (such as GS), providing a better data basis for high-resolution remote sensing tasks and supporting the improvement of the accuracy of downstream applications such as water body extraction and ground object classification.

[0115] In summary, the multi-source satellite remote sensing image fusion method and system provided by the method of the present invention significantly improve the accuracy of multi-source satellite remote sensing image fusion through a closed-loop of data preprocessing - hierarchical fusion - multi-method collaboration - fusion result evaluation, provide multi-spectral images with high-resolution characteristics, and provide important technical support for the application of high-resolution remote sensing images. The method of the present invention is easy to understand, simple to calculate, has a small workload, is convenient for engineering applications, and provides a theoretical basis and technical support for the further development of the aerospace information field.

[0116] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered by the scope of the claims and the specification of the present invention.

Claims

1. A multi-source satellite remote sensing image fusion method based on a multi-level framework, characterized in that It includes the following steps: Obtain multi-source satellite remote sensing image data, and preprocess the multi-source satellite remote sensing image data to obtain a fused image dataset; Construct a progressive multi-level fusion strategy for the multi-source satellite remote sensing image data based on a multi-level framework; According to the fused image dataset, use the progressive multi-level fusion strategy to obtain the multi-spectral high-spatial-resolution data of the multi-source satellite remote sensing image data; Establish a collaborative architecture of multiple fusion methods, and combine the multi-spectral high-spatial-resolution data according to the collaborative architecture of multiple fusion methods to obtain multi-source satellite remote sensing fused image data, realizing the fusion of multi-source satellite remote sensing images; The fusion process of the progressive multi-level fusion strategy satisfies the following relationship: ； Among them, is the high-resolution multispectral fusion image output after the -level fusion, is the spatial resolution input for the -level fusion, is the -level fusion model, are the preprocessed low-resolution multispectral images at each level, are the high-resolution auxiliary images at each level, is the index variable of the fusion model, is the total number of fusion times.

2. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 1, wherein, The obtaining of the multi-source satellite remote sensing image data and the preprocessing of the multi-source satellite remote sensing image data to obtain a fused image dataset include: Obtain first satellite image data and second satellite image data, and use the first satellite image data and the second satellite image data as the multi-source satellite remote sensing image data; Perform radiometric correction and atmospheric correction on the first satellite image data to obtain first surface reflectance data; Perform radiometric correction, atmospheric correction, and geometric correction on the second satellite image data to obtain second surface reflectance data; Construct the fused image dataset by combining the first surface reflectance data and the second surface reflectance data.

3. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 2, wherein The constructing of the fused image dataset by combining the first surface reflectance data and the second surface reflectance data includes: Register the first surface reflectance data and the second surface reflectance data; Resample the registered second surface reflectance data to obtain a low-resolution - high-resolution image pair; Construct the fused image dataset based on the low-resolution - high-resolution image pair.

4. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 3, characterized in that, The constructing of the progressive multi-level fusion strategy for the multi-source satellite remote sensing image data based on a multi-level framework includes: Construct the progressive multi-level fusion strategy based on homologous satellite image fusion, heterologous satellite image fusion, and multi-spectral - panchromatic image fusion.

5. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 4, characterized in that, The obtaining of the multi-spectral high-spatial-resolution data of the multi-source satellite remote sensing image data by using the progressive multi-level fusion strategy according to the fused image dataset includes: Based on the homologous satellite image fusion, perform multi-resolution data fusion on the first satellite image data to obtain multi-spectral enhanced data; According to the heterologous satellite image fusion, fuse the multi-spectral enhanced data with the second satellite image data to obtain high-spatial-resolution data; According to the multi-spectral - panchromatic image fusion, fuse the high-spatial-resolution data and the panchromatic image of the second satellite image data to obtain the multi-spectral high-spatial-resolution data.

6. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 1, characterized in that The establishing of the collaborative architecture of multiple fusion methods includes: Construct the collaborative architecture of multiple fusion methods by combining the component substitution method, the multi-scale decomposition method, and the deep learning method.

7. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 6, wherein The obtaining of the multi-source satellite remote sensing fused image data by combining the multi-spectral high-spatial-resolution data according to the collaborative architecture of multiple fusion methods to realize the fusion of multi-source satellite remote sensing images includes: Perform component substitution on the multi-spectral high spatial resolution data based on the component substitution method to obtain the first multi-source satellite remote sensing fusion image data; Perform multi-scale decomposition analysis on the multi-spectral high spatial resolution data according to the multi-scale decomposition method to obtain the second multi-source satellite remote sensing fusion image data; Based on the deep learning method, combine the multi-spectral high spatial resolution data to obtain the third multi-source satellite remote sensing fusion image data; Use the first multi-source satellite remote sensing fusion image data, the second multi-source satellite remote sensing fusion image data, and the third multi-source satellite remote sensing fusion image data as the multi-source satellite remote sensing fusion image data.

8. The multi-source satellite remote sensing image fusion method based on a multi-level framework according to claim 6, characterized in that, The multi-fusion method collaborative architecture includes: The component substitution method satisfies the following relationship: ； Among them, is the output high-resolution image, represents the inverse transformation of the color space, is the panchromatic image after histogram matching, represents the retained components that have not been replaced; The multi-scale decomposition method satisfies the following relationship: ； Among them, is the high-resolution multispectral image of the band after fusion, is the low-resolution multispectral image after upsampling of the band, is the gain coefficient, is the high-frequency component of the panchromatic image, is the original panchromatic image, represents low-pass filtering; The deep learning method satisfies the following relationship: ； Among them, is the output high-resolution image, is the parameterized neural network, represents the model parameters, is the original panchromatic image, is the upsampled low-resolution multispectral image.

9. A multi-source satellite remote sensing image fusion system based on a multi-level framework, characterized in that, The system includes an input device, an output device, a processor, and a memory. The input device, output device, processor, and memory are interconnected. Among them, the memory is used to store computer programs. The computer programs include program instructions. The processor is configured to call the program instructions to execute the multi-source satellite remote sensing image fusion method based on a multi-level framework according to any one of claims 1-8.

Images