Remote sensing image space-time fusion method based on multispectrum

Abstract

The invention discloses a multispectral-based remote sensing image space-time fusion method. The method comprises the following steps: (1) acquiring a Landsat image and an MODIS image by using an open source website, and preprocessing the Landsat image and the MODIS image; (2) constructing a PLSR regression relation in space-time fusion; (3) selecting an optimal PLS component number OPC; (4) self-adaptive residual compensation (ARC); and (5) obtaining a final prediction result. The accuracy of the prediction result is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multispectral-based spatiotemporal fusion method for remote sensing images. Background Art

[0002] With the launch of numerous satellites over the past few decades, the accumulation of vast amounts of data has provided opportunities to integrate data from diverse sensors to observe the Earth's surface at high spatial and temporal resolution. This is particularly challenging for single satellite sensors due to the inherent trade-off between the two resolutions, as well as technical and budgetary constraints. For example, the Landsat series and Sentinel-2 are capable of acquiring images with fine spatial detail at fine spatial resolution (10–30 meters, hereafter referred to as fine imagery). However, their long revisit periods (5–16 days) and cloud contamination severely reduce their observation frequency and data quality. On the other hand, the Advanced Very High Resolution Radiometer (AVHRR), the Moderate Resolution Imaging Spectroradiometer (MODIS), and the Visible Infrared Imaging Radiometer Suite (VIIRS) provide daily observations. However, their coarse spatial resolution (250–1000 meters, hereafter referred to as coarse imagery) hinders their application in heterogeneous regions. This imperfection has spurred the development of numerous spatiotemporal fusion methods in recent years to meet the demand for high spatial and temporal resolution data in various applications.

[0003] Spatiotemporal fusion methods aim to generate composite images with high spatial and temporal resolution by combining the strengths of data from two or more sensors. These data can be high spatial resolution but low temporal resolution (such as Landsat images) or high temporal resolution but low spatial resolution (such as MODIS images). However, existing fusion methods utilize low-resolution multispectral band images to reconstruct missing high-resolution multispectral band images. The spectra of the two bands correspond one-to-one, requiring separate fusion calculations for each spectral band. This is computationally intensive and inefficient. Existing spatiotemporal fusion methods can generally be divided into two categories. The first category is incremental estimation methods. However, these methods suffer from several key issues, including performance degradation when the temporal variation between the forecast date and the base date is large or spatially heterogeneous, weak resistance to geometric misregistration, and low computational efficiency. The second category is regression-based methods. Both of these methods predict a single band individually, generating a single-band prediction by inputting a spectrally similar band. This means that only the limited information of the single similar band is utilized, while spectral autocorrelations between different bands are ignored. Generally speaking, using more spectral bands from remote sensing imagery can improve regression model performance by incorporating valuable information from band autocorrelations. Evidence from studies on spatial-spectral fusion supports this view. For example, multiple linear regression (MLR), such as Area-to-Point Regression Kriging (ATPRK), uses multiband imagery as input to produce predictions for a single band, effectively improving the quality of spatial-spectral fusion results. Furthermore, spectral bands often have varying levels of data quality. For example, the shortwave infrared (SWIR) band is less adversely affected by aerosols than the visible band. Therefore, using spectral bands with better data quality in the fusion process can also improve the quality of fused images from other bands with poorer data quality, particularly for haze and cloudy areas. Furthermore, using all available spectral bands as input in MLR does not guarantee improved performance of spatiotemporal fusion methods. Despite its superior modeling capabilities, crosstalk between multiple bands can lead to Gaussian noise inflation when Gaussian noise is high, resulting in unstable results. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a spatiotemporal fusion method for remote sensing images based on multispectral technology to solve the problem that when the Gaussian noise is high, the cross-correlation between multiple bands will lead to the expansion of Gaussian noise, thereby causing unstable results.

[0005] Technical solution: The present invention provides a multispectral remote sensing image spatiotemporal fusion method, comprising the following steps: (1) Obtain Landsat and MODIS images using open source websites and perform preprocessing; (2) Constructing PLSR regression relationship in spatiotemporal fusion; (3) Select the optimal number of PLS ​​components OPC; (4) Adaptive residual compensation (ARC); (5) Obtain the final prediction results.

[0006] Furthermore, in step (1), the Landsat image and the MODIS image are resized, and the MODIS image is downscaled to unify the resolution with the Landsat image.

[0007] Furthermore, step (2) is specifically as follows: first, a partial least squares regression model is used in all spectra of all pixels in the moving window M, i.e., the 6-band MODIS images, to establish a regression relationship on the MODIS images at time t1 and t2, and obtain a regression model with n principal components; second, the regression model with n principal components is transferred to the Landsat data at time t1, and the predicted Landsat data image at time t2 is obtained.

[0008] Furthermore, in step (3), a cross-validation CV strategy is used to determine the number of selected components, including the following steps: (31) Divide the rough image pair, i.e., the MODIS images at time t1 and t2 of the corresponding position, into k subsets by equidistant segmentation; (32) Establish PLSR on k-1 subsets using the same number of components; (33) The predicted residual sum of squares (PRESS) is calculated for the remaining subset to quantify the error of the established model. The formula is as follows: ; Among them, C1(M) represents the MODIS image at time t2; C2(M) represents the MODIS image at time t2; (34) Steps (31)-(33) are repeated k times until all subsets are calculated once, and the average PRESS value of the model at each number of components is calculated; (35) Steps (31)-(34) are repeated n times to cover all component numbers; when the regression model reaches the minimum average PRESS value, its corresponding component number n is selected as the optimal component number F PLSR ; Then, the optimal n is used for the PLSR models of all local moving windows.

[0009] Furthermore, step (4) is as follows: Let the residual R(x, y, b) on MODIS be defined as the difference between the reflectance value of the MODIS pixel on the prediction date and the average of the predicted reflectance values ​​of all Landsat pixels corresponding to the pixel, and the formula is as follows: ; Among them, C(x, y, b) and F p (xi, yi, b) are the reflectance values ​​of the i-th Landsat pixel in band b at position (x, y) in MODIS and C(x, y), respectively. m is the number of Landsat pixels within a MODIS pixel; the residual values ​​R(x, y, b) of all coarse pixels are then spatially interpolated to the fine resolution to obtain the fine residual r, which is used to compensate the residual.

[0010] Furthermore, step (5) is as follows: similar pixels in the neighborhood are introduced to distribute the fine residual r, and the residual-compensated Landsat prediction result F is obtained. RC ; Then compare the PLSR regression results F PLSR , the result of residual compensation F RC And the image F obtained by downscaling the MODIS image C The size of the change between the two determines which prediction result is used as the final prediction result F PLSR-STF Among them, F RC The formula is as follows: ; Among them, F RC represents the prediction result; r represents the fine residual; F PLSR Represents the PLSR regression results.

[0011] The multispectral remote sensing image spatiotemporal fusion system of the present invention comprises: Acquisition module: used to obtain Landsat images and MODIS images using open source websites and perform preprocessing; PLSR module: used to construct PLSR regression relationship in spatiotemporal fusion; OPC module: used to select the optimal number of PLS ​​components OPC; Residual compensation module: used for adaptive residual compensation ARC; Prediction module: used to obtain the final prediction results.

[0012] An electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, it implements any one of the multispectral-based spatiotemporal fusion methods for remote sensing images.

[0013] The storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, it implements any one of the multispectral-based spatiotemporal fusion methods for remote sensing images.

[0014] Beneficial effects: Compared with the existing technology, the present invention has the following significant advantages: it breaks the limitation of the traditional spatiotemporal fusion method based on a one-to-one framework and adopts a more comprehensive many-to-many framework. This innovation enables the method to maximize the use of information from all available spectral bands and effectively reduce the impact of collinearity, thereby generating more accurate and information-rich multi-band fusion images. Compared with other typical spatiotemporal fusion methods, this method shows obvious advantages. In particular, when dealing with complex temporal variation patterns, it can continuously generate high-quality prediction results with significant spatial details. In the case of multiple pairs of input data, it also shows stable high fusion accuracy. It is worth noting that it also shows strong resistance to environmental noise, such as haze and thin cloud layers, and can maintain good performance even under these difficult conditions. In addition, since it can process multiple bands simultaneously, it is also acceptable in terms of computational efficiency, which is particularly important for processing large amounts of satellite data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the present invention; Figure 2 Schematic diagram of selecting the optimal number of components for PLS regression of the present invention; Figure 3 is a schematic diagram of adaptive residual compensation of the present invention; Figure 4 The accuracy of various methods in the CIA area at different time intervals is compared visually based on color images of the present invention; Figure 5 The accuracy of various methods at different time intervals in the LGC region of the present invention is compared visually based on color images; Figure 6 The accuracy of various methods at different time intervals in the southern Daxing area of ​​Beijing is compared visually based on color images. DETAILED DESCRIPTION

[0016] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0017] like Figure 1As shown, an embodiment of the present invention provides a multispectral remote sensing image spatiotemporal fusion method, comprising the following steps: (1) Use open source websites to obtain Landsat and MODIS images and perform preprocessing. Unify the size of Landsat and MODIS images and downscale the MODIS images to unify their resolution with that of Landsat images. The spatial resolution of Landsat images is 25m, and the spatial resolution of MODIS images is 500m.

[0018] (2) Constructing the PLSR regression relationship in spatiotemporal fusion; First, a partial least squares regression model is used in all spectra of all pixels in the moving window M, i.e., the 6-band MODIS image, to establish a regression relationship on the MODIS images at time t1 and t2, and obtain a regression model with n principal components; secondly, the regression model with n principal components is transferred to the Landsat data at time t1, and the predicted Landsat data image at time t2 is obtained. The formula is as follows: ; Among them, C1 and C2 are coarse resolution images at time t1 and t2, that is, MODIS images, R n is the residual after regression. By the fundamental assumption of spatiotemporal fusion, the regression model is locally scale-invariant, so the relationships established at the MODIS spatial resolution can be applied to fine spatial resolutions. Therefore, these regression coefficients can be transferred to the Landsat data at time T1 to produce the predicted Landsat image at time T2. The formula is as follows: ; in, and are the Landsat image at time t1 and the predicted Landsat image at time t2, and n is the number of components in the partial least squares regression (PLSR).

[0019] (3) Select the optimal number of PLS ​​components OPC; Figure 2 As shown, a cross-validation CV strategy is used to determine the size of the number of selected components, including the following steps: (31) Divide the rough image pair, i.e., the MODIS images at time t1 and t2 of the corresponding position, into k subsets by equidistant segmentation; (32) Establish PLSR on k-1 subsets using the same number of components; (33) The predicted residual sum of squares (PRESS) is calculated for the remaining subset to quantify the error of the established model. The formula is as follows: ; Among them, C1(M) represents the MODIS image at time t2; C2(M) represents the MODIS image at time t2; (34) Steps (31)-(33) are repeated k times until all subsets are calculated once, and the average PRESS value of the model at each number of components is calculated; (35) Steps (31)-(34) are repeated n times to cover all component numbers; when the regression model reaches the minimum average PRESS value, its corresponding component number n is selected as the optimal component number F PLSR ; Then, the optimal n is used for the PLSR models of all local moving windows.

[0020] (4) Adaptive residual compensation ARC; such as Figure 3 As shown, specifically as follows: Let the residual R(x, y, b) on MODIS be defined as the difference between the reflectance value of the MODIS pixel on the prediction date and the average of the predicted reflectance values ​​of all Landsat pixels corresponding to the pixel, and the formula is as follows: ; Among them, C(x, y, b) and F p (xi, yi, b) are the reflectance values ​​of the i-th Landsat pixel in band b at position (x, y) in MODIS and C(x, y), respectively. m is the number of Landsat pixels within a MODIS pixel; the residual values ​​R(x, y, b) of all coarse pixels are then spatially interpolated to the fine resolution to obtain the fine residual r, which is used to compensate the residual.

[0021] (5) Obtain the final prediction result. Specifically: Introduce similar pixels in the neighborhood to distribute the fine residual r, and obtain the residual-compensated Landsat prediction result F RC ; Then compare the PLSR regression results F PLSR , the result of residual compensation F RC and the image F obtained by downscaling the MODIS image C The size of the change between the two determines which prediction result is used as the final prediction result F PLSR-STF Among them, F RC The formula is as follows: ; Among them, F RC represents the prediction result; r represents the fine residual; F PLSR Represents the PLSR regression results.

[0022] The PLAR-STF method was tested in three regions: the Coleambally Irrigation in Australia, the LGC region in northern New South Wales, and the Daxing area of ​​Beijing. These three test regions were chosen because they present challenging spatiotemporal fusion problems. The cropland areas in the Coleambally Irrigation and the LGC region in northern New South Wales vary significantly, exhibiting strong heterogeneity, rapid changes in surface vegetation phenology, and severe abrupt surface changes in some areas. Over the past 20 years, Beijing has experienced significant land cover changes, such as from vegetation to buildings. Because PLSR-STF uses Landsat imagery from 2017, its application to Beijing is challenging. To validate the new method, quantitative evaluation was performed using clear satellite imagery available at that time. For validation, a pair of clear MODIS and Landsat images was selected for each experimental region. Then, at a different time when only clear MODIS images were available, predictions were made using Landsat remote sensing images, where none existed. To investigate the potential impact of cloud shadows, we selected remote sensing images with partial cloud contamination at time t1 for verification. For the CIA region, the baseline image and reference image were Landsat 5 images acquired on July 26 and June 8, 2001, respectively. For the LGC region, the baseline image and reference image were Landsat 8 images acquired on December 30 and October 11, 2014, respectively. For the Daxing region in Beijing, the baseline image and reference image were Landsat 8 images acquired on August 8 and October 11, 2010, respectively. After determining the experimental areas and preparing the necessary experimental data, we simultaneously fused and reconstructed multiple bands of the cloud-simulated images for each experimental area using the PLSR-STF method and the STARFM, Fit-FC, and FSDAF methods. The prediction results from the different methods were visually compared and quantitatively evaluated. The root mean square error (RMSE), correlation coefficient (CC), and structural similarity (SSIM) were calculated and compared with each other.

[0023] Figure 4-Figure 6The prediction performance of PLSR-STF, STARFM, Fit-FC, and FSDAF methods is compared for the CIA, LGC, and Daxing District of Beijing. For the CIA region, four fusion results and zoomed-in images of subregions are shown for increasingly longer time intervals between prediction dates (November 2, 2002, January 5, 2002, and May 4, 2002) compared to the MNSPI imagery. The same pair of Landsat and MODIS images from October 8, 2001, serves as the reference image. When the prediction date is close to the reference date, all fusion results achieve satisfactory results, showing good agreement with the reference imagery. However, when the time difference is large, STARFM, FSDAF, and Fit-FC produce blurry results, which can be more clearly observed in the fused subregion images, where patch boundaries are oversmoothed and color distortion is noticeable. In contrast, PLSR-STF maintains spatial detail and color fidelity across different prediction dates generated by PLS regression between different components. The images predicted from PLSR-STF are visually the most accurate. The quantitative evaluation results confirm the visual results and show that PLSR-STF has lower RMSE and higher CC and SSIM in all bands (see Appendix Figure 4 For the LGC farmland area, it can be seen from the visual results that STARFM, Fit-FC and FSDAF methods cannot reconstruct some linear features well, such as ridge boundaries and slender objects (see Appendix Figure 5 ). In terms of accuracy in all bands, PLS-STF is generally better than the other three methods. For the Beijing urban area, as shown in the attached Figure 6 As shown in Figure 3, PLSR-STF also has better results than FSDAF. In terms of accuracy in all bands, PLSR-STF has lower RMSE values.

[0024] Because the PLSR calculation process incorporates the autocorrelations between different bands into the regression model and performs a weighted combination to produce the final estimate, the three prediction results from the three experimental regions better illustrate the effectiveness of PLSR-STF. The figure shows that across all experimental regions and all bands, the RMSE of the estimates based on all bands is smaller than that of the single-band estimates. Furthermore, by combining estimates from multiple time points, the RMSE of PLSR-STF can be further reduced in some cases.

[0025] The embodiment of the present invention further provides a multispectral remote sensing image spatiotemporal fusion system, comprising: Acquisition module: used to obtain Landsat images and MODIS images using open source websites and perform preprocessing; PLSR module: used to construct PLSR regression relationship in spatiotemporal fusion; OPC module: used to select the optimal number of PLS ​​components OPC; Residual compensation module: used for adaptive residual compensation ARC; Prediction module: used to obtain the final prediction results.

[0026] An embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the method for spatiotemporal fusion of remote sensing images based on multispectral is implemented.

[0027] An embodiment of the present invention further provides a storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements any one of the multispectral-based spatiotemporal fusion methods for remote sensing images.

Claims

1. A method for spatiotemporal fusion of remote sensing images based on multispectral, characterized in that: The following steps are involved: (1) Use open source websites to obtain Landsat images and MODIS images and perform preprocessing; (2) Constructing PLSR regression relationship in spatiotemporal fusion; (3) Select the optimal number of PLS ​​components OPC; (4) Adaptive residual compensation ARC; (5) Obtain the final prediction results.

2. The method for spatiotemporal fusion of remote sensing images based on multispectral according to claim 1, characterized in that: In step (1), the Landsat image and the MODIS image are resized to the same size, and the MODIS image is downscaled to unify the resolution with the Landsat image.

3. The method for spatiotemporal fusion of remote sensing images based on multispectral according to claim 1, characterized in that: Step (2) is as follows: First, a partial least squares regression model is used in all spectra of all pixels in the moving window M, i.e., the MODIS images of 6 bands, to establish a regression relationship between the MODIS images at time t1 and t2, and obtain a regression model with n principal components; secondly, the regression model with n principal components is transferred to the Landsat data at time t1, and the predicted Landsat data image at time t2 is obtained.

4. The method for spatiotemporal fusion of remote sensing images based on multispectral according to claim 1, characterized in that: In step (3), a cross-validation CV strategy is used to determine the number of selected components, including the following steps: (31) Divide the rough image pair, i.e., the MODIS images at time t1 and t2 of the corresponding position, into k subsets by equidistant segmentation; (32) Establish PLSR on k-1 subsets using the same number of components; (33) The predicted residual sum of squares PRESS is calculated for the remaining subset to quantify the error of the established model. The formula is as follows: ； in, represents the MODIS image at time t2; C2 (M) represents the MODIS image at time t2; (34) Steps (31)-(33) are repeated k times until all subsets are calculated once, and the average PRESS value of the model at each number of components is calculated; (35) Steps (31)-(34) are repeated n times, covering all component numbers; when the regression model reaches the minimum average PRESS value, its corresponding component number n is selected as the optimal component number F PLSR ; Then, the optimal n is used for the PLSR models of all local moving windows.

5. The method for spatiotemporal fusion of remote sensing images based on multispectral according to claim 1, characterized in that: Step (4) is as follows: Let the residual R(x, y, b) on MODIS be defined as the difference between the reflectance value of the MODIS pixel on the prediction date and the average of the predicted reflectance values ​​of all Landsat pixels corresponding to the pixel, and the formula is as follows: ； Among them, C(x, y, b) and F p (xi, yi, b) are the reflectance values ​​of the i-th Landsat pixel in band b at the MODIS and C(x, y) locations at position (x, y), respectively. m is the number of Landsat pixels within a MODIS pixel; the residual values ​​R(x, y, b) of all coarse pixels are then spatially interpolated to a fine resolution to obtain the fine residual r, which is used to make the compensation residual.

6. The method for spatiotemporal fusion of remote sensing images based on multispectral according to claim 1, characterized in that: Step (5) is as follows: Introduce similar pixels in the neighborhood to distribute the fine residual r, and obtain the residual-compensated Landsat prediction result F RC ; Then, by comparing the PLSR regression results F PLSR , the result of residual compensation F RC And the image F obtained by downscaling the MODIS image C The size of the change between the two determines which prediction result is used as the final prediction result F PLSR-STF Among them, F RC The formula is as follows: 。 7. A multi-spectral remote sensing image spatiotemporal fusion system, characterized in that: include: Acquisition module: used to obtain Landsat images and MODIS images using open source websites and perform preprocessing; PLSR module: used to construct PLSR regression relationship in spatiotemporal fusion; OPC module: used to select the optimal number of PLS ​​components OPC; Residual compensation module: used for adaptive residual compensation ARC; Prediction module: used to obtain the final prediction results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, a spatiotemporal fusion method of remote sensing images based on multispectral is implemented according to any one of claims 1 to 6.

9. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a spatiotemporal fusion method of remote sensing images based on multispectral is implemented according to any one of claims 1 to 6.

Images