Image fusion method and system based on multi-scale smoothing and sharpening filter

Abstract

The present invention discloses an image fusion method and system based on a multiscale smoothing and sharpening filter, belonging to the technical field of remote sensing image processing, the method comprising: obtaining a Landsat-8 image at a target time and a Sentinel-2 image at a reference time, and preprocessing them respectively; extracting high-frequency components from the preprocessed images respectively by means of a smoothing and sharpening filter; extracting a multiscale detail image from the high-frequency components of the Sentinel-2 image by means of a multiscale smoothing and sharpening filter; summing the multiscale detail images of the Sentinel-2 image, and adding them to the preprocessed Landsat-8 image to obtain an image fusion result. The present invention improves the accuracy of the fusion of the Landsat-8 image and the Sentinel-2 image by re-extracting and transferring detail information from the high-frequency components of the Sentinel-2 image, and can accurately capture phenology and land cover changes.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing, and in particular relates to an image fusion method and system based on a multi-scale smoothing and sharpening filter. Background Art

[0002] Landsat-8 and Sentinel-2 are two widely used medium-resolution satellites that can provide image data with high spatial resolution and spectral resolution. These data have important application value in agriculture, forestry, urban planning, environmental protection and other fields. Landsat-8 and Sentinel-2 images have similar wavelengths and the same geographic coordinate system, which makes it feasible to monitor surface changes at a higher frequency by fusing Landsat-8 and Sentinel-2 data. Combining Landsat-8 and Sentinel-2 data can shorten the revisit period to 2-3 days, greatly improving the application value of the data series.

[0003] At present, researchers have developed a variety of Landsat-8 and Sentinel-2 image fusion methods, such as area-to-point regression kriging (ATPRK), based on the classic spatiotemporal image fusion (STIF) model, etc. In recent years, with the rapid development of deep learning, some image fusion methods based on deep learning have also been developed, such as image fusion methods based on extended super-resolution convolutional neural networks and degradation-constrained spatiotemporal fusion networks.

[0004] However, most of the image fusion methods in the prior art cannot preserve the details and texture of the image well, and there are problems of over-sharpening or over-smoothing. Studies have shown that ATPRK has the problem of over-sharpening. Moreover, ATPRK involves complex semi-variogram modeling, which has a large amount of calculation and limits its practical application. The classic STIF model is mainly used to complete the fusion of Landsat and MODIS images, in which the magnification factor of the spatial resolution is 16-20 times. However, the scale factor in the fusion of Landsat-8 and Sentinel-2 images is 3 times, and compared with the resolution of the Landsat image that is the target of STIF, the resolution of Sentinel-2 images is higher, which means that there are more fine structures to be reconstructed. Therefore, the classic STIF model is not perfectly applicable to the fusion of Landsat-8 and Sentinel-2 images, which easily leads to spectral distortion and loss of detail features in the fused image. Summary of the invention

[0005] In view of this, the present invention proposes an image fusion method and system based on a multi-scale smoothing and sharpening filter, which is used to solve the problem that the image fusion method in the prior art is prone to lose the detailed features of the image.

[0006] In a first aspect, the present invention discloses an image fusion method based on a multi-scale smoothing and sharpening filter, the method comprising:

[0007] Obtain the Landsat-8 image of the target time and the Sentinel-2 image of the reference time of the same area, and preprocess them respectively;

[0008] The high-frequency components were extracted from the preprocessed Landsat-8 image and the preprocessed Sentinel-2 image by smoothing and sharpening filters respectively;

[0009] The high-frequency components of Sentinel-2 images are used as input images and the high-frequency components of Landsant-8 images are used as guide images. Multi-scale smoothing and sharpening filters are used to extract multi-scale detail images from the high-frequency components of Sentinel-2 images.

[0010] The multi-scale detail images in the preprocessed Sentinel-2 image are summed and added to the preprocessed Landsat-8 image to obtain the image fusion result.

[0011] On the basis of the above technical solution, preferably, the pretreatment includes:

[0012] The Landsat-8 image is upsampled by thin plate spline interpolation. The size of the upsampled Landsat-8 image is the same as that of the Sentinel-2 image.

[0013] The upsampled Landsat-8 image is denoised by morphological opening and closing operations to obtain the preprocessed Landsat-8 image.

[0014] The Sentinel-2 image is denoised and enhanced by the Gaussian Laplace operator to obtain the preprocessed Sentinel-2 image.

[0015] On the basis of the above technical solution, preferably, the calculation formula of the smoothing and sharpening filter is:

[0016]

[0017] Where J(p) is the pixel value after filtering at position p of the input image, μ i is the pixel average of the i-th input image block, φ iis the covariance of the input image patch and the guide image patch, G(p) is the pixel value at position p of the guide image, and v i is the pixel average of the i-th guide image block; w i is the weight, α i is the parameter of the generalized gamma distribution, and M is the number of image patches at position p in the input image.

[0018] On the basis of the above technical solution, preferably, the extracting high-frequency components in the pre-processed Landsat-8 image and the pre-processed Sentinel-2 image respectively by using a smoothing and sharpening filter SSIF specifically comprises:

[0019] The preprocessed Landsat-8 image is input into a smoothing and sharpening filter SSIF to obtain a low-frequency component of the preprocessed Landsat-8 image, and the corresponding low-frequency component is removed from the preprocessed Landsat-8 image to obtain a high-frequency component of the preprocessed Landsat-8 image;

[0020] The preprocessed Sentinel-2 image is input into the smoothing and sharpening filter SSIF to obtain the low-frequency component of the preprocessed Sentinel-2 image, and the corresponding low-frequency component is removed from the preprocessed Sentinel-2 image to obtain the high-frequency component of the preprocessed Sentinel-2 image.

[0021] On the basis of the above technical solution, preferably, the calculation formula of the high frequency component of the preprocessed Landsat-8 image is:

[0022]

[0023] in, is the preprocessed Landsat-8 image, SSIF(·) represents the filtering operation of the smoothing and sharpening filter SSIF, L high is the high-frequency component of the preprocessed Landsat-8 image;

[0024] The calculation formula of the high frequency component of the preprocessed Sentinel-2 image is:

[0025]

[0026] in, is the preprocessed Sentinel-2 image, S high is the high-frequency component of the preprocessed Sentinel-2 image.

[0027] On the basis of the above technical solution, preferably, the step of extracting a multi-scale detail image from the high-frequency component of the Sentinel-2 image using a multi-scale smoothing and sharpening filter specifically includes:

[0028] A multi-scale smoothing and sharpening filter is constructed. The multi-scale smoothing and sharpening filter is used to repeatedly filter the filtering result of the smoothing and sharpening filter. The expression is:

[0029] S j =SSIF(S j-1 )

[0030] Among them, S j-1 , S j are the filtered images obtained by the j-1th and jth repeated filtering processes, respectively, j ≥ 1;

[0031] The high-frequency components of the Sentinel-2 image are repeatedly filtered by a multi-scale smoothing and sharpening filter to obtain a multi-scale detail image. The calculation formula for the detail image of the jth scale is:

[0032] D j =S j-1 -S j

[0033] Where j = 1, 2, ..., N, N is the number of scales, D j is the detail image of the jth scale; when j=1, S j-1 =S high .

[0034] On the basis of the above technical solution, preferably, the calculation formula of the image fusion result is:

[0035]

[0036] Where D is the superposition result of detail images of all scales; is the preprocessed Landsat-8 image, S proposed is the image fusion result.

[0037] In a second aspect of the present invention, an image fusion system based on a multi-scale smoothing and sharpening filter is disclosed, the system comprising:

[0038] Preprocessing module: used to obtain Landsat-8 images of the target time and Sentinel-2 images of the same area at the reference time, and perform preprocessing respectively;

[0039] Component extraction module: used to extract high-frequency components from the pre-processed Landsat-8 image and the pre-processed Sentinel-2 image respectively through a smoothing and sharpening filter;

[0040] Detail extraction module: It is used to extract multi-scale detail images from the high-frequency components of the Sentinel-2 image using a smoothing and sharpening filter, using the high-frequency components of the Sentinel-2 image as the input image and the high-frequency components of the Landsant-8 image as the guide image.

[0041] Image fusion module: used to sum the multi-scale detail images in the preprocessed Sentinel-2 image and add them to the preprocessed Landsat-8 image to obtain the image fusion result.

[0042] In a third aspect of the present invention, an electronic device is disclosed, comprising: at least one processor, at least one memory, a communication interface and a bus;

[0043] Wherein, the processor, memory, and communication interface communicate with each other via the bus;

[0044] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to the first aspect of the present invention.

[0045] According to a fourth aspect of the present invention, a computer-readable storage medium is disclosed, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method described in the first aspect of the present invention.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1) The present invention extracts high-frequency components from Landsat-8 images and Sentinel-2 images respectively through smoothing and sharpening filters, takes the high-frequency components of Landsant-8 images as guide images, and utilizes smoothing and sharpening filters to re-extract and transfer the multi-scale detail images missing from the Landsat-8 images from the high-frequency components of the Sentinel-2 images, thereby enhancing the spatial details of the fused image; the image fusion result of the present invention can both retain the spectral characteristics of the Landsat-8 image at the target time and transfer the spatial details of the Sentinel-2 image at the reference time.

[0048] 2) The present invention realizes separation of high-frequency components and low-frequency components through the smoothing or sharpening function of the smoothing and sharpening filter SSIF, and designs a multi-scale smoothing and sharpening filter. The scale feature is increased by repeated filtering processing, and a detail image is obtained according to the difference between two adjacent filtered images, thereby obtaining a multi-scale detail image of the Sentinel-2 image, which is convenient for accurately restoring the details and texture of the image and preventing over-sharpening or over-smoothing.

[0049] 3) The present invention improves the accuracy of Landsat-8 image and Sentinel-2 image fusion by re-extracting and transferring detail information from the high-frequency components of Sentinel-2 images, and its fusion accuracy is better than the existing Landsat-8 and Sentinel-2 image fusion method; the image fusion method of the present invention can not only accurately capture phenology and land cover changes, but also reconstruct fine structures and edges. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 It is a flow chart of the image fusion method based on multi-scale smoothing and sharpening filter of the present invention;

[0052] Figure 2 The flowchart of the algorithm of the multi-scale smoothing and sharpening filter of the present invention (taking dual scale as an example);

[0053] Figure 3 Data sets used by embodiments of the present invention;

[0054] Figure 4 The following is a comparison chart of the fusion results of five Landsat-8 and Sentinel-2 fusion methods on this dataset;

[0055] Figure 5 for Figure 4 A local magnified image of the area marked by the box. DETAILED DESCRIPTION

[0056] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Generally speaking, Sentinel-2 images have a higher resolution and contain more fine shapes, textures, and edges. Combining Landsat-8 with Sentinel-2 data can obtain a higher frequency high-resolution data sequence. However, most Landsat-8 and Sentinel-2 image fusion methods do not fully utilize the Sentinel-2 images of the reference date, cannot accurately restore the details and texture of the image, and have problems of over-sharpening or over-smoothing. Therefore, the present invention proposes a Landsat-8 and Sentinel-2 image fusion method based on a multi-scale smoothing and sharpening filter, which can not only retain the spectral characteristics of the Landsat-8 image at the target time, but also transfer the spatial details of the Sentinel-2 image at the reference time.

[0058] See also Figure 1 The present invention discloses an image fusion method based on a multi-scale smoothing and sharpening filter, the method comprising:

[0059] S1. Obtain the Landsat-8 image at the target time and the Sentinel-2 image at the reference time of the same area, and preprocess them respectively.

[0060] Step S1 specifically includes the following sub-steps:

[0061] S11. Remote sensing data acquisition.

[0062] In order to ensure the accuracy and effectiveness of fusion, it is usually necessary to obtain Landsat-8 images and Sentinel-2 images of the same target area. This ensures that the two remote sensing images obtained have similar geographical locations and resolutions, making it easier to perform image fusion.

[0063] The present invention takes the images of Landsat-8 30m band and Sentinel-2 10m band as examples to obtain remote sensing data.

[0064] S12: Image preprocessing.

[0065] Step S12 specifically includes the following sub-steps:

[0066] S121, upsampling the Landsat-8 image by thin plate spline interpolation (TPS), so that the size of the upsampled Landsat-8 image is the same as that of the Sentinel-2 image;

[0067] TPS is a robust spatial data interpolation algorithm based on spatial correlation. It uses thin plate spline functions to establish a surface passing through all control points and minimizes the curvature of the surface, which is measured by an energy function. The upsampled image obtained by TPS interpolation can remain smooth and retain most of the spatial information, which is an important component of the final fused image.

[0068] In the TPS interpolation algorithm, the calculation formula for the interpolation result at the pixel position (x, y) of the kth band is as follows:

[0069]

[0070] in, is the interpolation result at the pixel position (x, y) of the kth band, a0, a1, a2 and b l is a parameter determined by minimizing the energy function, and L is a known control point (x l ,y l ), l = 1, 2, ..., L,

[0071] b l The following constraints must also be met:

[0072]

[0073] S122, performing denoising on the upsampled Landsat-8 image through morphological opening and closing operations to obtain a preprocessed Landsat-8 image.

[0074] Affected by sensor accuracy and environmental conditions, there may be noise and artifacts in the original image, which infiltrate the fused image and reduce the fusion effect. In addition, upsampling is also prone to amplification and blurring of noise. Therefore, it is necessary to reduce the impact of noise on the image in the preprocessing step.

[0075] The present invention implements simple and effective denoising on the upsampled Landsat-8 image through morphological opening and closing operations. The function of the opening operation is to delete small objects and unnecessary edges, while the function of the closing operation is to fill the small holes in the object. The calculation formula of the morphological opening and closing operation is as follows. .

[0076]

[0077] in, is the pixel value at the pixel position (x, y) of the kth band before and after the morphological opening and closing operation, S(m, n) is the pixel value of the structural element at the position (m, n), represents the opening operation, represents the closing operation, and the calculation formulas for the opening operation and the closing operation are:

[0078]

[0079] where F k (x, y) is the pixel value at the pixel position (x, y) of the kth band after the opening operation. ⊙ represents morphological erosion, represents morphological dilation, and their calculation formulas are as follows:

[0080]

[0081] S123, optimizing the Sentinel-2 image by using the Laplacian of Gaussian operator, reducing noise by using Gaussian filtering, and then enhancing spatial details by using the Laplacian operator to obtain a preprocessed Sentinel-2 image.

[0082] For Sentinel-2 images, image optimization is performed using the Laplacian of Gaussian (LOG), which reduces noise through Gaussian filtering and then enhances spatial details through the Laplacian.

[0083] S2. Extract high-frequency components from the preprocessed Landsat-8 image and the preprocessed Sentinel-2 image using smoothing and sharpening filters.

[0084] The smoothing-sharpening filter (SSIF) is an edge-aware filter that adjusts the function and level of smoothing or sharpening through a tuning parameter. In SSIF, input images and guide images are usually used for processing. The input image refers to the original image to be processed. It is the target of SSIF's filtering operation. By smoothing or sharpening the input image, the image quality can be improved, the noise can be reduced, and the details and edges of the image can be improved. The guide image is used to guide SSIF to retain or enhance the edge information during the filtering process. The guide image often contains more details and structural information, which can help SSIF better grasp the characteristics of the image, making the filtering effect more accurate and natural. Through the edge information of the guide image, SSIF can better distinguish the structure and texture of the image, thereby achieving a better balance between smoothing and sharpening. The guide image can be the input image itself or another image.

[0085] Similar to the classic guided filter, SSIF designs a block interpolation model. An image block with a radius of r contains M = (2r + 1) 2 pixels, and the image blocks are divided with each pixel as the center, where the i-th image block is denoted as Ω i , i=1,2,…,M. Let Ii (q) represents Ω i The pixel value at position q in the image is, then the output J of SSIF is i (q) is:

[0086] J i (q) = μ i +sign(φ i )α i (G i (q)-v i ) (8)

[0087] where μ i is the average value of the i-th input image patch, φ i is the covariance between the ith input image patch and the guidance image patch, G i (q) is the pixel value at position q in the guidance image patch, v i is the average value of the i-th guidance image patch.

[0088] In order to determine a suitable parameter α i , calculate the non-negative log-likelihood of the i-th image patch:

[0089]

[0090] in is the covariance of the guidance image patch, and τ is set to 1.

[0091] Considering α i Satisfying the generalized gamma distribution as a priori, the cost function D(α i ):

[0092]

[0093] Among them, θ>0 is a scale parameter, and η≥0 is a parameter that controls the shape of the distribution. Adjust the parameter setting to η=κ∈, To ensure that the choice between smoothing (0≤κ<1) and sharpening (κ>1) is controlled only by κ.

[0094] In formula (10) We can get α i The calculation formula is:

[0095]

[0096] The pixel at position p belongs to M overlapping image blocks. The final SSIF filtering result is the weighted average of the outputs generated by each image block. Specifically, the pixel value after filtering at position p is:

[0097]

[0098] Where J(p) is the pixel value after filtering at position p of the input image, μ i is the pixel average of the i-th input image block, φ i is the covariance of the ith input image patch and the guide image patch, G(p) is the pixel value at position p of the guide image, and v i is the pixel average of the i-th guide image block; w i is the weight, α i are the parameters of the generalized gamma distribution.

[0099] Weight w i The calculation formula is as follows:

[0100]

[0101] where c i is a regularization parameter that ensures s is a user-defined scale parameter, Yes all The average value of .

[0102] The present invention separates the low-frequency component and the high-frequency component of the image on the basis of the above smoothing and sharpening filter SSIF, and extracts the high-frequency component.

[0103] Step S2 specifically includes the following sub-steps:

[0104] S21, input the preprocessed Landsat-8 image into a smoothing and sharpening filter SSIF to obtain a low-frequency component of the preprocessed Landsat-8 image, remove the corresponding low-frequency component from the preprocessed Landsat-8 image, and obtain a high-frequency component of the Landsat-8 image. The calculation formula is:

[0105]

[0106] in, is the preprocessed Landsat-8 image, SSIF(·) represents the filtering operation of the smoothing and sharpening filter SSIF, L high is the high frequency component of the Landsat-8 image.

[0107] S22, input the preprocessed Sentinel-2 image into the smoothing and sharpening filter SSIF to obtain the low-frequency component of the preprocessed Sentinel-2 image, remove the corresponding low-frequency component from the preprocessed Sentinel-2 image, and obtain the high-frequency component of the Sentinel-2 image. The calculation formula is:

[0108]

[0109] in, is the preprocessed Sentinel-2 image, S high is the high frequency component of the Sentinel-2 image.

[0110] S3, using the high-frequency component of the Sentinel-2 image as the input image and the high-frequency component of the Landsant-8 image as the guide image, a multi-scale smoothing and sharpening filter is used to extract a multi-scale detail image from the high-frequency component of the Sentinel-2 image.

[0111] The present invention uses the high frequency component of the Sentinel-2 image as the initial input image and the high frequency component of the Landsant-8 image as the guide image, and uses SSIF to extract Missing details.

[0112] In order to make the detail image closer to the surface coverage at the target time, the present invention designs a multi-scale smoothing and sharpening filter, which uses the multi-scale smoothing and sharpening filter to repeatedly filter the high-frequency components of the Sentinel-2 image, that is, the filtered output result of the image is repeatedly processed by SSIF, and each repetition will increase a scale to obtain a multi-scale detail image. The expression of the multi-scale smoothing and sharpening filter is:

[0113] S j =SSIF(S j-1 )

[0114] Where j is the number of repetitions. Since each repetition will increase a scale, j also represents the scale of the jth repetition, j = 1, 2, ..., N, N is the number of scales, S j-1 , S j They are the filtered images obtained by the j-1th and jth repeated filtering processes respectively.

[0115] Figure 2 This is an algorithm flow chart of the multi-scale smoothing and sharpening filter of the present invention (taking dual scale as an example).

[0116] The high-frequency components of the Sentinel-2 image are repeatedly filtered using a multi-scale smoothing and sharpening filter to obtain a multi-scale detail image. The calculation formula for the j-th scale detail image is:

[0117] D j =S j-1 -S j

[0118] D j is the detail image of the jth scale, which is the difference between two adjacent filtered images. When j = 1, S j-1=S high .

[0119] The present invention realizes separation of high-frequency components and low-frequency components through the smoothing or sharpening function of a smoothing and sharpening filter SSIF, designs a multi-scale smoothing and sharpening filter, increases scale features through repeated filtering processing, and obtains a detail image according to the difference between two adjacent filtered images, thereby re-extracting the multi-scale detail image missing from the Landsat-8 image from the high-frequency component of the Sentinel-2 image, enhancing the spatial details of the fused image, facilitating accurate restoration of the details and texture of the image, and preventing over-sharpening or over-smoothing.

[0120] S4. Sum the multi-scale detail images of the Sentinel-2 image and add them to the preprocessed Landsat-8 image to obtain the image fusion result.

[0121] The multi-scale detail images of the Sentinel-2 image are summed to obtain the total detail image D, and the formula is:

[0122]

[0123] The total detail image is added to the preprocessed Landsat-8 image to obtain the image fusion result. The calculation formula is:

[0124]

[0125] Where D is the sum of detail images of all scales; is the preprocessed Landsat-8 image, S proposed is the image fusion result.

[0126] The present invention makes full use of the Sentinel-2 image of the reference date, and improves the accuracy of the fusion of the Landsat-8 image and the Sentinel-2 image by re-extracting and transferring detail information from the high-frequency components of the Sentinel-2 image. It can not only retain the spectral characteristics of the Landsat-8 image at the target time, but also transfer the spatial details of the Sentinel-2 image at the reference time.

[0127] The Landsat-8 and Sentinel-2 image fusion method based on the multi-scale smoothing and sharpening filter of the present invention is verified below in conjunction with a specific example.

[0128] Figure 3It is the data set used in the experiment of the present invention, wherein the images shown in ①② are the input images of the experiment, which are the Sentinel-2 images at the reference time (April 20, 2018) and the Landsat-8 images at the target time (March 16, 2018), respectively, and the image shown in ③ is the Sentinel-2 image at the target time. Each image contains four bands, namely, the blue band, the green band, the red band and the near-infrared band. The spatial resolution of the Sentinel-2 image is 10m, containing 990×990 pixels, while the spatial resolution of the Landsat-8 image is 30m, containing 330×330 pixels. In the experiment, the scale of the present invention is set to 2, and STARFM (Spatial and Temporal Adaptive Reflectance FusionModel, spatiotemporal adaptive reflectance fusion model), FSDAF (Flexible Spatiotemporal Data Fusion, flexible spatiotemporal data fusion model) and Fit-Fc all use corresponding simplified versions to adapt to the scene with only two input images.

[0129] Figure 4 The following is a comparison of the fusion results of five Landsat-8 and Sentinel-2 fusion methods on this dataset, where Figure (a) is the reference image, and Figures (b) to (f) are the fusion results of ATPRK, STARFM, FSDAF, Fit-FC and the method of the present invention. Figure 5 It is shown in Figure 4 The locally enlarged images of the area marked by the box, wherein (A) is a locally enlarged image of the reference image, (B) is a locally enlarged image of the ATPRK fusion result, (C) is a locally enlarged image of the STARFM fusion result; (D) is a locally enlarged image of the FSDAF fusion result, (E) is a locally enlarged image of the Fit-FC fusion result; (F) is a locally enlarged image of the fusion result of the present invention. The heterogeneous area mainly experienced phenological changes, and some of them experienced land cover changes, such as the middle and upper part of the selected marked area. In the field of remote sensing image fusion, the difficulty of image fusion lies in capturing land cover changes while reproducing the fine structure of dense buildings in urban areas. ATPRK can capture temporal changes in detail, but there are obvious jagged edges and mosaics, resulting in low clarity. STARFM and Fit-FC can neither detect land cover changes nor reconstruct textures and details well. FSDAF and the method of the present invention reproduce land cover changes relatively well, and the fusion results produced by the method of the present invention are more uniform and retain most of the boundary information.

[0130] The evaluation indicators of all compared methods on this dataset are shown in Table 1. As shown in Table 1, the evaluation indicators used include SSIM (structural similarity), PSNR (peak signal-to-noise ratio), RMSE (root mean square error), CC (correlation coefficient) and SAM (spectral angular distance).

[0131] Table 1 Comparison of evaluation indicators of fusion results of five methods

[0132]

[0133] It can be concluded from Table 1 that, compared with other methods, the method of the present invention has higher SSIM, PSNR and CC, and smaller RMSE and SAM. This shows that the method of the present invention can produce the most accurate fusion results, in which RMSE is reduced by 0.0036-0.0049 and CC is improved by 0.0152-0.0192.

[0134] In general, the fusion accuracy of the present invention is better than that of the existing Landsat-8 and Sentinel-2 image fusion methods. In terms of visual effects, the image fusion method of the present invention can not only accurately capture phenology and land cover changes, but also reconstruct fine structures and edges. In terms of evaluation indicators, the method of the present invention achieves the highest SSIM, PSNR and CC, and the lowest RMSE and SAM among all the comparison methods.

[0135] On the basis of the above method embodiments, the present invention further discloses an image fusion system based on a multi-scale smoothing and sharpening filter, the system comprising:

[0136] Preprocessing module: used to obtain Landsat-8 images of the target time and Sentinel-2 images of the same area at the reference time, and perform preprocessing respectively;

[0137] Component extraction module: used to extract high-frequency components from the pre-processed Landsat-8 image and the pre-processed Sentinel-2 image respectively through a smoothing and sharpening filter;

[0138] Detail extraction module: It is used to extract multi-scale detail images from the high-frequency components of the Sentinel-2 image using a smoothing and sharpening filter, using the high-frequency components of the Sentinel-2 image as the input image and the high-frequency components of the Landsant-8 image as the guide image.

[0139] Image fusion module: used to sum the multi-scale detail images in the preprocessed Sentinel-2 image and add them to the preprocessed Landsat-8 image to obtain the image fusion result.

[0140] The above system embodiments and method embodiments correspond one to one, and for a brief description of the system embodiments, please refer to the method embodiments.

[0141] The present invention also discloses an electronic device, comprising: at least one processor, at least one memory, a communication interface and a bus; wherein the processor, memory, and communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the aforementioned method of the present invention.

[0142] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to implement all or part of the steps of the method described in the embodiment of the present invention. The storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0143] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be distributed on multiple network units. A person skilled in the art may select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment without creative work.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image fusion method based on multi-scale smoothing and sharpening filter, characterized in that: The method comprises: Obtain the Landsat-8 image of the target time and the Sentinel-2 image of the reference time of the same area, and preprocess them respectively; The high-frequency components were extracted from the preprocessed Landsat-8 image and the preprocessed Sentinel-2 image by smoothing and sharpening filters respectively; The high-frequency components of Sentinel-2 images are used as input images and the high-frequency components of Landsant-8 images are used as guide images. Multi-scale smoothing and sharpening filters are used to extract multi-scale detail images from the high-frequency components of Sentinel-2 images. The multi-scale detail images of the Sentinel-2 image are summed and added to the preprocessed Landsat-8 image to obtain the image fusion result; The extracting high-frequency components from the pre-processed Landsat-8 image and the pre-processed Sentinel-2 image by using a smoothing and sharpening filter specifically includes: The preprocessed Landsat-8 image is input into a smoothing and sharpening filter to obtain a low-frequency component of the preprocessed Landsat-8 image, and the corresponding low-frequency component is removed from the preprocessed Landsat-8 image to obtain a high-frequency component of the Landsat-8 image; Input the preprocessed Sentinel-2 image into a smoothing and sharpening filter to obtain a low-frequency component of the preprocessed Sentinel-2 image, and remove the corresponding low-frequency component from the preprocessed Sentinel-2 image to obtain a high-frequency component of the Sentinel-2 image; The calculation formula of the high frequency component of the Landsat-8 image is: in, is the preprocessed Landsat-8 image, SSIF(·) represents the filtering operation of the smoothing and sharpening filter, L high is the high frequency component of the Landsat-8 image; The calculation formula of the high frequency component of the Sentinel-2 image is: in, is the preprocessed Sentinel-2 image, S high is the high frequency component of the Sentinel-2 image; The method of extracting a multi-scale detail image from the high-frequency component of the Sentinel-2 image using a multi-scale smoothing and sharpening filter comprises: A multi-scale smoothing and sharpening filter is constructed. The multi-scale smoothing and sharpening filter is used to repeatedly filter the filtering result of the smoothing and sharpening filter. The expression is: S j =SSIF(S j-1 ) Among them, S j-1 , S j are the filtered images obtained by the j-1th and jth repeated filtering processes, respectively, j ≥ 1; The high-frequency components of the Sentinel-2 image are repeatedly filtered by a multi-scale smoothing and sharpening filter to obtain a multi-scale detail image. The calculation formula for the detail image of the jth scale is: D j =S j-1 -S j Where j = 1, 2, ..., N, N is the number of scales, D j is the detail image of the jth scale; when j=1, S j-1 =S high ; The calculation formula of the image fusion result is: Where D is the sum of detail images of all scales; is the preprocessed Landsat-8 image, S proposed is the image fusion result.

2. The image fusion method based on multi-scale smoothing and sharpening filter according to claim 1, characterized in that: The pre-processing comprises: The Landsat-8 image is upsampled by thin plate spline interpolation. The upsampled Landsat-8 image has the same size as the Sentinel-2 image. The upsampled Landsat-8 image is denoised by morphological opening and closing operations to obtain the preprocessed Landsat-8 image. The Sentinel-2 image is denoised and enhanced by the Gaussian Laplace operator to obtain the preprocessed Sentinel-2 image.

3. The image fusion method based on multi-scale smoothing and sharpening filter according to claim 1, characterized in that: The calculation formula of the smoothing and sharpening filter is: Where J(p) is the pixel value after filtering at position p of the input image, μ i is the pixel average of the i-th input image block, φ i is the covariance of the input image patch and the guide image patch, G(p) is the pixel value at position p of the guide image, and v i is the pixel average of the i-th guide image block; w i is the weight, α i are the parameters that obey the generalized gamma distribution, i = 1, 2, …, M, and M is the number of image blocks at position p of the input image.

4. An image fusion system based on multi-scale smoothing and sharpening filter, characterized in that: The system comprises: Preprocessing module: used to obtain Landsat-8 images of the target time and Sentinel-2 images of the same area at the reference time, and perform preprocessing respectively; Component extraction module: used to extract high-frequency components from the pre-processed Landsat-8 image and the pre-processed Sentinel-2 image respectively through a smoothing and sharpening filter; the extraction of high-frequency components from the pre-processed Landsat-8 image and the pre-processed Sentinel-2 image respectively through a smoothing and sharpening filter specifically includes: The preprocessed Landsat-8 image is input into a smoothing and sharpening filter to obtain a low-frequency component of the preprocessed Landsat-8 image, and the corresponding low-frequency component is removed from the preprocessed Landsat-8 image to obtain a high-frequency component of the Landsat-8 image; Input the preprocessed Sentinel-2 image into a smoothing and sharpening filter to obtain a low-frequency component of the preprocessed Sentinel-2 image, and remove the corresponding low-frequency component from the preprocessed Sentinel-2 image to obtain a high-frequency component of the Sentinel-2 image; The calculation formula of the high frequency component of the Landsat-8 image is: in, is the preprocessed Landsat-8 image, SSIF(·) represents the filtering operation of the smoothing and sharpening filter, L high is the high frequency component of the Landsat-8 image; The calculation formula of the high frequency component of the Sentinel-2 image is: in, is the preprocessed Sentinel-2 image, S high is the high frequency component of the Sentinel-2 image; A detail extraction module is used to extract a multi-scale detail image from the high-frequency component of the Sentinel-2 image using a multi-scale smoothing and sharpening filter, using the high-frequency component of the Sentinel-2 image as an input image and the high-frequency component of the Landsant-8 image as a guide image; the method of extracting a multi-scale detail image from the high-frequency component of the Sentinel-2 image using a multi-scale smoothing and sharpening filter includes: A multi-scale smoothing and sharpening filter is constructed. The multi-scale smoothing and sharpening filter is used to repeatedly filter the filtering result of the smoothing and sharpening filter. The expression is: S j =SSIF(S j-1 ) Among them, S j-1 , S j are the filtered images obtained by the j-1th and jth repeated filtering processes, respectively, j ≥ 1; SSIF(·) represents the filtering operation of the smoothing and sharpening filter; The high-frequency components of the Sentinel-2 image are repeatedly filtered by a multi-scale smoothing and sharpening filter to obtain a multi-scale detail image. The calculation formula for the detail image of the jth scale is: D j =S j-1 -S j Where j = 1, 2, ..., N, N is the number of scales, D j is the detail image of the jth scale; when j=1, S j-1 =S high , S high is the high frequency component of the Sentinel-2 image; Image fusion module: used to sum the multi-scale detail images in the preprocessed Sentinel-2 image and add them to the preprocessed Landsat-8 image to obtain the image fusion result; the calculation formula of the image fusion result is: Where D is the sum of detail images of all scales; is the preprocessed Landsat-8 image, S proposed is the image fusion result.

5. An electronic device, characterized in that: include: at least one processor, at least one memory, a communication interface, and a bus; Wherein, the processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method according to any one of claims 1 to 3.

Images