Non-reference Spatial Quality Assessment Method and Device for Remote Sensing Image Fusion

Abstract

The present invention relates to a no-reference spatial quality assessment method and device for remote sensing image fusion. A single-band high-resolution image is obtained from a multi-band high-resolution image. By respectively extracting texture features from the fused image and the single-band high-resolution image, the texture features of the fused image and the texture features of the single-band high-resolution image are obtained. Then, the mutual information metric is performed on the texture features of the fused image and the texture features of the single-band high-resolution image to obtain the feature mutual information between the fused image and the single-band high-resolution image. Furthermore, the spatial quality of the fused image is evaluated without reference based on the feature mutual information. This method has high accuracy and makes the no-reference spatial quality assessment of remote sensing image fusion more in line with practical applications.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image fusion, and in particular to a method and device for non-reference spatial quality assessment of remote sensing image fusion. Background Art

[0002] Orbiting high-resolution earth observation satellites can provide remote sensing images with different spatial resolutions and different spectral resolutions, such as high-spatial-resolution panchromatic (PAN) images, low-spatial-resolution multispectral (MS) images, and even lower-spatial-resolution hyperspectral (HS) images. Image fusion is an effective way to improve the spatial resolution of optical remote sensing images. Its purpose is to use high-resolution images to improve the spatial resolution of low-resolution images while minimizing spectral information distortion in the fusion results as much as possible.

[0003] Due to different fusion methods, there are differences in the spatial quality of the fusion results. Therefore, it is necessary to evaluate the spatial quality of the fusion results. The evaluation of the spatial quality of the fusion results can be divided into subjective evaluation and objective evaluation. Advanced satellite platforms or satellite ground equipment can use the objective evaluation values of the spatial quality of the fusion results as metadata for subsequent image classification, target detection and recognition, and other applications. According to whether a reference image is used to evaluate the quality of the fused image, the objective quality evaluation of remote sensing image fusion can be divided into two types: reference-based quality evaluation and non-reference quality evaluation. The assumption of reference-based quality evaluation is too ideal. In practical applications, there is no corresponding reference image (i.e., an image with higher spatial resolution) for the fusion results, and there are many uncertainties in the implementation process of reference-based quality evaluation. Therefore, non-reference quality evaluation is preferred for remote sensing image fusion.

[0004] Among the existing fusion image evaluation methods, the widely used non-reference spatial distortion index is the spatial distortion index of QNR The spatial distortion index of FQNR And the spatial distortion index of RQNR Are the most commonly used non-reference spatial distortion indices. When using this index for spatial quality evaluation, it is severely coupled with spectral distortion quality evaluation, and the results often tend to be more towards comprehensive quality evaluation. For Making improvements, only extracting high-frequency components for evaluation, the evaluation effect has been improved, but the upper limit of the evaluation range of this index is not necessarily 1. Whether it is Or Both are related to the degradation of panchromatic images, and the degradation filter generally selects a modulation transfer filter (MTF). The imaging sensors in orbit often experience parameter changes of the MTF filter due to component or instrument aging. Therefore, the degradation filter has uncertainty, which ultimately leads to insufficient accuracy in spatial quality evaluation when and are used as no-reference spatial distortion metrics. Summary of the Invention

[0005] Based on this, it is necessary to provide a no-reference spatial quality evaluation method and device for remote sensing image fusion with high accuracy in view of the above technical problems.

[0006] In a first aspect, the present invention provides a no-reference spatial quality evaluation method for remote sensing image fusion, including:

[0007] Obtaining a single-band high-resolution image from a multi-band high-resolution image;

[0008] Performing texture feature extraction on the fused image and the single-band high-resolution image respectively to obtain the texture features of the fused image and the texture features of the single-band high-resolution image;

[0009] Performing mutual information measurement on the texture features of the fused image and the texture features of the single-band high-resolution image to obtain the feature mutual information between the fused image and the single-band high-resolution image;

[0010] Performing no-reference evaluation of the spatial quality of the fused image based on the feature mutual information.

[0011] In one embodiment, the fused image has multiple bands, and performing texture feature extraction on the fused image means performing texture feature extraction on each band of the fused image to obtain the texture features of each band of the fused image.

[0012] In one embodiment, performing mutual information measurement on the texture features of the fused image and the texture features of the single-band high-resolution image to obtain the feature mutual information between the fused image and the single-band high-resolution image includes:

[0013] Performing mutual information measurement on the texture features of each band of the fused image and the texture features of the single-band high-resolution image to obtain the band feature mutual information between each band of the fused image and the single-band high-resolution image;

[0014] Calculating the mean of all band feature mutual information as the feature mutual information between the fused image and the single-band high-resolution image.

[0015] In one embodiment, obtaining a single-band high-resolution image from a multi-band high-resolution image is to perform weighted summation or selection on the high-resolution multi-band image to generate a single-band high-resolution image.

[0016] In one embodiment, the mutual information metric is calculated in blocks. Within each image block of the block calculation, the mutual information metric is calculated in a sliding window manner.

[0017] In one embodiment, define F i as the texture feature of the i-th band of the fused image, and F P as the texture feature of the single-band high-resolution image. Let the mutual information of the band features be calculating the mutual information metric between the texture feature of the i-th band of the fused image and the texture feature of the single-band high-resolution image is

[0018]

[0019] In the formula, FMI k (F i , F P ) is the mutual information between F i and F P on the k-th image block, L is the total number of image blocks generated when calculating the mutual information between F i and F P , and k = 1, 2... L.

[0020] In one embodiment, the mutual information FMI i (F P and F k (F i , F P ) of the k-th image block is calculated as

[0021]

[0022] In the formula, I j (F i , F P ) represents the mutual information value between F i and F P at the j-th sliding window, H j (F i ) represents the entropy of F i at the j-th sliding window, H j (F P ) represents the entropy of F P at the j-th sliding window, and n is the total number of sliding windows on each image block, and j = 1, 2... n.

[0023] In one embodiment, define D FMITo fuse the feature mutual information of the fused image and the single-band high-resolution image, the mean value of the feature mutual information of all bands is calculated as

[0024]

[0025] In the formula, N is the total number of bands of the fused image.

[0026] In one embodiment, a texture filter is used to extract texture features from the fused image and the single-band high-resolution image.

[0027] In one embodiment, the single-band high-resolution image is generated by weighted summation or selection of the high-resolution multi-band image.

[0028] In a second aspect, the present invention also provides a no-reference spatial quality assessment device for remote sensing image fusion, including a memory and a processor. The memory stores a computer program. It is characterized in that when the processor executes the computer program, the following steps are implemented:

[0029] Obtain a single-band high-resolution image according to the multi-band high-resolution image;

[0030] Extract texture features from the fused image and the single-band high-resolution image respectively to obtain the texture features of the fused image and the texture features of the single-band high-resolution image;

[0031] Perform mutual information measurement on the texture features of the fused image and the texture features of the single-band high-resolution image to obtain the feature mutual information between the fused image and the single-band high-resolution image;

[0032] Perform no-reference assessment on the spatial quality of the fused image according to the feature mutual information.

[0033] The above no-reference spatial quality assessment method and device for remote sensing image fusion obtain a single-band high-resolution image according to the multi-band high-resolution image, extract the texture features of the fused image and the texture features of the single-band high-resolution image by respectively performing texture feature extraction on the fused image and the single-band high-resolution image, then perform mutual information measurement on the texture features of the fused image and the texture features of the single-band high-resolution image to obtain the feature mutual information between the fused image and the single-band high-resolution image, and then perform no-reference assessment on the spatial quality of the fused image according to the feature mutual information. This method evaluates the spatial quality of the fused image according to the correlation between the texture features of the fused image and the texture features of the high-resolution remote sensing image, does not involve panchromatic image degradation, excludes the interference of the degradation filter, has high accuracy, and does not require the assumption of scale invariance, breaking through the limitation of the reference image, making the no-reference spatial quality assessment of remote sensing image fusion more in line with practical applications. Description of the Drawings

[0034] Figure 1 It is a schematic flowchart of a no-reference spatial quality assessment method for remote sensing image fusion provided in an embodiment of the present invention;

[0035] Figure 2 It is a schematic flowchart of a no-reference spatial quality assessment method for remote sensing image fusion in one embodiment of the present invention;

[0036] Figure 3 It is a schematic flowchart of a no-reference spatial quality assessment method for remote sensing image fusion in one embodiment of the present invention;

[0037] Figure 4 It is a schematic diagram of the principle of a no-reference spatial quality assessment method for remote sensing image fusion in one embodiment of the present invention;

[0038] Figure 5 It is a schematic diagram of the principle of mutual information measurement in one embodiment of the present invention;

[0039] Figure 6 It is a fusion result map of the IKONOS satellite dataset in one embodiment of the present invention, where Figure 6 (a) is the MS image, Figure 6 (b) is the PAN image, Figure 6 (c) is the fused image by the IHS method, Figure 6 (d) is the fused image by the PCA method, Figure 6 (e) is the fused image by the BT method, Figure 6 (f) is the fused image by the BDSD method, Figure 6 (g) is the fused image by the GS method, Figure 6 (h) is the fused image by the PRACS method, Figure 6 (i) is the fused image by the ATWT method, Figure 6 (j) is the fused image by the ATWT-M2 method, Figure 6 (k) is the fused image by the ATWT-M3 method, Figure 6 (l) is the fused image by the AWLP method;

[0040] Figure 7 It is an image of each band of the ATWT fusion result of the IKONOS satellite dataset in one embodiment of the present invention, where Figure 7 (a) is the MS image, Figure 7 (b) is the PAN image, Figure 7 (c) is the fused image by the ATWT method, Figure 7 (d) is the blue band image of the MS image, Figure 7 (e) is the blue band image of the ATWT image, Figure 7 (f) is the green band image of the MS image, Figure 7 (g) is the green band image of the ATWT image,Figure 7 (h) is the red - band image of the MS image, Figure 7 (i) is the red - band image of the ATWT image, Figure 7 (j) is the near - infrared - band image of the MS image, Figure 7 (k) is the near - infrared - band image of the ATWT image;

[0041] Figure 8 is the blue - band heat map of the ATWT image in one embodiment of the present invention;

[0042] Figure 9 is the green - band heat map of the ATWT image in one embodiment of the present invention;

[0043] Figure 10 is the red - band heat map of the ATWT image in one embodiment of the present invention;

[0044] Figure 11 is the near - infrared - band heat map of the ATWT image in one embodiment of the present invention. Detailed implementation manners

[0045] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] In one embodiment, as Figure 1 shown, Figure 1 is one of the schematic flowcharts of the no - reference spatial quality assessment method for remote - sensing image fusion provided by the embodiment of the present invention, and includes the following steps:

[0047] S101. Obtain a single - band high - resolution image according to the multi - band high - resolution image.

[0048] Preferably, in this embodiment, obtaining a single - band high - resolution image according to the multi - band high - resolution image is to perform weighted summation or selection on the high - resolution multi - band image to generate a single - band high - resolution image.

[0049] S102. Extract the texture features of the fused image and the single - band high - resolution image respectively to obtain the texture features of the fused image and the texture features of the single - band high - resolution image.

[0050] In this embodiment, using the texture image as the texture feature, a texture filter is used to extract the texture features of the fused image and the single - band high - resolution image.

[0051] It should be noted that the texture filter in this embodiment can be, but is not limited to, a bilateral texture filter, and the type of the texture filter can be specifically selected according to the actual situation.

[0052] S103. Perform mutual information measurement on the texture features of the fused image and the texture features of the single-band high-resolution image to obtain the feature mutual information between the fused image and the single-band high-resolution image.

[0053] Specifically, mutual information measurement is a useful information measurement in information theory. It refers to the correlation between two event sets. Mutual information measurement has the characteristics of no need for preprocessing, high automation degree, and strong robustness. The mutual information measurement in this embodiment is calculated in blocks. Within each image block of the block calculation, the mutual information measurement is calculated in a sliding window manner.

[0054] In this embodiment, the correlation between the texture features of the fused image and the texture features of the high-resolution remote sensing image is used as an evaluation index for the no-reference spatial quality. When calculating, it does not require the use of the scale-invariant hypothesis, breaking through the limitation of the reference image, making the evaluation of the no-reference spatial quality of remote sensing image fusion more in line with the actual situation.

[0055] It should be noted that using different texture filters will change the result of the feature mutual information.

[0056] S104. Perform no-reference evaluation on the spatial quality of the fused image according to the feature mutual information.

[0057] Specifically, the larger the value of the feature mutual information, the better the spatial quality of the fused image.

[0058] The no-reference spatial quality evaluation method for remote sensing image fusion in this embodiment evaluates the no-reference spatial quality of the fused image according to the correlation between the texture features of the fused image and the texture features of the high-resolution remote sensing image. It does not involve image degradation, excludes the interference of the degradation filter, and the evaluation result is more accurate.

[0059] In an optional embodiment, the fused image has multiple bands. Extracting the texture features of the fused image means extracting the texture features of each band of the fused image to obtain the texture features of each band of the fused image.

[0060] In one of the embodiments, as Figure 2 shown, Figure 2 is one of the flow diagrams of the no-reference spatial quality evaluation method for remote sensing image fusion provided by the embodiments of the present invention. This embodiment involves an optional way of how to perform mutual information measurement on the texture features of the fused image and the texture features of the single-band high-resolution image to obtain the feature mutual information between the fused image and the single-band high-resolution image. On the basis of the above embodiments, S103 includes the following steps:

[0061] S201. Measure the mutual information between the texture features of each band of the fused image and the texture features of the single-band high-resolution image to obtain the band feature mutual information between each band of the fused image and the single-band high-resolution image.

[0062] Specifically, the band feature mutual information refers to the feature mutual information obtained by measuring the mutual information between the single-band fused image and the single-band high-resolution image.

[0063] S202. Calculate the mean of all band feature mutual informations as the feature mutual information between the fused image and the single-band high-resolution image.

[0064] In one embodiment, define F i as the texture feature of the i-th band of the fused image, and F P as the texture feature of the single-band high-resolution image. Let the mutual information between the texture feature of the i-th band of the fused image and the texture feature of the single-band high-resolution image be

[0065]

[0066] where FMI k (F i , F P ) is the mutual information between F i and F P on the k-th image block, L is the total number of image blocks generated during the mutual information measurement of F i and F P , and k = 1, 2... L.

[0067] In one of the embodiments, the calculation formula for the mutual information FMI i (F P ) between F k (F i , F P ) on the k-th image block is

[0068]

[0069] where I j (F i , F P ) represents the mutual information value between F i and F P at the j-th sliding window, H j (F i ) represents the entropy of F i at the j-th sliding window, H j (F P ) represents the entropy of F P at the j-th sliding window, and n is the total number of sliding windows on each image block, and j = 1, 2... n.

[0070] Preferably, define D FMI as the feature mutual information between the fused image and the single-band high-resolution image, and calculate the mean value of the feature mutual information of all bands as

[0071]

[0072] In the formula, N is the total number of bands of the fused image.

[0073] In a specific embodiment, assume that the low-resolution multi-band image has N bands, denoted as {M1, M2,... M N}, and each band has E rows × F columns of pixels. The high-resolution multi-band image has S bands, denoted as {P1, P2,... P S}, and each band has H rows × W columns of pixels. The high-resolution multi-band image {P1, P2,... P S} can generate a single-band high-resolution image, denoted as P, with H rows × W columns of pixels through synthesis or selection. The fused image has N bands, denoted as and each band has H rows × W columns of pixels. As Figure 3 and Figure 4 shown, Figure 3 is the flow schematic diagram of the no-reference spatial quality assessment method for remote sensing image fusion in this embodiment, Figure 4 and the principle schematic diagram of the no-reference spatial quality assessment method for remote sensing image fusion. The no-reference spatial quality assessment method for remote sensing image fusion in this embodiment includes:

[0074] (1) Input the fused image into the texture filter to obtain the texture feature image F corresponding to the fused image. The texture feature images of N bands are denoted as {F1, F2,... F N}, where each band's texture feature image has H rows × W columns of pixels. Input the single-band high-resolution image P into the texture filter to obtain the texture feature image of the single-band high-resolution image P, denoted as F P , with H rows × W columns of pixels.

[0075] (2) Mutual information measurement of the texture features of the fused image and the texture features of the single-band high-resolution image.

[0076] Specifically, the mutual information measurement is calculated in blocks. The image (each band of the fused image or the single-band high-resolution image) is divided into non-overlapping image blocks with a size of R rows × R columns, where R < min{H, M}. When the image size cannot be exactly divisible by the block size, mirror filling is performed along the boundary to ensure that the filled image block size is R rows × R columns. Therefore, the entire image can generate ( The ceiling function is used to round up the number of image blocks.

[0077] Within each image block, the mutual information metric is calculated in a sliding window manner. The sliding window operates pixel by pixel, with a size of G×G (G is an odd number) and G < R. Before the sliding window operation, first pad the left and right edges of the current image block with ( The floor function is used to round down) columns of pixels, and then pad the top and bottom edges of the image block with rows of pixels. The padding is done by mirroring along the boundary, ultimately ensuring that there are n = R 2 corresponding sliding windows on the image block.

[0078] As Figure 5 shown, Figure 5 is a schematic diagram of the principle of the mutual information metric. Among them, F i (x, y) represents the pixel value of the texture feature image F i of the i-th band at the point (x, y). F P (z, w) is the pixel value of the image F P at the point (z, w). F i (x, y) and F P (z, w)'s joint probability density function is denoted as F i (x, y) and F P (z, w)'s correlation coefficient is denoted as The joint probability density function 's Fréchet upper and lower bounds are respectively and

[0079] When ,

[0080] When ,

[0081] F i (x, y)'s gradient marginal probability density function is denoted as and is defined as follows:

[0082]

[0083] Similarly, the gradient marginal probability density function of the texture feature image F P (z, w) is denoted as and is defined as follows:

[0084]

[0085] and The joint probability density function is denoted as

[0086] If is positive, that is then

[0087]

[0088] If is negative, that is then

[0089]

[0090] Among them,

[0091] F i The spatial information content contained in F P can be measured by mutual information. The calculation formula of mutual information is as follows:

[0092]

[0093] For the k-th image block, the value of the mean-normalized n mutual informations is the mutual information between F i and F P on this image block. Calculate FMI k (F i , F P ).

[0094] For the i-th band of the fused image, when calculating the mutual information between F P and F i there are image blocks generated. The feature mutual information i between F P is defined as the mean of the mutual informations of all image blocks on this band. Calculate according to formula (2).

[0095] It should be noted that the spatial quality measurement results of each band of the fused image are all in the range of [0,1]. The larger the value, the better the spatial quality assessment result.

[0096] The fused image has N bands, and N feature mutual information values are generated through mean operation 1 For the N-band images, the final spatial quality measurement result is the mean of the feature mutual informations generated by all bands (mean operation 2), which is D FMI , D​FMI Calculate according to Equation (3).

[0097] It should be noted that the output result of the overall spatial quality measurement (i.e., evaluation) of the fused image is D FMI . D FMI ranges from [0, 1]. The larger the value, the better the spatial quality evaluation result.

[0098] The no-reference spatial quality evaluation method for remote sensing image fusion in this embodiment evaluates the spatial quality by measuring the correlation between the texture features of the fused image and the texture features of the high-resolution remote sensing image. It does not require the assumption of scale invariance, breaks through the limitation of the reference image, makes the quality evaluation of remote sensing image fusion more in line with practical applications, and does not involve the degradation of the panchromatic image when measuring the correlation between the texture features of the fused image and the texture features of the high-resolution remote sensing image, excluding the interference of the degradation filter.

[0099] In a more specific embodiment, the no-reference spatial quality evaluation method for remote sensing image fusion is described by taking the spatial quality evaluation of the fusion of multi-spectral images and panchromatic images as an example. The method of this embodiment uses the IKONOS satellite images in the NBU database. Among them, the multi-spectral image MS has a total of 4 bands, namely red, blue, green, and infrared bands, and each band has 256 rows × 256 columns of pixels. The panchromatic image PAN is a single-band image with 1024 rows × 1024 columns of pixels, and the fused image has a total of 4 bands, which are respectively denoted as and each band has 1024 rows × 1024 columns of pixels.

[0100] The no-reference spatial quality evaluation method for remote sensing image fusion in this embodiment includes:

[0101] (1) Use a bilateral texture filter to extract texture features. Among them, the bilateral texture filter uses a 3×3 sliding window for calculation, and the number of iterations is 3.

[0102] Specifically, input the fused image with 4 bands and 1024 rows × 1024 columns into the texture filter. The difference between the output image and the fused image is the texture feature image of the fused image, denoted as {MF1, MF2, MF3, MF4}. This texture feature image has 4 bands, and each band has 1024 rows × 1024 columns of pixels.

[0103] Input the single-band panchromatic image PAN with 1024 rows × 1024 columns into the texture filter. The difference between the output image and the panchromatic image is the texture feature image of the panchromatic image, denoted as F PAN , and this texture feature image has 1024 rows × 1024 columns of pixels.

[0104] (2) Mutual information metric for fusing the texture features of the fused image and the texture features of the single-band high-resolution image

[0105] The image is divided into non-overlapping blocks with a block size of 32 rows × 32 columns, and the entire image generates image blocks.

[0106] On each image block, the mutual information metric is calculated in a sliding window manner. The sliding window is performed point by point, and the size of the sliding window is 3×3. Before the sliding window operation, first fill columns of pixels on the left and right edges of the image block, and then fill rows of pixels on the upper and lower edges of the image block. The filling is performed by mirroring with the boundary as the axis. Finally, there are n = 32×32 = 1024 corresponding sliding windows on each image block.

[0107] Assume MF i (MF i ∈{MF1, MF2, … MF4}) is the texture feature image of the i-th band of the fused image, and MF i (x,y) represents the pixel value of the MF i image at the point (x,y). F PAN is the texture feature image of the panchromatic image, and F PAN (z,w) is the pixel value of the F PAN image at the point (z,w). The joint probability density function of MF i (x,y) and F PAN (z,w) is MF i (x,y) and F PAN (z,w)'s correlation coefficient is denoted as Joint probability density function 's Fréchet upper and lower bounds are respectively

[0108] When time,

[0109] When time,

[0110] Then according to formula (4), calculate the gradient marginal probability density function of F i (x,y) According to formula (5), calculate the gradient marginal probability density function of F PAN (z,w) Calculate and joint probability density function Calculate the mutual information I(MF i , F PAN ) of each sliding window through formula (8), and calculate the mutual information FMI i between F P and F k on the k-th image patch through formula (2). i , F PAN ).

[0111] (3) Spatial quality measurement and evaluation of each band and the whole.

[0112] 1) Mean operation 1

[0113] For a certain band of the fused image, when calculating the mutual information between F PAN and MF i , L = 1024 image patches are generated. Calculate the spatial quality measurement results of each band of the fused image according to formula (1), denoted as

[0114] 2) Mean operation 2

[0115] The fused image has 4 bands, and 4 characteristic mutual information values are generated through mean operation 1 Obtain the overall spatial quality measurement output result D of the fused image according to formula (3) FMI .

[0116] In this embodiment, the image fusion result is as Figure 6 shown Figure 6 , which is the fused image generated by different fusion methods for the IKONOS dataset and can be used for subjective visual evaluation. Combining Figure 6 it can be obtained that, in terms of visual effect, generally speaking, the spatial quality of CS (component substitution) - type methods is higher than that of MRA (multi - resolution analysis) - type methods. For CS - type methods, the fused images generated by PCA and GS methods have the best spatial quality, followed by IHS, BT, and BDSD methods, while the fused image generated by the PRACS method is relatively blurred and has the worst spatial quality. For MRA - type methods, the fused images generated by ATWT and AWLP methods have general quality, while the fused images generated by ATWT - M2 and ATWT - M3 methods are very blurred and have poor spatial quality.

[0117] As shown in Table 1, Table 1 shows the numerical results and ranking results of the spatial distortion index for spatial distortion evaluation of the fused images generated by different fusion methods for the IKONOS dataset. The MRA-based CS-like fusion method gives better spatial quality assessment results. In particular, this metric shows that the spatial quality assessment results of the two fused images with obvious spatial blur, ATWT-M2 and ATWT-M3, are good, while the spatial quality assessment results of the fused images generated by the relatively effective PCA and GS methods are poor, showing a large difference from the subjective visual assessment. Compared with The effect has been improved to some extent, but this metric indicates that the fused image generated by the PCA method is worse than the fused images generated by the ATWT and AWLP methods, which is contrary to the subjective visual assessment method. D FMI The performance of the metric is better than the metric, and is similar to the visual assessment. Although D FMI and have a similar ranking, compared with D FMI the numerical changes in the assessment results for different fusion methods are obvious, and it can better distinguish the differences in fusion performance than can.

[0118] Among them, the regression-based QNR calculates the spatial distortion metric based on the spatial matching degree between the panchromatic image and the fused image. is currently considered the most accurate spatial quality assessment metric. This metric models the panchromatic image as a linear combination of the bands of the fused image, but the rationality of its assumption cannot be explained.

[0119] Table 1 Numerical results of spatial quality assessment of fusion results for the IKONOS satellite dataset

[0120]

[0121] As shown in Table 2, Table 2 shows the Kendall correlation coefficients between the spatial distortion index and the spatial distortion assessment methods for the fused images generated by different fusion methods on the IKONOS dataset. Among them, the metric D FMI and and have Kendall correlation coefficients of -0.6444, 0.3778, and 0.8667 respectively, indicating that D FMI is very close to , that is, D FMI is highly accurate and consistent with the subjective visual assessment results.

[0122] Table 2 Kendall correlation coefficient table between spatial quality assessment methods for fusion results of the IKONOS satellite dataset

[0123]

[0124] As Figure 7 , Figure 8 , Figure 9 , Figure 10 and Figure 11 shown, Figure 7 are the images of each band of the ATWT fusion result of the IKONOS satellite dataset in this embodiment, Figure 8 , Figure 9 , Figure 10 and Figure 11 are the heat maps corresponding to the blue, green, red, and infrared band images. Visually, there is less detail injected into the infrared band of the fused image, while there is more detail injected into the blue, green, and red bands. Taking the roof edge as an example, there are obvious artifacts in the blue and infrared bands of the fused image, and the characteristic mutual information value at the roof edge is smaller on the heat map. While there are no artifacts in the green and red bands, and the characteristic mutual information value at the roof edge is larger on the heat map.

[0125] It should be noted that the closer the value of each point on the heat map is to 1, the more detail the panchromatic image injects into the corresponding band of the fused image. The heat map reflects the mutual information calculated by the corresponding sliding window at each point, and each value is the mutual information value. Using the heat map, it is possible to directly see how much detail is injected at a certain point in a certain band of the fused image. The more detail is injected, the closer it is to 1, and the whiter the color of the heat map at that point. The color of the points on the heat map ranges from white to black, and the corresponding point values gradually decrease from 1 to 0 during the change. In this embodiment of Figure 8 , Figure 9 , Figure 10 and Figure 11 , the darker the gray color, the closer the color of the point on the heat map is to black, the smaller the point value, and the less detail is injected at the point, and the smaller the characteristic mutual information value.

[0126] As shown in Table 3, Table 3 shows the characteristic mutual information values of the blue, green, red, and infrared bands of the fused image generated by the ATWT fusion method of the IKONOS dataset and the single-band high-resolution image, which are 0.4238, 0.4450, 0.4365, and 0.3381 respectively. The fusion effects of the blue, green, and red bands are better than those of the near-infrared band, which is consistent with our visual evaluation. Among them, is the characteristic mutual information value of the blue band and the single-band high-resolution image, is the characteristic mutual information value of the green band and the single-band high-resolution image, is the characteristic mutual information value of the red band and the single-band high-resolution image, is the characteristic mutual information value of the infrared band and the single-band high-resolution image.

[0127] Table 3 Spatial quality assessment results of each band of the ATWT fusion result of the IKONOS satellite dataset

[0128]

[0129] According to Figure 6 , Figure 7 , Figure 8 , Figure 9 , Figure 10 and Figure 11 and the analysis of Table 1, Table 2 and Table 3, when the spatial quality of the remote sensing fusion image of the present invention is evaluated without reference and without relying on subjective evaluation, a relatively accurate evaluation effect can still be achieved.

[0130] Based on the same inventive concept, the embodiment of the present application also provides a device for evaluating the no-reference spatial quality of remote sensing image fusion for implementing the above-mentioned no-reference spatial quality evaluation method for remote sensing image fusion. The implementation solutions provided by this device to solve problems are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more of the following embodiments of the device for evaluating the no-reference spatial quality of remote sensing image fusion can refer to the limitations on the no-reference spatial quality evaluation method for remote sensing image fusion in the above text, and will not be elaborated here.

[0131] In one embodiment, a device for evaluating the no-reference spatial quality of remote sensing image fusion includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the no-reference spatial quality evaluation method for remote sensing image fusion in any one of the above embodiments are implemented.

[0132] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are displayed in sequence according to the arrows, these steps do not necessarily need to be executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily need to be executed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily need to be sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0133] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0134] The above-described embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the scope of the patent for the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the appended claims.

Claims

1. A no-reference spatial quality assessment method for remote sensing image fusion, characterized in that Including: Obtaining a single-band high-resolution image from a multi-band high-resolution image; Performing texture feature extraction on the fused image and the single-band high-resolution image respectively to obtain the texture features of the fused image and the texture features of the single-band high-resolution image; Performing mutual information measurement on the texture features of the fused image and the texture features of the single-band high-resolution image to obtain the feature mutual information between the fused image and the single-band high-resolution image; Performing a no-reference evaluation on the spatial quality of the fused image according to the feature mutual information; The fused image has multiple bands, and performing texture feature extraction on the fused image means performing texture feature extraction on each band of the fused image to obtain the texture features of each band of the fused image; Performing mutual information measurement on the texture features of the fused image and the texture features of the single-band high-resolution image to obtain the feature mutual information between the fused image and the single-band high-resolution image includes: Performing mutual information measurement on the texture features of each band of the fused image and the texture features of the single-band high-resolution image to obtain the band feature mutual information between each band of the fused image and the single-band high-resolution image; Calculating the mean of all the band feature mutual informations as the feature mutual information between the fused image and the single-band high-resolution image.

2. The no-reference spatial quality assessment method for remote sensing image fusion according to claim 1, wherein Obtaining a single-band high-resolution image from a multi-band high-resolution image means performing weighted summation or selection on the high-resolution multi-band image to generate a single-band high-resolution image.

3. The no-reference spatial quality assessment method for remote sensing image fusion according to claim 2, characterized in that The mutual information measurement is calculated in blocks, and within each image block of the block calculation, the mutual information measurement is calculated in a sliding window manner.

4. The no-reference spatial quality assessment method for remote sensing image fusion according to claim 3, wherein Define F i as the texture feature of the i-th band of the fused image, and F P as the texture feature of the single-band high-resolution image. is the mutual information of band features. The mutual information measurement of the texture feature of the i-th band of the fused image and the texture feature of the single-band high-resolution image is where FMI k (F i , F P ) is the mutual information between F i and F P on the k-th image block, L is the total number of image blocks generated when mutual information measurement is performed between F i and F P , and k = 1, 2... L.

5. The no-reference spatial quality assessment method for remote sensing image fusion according to claim 4, characterized in that F on the k-th image block i and F P The mutual information FMI k (F i , F P ) is calculated as Where, I j (F i , F P ) represents the mutual information value of F i and F P at the j-th sliding window, H j (F i ) represents the entropy of F i at the j-th sliding window, H j (F P ) represents the entropy of F P at the j-th sliding window, n is the total number of sliding windows on each image block, and j = 1, 2... n.

6. The no-reference spatial quality assessment method for remote sensing image fusion according to claim 5, characterized in that Definition D FMI To fuse the feature mutual information of the fused image and the single-band high-resolution image, the mean value of the feature mutual information of all bands is calculated as In the formula, N is the total number of bands of the fused image.

7. The no-reference spatial quality assessment method for remote sensing image fusion according to any one of claims 1 to 6, characterized in that, Using a texture filter to perform texture feature extraction on the fused image and the single-band high-resolution image.

8. A no-reference spatial quality assessment device for remote sensing image fusion, characterized in that, Including a memory and a processor, the memory stores a computer program, and it is characterized in that when the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images