Image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet

Abstract

The present invention provides an image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet, which can process and analyze remote sensing image data from different sensors, thereby obtaining characteristic information of remote sensing images from different sources. The processed image is usually richer and more comprehensive than the original remote sensing image, and is more in line with the visual characteristics of the human eye or the information processing method of the machine, so as to facilitate subsequent processing and better meet application requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to an image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet. Background Art

[0002] At present, remote sensing images play an important role in urban planning, environmental monitoring, military defense and other fields. Therefore, the fusion of remote sensing images has attracted more and more researchers in the world and has become a hot topic in the field. Due to the different incident wavelengths of remote sensing images in the same area, multi-band remote sensing images have significant differences. High-frequency band remote sensing images can provide an overall view of the scene, which is similar to optical imaging, while low-frequency band remote sensing images are relatively dim but have deeper penetration. Fusion of remote sensing images of different frequency bands can enhance the recognition and observation of ground objects. Pixel-level image fusion algorithms can be mainly divided into two categories: based on multi-scale transform (MST), based on sparse representation (SR), and based on deep learning. The human eye processes the received visual information at different scales and is very sensitive to detail information. The fusion algorithm based on multi-scale transform (MST) decomposes the source image into different scales and resolutions to obtain low-frequency components containing image energy information and high-frequency components mainly containing image detail information, and performs image fusion on the low-frequency and high-frequency components according to different fusion rules. Finally, the fused image is obtained through multi-scale reconstruction. In other words, the way the MST-based algorithm processes image information is highly similar to the visual perception of the human visual system (HVS). However, the fusion method based on multi-scale decomposition and reconstruction has its own inherent defects. In the process of decomposition and reconstruction, some useful information will inevitably be lost, resulting in the image fusion performance not being optimal. Algorithms based on sparse representation (SR) are mainly classified into two categories: fixed dictionary and dictionary learning. However, fixed dictionaries rely on specific transformations, and only a limited number of transformations can meet the diversity of image content and applications. Not only that, fixed dictionaries also have the disadvantages of being time-consuming and computationally complex. The method based on dictionary learning will have the problem of color distortion.

[0003] At present, deep learning is also widely used in the field of image fusion. According to research, the fusion algorithm based on convolutional neural network (CNN) has greater research potential than the traditional algorithm and is the most suitable architecture for image fusion. However, the algorithm based on deep learning not only has a large demand for data sets, but also has complex and time-consuming calculations and cumbersome parameter settings.

[0004] In order to better improve the fusion performance of the multi-band remote sensing image fusion algorithm, the present invention proposes an image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet. Summary of the invention

[0005] In view of this, an object of the present invention is to provide an image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet to at least solve the above problems.

[0006] The technical solution adopted by the present invention is as follows:

[0007] The image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet includes the following steps:

[0008] S1: Decompose the multi-source remote sensing image by non-subsampled shearlet transform to obtain high-frequency component images;

[0009] S2: The multi-source remote sensing image is decomposed into energy layer multi-source remote sensing image and structure layer multi-source remote sensing image by using the joint bilateral filtering model;

[0010] S3: Fusing the multi-source remote sensing images of the energy layer to form an energy layer fusion image;

[0011] S4: fusing the high-frequency component images to form a high-frequency component fused image;

[0012] S5: taking the energy layer fusion image as the low-frequency component and the high-frequency component fusion image and performing non-subsampled shearlet inverse transform to obtain the final fusion image;

[0013] S6: Evaluate the performance of the image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet using objective evaluation indicators.

[0014] Furthermore, in step S1, when performing non-subsampled shearlet transform decomposition on the multi-source remote sensing image, the maximum rule is used to obtain a high-frequency component image.

[0015] Furthermore, in step S3, the Abs-Max criterion is used when fusing the energy layer multi-source remote sensing images, and the absolute value of the pixel of the energy layer multi-source remote sensing image is used as the activity of the pixel.

[0016] Furthermore, in step S2, before performing dual-scale decomposition on the multi-source remote sensing image through the joint bilateral filtering model, the multi-source remote sensing image is smoothed and Gaussian filtered. The calculation formula for the smoothing is as follows:

[0017] R σ =G σ *I

[0018]

[0019] Where I is the input multi-source remote sensing image, G σ (x,y) represents the variance σ 2 Gaussian filter, R σThis is the smoothing output with a standard deviation of σ. 2 Indicates the scale; a weighted average Gaussian filter is used to obtain the energy layer multi-source remote sensing image. The calculation formula is as follows:

[0020]

[0021]

[0022] Where I and G represent the input image and the energy layer multi-source remote sensing image after filtering, respectively. j represents normalization, G(j) represents the global blur map output after filtering the j-th pixel of the input image, N(j) represents the local area of ​​the j-th pixel, i represents a pixel in its local area, represents the smoothing parameter.

[0023] Furthermore, when smoothing multi-source remote sensing images, joint bilateral filtering is used to prevent structural damage to the multi-source remote sensing images during the smoothing process. The calculation formula of the joint bilateral filtering is as follows:

[0024]

[0025] where s p represents the filtered output value of the multi-source remote sensing image I at pixel p, where I represents the multi-source remote sensing image, represents the introduced guided graph, p and q are both pixel indices, q is the pixel in the multi-source remote sensing image window Ω centered on p, G s represents the kernel function in the spatial domain, G r represents the weight distribution function of the pixel range domain, represents the normalized term, which is calculated as follows:

[0026]

[0027] The calculation formula of the result after joint bilateral filtering is obtained through the above formula:

[0028]

[0029]

[0030] Where J(j) represents the result image after the jth pixel of the input image is subjected to joint bilateral filtering. and are smoothing parameters, g d and g r Represent the spatial distance function and intensity range function respectively, and calculate g d and g r The weight is calculated as follows:

[0031]

[0032]

[0033] Furthermore, the calculation formula of the Abs-Max criterion is as follows:

[0034]

[0035]

[0036] Where FE represents the fused image of the energy layer, AE and BE represent the energy layers of two multi-source remote sensing images respectively, map represents the fusion coefficient, and (x, y) represents the position in the pixel.

[0037] Furthermore, the calculation formula for obtaining the high frequency component using the maximum rule is as follows:

[0038]

[0039] Where FH represents the fused image of the high-frequency components, AH and BH represent the high-frequency components obtained by non-subsampled shearlet transform of two multi-source remote sensing images, i represents the decomposition layer, and (x, y) represents the position in the pixel.

[0040] Furthermore, in step S6, the objective indicator evaluation includes information entropy, standard deviation, spatial frequency, feature mutual information and gradient-based metrics.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention can process and analyze remote sensing data from different sensors, thereby obtaining characteristic information of remote sensing images from different sources. The image is usually richer and more comprehensive than the original image, and more in line with the visual characteristics of the human eye or the information processing method of the machine, so as to facilitate the subsequent processing of the image and better meet the application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only preferred embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 It is a flowchart of an image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet provided by an embodiment of the present invention.

[0045] Figure 2 Schematic diagram of an image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet provided by an embodiment of the present invention

[0046] Figure 3 Schematic diagram of fusion of 9 pairs of multi-source remote sensing images based on the image fusion algorithm of joint bilateral filtering and non-subsampled shearlet provided by the embodiment of the present invention DETAILED DESCRIPTION

[0047] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The enumerated embodiments are only used to explain the present invention and are not used to limit the scope of the present invention.

[0048] Reference Figure 1 , Figure 2 and Figure 3 The present invention provides an image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet, comprising the following steps:

[0049] S1: Decompose the multi-source remote sensing image by non-subsampled shearlet transform to obtain a high-frequency component image; illustratively, first perform pyramid decomposition (NSP) on the multi-source remote sensing image by non-subsampled Laplacian pyramid filter bank (NSPFB) to obtain its low-frequency component and high-frequency component. There are four levels of decomposition, and the final decomposition result is 1 low-frequency component and 48 high-frequency components of different scales. The first level decomposition produces 8 sub-bands, the second level decomposition produces 8 sub-bands, the third level decomposition produces 16 sub-bands, and the fourth level decomposition produces 16 sub-bands.

[0050] S2: Perform a dual-scale decomposition of the multi-source remote sensing image into an energy layer multi-source remote sensing image and a structure layer multi-source remote sensing image through a joint bilateral filtering model; illustratively, in order to better distinguish the grayscale information and gradient structure of the image, the multi-source remote sensing image is decomposed at a dual scale.

[0051] S3: Fusing the multi-source remote sensing images of the energy layer to form an energy layer fusion image;

[0052] S4: fusing the high-frequency component images to form a high-frequency component fused image;

[0053] S5: taking the energy layer fusion image as the low-frequency component and the high-frequency component fusion image and performing non-subsampled shearlet inverse transform to obtain the final fusion image;

[0054] S6: Evaluate the performance of the image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet using objective evaluation indicators. For example, the evaluation of the algorithm performance by objective evaluation indicators can better verify the superior performance of the algorithm.

[0055] Specifically, in step S1, when performing non-subsampled shearlet transform decomposition on the multi-source remote sensing image, the maximum rule is used to obtain a high-frequency component image.

[0056] Specifically, in step S3, the Abs-Max criterion is used when fusing the energy layer multi-source remote sensing images, and the absolute value of the pixel of the energy layer multi-source remote sensing image is used as the activity of the pixel.

[0057] Specifically, in step S2, before performing dual-scale decomposition on the multi-source remote sensing image through the joint bilateral filtering model, the multi-source remote sensing image is smoothed and Gaussian filtered. The calculation formula for the smoothing process is as follows:

[0058] R σ =G σ *I

[0059]

[0060] Where I is the input multi-source remote sensing image, G σ (x,y) represents the variance σ 2 Gaussian filter, R σ This is the smoothing output with a standard deviation of σ. 2 Indicates the scale; a weighted average Gaussian filter is used to obtain the energy layer multi-source remote sensing image. The calculation formula is as follows:

[0061]

[0062]

[0063] Where I and G represent the input image and the energy layer multi-source remote sensing image after filtering, respectively. j represents normalization, G(j) represents the global blur map output after filtering the j-th pixel of the input image, N(j) represents the local area of ​​the j-th pixel, i represents a pixel in its local area, represents the smoothing parameter.

[0064] Specifically, when smoothing multi-source remote sensing images, joint bilateral filtering is used to prevent structural damage to the multi-source remote sensing images during the smoothing process. The calculation formula of the joint bilateral filtering is as follows:

[0065]

[0066] where s p represents the filtered output value of the multi-source remote sensing image I at pixel p, where I represents the multi-source remote sensing image, represents the introduced guided graph, p and q are both pixel indices, q is the pixel in the multi-source remote sensing image window Ω centered on p, Gs represents the kernel function in the spatial domain, G r represents the weight distribution function of the pixel range domain, represents the normalized term, which is calculated as follows:

[0067]

[0068] The calculation formula of the result after joint bilateral filtering is obtained through the above formula:

[0069]

[0070]

[0071] Where J(j) represents the result image after the jth pixel of the input image is subjected to joint bilateral filtering. and are smoothing parameters, g d and g r Represent the spatial distance function and intensity range function respectively, and calculate g d and g r The weight is calculated as follows:

[0072]

[0073]

[0074] Specifically, the calculation formula of the Abs-Max criterion is as follows:

[0075]

[0076]

[0077] Where FE represents the fused image of the energy layer, AE and BE represent the energy layers of two multi-source remote sensing images respectively, map represents the fusion coefficient, and (x, y) represents the position in the pixel.

[0078] Where FE represents the fused image of the energy layer, AE and BE represent the energy layers of two multi-source remote sensing images respectively, map represents the fusion coefficient, and (x, y) represents the position in the pixel.

[0079] Specifically, the calculation formula for obtaining the high-frequency component using the maximum rule is as follows:

[0080]

[0081] Where FH represents the fused image of the high-frequency components, AH and BH represent the high-frequency components obtained by non-subsampled shearlet transform of two multi-source remote sensing images, i represents the decomposition layer, and (x, y) represents the position in the pixel.

[0082] Specifically, in step S6, the objective index evaluation includes information entropy, standard deviation, spatial frequency, feature mutual information and gradient-based metrics. For example, a public multi-source remote sensing image dataset provided by Durga Prasad Bavirisetti is used as the original dataset, and 9 pairs of brain medical images are selected from the original dataset for experiments. These multi-source remote sensing images are from different imaging devices, have been registered, and have the same size of 256×256.

[0083] Evaluation indicators

[0084] Five objective evaluation indicators are used to quantitatively evaluate the fusion performance of the algorithm, namely information entropy (EN), standard deviation (SD), spatial frequency (SF), feature mutual information (FMI) and gradient-based metric (Q AB / F ). These five indicators are positive indicators. The higher the value, the better the fusion performance.

[0085] Information Entropy (EN)

[0086] Information entropy reflects the richness of image information and is a positive indicator. The higher the value, the richer the image information. Its calculation formula is as follows:

[0087] Where L and p(x i ) represent the grayscale levels and grayscale distribution of the multi-source remote sensing images respectively.

[0088] Standard Deviation (SD)

[0089] SD is often used to measure the discreteness of a set of values. The larger the value, the more discrete the grayscale of the multi-source remote sensing image is, that is, the higher the image quality is. The calculation formula is as follows:

[0090]

[0091] Where μ is the grayscale mean of the multi-source smoke image, and M and N represent the width and height of the multi-source remote sensing image, respectively.

[0092] Spatial Frequency (SF)

[0093] SF is proportional to the activity of the grayscale space of the multi-source remote sensing image and reflects the rate of change of the grayscale of the multi-source remote sensing image. The calculation formula is as follows:

[0094] Where RF and CP represent the row frequency and column frequency of the multi-source remote sensing image, respectively. The calculation process is as follows:

[0095]

[0096]

[0097] Feature Mutual Information (FMI)

[0098] FMI is an evaluation index of the final image fusion quality based on mutual information (MI). The larger the FMI, the better the quality of the final fused image. The calculation formula is as follows:

[0099]

[0100] Among them, H F is the information entropy of the multi-source remote sensing image F, H A and H B Similarly, I FA and I FB They represent the feature information from multi-source remote sensing images A and B contained in the multi-source remote sensing image F. The calculation process is as follows:

[0101]

[0102]

[0103] Gradient-based Q AB / F

[0104] Q AB / F It is a non-reference quality assessment indicator that uses local metrics to estimate the degree of representation of the input multi-source remote sensing images in the final fused image. The higher the value, the better the quality of the final fused image.

[0105] Assume that g A (i,j) and α A (i, j) is the edge strength and direction of the multi-source remote sensing image A, then the relative strength G between the multi-source remote sensing image A and the final fusion image F AF and direction value Δ AF The calculation formula is

[0106]

[0107]

[0108] Then the edge strength and direction retention Then

[0109]

[0110]

[0111] where Γ g , Γ α , κ g , κ αThe value of determines the shape of the Sigmoid function. Therefore, the edge information transfer value Q between the multi-source remote sensing image A and the final fusion image F is AF (i,j) is

[0112] Q AF (i,j)=Q g AF (i,j)Q α AF (i,j)

[0113] The fusion metric Q AB / F It is through Q AF The weighted average of (i,j) is defined as follows:

[0114]

[0115] where w A (i,j) and w B (i,j) is the weighting coefficient.

[0116] The weighting coefficient w(i,j) can be obtained by the edge strength g A (i,j) is calculated, and the calculation process is as follows, L

[0117] w(i,j)=[g(i,j)] L

[0118] Where L is a positive number.

[0119] Table 1 lists the objective evaluation indicators of different fusion algorithms. These indicators are positive indicators. The larger the value, the better the fusion performance. Each final image fusion metric in the table is the average of the nine source image pairs of different algorithms, and the value in bold is the best score under this indicator.

[0120] Table 1

[0121]

[0122]

[0123] From the table, we know that in addition to the indicator Q AB / F Except for EN, the algorithm proposed in this paper is significantly better than NSST-PAPCNN in all indicators. AB / F In terms of EN, our method is very close to NSST-PAPCNN. From the overall comparison of objective evaluation indicators, the fusion performance of the method proposed in this paper is better than that of other algorithms.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. Image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet, characterized by: The following steps are involved: S1: Decompose the multi-source remote sensing image by non-subsampled shearlet transform to obtain high-frequency component images; S2: The multi-source remote sensing image is decomposed into energy layer multi-source remote sensing image and structure layer multi-source remote sensing image by using the joint bilateral filtering model; S3: Fusing the multi-source remote sensing images of the energy layer to form an energy layer fusion image; S4: fusing the high-frequency component images to form a high-frequency component fused image; S5: taking the energy layer fusion image as the low-frequency component and the high-frequency component fusion image and performing non-subsampled shearlet inverse transform to obtain the final fusion image; S6: Evaluate the performance of the image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet using objective evaluation indicators.

2. The image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet according to claim 1 is characterized in that: In step S1, when performing non-subsampled shearlet transform decomposition on a multi-source remote sensing image, a maximum rule is used to obtain a high-frequency component image.

3. The image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet according to claim 1, characterized in that: In step S3, the Abs-Max criterion is used when fusing the energy layer multi-source remote sensing images, and the absolute value of the pixel of the energy layer multi-source remote sensing image is used as the activity of the pixel.

4. The image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet according to claim 1, characterized in that: In step S2, before performing dual-scale decomposition on the multi-source remote sensing image through the joint bilateral filtering model, the multi-source remote sensing image is smoothed and Gaussian filtered. The calculation formula for the smoothing process is as follows: R σ =G σ *I Where I is the input multi-source remote sensing image, G σ (x,y) represents the variance σ 2 Gaussian filter, R σ This is the smoothing output with a standard deviation of σ. 2 Indicates the scale; a weighted average Gaussian filter is used to obtain the energy layer multi-source remote sensing image. The calculation formula is as follows: Where I and G represent the input image and the energy layer multi-source remote sensing image after filtering, respectively. j represents normalization, G(j) represents the global blur map output after filtering the j-th pixel of the input image, N(j) represents the local area of ​​the j-th pixel, i represents a pixel in its local area, represents the smoothing parameter.

5. The image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet according to claim 4, characterized in that: When smoothing multi-source remote sensing images, joint bilateral filtering is used to prevent structural damage to multi-source remote sensing images during the smoothing process. The calculation formula of joint bilateral filtering is as follows: where s p represents the filtered output value of the multi-source remote sensing image I at pixel p, where I represents the multi-source remote sensing image, represents the introduced guided graph, p and q are both pixel indices, q is the pixel in the multi-source remote sensing image window Ω centered on p, G s represents the kernel function in the spatial domain, G r represents the weight distribution function of the pixel range domain, represents the normalized term, which is calculated as follows: The calculation formula of the result after joint bilateral filtering is obtained through the above formula: Where J(j) represents the result image after the jth pixel of the input image is subjected to joint bilateral filtering. and are smoothing parameters, g d and g r Represent the spatial distance function and intensity range function respectively, and calculate g d and g r The weight is calculated as follows:

6. The image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet according to claim 3, characterized in that: The calculation formula of Abs-Max criterion is as follows: Where FE represents the fused image of the energy layer, AE and BE represent the energy layers of two multi-source remote sensing images respectively, map represents the fusion coefficient, and (x, y) represents the position in the pixel.

7. The image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet according to claim 2, characterized in that: The calculation formula for obtaining the high frequency component using the maximum rule is as follows: Where FH represents the fused image of the high-frequency components, AH and BH represent the high-frequency components obtained by non-subsampled shearlet transform of two multi-source remote sensing images, i represents the decomposition layer, and (x, y) represents the position in the pixel.

8. The image fusion algorithm based on joint bilateral filtering and non-subsampled shearlet according to claim 1, characterized in that: In step S6, the objective evaluation indicators include information entropy, standard deviation, spatial frequency, feature mutual information and gradient-based metrics.

Images