Multi-scale structural feature enhanced multi-modal image fusion method in NSST domain

Through the combination of improved Laplace fusion strategy and ROEWA operator, the edge structure information of SAR images is extracted and enhanced, and the information distortion problem in the fusion of SAR images and full-color optical images in the prior art is solved, and the image is significantly improved The structure and retention ability of spectral information suppresses spot noise and improves the interpretability of the image.

CN114119440BActive Publication Date: 2024-12-03XIAN TECH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111360466.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-17
Publication Date
2024-12-03
Estimated Expiration
2041-11-17

AI Technical Summary

Technical Problem

When the prior art fuses SAR images with full-color optical images, spectral differences cannot be fully considered in low-frequency information fusion, resulting in information distortion; when the geometric structural characteristics of SAR images cannot be effectively mined in high-frequency component fusion, resulting in the fusion image. There is spot noise, which loses the geometric structure information of the image.

Method used

Multimode image fusion method in the NSST domain enhanced by multi-scale structural features is adopted to extract edge structure information of SAR images through improved Laplace fusion strategy and exponential weighted mean ratio operator (ROEWA), and inject high-frequency fusion through guide filtering. Components, combined with NSST inverse changes, ensure the retention of low-frequency details and the improvement of high-frequency structural characteristics.

Benefits of technology

It effectively suppresses spot noise, improves the structural characteristics and spectral information characterization ability of the fused image, significantly improves the image geometric structure and retention of spectral information, and improves the interpretability of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114119440B_ABST
    Figure CN114119440B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-mode image fusion method in the NSST domain with multi-scale structural feature enhancement, which mainly solves the problem of geometric structure information loss and spectral information distortion in the fusion results of SAR images and panchromatic optical images. The implementation steps are: 1) inputting SAR images and panchromatic optical images; 2) performing multi-scale decomposition on the two input images to obtain high-frequency components and low-frequency components respectively; 3) adopting an improved Laplace operator fusion strategy to fuse the low-frequency components of the two images; 4) adopting an absolute value maximum fusion strategy to fuse the high-frequency components of the two images; 5) using an exponential weighted mean ratio operator to extract edge structure information in the SAR image and injecting it into the high-frequency decomposition component of the SAR image through guided filtering; 6) performing inverse NSST changes on the low-frequency components and high-frequency components of the fused image to obtain the final fused image. The structural characteristics of the fused image of the present invention are improved by 2.4%, and the spectral information is improved by 4.3%, which effectively suppresses the influence of speckle noise on the fusion result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a multi-scale structural feature enhanced NSST domain multi-modal image fusion method. Background Art

[0002] With the rapid development of remote sensing satellites in orbit at home and abroad, a large amount of multi-mode remote sensing images have been accumulated. Among them, SAR data and panchromatic optical image data are the two main types of data on satellite remote sensing platforms. Therefore, the fusion of the two image data is becoming more and more important.

[0003] The fusion of SAR images and panchromatic optical images can combine the advantages of the two images, maintaining the spectral information of panchromatic optical images, highlighting the expressiveness of SAR images on ground details, and improving the interpretability of images. For the fusion of SAR images and panchromatic optical images, the methods currently proposed mainly include image fusion methods based on Laplace pyramid, image fusion based on wavelet transform, and variational multi-scale methods. Although these methods have mature algorithms and fast calculation speed, they fail to fully consider the spectral differences of the two images in the fusion of low-frequency information, which easily leads to distortion of the spectral information of the two images in the fusion results; in the fusion of high-frequency components, they fail to fully explore the important geometric structure characteristics of SAR images, so the fused images are often full of speckle noise, which seriously loses the geometric structure information of the images. Among them, the disclosed patent: An image fusion method based on GFRW and ISCM (CN202010420380) designed different fusion strategies for low-frequency and high-frequency components in multi-scale domains, but the SML in the low-frequency strategy failed to fully consider the influence of the diagonal direction and could not achieve adaptive threshold in the final fusion decision mapping, which is bound to lead to the loss of low-frequency energy; although the pulse coupled neural network model is used in the high-frequency fusion strategy, it is only for the case where the background noise interference of medical images is small, and it is impossible to extract useful structural information in the case of SAR images. Summary of the invention

[0004] The present invention proposes a multi-mode image fusion method in the NSST domain with multi-scale structural feature enhancement, which aims to solve the problems of loss of important geometric structure information and lack of prominence of spectral information in the fusion results of SAR images and panchromatic optical images.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] A multi-scale structural feature enhanced NSST domain multi-modal image fusion method comprises the following steps:

[0007] Step 1: Input two images, one of which is a SAR image A and the other is a panchromatic optical image B;

[0008] Step 2: Perform NSST decomposition on the two input images to obtain their decomposition bands respectively. and The symbol represents the high-frequency band coefficient of A in the decomposition layer l and direction k, and the symbol L A represents the low-frequency band coefficient of A. Similarly, the symbol represents the high-band coefficient of B in the decomposition layer l and direction k, while the symbol L B Represents the low-frequency band coefficient of B.

[0009] Step 3: An improved Laplace fusion strategy is used to fuse the low-frequency components of the SAR image and the panchromatic optical image to obtain the fused component F1. Two activity level metrics based on local energy and WLE improved ML are designed in the fusion strategy to maintain the low-frequency energy and improve the detail extraction capability.

[0010] Step 4: Fuse the high-frequency components of the SAR image and the panchromatic optical image to obtain the fused component F2:

[0011] The absolute value fusion strategy is used to fusion the high frequency coefficients of SAR images. and panchromatic optical image high frequency coefficient Fusion is performed to obtain the fused high-frequency coefficients:

[0012]

[0013] Step 5: Use the exponentially weighted mean ratio operator (ROEWA) to extract edge structure information F in the SAR image sar And inject it into the high-frequency fusion component F2 through guided filtering (GF) to obtain a new fusion component F3;

[0014] Step 6: Perform NSST inverse transformation on the fused structural component F1 and texture component F3 to obtain the fusion result F.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] (1) The present invention improves a Laplace fusion strategy when fusing low-frequency components, so that the detail information existing in the low-frequency components of the NSST multi-level decomposition can be completely retained, and at the same time, the representation ability of the spectral information of the fusion result is more obvious without information distortion; compared with the original Laplace matrix (ML), EML takes into account the influence of diagonal coefficients and makes full use of neighboring information.

[0017] (2) The present invention introduces a structural feature enhancement method in the high-frequency component fusion. The ROEWA operator is used to extract the ground object structure information of the SAR image separately and use it as a guide image to perform guided filtering on the high-frequency fusion component, so as to enhance the high-frequency structural characteristics of the SAR image and remove the influence of speckle noise, thereby greatly improving the high-frequency structural information of the fused image.

[0018] (3) The structural characteristics of the fused image are improved by 2.4%, and the spectral information is improved by 4.3%, which effectively suppresses the influence of speckle noise on the fusion result. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is an overall flow chart of a multi-modal image fusion method in the NSST domain with multi-scale structural feature enhancement of the present invention;

[0020] Figure 2 A diagram showing the integration effect of the urban area of ​​the method of the present invention;

[0021] Figure 3 This is a diagram showing the airport area integration effect of the method of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] According to an embodiment of the present invention, a multi-scale structural feature enhanced NSST domain multi-modal image fusion method is provided. Figure 1 As shown, according to an embodiment of the present invention, a multi-scale structural feature enhanced NSST domain multi-modal image fusion method includes the following steps:

[0024] Step 1: Input two images, one of which is a SAR image A and the other is a panchromatic optical image B;

[0025] Step 2: Perform L-level multi-scale NSST decomposition on the two input images to obtain their decomposition bands respectively. and The symbol represents the high-frequency band coefficient of A in the decomposition layer l and direction k, while the symbol L A represents the low-frequency band coefficient of A. Similarly, the symbol represents the high-band coefficient of B in the decomposition layer l and direction k, while the symbol L BIndicates the low frequency band coefficient of B. In the present invention, the number of decomposition layers is set to L=4 based on empirical values, and the number of directions corresponding to each layer is 16, 16, 8, 8.

[0026] Step 3: Use an improved fusion strategy to perform structural fusion on the low-frequency components of the SAR image and the panchromatic optical image to obtain the fusion component F1.

[0027] 3.1 First, calculate the activity level of SAR image and panchromatic optical image respectively, the formula is as follows:

[0028]

[0029] where S∈{A,B} and W is a (2r+1)×(2r+1) weighted matrix of radius r, and for each element in W, its value is set to 2 2r-d , where d is the distance from its four neighbors to the center.

[0030] 3.2 Then the weighted and activity level metrics of the SAR image and the panchromatic optical image are calculated as follows:

[0031]

[0032] Where S∈{A,B}, W is the weight matrix defined in the same way as above, and WSEML is defined as follows:

[0033]

[0034] Compared with the original improved Laplacian matrix, EML considers the influence of diagonal coefficients and makes full use of neighboring information. According to the Euclidean distance, the weights of the two diagonal items are set to

[0035] 3.3 Finally, the low-frequency fusion rule of the SAR image and the panchromatic optical image is defined as the product of WLE and WSEML, so the low-frequency fusion strategy is calculated by the following rule:

[0036]

[0037] Step 4: Perform texture fusion on the high-frequency components of the SAR image and the panchromatic optical image to obtain the fusion component F2.

[0038] The absolute value fusion strategy is used to fusion the high frequency coefficients of SAR images. and panchromatic optical image high frequency coefficient Fusion is performed to obtain the fused high-frequency coefficients:

[0039]

[0040] Step 5: Use the exponentially weighted mean ratio operator (ROEWA) to extract edge structure information F in the SAR image sar And inject it into the high-frequency fusion component F2 through guided filtering (GF) to obtain a new fusion component F3;

[0041] 5.1 First, the edge gradient of the SAR image is calculated by the ROEWA operator as follows:

[0042]

[0043] 5.2 Then find the norm and sum of the two gradient components to obtain the edge intensity map of the image:

[0044]

[0045] 5.3 Finally, through guided filtering, the high-frequency fusion component F2 is used as the original image and the edge intensity map is used as the R guided image. Through the following formula:

[0046]

[0047] After filtering, the new fusion component F3 is obtained, where j∈W k , j is just a subscript within a certain window, W ij (I) is the weight of the weighted average associated only with guide map I.

[0048] Step 6: Perform NSST inverse transformation on the fused structural component F1 and texture component F3 to obtain the fusion result F.

[0049] In order to better test the performance of the above technical solution of the present invention, the above technical solution of the present invention is described in detail by comparative experiments (selecting the currently commonly used multi-scale decomposition method and the method of the present invention for performance comparison); the simulation experiment platform of the present invention adopts Intel (R) Core (TM) I5 CPU 2.20GHz, 8GB memory, PC running Windows 10, and the programming language is Matlab2016b. The experimental data selects two different types of land features, city and airport.

[0050] like Figure 2 Shown are pairs of tightly registered images selected from urban areas, which contain rich geometric structure information. Figure 2 Among them, (a) is the original full-color optical image, (b) is the original SAR image, (c) is the SAR geometric structure extraction diagram, (d) is the fusion method based on wavelet transform, (e) is the fusion method based on rolling guided filtering, and (f) is the fusion method proposed in the present invention.

[0051] like Figure 3 Shown are pairs of tightly registered images selected from the airport area, which contain rich information about flat areas. Figure 3 Among them, (a) is the original full-color optical image, (b) is the original SAR image, (c) is the SAR geometric structure extraction diagram, (d) is the fusion method based on wavelet transform, (e) is the fusion method based on rolling guided filtering, and (f) is the fusion method proposed in the present invention.

[0052] from Figure 2 (d)-2(f) shows that for this set of measured SAR images and panchromatic optical images, the fusion results obtained based on the wavelet transform method are Figure 2 (d) The overall contrast is low. Although the fusion method based on rolling guided filtering has the brightest overall brightness, it is poor in terms of structure and spectrum preservation. Figure 2 (e) There is obvious speckle noise. The fusion method proposed in this invention achieves better results. Figure 2 In (f), the speckle noise is significantly suppressed and the geometric structure is clearer. Figure 3 (d)-3(f) shows that for this set of measured SAR images and visible light images, the fusion results based on the wavelet transform method are Figure 3 (d) The overall brightness is low and the visible light spectrum is poorly preserved; the fusion result obtained by the fusion method based on rolling guided filtering Figure 3 The noise in the flat area in (e) is obvious; the fusion method proposed in the present invention achieves better results. The speckle noise in 3(f) in the fusion result is significantly suppressed, and the structure is clearer.

[0053] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. Any partial modification or replacement within the technical scope disclosed by the present invention by anyone familiar with the technology should be included in the scope of the present invention.

Claims

1. A multi-modal image fusion method in NSST domain with multi-scale structural feature enhancement , The following steps are included: Step 1: Input two images, one of which is a SAR image A and the other is a panchromatic optical image B; Step 2: Perform NSST decomposition on the two input images to obtain their decomposition bands respectively. and The symbol represents the high-frequency band coefficient of A in the decomposition layer l and direction k, while the symbol L A Represents the low-frequency band coefficient of A; similarly, the symbol represents the high-frequency band coefficient of B in the decomposition layer l and direction k, while the symbol L B represents the low-frequency band coefficient of B; Step 3: Use an improved Laplace fusion strategy to fuse the low-frequency components of the SAR image and the panchromatic optical image to obtain the fusion component F 1 , where two activity level metrics based on local energy and WLE-improved ML are designed in the fusion strategy to maintain low-frequency energy and improve detail extraction capabilities; 3.1 First, calculate the activity level of SAR image and panchromatic optical image respectively, the formula is as follows: where S∈{A,B} and W is a (2r+1)×(2r+1) weighted matrix of radius r, and for each element in W, its value is set to 2 2r-d , where d is the distance from its four neighbors to the center; 3.2 Then the weighted and activity level metrics of the SAR image and the panchromatic optical image are calculated as follows: Where S∈{A,B}, W is the weight matrix defined in the same way as above, and WSEML is defined as follows: 3.3 The low-frequency fusion rule of the final SAR image and the panchromatic optical image is defined as the product of WLE and WSEML, so the low-frequency fusion strategy is calculated by the following rule: Step 4: Fuse the high-frequency components of the SAR image and the panchromatic optical image to obtain the fusion component F 2 : The absolute value fusion strategy is used to fusion the high frequency coefficients of SAR images. and panchromatic optical image high frequency coefficient Fusion is performed to obtain the fused high-frequency coefficients: Step 5: Use the exponentially weighted mean ratio operator (ROEWA) to extract edge structure information F in the SAR image sar And inject the high-frequency fusion component F through guided filtering (GF) 2 In the equation, we get the new fusion component F 3 ; Step 6: The above fused structural component F 1 and texture component F 3 Perform NSST inverse transformation to obtain the fusion result F.